> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getlilac.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Cluster Monitoring

> Monitor your GPU cluster connection status, workload activity, and operator health using the Lilac dashboard and Kubernetes debugging tools.

Monitor your cluster's connection status and workloads using the Lilac dashboard and Kubernetes tools.

## Dashboard

The **Clusters** section of the [Lilac console](https://console.getlilac.com) shows:

### Cluster Status

| Status        | Meaning                                                                    |
| ------------- | -------------------------------------------------------------------------- |
| **Connected** | Operator is syncing normally with the control plane                        |
| **Degraded**  | Sync has failed recently — operator will retry on next cycle               |
| **Draining**  | Disconnected for 10+ minutes — inference pods are being gracefully removed |

### Workload Activity

* Desired vs. reported workloads per pool
* Workload details: GPU count, pod phase, ready status, restart counts
* Draining workload counts
* Model assignments (which models are running on which nodes)

### GPU Allocation

* Per-pool breakdown of tenant vs. Lilac GPU usage
* Node details (GPU product, total GPUs)
* Last sync timestamps and lease expiration

<Info>
  Detailed usage statistics (tokens processed, revenue earned) are sent in your monthly report. See [Revenue & Payouts](/suppliers/revenue) for details.
</Info>

## Kubernetes Monitoring

### Check Pool Status

```bash theme={null}
kubectl get gpupool -n lilac-system
```

### View Operator Logs

```bash theme={null}
kubectl logs -n lilac-system deploy/lilac-gpu-operator -f
```

Key log events to watch:

| Event                           | Meaning                                 |
| ------------------------------- | --------------------------------------- |
| `control plane sync successful` | Normal sync completed                   |
| `workload created`              | New inference pod deployed              |
| `preemption triggered`          | GPUs being reclaimed for your workloads |
| `workload drained`              | Inference pod gracefully removed        |
| `sync failed`                   | Control plane unreachable — will retry  |

### View Running Inference Pods

```bash theme={null}
kubectl get pods -n lilac-system -l app.kubernetes.io/managed-by=lilac
```

### Kubernetes Events

The operator emits Kubernetes events for key state transitions:

| Event                      | Description                                       |
| -------------------------- | ------------------------------------------------- |
| `PoolCleanedUp`            | All managed workloads deleted from pool           |
| `ControlPlaneDegraded`     | Control plane stopped responding                  |
| `ControlPlaneDisconnected` | Disconnect timeout elapsed, draining workloads    |
| `WorkloadPreempted`        | Workload evicted after grace period               |
| `WorkloadDraining`         | Draining began (includes reason and grace period) |

View events:

```bash theme={null}
kubectl get events -n lilac-system --sort-by='.lastTimestamp'
```