> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getlilac.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Operator cluster monitoring

> Monitor Kubernetes operator connectivity, workload activity, and cluster health through the Lilac dashboard and Kubernetes tools.

Monitor your cluster's connection status and workloads using the Lilac dashboard and Kubernetes tools.

## Dashboard

The **Clusters** section of the [Lilac console](https://console.getlilac.com) shows:

### Cluster Status

| Status        | Meaning                                                                    |
| ------------- | -------------------------------------------------------------------------- |
| **Connected** | Operator is syncing normally with the control plane                        |
| **Degraded**  | Sync has failed recently — operator will retry on next cycle               |
| **Draining**  | Disconnected for 10+ minutes — inference pods are being gracefully removed |

### Workload Activity

* Desired vs. reported workloads per pool
* Workload details: GPU count, pod phase, ready status, restart counts
* Draining workload counts
* Model assignments (which models are running on which nodes)

### GPU Allocation

* Per-pool breakdown of tenant vs. Lilac GPU usage
* Node details (GPU product, total GPUs)
* Last sync timestamps and lease expiration

<Info>
  Detailed usage statistics (tokens processed, revenue earned) are sent in your monthly report. See [Revenue & Payouts](/suppliers/revenue) for details.
</Info>

## Kubernetes Monitoring

### Check Pool Status

```bash theme={null}
kubectl get gpupool -n lilac-system
```

### View Operator Logs

```bash theme={null}
kubectl logs -n lilac-system deploy/lilac-gpu-operator -f
```

Key log events to watch:

| Event                           | Meaning                                 |
| ------------------------------- | --------------------------------------- |
| `control plane sync successful` | Normal sync completed                   |
| `workload created`              | New inference pod deployed              |
| `preemption triggered`          | GPUs being reclaimed for your workloads |
| `workload drained`              | Inference pod gracefully removed        |
| `sync failed`                   | Control plane unreachable — will retry  |

### View Running Inference Pods

```bash theme={null}
kubectl get pods -n lilac-system -l app.kubernetes.io/managed-by=lilac
```

### Kubernetes Events

The operator emits Kubernetes events for key state transitions:

| Event                      | Description                                       |
| -------------------------- | ------------------------------------------------- |
| `PoolCleanedUp`            | All managed workloads deleted from pool           |
| `ControlPlaneDegraded`     | Control plane stopped responding                  |
| `ControlPlaneDisconnected` | Disconnect timeout elapsed, draining workloads    |
| `WorkloadPreempted`        | Workload evicted after grace period               |
| `WorkloadDraining`         | Draining began (includes reason and grace period) |

View events:

```bash theme={null}
kubectl get events -n lilac-system --sort-by='.lastTimestamp'
```