> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getlilac.com/llms.txt
> Use this file to discover all available pages before exploring further.

# GPU Preemption

> Learn how the Lilac operator gracefully reclaims GPUs when your workloads need them back, using LIFO eviction and configurable grace periods.

Your workloads always come first. When your cluster needs GPUs that Lilac is currently using, the operator automatically and gracefully reclaims them.

## How Preemption Works

When the operator detects that your workloads need GPUs, it:

1. **Selects inference pods to evict** — using last-in-first-out (LIFO) ordering, the most recently created Lilac pods are evicted first
2. **Initiates graceful drain** — the selected pods receive a shutdown signal and are given the configured `gracePeriod` to finish in-flight requests
3. **Force-deletes if needed** — pods that haven't terminated after the grace period are force-deleted
4. **Reports to control plane** — the operator notifies Lilac so traffic is rerouted to other available GPUs across the network

This entire process typically completes in under 60 seconds.

## Preemption Triggers

| Trigger                | Description                                                          |
| ---------------------- | -------------------------------------------------------------------- |
| **Tenant reclaim**     | Your pod needs a GPU that Lilac is currently using                   |
| **Schedule inactive**  | The availability window has closed                                   |
| **Inference disabled** | You set `workloads.inference: false` on the pool                     |
| **Disconnected**       | The operator lost contact with the control plane for over 10 minutes |
| **Scale down**         | The control plane decided to reduce workloads on your cluster        |
| **Unhealthy**          | The health tracker detected issues with a workload pod               |

## Grace Period

The `gracePeriod` in your GPU pool config controls how long inference pods have to finish in-flight requests:

```yaml theme={null}
preemption:
  gracePeriod: 30s   # Default: 30 seconds
  priority: tenant   # Your workloads always win
```

<Tip>
  30 seconds is usually plenty for inference requests to complete. Increase this if you serve very long completions (e.g., large max\_tokens values).
</Tip>

## What Happens to In-Flight Requests

When an inference pod is preempted:

* **Completed requests** are returned normally
* **Streaming requests** receive a clean stream termination
* **New requests** are automatically routed to other GPUs in the Lilac network — users experience no downtime

## Zero Impact on Your Workloads

The operator never modifies, evicts, or interferes with your pods. It only manages pods it created (labeled as Lilac inference workloads). Your workload scheduling, resource requests, and priority classes are untouched.