Capacity buffers help you reduce Pod startup latency for your Google Kubernetes Engine (GKE) workloads by letting you proactively declare tiers of active or standby capacity buffers in your cluster. By declaring spare capacity ahead of time, you can achieve faster workload startups in a cost-efficient manner.
This document explains how capacity buffers work. To learn how to enable and use capacity buffers, see Configure capacity buffers.
When to use capacity buffers
Use capacity buffers for applications that are sensitive to startup latency and need to scale rapidly. When you experience sudden increases in traffic, an active buffer provides pre-provisioned capacity that's designed for low-latency scaling. When you experience a sustained increase in traffic, a standby buffer provides Pod scheduling at a more affordable cost than pre-provisioning.
Capacity buffers provide the following benefits:
- Minimize scaling latency: active buffers provide running nodes, which help minimize latency. Standby buffers resume quickly, providing faster capacity availability than fresh nodes at a lower cost compared to active buffers.
- Cost-efficient over-provisioning: capacity buffers help you maintain a safety net. For large-scale workloads, this approach is often more cost-efficient than other over-provisioning methods, such as lowering horizontal Pod autoscaler (HPA) utilization targets, which can increase idle capacity linearly as your cluster grows.
- Meet workload requirements: you have full control over your capacity buffer configuration. Your options include incorporating custom daemonsets to preload images, tuning startup time, and controlling buffer sizes to fit your needs.
We recommend capacity buffers for latency-sensitive workloads that require rapid scale-up, such as AI agents, AI inference, retail applications during sales events, or game servers during peak player activity.
How capacity buffers work
Implement a capacity buffer by using a Kubernetes CapacityBuffer custom resource to define a buffer of spare capacity. The GKE cluster autoscaler monitors CapacityBuffer resources and treats them as pending demand to help ensure that spare capacity is available. If your cluster doesn't have enough capacity to satisfy the resource requests defined in the buffer, the cluster autoscaler provisions additional nodes.