Plan and create your AI infrastructure

This document summarizes how you create a cluster for your AI workloads on AI Hypercomputer. Specifically, this document guides you through the process and choices to make when you start a cluster. As an alternative to manual evaluation, you can prompt Gemini in the Google Cloud console to compare consumption options for your workload and budget. For more information, see Design AI infrastructure with Compute Advisor.

This document assumes that you are familiar with commonly used terminology for AI and ML workloads, such as model training and inference. It also assumes that you have identified the specific AI workload for which you need to determine the optimal machine type and capacity needs for your deployment—for example, foundation model pre-training, fine-tuning, or inference.

Start a cluster

Starting a cluster involves the following steps:

  1. Determine your workload and choose a machine type
  2. Choose a consumption option and obtain capacity
  3. Choose a deployment option
  4. Choose an orchestrator
  5. Choose the operating system and cluster image
  6. Create your cluster
  7. Provision storage for your workload

Determine your workload and choose a machine type

Select a machine type for your AI workload. AI Hypercomputer supports cluster creation for both clustered GPUs and general GPUs.

To help you choose, determine your workload requirements and match them with the recommended machine type and GPU type:

  • Clustered GPUs: best for large-scale, high-performance workloads such as pre-training foundation models, fine-tuning large models, and inference across multiple hosts.
  • General GPUs: best for mainstream inference and serving, retrieval-augmented generation (RAG), and cost-effective small-to-medium model training and fine-tuning.

To match your workload to the recommended machine type, use this table:

GPU type Workload or use case Recommended machine types
Clustered GPU Pre-training foundation models and inference across multiple hosts A4X Max (NVIDIA GB300)*, A4X (NVIDIA GB200)*
Large model training, fine-tuning, and inference A4 (NVIDIA B200), A3 Ultra (NVIDIA H200 141GB)
Mainstream model inference and fine-tuning A3 Mega (NVIDIA H100 80GB), A3 High (NVIDIA H100 80GB)
General GPU High-throughput edge serving and inference A3 Edge (NVIDIA H100 80GB)
High-performance single-node serving and small-scale fine-tuning A2 (NVIDIA A100)
Cost-optimized entry-level inference G4 (NVIDIA RTX PRO 6000), N1 (NVIDIA T4 or V100)
Mainstream inference, RAG, and small-to-medium model training G2 (NVIDIA L4)

For detailed information about each machine series, see About GPU accelerators.

Choose a consumption option and obtain capacity

Select a consumption option for your GPU resources based on your chosen machine type and whether you use General GPUs or Clustered GPUs.

Consumption options for general GPUs

Consumption option Available for Best for How to request
On-demand All general GPUs. Workloads that don't require assured capacity. Create a compute instance or cluster and specify the standard provisioning model. For instructions, see Create VM instances.

Tip: To increase your chances of obtaining general GPU capacity, use Flex-start or Spot.
Standard reservations and standard future reservations All general GPUs. Workloads that require assured capacity either immediately (standard reservations) or for a specific future date (standard future reservations). Create an on-demand reservation or a future reservation request. For instructions, see Reserve capacity.
Flex-start All GPU machine types except for A4X Max and A4X. Workloads that require short-lived, dense clusters that last up to seven days. Offers dense resource allocation and up to a 53% discount on supported machine types; otherwise, standard on-demand rates apply. Create a request by using Compute Engine, Cluster Director, Cluster Toolkit, or GKE. Resources are provisioned as soon as they become available (start time isn't immediate). For instructions, see Obtain capacity.
Spot All GPU machine types except A4X Max and A4X. Workloads that are fault-tolerant, batch, or short-lived. Offers the deepest discount (between 61% and 90%), but compute resources can be preempted at any time. Create instances or node pools immediately by using the Spot provisioning model. For instructions, see Obtain capacity.

Consumption options for clustered GPUs

Consumption option Available for Best for How to request
Future reservations in AI Hypercomputer All clustered GPUs and A3 Edge machine types. Workloads that require stability for an extended period of time, such as pre-training foundation models or foundation model inference across multiple hosts. Offers dense resource allocation and up to a 53% discount for vCPUs and GPUs.