GPU instances: what you are actually paying for

GPU instances cost many times what a comparable CPU instance costs, and the gap is justified for the right workload. It is not justified for a GPU that spends most of its life waiting for data.

The dominant factor in whether a GPU instance is good value is not which GPU you chose. It is what fraction of the time it is actually computing.

What you are renting

A GPU instance bundles several things, and it helps to see them separately.

The accelerator itself, with a fixed amount of very fast on-board memory. That memory capacity is frequently the binding constraint rather than the compute throughput, because a model that does not fit cannot run at all.

Host CPU and system memory, used to prepare and feed data. Insufficient host resources starve the GPU.

Storage throughput, which determines how fast training data reaches the host.

Network bandwidth, which matters enormously for multi-GPU and multi-node training, and barely at all for single-GPU inference.

Understanding which of these binds for your workload is the whole optimisation problem.

Memory is usually the first constraint

For model training and for inference on large models, the question “will it fit” comes before any question about speed.

Memory is consumed by the model weights, the optimiser state during training, the activations for the batch in flight, and various overheads. Training typically needs several times the memory that inference of the same model needs, because of gradients and optimiser state.

When it does not fit, the options are a smaller batch size, gradient accumulation to simulate a larger batch, mixed or lower precision, gradient checkpointing to trade compute for memory, model parallelism across several GPUs, or simply a GPU with more memory.

Reduced precision is usually the first thing to try, because modern accelerators are faster at lower precision as well as more memory-efficient, so it frequently improves both constraints at once.

Why utilisation is usually poor

The most common finding on a first inspection of a GPU workload is that the accelerator is idle a large fraction of the time. The usual causes:

Data loading is too slow. The GPU finishes a batch and waits. Fixed by more data loader workers, prefetching, storing data in a format that decodes quickly, and keeping the dataset on fast local storage rather than reading it over the network.

Preprocessing runs on the CPU. Image decoding and augmentation on an under-provisioned host CPU becomes the bottleneck. Either provision more host CPU or move preprocessing to the GPU.

The batch is too small. Small batches underuse the parallelism the hardware offers. Increase until memory limits you.

Frequent synchronisation. Copying results back to the host every step, or logging metrics that force a synchronisation, serialises what should overlap.

The workload is not GPU-shaped at all. Some work genuinely does not parallelise well, and a GPU is the wrong tool.

Measure utilisation before optimising anything else. A GPU at thirty percent utilisation is a workload where the fix is data pipeline work, not a bigger accelerator.

Getting the cost down

Fix utilisation first. Doubling utilisation halves the effective cost, and it is usually cheaper than any other lever.

Use interruptible capacity for training. Training with regular checkpointing is close to the ideal interruptible workload, and the discount is large. Without checkpointing it is a bad idea, because losing a long run to a preemption is worse than the saving.

Right-size the accelerator. The largest available GPU is not always fastest per unit of cost. If your model fits comfortably on a smaller one, the smaller one is often better value.

Separate training from inference. They have different shapes. Training wants large, possibly interruptible capacity in bursts. Inference wants smaller, always-available capacity sized to request volume. Running inference on training-sized hardware is a common and expensive mistake.

Batch inference requests where latency allows. Serving one request at a time on a GPU wastes most of its parallelism.

Shut it down. An idle GPU instance left running overnight is among the most expensive forms of forgetfulness available in cloud computing. Automate the shutdown.

Availability is a real constraint

Popular GPU types are frequently unavailable in popular regions, and this affects planning in ways CPU capacity does not.

Practical responses: be flexible about region and about accelerator type, use capacity reservations if your work is time-sensitive, and design so that a training run can start on whatever is available rather than requiring one specific configuration.

A short evaluation sequence

1. Establish that the workload needs a GPU at all. Some inference workloads run acceptably on CPU, especially with quantisation, and CPU capacity is far easier to obtain.
2. Determine the memory requirement, because that decides the shortlist.
3. Run on the smallest accelerator that fits and measure utilisation.
4. Fix the data pipeline until utilisation is high.
5. Only then consider a larger accelerator, because until utilisation is high you would be buying capacity you are not using.

Following that order routinely halves the cost of a first GPU deployment, and it does so before spending anything.

For a view of how the providers differ on the points above, the cloud account catalogue lays out their respective strengths and configurations.

For the vendor’s own reference on the services involved here, see the Google Cloud documentation.

If you want to work through this yourself, our Google Cloud accounts come in six configurations with every price shown, and the cloud account catalogue compares them against the other nine platforms.

Questions people ask