Running model inference on rented hardware

There are two ways to run a model: call somebody’s API and pay per token, or rent hardware and run it yourself. Both are reasonable, and which is cheaper depends almost entirely on how consistently busy the hardware would be.

The economics, simply

An API bills per unit of work. Ten requests cost ten units; zero requests cost nothing.

Rented hardware bills per hour regardless of use. The cost per request is the hourly rate divided by the requests served in that hour, which means it falls as volume rises and is effectively infinite when idle.

There is therefore a break-even volume. Below it the API is cheaper; above it, self-hosting is. Where that point sits depends on the model size, the hardware and the API’s pricing, and it moves as all three change.

The mistake is comparing the hourly rate against the API price without accounting for utilisation. A GPU instance busy ten percent of the time costs ten times its nominal per-request rate.

When self-hosting genuinely wins

Sustained, predictable volume. Consistent traffic keeps the hardware busy, which is the entire basis of the saving.

Batchable workloads. Offline processing of a queue can saturate the hardware completely, which is the best case for self-hosting by a wide margin.

Data that must not leave your infrastructure. Sometimes the constraint is not cost at all, and this is a perfectly good reason on its own.

A specific or fine-tuned model not offered by an API provider.

Latency requirements that a network round trip to an external service cannot meet.

Predictable billing as a requirement in itself, which per-hour pricing gives and per-token pricing does not.

When the API wins

Low or spiky volume. Paying for idle hardware to serve occasional requests is poor value, and no amount of optimisation fixes it.

Experimentation. Trying several models is trivial through APIs and a project on your own hardware.

Small teams without appetite for operating inference infrastructure, which is genuinely more involved than running a web service.

Needing the newest frontier models, which are generally not available to self-host.

Highly variable load where the peak is far above the average, since you would provision for the peak and pay for it continuously.

What running it actually involves

More than starting a process. In rough order of how often each is underestimated:

Fitting the model in memory. Quantisation to lower precision reduces memory substantially at some quality cost, and is usually necessary rather than optional.

Choosing a serving runtime. Purpose-built inference servers handle continuous batching, key-value caching and efficient scheduling. The difference in throughput between a naive loop and a proper serving runtime is large, frequently several times.

Batching. Serving one request at a time wastes most of the hardware. Continuous batching, where new requests join a batch already in progress, is what makes throughput acceptable.

Managing context length. Memory use grows with context, so a long-context request can consume the memory of many short ones. Without limits, one request can degrade service for everyone.

Cold starts. Loading a large model takes minutes. Autoscaling that assumes seconds does not work here, which pushes you toward keeping capacity warm and therefore back toward the utilisation problem.

Monitoring the right things. Tokens per second, time to first token, queue depth and memory headroom, rather than the CPU and memory metrics you would watch for a web service.

The arrangement that usually works best

Many teams end up with a hybrid, and it is a genuinely good answer rather than a compromise.

Self-host the steady baseline that keeps hardware reasonably busy, and burst to an API for traffic above it. You get the cost profile of owned capacity for the predictable majority and elasticity for the peaks, without provisioning for the peak.

The same split works by task. Route bulk, batchable, latency-tolerant work to your own hardware, and interactive or specialised requests to an API.

Working out the answer for your case

1. Measure your actual request volume and its distribution over a week. Peak-to-average ratio matters as much as the total.
2. Determine which model you need, and honestly whether a smaller one would do. Model size drives hardware, and hardware drives cost.
3. Benchmark throughput on a candidate instance with a proper serving runtime and realistic request lengths.
4. Calculate cost per request at your real utilisation, not at full utilisation.
5. Compare against the API price for the same volume.
6. Add the operational cost honestly: setup, monitoring, upgrades, incidents.

If self-hosting wins by a small margin, the API is probably still the better choice once step six is weighted properly. If it wins by a wide margin, the hardware is worth it.

Revisit the calculation periodically. API prices have fallen repeatedly, model efficiency keeps improving, and a decision made a year ago on solid reasoning may no longer hold.

For a view of how the providers differ on the points above, the cloud account catalogue lays out their respective strengths and configurations.

For the vendor’s own reference on the services involved here, see the AWS documentation.

Putting any of this into practice needs an account to work in. We stock AWS accounts in sixteen configurations, and the cloud account catalogue sets them beside nine other providers.

Questions people ask