The Infrastructure Cost of Running Frontier Models vs. Commercial Cloud Hosting Options for Developers

Most developers will never train a frontier model. The realistic decisions are: call a hosted model API, rent GPUs to serve an open-weight model, or fine-tune and then serve. This guide breaks down the real infrastructure cost drivers behind those choices and compares hosting options using a repeatable framework rather than a single headline price.

Three physical constraints drive cost:

For training, a common rule of thumb for mixed-precision training with the Adam optimizer is around 16 bytes per parameter of state, so a 70B model needs on the order of 1.1 TB of memory before activations. This is why full training is a cluster-scale problem while parameter-efficient fine-tuning (LoRA/QLoRA) is a single-node problem.

Strengths: compliance certifications, integrated networking and storage, enterprise support, and committed-use discounts. Weaknesses: highest on-demand prices, 8-GPU minimums on flagship instances, and egress fees commonly in the range of roughly $0.08 to $0.12 per GB leaving the cloud.

Strengths: usually lower per-GPU prices, faster access to new hardware, and in some cases free or low egress. Weaknesses: fewer managed services, more variation in reliability and compliance posture, and some require node-level or contract-level commitments.

Strengths: lowest prices and single-GPU rental. Weaknesses: variable hardware quality and uptime, weaker isolation guarantees, and more operational work. Best for experimentation and fault-tolerant batch jobs.

Why the spread? Hyperscaler figures often divide an 8-GPU node price by eight, include CPU, RAM and storage in the bundle, and differ by region. Some sources report mid-2025 price cuts on AWS H100 instances, while other sources reported H100 contract prices rising in late 2025 and early 2026 amid supply constraints. Treat any single number as a snapshot.

Discount levers reported by trackers: three-year reservations cutting roughly 50 to 60 percent off H100 on-demand pricing at major hyperscalers, and spot or interruptible capacity typically 40 to 65 percent below on-demand, with the trade-off that jobs can be interrupted.

The list price is not your price. Use:

Effective GPU cost = (hourly price ÷ utilization) + data movement + engineering overhead

Worked examples (illustrative arithmetic):

A key insight: utilization usually matters more than the sticker price. A cheap GPU that sits idle 70 percent of the time is more expensive than a pricier one that stays busy.

Self-hosting wins when your sustained token throughput times the GPU's cost-per-token at high utilization is lower than the API price for comparable quality. Estimate it like this:

If your traffic is bursty, the API almost always wins because you avoid paying for idle GPUs. If you have steady, predictable load and a model that meets your quality bar, self-hosting can win.

The cheapest option depends less on any provider's hourly rate and more on your utilization, your data movement and your tolerance for operational work. Use hosted APIs while demand is uncertain, move to specialist or reserved GPU capacity when traffic is steady and measurable, and always compute cost per million tokens from your own benchmarks rather than from list prices.

Pricing information in this guide reflects publicly reported ranges from third-party trackers and provider pages in 2026, which vary and change often. Verify current rates before making purchasing decisions. This is general information, not financial advice.