GPU inference cost per million tokens: an H100 example

What to take away

• Divide the full cost of billable capacity by the output tokens your service actually delivers. An hourly GPU rate alone is not a cost per token.
• Match the model, checkpoint, precision, GPU configuration, and input/output lengths when using a published benchmark.
• Treat a maximum-load benchmark calculation as a capacity scenario. Use billed hours and delivered tokens for an operating-cost estimate.
• Check response times as well as throughput before choosing the lowest calculated cost.

A GPU rental page gives you a price per hour. A language-model API often gives you a price per million tokens. It is tempting to turn one into the other by finding a tokens-per-second benchmark and doing the division. That is a useful first calculation, but it answers a narrow question: what would output tokens cost if the rented machine sustained that measured rate for every billable second?

Your service will have its own request lengths, traffic pattern, response-time target, and idle periods. This guide shows both calculations: a benchmark-based capacity scenario and a bill-based operating scenario. It uses one model and one GPU variant so the assumptions remain visible. For GPU sizing, start with our memory guide; for the choice between hourly and monthly rental, use our rental-term guide.

Fix the model, hardware, and price

The worked example uses NVIDIA's Llama 3.1 8B Instruct FP8 checkpoint on one H100 SXM 80 GB, with the TensorRT-LLM PyTorch backend. NVIDIA measured aggregate output throughput on a DGX H100. The published test sends synthetic requests to a local client without delay, creating a maximum-load scenario. It is a publisher measurement, not a benchmark run by Noach Ark. The source describes the v0.21 benchmarking workflow and the workload lengths for each row. NVIDIA's pinned performance overview

For the rental input, Hyperstack's public pricing page listed an on-demand H100 SXM at $3.20 per GPU-hour when checked on September 25, 2026. This example assumes one billable GPU at that advertised rate, continuously allocated during each illustrated hour. It excludes any separately billed storage, public IP, taxes, or other charges. The price page does not verify that a rented Hyperstack GPU reproduces NVIDIA's DGX benchmark result.

The hardware label matters. An H100 PCIe, H100 SXM, H100 NVLink offer, and an unspecified “H100 80 GB” listing should not share a benchmark row merely because their names contain H100. NVIDIA says the variants and host systems can differ in features that affect performance. For a rental comparison, also check the minimum GPU count and the full instance price, not only a normalized per-GPU price. NVIDIA's hardware and method notes, Noach Ark H100 SXM offers

Calculate a benchmark-based cost

For an hourly rental and an aggregate output throughput measurement, the arithmetic is:

Calculated dollars per million output tokens = full instance price per hour × 1,000,000 ÷ (output tokens per second × 3,600).

NVIDIA reports the following three workloads for the same model, precision, engine, GPU variant, and one-GPU configuration. Input and output lengths are separate because processing a long prompt takes work even when it produces only a short answer. The final column combines NVIDIA's measured rate with the September 25 Hyperstack price; it is our calculation, not a price offered by either company. NVIDIA benchmark rows, Hyperstack price

Calculated from one advertised $3.20 per-GPU-hour rate and NVIDIA's offline maximum-load measurements. These are output-token-only capacity scenarios, excluding extras and any time spent below the published throughput. The benchmark was not performed on the advertised rental.

For the middle row, the calculation is $3.20 × 1,000,000 ÷ (14,991.62 × 3,600) = $0.0593 per million output tokens. The first and third rows have similar total sequence lengths, but the prompt-heavy row yields far fewer output tokens per second. At the same rental rate, its calculated output-token cost is about 6.5 times as high. A single “H100 cost per million tokens” therefore hides the workload that produced it.

This metric charges the complete GPU hour to the output tokens. It includes the GPU time spent processing prompts, but it does not mean input tokens were free or that the figure is directly equivalent to an API's output-token price. An API may bill input and output separately and provide a different model, reliability level, and serving arrangement.

Replace benchmark capacity with delivered traffic

Maximum-load throughput measures what one test setup delivered under continuous demand. It does not say how many tokens your customers will request in an ordinary hour, whether your serving stack will reproduce the result, or whether users will accept the response time at that load. NVIDIA's reporting distinguishes total output throughput from per-user output speed and time to first token. NVIDIA measurement and reporting example

For a running service, use the bill and the observed output count:

Operating dollars per million output tokens = total billable service cost × 1,000,000 ÷ output tokens actually delivered.

Suppose the one-GPU service stays allocated for every hour of a 30-day, 720-hour month at $3.20/hour and delivers 100 million output tokens. The GPU rental subtotal is $2,304; divided by 100 million output tokens, that is $23.04 per million output tokens. This is a hypothetical traffic scenario, not a measured Noach Ark deployment or a Hyperstack quote. Storage and other applicable charges would raise the numerator. If the service can release the instance for part of the month, use the hours actually billed under the provider's terms.

There is another way to inspect the gap. If production delivers one quarter of the middle benchmark row's output rate averaged across all billable seconds, the arithmetic becomes $0.0593 ÷ 0.25 = about $0.237 per million output tokens, before extras. That quarter is an output-rate ratio that includes idle time, workload differences, and serving limits; it is not a GPU utilization reading and it is not a forecast of real traffic.

The monthly example and the quarter-rate example illustrate different volumes. Neither predicts the other's result. Once you have invoices and token counts, the bill-based calculation is the one to use for your own operating costs.

Test the response time you need

A service may improve total throughput by serving many requests together while making each person wait longer. Before treating a low calculated cost as a viable deployment, measure the delay to the first token, the pace of subsequent tokens, completed requests per second, and the share of requests that fail or exceed your response-time target.

vLLM's serving benchmark guide shows how to test a specified request rate and concurrency and reports output throughput alongside time to first token and time per output token. It also warns that repeated prompts can benefit from a warm prefix cache and inflate throughput. Use prompts resembling your own traffic, record whether caching is enabled, and keep the measurement method the same across candidates.

For every result, preserve the exact checkpoint, precision, engine and version, GPU variant and count, input/output lengths, request rate, concurrency, and billing terms. Compare only configurations that meet your application’s response-time requirement. A model that fits in memory still needs a serving test; our GPU memory guide explains why context and concurrency affect that first screen.

Turn the estimate into a rental decision

Use this sequence when comparing H100 SXM offers or another GPU shortlist:

1. Define one model checkpoint and an input/output length mix drawn from your expected requests.
2. Identify the full billable configuration, minimum GPU count, rental term, and separately charged services.
3. Treat published throughput as a dated reference. Test the candidate under realistic request rates and a response-time limit.
4. Divide the expected or observed total bill by the expected or observed delivered output tokens. Label estimates and measurements separately.
5. Recheck the provider's current offer before spending; the September 25 price used here is a dated observation.

The benchmark calculation tells you how a fully loaded test configuration converts rental time into output. The bill-based calculation tells you what your own service pays for the output it actually delivers. Keep both attached to their assumptions, and the price per token becomes a useful decision tool rather than a misleading headline.

Method and sources: Researched September 25, 2026. Benchmark measurements come from NVIDIA's TensorRT-LLM performance overview pinned to commit 0c9430e; the rental input comes from Hyperstack's public pricing page. Dollar-per-token results and the 30-day scenario are arithmetic examples. Noach Ark did not run the benchmarks or verify the advertised rental at checkout. Prepared with AI-assisted research and writing; the conceptual hero illustration is AI-generated. Read Noach Ark's methodology for how observations and estimates should be interpreted.