How to benchmark a rented GPU for LLM inference

What to take away

• Compare complete serving configurations using the same model revision, precision, workload, and cache policy.
• Measure first-token and complete-response latency alongside aggregate output throughput. A high tokens-per-second total does not establish a responsive service.
• Start with a controlled workload, then repeat with application prompts, natural stopping, and representative traffic.
• Record errors, incomplete requests, queue behavior, and sample counts. A fast successful subset can hide overload.
• Compare cost at a load that meets your response-time and reliability targets, then confirm it with a longer run.

A rental listing tells you the GPU name and the hourly price. Your application needs a different answer: how much work can this particular machine serve while responses remain usable?

The answer depends on the model, request lengths, serving software, traffic pattern, and the rest of the instance. A benchmark is useful when it preserves those conditions and measures the experience your application requires. This guide gives you a repeatable procedure for collecting that evidence.

The commands below are templates checked against official documentation and implementation source on September 25, 2026. Noach Ark has not executed them on a GPU or measured a rental's performance. The example rates and durations are exercise inputs, not capacity claims. Use the procedure to produce your own measurements.

Define the comparison before you start

Write down the task first. A short chat response, a long document summary, and a coding response may have very different input and output lengths. Choose a model and a representative mix of requests, then set acceptable first-response time, complete-response time, and error rate. These are your service targets; the benchmark should test them.

For a controlled comparison, hold the model checkpoint, tokenizer, precision, endpoint format, prompt/output lengths, and cache policy constant. If you change precision or the model, record that as a different configuration and evaluate answer quality as well as performance.

Record the exact GPU variant and count, available memory, dedicated or fractional allocation, host CPU and RAM, region, driver/runtime, server arguments, and price of the complete instance. Keep the client location in the record too. A client on the GPU host measures a different network path from an application calling across regions.

Check that the configuration fits before loading it. Our GPU memory guide explains why weights alone do not describe a serving workload's memory needs. After startup, send ordinary requests and inspect their answers. Performance measurements do not establish that the model is suitable for the task.

Choose metrics that describe the service

Collect response times and total output throughput together. Throughput describes work delivered across requests; latency describes how long an individual request takes. Keep output-token throughput separate from a total that adds input and output tokens. GuideLLM metric definitions

Report units explicitly: milliseconds or seconds for latency, requests/s for request rate, and output tok/s for throughput. A p95 latency describes the point at or below which approximately 95% of the relevant observations fall. Preserve the sample count behind it and the tool's measurement window.

Token-latency labels need extra care. In vLLM 0.30.0's benchmark, time per output token, or TPOT, is calculated as end-to-end latency minus TTFT, divided by output tokens minus one. In GuideLLM 0.7.4, TPOT uses the interval from request start to the last token divided by output-token count, so it includes the initial wait. Those values answer different questions. vLLM calculation, GuideLLM calculation

GuideLLM's inter-token latency, or ITL, is a per-request average between the first and last token; the benchmark combines those averages with weighting. It is not a distribution of every individual pause in a stream. vLLM's benchmark instead aggregates gaps between streamed outputs, which may themselves contain multiple tokens. Keep the tool and version attached to these metrics rather than merging them into one comparison column. GuideLLM aggregation, vLLM streaming gaps

For the rental decision, begin with clearly labeled TTFT, complete-request latency, request counts, and aggregate output throughput. Add a streaming-smoothness measure when the application requires it and explain exactly what it measures.

Start a pinned server

This example uses vLLM for serving and GuideLLM for load generation. vLLM's benchmark guide recommends GuideLLM for production server benchmarking; its built-in benchmark examples primarily support feature evaluation and regression testing. vLLM benchmark guide

The source review uses these pins:

The model is an illustrative text-generation checkpoint, not a recommendation that it fits every GPU or is the right model for every application. Preserve the installed package versions and dependency lock or container digest as well as the model revision. vLLM release, GuideLLM release, pinned model card

Use a compatible Linux GPU host with the selected software already installed. Match the driver to the actual vLLM artifact: at the review date, the v0.30.0 release described a CUDA 13.0 default while its versioned installation page still described CUDA 12.9 defaults. Verify the artifact you install rather than assuming those descriptions are interchangeable. Installation documentation

In the server terminal, run:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --revision a09a35458c702b33eeacc393d103063234e8bc28 \
  --tokenizer-revision a09a35458c702b33eeacc393d103063234e8bc28 \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 4096 \
  --no-enable-prefix-caching \
  --host 127.0.0.1 \
  --port 8000

This configuration uses one GPU, BF16 precision, a 4,096-token context limit, and disabled prefix caching. These are explicit experiment settings, not a memory-fit guarantee. Pinning the model and tokenizer prevents a later revision from silently changing the comparison. vLLM server arguments

The server listens on loopback. Run the client on the same host for this exercise; the measured path includes local HTTP and server overhead. Confirm that startup finishes and an ordinary request produces a sensible response before generating load. Record startup separately if it matters to the application.

Run a sequential baseline

In a separate client environment with GuideLLM 0.7.4, run:

guidellm run \
  --backend '{"kind":"openai_http","target":"http://127.0.0.1:8000","model":"Qwen/Qwen2.5-7B-Instruct","request_format":"/v1/chat/completions","stream":true,"timeout":60,"extras":{"body":{"temperature":0}}}' \
  --tokenizer '{"kind":"huggingface_auto","model":"Qwen/Qwen2.5-7B-Instruct","load_kwargs":{"revision":"a09a35458c702b33eeacc393d103063234e8bc28"}}' \
  --data kind=synthetic_text,prompt_tokens=1024,output_tokens=256 \
  --profile kind=synchronous,warmup=0.1,cooldown=0.1 \
  --constraint kind=max_duration,seconds=120 \
  --seed kind=static,value=42 \
  --output kind=json,path=results/baseline.json

The synchronous profile sends requests one at a time. It establishes a useful starting point before introducing concurrent demand. The duration is a short exploratory budget, not evidence of sustained performance. The seed records the synthetic workload and scheduling randomness. GuideLLM CLI and synchronous profile

The backend selects streaming chat completions and a 60-second response-read timeout. The tokenizer uses the same revision as the server. Preserve these choices when comparing candidates. HTTP backend configuration, tokenizer revision support

The synthetic prompt setting is 1,024 tokens; inspect the observed input count after chat formatting. Supplying 256 output tokens causes this GuideLLM HTTP handler to set an output cap and ignore the model's end-of-sequence signal. That deliberately creates fixed-length generation. A naturally stopping chat response is a different workload. Synthetic dataset options, fixed-length request construction

Here, warmup and cooldown are each configured as 10% of the duration budget. Preserve that configuration in the result record: GuideLLM's request-level and token-event metrics apply the measurement window differently. Their sample counts need not match. Phase configuration, measurement windows

Save the raw JSON. It preserves more detail than a CSV summary and lets you revisit failures and measurement scope instead of relying on one terminal screenshot. GuideLLM output formats

Increase arriving work and inspect the queue

A fixed-concurrency test keeps a chosen number of requests in flight. An arrival-rate test instead schedules requests at a specified pace. If requests become slower, fixed concurrency can reduce the rate at which new work arrives; an arrival-rate test can reveal work accumulating. Choose the demand model that represents the application. GuideLLM scheduling profiles

To explore increasing offered rates, reuse the baseline command with these three changes:

1. Replace the profile line with the Poisson profile below.
2. Keep the duration constraint and add the over-saturation constraint below.
3. Change the output path to results/rate-stages.json so you retain the baseline.

--profile '{"kind":"poisson","rate":[1,2,4],"warmup":0.1,"cooldown":0.1}'

--constraint kind=over_saturation

These are replacement/additional arguments, not standalone shell commands. The Poisson profile schedules requests with randomized intervals around each target rate. The example asks for stages at 1, 2, and 4 requests/s; it does not predict that the GPU can sustain them. A Poisson process is also a traffic assumption, so test bursts or other patterns separately when they matter. Rate-profile implementation

The 120-second constraint applies to each strategy, rather than the entire multi-stage run. Keep each stage's actual duration, achieved rate, and stopping reason. Increase rates only as the earlier results justify it. Per-strategy constraints

GuideLLM's over-saturation detection looks for rising concurrency and TTFT. Its thresholds do not represent your application's response-time target. A run may miss that target before the detector stops it; a stopped overload run is useful evidence about the tested load. Over-saturation behavior

Inspect queues alongside client timings. vLLM exposes the vllm:num_requests_waiting gauge and latency histograms. Keep server telemetry and client timings identifiable because they measure different boundaries. Also check that the client itself generated the intended load: a concurrency limit or constrained client can suppress demand. vLLM production metrics

Repeat with the workload people will use

The controlled run makes comparisons easier. The application pass establishes whether the result survives different request lengths and behavior.

Use representative prompts, the application's endpoint and chat format, actual sampling settings, output limits, and natural stopping. GuideLLM supports local datasets and column mapping. Inspect the request body when using output-length annotations: they can force generation length, while backend extras can override those settings. Dataset configuration, backend extras

Test caching deliberately. Prefix caching reuses computation for shared prompt prefixes and benefits prompt processing rather than the generation of new tokens. A repeated synthetic dataset can therefore give a different result from mostly unique prompts. Keep a cache-disabled controlled baseline, then run a separate cache-enabled test with the prefix reuse your application actually has. Record whether each test starts with cold or warmed caches. vLLM prefix caching

Repeat important settings and retain every run. Three repeats are a practical starting point for this procedure, not a statistical guarantee. Report variability, investigate outliers, and collect enough observations for the percentiles you intend to publish. With only 100 observations, roughly one observation occupies the upper 1%; a p99 value alone provides little evidence about the tail.

Confirm a shortlisted configuration with a longer run at representative demand. Include traffic variation and the client route that matter to your service. Leave operating headroom rather than treating the last passing exploratory stage as a guaranteed production limit.

Keep a comparison record and calculate cost

Use this worksheet for each run. It contains fields to fill, not measured GPU results.

Compare configurations at a workload and load that meet the same targets. If a cheaper instance only passes at less demand than your application requires, its price does not resolve the decision. Use our hourly versus monthly rental guide to check how the billing arrangement affects the comparison.

For a continuously billed capacity scenario, calculate dollars per million output tokens as:

Full instance dollars/hour × 1,000,000 ÷ (measured output tok/s × 3,600).

This is a calculation based on sustaining the measured output rate throughout each billed hour. It does not account for idle periods by itself. For an operating service, calculate total bill × 1,000,000 ÷ delivered output tokens over the same period. Keep failed or unusable work from silently becoming useful delivered output. Our cost-per-million-tokens guide explains the distinction between a benchmark capacity scenario and an operating bill.

Choose the rental after the evidence supports the workload, response-time targets, reliability, and billing assumptions together. Keep the workload and result files so the next model change, software upgrade, or rental renewal can be evaluated with the same procedure.

Research and validation note

Documentation and source reviewed September 25, 2026, using vLLM 0.30.0 and GuideLLM 0.7.4. Prepared with AI-assisted research and writing. Noach Ark did not purchase a rental, run the GPU benchmarks, or validate the commands against installed CLIs. The templates require runtime verification on the selected host; no provider ranking or measured GPU performance is presented. The hero is a conceptual AI-generated illustration.