Gemma 4 26B A4B: When Is Self-Hosting Worth Testing?

For Gemma 4 26B A4B IT, an API bill around 1,344.00 USD per 30 days can fund the modeled candidate compute budget. At 4,096 billed input and 1,024 billed output tokens per request, that is about 2.1 million requests, or 2.2 billion output tokens plus their input. Near this spending line, investigate the candidate below. Moving traffic requires measured capacity, acceptable task quality and a complete operating budget.

Your Gemma 4 26B A4B API spending justifies investigating self-hosting when it approaches the live candidate budget below. The proposed experiment is the pinned original BF16 Instruct checkpoint on one H100 NVL 94 GB, compared with DeepInfra's exact FP8 endpoint. This model activates only part of its experts per token, but the full checkpoint still needs memory. Its documented one-GPU BF16 launch also uses much shorter conversations than the API accepts. Use the spending line to fund a test at your real context lengths and latency target; it does not establish GPU fit, capacity, savings or equivalence to the API.

Capacity unverified

This article provides a financial investigation screen. It has no qualified aggregate output-throughput measurement for the candidate. The traffic figure does not establish a reachable saving, supported concurrency, response time or equivalence to the API.

Start with the 32k BF16 example, then test your longest conversations

The official vLLM guide lists a 64 GB default BF16 minimum and shows this exact 26B MoE model on one A100/H100 with max-model-len 32768. Its full-feature launch, including image, thinking and tool support, instead caps length at 16384 and images at four per prompt. Those launch limits matter more to a migration test than a general claim that the model supports long context. The guide does not identify an H100 variant or the tested publisher weight revision.

There is a documentation discrepancy to preserve rather than silently resolve. The pinned publisher configuration allows 262,144 text positions, as does the exact API's recorded context limit. The guide's model metadata says 131,072, and its Blackwell quantization section describes 262,144 as exceeding the configuration. That statement does not match this pinned original configuration. It is not evidence that a one-H100 BF16 instance can serve full-length conversations.

The pinned model has 30 text layers: 25 use 1,024-token sliding attention and five use full attention. The local and global paths use different head dimensions and KV-head counts. Its 128 experts select eight per token. The 4B-active label therefore does not supply either a resident-memory figure or a uniform KV-cache estimate. No cache substitution or concurrent-conversation count is assigned here.

Replay short requests first, then your longest real conversation, including media and the required output allowance. Measure peak memory, queueing and completion latency as you raise context and concurrency separately. If the application depends on requests beyond the documented launch limits, passing the shorter test leaves that requirement unresolved. Keep those traffic segments separate when interpreting the spending screen.

Evidence: architecture source; serving source; architecture source.

Keep the assistant checkpoint out of the baseline experiment

The captured guide's MTP option names a second repository, google/gemma-4-26B-A4B-it-assistant, and recommends four speculative tokens. That is an additional model to load and configure, not four extra tokens of ordinary user demand. The guide warns that Gemma 4 MTP needs a nightly build rather than the stable latest image. Its minimum-version field is not a promise that every optional command works in that stable version.

For this investigation, begin with the original BF16 checkpoint without the assistant. Record the exact engine image, dependencies and launch arguments. If you later test MTP, pin the assistant revision too, then measure its extra memory, draft acceptance, complete-response latency and aggregate output under the same requests. An acceptance rate or a fast draft stage cannot replace a served-output capacity measurement. No assistant memory or speed improvement is included in this article's candidate.

The guide's full-context Blackwell examples use NVIDIA's separately named NVFP4 checkpoint with an assistant, different batch limits and hardware-specific settings. One uses a pinned nightly image; the other reported builds are internal nightlies. Those examples do not validate this original BF16 H100 NVL setup. Similarly, the guide's smaller FP8 and NVFP4 memory figures belong to different quantized checkpoints. The API's FP8 declaration does not identify which downloadable revision it serves.

Treat baseline BF16 and assistant-assisted decoding as separate experiments. Preserve task success and full output lengths in both. A configuration change that merely truncates or changes the accepted answer does not demonstrate that the API workload migrated.

Evidence: model source; model source; serving source.

Check the attention build before drawing a capacity conclusion

A September 2026 report in the vLLM project describes this exact model's H100 decode step as 48% slower at batch one in v0.30.0 than in v0.26.0. The reporter traced it to the FA4 attention path ignoring a KV-split setting, with a particularly large effect on the five global layers. The report supplies an FP8, text-only reproduction on an H100 80GB HBM3 at about 6k context. It does not measure our original BF16 H100 NVL candidate.

This is a dated diagnostic report, not a recommendation to run an old release or a statement that every later build retains the issue. Record the serving image and attention-library revision, and compare representative short and long requests when changing versions. Separate prefill, decode and complete-task latency in the diagnosis. The report's decode percentages and kernel timings cannot fill the missing aggregate-output reference or alter the financial threshold below.

Evidence: architecture source; serving source.

The candidate to test

Test 1 × H100 NVL (94 GB per GPU) with the pinned BF16 checkpoint. Serving-engine guidance: the captured official vLLM recipe lists a 64 GB minimum for this model and precision, with vLLM 0.25.0 or newer. Serving recipe, captured October 15, 2026 at 9:29 am EDT; the reviewed snapshot is fixed by its capture hash, even if the linked document changes. Reported validation scope: The captured guide marks H100/H200, AMD, Intel and Blackwell hardware supported/verified. Its exact 26B BF16 one-A100/H100 example uses a 32,768-token limit and 0.90 memory utilization; the full-feature launch uses 16,384 tokens and four images per prompt. The H100 variant, concurrency and tested publisher revision are not identified. The full-context Blackwell runs use a different NVFP4 checkpoint with an assistant and nightly builds. No measured one-H100 NVL BF16 capacity or fit is supplied. GPU fit remains unverified. GPU fit unverified: this guidance identifies a candidate to investigate; it does not verify the chosen GPU, checkpoint revision, context length, simultaneous requests or runtime memory. The checkpoint is pinned for your test; the engine recipe names the model without identifying its tested checkpoint revision. The API is separately tracked as fp8. Verify fit, formatting, task quality and checkpoint/precision equivalence before moving traffic. The calculation assumes the same billed token workload, without proving that the candidate generates the same tokens.

Spending investigation threshold

The current API observation is DeepInfra's google/gemma-4-26B-A4B-it, fp8, at 0.07 USD/million input and 0.34 USD/million output tokens. The lowest qualifying quote in our catalog for this exact candidate is Vast.ai, Marketplace offer #33610858, , US, 1.87 USD/hour for the whole instance. It is listed on-demand hourly pricing with no minimum commitment; confirm stock and final charges. Stock status: listed. This is catalog coverage, not a whole-market comparison.

# Display values are rounded; totals use full stored rates.
Period budget ≈ 1.87 USD/hour × 720 hours × 1 replica(s) + 0.00 USD
              ≈ 1,344.00 USD
API USD/million output, including input = 0.34 + (4096 / 1024) × 0.07
                                       = 0.62
Spending investigation threshold = budget / API USD per million output × 1,000,000
                                ≈ 2.2 billion billed output tokens
Requests ≈ 2.1 million per 30 days

At the spending threshold the workload demands 836 billed output tokens/second averaged over the calendar period, or 836 per replica if traffic is evenly shared. These are measurement targets, not measured GPU speeds. Replay the actual request lengths, bursts, reasoning settings and tool turns, then check aggregate output throughput, tail latency, failures and task success.

With the same 4,096 input tokens but 8,192 billed output tokens per request, the same budget corresponds to 440 thousand requests and 1,383 average output tokens/second. This longer-output sensitivity illustrates the effect of billed generation; it does not predict a reasoning setting's output length or the GPU's performance.

The period assumes 720 always-on hours and 1 replica(s). The explicit additional operating allowance is 0.00 USD. Include storage, network, taxes, monitoring, operational work and redundancy before comparing total service cost. When this allowance is zero the screen covers compute only. Extra replicas increase the bill; they are not a tested availability design.

API checked October 21, 2026 at 2:16 am EDT, collector source; rental checked October 21, 2026 at 2:02 am EDT, quote source. Recorded terms: On-demand compute billed for actual active runtime; no monthly commitment. Storage persists and remains billable until deleted; transfer and other charges are separate. Quote notes: Targeted contract refresh. Rentable at check; capacity can change. Compute only; storage and traffic extra. Prices and calculations refresh automatically. Figures are rounded for readability; calculations use the full stored precision. Stale, retired or changed-identity observations suppress this calculation.

Self-hosting terms and evidence

The publisher metadata and captured pinned checkpoint page declare Apache 2.0. The publisher links to Google's Gemma 4 license page; the captured terms here are Google's exact canonical Apache 2.0 URL under its own original receipt. These terms have been reviewed for this self-hosting comparison. When redistributing, provide the license, preserve applicable notices and identify modified files, including applicable NOTICE requirements. The license does not establish suitability or compliance for a particular application.

Publisher terms. Analysis reviewed October 15, 2026. Noach Ark did not rent this allocation or run inference, capacity or quality tests for this article. Prices and financial calculations refresh independently of this editorial review.