Qwen3.5-27B: When to Test Self-Hosting on an H200

For Qwen3.5-27B, an API bill around 2,917.44 USD per 30 days can fund the modeled candidate compute budget. At 4,096 billed input and 1,024 billed output tokens per request, that is about 780 thousand requests, or 800 million output tokens plus their input. Near this spending line, investigate the candidate below. Moving traffic requires measured capacity, acceptable task quality and a complete operating budget.

Your Qwen3.5-27B traffic justifies investigating self-hosting when its billed usage approaches the live candidate budget below. The concrete test is the original BF16 checkpoint on one H200 NVL 141 GB, compared with DeepInfra's exact FP8 endpoint. This is a dense model: the language-model work can dominate even when requests contain images. Investigate whether your complete requests fit and finish successfully before adding a more complicated vision-serving pipeline. The spending line funds an investigation; the decision to move traffic still requires measured capacity, latency, task quality and a full operating budget.

Capacity unverified

This article provides a financial investigation screen. It has no qualified aggregate output-throughput measurement for the candidate. The traffic figure does not establish a reachable saving, supported concurrency, response time or equivalence to the API.

For this 27B model, separating the encoder can reduce goodput

An image-serving optimization that helps a smaller Qwen model can hurt this one. In NVIDIA's Qwen3.5 size ablation, colocated encoder separation produced 2.62 times the goodput of aggregated serving for 4B, 1.50 times for 9B, but only 0.65 times for 27B. NVIDIA attributes the 27B result to language-model work outweighing the benefit of isolating the encoder.

That relative goodput result comes from a Dynamo study with a four-GB200 test environment and workload-specific latency criteria. It is not aggregate output capacity for our BF16 H200 NVL candidate, and it cannot supply its traffic threshold. Its useful consequence is a test order: start with aggregated serving, then compare encoder separation on the same requests and task-success criteria. Measure the whole response, not only time to first token. Keep any additional workers and resources in the operating budget. Do not assume a better vision-stage metric means more completed 27B requests.

Evidence: serving source.

Why the BF16 test starts with one H200

The official vLLM recipe calls Qwen3.5-27B dense and lists 27B active parameters. Its default BF16 variant has 65 GB minimum-VRAM guidance, but its BF16 hardware recommendations are one H200 or two H100 GPUs. The shown NVIDIA BF16 launch uses tensor parallelism two. The publisher card uses eight GPUs in its full-context examples. None supplies a measured one-H100 comparison.

This article therefore investigates one H200 NVL rather than treating the numerical minimum as a validated one-H100 deployment. The guide names H200 without a variant. Its hardware verification is for Intel Arc Pro B60/B70, and its displayed XPU command uses tensor parallelism four, eager execution and an 8,192-token maximum model length. This is serving guidance, not a measured result for the chosen H200 NVL or the pinned publisher revision. GPU fit remains unverified.

The single-GPU example instead loads the separately named Qwen3.5-27B-FP8 checkpoint; the guide assigns that variant 33 GB and its GPTQ Int4 variant 17 GB. Those are different weight/precision experiments. They do not shrink this original BF16 candidate, and the API's FP8 label does not identify which downloadable weight revision the provider serves. Replay your tasks before assuming output equivalence.

The pinned BF16 configuration has 48 linear-attention layers, 16 full-attention layers and float32 recurrent state. Its native 262,144-token context does not establish concurrent-request memory. No uniform BF16 cache estimate or conversation count is assigned here.

Evidence: model source; architecture source; serving source.

Fix the reasoning policy before calling the migration successful

Qwen3.5 thinks by default. The pinned template starts a thinking block unless enable_thinking is explicitly false; a prompt instruction alone does not select the documented non-thinking template. DeepInfra's reasoning documentation says reasoning tokens count toward output billing. Use reported billed output rather than only the answer displayed to your user, without adding the reasoning count a second time.

The 27B card has a reproducibility issue worth resolving before the replay: its API-usage section and Best Practices give different non-thinking reasoning-task settings. One specifies top_p 0.95, top_k 20 and presence_penalty 1.5; the other specifies top_p 1.0, top_k 40 and presence_penalty 2.0. Do not silently combine them. Record the explicit policy that produced your accepted API results and test that same policy on the candidate, including parameters the serving framework actually supports. Precise coding has another documented policy, including temperature 0.6 and no presence penalty.

The card recommends allowing 32,768 output tokens for most queries and 81,920 for highly complex competition problems. These are output limits, not predicted usage or measurements. Check actual generated lengths, truncation, repeated output and successful-task rate. A shorter completion obtained by changing thinking, sampling or the output limit is not evidence that the original workload migrated successfully. The live block's longer-output example shows the financial effect without promising a GPU speed.

Evidence: model source; architecture source; serving source.

A long-video request can consume the context before reasoning begins

For hour-scale video, the pinned publisher card recommends increasing the video processor's longest_edge setting to 469,762,048, which it describes as a 224k-video-token budget for higher frame-rate sampling. The shipped video configuration instead uses 25,165,824. Copying the long-video setting changes the request you are asking the dense 27B model to process. It is not a free quality improvement within the text example.

Compare that visual budget with the native 262,144-token context and the card's separately recommended output allowances. Media, the conversation and generated reasoning must all share the configured request window. Large video input plus a generous generation allowance can leave little room for the rest of the conversation. Verify the actual processed token count; the configured budget does not mean every video consumes that many tokens.

Build a video replay from your real duration, frame sampling and processor settings. Include a longest video followed by the next user turn, and mix those requests with your ordinary text traffic. Record accepted frame counts, processed tokens, peak memory, queueing, completion latency and task success. The official guide offers a language-model-only flag that removes the vision path; a test using it cannot establish video service capacity. The spending calculation below remains a billed text-input/output screen, with no measured image or video capacity.

Evidence: model source; serving source; architecture source.

The candidate to test

Test 1 × H200 NVL (141 GB per GPU) with the pinned BF16 checkpoint. Serving-engine guidance: the captured official vLLM recipe lists a 65 GB minimum for this model and precision, with vLLM 0.17.0 or newer. Serving recipe, captured October 12, 2026 at 9:35 am EDT; the reviewed snapshot is fixed by its capture hash, even if the linked document changes. Reported validation scope: The captured recipe marks Intel Arc Pro B60/B70 hardware verified and describes validation on four B60 or two B70 cards. Its displayed XPU command uses tensor parallel size four, eager mode and an 8,192-token maximum model length. BF16 prerequisites name one H200 without a variant or two H100s; the NVIDIA BF16 launch uses two H100s at 262,144 tokens. The publisher card shows eight-GPU full-context launches. No measured one-H200 NVL workload or tested publisher weight revision is provided. GPU fit for this single-device candidate remains unverified. GPU fit unverified: this guidance identifies a candidate to investigate; it does not verify the chosen GPU, checkpoint revision, context length, simultaneous requests or runtime memory. The checkpoint is pinned for your test; the engine recipe names the model without identifying its tested checkpoint revision. The API is separately tracked as fp8. Verify fit, formatting, task quality and checkpoint/precision equivalence before moving traffic. The calculation assumes the same billed token workload, without proving that the candidate generates the same tokens.

Spending investigation threshold

The current API observation is DeepInfra's Qwen/Qwen3.5-27B, fp8, at 0.26 USD/million input and 2.6 USD/million output tokens. The lowest qualifying quote in our catalog for this exact candidate is Vast.ai, Marketplace offer #55583626, Texas, US, 4.05 USD/hour for the whole instance. It is listed on-demand hourly pricing with no minimum commitment; confirm stock and final charges. Stock status: listed. This is catalog coverage, not a whole-market comparison.

# Display values are rounded; totals use full stored rates.
Period budget ≈ 4.05 USD/hour × 720 hours × 1 replica(s) + 0.00 USD
              ≈ 2,917.44 USD
API USD/million output, including input = 2.6 + (4096 / 1024) × 0.26
                                       = 3.64
Spending investigation threshold = budget / API USD per million output × 1,000,000
                                ≈ 800 million billed output tokens
Requests ≈ 780 thousand per 30 days

At the spending threshold the workload demands 309 billed output tokens/second averaged over the calendar period, or 309 per replica if traffic is evenly shared. These are measurement targets, not measured GPU speeds. Replay the actual request lengths, bursts, reasoning settings and tool turns, then check aggregate output throughput, tail latency, failures and task success.

With the same 4,096 input tokens but 8,192 billed output tokens per request, the same budget corresponds to 130 thousand requests and 412 average output tokens/second. This longer-output sensitivity illustrates the effect of billed generation; it does not predict a reasoning setting's output length or the GPU's performance.

The period assumes 720 always-on hours and 1 replica(s). The explicit additional operating allowance is 0.00 USD. Include storage, network, taxes, monitoring, operational work and redundancy before comparing total service cost. When this allowance is zero the screen covers compute only. Extra replicas increase the bill; they are not a tested availability design.

API checked October 12, 2026 at 9:39 am EDT, collector source; rental checked October 12, 2026 at 1:01 pm EDT, quote source. Recorded terms: On-demand compute billed for actual active runtime; no monthly commitment. Storage persists and remains billable until deleted; transfer and other charges are separate. Quote notes: API discovery sample, capped at 12/GPU model. Rentable at check; capacity can change. Compute only; storage and traffic extra. Prices and calculations refresh automatically. Figures are rounded for readability; calculations use the full stored precision. Stale, retired or changed-identity observations suppress this calculation.

Self-hosting terms and evidence

The pinned Qwen publisher checkpoint declares Apache 2.0. Its captured license has been reviewed for this self-hosting comparison. When redistributing, provide the license, preserve applicable notices and identify modified files; follow any applicable NOTICE requirements. The terms do not establish suitability or compliance for a particular application.

Publisher terms. Analysis reviewed October 12, 2026. Noach Ark did not rent this allocation or run inference, capacity or quality tests for this article. Prices and financial calculations refresh independently of this editorial review.