Case study · Jul to Oct 2026
A 35B LLM for a Thai hospital, on one DGX Spark.
Hospital staff needed an assistant that answers in Thai from the hospital's own knowledge base. Patient data could not leave the building. I served Qwen3.6-35B-A3B on a single NVIDIA DGX Spark and spent three months measuring every change. Single-user Thai generation went from 58.0 to 108.9 tokens per second.
- Single user, Thai
- 108.9 tok/s, up from 58.0 (1.88×)
- 32 concurrent requests
- 556 tok/s, up from 454 (+22%), 0 of 200 failed
- Golden set
- 11 of 11 answers correct on 10 of 10 runs
- Data residency
- One machine. Inference, embeddings, index and chat all stay on it
The constraint
The DGX Spark has 128 GB of unified memory and about 273 GB/s of memory bandwidth. Generating each token means reading the active weights, so bandwidth sets the speed limit, not compute. A mixture-of-experts model with 3B active parameters fits that box far better than a dense model of the same quality.
Thai makes it harder. A token here carries 2.5 Thai characters against 6.3 English ones. The same tokens per second therefore gives a Thai reader half the text per second, and Thai is the number that matters to staff.
What moved the number
- Shipped: NVFP4, 3 drafted tokens108.9 tok/s
- Before: the incumbent checkpoint58.0 tok/s
- Turned down: 125B mixture-of-experts19.4 tok/s
- Turned down: dense 27B medicalabout 17 tok/s
- Turned down: 284B at 2 bits13.75 tok/s
- Batch capacity. Raising the concurrent-sequence cap from 4 to 32 roughly tripled throughput, to 454 tok/s at 32 requests with no failures.
- Engine choice. vLLM beat SGLang end to end: 53.3 against 35.6 tok/s single-stream, and 162 against 292 ms to first token. The difference was speculative decoding. vLLM could use the model's own multi-token prediction head, with 86% of drafted tokens accepted.
- Bytes per token. I priced each candidate checkpoint from its safetensors headers without downloading it. The incumbent read 4.00 GB per decode step, which caps it at 58.4 tok/s on this box.
- The swap. NVIDIA's own NVFP4 checkpoint reads 2.384 GB per step. With vLLM v0.28 and three drafted tokens per step, accepted tokens per step rose from 1.78 to 2.76. Depth 3 was the measured best: 105.5 tok/s at depth 2, 108.9 at 3, 95.2 at 4.
- Context. The window went from 131,072 to 262,144 tokens, the checkpoint's ceiling, so the whole 172,907-token reference knowledge base fits. Needle recall held at 12 of 12 up to 131k tokens.
Keeping the comparisons honest
Live traffic swung results by about 35%, so every benchmark waits until no requests are running. Both arms of an A/B run on the same day, because day-to-day variance is about ±6%.
The golden set grew from 4 to 11 questions after every model scored 100% on the first 4. A score that never moves decides nothing. Answers are scored on required and forbidden content, with retrieval and answer graded separately.
Thai text gets its own checks: tone and vowel-mark density, scans for stripped words, and reading the raw Thai across at least three reruns. The 11 of 11 result also needed fixes to the harness and the app, such as a fresh session per question. The model swap alone scored the same as the incumbent.
What I turned down
| Option | Why not |
|---|---|
| A faster 4.75-bit quantization | 8% faster, but dropped Thai tone marks in 2 of 3 reruns (ผู้ป่วย became ผูปวย). |
| A faster community NVFP4 build | 41% faster, but lost golden-set answers. I traced the cause to its tokenizer, so it can be reopened. |
| A general vision model for OCR | Changed clinical words in 3 of 3 reruns (ภาวะช็อก became ภาวะซีอก). |
| A 125B mixture-of-experts successor | Speculative decoding would not load. 19.4 tok/s, with the same golden-set score. |
| A dense 27B medical model | About 17 tok/s at best, and its context is smaller than the knowledge base. |
| A 284B model at 2 bits | Fits in memory, but decodes at 13.75 tok/s. |
Around the model
- Retrieval. RAG over the hospital's Wiki.js knowledge base, embedded with bge-m3 and synced nightly.
- Thai OCR. Typhoon OCR 1.5 reads scanned documents with a character error rate of 0.000, against 0.012 to 0.031 for the alternatives, and about 3× faster per page. Each scan becomes an unpublished draft, and a person publishes it before it reaches the index.
- Report bot. Staff ask in Thai and the bot picks the right hospital report in 93.4 to 95.3% of 242 test cases. The model only chooses from a fixed menu. The app owns the SQL, and identifying columns are stripped before rows reach the model.