DwarfStar
DeepSeek V4, locally.
A narrow, self-contained C inference engine built by antirez (Salvatore Sanfilippo, creator of Redis) for running DeepSeek V4 Flash and PRO on high-end personal machines. Metal-first. CUDA arrived in May 2026.
At a glance
What DS4 ships today
A purpose-built inference engine — not a generic GGUF runner. One model family, end-to-end.
makeMetal + CUDA backends
Primary target is Apple Silicon Metal. In May 2026, a separable-backend architecture added first-class CUDA support, with special care for the NVIDIA DGX Spark (GB10).
Disk-based KV cache
Treats SSD storage as a first-class citizen for KV state. Persist session state across restarts. Enables long-context inference without keeping everything in RAM.
OpenAI / Anthropic API
ds4-server on port 8000 serves /v1/chat/completions and /v1/messages endpoints. Drop-in compatible with coding agents like Claude Code and opencode.
Asymmetric quantization
Only routed MoE experts are compressed to 2-bit (IQ2_XXS + Q2_K). Shared experts, attention, routing, and embeddings stay at Q8 full quality. Quality stays usable.
Official-vector validation
Logits are checked against official DeepSeek V4 Flash continuations at different context sizes. DS4 ships with test vectors and a regression test suite.
Integrated coding agent
ds4-agent (alpha) provides an integrated coding agent. Tool calling is reliable under the quantized model. Also works with Claude Code, opencode, and Pi.
Philosophy
One model, done end-to-end
DS4 is intentionally narrow. It takes a deliberately focused bet: one model family at a time, exhaustively validated, fully integrated.
Most inference engines chase the newest model as soon as weights drop. DS4 does the opposite — it treats DeepSeek V4 Flash as the only target, building a self-contained C engine around its specific architecture, quantization mix, and KV cache design.
The result is not a generic GGUF wrapper. It is a completely standalone runtime: custom GGUF files, model-specific loading, prompt rendering, tool calling, KV state management (both RAM and on-disk), and an HTTP server API — all tested end-to-end with coding agents.
"This project takes a deliberately narrow bet: one model at a time, official-vector validation, long-context tests, and enough agent integration to know if it really works." — antirez, ds4 README
Why DeepSeek V4 Flash specifically?
After extensive comparison with dense models, antirez concluded that Flash punches above its weight class:
- Thinking-mode proportional complexity — Flash produces thinking sections that are ~1/5 the length of competitors, and the length scales with problem complexity.
- Broad knowledge at the edge — 284B parameters means better coverage on niche topics (Italian politics, technical edge cases) than 27B or 35B models.
- Runs on real hardware — With 2-bit asymmetric quants, Flash fits on 96–128GB MacBooks and runs at usable speeds. PRO is available for 512GB Mac Studios.
- 1M token context — Unusually long context window with an incredibly compressed KV cache design.
Model weights & how to get started
DS4 only works with its custom GGUFs published on HuggingFace. Arbitrary DeepSeek/GGUF files will not have the correct tensor layout, quantization mix, or metadata. Download in one command:
./download_model.sh q2-imatrix # 96–128 GB RAM
./download_model.sh q4-imatrix # ≥ 256 GB RAM
./download_model.sh pro-imatrix # 512 GB RAM
Build with make (macOS Metal), make cuda-spark (Linux DGX Spark), or make cuda-generic (other CUDA GPUs). Compilation finishes in under 10 seconds.
Timeline
What's shipped, what's next
A rapidly moving project. Here is the commit-by-commit arc from initial release to the present day.
q4-imatrix enters the download script.rocm branch created. Antirez does not have AMD hardware, so this is maintained by the community with periodic rebasing.Analysis
Could DS4 serve from Tinfoil?
Tinfoil provides verifiably private AI via secure hardware enclaves on NVIDIA GPUs. The question: can DS4 run inside that envelope?
What Tinfoil is
Tinfoil (tinfoil.sh) is a YC-backed service that hosts open-source LLMs inside NVIDIA confidential computing enclaves — isolated GPU environments that hardware-guarantee zero data access and zero data retention. Think of it as "local AI privacy, but in the cloud." Users can verify attestation documents to confirm the enclave is running the expected code.
Tinfoil currently supports models like DeepSeek R1 70B, Llama 3.3 70B, Kimi K2, GPT-OSS 120B, and Qwen3 Coder 480B via a drop-in OpenAI-compatible API. They charge for chat ($20/mo) and API usage ($5 free credits).
What DS4 needs
- Access to CUDA GPUs (NVIDIA, preferably GB10 / DGX Spark class)
- ~128GB unified or device memory for Flash q2-imatrix
- Fast SSD for disk-based KV cache persistence
- Ability to run a custom compiled binary (
ds4-server) - Access to custom GGUF files (not generic Hugging Face weights)
| Requirement | Tinfoil | DS4 needs |
|---|---|---|
| CUDA GPU access | Yes (NVIDIA enclaves) | Yes (CUDA backend) |
| ≥128GB memory | Depends on SKU | 128GB for Flash q2 |
| Fast SSD for KV cache | Unknown / cloud NVMe | Disk KV is optional |
| Custom binary execution | Containers, not binaries | Compiled ds4-server |
| Custom model weights | Managed model list | Custom GGUFs required |
| Metal (Apple Silicon) | No — NVIDIA only | Metal is primary target |
On paper, DS4 CUDA on Tinfoil is viable. The CUDA backend targets NVIDIA, Tinfoil runs on NVIDIA enclaves, and the OpenAI/Anthropic API format is what Tinfoil already exposes. If Tinfoil can provision a 128GB+ GPU instance and allow custom container images with ds4-server baked in, the stack lines up.
But there are gaps. Tinfoil manages a curated model list — you can't just point it at arbitrary GGUF files. DS4's custom weights (asymmetrically quantized, model-specific) would need to be bundled into the deployment image or hosted from inside the enclave. KV cache on fast SSD is a DS4 superpower; in a cloud enclave, NVMe may be available but persistence semantics differ.
Most importantly, DS4's value proposition — a narrow, single-model, hand-tuned engine — is the opposite of Tinfoil's generic multi-model router. You'd be asking Tinfoil to host a bespoke binary instead of their standard inference stack. That might defeat the purpose of using Tinfoil at all.
Bottom line: If you're evaluating DS4 for inference and Tinfoil for privacy guarantees, they serve different problems. DS4 is for "I own the hardware and want the best possible local experience." Tinfoil is for "I can't own the hardware but want verifiable privacy." A Tinfoil-powered DS4 would be a neat demo, but you'd more likely use Tinfoil's existing DeepSeek R1 endpoint or run DS4 on your own DGX Spark / CUDA box.
Resources