Tiny-vLLM Open Source: Inference Engine Rewritten Entirely in C++/CUDA

Another Rebellion Against the Python Inference Stack

At the end of May, a project named tiny-vllm quietly appeared on Hacker News' Show HN section. The author, jmaczan, wrote an LLM inference engine from scratch in pure C++ and CUDA, with a clear mission—to extract vLLM’s core ideas (PagedAttention, Continuous Batching, and the like) from the Python framework and turn them into a minimal implementation that one could read in a week.

Despite the “tiny” in its name, its ambitions aren’t small. The author makes it clear in the README: this isn’t a toy—it aims to be a production-ready inference engine with code maintained at a few thousand lines. For comparison, the vLLM codebase now contains hundreds of thousands of lines—tens of thousands just in Python, plus CUDA kernels, backend adapters, schedulers, and distributed systems logic. For a newcomer, understanding exactly what happens during a single forward pass can take days of effort.

That’s exactly the problem tiny-vllm wants to solve.

Why “Rewrite vLLM” Again

Similar projects have actually appeared many times in the past two years. Karpathy’s llm.c and llama2.c take an educational approach—hundreds of lines of C to run inference; llama.cpp, meanwhile, pushes engineering to the extreme, dominating the local inference market for consumer hardware. But between the two—a service-grade inference engine with vLLM-level throughput that still stays readable—there hasn’t been a perfect fit.

vLLM itself is in an awkward position. It began as a UC Berkeley Sky Computing Lab research project, gained fame after the PagedAttention paper, and was then widely adopted in industry. Since then, it’s accumulated an endless pile of features: support for NVIDIA, AMD, Intel, TPU, AWS Trainium, Gaudi; tensor/pipeline/expert parallelism; compatibility with GPTQ/AWQ/INT4/INT8/FP8 quantization; integrations with FlashAttention and FlashInfer. With so much added, the core scheduling logic is buried under layers of abstraction.

The bigger issue is Python overhead itself. vLLM uses a “Python frontend + precompiled C++/CUDA backend” hybrid design. The scheduler, request queue, sampling logic, and KV cache management all run mostly in Python. In small-batch, low-latency settings, the GIL and interpreter overhead become measurable in real latency. Community discussions about “Python overhead in vLLM” have lasted more than a year; the maintainers are trying async engines and ZMQ-based process communication, but the fundamental problem remains.

Tiny-vllm takes a radical approach—the entire engine is written in C++, directly calling CUDA kernels, with no Python layer at all.

Technical Implementation: Focusing on Three Core Parts

From the current implementation, the author focuses on three main aspects:

1. C++ Implementation of PagedAttention

This is the soul of vLLM. In traditional attention mechanisms, the KV cache is allocated contiguously per request, leading to severe memory fragmentation—a slot with max_seq_len=4096 might only use 200 tokens, wasting the rest. PagedAttention borrows the virtual memory idea from operating systems, splitting the KV cache into fixed-size blocks (usually 16 tokens each), allocating them on demand. The result is logically contiguous sequences built from physically fragmented blocks.

Tiny-vllm rewrites this entire block manager in C++. The block table is a simple std::vector, allocation and recycling use a basic free list. Compared to vLLM’s multi-tiered BlockManager inheritance hierarchy, this version is indeed far easier to read.

2. Continuous Batching Scheduler

This is the key reason vLLM’s throughput is an order of magnitude higher than traditional static batching. Cade Daniel’s often-cited “continuous batching enables 23x throughput” article refers to precisely this. In simple terms, during decoding, the batch is reshuffled after every generated token—finished requests are immediately replaced with new ones, keeping the GPU constantly busy.

Tiny-vllm’s scheduler is a pure C++ implementation: an event loop runs on the main thread, and asynchronous CUDA streams handle launching kernels. No Ray, no multiprocessing, no asyncio—meaning operators no longer have to wrestle with timeouts and deadlocks.

3. Direct Reuse of CUDA Kernels

The author takes a pragmatic approach—no attempt to rewrite FlashAttention itself, since Tri Dao’s team spent years refining it. Instead, tiny-vllm directly includes FlashAttention’s .cu files, implementing only the paged attention dispatch and a few auxiliary kernels (for rotary embeddings, RMSNorm, sampling, and similar small components).

How Does It Compare to llama.cpp and SGLang

This is the question everyone’s asking. My take is—the three solve different problems.

llama.cpp excels at quantization and cross-platform support. It can run a 4-bit Llama on a Raspberry Pi, leverage Metal acceleration on an M-series Mac, or perform pure CPU inference on servers without GPUs. But its concurrency model is weak—it isn’t built for online serving.

SGLang takes another path—stacking even more aggressive optimizations atop vLLM with RadixAttention and a zero-overhead scheduler, focusing on highly concurrent structured output. But its codebase is even larger than vLLM’s—definitely not meant for casual reading.

Tiny-vllm occupies a gap: teams that want vLLM’s service power but also need full control over the code. This demand is real—especially from cloud providers and inference-service startups. They often need to embed private logic—custom load balancers, specialized sampling, or integrations with proprietary KV cache stores. In vLLM’s hundreds of thousands of lines of code, modifying such things—and surviving version upgrades—is a nightmare.

What’s Still Missing

Let’s pour some cold water. Tiny-vllm’s README openly acknowledges that the project is still early-stage:

• Limited model support. Currently validated only on the Llama family; Qwen, DeepSeek, Mixtral, and other mainstream models haven’t been adapted yet. Each model architecture requires new kernel dispatch logic.
• Little quantization support. FP16/BF16 inference works, but GPTQ/AWQ/FP8 and other quantization schemes aren’t implemented.
• Distributed inference remains a challenge. Tensor parallelism might be manageable, but adding pipeline or expert parallelism would quickly bloat the codebase, risking the “tiny” goal.
• Missing ecosystem. vLLM now offers an OpenAI-compatible API, Prometheus metrics, LoRA hot-reloading, and speculative decoding. Tiny-vllm would need to rebuild all that infrastructure from scratch.

So for now, the project best suits two types of people: those who want to understand PagedAttention and Continuous Batching by reading a well-commented minimal implementation; and teams with special customization needs willing to fork and heavily modify a compact codebase.

Use it as a drop-in replacement for production vLLM? Not yet.

A Bigger Trend

Stepping back, tiny-vllm is not an isolated case. Over the past year, “bypassing Python to rebuild inference stacks in systems languages” has become a visible trend: Mistral’s own mistral.rs is written in Rust; Hugging Face’s text-generation-inference is migrating core logic to Rust; Karpathy’s experimental projects increasingly use C/CUDA directly.

Behind this lies a shared understanding—LLM inference has moved beyond ‘just make it work’; now every millisecond of latency and every fraction of GPU utilization matters. Python remains unbeatable for fast research iteration, but in production inference pipelines, its marginal cost is becoming too steep.

vLLM’s maintainers seem aware of this too—the latest “V1 engine refactor” aims to move more scheduling logic down into C++. But ships that large turn slowly; a from-scratch rewrite like tiny-vllm can simply move lighter and faster.

The rise of these “small but sharp open-source replacements” may be happening faster than we expect.

References

• jmaczan/tiny-vllm - GitHub — Tiny-VLLM project homepage with full source, build instructions, and design documentation.