NanoEuler: A person handcrafts GPT-2 using pure C/CUDA

Another Handcrafted GPT-2 Project from Scratch, This Time Called NanoEuler

At the end of June, a developer with the GitHub ID justvugg posted their side project NanoEuler on Hacker News: implementing a GPT-2–scale language model from scratch using pure C and CUDA, without relying on PyTorch, without relying on any Python deep learning frameworks, with even the training loop hand-written.

This might not sound new—Karpathy’s llm.c already paved the way, and nanoGPT has almost become the go-to textbook for LLM beginners. But NanoEuler is worth mentioning for two reasons: first, it’s not a simple clone of llm.c—the author has their own approach to kernel organization and backpropagation implementation; second, in 2026, with TensorRT-LLM, vLLM, and SGLang everywhere, someone still willing to handwrite every single GPT-2 operator in CUDA shows that the community’s demand for “understanding the lower levels” has never disappeared.

What Exactly This Project Does

A quick glance at the repo reveals a very straightforward structure. No setup.py, no requirements.txt, just a few .c, .cu, and .h files. The whole project breaks GPT-2 training into several parts:

• Tokenizer: Minimal BPE implementation capable of loading the original GPT-2 vocab and merges
• Forward Pass: Embedding, LayerNorm, Causal Self-Attention, MLP (GELU activation), and the final lm_head
• Backward Pass: Each forward kernel has a corresponding handwritten backward kernel
• Optimizer: AdamW, with custom m/v updates
• Data Loading: Directly reads binary token files and batches them

Anyone familiar with PyTorch might pause here: in PyTorch, F.layer_norm is done in one line, and loss.backward() handles automatic backpropagation. But in C/CUDA, LayerNorm’s forward involves calculating mean, variance, normalize, scale + shift; backward requires computing gradients with respect to input x, gamma, and beta separately, each involving reduction; every reduction must consider warp-level shuffle and shared memory bank conflicts. A LayerNorm forward and backward combined can take dozens of lines of kernel code. Attention needs careful handling of the causal mask and numerical stability of softmax.

NanoEuler implements all of this. The model size matches GPT-2 small (124M) and can be scaled to medium or large.

Compared with llm.c — What’s Lacking, What’s Strong

The inevitable comparison is with Karpathy’s llm.c. Objectively, llm.c is the benchmark in terms of engineering completeness and performance optimization—it does mixed precision, multi-GPU training, integrates FlashAttention, and even reproduces GPT-2’s full training wall-clock time. NanoEuler isn’t there yet: single GPU FP32 training, no advanced optimization libraries aside from cuBLAS.

But NanoEuler’s code organization is lighter. llm.c, in pursuit of performance, has ballooned into a single file daunting for beginners, with macros and conditional compilation interleaved—newcomers need to digest it first. NanoEuler splits kernels by function into multiple files, with less surrounding code for each kernel, which makes it friendlier as a teaching material for “understanding exactly what GPT-2 does at the lower level.”

This tradeoff is typical: llm.c says “I want to prove this path can be pushed to the extreme,” NanoEuler says “I want to fully understand what every line does.” The goals don’t conflict, but the final product looks different.

Why Is Someone Doing This in 2026?

People might ask: With vLLM and TensorRT-LLM for inference, Megatron and DeepSpeed for training, and Python implementations like nanoGPT for small research models, what’s the point of rewriting in C/CUDA?

Several practical reasons:

Understanding what happens below the abstraction layer. PyTorch hides too much. When you call scaled_dot_product_attention, do you know whether internally it uses FlashAttention-2 or cuDNN’s fused kernel? Do you know its memory access pattern? Most application engineers don’t need to know, but those working on inference acceleration, training frameworks, or new hardware adaptation must know.

Hard currency for interviews and hiring. In recent years AI infra interviews have leaned more toward the low-level. Being able to explain how to tune a CUDA kernel’s launch config or use shared memory is more valuable than doing ten LeetCode problems. A working C/CUDA GPT-2 project is one of the most convincing proofs you can put on a resume.

Preparing for next-generation hardware. In 2026, there are more new accelerator cards—not just from NVIDIA. If you want to port a training stack to these new devices, understanding low-level operator writing is your entry ticket.

A Few Implementation Details Worth Digging Into

When reading such a project, just knowing “it implements GPT-2” is useless—you need to see how it handles the pitfalls.

Causal mask in Attention. The naive approach is to construct an upper-triangular matrix, add it to attention scores, and then softmax—but that wastes half the computation. NanoEuler follows llm.c: skip the upper-triangular portion directly in the softmax kernel, saving both compute and memory.

Backward for LayerNorm. One of the easiest places to get wrong. dL/dx includes three terms: direct term, term through mean, term through variance. The latter two are global reductions—you must use warp shuffle for intra-warp reduction before block-level reduction, or performance drops by an order of magnitude.

AdamW implementation. Adam’s m and v states are as large as the parameters—about 1.5GB (FP32) for a 124M model; add parameters and gradients, and single-GPU memory pressure spikes. NanoEuler currently uses all FP32 without mixed-precision optimizer states—something worth optimizing later.

Data pipeline. The project assumes you already have preprocessed binary token files, each token stored as uint16. This simplification is reasonable—doing tokenization inside the training loop becomes a bottleneck; offline preprocessing and then streaming batches in is standard in industry.

Who This Project Is For

Straight talk:

• If you’re an LLM app developer working with APIs and prompts daily, this project won’t help much—just glance at it for broader perspective.
• If you’re an engineer aiming to switch to AI infra, reading this code carefully is more useful than reading ten papers.
• If you’re working on inference acceleration, custom frameworks, or adapting to new hardware, such from-scratch code is an excellent reference.
• If you mentor students or newcomers in AI, this project can be an alternative to llm.c for teaching, with a lower barrier to entry.

A Side Note

Training GPT-2 isn’t hard—a single A100 can finish in a few hours—but many people get stuck at “wanting to understand but never having the chance to read it from scratch.” NanoEuler’s value is in making that chance available to everyone.

As for inference, if you just want a GPT-2–level (or stronger) model for business, there’s no need to train it yourself. OpenAI Hub has aggregated mainstream models like GPT, Claude, Gemini, and DeepSeek—one key to connect, domestic access, OpenAI format compatible—save your energy for upper-layer products, and let those who understand the lower layer handle it.

References

• NanoEuler GitHub Repository - Project source code and README
• Reddit Discussion: Building GPT-2 in C - Community discussion on implementing GPT-2 from scratch
• Zhihu: Line-by-line Comparison of nanoGPT and llm.c - Code breakdown from an inference engineer’s perspective, including comparisons with llm.c