GigaToken Open Source: Tokenization Speed Increased by 1000×

A Bottleneck Neglected for Too Long

On July 22, Stanford's Marcel Rød released GigaToken on GitHub — a BPE tokenizer rewritten from scratch in Rust. The project page features some striking numbers: when processing large-scale text corpora using the GPT-2 and Qwen vocabularies, it’s nearly 1,000× faster than OpenAI’s official tiktoken.

For those who’ve done pretraining, that number needs no explanation. Anyone who has run data on the scale of CommonCrawl knows that tokenization can often be more frustrating than the training itself: tens of terabytes of raw text, fed into tiktoken or HuggingFace tokenizers, can take days to process, and one mid-run OOM means starting over. Meanwhile, GPUs sit idle as the cloud bill keeps ticking.

That’s exactly the pain point GigaToken targets — and its approach is straightforward: remove Python completely from the critical path.

How the 1000× Speedup Was Measured

Let’s break down that number to avoid misinterpretation.

As the author clearly notes in the README: the near-1000× acceleration was observed when directly using GigaToken’s native Rust API — meaning the entire process of reading files, tokenizing, and writing out results is handled within Rust, without touching the Python interpreter. The vocabularies used were GPT-2 and Qwen’s BPEs.

If, however, you import it from Python and feed in strings chunk by chunk, the speedup will be significantly reduced. That’s due to PyO3 boundary overhead, GIL contention, and string encode/decode costs that even Rust can’t eliminate. This isn’t GigaToken’s fault — it’s a common limitation of any "Rust-accelerated Python" library, polars and tokenizers included.

The true 1000× scenario is when: you have a huge collection of text files and want to pretokenize them into token IDs for training. That’s precisely the slowest part of any pretraining pipeline — clearly, this project emerged from the author’s own hard-earned experience.

Why tiktoken Is No Longer Fast Enough

tiktoken itself is written in Rust, and when OpenAI open-sourced it in 2022, it was already an order of magnitude faster than HuggingFace’s implementation. So where does GigaToken’s additional performance come from?

From the project structure, a few key factors stand out:

• Zero-copy I/O: Uses mmap to read files directly, avoiding the expensive "read → UTF-8 decode → pass string → re-encode to bytes" Python loop
• Batch processing: The native mode naturally allows parallel processing of thousands of files, while tiktoken’s Python interface usually handles one chunk at a time
• Optimized BPE merge data structures: The BPE bottleneck lies in repeatedly looking up the merge table and updating the priority queue — the data structure choice here is critical
• Eliminated regex-based pre-tokenization overhead: GPT-2’s well-known pre-tokenization regex is one of tiktoken’s hotspots

In other words, tiktoken is fast in the "BPE inner loop," but its outer glue code is still Python. GigaToken moves the entire pipeline into Rust, eliminating all those hidden costs.

Who Benefits Most

First, who won’t benefit: if you’re just feeding prompts of a few hundred tokens to an LLM during inference, tiktoken is perfectly fine. Switching to GigaToken won’t make a difference — the latency savings drown in network round trips.

The real beneficiaries:

1. Teams training their own models. Even for modest 1B-parameter continue pretraining, shrinking a few hundred GBs of tokenization from two days to a few minutes is game-changing.
2. Those running data-mixing experiments. Trying different domain ratios? That used to mean rerunning the whole tokenization pipeline — now it's almost instant.
3. Batch tasks for RAG or long-document handling. Precomputing token counts for billions of documents to plan slicing or cost estimation—this is GigaToken’s sweet spot.
4. Tokenizer researchers themselves. Comparing vocabularies on the same corpus used to mean one experiment per day; now it’s ten per hour.

Don’t Confuse It with GigaTok

A quick note: there’s another recent project called GigaTok — ByteDance and HKU’s ICCV 2025 paper “Scaling Visual Tokenizers to 3 Billion Parameters.” The names differ by just one letter, but they’re entirely unrelated:

• GigaTok (vision): scales an image tokenizer up to 3B parameters for autoregressive image generation.
• GigaToken (text): pushes BPE tokenizer execution speed to the limit for language-model data preprocessing.

One scales model size, the other compresses time. The name clash is pure coincidence, but it’s causing confusion in the community already.

Rust’s Continued Infiltration of the ML Toolchain

Zooming out, GigaToken is part of a broader trend.

From HuggingFace tokenizers and tiktoken in 2022, to polars, candle, burn, and numerous Rust kernels inside vLLM, then to uv dominating Python package management in 2024 — Rust’s footprint in the ML infrastructure layer grows year by year. The reason is simple:

ML engineering bottlenecks increasingly lie not in algorithms but in data pipeline throughput. Python is fine for glue code, but once any stage must handle TB-scale data, the GIL and interpreter overhead become serious liabilities.

Three years ago, people might have said “we can do it in C++,” but Rust’s advantage is that it integrates into the Python ecosystem at low friction—just a pip install, and users barely notice the language underneath. This kind of “invisible acceleration” model is particularly friendly to open-source projects: authors don’t have to convince anyone to switch languages.

Open Questions

With the project newly open-sourced, several key issues remain unresolved:

Vocabulary coverage — Currently supports GPT-2 and Qwen BPEs. Variants like Llama 3’s tiktoken, DeepSeek’s tokenizer, and the SentencePiece family (Gemma, T5) are not yet fully supported. SentencePiece is tricky because it’s not pure BPE—it uses a unigram LM or a BPE variant, requiring different algorithms.

Training mode — GigaToken currently accelerates “encoding with an existing vocabulary,” not “training a new vocabulary from scratch.” The latter is also time-consuming — training a 50K-token GPT-2-style vocab over hundreds of GBs of corpus typically takes hours with tokenizers.

Cross-platform behavior — Rust projects are generally portable, but mmap behaves differently between Windows and Linux. Real-world large-scale concurrent I/O tests are needed.

Correctness verification — At this level of acceleration, the most common suspicion is “did it skip edge cases?” The author needs to produce large-scale, token-by-token alignment tests against tiktoken. Current repo tests are still basic.

Final Thoughts

This project has a clear focus: it’s not for inference service; it’s for data engineering.
Once you understand that, you won’t overhype the “1000×” claim or assume it competes with tiktoken — the two serve different roles.

tiktoken is an SDK for app developers — stable, cross-language, and aligned with OpenAI. GigaToken is a tool for infrastructure engineers — built to push hardware to its limits. Both will likely coexist for the foreseeable future.

For teams doing model training or large-scale data processing, GigaToken is absolutely worth trying — especially by running their own data through the Rust native API and measuring the end-to-end time saved. Optimization at this layer pays compound interest: every experiment runs twice as fast, and over a year, you can run exponentially more.

Open-source projects that achieve wide adoption typically follow a cycle: “benchmark surprise → growing pains → fixes → stability.” GigaToken is at step one. The community’s feedback in the next few months will decide whether it matures — but at least it’s tackling a long-ignored problem: BPE tokenization has been overdue for serious optimization.

References

• marcelroed/gigatoken - GitHub — GigaToken project page with performance data, usage instructions, and native API documentation
• SilentView/GigaTok - GitHub — Similarly named but entirely different visual tokenizer project (ICCV 2025), for comparison