18-Year-Old Independent Developer Builds Tunisian Darija Translation Model from Scratch: A Honest Start with BLEU 3.89

Last weekend, an 18-year-old Tunisian student posted a niche but fascinating project on r/MachineLearning: a fully from-scratch machine translation pipeline for Tunisian Darija, complete with a manually annotated parallel corpus — all open source. The first version’s BLEU score was only 3.89, which he openly stated in the README.

It’s not a groundbreaking SOTA result, but it’s worth discussing. Because even in 2026 — when large models like Llama 4 and GPT-5 easily handle multiple languages — there’s still a whole set of languages these giants haven’t touched at all. Tunisian Darija is one of them.

A Language Ignored by Mainstream NLP

First, let’s clarify what this language is. Tunisian Darija is the local spoken Arabic dialect in Tunisia. Its difference from Modern Standard Arabic (MSA), the one taught in textbooks, is comparable to Cantonese versus Mandarin — perhaps even greater. What makes it even trickier is its writing system: young people often write it using Latin letters mixed with numbers, known as Arabizi in the field. For example:

• 3 represents the Arabic letter ع (ayn)
• 7 represents ح (ha)
• 9 represents ق (qaf)
• 5 represents خ (kha)

So a word like "3aychek" (meaning “thank you”) looks completely garbled to any standard Arabic tokenizer. As a result, existing Arabic NLP toolchains — whether it’s AraBERT or Google Translate’s Arabic module — tend to forcibly normalize Arabizi into MSA before translation, causing massive loss along the way.

The author summed it up perfectly: "Existing Arabic tools route it through MSA and mishandle the orthography." One sentence that nails the pain point.

What He Did

The technical stack of this pipeline is actually quite restrained — no parameter overloads, no GPT-4 distillation — just a straightforward Transformer encoder-decoder model:

• Arabizi-aware SentencePiece BPE tokenizer: 16k shared vocabulary, with a crucial detail — treating numbers like 3/7/9/5 as protected symbols to prevent BPE from splitting them or mixing them with ordinary digits. This is one of the most clever engineering decisions in the project.
• ~15.6M-parameter Transformer: trained completely from scratch, not based on any pre-trained language model.
• Transfer learning path: first pre-trained on cleaned Moroccan Darija data (since Maghrebi Arabic dialects share certain mutual intelligibility), then fine-tuned on the manually constructed Tunisian parallel data.
• Complete reproducible cleaning/training/evaluation pipeline.

And the data size? About 553 manually annotated parallel sentence pairs. Yes, you read that right — just over five hundred.

Why This Project Deserves Attention

In short: from an engineering and results standpoint, this v1 is basically at toy level. A BLEU of 3.89 means the outputs are not practically usable, with most n-grams mismatched. But if you’ve worked in low-resource NLP before, you know the methodology itself is right.

Compare it with the 2020 BUPT paper on Tibetan–Chinese parallel corpus for the judicial domain — that was a large team with around 160,000 sentence pairs and a curated terminology set for prior knowledge. Here, we have a high school student working alone with 553 sentences and building everything from tokenizer up. The scale difference is massive, yet he got several key things right:

1. Addressing orthographic uniqueness of Arabizi starting from the tokenizer, instead of crudely converting back to Arabic script.
2. Using Moroccan Darija for transfer learning, a typical "related-language transfer" strategy in low-resource NMT, discussed in the 2021 survey Research Review of Neural Machine Translation for Scarce Resource Languages.
3. Opening data construction for community participation, rather than building everything solo. The 553 pairs are just a seed — the pipeline enables others in Tunisia (developers, linguists) to keep adding more.

In other words, his goal wasn’t to solve the problem with v1, but to build a scaffold anyone can contribute to.

The Old Problem of Low-Resource Translation

Low-resource machine translation isn’t a new topic. The field has been wrestling with it since the days of statistical MT, and in the neural MT era we’ve seen numerous methods like back-translation, multilingual transfer, and pivot translation. Alibaba’s Tianchi platform even held evaluation contests specifically for low-resource multilingual translation.

But dialects have always been a blind spot. The reasons are simple:

• Dialects lack orthographic standardization; a single word can have five spellings (e.g., “hello” might appear as slem, slam, or sale7).
• There’s no authoritative corpus — legal texts and news reports are all in MSA.
• Unclear commercial value — big corporations like Google and Meta have no incentive to train a dedicated Tunisian dialect model.

So this kind of work can only come from the community itself. The author mentioned that, to his knowledge, there was no open-source parallel corpus for Tunisian Darija before his. That fact alone speaks volumes: a country of 12 million people has its everyday spoken language completely absent from the 2026 open-source NLP world.

Some Things Worth Critiquing — and Hoping For

Critiques:

• A 15.6M-parameter model trained on 553 data samples will inevitably overfit. The author admits the test set is very small, so BLEU on a locked test set has limited statistical significance.
• BLEU is a poor evaluation metric for dialect translation. Arabizi spelling freedom means multiple valid outputs can represent the same correct translation, causing BLEU to severely underestimate performance. COMET or human evaluation would be better.
• No back-translation augmentation. That’s a standard move in low-resource NMT; the author may add it later.

Hopes:

• With community expansion of the dataset to around 10k sentence pairs, combined with the current pipeline, results should improve dramatically.
• The Arabizi-aware tokenizer design can be transferred directly to other Maghrebi dialects (Algerian Darija, Libyan Arabic), and even to colloquial Egyptian Arabic.
• If the community adopts it, we might eventually see a fine-tuned version based on small open base models like Qwen or Gemma — far more effective than a from-scratch 15M-parameter Transformer.

What It Means for Developers

If you’re building multilingual applications, especially for North African or Middle Eastern markets, this project offers at least two directly useful resources:

1. An Arabizi-aware SentencePiece tokenizer, which you can integrate into your own models to handle Arabizi user input.
2. A reproducible low-resource NMT training template, perfect for small-dataset dialect scenarios — you can almost copy it as is.

For developers who usually rely on mainstream large models, projects like this are a reminder that “multilingual support” in major LMs has boundaries. GPT-5 and Claude 4 can handle MSA fine, but when it comes to Arabizi, you’ll need your own preprocessing pipeline. In real-world product scenarios, these practical details matter more than leaderboard scores.

By the way, if you want to quickly benchmark multiple models — say, feed the same Arabizi text to GPT, Claude, Gemini, and DeepSeek to see whose dialect handling works best — OpenAI Hub’s one-key multi-model API is handy. It supports direct API calls from China and follows OpenAI-compatible formats, so a quick evaluation script will do the job.

A Few Reflections

It’s 2026. The AI community’s attention is all on agents, multimodality, and reasoning models. But outside those spotlights, there are still numerous languages, scenarios, and users whose needs remain unmet. An 18-year-old student, armed with 553 sentences and a 15M-parameter model, tore open that silence. BLEU 3.89 doesn’t look good, but the path he’s taken is right.

Low-resource NLP has never needed bigger models — it’s always needed more people willing to start from the tokenizer, writing code line by line and annotating sentence by sentence.

References

• I built an open, from-scratch MT pipeline + parallel corpus for Tunisian Darija - Reddit — Original project post by the author, with technical details and GitHub link