Building a Transformer from Scratch: English-to-Thai Translation in Practice

A Transformer Reimplementation Worth Debugging Line by Line

Recently, developer imrancoder released a project that implements and trains a Transformer from scratch. Rather than directly calling a high-level, prepackaged Transformer module, the project uses PyTorch’s foundational torch.nn components to build an encoder–decoder architecture following Attention Is All You Need, then trains it as an English-to-Tamil machine translation model.

The project uses the gopi30/english-tamil parallel corpus from Hugging Face and was trained in a Kaggle environment with two NVIDIA T4 GPUs. The author also compiled the mathematical derivations, tensor shape transformations, model code, and training pipeline.

Projects like this are, of course, not going to challenge mature large models in translation quality, nor are they likely to become production services directly. But if developers have remained at the stage of calling nn.Transformer, Trainer, or inference APIs, this is more useful than looking at another Transformer architecture diagram. Which dimension should attention apply softmax over? Why does the decoder need two kinds of masks? Why can training be parallelized while inference must generate one token at a time? These questions only truly surface when the code throws errors or the loss refuses to decrease.

Our assessment is: this is a competent Transformer dissection project, not a competitive machine translation product. It is worth reproducing, but do not confuse “implemented from scratch” with “invented from scratch,” and do not substitute the ability to output Tamil sentences for rigorous evaluation.

What the Complete Pipeline Does, from Data to Output

English-to-Tamil translation remains a classic Seq2Seq task. Training samples consist of paired sentences: English is the source sequence, and Tamil is the target sequence. The model must learn a conditional probability: given an English sentence and the Tamil prefix generated so far, predict the next target token.

A complete training pipeline can be broken down into seven steps:

1. Clean and pair the English and Tamil sentences;
2. Train or load a tokenizer to convert text into token IDs;
3. Add special symbols such as BOS, EOS, and PAD;
4. Construct the source-sequence padding mask and target-sequence causal mask;
5. Use the Encoder to encode the full English sentence;
6. Have the Decoder read the shifted-right Tamil sequence and the Encoder output;
7. Use a linear layer to predict the next token over the target vocabulary and compute cross-entropy loss.

Assume the batch size is B, the English sequence length is S, the Tamil sequence length is T, the hidden dimension is D, and the main tensors generally flow as follows:

src_ids                 [B, S]
tgt_input_ids           [B, T]

src_embedding           [B, S, D]
tgt_embedding           [B, T, D]

encoder_output          [B, S, D]
decoder_output          [B, T, D]
logits                  [B, T, V_tgt]

Here, V_tgt is the size of the target vocabulary. Many bugs in handwritten Transformer implementations do not come from the formulas themselves, but from mixing up B, H, S, T, D, after which a single transpose or view silently changes the tensor’s semantics.

Data Processing Is Not a Supporting Role, Especially for Tamil

The project uses gopi30/english-tamil, an English–Tamil parallel dataset. For an educational project, its advantage is that the task boundary is clear: English goes in, Tamil comes out, making the responsibilities of the Encoder and Decoder easy to observe.

But Tamil is not a target language to which simple English-style whitespace tokenization can be applied. It has relatively rich morphology, and the same root may take multiple surface forms depending on case, number, person, or tense. A word-level tokenizer can easily lead to vocabulary explosion and a large number of low-frequency words; a character-level tokenizer avoids out-of-vocabulary words but significantly lengthens sequences.

A more practical approach usually uses subword methods such as BPE, WordPiece, or Unigram. When reproducing the project, verify at least four things:

• Whether English and Tamil use a shared vocabulary or separate vocabularies;
• Whether special-token IDs remain consistent across the dataset, masks, and loss;
• Whether the tokenizer is fitted only on the training set to prevent validation-set leakage;
• Whether the text uses consistent Unicode normalization and whether abnormal whitespace and duplicate samples have been cleaned.

If the tokenizer is unstable, the model may appear to converge while actually only memorizing frequent fragments. For low-resource or morphologically complex languages, data quality is often more important than adding two more Transformer layers.

Multi-Head Attention: Simple Formula, Error-Prone Shapes

The core formula for scaled dot-product attention is only one line:

Attention(Q, K, V) = softmax(QKᵀ / sqrt(d_k))V

Dividing by sqrt(d_k) is not decorative. As the dimension grows, the absolute values of dot products also tend to increase, making softmax more likely to enter a saturated region and degrading gradients. Scaling keeps attention scores within a range better suited for optimization.

Multi-head attention divides the hidden dimension D into H heads, with each head having dimension D/H. The typical shape transformations are as follows:

Input X                    [B, L, D]
Q/K/V after linear mapping [B, L, D]
Split into multiple heads  [B, H, L, D/H]
Attention scores           [B, H, Lq, Lk]
Output of each head        [B, H, Lq, D/H]
Merge multiple heads       [B, Lq, D]

The following is not a line-by-line copy of the project repository’s code, but a condensed version of the core logic:

import math
import torch
import torch.nn as nn

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        assert d_model % n_heads == 0
        self.n_heads = n_heads
        self.d_head = d_model // n_heads

        self.q_proj = nn.Linear(d_model, d_model)
        self.k_proj = nn.Linear(d_model, d_model)
        self.v_proj = nn.Linear(d_model, d_model)
        self.out_proj = nn.Linear(d_model, d_model)

    def split_heads(self, x):
        b, length, _ = x.shape
        x = x.view(b, length, self.n_heads, self.d_head)
        return x.transpose(1, 2)

    def forward(self, query, key, value, mask=None):
        q = self.split_heads(self.q_proj(query))
        k = self.split_heads(self.k_proj(key))
        v = self.split_heads(self.v_proj(value))

        scores = q @ k.transpose(-2, -1)
        scores = scores / math.sqrt(self.d_head)

        if mask is not None:
            scores = scores.masked_fill(~mask, float('-inf'))

        weights = torch.softmax(scores, dim=-1)
        context = weights @ v

        context = context.transpose(1, 2).contiguous()
        b, length, _, _ = context.shape
        context = context.view(b, length, -1)
        return self.out_proj(context)

There are three common issues here: softmax must operate over the final dimension corresponding to key length; after transpose, call contiguous before view; and the Boolean semantics of the mask must be consistent. Whether True means “keep” or “mask out” cannot be reversed across different modules.

The Decoder’s Two Types of Attention Perform Different Jobs

Self-attention in the Encoder allows each English token to examine the entire English sentence. The Decoder, meanwhile, has two attention mechanisms:

• Masked self-attention: Query, Key, and Value all come from the target sequence, but the current position cannot see future tokens;
• Cross-attention: Query comes from the Decoder, while Key and Value come from the Encoder output, aligning the target-language generation process with information from the source sentence.

If cross-attention is compared to looking up information, the Decoder’s current state is the search query, while the Encoder output is the already indexed English sentence. Each time a Tamil token is generated, the model reassesses which positions in the source sentence are most relevant.

During training, the complete target sentence is already available, so every position can be processed in parallel. However, a lower-triangular causal mask must block future answers:

def causal_mask(length, device):
    mask = torch.tril(
        torch.ones(length, length, dtype=torch.bool, device=device)
    )
    return mask[None, None, :, :]  # [1, 1, T, T]


def padding_mask(token_ids, pad_id):
    return (token_ids != pad_id)[:, None, None, :]  # [B, 1, 1, L]

Padding masks and causal masks do not solve the same problem. The former prevents the model from attending to padded positions, while the latter prevents the target side from peeking at the future. In many implementations where “loss falls unusually fast but inference is disastrous,” the root cause is that the causal mask is reversed or never takes effect.

The Target Sequence Must Be Shifted by One Position During Training

Suppose the correct translation tokens are:

<BOS> நான் வீட்டிற்கு செல்கிறேன் <EOS>

The Decoder input should be the sequence with the final token removed, while the supervision labels should omit the first token:

Input:  <BOS> நான் வீட்டிற்கு செல்கிறேன்
Label:  நான் வீட்டிற்கு செல்கிறேன் <EOS>

The corresponding code generally looks like this:

tgt_in = tgt_ids[:, :-1]
tgt_out = tgt_ids[:, 1:]

logits = model(src_ids, tgt_in)
loss = criterion(
    logits.reshape(-1, logits.size(-1)),
    tgt_out.reshape(-1)
)

The loss function should use ignore_index=pad_id; otherwise, PAD tokens added to sentences of different lengths will be treated as real labels. Common training settings for the original Transformer also include Adam, learning-rate warmup, dropout, and label smoothing. An educational reproduction can simplify these, but change only one variable at a time. Otherwise, it becomes difficult to determine whether a failure to converge comes from the architecture, data, or optimizer.

Before full training, first make the model overfit tens to hundreds of sentence pairs. If it cannot even memorize a tiny dataset, spending more GPU time is usually pointless. Check the following first:

1. Whether the input and labels are shifted by one position;
2. Whether PAD is ignored by the loss;
3. Whether the causal mask points in the correct direction;
4. Whether positional encoding covers the maximum sequence length;
5. Whether logits are passed directly to cross-entropy instead of applying softmax first;
6. Whether training and inference use the same BOS and EOS conventions.

Dual T4s Describe the Training Environment, Not a Two-GPU Requirement

The author completed training on two NVIDIA T4 GPUs in Kaggle, but “dual GPUs” alone says nothing about implementation efficiency. Anyone reproducing the project should also confirm whether it uses DataParallel, DistributedDataParallel, or merely assigns different work to different devices. They should also monitor the effective batch size, gradient synchronization, GPU memory usage, and whether data loading becomes a bottleneck.

For a Transformer of this educational scale, a single GPU can usually run it as well, though training will take longer. The most common benefit of two GPUs is increasing the batch size or shortening the experiment cycle, not improving single-sample inference speed. If the model is very small and the sequences are short, communication overhead may even cancel out the benefits of parallelism.

What truly needs to be recorded is a reproducible configuration: number of model layers, d_model, number of attention heads, feed-forward dimension, vocabulary size, maximum sequence length, batch size, learning-rate schedule, random seed, and checkpoint-selection rules. Merely stating “trained on dual T4s” offers limited help to other developers.

Inference Cannot Copy Training: It Is Sequential Generation

During training, the ground-truth target sequence is available, allowing parallel computation through masking. During inference, there is no answer available, so generation must begin with BOS, repeatedly predict the next token, and append the prediction back to the input.

@torch.no_grad()
def greedy_decode(model, src_ids, bos_id, eos_id, max_len):
    generated = torch.full(
        (src_ids.size(0), 1), bos_id,
        dtype=torch.long,
        device=src_ids.device
    )

    for _ in range(max_len - 1):
        logits = model(src_ids, generated)
        next_token = logits[:, -1].argmax(dim=-1, keepdim=True)
        generated = torch.cat([generated, next_token], dim=1)

        if torch.all(next_token.squeeze(1) == eos_id):
            break

    return generated

Greedy decoding is suitable for verifying that the implementation is correct, but it does not necessarily produce the best translation. A more complete system would use beam search and handle length penalties, repeated fragments, minimum output length, and caching. If the entire Decoder history is recomputed at every step, inference complexity and latency increase significantly; modern generation systems usually cache historical Keys and Values.

How to Tell Whether It Has Actually Learned to Translate

Based on the currently available project summary alone, it is not possible to determine the translation quality achieved by the model. A decreasing training loss and a few randomly selected translations that appear fluent are not enough to demonstrate generalization.

At a minimum, the following should be reported:

• BLEU on an independent test set;
• chrF or chrF++, which is more sensitive to morphological variation;
• Comparisons against simple baselines and an nn.Transformer version;
• Performance across different sentence-length ranges;
• Human evaluation by native Tamil speakers;
• Checks for training-set duplicates, data leakage, and incorrect alignments.

For a language pair such as English–Tamil, a single BLEU score also has limitations. A valid translation may differ substantially from the reference on the surface, while morphological variation further amplifies token-level mismatches. Metrics should be used to compare experiments, not replace judgments of linguistic quality.

The Most Valuable Lesson Is Not Rebuilding a Translator

Compared with directly using a mature model, the results of a handwritten Transformer will almost certainly be worse: the dataset is smaller, the architecture is older, there is no large-scale pretraining, and it lacks the normalization, positional-encoding, optimization, and inference techniques used by modern models.

But it offers three irreplaceable educational benefits.

First, it builds intuition for tensor shapes. Developers come to truly understand that multi-head attention splits the hidden dimension rather than duplicating the input multiple times.

Second, it clarifies the difference between training and inference. Teacher forcing, causal masking, and autoregressive generation stop being mere terminology.

Third, it builds the ability to debug model errors. Later, when encountering KV Cache, FlashAttention, Grouped Query Attention, or large-model parallelism, developers can understand which portion of the computation an optimization replaces instead of merely memorizing parameter names.

Therefore, the most sensible way to reproduce the project is not to copy it in full and take a screenshot afterward. Instead, add assertions to key tensors, visualize attention matrices, deliberately remove masks to observe the resulting failures, and compare numerical results against PyTorch’s high-level implementation. The project is only truly complete when every dimension can be explained.

Minimum Checklist Before Reproduction

• Fix the random seed and save the tokenizer and vocabulary;
• First overfit on a very small dataset;
• Write shape tests for every module;
• Inspect the broadcasted result of attention masks;
• Specify whether the implementation uses the original paper’s post-norm or the more trainable pre-norm;
• Save the checkpoint with the best validation performance rather than only the final step;
• Record loss, BLEU/chrF, throughput, and GPU memory usage;
• When randomly sampling results, retain the source sentence, reference translation, and model output to make data problems easier to locate.

This English-to-Tamil project does not introduce a new architectural breakthrough for Transformers, but as an exercise in “turning formulas into a trainable system,” it is more substantial than many tutorials that merely present a few dozen lines of wrapper code. For developers who already know how to call models but still cannot clearly explain masks and cross-attention, the most valuable action is not bookmarking the repository—it is breaking the implementation once by hand and then fixing it.

References

• Reddit: Project announcement and training environment: The author introduces the pure-PyTorch Transformer, the English-to-Tamil dataset, and the dual-T4 training environment.
• GitHub: Transformer from scratch project code: Contains the relevant code and project files for implementing a Transformer from scratch.
• Hugging Face: gopi30/english-tamil dataset: The page for the English–Tamil parallel corpus used by the project.
• GitHub: Chinese–English Transformer translation implementation: Useful for comparison with a PyTorch Transformer implementation for another language pair.
• Zhihu: Practical English-to-Chinese translation with PyTorch Transformer: Provides additional practical experience with machine translation, including data processing, BLEU evaluation, and GPU memory management.