He hand-coded YOLO26n in assembly.

Recently, a developer publicly shared his undergraduate capstone project in Reddit’s MachineLearning community: implementing complete YOLO26n inference from scratch on a Raspberry Pi 4 using only ARM64 assembly and C, without relying on ready-made inference frameworks such as PyTorch, ONNX Runtime, or NCNN.

The project can already produce correct object detection results. More notably, the author did not stop at a bare-bones educational interpreter that merely “runs.” Instead, he packed ARM NEON SIMD, Winograd convolution, GEMM microkernels, cache-aware tiling, operator fusion, and attention modules into the inference engine.

But the conclusion is not a triumphant one: the optimized performance improvement fell short of the author’s initial expectations.

That is precisely what makes the project so valuable. It shows that in edge AI engineering in 2026, “handwritten assembly” is no longer a blank check that can automatically be cashed for high performance. Actual speed is often determined by the model architecture, data layout, memory traffic, quantization strategy, and scheduling of the entire computation graph—not merely by how elegantly a particular convolution loop is written.

Not Calling Operators, but Building an Entire Inference Engine

Strictly speaking, this project was not “written entirely in assembly.” The author adopted a more practical hybrid approach: C handles model loading, operator scheduling, memory management, and flow control, while ARM64 assembly handles the performance-critical compute kernels.

This division of labor is also common in industrial inference libraries. Making assembly handle the entire model graph would not inherently make it faster; it would only cause maintenance costs to skyrocket. The parts worth pushing down to the assembly level are usually those executed most frequently and offering the greatest degree of data parallelism, such as inner convolution loops, matrix multiply-accumulate operations, activation functions, and data rearrangement.

The author implemented:

• A hybrid ARM64 assembly and C inference engine;
• ARM NEON SIMD optimization;
• Winograd convolution;
• Optimized GEMM kernels;
• Cache-aware tiling, or partitioning data according to cache capacity;
• Custom ARM64 microkernels;
• Operator fusion;
• Attention mechanisms;
• Components required by YOLO26n, including Conv, C3K2, SPPF, C2PSA, PSA, BottleNeck, and Detect;
• A binary weight format redesigned for his own pipeline.

In other words, rather than exporting the model and invoking an existing runtime, he revisited all the questions an inference framework must answer: where tensors should be stored, how weights should be arranged, how convolution should be lowered to matrix multiplication, when intermediate results should be released, and how each operator can run faster on an ARM CPU.

A simplified execution pipeline can be abstracted roughly as follows:

Read custom binary weights
        ↓
Allocate buffers for input and intermediate tensors
        ↓
Execute Conv / C3K2 / SPPF in topological order
        ↓
Execute attention-related modules such as C2PSA / PSA
        ↓
Enter the Detect head
        ↓
Decode bounding boxes, classes, and confidence scores
        ↓
Output object detection results

It may look like a straightforward sequential pipeline, but every arrow conceals a large number of engineering decisions. For example, the same convolution can be computed directly with a sliding window or transformed into GEMM. Winograd can be used to reduce the number of multiplications, but its additional data transformations may instead slow down the overall process. Weights can be stored in the original layout used by the training framework, or they can be rearranged in advance into a format that the microkernel can read sequentially with minimal effort.

What Problems Do NEON, GEMM, and Winograd Each Solve?

The Raspberry Pi 4 uses an ARM Cortex-A72 CPU with support for 128-bit NEON SIMD. With FP32, for example, a vector register can hold four floating-point values simultaneously. Compared with scalar, element-by-element computation, NEON allows a single instruction to process data from multiple channels in parallel.

However, SIMD gains come with a prerequisite: data must be arranged in a way that supports efficient vector loads. If trailing channels are not aligned, access strides are too large, or layout conversions occur frequently during computation, the theoretical four-way parallelism can easily be consumed by loading, rearrangement, and boundary handling.

GEMM microkernels address the innermost part of matrix multiplication. Convolution can usually be transformed into matrix multiplication through im2col or other transformations. A generic implementation repeatedly reads data from memory, whereas an optimized microkernel attempts to keep a small block of inputs and weights in registers for an extended period, complete multiple multiply-accumulate operations in succession, and then write the results back all at once.

Cache-aware tiling does not simply mean “cutting the matrix into smaller pieces.” The tile size must simultaneously account for:

1. L1 and L2 cache capacity;
2. Cache-line size and set associativity;
3. The number of NEON registers;
4. How inputs, outputs, and weights are reused;
5. The channel counts, strides, and feature-map sizes of different convolution layers.

If a tile is too large, the working set will not fit in cache. If it is too small, loop control and boundary handling will consume a larger proportion of execution time. Mature frameworks generally provide multiple kernels for different shapes and select among them using heuristics or even runtime benchmarking. A personal project using only one fixed tiling strategy will struggle to cover all the dramatically different layer shapes found in YOLO.

Winograd convolution is likewise not something that “accelerates as soon as it is enabled.” It reduces the number of multiplications in certain 3×3 convolutions through input and weight transformations, making it suitable for specific kernel sizes, strides, and tensor dimensions. But the transformations themselves take time and generate additional intermediate data. For layers with 1×1 convolutions, small feature maps, low channel counts, or constrained memory bandwidth, Winograd may not be worthwhile.

Consequently, an optimization that performs better in a microbenchmark does not mean the entire YOLO26n computation graph will become proportionally faster.

A Custom Weight Format May Matter More Than Handwritten Instructions

The author extracted parameters from YOLO26n and redesigned them into a binary format suited to his own inference pipeline. This step can easily be overshadowed by the “assembly implementation,” but it may actually be closer to the core work of building an inference framework.

When training frameworks save weights, they prioritize generality, recoverability, and ecosystem compatibility—not the read order of a particular microkernel on a specific ARM CPU. If an inference engine waits until runtime to transpose weights, pack channels, and add alignment padding, it increases startup time and temporary memory usage.

Offline rearrangement allows weights to be stored sequentially in the order consumed by the microkernel. For example, output channels can be grouped according to the vector width, input channels can be padded, and frequently used bias or scaling parameters can be placed in adjacent regions. At runtime, the engine no longer needs to “tidy up the data while computing”; it can directly load blocks that have already been packed.

The trade-off is equally clear: such formats are generally tightly coupled to their implementations. Changing the kernel’s register blocking, precision type, or channel arrangement may require regenerating the weight file. It is more like an executable asset for a specific hardware backend than a general-purpose model file.

Why Handwritten Assembly Did Not Deliver the Expected Gains

The author did not provide sufficiently complete end-to-end benchmark data in the original post, stating only that the performance gains fell short of expectations. Therefore, it is not yet possible to determine whether the kernel efficiency was inadequate or the benchmark baseline was simply too strong. Judging from the typical bottlenecks of edge inference, however, there are at least several possibilities.

1. YOLO Is Not One Giant GEMM

Large matrix multiplications in language models tend to have relatively regular shapes, whereas object detection networks contain multiscale feature maps, branch connections, concatenation, upsampling, convolutions, and detection heads. Arithmetic intensity varies significantly across layers.

Even if a particular GEMM kernel becomes 50% faster, if it accounts for only 20% of end-to-end latency, the theoretical overall gain cannot exceed 10%. This is the everyday inference-engineering version of Amdahl’s law.

2. The Raspberry Pi 4 Is Easily Constrained by Its Memory System

The NEON compute capability of the Cortex-A72 does not equal the throughput an application can sustain. Model inference requires weights and intermediate feature maps to be read continuously. If arithmetic intensity is insufficient, the CPU may spend most of its time waiting for data.

In that case, eliminating a few more multiplication instructions has limited value. More effective approaches may include reducing precision, avoiding intermediate tensor write-backs, fusing adjacent operators, or redesigning buffer allocation so that data remains in cache as much as possible.

3. Operator Fusion Is Easy; Graph-Level Fusion Is Hard

Conv, Bias, and activation can be fused into the same loop relatively directly. But with residual paths, concatenations, and attention branches, a tensor may be reused by multiple downstream nodes. Overwriting it too early breaks correctness, while retaining it consumes memory.

Mature runtimes perform lifetime analysis, memory reuse, and graph optimization. Even with very fast individual operators, a personal implementation may still lose in end-to-end performance because of intermediate tensor movement.

4. Comparing Against Mature Frameworks Is Inherently Unfair

If the baseline is unoptimized pure C, implementing NEON kernels should provide obvious gains. If the baseline is an engine such as NCNN, which has been optimized specifically for ARM for many years, a personal project is competing against an entire body of engineering work, including kernel tuning for different CPUs, thread pools, memory allocators, quantization paths, and operator-selection strategies.

This does not mean that “assembly no longer works.” It means mature frameworks already make extensive use of assembly and intrinsics. Not using a framework does not mean entering an untouched field with no existing optimization. Quite the opposite: the comparison target may already have harvested most of the low-hanging fruit.

5. FP32 May Not Be the Best Choice for Edge Deployment

The referenced material does not clearly state whether the implementation includes INT8, FP16, or other low-precision paths. If the core implementation still uses FP32, both performance and memory bandwidth can easily become constrained on a device such as the Raspberry Pi.

Edge vision deployments intended for real products generally treat quantization as a central variable. INT8 is not merely an instruction-level substitution; it also involves calibration, quantization scales, accumulation precision, activation ranges, and regression testing for detection accuracy. This work is more complex than optimizing an FP32 microkernel, but it is often more closely tied to practical gains.

“Correct Results” Do Not Yet Mean “Complete Validation”

The author states that the implementation can already produce correct object detection results, which is an important milestone for implementing an entire network from scratch. However, determining whether it is competitive as an engineering solution would require a set of additional key metrics:

• Per-layer errors relative to the original model’s outputs;
• Changes in mAP on a standard dataset;
• Single-threaded and multithreaded latency;
• Latency distributions during cold starts and after reaching steady-state operation;
• Peak memory usage;
• Performance at different input resolutions;
• Same-precision comparisons with backends such as NCNN, TFLite, and ONNX Runtime;
• Separate timing for preprocessing, model execution, and postprocessing;
• Raspberry Pi temperature, clock frequency, and whether thermal throttling occurred.

Benchmarks on a Raspberry Pi are particularly susceptible to cooling conditions and dynamic frequency scaling. Reporting only the single fastest result is not very meaningful. A more reasonable approach would be to perform hundreds of runs after warm-up while reporting the median, P90 latency, CPU frequency, and thread count.

Without these figures, “correct” only demonstrates that the functional pipeline works. “Faster” still needs to be proven.

The Project’s Value Is Not in Replacing NCNN

From a practical standpoint, ordinary developers have no reason to handwrite an entire ARM64 inference engine just to deploy YOLO26n. Mature frameworks offer broader operator coverage, more complete quantization tools, and more reliable device adaptation. Business projects prioritize delivery and maintenance, not proving that their developers can write matrix multiplication.

As an undergraduate capstone project, however, this work has substantial technical depth. It genuinely connects compilers, computer architecture, numerical computing, and deep learning models. Many developers know that convolution is “matrix multiplication under the hood,” but have never personally dealt with register blocking, cache conflicts, weight packing, or tensor lifetimes. They may also know that operator fusion reduces memory access, yet rarely encounter the boundaries in branched networks where fusion cannot be applied arbitrarily.

More importantly, the project reached an honest conclusion: knowing all the optimization buzzwords and implementing them does not mean end-to-end performance will automatically take the lead.

Inference performance is not a simple sum of individual techniques. NEON, Winograd, GEMM, tiling, and fusion interact with one another. A layout suitable for convolution may not suit attention. A rearrangement that improves the current layer may force the next layer to pay a conversion cost. Reducing the number of multiplications may also introduce more memory accesses.

That is precisely why frameworks exist. They do not merely save developers from writing code; they manage a complex set of trade-offs among models, hardware, and a vast range of tensor shapes.

More Practical Lessons for Developers

For anyone preparing to undertake similar edge inference optimization, the correct sequence suggested by this project is not “learn assembly first,” but rather:

1. Build a per-layer profiler first. If you do not know where the time is going, do not start writing microkernels.
2. First determine whether the bottleneck is compute or bandwidth. The two require completely different optimization strategies.
3. Select kernels for each class of shape. Large matrices, small matrices, 1×1 convolutions, and 3×3 convolutions should not be forced to share the same approach.
4. Treat data layout as a first-class concern. Eliminating one rearrangement may be more effective than saving several instructions.
5. Evaluate operator fusion end to end. Do not look only at single-operator benchmarks.
6. Assess quantization early. On edge devices, numeric precision often has a greater impact on final performance than assembly techniques.
7. Compare against mature runtimes under identical conditions. Inputs, thread counts, precision, postprocessing, and cooling conditions must all be consistent.

This implementation may not become a better way to deploy YOLO26n, but it serves as an excellent “inference framework autopsy report.” It proves that an individual developer can indeed go all the way from a weight file to detection output, while once again reminding the industry that handwritten assembly is only the final mile. The truly difficult part is deciding where that final mile should lead.

References

• Reddit: I implemented the YOLO26n model inference from scratch using ARM64 Assembly Language: The project author’s original introduction, listing the inference engine’s components, ARM optimization methods, and feedback that performance fell short of expectations.
• Zhihu: YOLO26 officially open-sourced, with discussion of NMS-free inference and CPU performance: Additional background on YOLO26’s model characteristics, CPU inference direction, and common deployment methods.