ARPL Brings Automatic LLM Tuning to ARM Devices

ARPL recently released its first usable version, attempting to solve a long-underestimated problem: using the same set of llama.cpp parameters across different ARM devices is usually not a sensible approach.

When an application starts, the project reads the ISA capabilities and CPU topology actually exposed by the device, including instruction extensions such as SDOT, I8MM, and SME2, as well as how the cores are clustered. It then recommends a thread count for llama.cpp and adjusts context-related parameters such as Flash Attention and KV Cache quantization. Developers do not need to compile separately for every phone model, nor do users have to manually change parameters according to a device compatibility table.

The first public release primarily targets Android. It includes a reference application written in Kotlin and Jetpack Compose, along with a JNI bridge to llama.cpp. The author says the project was developed around the Snapdragon 8 Elite and has been tested on the Samsung Galaxy S25 Ultra (SM-S938B).

However, ARPL is not currently a general-purpose feature that has already been merged upstream into llama.cpp. More precisely, it is a runtime hardware detection and tuning framework built around llama.cpp. Heterogeneous workload distribution across the CPU, GPU, and NPU is still under development. This release covers only ISA, threading, and context strategies.

Just Because llama.cpp Runs Does Not Mean It Runs Well

Running llama.cpp on ARM devices is nothing new. It supports AArch64 capabilities such as NEON, I8MM, SVE, SVE2, SME, and SME2, and can also use different hardware through paths such as KleidiAI microkernels, OpenCL, and Vulkan.

The real challenge is the configuration strategy.

A new flagship powered by the Snapdragon 8 Elite and a five-year-old mid-range phone may both successfully load the same GGUF model. But if the application uses a fixed thread count, a fixed KV cache format, and the same attention configuration on both, the results can be completely different:

• The flagship chip may fail to fully utilize new matrix and dot-product instructions, wasting performance;
• The older device may suffer from scheduling contention and more severe throttling because too many threads are used;
• Treating performance and efficiency cores identically and maxing them all out may produce fast initial inference but increasingly slow sustained generation;
• Without KV cache quantization, long contexts can quickly exhaust memory capacity and bandwidth;
• Mechanically enabling or disabling Flash Attention may not suit the current backend, model, or hardware capabilities.

On desktop systems, developers can usually tune performance over time through launch parameters, build options, and benchmarks. This is much harder for mobile applications: phones use a wide range of SoCs, vendor kernels differ, and even the same chip may expose different capabilities under different OS versions. Application developers cannot realistically maintain an ever-growing “device model–parameter” lookup table.

ARPL aims to replace that static table with a startup-time hardware checkup.

Three-Layer Detection: Instructions, Cores, and Inference Context

1. Determine the ISA Through HWCAP Instead of Guessing from the Chip Model

ARPL uses runtime HWCAP information to identify CPU capabilities. HWCAP can be understood as a list provided by the operating system that tells user-space programs which instructions are safe to use. Instead of guessing what a chip supports from a commercial name such as “Snapdragon 8 Elite,” the program directly queries which capabilities the current system actually exposes.

This is far more reliable than a device-model allowlist.

A device’s advertised hardware, CPU microarchitecture, and the instructions actually available to an Android application are not always the same thing. The system kernel, virtualization environment, vendor firmware, and even the ABI under which the application runs can all affect the capabilities ultimately exposed. Runtime detection can at least prevent an application from executing an instruction that the system has not declared as supported and crashing immediately.

ARPL currently focuses on capabilities including:

• SDOT: Instructions for dot-product calculations, which are important for low-precision matrix operations;
• I8MM: Enhanced 8-bit integer matrix multiplication, suitable for frequently used computations in quantized models;
• SME2: A newer Arm Scalable Matrix Extension designed to improve the efficiency of matrix workloads.

However, two stages must be distinguished here: detecting an instruction does not automatically provide acceleration. llama.cpp or its underlying microkernels must already implement the corresponding path and dispatch to it correctly at runtime before the hardware capability can translate into faster token generation. ARPL solves the problem of “discovering capabilities and selecting a strategy”; it does not reinvent matrix kernels.

It is more like a dispatcher than an engine.

2. Thread Count Is No Longer Equal to Logical Core Count

Mobile SoCs commonly use heterogeneous CPU designs. Performance and efficiency cores may be divided into multiple clusters, each with different frequency, cache, and power characteristics. Simply querying the total number of logical cores through a system API and setting the thread count to that number is often the easiest—and crudest—approach.

LLM inference does not guarantee that more threads will produce better performance. Matrix computations are affected by memory bandwidth, cache hit rates, and synchronization overhead. Adding a few more small cores may only create additional contention. Phones also have far less thermal headroom than servers, so after dozens of seconds at full multithreaded load, the frequency curve is usually more important than the peak number seen at startup.

ARPL reads core-cluster information and provides topology-aware thread-count recommendations. Its value lies not only in allowing flagship devices to use more threads, but also in proactively using fewer threads on devices where additional threads would be counterproductive.

This is particularly suitable for the following scenarios:

1. Offline chat or document summarization on phones that requires sustained generation of hundreds of tokens;
2. Applications performing embedding, classification, or small-model extraction in the background without monopolizing foreground resources;
3. A single APK that must support flagship phones, mid-range devices, and tablets;
4. Enterprise deployments where device models cannot be controlled and developers cannot test every device in advance.

Of course, topology awareness still does not amount to complete performance scheduling. Android cpusets, thread affinity, thermal state, and vendor schedulers can all affect the result. ARPL currently provides a more sensible starting point, not an optimal thread count that applies to every workload.

3. Propagating Hardware Information into Context Strategies

ARPL does not stop at identifying CPU features. The public release also adjusts parameters such as Flash Attention and KV Cache quantization according to hardware support.

This may be more practical than gaining a few additional percentage points of CPU performance.

As a model’s context grows, the KV cache continuously consumes memory and creates significant bandwidth pressure. A phone’s unified memory must serve the operating system, graphical interface, and other applications while also holding model weights and runtime caches. With the wrong parameter settings, a local model may not merely run “a little slower”—the operating system may terminate it outright.

KV Cache quantization trades precision for lower memory usage and bandwidth pressure, while Flash Attention attempts to reduce intermediate data reads and writes during attention computation. Neither setting should be enabled indiscriminately on every device, but both should be selected in conjunction with hardware capabilities, model size, and context length.

ARPL’s logic can be abstracted into the following pipeline. This is only a process illustration, not an actual API provided by the project:

Application startup
  ├─ Read system HWCAP
  │    └─ Check for capabilities such as SDOT / I8MM / SME2
  ├─ Read CPU topology
  │    └─ Identify core count, clustering, and performance differences
  ├─ Generate runtime recommendations
  │    ├─ llama.cpp thread count
  │    ├─ Flash Attention strategy
  │    └─ KV Cache quantization strategy
  └─ Initialize llama.cpp through JNI and load the GGUF model

The greatest benefit of this design is that the application package can remain relatively uniform, while decisions previously made during the build process or through manual developer testing are deferred until runtime on the user’s device.

ARPL Is Not a Replacement for KleidiAI

From a technical-stack perspective, ARPL and Arm KleidiAI operate at different layers.

KleidiAI provides optimized microkernels for Arm CPU features and is responsible for making tensor operations actually run faster. llama.cpp can enable the corresponding CPU backend at build time and select a more efficient implementation when supported capabilities are detected. Some SME paths must also be enabled separately, while higher-priority backends such as Metal or GPU backends may change where execution ultimately takes place.

ARPL addresses higher-level questions:

• Which ISAs does the current device actually expose?
• How should the cores in this SoC be used?
• How should llama.cpp startup parameters vary by device?
• Which context optimizations should be enabled on this device?

The two do not conflict. An ideal combination would use KleidiAI or llama.cpp kernels to provide high-performance operators at the lower level, while a runtime strategy layer such as ARPL selects the correct kernels and configuration at the upper level.

This is also ARPL’s most commendable aspect at present. The ARM ecosystem does not lack isolated optimizations; what it lacks is an automation layer that connects hardware detection, backend capabilities, and application parameters. For independent developers, avoiding the need to maintain dozens of device configurations may be more valuable than a 10% improvement in a particular benchmark.

It Is Too Early to Treat It as “Self-Driving Inference” for Phones

ARPL is moving in the right direction, but its first release still has clear limitations.

First, the only test device currently mentioned by the author is the Samsung Galaxy S25 Ultra. Whether the successful experience on the Snapdragon 8 Elite can be transferred to MediaTek, Samsung Exynos, older Qualcomm platforms, ARM servers, or development boards will require validation on more devices.

Second, the publicly available information does not yet include a complete, reproducible performance table. Claims that it has “already made a noticeable difference” still need to be broken down into metrics such as time to first token, sustained generation speed, peak memory usage, power consumption, and temperature increase. Mobile benchmarks in particular cannot focus only on peak tokens/s over a few dozen seconds; if thermal throttling occurs after five minutes, the earlier advantage may quickly disappear.

Third, heterogeneous task allocation across the CPU, GPU, and NPU is not included in this release. For Snapdragon platforms, this is precisely the more difficult and potentially more valuable part. The Hexagon backend in the llama.cpp repository is still marked as under development, while Adreno GPUs involve OpenCL or Vulkan paths. Reliably partitioning a model across the CPU, GPU, and NPU requires more than hardware detection; it also requires handling operator coverage, memory copies, synchronization overhead, and differences between vendor drivers.

Finally, automatic recommendations should not become a black box. Developers still need visibility into detection results, final parameters, and fallback reasons, along with the ability to override settings manually. Otherwise, if a particular device suffers a performance regression, debugging may become more difficult than with explicit configuration.

A mature version should provide at least:

• Readable ISA and core-topology logs;
• A comparison between automatic and manual configurations;
• Safe fallbacks when a capability is unsupported;
• Benchmark data across different models, quantization formats, and context lengths;
• Sustained performance before and after thermal throttling, rather than only during a cold start;
• Configuration-strategy version numbers to help identify performance changes after upgrades.

Local LLMs Need Exactly This Kind of “Unsexy” Infrastructure

Over the past two years, mobile LLM demonstrations have often focused on the fact that “a model with several billion parameters can finally run.” But when moving from demos to products, what usually consumes development time is not loading the model, but compatibility, memory, threading, thermal management, and drivers.

ARPL does not release a new model or create a new inference backend. It does something more straightforward: it enables llama.cpp to first understand what device it is running on and then decide how it should run.

This work will not suddenly double inference speed across all ARM phones, but it could lower the engineering barrier for local AI applications. In particular, as the same GGUF model must support an ever-growing range of Android devices, runtime ISA and topology detection will gradually shift from an “advanced optimization” to a basic capability.

At this stage, the most appropriate assessment of ARPL is: the direction is right and the engineering value is clear, but the evidence and device coverage are still at an early stage. It is already far more sophisticated than hard-coding parameters by device model, but it still needs heterogeneous execution, long-term performance data, and broader device validation before it becomes a truly cross-chip, cross-backend automatic scheduling system.

If it is eventually integrated upstream into llama.cpp or develops into a stable hardware-strategy interface, its significance will extend beyond an Android reference application: local LLMs will finally be able to identify the device and then choose an inference path, much like modern graphics applications select a rendering backend, instead of requiring every developer to stumble over the same problems all over again.

References

• Reddit: ARPL Runtime ISA and Topology Detection Release Notes: The project author introduces the first public release, Android reference application, scope of hardware detection, and current limitations.
• GitHub: llama.cpp Repository: Information on the quantization formats, CPU instruction sets, and inference backends supported by llama.cpp.
• GitHub: llama.cpp Snapdragon Backend Documentation: Details on the Qualcomm Hexagon backend support that remains under development.