Liquid AI Makes CPUs Handle Long Contexts

Liquid AI Shifts the Optimization Target from “Generating” to “Finishing the Read”

Liquid AI recently released LFM2.5 Encoders, extending LFM2.5’s on-device roadmap to encoder models, with a focus on addressing slow inference and high memory usage for long contexts on CPUs.

The phrase “long-context inference” in the headline can easily be misleading. LFM2.5 Encoders are not another generative model for chatting or writing code, nor do they output answers token by token like GPT or Claude. They perform a different, more fundamental task—one that is often invoked more frequently in production systems: reading an entire passage at once and generating contextual representations for each token or the whole document.

In practical applications, this includes text classification, semantic retrieval, reranking, entity extraction, long-document matching, and the indexing and retrieval stages of RAG pipelines.

Liquid AI’s assessment is clear: generation is not the only bottleneck for on-device AI. Many enterprise tasks do not need a large model running on a GPU to generate an answer word by word. They simply need to read through dozens of pages of contracts, code repositories, or support-ticket records at low cost. Keeping these workloads on CPUs often delivers more practical value than compressing yet another chat model.

An Encoder Is Not a “Mini Chat Model”

Developers first need to distinguish among three types of models that are often conflated:

• Generative models: Predict the next token based on existing tokens, making them suitable for question answering, writing, and agents;
• Base encoders: Read the complete input bidirectionally and output contextualized hidden states;
• Embedding or reranker models: Build on encoders and are further trained for vector retrieval, similarity, or ranking tasks.

LFM2.5 Encoders belong to the second category and serve as infrastructure for its downstream applications. Encoders can use information on both sides of a token simultaneously and do not need to maintain the causal mask used in autoregressive generation, making them better suited to “understanding the full text.” However, this also means that a base encoder is not the same as a general-purpose, ready-to-use embedding model.

If a model has not been trained with contrastive learning, an appropriate pooling method, and retrieval data, simply averaging the last-layer hidden states may not produce reliable document vectors. Similarly, classification and named entity recognition generally still require task-specific heads or fine-tuning. What Liquid AI is providing here is a more efficient foundation, not a complete replacement for all downstream adaptation work.

This distinction matters. Over the past two years, many teams have packaged any Transformer capable of outputting vectors as an embedding model, only to encounter lower recall rates and length-dependent drift in vector distributions in production. Running faster solves an engineering problem; whether the vectors are useful still depends on task-specific data.

Why CPU, Specifically?

Cloud GPUs are well suited to high-concurrency generation, but encoder workloads do not always keep GPUs fully utilized.

Consider an enterprise knowledge base. When a new document enters the system, it typically goes through parsing, chunking, encoding, and ingestion. Model calls are interleaved with disk reads, OCR, and database writes, and batch sizes may not remain stable. Keeping a GPU occupied over the long term for this kind of intermittent workload produces poor utilization and unattractive costs.

CPUs, by contrast, are ubiquitous in application servers, database nodes, laptops, and edge gateways. As long as encoding is fast enough, many tasks can be completed directly on existing machines:

1. Documents do not need to leave the local network, making privacy boundaries clearer;
2. There is no need to maintain a separate GPU service, shortening the deployment pipeline;
3. Low-concurrency scenarios do not incur the cost of idle GPUs;
4. Edge devices can continue performing classification, retrieval, and content moderation even when offline.

The problem is that the self-attention computation and intermediate states of traditional Transformers grow rapidly as sequence length increases. The difference may be imperceptible for short sentences, but once inputs expand to thousands or even tens of thousands of tokens, the real CPU constraints are often not just compute, but memory bandwidth, cache hit rates, and the cost of moving intermediate activations.

In other words, long-context processing is not simply a matter of “running short-text computation a few more times.” The longer the sequence, the more likely the CPU is to spend its time moving data around.

The Core of the LFM Approach Is to Stop Making Attention Do Everything

The LFM2 series has consistently emphasized the co-design of architecture, training, and runtime for on-device deployment. The previously released LFM2 family spans dense models from 350M to 2.6B parameters and also includes mixture-of-experts variants. LFM2.5 further strengthens CPU inference, low-precision deployment, and long-context capabilities.

The key idea behind the LFM architecture is to delegate local or compressible sequence mixing to modules with more manageable computational costs, while retaining Attention’s ability to model global dependencies. It does not eliminate Attention entirely; instead, it avoids rereading the full context in the most expensive way at every layer.

Think of it as reading a long report. Traditional full attention is like going back and rereading the entire report after every sentence. A hybrid architecture first uses cheaper mechanisms to process adjacent sections and invokes a global view only when long-range connections actually need to be established.

This design is particularly well suited to encoders. Generative models have two stages: they first prefill the prompt, then decode one token at a time. Encoders do not have the subsequent decoding stage; nearly all of their cost is concentrated in a single complete forward pass. Therefore, the “faster inference” claimed for LFM2.5 Encoders mainly refers to:

• Ingesting an entire long passage more quickly;
• Reducing the intermediate overhead of long-sequence computation;
• Achieving more stable latency on general-purpose CPUs;
• Making it unnecessary to default to GPUs for longer inputs.

This is not the same metric as a generative model’s advertised token output rate. Directly comparing the two is meaningless.

Long-Context Acceleration Cannot Be Judged by a Single Peak-Performance Chart

Liquid AI’s release materials use CPUs and long sequences as the primary benchmark scenarios. This better reflects the product’s positioning than merely showing small-batch peak performance on high-end GPUs. Even so, developers should still pay close attention to the test conditions when evaluating such results.

CPU benchmarks are highly sensitive to variables including:

• Instruction-set and vector-extension support, such as AVX2 and AVX-512;
• Physical cores, logical threads, and NUMA configuration;
• FP32, BF16, INT8, or lower precision;
• Batch size and input-length distribution;
• Whether tokenization, padding, and data copies are included;
• Whether the measurement covers the first run or steady-state latency after warm-up;
• Whether the runtime has fused operators specific to the model.

Official results are therefore more useful for demonstrating the architectural direction than as a substitute for retesting on production hardware. What really matters is not the percentage lead at a particular short-input data point, but whether throughput and memory usage remain manageable as context length increases.

A more reliable local benchmark should cover at least several lengths, such as 512, 2K, 8K, and 16K, while recording p50 and p95 latency as well as peak process memory. The test framework can be organized roughly as follows:

for sequence_length in [512, 2048, 8192, 16384]:
    Construct fixed-length inputs to avoid variation in sample length
    Run several warm-up rounds and exclude them from the statistics
    Measure interactive latency at batch_size = 1
    Measure offline throughput at multiple batch sizes
    Record p50, p95, peak memory, and actual CPU utilization separately
    Validate output values to prevent accuracy anomalies caused by quantization or operator substitution

Another common misconception should also be noted: if business documents will ultimately still be split into 512-token chunks, support for long contexts will not automatically improve retrieval quality. Longer inputs are truly valuable only for cross-paragraph relationships, full-document classification, or long-query–long-document matching.

The Most Direct Use Case Is Local RAG, but It Does Not Replace a Vector Database

The most suitable use cases for LFM2.5 Encoders can be broadly divided into four categories.

1. Long-Document Classification

Contract-type recognition, support-ticket routing, compliance assessment, and email archiving all require reading complete documents. Traditional approaches split documents into multiple sections and then vote on the results, which can lose relationships between sections. A long-context encoder can reduce chunking and directly output document-level representations.

2. RAG Indexing and Retrieval

When data cannot be uploaded to the cloud, a CPU encoder can be deployed on the same node as document parsing and the vector database. For small and medium-sized knowledge bases, this integrated deployment is simpler than maintaining a separate GPU inference service.

However, an encoder does not replace a vector database. The encoder converts text into representations, while the vector database handles storage, filtering, and approximate nearest-neighbor search. They are upstream and downstream components of the same pipeline.

3. Reranking and Long-Query Matching

Search and recommendation systems often retrieve a large set of candidates in the first stage and then use a stronger model to rerank them. Compared with models that encode only individual sentences, an encoder capable of reliably reading longer queries and candidate documents is better suited to technical documentation, regulatory provisions, and code search.

4. Privacy-Sensitive On-Device Tasks

Tasks such as preliminary classification of medical records, enterprise intranet search, and offline content moderation may not permit sending source text to third-party APIs. As long as sufficient memory is available, CPU-optimized models can keep sensitive data on the device or within the private network.

Its Real Competitors Are Not GPT, but Models Like ModernBERT

The competitors of LFM2.5 Encoders are not large chat models, but ModernBERT, long-context BERT variants, the Jina family of encoding models, and various lightweight encoders optimized for retrieval.

Compared with these approaches, Liquid AI has two main differentiators.

First, it treats CPUs and on-device hardware as the starting point for architectural design, rather than training a standard Transformer first and then relying on quantization and runtime optimizations as remedies. For long sequences, the benefits of this kind of hardware-aware co-design are generally more consistent than merely reducing parameter count.

Second, LFM2.5 has already developed into a model family spanning language, reasoning, vision, and audio. Liquid AI previously provided llama.cpp-compatible GGUF builds for LFM2.5 and has continued expanding its on-device ecosystem with MLX, ONNX Runtime, and other platforms. Adding encoders fills in the first half of the “retrieve first, generate second” workflow, helping it build a complete local AI stack.

However, Liquid AI also has an obvious weakness: its ecosystem and task-specific fine-tuning assets still cannot match those of the BERT family. The latter has accumulated a large collection of classification heads, retrieval recipes, distillation methods, and industry datasets. Even if a new encoder is faster at the underlying level, it still needs to prove that its performance can be reproduced consistently across common frameworks, quantization formats, and downstream tasks.

Speed is the ticket to production, not the final answer.

The Five Most Important Things to Check Before Deployment

If you plan to evaluate LFM2.5 Encoders, first verify the following:

1. Whether the task actually requires token representations or sentence vectors. For retrieval, you must verify whether an official version with appropriate pooling and retrieval training is available;
2. The actual distribution of context lengths. Do not let extremely rare, very long documents dictate the entire infrastructure;
3. Task accuracy after quantization. Encoding models are sensitive to changes in vector distributions, so merely verifying that the model runs is not enough;
4. Runtime operator support. The theoretical advantages of a hybrid architecture can be realized only if the inference framework implements efficient operators;
5. End-to-end latency. Tokenization, document chunking, pooling, and vector writes should all be included, rather than measuring only the model’s forward pass.

In addition, CPU inference does not necessarily mean low memory usage. Activations and output tensors for long inputs still consume memory. If hidden states must be retained for every token, the output itself can become very large. Extracting only the CLS token, pooled vectors, or a small number of task-head outputs is much closer to the ideal form of on-device deployment.

Assessment: The Direction Matters More Than the Speedup in Any Single Benchmark

The value of LFM2.5 Encoders lies not in proving that CPUs can replace all GPUs, but in redrawing the boundary between their respective workloads.

Over the past few years, the industry has become accustomed to pushing every AI workload into GPU services: embeddings run on GPUs, reranking runs on GPUs, and classification runs on GPUs as well. This may be reasonable for high-concurrency platforms, but for enterprise intranets, edge devices, and infrequently accessed knowledge bases, such an architecture is often unnecessarily heavy.

Liquid AI is attempting to demonstrate that purpose-built hybrid encoders can return some long-document processing workloads to CPUs. As long as downstream quality does not degrade significantly, this can directly reduce the number of services, network transfers, and idle compute resources.

At this stage, it would be premature to declare it the best long-context encoder based solely on official benchmarks. The model still needs to be tested for retrieval quality, fine-tuning stability, quantization error, and runtime support across multiple platforms. But from a product-direction perspective, this release is more useful than yet another small chat model: before a generative model can answer, someone has to finish reading the material. LFM2.5 Encoders aim to take on precisely this inconspicuous but expensive work.

References

• Hugging Face: LFM2.5-Encoders for Fast Long-Context Inference on CPU: Liquid AI’s announcement covering LFM2.5 Encoders, CPU long-context optimization, and the direction of its benchmarks.
• Zhihu: Liquid AI—The American Story of an On-Device AI Unicorn: Chinese-language background material introducing Liquid AI, the LFM2 hybrid architecture, and its on-device deployment roadmap.