Open-Source Offline Transcription Arrives on iPhone

One iPhone Takes On the Entire Speech Processing Pipeline

As of August 5, developers have recently released the open-source iOS app LiveTranscriber. It does more than simply bring Whisper to a phone. Instead, it attempts to complete a relatively comprehensive speech workflow entirely on an iPhone: recording, real-time recognition, multi-speaker differentiation, translation, summarization, keyword extraction, and history search—all without relying on cloud inference.

The currently announced combination of local models includes:

• Whisper: general-purpose offline transcription;
• Qwen3-ASR: multilingual speech recognition;
• NVIDIA Nemotron Streaming: low-latency streaming transcription;
• MOSS Multi-Speaker: meeting transcription with speaker labels;
• Qwen3: on-device generation of summaries, titles, and key points, as well as analysis of transcribed text.

The app also supports automatic syncing of Apple Watch recordings, model downloads and switching, real-time translation, and searchable transcription history.

What truly deserves attention here is not that “the iPhone can run yet another model,” but that multiple open-source speech models are beginning to be organized into a usable mobile product. There have been plenty of on-device ASR demos over the past two years. What remains rare is a project willing to tackle the less glamorous issues that determine whether a product is actually usable: model routing, memory reclamation, streaming audio, foreground-background transitions, and mobile interaction design.

It Does Not Run Four Models at Once—Each Has Its Own Job

Based on the feature list, the most sensible design for LiveTranscriber is not to keep Whisper, Qwen3-ASR, Nemotron, and MOSS resident in memory simultaneously, but to use them as switchable backends for different tasks.

For example, when organizing a previously recorded interview, users can prioritize an offline model with higher accuracy but not necessarily the fastest processing speed. For real-time captions, they can switch to the natively streaming Nemotron Streaming model. In multi-person meetings, MOSS adds value by directly incorporating “who said what” into the output. Cross-language recordings can be handled by Qwen3-ASR, followed by summary generation using a local language model.

At a high level, the pipeline can be understood roughly as follows:

Microphone / Apple Watch recording
        ↓
VAD and audio chunking
        ↓
Ring buffer + streaming feature extraction
        ↓
Select ASR based on the scenario
  ├─ Whisper: general-purpose offline transcription
  ├─ Qwen3-ASR: multilingual recognition
  ├─ Nemotron: real-time streaming captions
  └─ MOSS: multi-speaker transcription and speaker labels
        ↓
Timestamps, segmentation, and text cleanup
        ↓
Local Qwen3: summaries / titles / key points / translation
        ↓
Local database indexing and search

This is far more complex than “installing a Whisper model and calling it.” Streaming recognition requires the app to continuously receive audio chunks while retaining sufficient context. Speaker recognition must maintain identity consistency throughout long recordings. The summarization model further consumes unified memory. If any layer runs out of control, iOS may terminate the process, the device may heat up rapidly, or caption latency may gradually accumulate from one or two seconds to the point of becoming unusable.

The LiveTranscriber developers have likewise focused the engineering challenge on “how to make models usable on an iPhone,” rather than whether they can perform a single inference. That assessment is accurate. Getting a mobile chip to output a piece of recognized text is no longer unusual. The difficult part is ensuring that, after running continuously for half an hour, the app still has not slowed down noticeably, exhausted its memory, or drained the battery.

The Bottleneck for On-Device Speech Has Shifted From “Can It Run?” to “How Should It Be Scheduled?”

Apple Silicon’s CPU, GPU, and Neural Engine share unified memory. This provides a clear advantage for on-device inference: developers do not need to move data around as frequently as they would in an environment with a discrete GPU. But sharing also means that model weights, audio buffers, KV caches, UI resources, and system processes all compete for the same pool of memory.

For an app like LiveTranscriber, there are at least four practical issues.

1. Models Cannot Remain Resident Indefinitely

If the ASR and local language models are loaded simultaneously, peak memory usage will rise quickly. A more practical approach is to load them by stage: retain only the streaming ASR model during recording, then unload the speech model and load Qwen3 after the user taps “Generate Summary.”

The ability to download and switch models is not merely a UI feature; it is also a resource-management strategy. Allowing users to choose between accuracy, latency, and storage usage is more honest than pretending that a single model is suitable for every device.

2. Whisper Is Not Inherently Suited to True Real-Time Streaming

Whisper was originally designed more like an architecture that processes audio windows in batches. Many so-called real-time implementations essentially keep extracting the latest segment of audio, decoding it repeatedly, and revising the previous result. This approach is mature and benefits from a rich model ecosystem, but it introduces redundant computation and may cause the text displayed on screen to be repeatedly rewritten.

This is where natively streaming models such as Nemotron Streaming matter. They are better suited to producing text as someone speaks and can use incremental state to avoid recomputing everything from scratch in each round. The trade-off is that their language coverage, final accuracy, and maturity of mobile adaptation may not comprehensively surpass Whisper.

Keeping multiple models available, rather than declaring that a new model has completely replaced Whisper, is therefore a more pragmatic product decision.

3. Multi-Speaker Transcription Is Not Ordinary ASR With an Extra Label Column

The hardest part of producing meeting minutes is often not understanding what was said, but identifying the speaker. Traditional solutions usually perform ASR first, then run a separate speaker diarization or clustering process, and finally align the timestamps. The longer the pipeline, the more easily errors accumulate: even if a sentence is recognized correctly, the meeting minutes are still unusable if it is assigned to the wrong person.

MOSS Multi-Speaker focuses on transcription with speaker information, aiming to shorten this processing chain. For interviews, classrooms, and multi-person meetings, this has more product value than merely reducing the word error rate by another small margin. However, developers still need to address complex scenarios such as overlapping speech, far-field audio capture, and new speakers joining midway through a session. Distinguishing two people in a demo does not mean the system will also work reliably in a ten-person conference room.

4. Battery Life and Temperature Are the Real Mobile Benchmarks

Desktop tests commonly examine real-time factor, time to first token, and word error rate. On an iPhone, sustained power consumption, thermal throttling, background survival time, and battery drain per minute must also be measured.

A model that can process five minutes of audio at 0.5× real-time does not necessarily suit continuously enabled real-time captions. Once the device heats up, the system may reduce its clock speed. Recording and inference permissions may also change when the app moves into the background. Automatic Apple Watch syncing likewise involves file transfers, resumable transfers, and state consistency across multiple devices—not simply copying audio to the phone.

Whether LiveTranscriber truly meets the standard of a “practical product” will ultimately depend on long-duration recording tests, not a demonstration lasting a few dozen seconds.

Compared With Existing Whisper Apps, It Goes Two Steps Further

Running Whisper offline on an iPhone is nothing new. Several apps already support local recording transcription, file imports, search, and offline translation. The open-source community has also long demonstrated native iOS apps running Whisper Base, Small, and even Medium models.

LiveTranscriber differs in two main respects.

First, it does not treat Whisper as the only backend. Instead, it acknowledges that different models have distinct roles in real-time, multilingual, and multi-speaker scenarios. For developers, this kind of “model routing layer” is more instructive than a single-model interface, because mobile AI products are moving from integrating one model to managing a collection of models with different capabilities.

Second, it also moves post-ASR text processing onto the device. A typical offline transcription app may not upload audio, yet could still send transcribed text to the cloud to generate a summary. By using local Qwen3 to process titles, key points, and content analysis, LiveTranscriber truly completes the privacy loop.

This is especially valuable for medical consultations, attorney interviews, corporate meetings, unpublished interviews, and closed-network environments. The privacy it provides is not based on a “service provider’s promise not to retain data”; rather, the data physically never leaves the device.

Of course, fully offline operation comes with clear costs. Models occupy several gigabytes of storage, older iPhones may lack sufficient speed or memory, and summaries generated by small on-device models are generally not as good as those produced by large cloud models. Local summaries may already be sufficient for ordinary meetings, but when precise decisions and specialized terminology are involved, users still need to verify the original recordings and verbatim transcripts.

The Value of Open Source Goes Beyond Being Free

LiveTranscriber’s greater significance lies in providing a deconstructable example of an on-device iOS speech product.

In the past, developers building similar apps often had to separately find Core ML or MLX models, handle microphone permissions, implement audio chunking, adapt different decoders, and then independently solve database and summarization-model issues. Apple Silicon speech toolchains such as speech-swift have already consolidated models including Qwen3-ASR, Whisper, MOSS, Parakeet, and Nemotron behind similar Swift interfaces. LiveTranscriber goes a step further by demonstrating how these capabilities can be incorporated into a user-facing application.

The direct value of projects like this for developers includes:

• Learning how streaming audio is passed into different ASR backends;
• Observing how tasks are allocated among Core ML, MLX, and the Neural Engine;
• Learning about model downloading, verification, switching, and version management;
• Studying model unloading and caching strategies under memory pressure;
• Reusing local transcription history, full-text search, and summarization pipelines;
• Testing recording synchronization between Apple Watch and iPhone.

However, “source code available” does not automatically mean it can be used commercially without restrictions. Teams planning to build products based on it must still individually review the licenses for the application code, model weights, converted artifacts, and third-party dependencies. In particular, converting a model to Core ML or a quantized format generally does not eliminate the usage restrictions attached to the original weights.

Our Assessment: The Direction Is Sound, but the Experience Depends on Worst-Case Performance

LiveTranscriber is a more meaningful open-source project than a single-model demo. What it validates is not whether a particular ASR model can produce text on an iPhone, but whether an on-device speech app can form a complete closed loop from capture to analysis.

At this stage, however, it would be premature to view it as a comprehensive replacement for cloud speech APIs based solely on its feature list. The advantages of on-device solutions center on privacy, offline availability, and zero marginal inference cost. Cloud solutions still make it easier to provide stable large-model capabilities, cross-device synchronization, stronger adaptation to specialized vocabularies, and server-side elasticity.

What will truly determine whether LiveTranscriber endures are the metrics that do not easily appear in demonstration videos:

1. Temperature, battery consumption, and latency after one hour of continuous recording;
2. The minimum viable configuration across different iPhone generations;
3. Accuracy with mixed Chinese and English, accents, noise, and far-field recordings;
4. Whether speaker labels remain stable during overlapping multi-speaker speech;
5. Whether alternating between ASR and Qwen3 introduces noticeable waits or memory spikes;
6. Whether the licenses for the models and application code permit commercial integration.

If these issues are handled well, LiveTranscriber will represent more than an offline recording app. It will embody a new form of mobile AI product in which models are not remote interfaces, but local components that can be downloaded, switched, and scheduled by the operating system.

Whisper made on-device transcription possible. Qwen3-ASR, Nemotron, and MOSS are beginning to break it down into more specific capabilities, such as multilingual support, low latency, and multi-speaker meetings. What LiveTranscriber does is assemble these components into an iPhone in a relatively complete way for the first time. The direction is right; what remains is a test of engineering endurance.

References

• Reddit: LiveTranscriber project introduction: The developer introduces the project’s goals, supported models, offline features, and mobile engineering challenges.
• GitHub: speech-swift: A native speech toolchain for Apple Silicon that lists support for models including Qwen3-ASR, Whisper, MOSS, and Nemotron Streaming.
• GitHub: ScribeAI and Whisper iOS discussion: An earlier implementation of on-device Whisper for iOS, useful for comparing the evolution of features such as real-time transcription, model sizes, and local search.