Apple SpeechAnalyzer hands-on test: Local ASR goes head-to-head with Whisper for the first time

This Time Apple Really Wants to Take Whisper’s Turf

In early July, a benchmark report from the Inscribe team spread through the developer community. They took the newly released SpeechAnalyzer API from iOS 26 / macOS 26 and compared it head-to-head against OpenAI’s Whisper series and Apple’s own previous SFSpeechRecognizer — it’s the most systematic third-party evaluation of SpeechAnalyzer since its debut at WWDC25.

The conclusion was even more striking than expected: in the long English transcription scenario — Whisper’s best field — SpeechAnalyzer’s word error rate (WER) is already comparable to Whisper Large-v3, while inference speed is nearly an order of magnitude faster, still running fully on-device. For developers building voice note, meeting transcription, or podcast tools, this effectively rewrites the answer to “which ASR to choose for iOS.”

A Replacement for an API Neglected for Ten Years

Let’s set the background. Apple’s Speech framework actually dates back to iOS 10; SFSpeechRecognizer was many veteran iOS developers’ first exposure to ASR. But the API had some long-standing pain points:

• Initially it was a cloud-based service, requiring a network connection and subject to quota limits—effectively unusable in privacy-sensitive contexts;
• Later it added an on-device mode via supportsOnDeviceRecognition, but the model’s quality lagged far behind Whisper, especially for proper nouns, accents, and long sentences;
• The API itself was old-school Objective-C style—callbacks, delegates, and state machines galore, at odds with modern Swift asynchronous patterns.

As a result, in recent years nearly every serious iOS app implementing speech features integrated Whisper.cpp, MLX-Whisper, or directly called OpenAI’s /audio/transcriptions. In most developers’ minds, Apple’s own Speech framework was “usable but undesirable.”

At WWDC25, Apple announced that SpeechAnalyzer would completely replace the old API and would be used behind system apps like Notes, Voice Memos, and Journal. The message was clear: Apple knows that if it can’t win the on-device AI battle, features like real-time captions on Vision Pro, AirPods Live Translation, and Apple Intelligence voice interaction just won’t work.

What the API Looks Like: Swift‑First, Modular, AsyncSequence‑Based

From a coding perspective, SpeechAnalyzer is a truly modern Swift API. The entire data flow revolves around AsyncSequence, and the recognition process is modularized into SpeechModules — SpeechTranscriber handles transcription, SpeechDetector handles endpoint detection, and future modules (like speaker separation or emotion analysis) could be added, though not yet officially opened.

A minimal working transcription flow looks like this:

import Speech

func setUpTranscriber() async throws {
    let transcriber = SpeechTranscriber(
        locale: Locale(identifier: "zh-CN"),
        preset: .progressiveTranscription
    )

    // Ensure the model is installed on device
    guard await SpeechTranscriber.supported(locale: transcriber.locale) else {
        throw TranscriptionError.unsupportedLocale
    }
    if await !SpeechTranscriber.installed(locale: transcriber.locale) {
        try await AssetInventory.download(for: [transcriber])
    }

    let analyzer = SpeechAnalyzer(modules: [transcriber])
    let (inputStream, continuation) = AsyncStream.makeStream(of: AnalyzerInput.self)

    try await analyzer.start(inputSequence: inputStream)

    // Consume transcription results
    for try await result in transcriber.results {
        print(result.text, result.isFinal)
    }
}

Some noteworthy details:

1. Models are no longer bundled in the system but downloaded per language as needed, managed through AssetInventory. Unlike Whisper’s “one multilingual model for all,” Apple chose a specialized model per language — smaller packages, faster loading.
2. AnalysisContext allows domain-specific vocabulary injection, such as medical terms, proper nouns, or code snippets — equivalent to Whisper’s initial_prompt, but more structured.
3. Real‑time and batch modes share one interface — feed an AVAudioFile into analyzeSequence for batch, or an AsyncStream for streaming. The old API’s fragmentation between live and offline modes is gone.

Benchmark Details: Not a Total Sweep, But an Incredible Trade‑off

Going back to Inscribe’s comparison, they tested across LibriSpeech test-clean and real meeting audio; several key metrics stand out:

• English WER: SpeechAnalyzer achieved ~3.2% on test-clean, Whisper Large-v3 ~2.8%, Whisper Small ~4.5%. So Apple’s on-device model sits between Whisper Small and Large, leaning closer to Large.
• Speed: On an M2 MacBook Air, transcribing a 60-second clip took ~0.8 seconds with SpeechAnalyzer versus ~6 seconds for Whisper Large-v3 (MLX version) — over 7× faster. The gap on iPhone 15 Pro was even wider.
• Memory: SpeechAnalyzer’s peak memory usage stayed under 500 MB, while Whisper Large’s model weights alone start at 3 GB.
• Old API Comparison: The on-device SFSpeechRecognizer registered about 8% WER — far behind the new API.

Its weaknesses are also clear. Chinese, Japanese, and certain smaller languages still trail Whisper Large, especially in accented or multi‑speaker conditions. Also, SpeechAnalyzer doesn’t yet handle translation (Whisper can directly translate from any language → English) and doesn’t expose timestamp-level confidence scores, so you’d need custom logic for subtitle alignment.

The Verdict: Not a Leaderboard Stunt, but a Scenario Match

Apple played this one smart. It’s not aiming to outdo Whisper in “every language, every scenario, highest accuracy,” but is laser‑focused on the on‑device, low‑latency, common-language segment — precisely the sweet spot for native iPhone, iPad, and Mac apps.

Developers can almost rewrite their selection logic:

• For native iOS/macOS apps covering English plus major European languages, SpeechAnalyzer is the new default — zero cost, zero network, zero privacy worries, good enough results.
• For cross‑platform or ultra‑high‑accuracy, multilingual, translation needs, Whisper or cloud ASR remain essential.
• For Chinese‑heavy use cases, don’t expect SpeechAnalyzer to shine yet; running Whisper Large‑v3 locally or using a cloud recognizer (Volcengine, iFlytek, or cloud Whisper) is still more reliable.

It’s worth noting that SpeechAnalyzer and the Foundation Models framework (also new in iOS 26) were designed to work together: speech goes into SpeechAnalyzer, transcribed text goes straight into the on‑device Apple Intelligence model for summarization, rewriting, or intent detection — the entire pipeline stays on‑chip. This “local ASR + local LLM” combo may be Apple’s most practically valuable developer‑facing update this year, far more tangible than the yet‑to‑ship “new Siri experience.”

Another Battle with Cloud ASR

Zooming out, SpeechAnalyzer’s progress also encroaches on cloud ASR territory. Many products once chose cloud transcription simply because on‑device ran too slowly or poorly; that justification is weaker in Apple’s ecosystem now.

Of course, the cloud still has its strengths — cross‑platform consistency, maximal accuracy from huge models, tight coupling with LLMs. Most teams now adopt hybrid deployments: run locally when possible, fall back to the cloud for unsupported languages or complex audio. For these setups, a unified API gateway saves effort: aggregators like OpenAI Hub let you call Whisper, GPT, Claude, Gemini, etc., with one key, OpenAI‑compatible format, direct mainland access — a pragmatic “fallback channel” for on‑device models.

A Few Unanswered Questions

The community still has several open points worth watching:

• When will the Chinese model catch up? Apple gave no timeline at WWDC25, but judging from the Notes app’s real‑time Chinese transcription, their internal version may already be stronger.
• Speaker diarization: The SpeechModule protocol clearly leaves hooks for extra modules — can third parties implement and attach their own diarization modules? Documentation is silent.
• Privacy and enterprise compliance: Fully on‑device processing means audio never leaves the device — a major plus for healthcare, legal, and finance sectors. But can enterprise‑distributed apps pre‑install models and skip user downloads? We’ll need to see MDM policy support.

For teams building iOS speech tools, the next couple of months will likely bring a wave of “migrating to SpeechAnalyzer” updates. Veteran tools like Just Press Record, AudioPen, and various AI meeting assistants — whoever switches first will gain big on battery life and UX. For the Whisper camp, it’s not the end, but it is a signal: Apple is finally back in the on‑device ASR game.

References

• huggingface.co/openai/whisper-large-v3 — Official Whisper Large‑v3 model card, one of the comparison benchmarks
• github.com/openai/whisper — Official Whisper repo, reference for benchmark reproduction
• github.com/ml-explore/mlx-examples — Apple MLX Whisper examples, basis for Mac‑side performance comparison