OpenAI Moves Voice Agents into Production: Full Launch of gpt-realtime and Realtime API

On May 7, OpenAI moved the Realtime API from public beta to GA in one go and released its most powerful voice-to-voice model to date, gpt-realtime. The keyword of this update can be summed up in one phrase: production-ready. MCP remote servers, image input, and SIP phone calls are all bundled into the API, with pricing slashed by 20% compared to gpt-4o-realtime-preview.

For developers building voice agents, this is the most substantial update since the public beta of the Realtime API last October—before it was something you could play with, now it's something you can actually deploy.

No Longer an STT + LLM + TTS “Sandwich”

Traditional voice agents string together three components: speech-to-text, large language model, and text-to-speech. The problems are obvious: high latency, lost emotion, and the flattening of non-verbal cues like pauses and laughter in intermediate steps. OpenAI’s Realtime API takes a different path—one single model directly ingests audio and outputs audio, end to end.

The benefits are obvious: a few hundred milliseconds of latency can decide whether a conversation feels like “natural communication” or “a robot hotline.” gpt-realtime pushes quality to a new level on this front. According to official data:

• Big Bench Audio (reasoning): 82.8% accuracy vs. 65.6% in the December 2024 version
• MultiChallenge Audio Benchmark (instruction following): 30.5% vs. 20.6% in the previous generation
• ComplexFuncBench (function calling): 66.5% vs. 49.7% previously

The improvement in instruction following is especially noteworthy. The hardest part of building voice agents isn’t “can it talk,” but “can it say exactly what it should, nothing more, nothing less.” For example, a legal disclaimer in customer service must be read word for word; when confirming order or vehicle numbers, letters and digits can’t be mistaken. OpenAI specifically highlights this— the new model has finer-grained interpretation of system messages and developer instructions, and can even consistently execute tonal cues like “agile yet professional” or “friendly and empathetic.”

Two new voices, Cedar and Marin, are also added and available exclusively in the Realtime API.

Three Capabilities That Truly Change Engineering Implementation

1. Remote MCP Servers

This is, in my opinion, the most important update. By inserting an MCP server URL into the session configuration, the API automatically handles tool calls—no more writing heaps of function-calling glue code.

POST /v1/realtime/client_secrets
{
  "session": {
    "type": "realtime",
    "tools": [
      {
        "type": "mcp",
        "server_label": "stripe",
        "server_url": "https://mcp.stripe.com",
        "authorization": "{access_token}",
        "require_approval": "never"
      }
    ]
  }
}

This means extending a voice agent’s capabilities shifts from “coding integrations” to simply “switching URLs.” Want to add refund functionality to a customer service agent? Point it to the Stripe MCP. Need it to query logistics? Change the server_url. This decoupling greatly boosts back-end development efficiency.

2. Image Input

Users can send an error screenshot or product photo, and the voice agent can converse directly based on the visual content. This brings “multimodality” from demos into real-world use—when users struggle to describe an issue in after-sales scenarios, asking for a picture is always more reliable than verbal explanations.

3. SIP Phone Calls

Direct integration with traditional phone systems—PBX and desk phones can connect. This feature is tailor-made for the B2B customer service market. Early partners like T-Mobile have already started pilot runs on their support lines, while Zillow integrated it into voice property search—according to Zillow, chatting to set filter conditions “feels just like talking to a friend.”

Asynchronous Function Calling: Long Tasks No Longer Block Conversations

A small detail, but crucial for engineering. Previously, a long-running function call would freeze the entire conversation while waiting for results. gpt-realtime natively supports asynchronous function calls— the model can keep conversing while waiting, with no code changes required.

This is almost essential for transactional or query-based voice apps. When a customer asks, “Has my refund arrived?” and the backend needs to query several systems, the AI used to sit in awkward silence for a few seconds; now, it can say, “I’m checking that for you, please hold on. Meanwhile, could you confirm which card you used for payment?”

Pricing and Context Control

Pricing for gpt-realtime is 20% lower than the previous generation:

• Audio input: $32 per million tokens (cached input $0.40)
• Audio output: $64 per million tokens

More importantly, fine-grained context control has been added—developers can set token limits and truncate multiple conversation turns at once. Costs in long-session scenarios (like 10+ minute customer service calls) will significantly decrease. OpenAI didn’t highlight this much, but anyone who’s run real billing knows how valuable it is.

API Call Example

OpenAI Hub now supports gpt-realtime, with direct access available in China, fully OpenAI-compatible using a single API key. Basic session creation:

curl -X POST https://api.openai-hub.com/v1/realtime/client_secrets \
  -H "Authorization: Bearer $OPENAI_HUB_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session": {
      "type": "realtime",
      "model": "gpt-realtime",
      "voice": "cedar",
      "instructions": "You are a polite, professional customer service agent who gives concise answers and always repeats and confirms any monetary amounts."
    }
  }'

WebSocket access follows the same pattern as the official version:

const ws = new WebSocket(
  'wss://api.openai-hub.com/v1/realtime?model=gpt-realtime',
  { headers: { Authorization: `Bearer ${process.env.OPENAI_HUB_KEY}` } }
);

ws.on('open', () => {
  ws.send(JSON.stringify({
    type: 'session.update',
    session: {
      modalities: ['audio', 'text'],
      voice: 'marin',
      input_audio_format: 'pcm16',
      output_audio_format: 'pcm16',
      turn_detection: { type: 'server_vad' }
    }
  }));
});

How to View This Update

Placing this Realtime API GA milestone on a longer timeline, OpenAI’s speech roadmap has gone through three phases:

1. October 2024: Realtime API public beta—proved that the “voice-to-voice” technical path works;
2. March 2025: Release of GPT-4o-transcribe / mini-transcribe / mini-tts—completed the traditional modular stack for developers not ready to go all in;
3. May 2026: GA + gpt-realtime + MCP/Image/SIP—officially targeting production environments.

The voice AI race was never about “who has the flashiest demo,” but “who can make enterprises actually outsource their customer service to AI.” On this front, Google’s Gemini Live and ElevenLabs’ Conversational AI are both strong, but OpenAI has now raised the bar with an MCP ecosystem, SIP telephony, and end-to-end audio model quality.

For developers, the most practical takeaway is: if you previously shelved a voice project due to latency, naturalness, or engineering complexity, now is the time to reevaluate. Especially for customer service, educational copilots, and creator-companion products—these are the three areas OpenAI explicitly emphasized in its blog post, and they’re also the clearest routes to commercialization.

References

• InfoQ: OpenAI Launches gpt-realtime — detailed Chinese analysis, including benchmarks and enterprise pilots