ByteDance Lets Doubao Talk While It Watches

ByteDance Extends Full-Duplex from Voice to Video

On August 5, ByteDance’s Seed team officially released SeedRealtime, a native full-duplex audio-video foundation model. Using a unified architecture that integrates audio, video, and text, the model can interact with continuous multimodal streams in real time—watching, listening, and speaking simultaneously.

SeedRealtime has been fully rolled out in the Doubao App. After updating to the latest version, users can select “Call” in the chat box to enter the video-call interface and try it out. ByteDance says this marks the first deployment of full-duplex audio-video technology in a large-scale consumer product.

The main point of this release is not that “Doubao can finally watch video.” Sending camera footage to a model is nothing new. The real challenge is whether a model can continuously understand a changing scene while listening to what the user is saying, deciding whether it should respond, and continuing to receive external information while generating speech of its own.

In other words, SeedRealtime aims to solve the orchestration problem in real-time interaction, not merely the problem of multimodal recognition.

No Longer “Capture a Frame, Ask a Question, Get an Answer”

Most visual conversation products in the past were still fundamentally turn-based systems. The user took a photo or uploaded an image, the model performed a single inference, and then returned text or speech. Even when connected to a live camera, the backend often sampled frames at a fixed frequency and sent a series of static images to a vision model for processing.

This type of approach is sufficient for tasks such as “What kind of plant is this?” or “Help me read this menu,” but it struggles with scenarios like these:

• The user turns the camera from the table toward the doorway and says, “What was the person who just walked by carrying?”;
• The model is explaining how to operate a device when an indicator light in the frame suddenly turns red;
• The user pauses mid-sentence to think, and someone nearby interjects;
• The model is answering when the user suddenly adjusts the camera or says, “Not this one—look to the left”;
• An object appears in the camera only briefly and has already left the frame by the time the user asks about it explicitly.

All these scenarios require the model to process continuous time rather than isolated image frames. It must know not only what is in the scene, but also what happened first and what happened later, as well as which segments of visual information correspond to expressions such as “just now,” “that one,” and “the second one from the left.”

SeedRealtime’s approach is to natively integrate audio, video, and text within a unified architecture, allowing the model to directly process continuous multimodal streams. Based on the information released by ByteDance, its capabilities can be divided into three areas: joint audio-video understanding, proactive interaction, and real-time control over conversational pacing.

Vision and Speech Must Share the Same Timeline

Joint audio-video understanding may sound like a familiar multimodal capability, but in real-time scenarios, it is first and foremost a temporal alignment problem.

For example, suppose a user points the camera at a row of products and says, “Help me take a look at this ingredients list.” If audio transcription, video frame sampling, and language inference are handled by separate modules, the system must additionally determine which frame and which product “this” refers to. Even slight discrepancies caused by network jitter, sampling frequency, or module latency can misalign the reference.

Chinese also contains many homophones. Based on speech alone, the model may be unable to determine which word the user said. The ambiguity may only be resolved by combining the speech with the objects visible through the camera and the actions being performed. Conversely, visual information may itself be ambiguous and require the user’s tone, context, and ongoing actions to complete the meaning.

This is the significance of SeedRealtime’s emphasis on native joint modeling: audio, video, and text do not sequentially enter three unrelated pipelines, but are understood within a shared context. The model sees more than image tokens and hears more than ASR transcripts; it must also preserve the temporal correspondence between the two.

The difference from a traditional cascaded approach is similar to having one person observe and communicate on-site rather than assigning three people to watch surveillance footage, listen to recordings, and write summaries, then combining their notes afterward. The latter is easier to develop and makes modules easier to replace, but every additional transcription step, queue, and protocol handoff introduces latency and information loss.

Of course, a “unified architecture” does not mean the server has no separate components at all. A real product still needs video encoding, network transmission, session management, safety review, audio playback, and tool-calling systems. The key question is whether the model responsible for core understanding and generation can jointly model audio-video timing, rather than simply inserting video descriptions and speech transcripts into a text context window.

The Hardest Part of Full-Duplex Is Not Low Latency, but Knowing When to Stay Silent

Full-duplex is often simplified as the ability for a model to listen and speak at the same time. That definition is not wrong, but it tells only half the story.

Traditional half-duplex voice assistants work more like walkie-talkies: the user finishes speaking, the system detects silence, and then it begins generating an answer. While the model is speaking, the input channel may be attenuated or even closed. A full-duplex system is more like a telephone call: both input and output channels remain open at all times, the user can interrupt whenever needed, and the model can pause, correct itself, or continue based on new information.

But having both channels open at once does not automatically make a conversation natural. The system must also continuously make a series of implicit judgments:

1. Is the current voice coming from the primary user, another person, or speaker echo?
2. Is the user merely pausing briefly, or have they finished speaking?
3. Does the user’s “mm-hmm” indicate agreement, or are they preparing to continue?
4. Is a new visual event important enough to interrupt the current response?
5. Should the model immediately stop the speech currently being played?
6. Should background television, conversations among passersby, and ambient noise be ignored?

The experience of a real-time model therefore depends on more than time to first token. A system that responds extremely quickly but constantly talks over users whenever they pause is often harder to use than one that is half a second slower but has better conversational judgment.

According to ByteDance’s end-to-end human evaluation, SeedRealtime reduced audio-video conversational pacing issues by half compared with cascaded models. These issues included interrupting users before they had finished speaking, failing to respond promptly after they had finished, and being falsely triggered by background noise or nearby conversations. The probability of completing a single conversation smoothly and without disruption also improved.

It should be noted that ByteDance has not yet disclosed the complete evaluation set, sample size, latency distribution, or detailed results across different network environments in its public materials. The claim that “issues were reduced by half” is therefore better viewed as a vendor test result rather than a general conclusion that has been independently reproduced.

Nevertheless, the metrics it selected are the right ones. If real-time interactive models continue to be compared only on question-answering accuracy, it becomes easy to overlook the factors that truly affect product usability. Rates of talking over users, false interruptions, effective interruption latency, false triggers caused by background speech, and audio-video synchronization drift during long sessions are more meaningful metrics for this type of model.

From Passively Answering to Proactively Monitoring and Alerting

Another important capability of SeedRealtime is proactive interaction. The model can continuously perceive changes in the environment, proactively speak when a key object appears or the visual state changes, and incorporate tool-call results into its spoken output.

This will change how visual assistants are used.

In the past, users generally had to ask an explicit question before the model began processing: first point the camera at a device, then ask, “Was this installed incorrectly?” A proactive model, by contrast, can continuously observe the process and immediately alert the user when it detects a part facing the wrong direction, a pot about to boil over, or a target object entering the frame.

Practical applications include:

• Equipment maintenance: Field personnel can keep their hands free while the model continuously checks each step through the camera and alerts them when an anomaly appears;
• Accessibility assistance: Identify roads, steps, approaching vehicles, and changes in signage rather than waiting for the user to ask about each item;
• Remote instruction: Observe writing, experiments, or musical instrument practice and provide immediate feedback in response to changes in movement;
• Shopping and navigation: Continuously offer suggestions based on products and locations visible through the camera, combined with the user’s preferences;
• At-home companionship: Respond according to real-time progress while reading picture books, building with blocks, or doing crafts together.

Tool calling can take this type of interaction even further. After seeing a flight number, product, or location, the model can look up real-time information and then continue the conversation in natural speech. Ideally, the user would not need to leave the video interface or formulate a new, complete instruction.

But proactivity is a double-edged sword. If the model comments on every visual change, it will quickly go from being an assistant to a nonstop surveillance monitor. Developers must design not only “when to trigger,” but also event priorities, cooldown periods, suppression of repeated alerts, and user-configurable levels of proactivity.

Proactive vision also increases privacy and compliance pressures. Continuous camera use means the footage may capture bystanders, screen content, home addresses, identity documents, and other sensitive information. Consumer deployment cannot focus solely on model capabilities; it must also address whether video is uploaded to the cloud, how long it is retained, whether it is used for training, and whether users can clearly tell when the model is actively observing.

A Visual Extension of Seeduplex—and a More Difficult Step

In April this year, ByteDance released Seeduplex, a native full-duplex speech model, and launched it in the Doubao App. That version focused on continuous listening, interference resistance, and dynamic end-of-turn detection: the model could continue listening to user input while generating speech and use both acoustic and semantic information to determine when to respond.

SeedRealtime can be seen as an extension of this technical approach into video, but adding video is not as simple as connecting one more input stream.

Audio typically arrives continuously at a high frequency, while video involves resolution, frame rate, encoding, and visual-token costs. If the model observes too sparsely, it will miss brief events. If it processes every high-resolution frame, inference costs and end-to-end latency will rise rapidly. The system must strike a balance among visual sampling, context compression, and event retention.

Long video calls also cause the context to expand. The model cannot retain every frame indefinitely. It must compress past audio and video into memory while preserving key information that may be referenced later. Deciding which details to forget and which events to retain in long-term context is itself a core challenge for real-time multimodal systems.

In addition, speech generated by the model may re-enter the microphone through the device’s speaker. In video scenarios, mirrors, content playing on screens, and multi-person environments introduce additional interference. On-device echo cancellation can address part of the problem, but the system must still determine at the semantic level whether a particular utterance is actually directed at the model.

SeedRealtime’s value therefore lies not in presenting a flashier demo, but in ByteDance placing the system directly into Doubao, where it will be tested under real-world network conditions, across complex devices, and at massive concurrency. There is a wide gap between getting a model to work and making it available to hundreds of millions of users—a gap filled by streaming-media transmission, GPU scheduling, session-state management, fault tolerance, and cost control. For real-time models, the engineering system itself is part of the product’s capabilities.

For Developers, the API Matters More Than the App Demo

As of August 5, ByteDance had confirmed that SeedRealtime had been fully rolled out in the Doubao App, but public information had not yet specified when a standalone API would become available, how it would be priced, its context specifications, supported video encoding formats, or concurrency limits. This article therefore does not provide a usage example.

If it is later made available through Volcano Engine, developers will need to pay attention not only to the model name, but also to the entire real-time protocol and control interface, including:

• Whether it is based on WebSocket, WebRTC, or another bidirectional streaming protocol;
• Audio sample rate, video resolution, and recommended frame rate;
• Whether server-side voice activity detection and interruption policies are configurable;
• Whether input audio transcripts, model output text, and event timestamps are provided;
• Whether proactive observation can be configured with trigger conditions based on task, region, or target type;
• Whether the system can continue receiving and understanding audio and video during tool calls;
• Whether unplayed audio can be canceled after an interruption and the generation state rolled back;
• Retention and data-use policies for video, audio, and transcription logs;
• Time to first packet, ongoing transmission fees, and visual-token pricing rules.

This type of model is also difficult to integrate like an ordinary text model through a JSON request compatible with OpenAI Chat Completions. Text interfaces revolve around one request and one response, while full-duplex systems require the client to continuously upload audio and video, receive incremental audio, and send interruption, mute, tool-result, and session-state events at any time.

Even if aggregation platforms provide unified access in the future, the compatibility layer is more likely to be based on the Realtime API or a WebRTC session protocol than on a traditional chat-completions interface. Whether OpenAI Hub can integrate SeedRealtime will likewise depend on when ByteDance provides a public API and real-time streaming protocol. Until an official interface is announced, claiming that it is already supported would be premature.

Assessment: Real-Time Multimodality Is Shifting from “Understanding” to “Getting Along”

SeedRealtime is not the first model capable of processing audio and video, and “watching while chatting” is not an entirely new product concept. What makes it more noteworthy is that it brings three problems that were previously optimized separately into a single real-time system: continuous audio-video understanding, concurrent bidirectional communication, and proactive expression.

This means the evaluation criteria for multimodal competition are changing. In the past, the question was whether a model could recognize an image or summarize a video. Going forward, the question will be whether it can maintain attention over interactions lasting several minutes or even tens of minutes, remember what just happened, and say the right thing at the right time.

ByteDance’s advantage is also clear: Doubao provides direct access to a massive user base, allowing it to quickly gather feedback on conversational pacing in real-world scenarios. Fully deploying the model in a consumer app carries more weight than releasing only a research demo.

At this stage, however, three information gaps remain: there are no reproducible public evaluations, key latency and cost data have not been disclosed, and the developer API remains unclear. SeedRealtime has already demonstrated ByteDance’s willingness to bring full-duplex audio-video technology into products, but whether it can become foundational industry infrastructure will depend on how openly it is made available.

In the short term, this technology is most likely to improve video companionship, real-time education, and visual assistance. Its longer-term impact will be felt in smart glasses, in-vehicle assistants, and embodied AI. These devices will not wait for users to take a photo before asking a question; they must continuously observe the world and collaborate with users as they act.

From this perspective, SeedRealtime is not about “adding a camera to a chatbot.” It is an attempt to establish a new interaction paradigm: AI no longer waits for a complete instruction in a text box, but instead shares the same unfolding moment in time with the user.

References

• ITHome: ByteDance Releases SeedRealtime Full-Duplex Audio-Video Foundation Model — Introduces the model’s unified audio-video architecture, three core capabilities, human-evaluation results, and the entry point for trying it in the Doubao App.