Gemini Robotics ER 2 Released: Robots Now Not Only Understand Their Surroundings, but Also Work as a Team

Google DeepMind recently released Gemini Robotics ER 2. Compared with the previous-generation Gemini Robotics-ER, which focused on spatial understanding and embodied reasoning, the new model pushes the capability boundary one step further: it must not only understand what is in front of it, but also determine what is happening from video, decide what to do next, and divide a complex task among multiple robots so they can complete it collaboratively.

This is a highly purposeful upgrade.

Over the past year, the most active area in embodied AI has been VLA (vision-language-action) models: cameras provide visual input, and the model directly generates actions for a robotic arm or robot. But in real factories, warehouses, and homes, motion control is only the final step. More often, robots fail not because a gripper rotated two degrees too little, but because they never understood the task's progress, did not know which device to invoke, or caused multiple robots to become stuck in the same passageway.

Gemini Robotics ER 2 targets precisely this system layer between perception, planning, and execution.

From "Seeing a Frame" to "Understanding a Process"

The most notable upgrade in ER 2 is video understanding.

An image can tell a robot that there is a cup, a plate, and a cloth on the table, but it has difficulty answering who just put down the cup, whether the plate has already been washed, which area the cloth has wiped, or whether the current action has deviated from the original plan. For a robot, video is not a collection of independent images, but a sequence of state changes with causal relationships.

This distinction is critical.

Consider warehouse sorting. A single-frame vision model can identify a cardboard box on a conveyor belt and estimate where to grasp it. A model that understands video may go further by determining which branch the box came from, whether the previous robotic arm failed to pick it up, whether the conveyor is congested, and whether it should continue picking or pause the equipment first.

The same applies to household scenarios. If a user says, "Clean up the things the child just knocked over," the robot must first locate the event in the video context, distinguish between items that were already on the floor and those that just fell, and then generate an order for tidying them up. Object detection alone is nowhere near enough.

By combining ER 2's video capabilities with embodied tasks, DeepMind is signaling that the model's focus is not merely video question answering, but extracting actionable state from continuous observation:

• Which subtasks have already been completed;
• What changed in the environment during execution;
• Whether the current plan is still valid;
• Whether replanning or other tools are required;
• What humans or other robots are doing.

This is closer to the "process memory" that robots truly need. If a traditional vision module is like a constantly refreshed surveillance snapshot, ER 2 aims to understand the entire surveillance recording and make scheduling decisions at critical points.

Task Orchestration Is Closer to a Production System Than "Knowing How to Perform Actions"

The second focus of ER 2 is task orchestration.

A real-world task is rarely completed through a single model inference. Consider the task of "cleaning a conference room and restoring its layout." The system may need to invoke, in sequence, mobile-base navigation, robotic-arm grasping, access control, object recognition, cleaning equipment, and an inventory database. Exceptions may also occur along the way: a chair may block the passage, the trash bin may be full, a cup may still contain liquid, or a robot may be charging.

These problems are fundamentally similar to workflow orchestration in software systems. A large model does not personally execute every low-level instruction. Instead, it decides how to decompose the task, in what order to invoke tools, and which branch to follow after a failure.

The previous-generation Gemini Robotics-ER already demonstrated strong spatial understanding, state estimation, planning, and code generation capabilities. DeepMind said at the time that its success rate on end-to-end robot control tasks could reach two to three times that of Gemini 2.0. When code generation was insufficient to solve a problem, it could also perform in-context learning from a small number of human demonstrations.

ER 2 continues along this path, but expands its focus from "generating an executable plan for one robot" to "organizing an entire robotic system to complete a task." This means the model's output should not merely be a natural-language plan, but a structured task graph that can be tracked, verified, and retried.

Ideally, the content generated by the orchestration layer would resemble the following abstract structure:

{
  "goal": "Clean the workbench and resume production",
  "tasks": [
    {"id": "inspect", "worker": "camera_robot", "depends_on": []},
    {"id": "remove_obstacle", "worker": "arm_robot", "depends_on": ["inspect"]},
    {"id": "clean_surface", "worker": "cleaning_robot", "depends_on": ["remove_obstacle"]},
    {"id": "verify", "worker": "camera_robot", "depends_on": ["clean_surface"]}
  ]
}

The real difficulty is not generating this JSON, but ensuring that it matches the real-world state: whether the robotic arm can reach the obstacle, whether the cleaning robot has enough battery power, whether someone enters the workspace during execution, and which step to roll back to if verification fails.

Therefore, ER 2's value cannot be judged solely by whether it successfully completes a task in a model demonstration. More important metrics include long-horizon task success rate, exception recovery rate, state-tracking accuracy, task latency, and whether incorrect decisions can be intercepted by the safety system in time. Ultimately, robot products compete on system reliability, not on a single polished demo.

Multi-Robot Collaboration Is Not About Having Two Robotic Arms Chat with Each Other

Multi-robot collaboration is the most imaginative—and the easiest to overestimate—part of this release.

Having multiple robots complete a task together involves far more than simply "assigning each robot something to do." It requires addressing at least four issues: capability discovery, task allocation, spatiotemporal coordination, and failure takeover.

Different robots have different sensors, payload capacities, operating ranges, and control interfaces. A wheeled robot may excel at transportation but be unable to retrieve a box from the top shelf. A robotic arm may perform precise grasping but be unable to leave its fixed workstation. A drone can conduct inspections quickly, but has limited endurance and payload capacity. The orchestration model must first understand each robot's capability boundaries and then assign work according to the current conditions.

Spatial conflicts are equally troublesome. Two robots may each execute a reasonable local plan, yet block each other when those plans are combined. A multi-robot system therefore cannot simply invoke multiple agents in parallel. It must maintain a shared world state and handle resource locks, priorities, and communication latency.

From this perspective, ER 2 is more like a "scheduling brain" for robotics than a universal model intended to replace every controller. It is responsible for understanding tasks, decomposing goals, and coordinating execution, while lower-level motion planners, collision detection, real-time controllers, and safety rules must still provide safeguards.

This layered approach is actually more realistic. Large models excel at handling open-ended semantics, ambiguous instructions, and long-horizon planning, but they are ill-suited to millisecond-level closed-loop control. Letting a model directly determine motor torque at every moment would be both expensive and difficult to verify. A more plausible route into production environments is to place ER 2 in a high-level decision-making role while leaving deterministic execution to conventional control systems.

How Does It Relate to VLA Models?

Gemini Robotics ER 2 is not equivalent to a conventional end-to-end VLA model.

A VLA model is more like a worker with strong hands-on skills: it sees an image, understands an instruction, and outputs actions. The ER series is more like an on-site supervisor: it understands spatial relationships and task state, decides which capabilities to invoke, and generates plans or code for robots. The two are not mutually exclusive. On the contrary, they may form a complete stack.

A typical architecture can be divided into four layers:

1. Multimodal perception layer: Processes camera feeds, depth maps, audio, and sensor data;
2. Embodied reasoning and orchestration layer: Understands goals, tracks state, decomposes tasks, and selects robots;
3. Skill or VLA layer: Executes relatively self-contained action skills such as grasping, opening doors, and transporting objects;
4. Real-time control and safety layer: Handles trajectories, force control, emergency stops, and collision constraints.

ER 2 primarily strengthens the second layer, while extending toward the first through video understanding and influencing the third through tool invocation and robot collaboration.

Compared with a purely end-to-end approach, this architecture allows modules to be replaced, makes faults easier to isolate, and facilitates integration with existing industrial equipment. Its drawback is the high cost of system integration: the more interfaces there are, the easier it becomes for states to fall out of sync. The fact that the model believes a task is complete does not mean that the PLC, robotic-arm controller, and warehouse management system hold the same state.

Whether ER 2 can be deployed in practice therefore ultimately depends on whether DeepMind provides sufficiently stable tool protocols, state representations, and evaluation methods—not merely on the model's reasoning capabilities.

How Developers Can Use It Now

It is important to emphasize that DeepMind's release of Robotics ER 2 does not mean developers can already control robots through a public endpoint as easily as they can invoke a regular Gemini chat model. Robotics models usually involve hardware adaptation, sensor input, skill registration, safety permissions, and real-time system integration. Access is therefore likely to be more restricted than for general-purpose large models.

If developers only want to use the Gemini family for image understanding, task decomposition, or tool selection, they can access it through a platform compatible with the OpenAI format. OpenAI Hub already supports mainstream Gemini models, but whether Robotics ER 2 is available as a standalone model, its exact model ID, and its video input format should all be verified against the list of models actually available in the console. A regular Gemini multimodal interface should not be treated as ER 2's robot execution API.

The following example is suitable for validating a "visual understanding plus task planning" workflow. It does not directly control real equipment:

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_OPENAI_HUB_KEY",
    base_url="https://api.openai-hub.com/v1",
)

response = client.chat.completions.create(
    model="YOUR_AVAILABLE_GEMINI_MODEL_ID",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        "Analyze the current state of the workbench and break the cleanup task into verifiable steps."
                        "Output only the tasks, prerequisites, risks, and completion criteria. Do not generate motor-control instructions."
                    ),
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/workbench.jpg"},
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)

There is currently no standardized OpenAI-compatible field specification for video input. Different Gemini integrations may use file uploads, frame extraction, or native video objects. Production systems should not assume that placing an MP4 URL in image_url will provide video-understanding capabilities equivalent to ER 2.

A more reliable development path is to have the model output a constrained task graph in a simulation environment first, then use a deterministic executor to perform parameter validation and permission checks. The model should only be allowed to invoke preregistered skills, and every action involving movement, grasping, or shared human-robot workspaces should require approval from an independent safety layer.

This Upgrade Is Genuinely Useful, but the "ChatGPT Moment for Robots" Is Still Far Away

Gemini Robotics ER 2 is moving in the right direction, and its improvements matter more than simply increasing grasping success rates.

Once robots enter real environments, the greatest costs often arise from long-tail exceptions. A failed grasp can be retried, but an incorrect assessment of task state may cause the entire process to continue from a false premise. Video understanding gives the model context across time, task orchestration enables it to manage long workflows, and multi-robot collaboration raises the ceiling for handling complex tasks. Taken together, these three capabilities do constitute an architectural upgrade.

But this is not yet a decisive breakthrough for general-purpose robotics.

First, there is usually an enormous gap between capabilities shown in official demonstrations and production-grade reliability. A successful ten-minute demonstration does not prove that a system can operate continuously in a factory for weeks.

Second, multi-robot collaboration amplifies errors. A single robot's misjudgment affects only one action, while a central orchestration model's misjudgment may distribute an incorrect plan across the entire robot fleet.

Third, video context introduces significant computational and latency pressure. The system must retain a sufficiently long history while also responding rapidly to real-world changes. It cannot wait tens of seconds as offline video analysis might.

Fourth, advances in model capabilities do not inherently resolve questions of safety and accountability. Who approves high-risk actions, how model decisions are audited, and what state robots should enter when communication is interrupted are questions that still require engineering processes and conventional safety systems.

ER 2 is therefore better understood as a maturation of the "middle layer of the robotic brain," rather than as a form of universal intelligence that can be installed directly into any hardware. It advances Gemini's multimodal reasoning from answering questions about content on a screen to understanding the progression of real-world tasks, while extending single-robot planning into multi-device orchestration.

If these capabilities become available through stable APIs, standardized skill interfaces, and reproducible evaluations, they will deliver practical value in warehousing, manufacturing, laboratory automation, and service robotics. Conversely, if they remain limited to controlled demonstrations and a small number of partner hardware platforms, ER 2's impact will be felt more in research direction than in the developer ecosystem.

DeepMind has made it clear that the next phase of competition in robotics is not merely about "whose robotic arm is more dexterous," but about "who can understand longer real-world processes and enable a group of different machines to reliably accomplish the same goal." This competition is only just beginning to move from individual-machine intelligence to system intelligence.

References

• Zhihu: An In-Depth Analysis of Gemini Robotics-ER 1.6: Used to understand the previous positioning of the Gemini Robotics-ER series in embodied reasoning, visual understanding, and high-level planning. This article's description of ER 2's new capabilities is based primarily on Google DeepMind's latest official release information, but no external link to the official website is included due to link-domain requirements.