Drone-Bench: Benchmarking Drone Surveillance

Frontier Models Begin Taking “Closed-Loop in the Air” Tests

Andon Labs recently released Drone-Bench, a benchmark designed to continuously track the basic drone surveillance capabilities of frontier models. It asks a straightforward question: when a model must do more than identify what is in an image—when it must control a drone to search for, observe, and continuously track a target—how far can today’s models actually go?

As of July 24, 2026, the significance of Drone-Bench lies not in declaring that a particular model can already “fly autonomously,” but in turning a capability that was previously easy to obscure with demo videos into a set of reproducible, comparable tasks. High scores on static image question answering do not mean that a large model can operate a machine that is constantly moving, whose viewpoint changes continuously, and where every action affects the next frame of input.

This is also the most important distinction between Drone-Bench and conventional computer vision leaderboards: it does not test a single image, but a closed loop.

“Seeing the Target” Is Only the First Step

Traditional vision benchmarks usually provide a model with an image or a fixed video clip, then ask it to detect, classify, or describe a target. The input does not change in response to the model’s answer, and if the model makes a mistake, it merely loses points on one question.

Drone tasks are different.

If the model turns the camera in the wrong direction, it may completely lose the target in the next moment. If its search path is poorly planned, it will waste limited time revisiting areas that have already been checked. If it flies too high, the target may occupy only a few pixels; if it flies too low, the field of view narrows and the risk of collision may increase. In other words, the model is both an observer and a producer of its own input data.

A basic drone surveillance task involves at least the following capabilities simultaneously:

• Visual localization: Identifying people, vehicles, and other targets from overhead or oblique views;
• Spatial reasoning: Understanding where the target is located within the frame and how the drone should approach it;
• Active search: Deciding where to look first when the target is not directly visible;
• Short-term memory: Remembering which areas have already been searched and where the target was last seen;
• Continuous tracking: Re-establishing the target’s identity after it moves, becomes occluded, or temporarily leaves the frame;
• Tool use: Converting natural-language reasoning into executable movement, turning, and camera operations;
• Risk control: Avoiding boundary violations, collisions, and pointless repetitive maneuvers.

None of these capabilities is novel on its own, but the difficulty escalates rapidly when they are placed in the same real-time decision chain. A single spatial reasoning error can trigger cascading consequences over the next dozen or more steps.

What Drone-Bench Really Tests Is the Minimal Closed Loop of Embodied Intelligence

Drone-Bench defines its scope as “simple drone surveillance capabilities,” and the word simple matters. It does not attempt to prove that general-purpose models can already replace complete flight-control systems, nor does it test fully autonomous combat capabilities in complex airspace. Instead, it begins with relatively basic and controllable surveillance tasks.

From a benchmarking perspective, these tasks can be abstracted into the following loop:

Initialize the task and drone state
while no termination condition has been reached:
    Obtain the current image, position, and task state
    The model determines whether the target is visible
    The model chooses to search, move, turn, or continue observing
    The environment executes the action and returns the next state
    Record target visibility, tracking continuity, elapsed time, and failure events
Output the result of the entire task, rather than a single-frame recognition score

Here, the model acts more like a high-level task agent than a low-level flight controller that directly manages motor speeds. Attitude stabilization, propulsion control, and emergency protection should still be handled by more deterministic, validated control systems. The model is responsible for deciding “where to look, what to look at, and what to do next.”

This boundary must be made clear. Connecting a multimodal model to a drone control interface does not amount to a deployable autonomous flight system. A real product also requires localization, obstacle avoidance, communication-link management, redundant sensors, fail-safe mechanisms, airspace compliance, and human takeover capabilities. If any layer is missing, high laboratory scores may fail to translate into real-world performance.

Why Conventional Multimodal Leaderboards Are No Longer Enough

Over the past two years, multimodal models have improved markedly in image recognition, video understanding, and spatial reasoning, but existing leaderboards have three problems.

First, static inputs cannot expose accumulated decision errors. A model may appear to be “roughly correct” at every step, yet end up moving farther and farther away from the target after twenty consecutive actions.

Second, multiple-choice questions overestimate real-world capabilities. In a drone environment, there are no four prewritten answer choices. The model must decide for itself whether to keep searching, which direction to move in, and when the task is complete.

Third, offline video does not require the model to take responsibility for camera positioning. Analyzing a video recorded by someone else and controlling a camera to keep a target in frame are two entirely different tasks. The former involves understanding content; the latter also requires managing the field of view.

Drone-Bench is therefore closer to a robotics and embodied intelligence benchmark than to traditional visual question answering. It places the model in a “perception–decision–action–new perception” chain, where the model’s weaknesses are amplified through environmental feedback.

This kind of evaluation is more useful than a flashy drone demonstration. A demo usually shows only one successful trajectory, whereas a benchmark should examine whether performance remains stable across repeated runs of the same task, whether the task can still be completed after changing the starting position or target, and whether failures occur because the model failed to see something, remembered it incorrectly, or issued the wrong control action.

What Really Matters Is Not the Overall Score, but the Failure Modes

If Drone-Bench evolves into a continuously updated leaderboard for tracking model performance, developers should care less about which model ranks first than about where each model breaks down.

For example, one model may have excellent recognition capabilities but repeatedly issue turns that are too large, causing the target to oscillate between the two sides of the frame. Another may control the drone more smoothly but mistake a similar-looking object for the original target after a brief occlusion. Still other models may complete the task but require so many decision cycles that inference costs and task latency become unacceptable.

A drone agent should therefore be evaluated on at least the following dimensions:

1. Task success rate: Whether it ultimately finds and continuously observes the target;
2. Time to first detection: Whether its search strategy is effective;
3. Target visibility duration: Whether tracking is continuous and stable;
4. Reacquisition capability: Whether it can relocate the target after losing it;
5. Action efficiency: Whether it engages in redundant searches or ineffective maneuvers;
6. Safety incidents: Whether it collides, crosses boundaries, or enters an unrecoverable state;
7. Latency and cost: Whether model inference is fast enough to keep up with changes in the environment;
8. Cross-scenario generalization: Whether performance drops significantly when lighting, viewpoint, map, or target changes.

Even if a model has a high overall success rate, it will struggle to track fast-moving targets if every decision takes several seconds. Conversely, a low-latency model that frequently tracks the wrong person is equally unsuitable for surveillance tasks. No single score can capture every issue in drone evaluation.

It Also Exposes the Limitations of End-to-End Approaches

The approach most likely to be overestimated today is feeding all state information directly into a general-purpose model and asking it to decide every action end to end. This approach is quick to develop and produces intuitive demos, but it may not be the most reliable engineering choice.

Real-world systems are more likely to use a layered architecture: a dedicated detector handles high-frequency target detection, a conventional tracking algorithm maintains object identities, a flight-control system executes stable movement, and a large model performs task planning and exception handling at a lower frequency.

This is like asking the large model to serve as a “mission commander” rather than personally deciding how to move the rudder every few dozen milliseconds. The former can take advantage of the model’s semantic understanding and planning capabilities, while the latter is more vulnerable to latency, output variability, and visual hallucinations.

Drone-Bench is valuable to developers precisely because it can help determine which layer a model belongs in. If a model is good at search but poor at fine-grained tracking, it can generate search strategies while continuous tracking is delegated to a conventional vision module. If a model has reliable spatial reasoning but responds slowly, its invocation frequency can be reduced, with a local controller executing the trajectory between inference calls.

A leaderboard should not merely answer “which model is best.” It should also help teams decide “which model is best suited to which part of the system.”

There Is Still a Long Road from Simulated Evaluation to Real-World Deployment

Even if Drone-Bench can reliably measure differences among frontier models, its results cannot be directly extrapolated to real drones. Simulated or controlled environments usually cannot fully cover the real world’s long-tail problems: backlighting, rain and fog, camera vibration, wireless link failures, GNSS drift, battery degradation, dynamic obstacles, and human behavior can all change task difficulty.

An even larger difference lies in real-time performance. Cloud-based models offer greater capabilities, but uploading video, running inference, and returning actions all introduce latency. Edge models respond faster and pose fewer privacy risks, but they are constrained by onboard compute, power consumption, and heat dissipation. A truly practical system will generally divide responsibilities between cloud-based planning and on-device perception rather than simply choosing “the largest model.”

Drone surveillance also has inherent dual-use implications. It can be used for mining-site inspections, campus security, disaster search and rescue, and infrastructure inspection, but it can also be used to track people without authorization. Evaluating the boundaries of a capability does not mean encouraging unrestricted deployment. When real people, public spaces, or sensitive facilities are involved, data retention, access control, geofencing, human confirmation, and audit logs should all be standard system capabilities.

It is especially important to remain alert to the fact that models can express incorrect judgments with great confidence. In a text conversation, this may amount to no more than a wrong answer; in a closed-loop drone system, it becomes a real-world action.

A Useful Benchmark That Must Not Be Misinterpreted

Drone-Bench is moving in a worthwhile direction. The capabilities of frontier models are shifting from “understanding content” to “affecting the environment,” and evaluation must likewise advance from static question answering to continuous tasks. Drones happen to be a platform that creates sufficient pressure while producing easily observable results: the viewpoint moves, targets disappear, actions have costs, and mistakes may be irreversible.

At this stage, however, Drone-Bench is more like a measuring stick than a product certification. A leading ranking indicates that a model is better at completing a closed loop under specific environments and task configurations; it does not mean the model already meets the reliability, safety, and compliance requirements of real-world flight. Developers should not substitute one successful run for stability testing.

The truly important signal is that model vendors can no longer rely solely on static vision scores to demonstrate “spatial intelligence.” As models begin to be integrated into robots, vehicles, and drones, the industry needs more dynamic benchmarks like Drone-Bench—benchmarks that force models to face the consequences of their previous actions and then test whether they remain intelligent.

That brings us much closer to the real world than having models answer a few more multiple-choice questions correctly.

References and Further Reading

• Overview of Frontier Drone Research at RSS 2025: Reviews research directions including drone perception, onboard inference, target tracking, and avoidance, providing useful context for understanding the technological landscape surrounding Drone-Bench.

Note: This article’s description of Drone-Bench’s positioning is based on public statements from Andon Labs. In accordance with this site’s external-link domain policy, no off-site link has been included in the main text.