NVIDIA releases LocateAnything: When the Agent finally learns to “hit where it points”

On May 29, NVIDIA, together with teams from Hong Kong Polytechnic University, Nanjing University, and others, launched a visual localization model called LocateAnything. From the name, one might think it’s another derivative of Segment Anything, but a closer look at the technical report reveals that the focus this time isn’t on what it can locate, but on how fast it can locate. On a single H100, it achieves 12.7 Boxes Per Second, while Qwen3-VL reaches only 1.1 and Rex-Omni 5.0. A throughput gap of more than tenfold is beyond what ordinary optimization can explain.

In the Agent-driven era of 2026, this means far more than just a new SOTA number.

The problem isn’t “seeing,” it’s “seeing fast enough”

Over the past year, multimodal large models have turned “image captioning” into a red ocean; the visual understanding capability of VLMs is no longer the bottleneck. But anyone who has done embodied intelligence or GUI Agent deployment knows a recurring pain point: the model knows where the target is, but it takes too long to tell you.

A typical case: asking an Agent to click a button on screen. The model first outputs a segment of natural language reasoning, then spits out coordinate tokens, decoding x1, y1, x2, y2 one number at a time. For humans, a glance takes tens of milliseconds; for models, it takes several seconds. Robot control exaggerates this further—if perception lags in the perception–decision–execution loop, the entire pipeline collapses.

This time, NVIDIA directly targeted the “decoding method” itself.

Parallel Box Decoding: treat coordinates as atoms, one-step output

The core innovation of LocateAnything is called Parallel Box Decoding (PBD). The idea is quite straightforward: since bounding boxes are just four fixed numbers (or two for a point), why output them autoregressively token by token?

PBD treats each bounding box as a fixed-length atomic unit, predicting x1, y1, x2, y2 coordinates in parallel in a single forward pass, rather than sequentially. It sounds reminiscent of Diffusion Language Models (DLM)—the paper does cite works like LLaDA and Dream—but LocateAnything doesn’t fully adopt a diffusion pipeline; instead, it introduces “block-level outputs” within the generative VLM framework.

This design brings direct benefits:

• Throughput scales linearly with box count. Whether detecting 100 targets or 10, the decoding cost difference is far smaller than for autoregressive methods.
• Stable latency. Each box is an atomic operation, not slowed down by high-entropy predictions of any single coordinate token.
• Friendly to dense detection. In datasets like LVIS, with dozens or hundreds of objects per image, traditional VLMs struggle to run, but PBD can handle such decoding workloads.

Three modes: leave engineering trade-offs to the user

LocateAnything doesn’t aim for “one mode fits all,” but instead offers three configurations:

• Fast Mode: pure parallel decoding, maximizing throughput—ideal for real-time scenarios like edge robotics or embodied intelligence.
• Slow Mode: reverts to token-by-token autoregressive decoding for higher precision—suited to offline labeling or benchmark testing.
• Hybrid Mode: defaults to Fast, but automatically falls back to Slow in cases of format errors or spatial ambiguities. This is the recommended default and the basis for the 12.7 BPS result.

The Hybrid design is pragmatic. The biggest risk with parallel decoding is “hallucinated coordinates”—predicting four numbers at once with weak mutual constraints can sometimes yield absurd results like x2 < x1. Hybrid employs a lightweight format checker as a safeguard, rerouting abnormal cases back through the autoregressive path for redo—a very efficient engineering trade-off.

Data: 12M images, 785M boxes, covering six localization scenarios

Decoding innovation alone isn’t enough—data scarcity has always plagued localization tasks. LocateAnything also released LocateAnything-Data, pushing the scale to extremes:

• 12M distinct images
• 138M language queries
• 785M bounding boxes

More importantly, it spans a broad range of tasks. It’s not just an object detection dataset, but covers six localization types:

1. General Object Detection (General OD)
2. GUI Element Localization — crucial for Computer Use Agents
3. Referring Expression Comprehension
4. OCR Text Localization
5. Layout Detection
6. Point Grounding

Training these six tasks together in a generative framework gives the model the ability to handle any visual task expressible via boxes or points, controllable using natural language—this is the true meaning of “Locate Anything.”

Benchmark comparison: speed without sacrificing accuracy

Speed alone isn’t enough—the worst scenario for a localization model is “fast but inaccurate.” Let’s look at key benchmarks:

| Metric | LocateAnything | Rex-Omni | Qwen3-VL | |---|---|---|---| | Throughput (H100, BPS) | 12.7 | 5.0 | 1.1 | | LVIS IoU=0.95 | 31.1 | 20.7 | - | | ScreenSpot-Pro Avg F1 | 60.3 | - | - | | DocLayNet | 76.8 | - | - | | M6Doc | 70.1 | - | - |

Some highlights:

LVIS IoU=0.95 at 31.1 is a tough metric—much stricter than the commonly used IoU=0.5, requiring predicted and ground-truth boxes to overlap by 95%. Rex-Omni scores 20.7, while LocateAnything exceeds it by over 10 points, showing PBD improves stability under high IoU thresholds. That matters greatly for industrial applications, since in robot grasping or screen clicking, even a slightly off box means a missed grab or wrong click.

ScreenSpot-Pro F1 of 60.3 directly reflects Computer Use Agent capability. Anthropic’s Computer Use and OpenAI’s Operator have been public for over a year, and it’s well-known that screen element localization failure rates are the biggest bottleneck to Agent practicality. A model that runs at 12.7 BPS and scores above 60 on ScreenSpot-Pro could theoretically reduce GUI Agent end-to-end latency to a fraction of current levels.

DocLayNet 76.8 and M6Doc 70.1 are tough layout analysis benchmarks. Scoring this high in layout detection suggests LocateAnything could be used directly as a layout segmenter in document RAG pipelines, far cleaner than traditional LayoutLMv3 plus post-processing.

How to use: open-source, HF demo, ready to deploy

NVIDIA chose an open-source route—model weights and code are hosted on Hugging Face and GitHub, with an online demo on the project page. Typical use cases for developers include:

• Embodied Intelligence: replace OWL-ViT or GroundingDINO in existing pipelines—throughput increases by an order of magnitude.
• GUI Agent: serve as the visual localization backend for Computer Use systems, paired with a reasoning VLM for “see + think + click” task division.
• Document Intelligence: handle layout segmentation and OCR box localization in one go, cutting out glue code.
• Data Labeling: run Slow Mode for large-scale pre-labeling, with manual verification only.

Note that LocateAnything itself isn’t a “dialogue” model—it’s dedicated to localization. In actual products, it’s likely paired with an inference VLM (e.g., Qwen3-VL or GPT series): the reasoning model interprets intents, LocateAnything converts intents into coordinates. This “specialist + generalist” pairing could be the mainstream multimodal Agent architecture in the near term.

A brief assessment

The most interesting aspect of LocateAnything isn’t hitting SOTA on a benchmark, but repositioning “visual localization” as an independent module rather than a side-task of VLMs.

The past two years have favored “one big model for everything,” where VLMs both describe images and draw boxes—excelling at neither. LocateAnything goes the opposite way: localization is localization, and deserves specialized decoding, dedicated data, and a tailored output format. This “anti end-to-end” engineering philosophy actually makes sense in the Agent age—since the bottleneck isn’t “how powerful one model is,” but “how smoothly the whole pipeline runs.”

The PBD approach might not be limited to visual localization. Any task with “structured, fixed-length, low-degree-of-freedom output”—such as function calls, table generation, or JSON output—could theoretically adopt parallel decoding to bypass autoregressive latency. If this direction proves viable, it may offer more potential than speculative decoding alone.

As for whether general VLMs like Qwen3-VL will be replaced: not soon. But if you’re building a system requiring frequent visual localization, swapping out the “box drawing” part of a general VLM for LocateAnything is almost guaranteed payoff.

References

• ITHome: NVIDIA launches LocateAnything, featuring high-speed, high-accuracy object detection – First domestic report, includes key data and mode descriptions
• Hugging Face Model Hub – location for LocateAnything weights and online demo (search “LocateAnything”)
• GitHub – source code and training data repository (official NVIDIA repo)