Signals: An Agent Trajectory Filtering Algorithm Without LLM Evaluation

Katanemo Labs (a subsidiary of DigitalOcean) has just released a new research paper on agent systems called Signals, which directly addresses the core pain point in current AI Agent development: how to quickly identify the trajectories truly worth analyzing from a huge volume of data.

The Core Problem: Trajectory Explosion and Review Cost

If you’re building an AI Agent, you’ve likely experienced this — every time the agent executes a task, it generates a complete interaction trajectory (trace/trajectory), including reasoning steps, tool calls, and environmental feedback. The issue is that there are simply too many of these trajectories.

Traditionally, there are two approaches: manually auditing each trajectory, or using additional LLM calls to evaluate each trajectory’s quality. The first method doesn’t scale; the second is costly — every trajectory consumes extra API quota, and the LLM’s judgments can be inaccurate.

Signals’ approach is different: instead of relying on humans or LLMs, it computes structured “signals” to quantify the informativeness of trajectories, automatically filtering for the most relevant ones. Critically, this computation doesn’t require a GPU and can run in real time in production environments without affecting the agent’s normal behavior.

Signal Taxonomy: Three Dimensions, Nine Patterns

Signals introduces a concise taxonomy that covers three core dimensions of agent behavior:

1. Interaction Dimension

• Misalignment: The agent’s output is inconsistent with the user’s intent or task goal
• Disengagement: The agent stops participating effectively, e.g., repeating meaningless responses
• Stagnation: The agent gets stuck in a state without progressing, yet doesn’t explicitly fail

2. Execution Dimension

• Failure: Tool call failure, API error, insufficient permissions, and other explicit errors
• Looping: The agent repeatedly executes the same or similar operations, entering an infinite loop
• Exhaustion: Reaching the maximum step, token limit, or time constraint

3. Environment Dimension

• Anomaly: Returned data or state from the environment exceeds the expected range
• Constraint Violation: Violates task-defined hard constraints
• Resource Contention: Conflicts caused by multiple agents or processes competing for the same resource

This taxonomy isn’t arbitrary — it’s derived from the team’s observation of common issue patterns in real agent systems. Each signal corresponds to a concrete, quantifiable behavioral feature.

Mechanism: Lightweight Feature Engineering

The core of Signals is a set of carefully designed feature extractors that compute signal strengths from structured trajectory data. These features include:

• Sequence patterns: Detecting repetitive action sequences and state transition patterns
• Statistical anomalies: Measuring deviation of key metrics (e.g., tool call success rate, response time)
• Semantic similarity: Using lightweight embedding models (non-LLM) to measure output similarity and detect stagnation or loops
• Constraint checks: Validating trajectory compliance against task rules

Importantly, these computations are deterministic and explainable — no black-box LLM evaluations involved. You can clearly know why a trajectory was flagged as informative: was it due to a failure signal, or an unexpected looping pattern?

Experimental Results: 1.52× Efficiency Improvement

Katanemo Labs conducted annotation experiments on τ-bench, an agent evaluation benchmark, comparing three sampling strategies:

1. Random sampling – randomly selects trajectories
2. Signals-based sampling – selects trajectories based on ranked signal strength
3. Oracle sampling – ideal post-hoc human annotations (used as the upper bound)

Results show:

• Random sampling informativeness rate: 54%
• Signals sampling informativeness rate: 82%
• Meaning: Each trajectory reviewed under Signals provides 1.52× more useful information

In other words, if you have a budget to review 100 trajectories, using Signals to pre-filter is equivalent to randomly reviewing 152. For fast-iterating agent development teams, that’s a tangible efficiency boost.

Engineering Value: A New Dimension of Observability

From an engineering perspective, Signals provides more than a filtering algorithm — it’s a new framework for agent observability.

Traditional metrics like latency, throughput, and error rate don’t suffice for agent systems. You also need to know:

• Is the agent performing useless work?
• Which task types tend to trigger loops?
• What environmental changes lead to performance drops?

Signals transforms these questions into measurable signals that can integrate with existing monitoring stacks. You can set alerts — e.g., notify the team automatically when the frequency of a particular signal exceeds a threshold.

Further applications include:

• Active learning: Prioritize labeling high-signal trajectories for fine-tuning or reinforcement learning
• A/B testing: Compare versions of agents based on their signal profiles
• Root cause analysis: When the agent malfunctions, quickly identify whether the issue lies in interaction, execution, or environment dimensions

Limitations and Future Directions

The paper openly discusses current limitations:

1. Completeness of signal definitions – Nine patterns cover most cases but not all; domain-specific agents may need custom signals
2. Threshold tuning – Signal strength thresholds must be task-adjusted; no automated tuning yet
3. Multi-agent systems – Current focus is single-agent trajectories; collaborative multi-agent signal calculation is more complex

The team’s next steps include:

• Open-sourcing a reference implementation of Signals for community contributions
• Exploring causal (not just correlational) links between signals and agent performance
• Studying how signals can guide online learning and adaptive behavior

Implications for Developers

If you’re building production-level AI Agents, Signals offers valuable insights:

1. Don’t blindly rely on LLM evaluation – “LLM-as-a-judge” is popular but costly and unstable; often structured rules and statistics are sufficient
2. Treat observability as a first-class citizen – Build monitoring and debugging into your design from day one, not as an afterthought
3. Prioritize informativeness over quantity – Better to sample fewer but higher-value trajectories than to collect massive low-value logs

On a broader level, Signals reflects a trend in agent engineering: shifting from “making it run” to “making it controllable, observable, and optimizable.” As AI Agents increasingly enter production, these engineering methodologies will become essential.

Comparison with Existing Solutions

Existing agent trajectory analysis tools such as LangSmith, Weave (by Weights & Biases), and Arize Phoenix mainly offer visualization and basic statistics but limited automated filtering capabilities.

Signals stands out in that it:

• Requires no extra LLM calls – reducing cost and improving real-time performance
• Provides a structured signal taxonomy – interpretable categorization instead of opaque scoring
• Uses lightweight computation – runnable on edge devices or resource-constrained environments

Signals isn’t meant to replace these tools but rather complement them — pre-filter with Signals, then perform in-depth analysis in your visualization environment.

Open Source and Community

The paper is publicly available, and code is expected to be open-sourced on GitHub soon. Katanemo Labs plans to provide implementations in Python and TypeScript, along with integrations for major agent frameworks (e.g., LangChain, LlamaIndex, AutoGPT).

Early community feedback highlights:

• Custom signals: Easy addition of domain-specific signal types desired
• Real-time monitoring: Integration with Prometheus, Grafana, and similar stacks requested
• Multimodal support: Current focus is text-based trajectories — what about visual or multimodal agents?

These are all valid directions for future collaborative development post open-sourcing.

Conclusion

Signals tackles a very practical issue: how to extract valuable information from massive agent interactions without increasing cost. The method is simple yet effective — precisely the kind of pragmatic innovation engineering-grade AI systems need.

For teams building AI Agents, this paper is worth a careful read. Even if you don’t directly adopt Signals, its taxonomy of signals and observability concepts can inspire your system design.

After all, agents are not magic — they’re complex systems requiring engineering management. And Signals provides one foundational building block for that management capability.

---

References

• Reddit r/MachineLearning discussion thread — Signals research introduction by Katanemo Labs members Salman, Shuguang, and Adil, summarizing the core ideas and experimental results.