Wattage Gives Agents a Cost Meter

Agent Costs Belong in CI, Too

Recently, an open-source project called Wattage appeared on GitHub. It positions itself as a token-spend profiler and cost-regression gate for AI agents: it records how many tokens an agent consumes while performing a task and prevents changes from reaching the main branch when a new version causes a significant cost increase.

This may sound like little more than a meter for API bills, but the second part is what truly deserves attention: cost-regression gating.

Over the past year, developers have grown accustomed to using CI to catch unit test failures, latency increases, bundle-size bloat, and declines in code coverage. Wattage aims to make “how much it costs to complete the same task” another engineering metric that can be continuously compared. If a prompt change, an expanded tool description, or an adjustment to agent loop logic increases average token consumption by 30%, the problem should not remain hidden until the bill arrives at the end of the month.

This is not an ambitious agent infrastructure project, but it addresses a specific—and increasingly expensive—problem in agent engineering today.

Why Ordinary Token Counters Are Not Enough

The cost of a single-turn chat is relatively easy to understand: add the input and output tokens, then calculate the cost using the model’s pricing. Agents are different. They are more like task dispatchers that autonomously call on multiple “outsourced teams.”

A seemingly simple task such as “read the repository, locate the bug, and generate a patch” may involve:

1. The initial system prompt and user task entering the context;
2. The model deciding to search files and generating a tool call;
3. The tool returning thousands of lines of code, which are added back into the context;
4. The model determining that it lacks sufficient information and calling search or the terminal again;
5. A tool call failing and the framework automatically retrying it;
6. The agent generating a patch, then calling the model again to review it;
7. The review failing and the task loop starting over.

The user sees only one final answer, but the billing system may see a dozen—or even dozens of—model requests. More importantly, agent costs are not determined solely by how long the response is. They are also affected by:

• The length of the system prompt, tool definitions, and message history;
• The volume of tool output returned to the model at each step;
• The number of agent loop iterations and the stopping conditions;
• Failure retries, timeout retries, and fallback paths;
• The model selected by the routing logic and the different input and output prices of each model;
• Whether the context cache is hit;
• Whether parallel branches are canceled early;
• Whether the provider’s usage fields are complete and measured consistently.

As a result, an agent cost problem is often not simply “one request used 500 extra tokens.” Instead, a code change may alter the entire execution trajectory.

For example, a developer might expand each tool’s description from two sentences to an entire page to improve tool-selection accuracy. Viewed in isolation, the prompt may appear to add only a few thousand tokens. But tool definitions are sent repeatedly with every request, so if the agent averages ten iterations, that increase is amplified tenfold. Similarly, raising the maximum iteration count from 8 to 20 may not affect every task immediately, but it can allow failed samples to enter much deeper loops, creating long-tail bills.

Wattage’s value lies not in its ability to count tokens, but in its attempt to connect those changes to code versions.

What It Aims to Block Is “Cost Regression”

A cost regression is analogous to a software performance regression: the functionality remains correct, but more resources are required to complete the same work.

Typical agent cost regressions include:

• The same set of tasks can still be completed, but average input token usage rises;
• The success rate changes little, but the number of tool-call rounds increases significantly;
• P50 cost remains stable, while P95 cost deteriorates sharply because of a few infinite loops;
• A prompt change improves the quality of one class of tasks but causes other tasks to retry frequently;
• After a change to model-routing strategy, requests intended for a low-cost model are accidentally sent to an expensive one;
• Context trimming fails, causing message history to keep growing on every iteration.

A reasonable cost gate should not rely solely on an absolute cap such as “no more than $1 per run.” It also needs to compare the baseline and current versions while distinguishing between totals, averages, and percentiles.

Conceptually, such a check might look like the following. This is only an illustration of the gating approach and is not the actual configuration format used by the Wattage repository:

suite: agent-regression
baseline: main
candidate: pull-request

budgets:
  average_cost_increase: 10%
  p95_token_increase: 15%
  max_cost_per_task: 0.20
  max_agent_steps: 12

policy:
  fail_on_regression: true
  minimum_samples: 30

After CI runs a fixed suite of tasks, the gate might produce a verdict like this: the current version delivers no significant improvement in success rate, but average cost has risen by 18% and P95 token usage by 27%. The budget has been exceeded, so the build fails.

This is closer to traditional software engineering than having developers manually inspect a few traces. Traces are useful for explaining why a particular task became more expensive; a regression gate answers a different question: Did this commit make the entire system systematically more expensive?

It Is Best Suited Not to Demos, but to Agents That Have Started Burning Money

Wattage has limited value for chatbots that call a model only once. For single-turn applications, provider bills, logging platforms, or simple usage aggregation are usually sufficient.

Cost-regression tools are most useful for systems with unstable execution paths that repeatedly call models and tools, such as:

• Coding agents that read files, search for symbols, run tests, and repeatedly attempt fixes;
• Deep-research agents that perform concurrent searches, summarize findings, cross-check sources, and generate reports;
• Customer-service agents that query orders, search knowledge bases, call business APIs, and decide whether to escalate to a human;
• Data-analysis agents that generate SQL, execute queries, correct errors, and visualize results;
• Multi-agent workflows in which planners, executors, and reviewers pass context back and forth over multiple rounds.

These applications share one characteristic: one user request does not equal one model request. Product teams count tasks, model providers count calls, and finance ultimately bears the cost of the ever-branching execution tree between them.

Without task-level attribution, a team may see that total token usage increased on a given day but have no way to determine whether there were more users, the tasks became harder, or a code commit introduced additional loops. The direction represented by Wattage is to attribute costs back to test tasks, execution traces, and code versions rather than leaving them at the level of the monthly bill.

The Real Challenge Is Not Multiplication, but Measurement Methodology

Multiplying token counts by unit prices is not difficult. The challenge is that billing rules vary across models and providers.

Input and output tokens usually have different prices, while cached input may have separate pricing. Some models also distinguish between long-context tiers, batch requests, or special capabilities. For certain reasoning models, the text visible to developers may not correspond directly to the usage that is actually billed. Prices also change over time. If historical runs are recalculated using the latest prices, a “price change” may be mistaken for a “code cost regression.”

A reliable cost profiler therefore needs to handle at least three layers of data:

1. Raw Usage

Retain the input, output, cached-token, and any other available usage data for each call. The raw data cannot be reduced to a single total-token figure, because that would make it impossible to recalculate costs under a new pricing structure later.

2. Pricing Snapshot at the Time

Cost comparisons must specify which point in time, provider, and model version’s pricing they use. Otherwise, even if the agent’s behavior remains completely unchanged, an official model price adjustment will cause its recorded cost to jump.

3. Attribution to Tasks and Steps

Each call must be mapped to a task, a run, and an agent step. Aggregating only by model can tell a team “how much was spent on this model,” but not “which commit caused the planning stage to run three extra rounds.”

This is why we believe Wattage is heading in the right direction, but also why implementation will not be easy: it is not facing a simple tokenizer problem, but a cost-observability problem that spans models, frameworks, and execution steps.

Lower Cost Does Not Necessarily Mean a Better System

Cost gates have another easy-to-miss pitfall: they cannot be used independently of quality metrics.

The cheapest agent is one that never calls a model at all, but it clearly provides no value. Developers can also reduce token usage by compressing context, returning fewer search results, or terminating loops early, but the trade-off may be lower success rates, more hallucinations, or omitted critical steps.

A more reasonable question is therefore not “Did the cost increase?” but “Did the cost per useful outcome deteriorate?” For example:

• Average cost per successful task;
• Token changes with no decline in success rate;
• The additional cost required for each percentage-point improvement in success rate;
• The number of tasks that can be completed within a fixed budget;
• Cost and success rates after stratifying tasks by difficulty.

Suppose a new version raises the task success rate from 60% to 85% while increasing cost by 15%. That would generally be acceptable. If the success rate improves by only 1% while the cost doubles, the gate should block it.

For nondeterministic agents, a single run is also insufficient. The same task may follow different paths because of sampling, search results, or the state of external tools. Developers need repeated runs, minimum sample sizes, and attention to confidence intervals—or at least P50 and P95—rather than drawing conclusions from two random results.

This means tools such as Wattage are best used alongside agent evaluation suites: the evaluation system determines whether the task was completed, while the cost profiler determines how many resources were spent completing it. Only by combining the two datasets can teams avoid either “ruining the product to save money” or “spending unlimited tokens for a slightly better score.”

It Does Not Entirely Conflict with Existing Observability Platforms

Many LLM observability products already support request tracing, token inspection, and cost estimation. Wattage’s differentiator should not be yet another cost dashboard, but tighter integration with the development workflow:

• Automatically comparing pull requests against the main branch;
• Blocking merges through exit codes or check statuses;
• Storing cost budgets in version control;
• Reporting regression causes at the task and step levels;
• Allowing developers to reproduce expensive execution traces locally.

In other words, an observability platform is more like an electric utility’s monthly bill and consumption chart; Wattage aims to be the circuit breaker inside CI. The former tells you where the money went, while the latter prevents a new device from being deployed when its power consumption is abnormal.

If Wattage ultimately does no more than display total token usage, it will have many alternatives. If it can reliably handle baseline management, noise filtering, pricing snapshots, and CI threshold decisions, it could become a small but practical engineering component for agent teams.

Several Issues to Watch at This Stage

As an open-source project that is still evolving rapidly, whether Wattage can truly enter production workflows will depend on several practical issues.

The first is integration cost. There are many agent frameworks: some provide a unified wrapper around model calls, while others allow application code to directly call different providers. If adopting the tool requires major changes to the call chain, teams may prefer to write their own aggregation scripts on top of an existing tracing system.

The second is the reliability of usage data. When a request fails, a streaming response is interrupted, a proxy layer retries a call, or a provider does not return complete statistics, how the tool fills in and labels the missing data will directly affect the results. Estimates can be useful for trend analysis, but they should not be presented as exact billing figures.

The third is test reproducibility. External search results, web content, and business APIs are constantly changing. If agent regression tests do not run in a fixed environment, cost fluctuations may not come from the code at all. Recording and replaying tool outputs is often more effective than simply running the tests more times.

The fourth is credential and content security. Cost analysis often requires observing request metadata and may even involve linking prompts with tool outputs. Teams should clearly define which data remains local, which data is uploaded, and whether logs contain credentials, user information, or proprietary code.

The fifth is maintaining multi-provider pricing data. Model names, versions, and billable items change quickly. An outdated pricing table can produce dollar figures that look precise but are actually wrong. For serious use cases, token regressions may be more stable than monetary regressions, while monetary comparisons require fixed pricing snapshots.

Our Verdict: A Small Tool That Captures the Next Step in Agent Engineering

The most commendable aspect of Wattage today is not that it has invented a new way to count tokens, but that it proposes a clear engineering action: when an agent’s resource consumption regresses, block the code merge just as you would for a failed test.

Agent development has previously focused more on whether a task can be completed. Going forward, the focus will increasingly shift to whether it can be completed reliably and predictably. As model capabilities become sufficient for more tasks, product margins will often be determined by irrelevant context, redundant tool calls, failed retries, and runaway loops. These problems will not naturally disappear as model prices fall. On the contrary, cheaper models may encourage teams to make even more calls, concealing waste more deeply.

Cost gates should not be overhyped, however. They must be considered alongside task success rates, output quality, and latency, while correctly accounting for randomness, caching, price changes, and changes in the test environment. Otherwise, teams will merely get another noisy red light in CI.

As of July 27, 2026, the direction represented by Wattage is more noteworthy than its current implementation: an agent’s token consumption should not be visible only to finance at the end of the month; developers should see it with every commit. For teams that have already placed agents into real business workflows, it is indeed time to install this “meter.”

References

• Wattage GitHub Repository: Project source code and documentation. It is positioned as a token-spend profiler and cost-regression gate for AI agents. For installation instructions, supported features, and the latest functionality, refer to the repository’s current version.