Amazon AI Project Burns Through $1.8 Million, 860% Over Budget Before Anyone Notices

Amazon recently uncovered an embarrassing set of internal AI cost figures: a project originally intended to match author information with product listings used Anthropic’s Claude Sonnet model and ultimately cost about $1.8 million—860% over its original budget. Worse still, the spending continued for roughly five months before Amazon discovered the problem.

According to the Financial Times, the project ultimately failed to deliver the expected results and was never even launched. What initially sounded like a typical data-cleaning and information-matching task turned into a “catastrophically expensive” experiment because the AI execution process lacked adequate cost controls.

This was not Amazon’s only case of an AI budget spiraling out of control. At an internal employee meeting on July 29, a senior Amazon engineer mentioned two other incidents: a project developing a financial auditing tool incurred approximately $541,000 in unexpected costs; another project attempting to use AI to improve logistics and delivery speeds overspent by about $134,000, and the issue was not discovered until more than two weeks later.

Amazon said in response that the company is “experimenting, learning, and improving how it uses AI, including increasing cost efficiency,” while emphasizing that these cases involved only a small number of teams and were not representative of AI use across the company.

From an engineering management perspective, however, the problem is probably not simply that “a few teams made a few mistakes.” What these cases truly reveal is that as companies move from using AI as a tool requiring human confirmation to entrusting work to Agents capable of autonomous loops, external service calls, and code modification, their existing cost-control systems are breaking down.

How Did a Simple Matching Task Burn Through $1.8 Million?

The publicly available information does not disclose the complete technical architecture, number of calls, or specific prompts used in the Claude Sonnet project, so it is impossible to reconstruct exactly how the $1.8 million accumulated line by line. However, based on the type of task and common AI Agent operating patterns, several cost multipliers are easy to identify.

Matching author information with product listings is essentially an entity resolution or record linkage task. The system must determine whether “a particular author” corresponds to “a particular product page,” potentially accounting for spelling variations, pen names, publisher information, category tags, and duplicate records. This type of work can generally be handled first with rules, database queries, vector retrieval, or smaller models, with only difficult cases passed to a more capable model for judgment.

If a premium model is used directly to process all data in bulk, costs can easily spiral out of control—especially when the system uses workflows such as the following:

• Sending lengthy product descriptions, historical information, and context for every record;
• Automatically retrying after failed matches without a maximum retry limit;
• Allowing the model to generate queries, modify scripts, and rerun tasks on its own;
• Using “self-reflection” or multi-round review, turning a single decision into multiple model calls;
• Failing to terminate tasks promptly after errors, allowing background jobs to continue for weeks or even months;
• Monitoring only cloud server costs without including third-party model Token charges in a unified alerting system.

In traditional software, an infinite loop usually saturates CPU or memory quickly, making it relatively easy for monitoring systems to detect. In large-model workflows, however, an infinite loop may look like “the model continuously trying to solve the problem”: breaking down the task again, adding context, calling tools, checking the results, and then starting over.

To a business system, this kind of loop may look like smarter, more resilient automation. To the billing system, it is a Token-burning machine with no brakes.

The Biggest Danger Is Not an Expensive Model, but the Lack of a “Spending Cap”

Claude Sonnet is not the most expensive model on the market, and Amazon has stronger cloud infrastructure and greater bargaining power than an ordinary development team. The key issue, therefore, is not that the project “used Claude Sonnet,” but why it was able to remain 860% over budget for five months without triggering any effective intervention.

AI projects generally incur three types of costs simultaneously:

1. Model call costs: Charged according to input and output Tokens, with long contexts, multi-round conversations, and large data volumes rapidly inflating the bill.
2. Compute and infrastructure costs: Including containers, GPUs, databases, logs, queues, object storage, and data transfer.
3. Engineering and failure costs: Including retries, erroneous calls, unusable outputs, manual review, and investment in projects that ultimately never launch.

In the past, budgeting for a back-end script often required considering only machine specifications and runtime. Today, companies must answer much more granular questions: How many model calls may be made per record? How much may a single task cost? What is each team’s daily Token quota? Who receives an alert when model outputs grow longer? After how many consecutive failures should an Agent automatically trip a circuit breaker?

There are no universal answers to these questions, but they cannot be ignored.

At the internal meeting, an Amazon employee noted that coding errors that incur “virtually no cost” in traditional systems can become catastrophically expensive when AI models execute tasks. Every company deploying AI Agents should take this observation seriously: model calls are themselves billable side effects, and any flaw in code logic can be converted into a real invoice.

AI Agents Have Changed Both the Cost Curve and the Nature of Failures

Failures in traditional software generally have relatively clear boundaries. An incorrect SQL statement may cause a query to fail; a misconfigured scheduled task may run a few extra times; a service caught in a loop may be detected by monitoring because of abnormal CPU usage, memory consumption, or request volume.

Problems with AI Agents are more subtle. An Agent may continuously produce seemingly reasonable content, repeatedly call external APIs, or carry an increasingly long history of context into each round of “correction.” The system does not crash, and the business process may show no obvious errors, yet costs continue accumulating in the background.

More complicating still, an Agent’s behavior is not always determined by a fixed process. A developer may set only a goal, such as “complete the matching of author information with products,” while the model dynamically determines task decomposition, tool selection, retry strategies, and result validation. This flexibility is precisely what gives Agents value over traditional automation scripts—and precisely what makes them difficult to predict.

In low-risk scenarios, this unpredictability may cost only a few extra dollars. In large-scale enterprise data-processing scenarios, the price can rise to hundreds of thousands or even millions of dollars.

From this perspective, the Amazon incident cannot simply be attributed to an engineer who “forgot to shut down a task.” It looks more like a traditional FinOps system failing to keep pace with changes in model-based applications: costs are no longer determined solely by the number of servers, but jointly by prompt length, model selection, context retention, the number of tool calls, and Agent decisions.

The Lesson Amazon Is Now Learning: Build Cost Controls Into Automation

Amazon engineers are reportedly creating “automated” safeguards for future projects to control AI usage costs. The specific measures have not all been disclosed, but a sound AI cost-governance system should include at least the following layers.

Layer One: Budgets and Hard Limits

Every project, team, environment, and task should have its own independent budget; companies cannot simply review the total bill at the end of the month. More importantly, there must be “hard limits”: once a monetary amount, Token count, number of calls, or runtime threshold is reached, the task must automatically pause rather than merely send an alert email.

For batch-processing tasks, a maximum cost can be set for each batch. For example, when a batch’s projected budget exceeds the threshold, it can be routed to human review or automatically switched to a cheaper model. Cost limits should not depend on developers remembering to configure them; they should be a default platform-level capability.

Layer Two: Call-Level Observability

Companies need to know who initiated every model call, which project it belonged to, how many input and output Tokens it processed, which tools it called, and whether it was generated by a retry.

Looking only at the vendor’s final bill is nowhere near enough. The bill tells you how much was spent, but it cannot explain which data batch, prompt, or piece of Agent logic caused the cost increase.

Layer Three: Circuit Breakers for Retries and Loops

Every automated retry mechanism should have a limit, and every Agent loop should have an exit condition. In addition to a maximum number of attempts, companies can set limits on total Tokens, runtime, tool calls, and context length.

Such restrictions may appear to sacrifice some task success rate, but compared with “letting the system keep trying indefinitely until it succeeds,” they are more consistent with a fundamental principle of production environments: failure is acceptable; loss of control is not.

Layer Four: Model Routing

Not every task requires a premium model such as Sonnet, GPT, or Gemini. Tasks such as entity matching, classification, format conversion, and deduplication can generally be handled first with rules, embedding models, or lower-cost models, with only low-confidence samples passed to a premium model.

A more sensible workflow is “low-cost model for initial filtering, premium model as a fallback,” rather than “use the most powerful model for everything from the start.” Caching should also be enabled for highly repetitive requests, while batch tasks that do not require real-time responses should use pricing plans better suited to offline processing.

What Does This Mean for Ordinary Companies?

Amazon operates at enormous scale, with annual capital expenditure expected to reach $200 billion, a substantial portion of which will go toward AI infrastructure and data centers. Some may argue that $1.8 million is merely a rounding error for a company of that size.

Yet Amazon’s scale is precisely what makes this case worth watching. Large companies have more mature procurement, financial, and cloud-management systems, but even they can allow a seemingly ordinary AI project to spiral out of control. For small and midsize companies with tighter budgets and less developed internal platforms, the risk is only greater.

This is especially true as companies move from fixed subscription models to Token- or usage-based pricing. Fixed per-seat fees make budgets relatively predictable; usage-based pricing ties costs directly to user behavior. A team adding more context, a workflow calling a model several more times, or a task being left running for several additional months will all eventually show up on the bill.

When purchasing AI services, companies therefore cannot compare only the “price per million Tokens.” What they really need to compare is the total cost per task: the average number of calls per task, the length of the input context, the failure rate, support for caching and batch processing, the availability of real-time usage APIs, the ability to break down bills by team and project, and whether spending can be automatically blocked when it exceeds the budget.

For developers, the most practical lesson is equally straightforward: do not treat a model call like an ordinary function call. Even if a normal function runs 10,000 times, its cost may remain close to zero. If a model call runs 10,000 times, the result may be an invoice that requires an explanation.

Putting AI Into Production Ultimately Comes Down to the “Braking System”

This incident will not diminish the value of Claude Sonnet, GPT, or other models in the enterprise. For information extraction, code generation, customer-service automation, and data processing, powerful models can indeed shorten development cycles and solve problems that once required extensive manual rule-writing.

But the more capable the model and the more autonomous the Agent, the less acceptable it is to leave cost control to manual inspections after a project goes live. Companies need to make budgets, permissions, rate limits, logging, circuit breakers, and audits part of the default AI platform configuration, just like identity authentication and data security.

The most important number to remember from Amazon’s incident is not $1.8 million, but “five months.” If overspending is discovered within a few hours, it is usually just an engineering incident. If it continues for five months before anyone notices, the organization lacks a complete feedback loop from API calls to financial reporting.

The true sign that an AI project is ready for production is not whether the model can complete the task, but whether it can stop promptly when it fails.

Companies preparing to deploy Agents should first answer at least three questions:

• What is the maximum number of Tokens and amount of money the task may consume?
• Which system is responsible for stopping it automatically before it exceeds the limit?
• When an anomaly occurs, can it be traced to a specific project, team, and call chain within minutes?

If the answer is simply “we review the vendor’s bill at the end of the month,” then the system is not ready for production. Amazon has already paid $1.8 million for this lesson. Other companies do not need to pay their tuition the same way.

References

• ITHome: Amazon’s AI Overspending Went Undetected for Five Months, With One Project Costing $1.8 Million—860% Over Budget — A summary of the Financial Times report and multiple cases of runaway AI costs disclosed at an internal Amazon meeting.

Note: The sections discussing the project’s specific technical architecture, number of calls, and prompts are technical analyses based on publicly available information and common AI Agent workflows. They do not represent implementation details that Amazon has publicly confirmed.