An Agent Retried 31,000 Times. How to Cap AI Spend per Customer

A three-person B2B team shared a bad Friday morning on r/SaaS this week. Their customers run agents on the team's infrastructure, the team pays the model bill, and customers pay a monthly plan. Overnight, one customer's agent hit a tool that kept returning an error, and it decided the fix was to retry with a slightly different prompt. It did that 31,000 times between 1:12am and 6:50am.

The bill was $4,700 in tokens, on a $400-a-month plan. The team ate most of it and sent a very awkward email.

If customers run agents on your account, you are exposed to the same thing. Below: why the limits most teams already have do not catch this, the per-customer budget design that does, the agent-level guards that stop a loop earlier, and what to put in your pricing so you are not negotiating after the fact.

The math of a runaway agent

It is worth running the numbers, because they show how little room you have:

• 1:12am to 6:50am is 338 minutes, so about 92 retries a minute.
• $4,700 over 31,000 retries is about $0.15 per attempt.
• At that rate the agent burned through the customer's entire $400 monthly plan in under 29 minutes.

Nobody is awake for that. Anything that depends on a human noticing a dashboard is too slow, which means the stop has to be automatic and it has to happen before the model call, not after the invoice.

Why global limits and spend alerts miss it

This team was not careless. They had a global rate limit and alerts on total spend. Neither fired in time, and it is easy to see why.

A global rate limit protects your infrastructure, not your margin. 92 calls a minute from one customer is often well under it. A total-spend alert watches the whole account, so one customer's spike has to be large enough to stand out against everyone else's normal usage, and if the threshold is set for the whole business, the damage is done by the time it trips.

The real gap was in the data. As the team put it, they had nothing per customer, because usage only got rolled up into the invoice at month end and nobody looked at it in between. Per-customer cost existed only as a billing artifact, not as something a running system could check.

Nobody did anything wrong, which is the point

The team's own read is the right one: the customer was not at fault. The agent did exactly what agents do when nothing tells them to stop. An error came back, so it tried again, slightly differently, which is often good behaviour. Retrying with a varied prompt is what makes agents resilient. It is also what makes them expensive when the tool is simply broken.

That means you cannot solve this with better customer behaviour or a stricter prompt alone. A system that spends real money in a loop needs limits outside the model, the same way a payment system does not rely on the caller to be sensible.

The fix: a budget check before every model call

After the incident, the team moved to a per-customer budget checked before every model call, with a soft alert at 1x plan and a hard stop at 2x. That design is a good default. To build it:

1. Keep a running spend counter per customer, per billing period, in something fast like Redis or a single indexed row. Update it after every model response using the tokens actually used.
2. Wrap every model call in one function that reads the counter first. If there is only one path to the model, the check cannot be skipped.
3. At 1x plan, alert the customer and your team, and keep running.
4. At 2x plan, refuse the call and return a clear error the agent and the customer can both understand.

Do not read from your billing system for this. Billing data is often delayed by minutes or hours, and at 92 calls a minute that is the whole problem. Estimate the cost of the call before sending it if you can, using input tokens plus the max output tokens, so one huge request cannot jump over the limit.

Stop the loop earlier with agent-level guards

A budget stop limits the damage. These guards stop the loop long before the budget matters:

• A maximum retry count per tool call. The team added this too, and said it should have existed before. Three to five attempts is plenty for most tools.
• Treat repeated identical errors as a stop signal. If a tool returns the same error twice, more attempts with different wording will not fix a broken tool. End the run and report the failure.
• A hard cap per run. Limit model calls, tokens and wall-clock time for a single agent run. No normal task needs 31,000 calls.
• A circuit breaker on the tool. When a tool fails repeatedly across runs, stop calling it for a few minutes. That protects every customer at once.
• Spike alerts per customer. Alert when one customer's hourly spend is many times their own normal, not when the whole account crosses a line.

See it when it happens

Limits stop the bleeding, but you also want to see the pattern. Tracing every run makes a loop obvious: the same tool span repeating with an error, again and again. Tools like telemetry.dev show each run as a single trace with cost per span and alert on cost spikes and agent loops. Whatever you use, log a cost per run and per customer from day one, because that data is painful to reconstruct later.

It is also worth testing your limits on purpose. Point an agent at a tool that always fails in a staging environment and confirm that it stops where you think it should. A retry cap nobody has ever seen fire is a guess. And if you are capping your own coding agent rather than your customers', the same thinking applies: see why your coding agent needs a hard budget cap.

Put it in your pricing and terms

The awkward email happened because nobody had decided in advance who pays. Decide now, and write it down:

• Say what happens at the limit. Hard stop, or overage at a stated rate. Either is fine; silence is not.
• Let customers set their own cap. A customer-controlled monthly limit below your default builds trust and removes the argument.
• Show usage in the product. A running number and a warning at 80% mean the first time they hear about a spike is not your email.
• Have a goodwill policy for loops. Runaway retries are usually nobody's fault. Deciding ahead of time how much you will absorb is cheaper than deciding at 7am.

If your plans bundle usage, also check your margin at the 2x stop. A customer who hits the hard stop every month at twice their plan price in tokens is a loss you have designed into your pricing.

What should happen when the hard stop fires

A hard stop that just throws an error is only half the job, because the customer's agent will read it and may try again. Return a specific, machine-readable error that says the budget is exhausted and that retrying will not help, so a well-behaved agent ends the run instead of looping on your limit.

Then tell a person. Email the account owner with what happened: which agent, which tool, how many calls, what it cost, and what to do next. Give them a clear way to raise the cap if the spend was intentional, because some customers will have a good reason, and a surprise outage on a legitimate batch job is its own awkward email. Keep the failed run's trace, so that the conversation starts from facts.

A one-afternoon checklist

1. Route every model call through one function.
2. Add a per-customer spend counter and check it in that function.
3. Set a soft alert at 1x plan and a hard stop at 2x.
4. Add a retry cap per tool call and a cap on calls per run.
5. Stop a run when a tool returns the same error twice.
6. Run a test agent against a tool that always fails and confirm that it stops.

Practical takeaways

• Global rate limits protect your servers, not your margin. Per-customer budgets protect your margin.
• Check the budget before the model call, from a fast counter, not after the invoice.
• Retrying is good agent behaviour until the tool is broken. Cap retries and treat repeated errors as a stop signal.
• Cap calls, tokens and time per run. No normal task needs thousands of calls.
• Test your limits on purpose against a tool that always fails.
• Decide who pays for runaway usage before it happens, and put it in your terms.

Conclusion

Agents do not know what money is. They will keep trying as long as nothing tells them to stop, and most of the time that persistence is what you are paying for. The fix is not a smarter agent. It is a small piece of boring infrastructure outside the agent: a counter, a check before every call, a retry cap, and a hard stop that nobody has to be awake for.

For a weekly note on what is actually working for builders shipping with AI, subscribe to the HelloBuilder newsletter.