Why AI API rate limits are not a spend cap

Four controls answer different questions

A support workflow can remain below ten requests per minute while repeatedly sending a long conversation to an expensive model. Conversely, a brief traffic burst can hit a rate limit without exceeding a task’s financial allowance. Set both controls around the resource you intend to protect.

• Rate: how quickly requests or tokens may arrive.
• Concurrency: how many calls may be outstanding together.
• Request bounds: permitted input size, output ceiling, model, and chargeable features.
• Budget: how much accounted usage the tenant, project, or task may authorize across calls.

OWASP’s Unbounded Consumption guidance recommends quotas, resource management, timeouts, and monitoring. None of those labels proves your application has connected every outbound path to the control.

Check what your provider actually enforces

OpenAI’s rate-limit documentation describes request and token windows separately from spend controls. Its current spend-limit documentation distinguishes alerts, which permit traffic to continue, from configurable hard organization and project limits, which reject affected requests. It also warns that enforcement is not instantaneous and recorded spend can slightly exceed the configured amount.

Do not assume every provider’s “budget” setting behaves the same way, or that a provider project limit allocates a budget to each customer task. Check the exact account, billing scope, error codes, reset period, and enforcement behavior. Treat provider controls as another layer, with their documented limits.

An original reservation ledger with invented prices

Consider a fabricated task with 100 demo credits. These are invented accounting units, not dollars, provider prices, measurements, or savings. In this example only, input costs one credit per unit and output costs two. Worker A proposes at most 20 input units and 30 output units: reserve 20 × 1 + 30 × 2 = 80 credits before dispatch.

Two workers must not independently read “100 available” and both dispatch an 80-credit call. Check eligibility and create the reservation in one atomic shared-store operation. Give each attempt a unique ledger identifier and make settlement idempotent. A repeated completion event must not release the same reservation twice.

OpenAI’s per-run spending controller example also distinguishes reservations from settlement and notes that a process-local lock does not coordinate multiple workers. The sequence above is our own illustrative accounting design, not an implemented Boundary service.

A reservation needs a defensible bound

An average expected response cost is not a safe reservation ceiling. Validate the maximum supported input and output, the selected model, the applicable pricing revision, and every enabled charge category. Include separately billed tools or exclude them from the supported workflow. Do not double-count categories that a provider already includes in another total.

Recheck the reservation when a fallback model, longer conversation, or additional tool changes the request. If you cannot bound a feature’s cost, decline it or require a separately scoped allowance. If authoritative usage exceeds the reservation, record the discrepancy and stop further dispatch for that run; do not silently reset its allowance.

Retries are attempts, not free recovery

A timeout tells you that the caller stopped waiting, not necessarily whether upstream work occurred. For worker B, keep the 30-credit hold until you can reconcile the outcome. A replacement costing 30 cannot be admitted against the remaining 15. Do not release the hold merely because a lease expires or a worker disappears.

Inventory retries in the application, queue, proxy, and SDK. OpenAI’s JavaScript SDK documentation describes automatic retries for certain failures; wrapping an SDK in another retry loop can multiply attempts. Set a total attempt budget and operation deadline, and account for each possible dispatch. Respect documented server delays for transient failures. Retrying a billing-limit error does not create more budget.

Use provider-supported retrieval or idempotency behavior only where the specific endpoint documents it. An application correlation identifier alone does not prove that repeating a request avoids duplicate inference charges.

A stopped stream is not a zero-cost request

A user closing a tab can leave generated work and uncertain usage. OpenAI’s Chat Completions streaming reference warns that an interrupted or cancelled stream may lack the final total-usage chunk. Missing usage is not evidence of zero consumption.

Stop future dispatch, cancel through the supported API where applicable, and retain the unresolved reservation. Reconcile using the provider evidence available for that endpoint. Delayed account totals can help accounting later, but they cannot automatically authorize the next call now.

Check the boundary with synthetic failures

Before connecting real spend, use a recording stub and a small ledger fixture. Race two reservations; return the same completion twice; crash after reservation; time out after simulated acceptance; and interrupt a stream before usage arrives. Assert that no denied attempt reaches the outbound stub and unknown outcomes retain their holds. Include a normal call that settles below its reservation, so an always-blocked system cannot pass as useful.

These are proposed checks, not reported test results. Boundary is a prelaunch commercial model router with planned request usage policies. This article does not establish a shipped budget engine, a guaranteed financial cap, or control over requests that bypass a router.