Per-User Rate Limiting for LLM Features

Per-user LLM rate limiting bounds how fast a single user can call your AI features. It's a close cousin of a spend cap, but it solves a different problem: a cap bounds *total cost over time*, a rate limit bounds *burst* — the number of calls (or tokens) in a short window.

Why you need both

• A spend cap stops a user from costing more than $X this month.
• A rate limit stops a user from firing 500 calls in a minute — protecting latency, provider quotas, and cost all at once.

Without a rate limit, a single script can exhaust your provider's account-wide rate quota and take the feature down for everyone, long before any monthly cap is reached.

What to limit on

For LLMs, requests-per-minute is a blunt instrument — one request can be 200 tokens or 200,000. Prefer a tokens-per-window limit, or combine both: a request cap for abuse and a token cap for cost.

const guard = new Guard(anthropic, {
  project: "prod-app",
  limits: {
    perUser: {
      requestsPerMinute: 20,
      tokensPerMinute: 50_000,
    },
  },
});
// Bursts past either limit are rejected with RateLimitedError.

Choosing the window

Short windows (per-minute) catch abuse fast but can frustrate legitimate power users. A common pattern is a token bucket: a steady refill rate with a burst allowance, so normal usage never notices the limit and only sustained hammering hits it.

Rate limiting and cost caps together

The complete per-user picture is: a hard spend cap for the month, a rate limit for the minute, and token limits per call. Together they make one user's worst behaviour bounded on every axis.