虚拟密钥与预算（2/6）

TPM limits are enforced by reserving tokens before the call and reconciling against real usage after it. When a request omits max_tokens / max_completion_tokens, LiteLLM has to guess how many output tokens to reserve, and the built-in guess is a single static estimate shared by every key, team and model.

That guess is wrong in both directions. If your model really emits more than the estimate, concurrent requests are all admitted against an under-reservation and the window overruns the limit once they finish. If it emits far less, the over-reservation blocks requests the budget could have served.

Declare what your models actually emit with default_estimated_output_tokens (one value) and default_estimated_output_tokens_per_model (a map of model name to value). Both are settable on a key and on a team.

curl --location 'http://0.0.0.0:4000/key/generate' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
  "team_id": "my-prod-team",
  "tpm_limit": 1000000,
  "default_estimated_output_tokens": 2048,
  "default_estimated_output_tokens_per_model": {
    "gpt-5.6-terra": 4096,
    "gpt-5.6-luna": 1024
  }
}'

The same two fields work on /team/new and /team/update, and both are editable from the Admin UI on the key and team settings pages.

curl --location 'http://0.0.0.0:4000/team/update' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
  "team_id": "my-prod-team",
  "default_estimated_output_tokens": 4096,
  "default_estimated_output_tokens_per_model": {"gpt-5.6-terra": 8192}
}'

Resolution order for the reserved output budget, first match wins:

Priority Source
1 Request max_tokens or max_completion_tokens
2 Key default_estimated_output_tokens_per_model[model]
3 Key default_estimated_output_tokens
4 Team default_estimated_output_tokens_per_model[model]
5 Team default_estimated_output_tokens
6 Built-in static estimate

Values must be positive integers, and the management endpoints reject anything else with a 422 naming the offending field. A value that is missing, or malformed because it was written straight into metadata rather than through these fields, falls through to the next tier, so a key that declares nothing behaves exactly as it does today. Embedding requests are unaffected since they produce no output tokens.

:::tip Size the estimate from your observed output distribution for that model, around the p95, not from its maximum context. Both directions cost you something:

  • Declaring more than the model emits throttles traffic the budget could have served. If the declared value plus the input estimate exceeds the limit the request is charged against, every such request is refused, and the proxy logs the reservation and the limit at debug level so you can see why.
  • Declaring less than the model emits is worse than declaring nothing, because the reservation is then smaller than the built-in estimate and more concurrent requests are admitted before the real usage lands. :::

:::note The proxy already hard-caps generation for tenants whose smallest applicable TPM limit is under 4096, by injecting a max_tokens of a quarter of that limit. A declaration larger than that cap raises it, so you are never truncated below what you said your model emits. A declaration smaller than it is ignored, because an estimate describes the typical response and must not silently truncate the long tail. :::

Per Team

Use /team/new or /team/update, to persist rate limits across multiple keys for a team.

curl --location 'http://0.0.0.0:4000/team/new' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{"team_id": "my-prod-team", "max_parallel_requests": 10, "tpm_limit": 20, "rpm_limit": 4}'

See Swagger

Expected Response

{
    "key": "sk-sA7VDkyhlQ7m8Gt77Mbt3Q",
    "expires": "2024-01-19T01:21:12.816168",
    "team_id": "my-prod-team"
}

Per Team Per Model

Set rate limits per model for a team

Use model_rpm_limit and model_tpm_limit to set rate limits per model for all keys belonging to a team. These limits apply across all keys in the team and are inherited by keys unless overridden at the key level.

Use /team/new or /team/update with model_rpm_limit and model_tpm_limit as dictionaries mapping model names to their limits:

curl --location 'http://0.0.0.0:4000/team/new' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
  "team_id": "my-prod-team",
  "model_rpm_limit": {"gpt-5.6-terra": 100, "gpt-5.6-luna": 200},
  "model_tpm_limit": {"gpt-5.6-terra": 10000, "gpt-5.6-luna": 20000}
}'

Update existing team with per-model limits:

curl --location 'http://0.0.0.0:4000/team/update' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
  "team_id": "my-prod-team",
  "model_rpm_limit": {"gpt-5.6-terra": 100, "gpt-5.6-luna": 200},
  "model_tpm_limit": {"gpt-5.6-terra": 10000, "gpt-5.6-luna": 20000}
}'

Alternative: Use metadata

You can also pass per-model limits via the metadata field:

curl --location 'http://0.0.0.0:4000/team/update' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
  "team_id": "my-prod-team",
  "metadata": {
    "model_rpm_limit": {"gpt-5.6-terra": 100, "gpt-5.6-luna": 200},
    "model_tpm_limit": {"gpt-5.6-terra": 10000, "gpt-5.6-luna": 20000}
  }
}'

Resolution order: When a key belongs to a team, rate limits are resolved as: Key metadata > Key model_max_budget > Team metadata. Keys can override team-level per-model limits with their own model_rpm_limit or model_tpm_limit.

Verify: Make a /chat/completions request and check response headers x-litellm-key-remaining-requests-{model} and x-litellm-key-remaining-tokens-{model} for the model-specific limits.

See Swagger

Per Internal User

Use /user/new or /user/update, to persist rate limits across multiple keys for internal users.

curl --location 'http://0.0.0.0:4000/user/new' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{"user_id": "krrish@berri.ai", "max_parallel_requests": 10, "tpm_limit": 20, "rpm_limit": 4}'

See Swagger

Expected Response

{
    "key": "sk-sA7VDkyhlQ7m8Gt77Mbt3Q",
    "expires": "2024-01-19T01:21:12.816168",
    "user_id": "krrish@berri.ai"
}

Per Key

Use /key/generate, if you want them for just that key.

curl --location 'http://0.0.0.0:4000/key/generate' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{"max_parallel_requests": 10, "tpm_limit": 20, "rpm_limit": 4}'

Expected Response

{
    "key": "sk-ulGNRXWtv7M0lFnnsQk0wQ",
    "expires": "2024-01-18T20:48:44.297973",
    "user_id": "78c2c8fc-c233-43b9-b0c3-eb931da27b84"  // 👈 auto-generated
}

Per API Key Per model

Set rate limits per model per api key

Set model_rpm_limit and model_tpm_limit to set rate limits per model per api key

Here gpt-5.6-terra is the model_name set on the litellm config.yaml

curl --location 'http://0.0.0.0:4000/key/generate' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{"model_rpm_limit": {"gpt-5.6-terra": 2}, "model_tpm_limit": {"gpt-5.6-terra": 1000}}'

Expected Response

{
    "key": "sk-ulGNRXWtv7M0lFnnsQk0wQ",
    "expires": "2024-01-18T20:48:44.297973"
}

Verify Model Rate Limits set correctly for this key

Make /chat/completions request check if x-litellm-key-remaining-requests-gpt-5.6-terra returned

curl -i http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-ulGNRXWtv7M0lFnnsQk0wQ" \
  -d '{
    "model": "gpt-5.6-terra",
    "messages": [
      {"role": "user", "content": "Hello, Claude!ss eho ares"}
    ]
  }'

Expected headers

x-litellm-key-remaining-requests-gpt-5.6-terra: 1
x-litellm-key-remaining-tokens-gpt-5.6-terra: 179

These headers indicate:

  • 1 request remaining for the gpt-5.6-terra model for key=sk-ulGNRXWtv7M0lFnnsQk0wQ
  • 179 tokens remaining for the gpt-5.6-terra model for key=sk-ulGNRXWtv7M0lFnnsQk0wQ

Per Agent

Set rate limits on agents registered with the Agent Gateway.

Agent-level limits cap total throughput across all sessions:

curl -X POST 'http://0.0.0.0:4000/v1/agents' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{"agent_name": "my-agent", "agent_card_params": {"name": "my-agent", "description": "My agent", "url": "http://my-agent:8080", "version": "1.0.0"}, "tpm_limit": 100000, "rpm_limit": 100}'

Session-level limits cap throughput per individual session:

curl -X POST 'http://0.0.0.0:4000/v1/agents' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{"agent_name": "my-agent", "agent_card_params": {"name": "my-agent", "description": "My agent", "url": "http://my-agent:8080", "version": "1.0.0"}, "session_tpm_limit": 50000, "session_rpm_limit": 50}'

You can also set max_iterations (call count cap) and max_budget_per_session (dollar cap) per session via litellm_params. See Agent Iteration Budgets for details.

For customers

:::info

You can also create a budget id for a customer on the UI, under the 'Rate Limits' tab.

:::

Use this to set rate limits for user passed to /chat/completions, without needing to create a key for every user

Step 1. Create Budget

Set a tpm_limit on the budget (You can also pass rpm_limit if needed)

Both are optional; a budget with neither set applies no LiteLLM TPM or RPM limit to its customers, and only provider rate limits apply

curl --location 'http://0.0.0.0:4000/budget/new' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
    "budget_id" : "free-tier",
    "tpm_limit": 5
}'

Step 2. Create Customer with Budget

We use budget_id="free-tier" from Step 1 when creating this new customers

curl --location 'http://0.0.0.0:4000/customer/new' \
--header "Authorization: Bearer $LITELLM_API_KEY" \
--header 'Content-Type: application/json' \
--data '{
    "user_id" : "palantir",
    "budget_id": "free-tier"
}'

Step 3. Pass user_id id in /chat/completions requests

Pass the user_id from Step 2 as user="palantir"

curl --location 'http://localhost:4000/chat/completions' \
    --header "Authorization: Bearer $LITELLM_API_KEY" \
    --header 'Content-Type: application/json' \
    --data '{
    "model": "llama3",
    "user": "palantir",
    "messages": [
        {
        "role": "user",
        "content": "gm"
        }
    ]
}'

Set default budget for ALL internal users

Use this to set a default budget for users who you give keys to.

This will apply when a user has user_role="internal_user" (set this via /user/new or /user/update).

This will NOT apply if a key has a team_id (team budgets will apply then). Tell us how we can improve this!

  1. Define max budget in your config.yaml
model_list:
  - model_name: "gpt-5.6-luna"
    litellm_params:
      model: gpt-5.6-luna
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  max_internal_user_budget: 0 # amount in USD
  internal_user_budget_duration: "1mo" # reset every month
  1. Create key for user
curl -L -X POST 'http://0.0.0.0:4000/key/generate' \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H 'Content-Type: application/json' \
-d '{}'

Expected Response:

{
  ...
  "key": "sk-X53RdxnDhzamRwjKXR4IHg"
}
  1. Test it!
curl -L -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-X53RdxnDhzamRwjKXR4IHg' \
-d '{
    "model": "gpt-5.6-luna",
    "messages": [{"role": "user", "content": "Hey, how's it going?"}]
}'

Expected Response:

{
    "error": {
        "message": "ExceededBudget: User=<user_id> over budget. Spend=3.7e-05, Budget=0.0",
        "type": "budget_exceeded",
        "param": null,
        "code": "400"
    }
}

Multi-instance rate limiting

Important Notes: - Rate limits apply to any key, user or team that has tpm_limit, rpm_limit or max_parallel_requests set, regardless of role. The master key has no limits unless you configure them. - When testing rate limits, use a virtual key with explicit limits so the limiter has something to enforce.

Changes: - This moves to using async_increment instead of async_set_cache when updating current requests/tokens. - The in-memory cache is synced with redis every 0.01s, to avoid calling redis for every request. - In testing, this was found to be 2x faster than the previous implementation, and reduced drift between expected and actual fails to at most 10 requests at high-traffic (100 RPS across 3 instances).

Hard rate limit enforcement (fail closed)

Across several instances, the tpm, rpm, and max_parallel_requests counters live in Redis (general_settings.coordination_redis or the REDIS_* environment variables) so every instance enforces the same limit. While Redis is unreachable, each instance falls back to counters in its own memory and keeps serving, so a key with rpm_limit: 2 is admitted up to 2 requests per instance, N times the limit across N instances, until Redis is back

For deployments where a configured rate limit must be a hard ceiling even while Redis is down, set fail_closed_rate_limit_enforcement:

general_settings:
  fail_closed_rate_limit_enforcement: true

With it enabled, a request whose counters cannot be verified against Redis is rejected with a 503 instead of being admitted against a per-instance counter. It is a 503 rather than a 429 so clients and load balancers can tell a Redis outage from a rate limit. The setting changes nothing while Redis answers, requests that carry no rate limit are unaffected, and post-request accounting stays best effort, so a request that was already admitted is never failed after the fact

Leave the setting off (the default) to keep serving through a Redis outage on per-instance limits. Without Redis the setting has no effect: a proxy that starts with it on and no Redis configured logs a warning and keeps enforcing limits per instance. The legacy limiter selected by LEGACY_MULTI_INSTANCE_RATE_LIMITING=true ignores the setting as well

Grant Access to new model

Use model access groups to give users access to select models, and add new ones to it over time (e.g. mistral, llama-2, etc.).

Difference between doing this with /key/generate vs. /user/new? If you do it on /user/new it'll persist across multiple keys generated for that user.

Step 1. Assign model, access group in config.yaml

model_list:
  - model_name: text-embedding-ada-002
    litellm_params:
      model: azure/azure-embedding-model
      api_base: "os.environ/AZURE_API_BASE"
      api_key: "os.environ/AZURE_API_KEY"
      api_version: "2023-07-01-preview"
    model_info:
      access_groups: ["beta-models"] # 👈 Model Access Group

Step 2. Create key with access group

curl --location 'http://localhost:4000/user/new' \
-H 'Authorization: Bearer <your-master-key>' \
-H 'Content-Type: application/json' \
-d '{"models": ["beta-models"], # 👈 Model Access Group
            "max_budget": 0}'

Create new keys for existing internal user

Just include user_id in the /key/generate request.

curl --location 'http://0.0.0.0:4000/key/generate' \
--header 'Authorization: Bearer <your-master-key>' \
--header 'Content-Type: application/json' \
--data '{"models": ["azure-models"], "user_id": "krrish@berri.ai"}'

API Specification

GenericBudgetInfo

A Pydantic model that defines budget information with a time period and limit.

class GenericBudgetInfo(BaseModel):
    budget_limit: float  # The maximum budget amount in USD
    time_period: str    # Duration string like "1d", "30d", etc.

Fields:

  • budget_limit (float): The maximum budget amount in USD
  • time_period (str): Duration string specifying the time period for the budget. Supported formats:
  • Seconds: "30s"
  • Minutes: "30m"
  • Hours: "30h"
  • Days: "30d"

Example:

{
  "budget_limit": "0.0001",
  "time_period": "1d"
}