How to Set AI API Concurrency Limits? Four Criteria Plus Load Testing

AI API concurrency limits are not constant—they're determined by the first wall you hit among four constraints

The concurrency limit for an AI API isn't about how high you want to set it, but about four mutually constraining criteria: account-side RPM/TPM quotas, the inference-side throughput collapse point, the slot duration occupied by single-request output length, and the p95 tail latency your business can tolerate. Your concurrency pool can only be the intersection of these four constraints, and that intersection changes with model architecture, context length, and account tier. On 2026-08-11, NVIDIA open-sourced the Nemotron-3.5-Lightning-30B-A3B model (official Hugging Face model card), with 30B total parameters, only 3B activated per token, native support for up to 1M context, and designed specifically for low-latency, high-throughput agent loops; DeepInfra also announced the same day in a blog post the launch of its OpenAI-compatible Serverless API, with specs per the official page. Such low-activation-parameter MoE changes the per-request compute and memory costs, but how much concurrency a client can actually open still requires measuring against the four criteria in this article; you cannot reuse hard-coded constants from old models.

First, separate three quantities: concurrency, RPM/TPM quotas, and actual throughput

Many people conflate concurrency, RPM/TPM quotas, and throughput. Concurrency is the number of in-flight requests; RPM/TPM is the quota on the account's billing window; throughput is the number of tokens completed per unit time. The relationship among them is intuitively close to Little's Law: in-flight requests ≈ arrival rate × average request duration. In other words, if the average request duration increases, the same arrival rate requires larger concurrency to sustain throughput; quotas cap the arrival rate.

Criterion 1: How much concurrency can account-side quotas support? Start with RPM/TPM reverse calculation

First, use the average request duration and RPM to calculate the theoretical in-flight limit: concurrency ≈ RPM × average request duration / 60. Then use the average per-request token count and TPM to calculate the token-dimension limit: concurrency ≈ TPM / (average tokens per request × average request duration). Take the smaller of the two as the first ceiling. Note that TPM typically includes both input and output tokens, and in long-context scenarios TPM often depletes before RPM. For specific tiers, check the rate limit and quota documentation of the platform you use.

Criterion 2: The inference-side throughput collapse point—only measurable via concurrency ramp tests

Beyond a certain concurrency point, total throughput stops increasing, TTFT and p95 lengthen first, and then queuing and 429 responses appear—requests are queued on the server rather than executed in parallel. This collapse point can only be read from a concurrency ramp experiment: x-axis concurrency, y-axis total token/s and p95, the inflection point is the collapse point. Use 70%–80% of the inflection point as the conservative starting point for your production concurrency pool; this is an engineering convention, not an industry standard. For example, if the inflection point is N concurrency, take 0.7N–0.8N, floor. Specific steps are in the load-testing section below.

Criterion 3: Single-request output length determines how long a concurrency slot is occupied

Decoding time grows almost linearly with output tokens. The longer the output, the longer the in-flight request residence, resulting in lower actual RPM and higher TPM. Therefore, set different concurrency pools for different task types:

Additionally, max_tokens and stop conditions are part of concurrency governance; limiting output length improves concurrency efficiency.

Criterion 4: Reverse-calculate concurrency limit from the p95 tail latency your business can accept

The first three criteria give a feasible range for AI API concurrency; the final value is determined by business SLOs. For interactive scenarios, lock the p95/p99 target first and then reverse-calculate concurrency; for offline batch processing, you can sacrifice tail latency for total throughput.

Specific thresholds must be set according to your own SLO and the platform's rate limit documentation.

Model architecture rewrites concurrency economics: low-activation MoE and million-token context

Take Nemotron-3.5-Lightning-30B-A3B (released by NVIDIA on 2026-08-11, Hugging Face model card) as an example: its hybrid Mamba-2 and MoE architecture activates only 3B parameters per token, theoretically lowering per-request compute and memory costs, allowing the same compute resources to accommodate more in-flight requests. However, the service provider's batching strategy, KV cache budget, and account quotas remain practical constraints, especially under long context, where KV cache can become the bottleneck again. Therefore, "fewer activated parameters so concurrency can be higher" is just a hypothesis that must be verified by testing; you cannot directly reuse concurrency constants from old models.

How to load test LLM API concurrency: fixed prompts, fixed output length, and concurrency ramp

To find your own inflection point, follow these steps:

1. Fix the prompt set and input length distribution.
2. Use max_tokens and ignore EOS to lock output length.
3. Increase concurrency exponentially from 1 (1, 2, 4, 8...).
4. Run each level long enough to obtain a stable p95 (sample size sufficient that the quantile no longer drifts significantly with additional samples).
5. Record TTFT, end-to-end latency, total token throughput, 429 and timeout ratios.
6. Sample at multiple times to avoid shared capacity fluctuations (shared capacity varies by time).
7. Keep per-request logs for quantile calculation.

How to implement the concurrency pool: semaphore, connection reuse, backoff, and timeout budget

Go from numbers to implementation: use semaphores or queues to limit in-flight requests, not unbounded coroutines; reuse HTTP connections and client instances; implement exponential backoff with jitter on 429 responses; distinguish retryable and non-retryable errors; include retry count in the timeout budget to avoid tail latency amplification. An advanced approach is AIMD-style adaptive concurrency: multiplicative decrease on 429, additive increase when stable. See AI API 429 error troubleshooting and AI API latency optimization for details.

Why concurrency limits must be re-tested per model: one script for multiple models

When you change models, versions, or providers, the concurrency inflection point shifts globally; you must rerun regression. With an OpenAI-compatible interface, the same test script only needs to change the model field. For example, NexAIX provides an OpenAI Chat Completions compatible interface with base URL https://api.nexaix.net/v1，支持流式输出与工具调用；满载时返回标准, 429 and retry recommendations, no silent switching to cheaper models, and the returned model field corresponds to the actual executing model, so the inflection point read from load tests is reproducible and attributable. More methods can be found in TTFT/throughput load testing for self-hosted AI APIs and Model migration and regression validation for OpenAI-compatible APIs.

Concurrency regression checklist after changing models or providers

After changing models, re-confirm the AI API concurrency value using this checklist.

• Rerun the concurrency ramp to confirm the new inflection point.
• Verify the RPM/TPM quotas for the new tier.
• Re-check whether the average output length has changed.
• Recalculate timeout and backoff parameters.
• Confirm whether 429 and error code semantics are consistent.
• Verify that the returned model field matches the actual executed model.
• After deployment, monitor p95 and 429 rates and set alert thresholds.

Before setting the concurrency pool, run a ramp on the model you'll actually use, and record the inflection point, p95, and 429 rate as a baseline. If you want to compare multiple models with the same script, first check the interface description in NexAIX's rate limit and quota documentation, then run a ramp using the test quota.

FAQ

What concurrency number should I set?

There is no universal optimal value. First test the inflection point, use 70%–80% of it as the production pool starting point, then adjust based on account quotas and business SLOs.

Why does throughput drop after increasing concurrency?

When concurrency exceeds the inference-side throughput collapse point, requests are queued rather than executed in parallel, causing total throughput to drop and latency to increase. You need to find the inflection point via load testing.

What is the relationship between RPM/TPM and concurrency?

RPM is the requests-per-minute quota, TPM is the tokens-per-minute quota. Concurrency is constrained by both; you can derive the theoretical upper limit via formulas and take the smaller value.

How to control concurrency in agent loops?

Distinguish between outer task concurrency and inner sequential steps within a task. For the outer layer, use a semaphore to limit the number of concurrently executing tasks; for the inner layer, keep multi-step calls within a single task sequential to avoid a single task filling the concurrency pool.

How to adjust concurrency if batch calls keep getting 429?

First lower concurrency below the inflection point, and implement exponential backoff with jitter to avoid retry storms; if still 429, check if the quota is exhausted.

Can I open higher concurrency with MoE that activates fewer parameters?

Theoretically lower per-request cost, but it depends on the provider's batching, KV cache, and quotas, so it must be empirically verified; you cannot assume it.