How to Choose a Long-Context API: 5 Cost Criteria for 1M Windows

When choosing a long-context API, look at only 5 criteria: effective usable length, per-call cost, prefix cache hit rate, TTFT degradation curve, and multimodal conversion rate. You don't need to be dazzled by the "1M window" hype, nor should you blindly revert to RAG. First calculate these 5 numbers, then decide whether to expand the window or do retrieval.

Recently, Kimi K3 (2.8T total params / 104B active MoE model) has gone live on several third-party inference platforms, natively supporting 1,048,576 token multimodal context, making million-level windows an API option for the first time. But this doesn't mean you can blindly fill it up—the criteria below will tell you why.

Conclusions First: What Scenarios Should Expand the Window, and What Scenarios Still Need Retrieval Chunking

If your business documents are frequently re-queried and the prefix (system prompt + fixed documents) can remain unchanged long-term, directly expanding the window is usually more cost-effective; if documents are used once, the corpus far exceeds the window, or precise traceability is needed, then RAG retrieval chunking remains a better choice; if both are mixed—first retrieve rough, then use long window for deep reading—is the actual form of most production systems.

Criterion 1: Declared Context Length ≠ Effective Usable Length, How to Test the Inflection Point Yourself

The maximum input sequence length advertised by vendors has a gap with the effective usable length in business. Literal "needle in a haystack" retrieval tends to pass, but cross-segment synthesis and implicit associations often start degrading at 32K~64K.

Test method: Use your own business Q&A set for length gradients, placing answer anchors at the beginning, middle, and end of the document, observing the inflection point where accuracy changes with length. This inflection point must be tested by yourself—don't copy others' numbers. Make the length gradient a repeatable regression task; the approach can reference AI Model Evaluation.

Criterion 2: Per-call Input Cost = Actual Filled Tokens × Unit Price, Calculate First Then Choose

Per-call input cost = actual filled tokens × unit price; multiply by daily call volume to get monthly total. In most scenarios, you won't fill the window; what really determines your bill is average fill and call frequency.

So, separating window cap and actual fill, you'll find you don't need 1M. Include average fill, cache hit rate, and call frequency in accounting; see AI API Cost Optimization for line-item methods.

Criterion 3: Whether Fixed Prefix Can Be Covered by Cache, Determining Marginal Cost of Long-Context APIs

In high-frequency multi-turn and Agent scenarios, the marginal cost of long-context APIs heavily depends on prefix prompt caching. When prefix hits, input billing often gets a significant discount; industry range is commonly 80%~90% tier, but discount ratios, minimum cacheable length, and cache retention times vary by platform. You must refer to your platform's current documentation.

Engineering advice: Put system prompts and long documents in a constant prefix segment, postpose changing content, and monitor cache hit rate continuously. For cache retention policies, see How Long Can API Caches Be Retained.

Criterion 4: How TTFT Degrades with Fill Length, Test with Length and Concurrency Gradients Separately

Prefill compute and communication overhead growing with context length is a physical law. When cache misses, near-million-level fill pushes first-token latency from seconds to tens of seconds—architecturally determined.

Test method: Fix concurrency to run length gradients; fix length to run concurrency gradients; record TTFT and overall completion time separately. With streaming output, users perceive TTFT, not total time. For deeper understanding of latency components, read AI API Latency.

Criterion 5: Multimodal Input Token Conversion Rate, How to Incorporate Images and Videos into Budget

A 1M native multimodal window means images and video frames must be converted to tokens and occupy the same budget; conversion rates may differ across platforms. Method: Use a fixed material to test the token count returned by usage, reverse-engineer the conversion ratio, then write it into your context budget table. Specific coefficients follow platform docs.

What the 1M Tier Looks Like Now: Reference with Kimi K3's MoE Architecture and Native Multimodal Window

Kimi K3 is an open-source MoE model by Moonshot AI with 2.8T total parameters, 104B activated, using Stable LatentMoE architecture (16 of 896 experts activated) and Kimi Delta Attention mechanism, natively supporting 1,048,576 token multimodal long context for text, images, and videos. As of August 2026, it has been launched on platforms like DeepInfra and OpenRouter via API. These specs and availability are public info from August 2026, from third-party inference platform model pages and developer guides, not a complete vendor-official parameter sheet; specific context limits, multimodal support, and availability are subject to official docs and each platform's model page current info.

Note: MoE sparse activation reduces per-token compute, but doesn't change the overhead of long-context Prefill and KV cache growing with length, so "parameter sparsity" ≠ "cheap long window."

Expansion vs. Retrieval Chunking Rules: Based on Document Reuse Rate and Q&A Rounds

Hybrid (retrieve rough, then deep read with long context API) is most practical in most production systems.

Minimal Reproducible Script: Run Length Gradient Comparisons Across Models with the Same Long Document

The script below uses OpenAI-compatible interface; after setting base_url to point to https://api.nexaix.net/v1, just change the model parameter to run all length gradients on the same code, avoiding SDK differences polluting comparisons. Streaming output for TTFT; check the model field in the response body to verify actual model executed; for 429, back off per retry advice.

import openai, csv, time

client = openai.OpenAI(base_url="https://api.nexaix.net/v1", api_key="YOUR_KEY")

original_doc = open("your_doc.txt", encoding="utf-8").read()
# 按字符近似截断，真实 token 数以 usage 为准
char_lengths = [8192, 32768, 131072, 524288]
questions = ["文档的核心结论是什么？", "第二处的论据是什么？"]

with open("results.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["len", "question", "input_tokens", "ttft", "total_time", "answer"])
    for L in char_lengths:
        doc = original_doc[:L]
        for q in questions:
            messages = [{"role":"system","content":"你是一个严谨的分析师。"}, {"role":"user","content": doc + "\n\n" + q}]
            t0 = time.time()
            stream = client.chat.completions.create(model="kimi-k3", messages=messages, stream=True, stream_options={"include_usage": True})
            chunks = []
            ttft = None
            usage = None
            for chunk in stream:
                if ttft is None and chunk.choices[0].delta.content:
                    ttft = time.time() - t0
                if chunk.choices[0].delta.content:
                    chunks.append(chunk.choices[0].delta.content)
                if chunk.usage:
                    usage = chunk.usage
            total = time.time() - t0
            prompt_tokens = usage.prompt_tokens if usage else L
            writer.writerow([L, q, prompt_tokens, ttft, total, "".join(chunks)])

For specific model context limits, multimodal support, and pricing, check current info on the NexAIX model page and pricing page.

Regression Checklist for Long-Context API Before Switching Models

Before switching long-context API providers, the following six items must be re-tested item by item.

• [ ] Re-test effective length inflection (with business Q&A set)
• [ ] Prefix cache still hits (monitor hit rate)
• [ ] TTFT meets target concurrency (length & concurrency gradients)
• [ ] Multimodal conversion ratio changed (test with fixed material)
• [ ] Degradation path for context length exceeded is ready (auto-truncate / chunked summary / fallback retrieval)
• [ ] Response body model field matches expectation

Copy this checklist directly into internal docs and check off each item before launch.

FAQs

How many characters can a million context actually hold?

Different tokenizers have very different segmentation granularity for Chinese; don't estimate with a fixed ratio. Method: Take a 10,000-character real business text, call the API once, read usage.prompt_tokens in the response, get your actual conversion factor with "characters ÷ tokens", then multiply by 1,048,576 to back-calculate capturable characters; the coefficient can vary by over 30% across models for the same text. Effective length is far below the window cap; still rely on the measured inflection from criterion 1.

Why is the first token of a long context so slow?

When cache misses, Prefill must process all input tokens; compute grows linearly with length, and 1M level can cause tens of seconds of TTFT. This is a physical limit, not a vendor defect.

Does MoE model long-context throughput drop?

MoE's sparse activation reduces per-token activation compute, saving inference power compared to dense models of equivalent total parameter scale; but KV cache and Prefill communication overhead for long context are not reduced by sparsity, so throughput still drops as fill length increases. The magnitude is highly correlated with deployment (parallel strategy, quantization precision); self-test on target platform using concurrency gradients from criterion 4.

How to resolve context length exceeded errors?

First, split the request into multiple shorter segments or use summarization. Second, enable auto-truncation or fallback retrieval. Long-term, evaluate whether window expansion is needed, or switch to hierarchical summarization.

Which is more cost-effective: 1M context model or RAG?

Depends on document reuse rate: high-frequency multi-turn with cacheable prefix, window expansion is more cost-effective; one-time or huge corpus, RAG is more frugal. Recommend deciding after measuring cost and latency with this article's length gradient script.