OpenAI max_tokens vs Context Window —

The model stops mid-sentence. Finish reason is length. Your prompt was a short question plus a pasted 40-page PDF dump. You set max_tokens: 2048 thinking that reserved two thousand tokens for the answer. Instead the API counted the prompt against the context window, left almost no room, and truncated. max_tokens (or max_completion_tokens) caps the completion; it does not expand the context window.

Two different budgets

Concept	Meaning
Context window	Hard limit on prompt + completion (+ tool overhead) for the model
`max_tokens` / `max_completion_tokens`	Upper bound on generated tokens for this response
Prompt tokens	Everything you already sent: system, history, tools, files

Approximate invariant:

prompt_tokens + completion_tokens ≤ context_window
completion_tokens ≤ max_tokens

If prompt_tokens is 120_000 on a 128k model, the model can emit at most ~8k tokens even if you asked for max_tokens: 16000. Some APIs will error; others silently clamp. Either way you do not get a 16k essay.

Why replies die early

Chat history bloat — You append every turn without summarization. By turn 30 the window is mostly old tool JSON.
RAG over-retrieval — Top-20 chunks × long documents leave no space for the answer.
System prompts with novels — Policy text + few-shot examples eat 4–8k before the user speaks.
Wrong parameter — Setting a high max_tokens while the prompt already filled the window only states an impossible wish.

Measure before you guess

import { encoding_for_model } from "tiktoken"; // or a modern tokenizer for your model

function count(text: string, model: string) {
  const enc = encoding_for_model(model as any);
  try {
    return enc.encode(text).length;
  } finally {
    enc.free();
  }
}

Log usage.prompt_tokens and usage.completion_tokens from the API response. When finish_reason === "length", compare prompt size to model limit. Also use a token counter for rough local estimates before you burn paid calls.

Budgeting pattern that works

const MODEL_LIMIT = 128_000;
const RESERVE_FOR_ANSWER = 2_048;
const SAFETY = 512;

function fitPrompt(parts: string[]): string {
  const budget = MODEL_LIMIT - RESERVE_FOR_ANSWER - SAFETY;
  // keep system + latest user; truncate or drop oldest history / RAG hits
  // ...
  return assembled;
}

const completion = await client.chat.completions.create({
  model: "gpt-4.1",
  messages,
  max_completion_tokens: RESERVE_FOR_ANSWER,
});

Rules of thumb:

Reserve answer space first, then fill the prompt with what fits.
Prefer summarize old turns over sending full history.
Cap RAG characters aggressively; cite pointers instead of pasting whole files.
For long outputs, use continuation (finish_reason === "length" → ask to continue) only after you verify the window actually has room.

API naming drift

Older Chat Completions used max_tokens. Newer models/APIs prefer max_completion_tokens or Responses API equivalents. Reasoning models may spend hidden tokens on chain-of-thought that also compete with visible output — read the model’s docs for whether reasoning counts against the same cap. Treat vendor docs as source of truth; the invariant “prompt + out ≤ window” still guides debugging.

Failure mode cheat sheet

Observation	Likely cause
`finish_reason: length`, short answer	Prompt too large or `max_*` too small
API 400 context length exceeded	Prompt alone past the window
Good first paragraph then cut	Hit `max_tokens` while window still had room — raise completion budget
Quality drops late in long chats	History noise; not a tokenizer bug

Room for the reply is a first-class budget. Allocate it explicitly. Everything else — RAG, memory, tools — competes for what remains.

Tools

OpenAI max_tokens vs Context Window — Room for Reply