AI Chat Conversation-History Management — Sliding Window & Summarization to Cut Tokens

AI LLM tokens web-development
The short version

Manage AI-chat history with a combination of a recent-message sliding window and a summary text of older turns. Token consumption stays roughly constant, and response speed stays stable.

In a multi-turn AI chat, the number of tokens per request snowballs as the conversation grows. Without mitigation you hit the LLM’s context limit, response speed degrades, and metered API costs jump.

Comparison matrix of history-management techniques

The three representative ways to keep token usage down, with pros and caveats.

TechniqueHow it worksToken savingsContext fidelityDifficulty
Send everything (no mitigation)Resend all messages every request❌ Keeps getting worse (O(N²))○ Fully preserved★☆☆ Low
Sliding windowKeep only the most recent N (e.g. 6)○ Fixed constant (O(1))△ Context beyond the window is lost★★☆ Medium
Context summarizationCompress old turns into a summary with an LLM○ Large reduction (summary only)○ Key context preserved★★★ High

Implementation steps

How to keep the history from ballooning.

  1. Slice the most recent N messages from the received history array.
  2. Compute the approximate token count after slicing and trim if it exceeds the limit.
  3. Insert a summary text into the system prompt’s context area and send.

1. Sliding extraction of recent messages

Extract the latest N turns (e.g. the last 3 exchanges = 6 messages) from the client-sent history.

interface ChatMessage {
  role: "user" | "assistant" | "system";
  content: string;
}

export function truncateHistory(history: ChatMessage[], maxMessages = 6): ChatMessage[] {
  // Keep only the most recent maxMessages
  if (history.length <= maxMessages) {
    return history;
  }
  return history.slice(-maxMessages);
}

2. Token-limit check and trimming

Estimate the character/token count, and when it exceeds the configured limit (e.g. 2000 tokens), drop the oldest messages first.

// Rough estimate — CJK characters run ~1.5–2 tokens each
function estimateTokenCount(text: string): number {
  return Math.ceil(text.length * 1.8);
}

export function fitHistoryWithinTokenLimit(
  history: ChatMessage[],
  maxTokens = 2000
): ChatMessage[] {
  const result: ChatMessage[] = [];
  let currentTokens = 0;

  // Accumulate from the newest message backwards
  for (let i = history.length - 1; i >= 0; i--) {
    const msgTokens = estimateTokenCount(history[i].content);
    if (currentTokens + msgTokens > maxTokens) {
      break; // Stop once the limit is exceeded
    }
    currentTokens += msgTokens;
    result.unshift(history[i]); // Prepend to keep chronological order
  }

  return result;
}

3. Injecting the summary into the system prompt

Once the conversation passes ~10 exchanges, extract the older context into a 1–2 sentence summary and inject it dynamically into the system prompt’s context area.

// Building the payload structure
const systemPrompt = `You are an assistant.
Summary of the conversation so far:
${summaryText || "none"}

Answer based on the context above.`;

const finalPayload = [
  { role: "system", content: systemPrompt },
  ...fitHistoryWithinTokenLimit(truncateHistory(rawHistory)),
];

Failure boundary — real incidents and fixes

Edge cases that commonly break history-trimming logic.

[Sample error log]: Error: 400 Invalid argument: role sequence must alternate between 'user' and 'model'
[Root cause]: Slicing the array with a sliding window left 'assistant' at the start, violating the LLM's role-alternation rule.

First message role mismatch

A naive .slice(-5) (odd count) can produce an array that starts with an assistant message, violating the API’s requirement that turns start with user.

  • Fix: After sliding, if the first element is assistant, drop it so the array always starts with a user message.

Broken pronoun references (“that”, “the thing you mentioned”)

Trimming too aggressively removes the referents of “the thing you mentioned earlier” / “that”, so the AI answers about something completely unrelated.

  • Fix: Always keep at least the most recent 2 exchanges (4 messages) in the sliding region, and hold important proper nouns in the summary area of the system prompt.