ChatGPT Token Counting — API Billing and Context Window Math
Added RAG, pasted the whole wiki into system prompt, enabled JSON mode with a 400-line schema — invoice doubled. r/OpenAI thread says “tokens are scam” while another comment correctly notes you sent 40k tokens per call. Both happen because token math is invisible until billing arrives.
- Words × 1.3 = tokens always — English prose ≈ 0.75 tokens/word; code, JSON, and CJK are different. Measure, don’t guess.
- Only user message counts — System prompt, few-shot examples, tool definitions, and chat history all bill as input.
- Output tokens are cheap so who cares — At scale, long completions (chain-of-thought leaking into output) dominate cost.
- Context window = free space — Filling 128k context still costs input token rate for every token you send.
What counts as input
| Component | Bills as input? |
|---|---|
| System message | Yes |
| Prior turns in chat | Yes (unless truncated server-side) |
| User message | Yes |
| Image tokens (vision) | Yes, tile-based |
| Function/tool schemas | Yes |
| Retrieved RAG chunks | Yes — every request |
Truncation strategies (keep last N turns, summarize old context) exist because unused history still costs money.
Rough English estimates
| Content | Approx tokens |
|---|---|
| 1 word | ~1.3 tokens (inverse: ~0.75 words/token) |
| 1 page (~500 words) | ~650–750 tokens |
| 10k word doc | ~13k–15k tokens |
Use token counter tool or official tiktoken for exact counts on your model.
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
print(len(enc.encode("Your prompt here")))Context window vs billing
Model advertises 128k context — that’s capacity, not price. Sending 100k tokens input at $/1M input rate is the real constraint. Design for:
- Smaller retrieved chunks (top-k, MMR)
- Summarized memory instead of full transcript
- Structured outputs to cap completion length (
max_tokens)
Cost reduction that actually works
- Count before ship — Wire tokenizer into CI for prompt templates.
- Shrink system prompt — Remove duplicate instructions buried in user messages too.
- Cache (where supported) — Prompt caching reduces repeated prefix cost on supported models.
- Cheaper model for routing — Classify intent with small model; escalate only when needed.
- Don’t log full prompts to Slack — Also a security issue.
ChatGPT UI vs API
Plus subscription ≠ API credits. Playground and ChatGPT app token usage doesn’t map 1:1 to API billing rates. Production apps need API dashboard metrics.
Vision token intuition
Images aren’t “one token” — they’re tiled and encoded. A 1920×1080 screenshot can be thousands of tokens. Resize before send if detail isn’t needed.
Streaming and partial output still bill
Streaming completions (stream: true) charge for tokens actually generated, not the max_tokens ceiling. Stopping early saves output cost — but input was already sent and billed in full when the request started. Client-side abort does not refund input tokens.
Model-specific tokenizer drift
gpt-4, gpt-4o, and open-weight models each use different tokenizers. A prompt that fits 8k on one model may overflow another. Never copy max_tokens settings across model migrations without re-measuring — migration posts on r/LocalLLaMA are full of “same prompt, different context usage” surprises.
Batch API vs realtime
OpenAI Batch (and similar queue products) discounts latency for offline jobs. Token counts are identical; only price and SLA differ. If you’re embedding 2M product descriptions overnight, batch + pre-counted chunks beats hammering realtime endpoints and wondering why rate limits fired at 2am.
Tool: count before you call
The token-counter tool estimates tokens for prompts locally — sanity-check RAG payloads before they hit production.
TL;DR
Billed tokens = everything you send + everything the model returns. English ≈ ¾ word per token is a rough guide; code/RAG/system prompts blow estimates. Measure with tiktoken, trim history, size RAG chunks deliberately.