LLM Token Counter — Prompt Budget Before Batch Jobs
Someone estimated “about 500 tokens per row,” multiplied by 10,000 rows, and green-lit a batch. The real average was closer to 800, the system prompt added another 1,200 every call, and retries doubled traffic after rate limits. The invoice was not a surprise to the tokenizer — only to the spreadsheet.
Token counting is budgeting. Treat it like capacity planning, not like a vibe.
Why “characters ÷ 4” lies to you
That heuristic is a rough average for English prose on some older models. It fails when:
- Code and JSON appear (punctuation-heavy → more tokens)
- Non-English text appears (many languages use more tokens per character)
- You count characters in your editor but the API counts tokens after template markup
- Multimodal or tool-call wrappers add hidden structure
Use the tokenizer family that matches your model when precision matters (for example, OpenAI’s tiktoken encodings for compatible models). For quick planning, a token counter beats guessing — still verify against the provider’s tokenizer for production estimates.
What must be in the budget
| Component | Counts toward limit? | Notes |
|---|---|---|
| System prompt | Yes | Every request |
| Developer / tool schemas | Yes | Easy to forget |
| User message | Yes | Your per-row payload |
| Conversation history | Yes | Chat endpoints |
Output max_tokens | Reserved | Leaves less room for input |
People forget the system prompt because it lives in code, not in the CSV. A 10k batch with a 1k-token system message pays that tax 10k times unless you redesign the job (cache, batch APIs, shorter instructions).
A preflight ritual for batch jobs
- Sample 50–100 real documents, including outliers (legal PDFs, giant stack traces, empty bodies).
- Measure tokens for system + tools + each sample.
- Take p95, not the mean — mean hides the row that blows the context window.
- Add margin (10–20%) for template drift and provider-side wrappers.
- Multiply by row count and retry assumptions.
- Set hard caps — truncate or summarize inputs that exceed a threshold instead of failing mid-run.
cost ≈ rows × (input_tokens_p95 + expected_output_tokens) × price_per_token
× (1 + retry_rate)If you chunk documents, count all chunks, not just the parent row.
Designing prompts that stay affordable
- Put stable instructions in a short system prompt; put variable data in the user message once.
- Prefer structured extraction (“return JSON with keys X”) over essay answers when you only need fields.
- Strip HTML to text before sending pages.
- Deduplicate repeated boilerplate (identical headers across rows).
- Log token usage from the API response (
usagefields) and reconcile weekly against finance.
Context windows vs output budgets
Hitting the model’s maximum context is a different failure mode from “we spent too much.” A 128k context does not mean you should stuff 120k tokens of HTML into every call. Latency and cost climb nonlinearly for some providers, and giant inputs often dilute instruction following.
Split the budget deliberately: reserve a fixed slice for instructions and tools, a slice for retrieved context, and a hard ceiling for generated output. If retrieval returns 40 chunks, count them before the request — do not assume the vector DB “probably fits.” When a single document exceeds the input budget, summarize or map-reduce instead of silently truncating the middle of a contract.
FAQ
Does the system prompt count?
Yes. Always.
Is the counter on the provider dashboard identical to local tiktoken?
Usually close for supported models; small differences appear with special tokens and chat templates. Budget with margin.
Can I bill customers using chars÷4?
Not if you care about accuracy. Meter with real tokenizer counts.
Estimate per row from worst realistic samples, include system and tools, and only then multiply. Batch LLM jobs are where optimistic token math goes to become a line item.