LLM Tokenizer Visualizer
Inspect real BPE token IDs, colored byte slices and exact roundtrip for two encodings locally.
Enabling local controls…
Results
Review the inputs, then run to create a readable result and complete export. Editing cancels active work and clears stale results.
Formula, sources and limits
Exact count for the selected encoding only. Chat wrappers, tools, images and provider-specific tokens are excluded. A token can contain partial UTF-8; the visualizer groups those tokens for legible text and retains every ID. Rates, limits, subscriptions and latency defaults are illustrative custom assumptions, not current vendor quotes or benchmarks. Check official sources and replace them. Each text field ≤32,768 UTF-16 units, total input ≤128 KiB, output ≤1 MiB; local job timeout 30s.
Methodology checked:
How to use this tool
Load the safe sample, review every labeled assumption, then run. Edit any field to clear old results. Copy or download the complete result. Processing stays in this tab.
When not to use it
Exact count for the selected encoding only. Chat wrappers, tools, images and provider-specific tokens are excluded. A token can contain partial UTF-8; the visualizer groups those tokens for legible text and retains every ID. Rates, limits, subscriptions and latency defaults are illustrative custom assumptions, not current vendor quotes or benchmarks. Check official sources and replace them. Each text field ≤32,768 UTF-16 units, total input ≤128 KiB, output ≤1 MiB; local job timeout 30s.