Home/ Tool index/ Tokenizer + Temperature
Launch ready Session 1 · Text Interactive

Tokenizer + Temperature Visualizer

The core text tool. It pulls apart three things that usually hide inside a single fluent sentence: how text becomes tokens, how the next token is predicted, and how temperature changes what gets sampled.

tools/tokenizer-temperature-visualizer/

Live preview · launch for the interactive version

§ A · What it makes visible

three hidden mechanisms
Fig. 01

Token boundaries

Type any text and watch it split into the chunks a model counts — frequently mid-word, and rarely where you'd expect.

Fig. 02

Next-token probability

For each step, see the ranked candidates and their probabilities — the guess underneath every "sentence."

Fig. 03

Temperature & sampling

Slide from greedy to high temperature and watch the same prompt move from predictable to surprising to unstable.

§ B · How to investigate it

run it like an experiment, not a toy
01 · Predict

Before you touch the slider

Write down what you think high temperature will do to this exact prompt.

prompt: "The museum was completely ___"
02 · Change one thing

Move only the temperature

Keep the prompt identical. Run it at 0.2, then 0.9. Change nothing else.

temp 0.2 → temp 0.9 · same prompt
03 · Compare evidence

Greedy vs. sampled

What changed in the output? What stayed the same about the mechanism underneath?

"empty" 71% → "empty / silent / flooded / on fire"
04 · Name it

Not "it got weird"

Name the behaviour precisely: a default, a failure, or a pattern — then decide what a human does next.

default · the model's safest continuation

§ C · Debrief questions

after the investigation
What does the tokenizer split that you wouldn't split as a reader?
At low temperature, what becomes more predictable?
Where did probability get confused with truth?
When did fluency start to feel like understanding?

§ D · Related

pairs well with · use in context