visualizer@text:~$ run tokenizer-temperature --inspect
loading token boundaries ............ ok
loading next-token probabilities .... ok
loading temperature sampler ......... ok
The core text tool. It pulls apart three things that usually hide inside a single fluent sentence: how text becomes tokens, how the next token is predicted, and how temperature reshapes what gets sampled.
Type any text and watch it split into the chunks a model counts — frequently mid-word, and rarely where you'd expect.
For each step, see the ranked candidates and their probabilities — the guess underneath every "sentence."
Slide from greedy to high temperature and watch the same prompt move from predictable to surprising to unstable.
Write down what you think high temperature will do to this exact prompt.
Keep the prompt identical. Run it at 0.2, then 0.9. Change nothing else.
What changed in the output? What stayed the same about the mechanism underneath?
Name the behaviour precisely: a default, a failure, or a pattern — then decide what a human does next.