How can the same collection of words
produce different writing?
And what makes an output worth keeping?
Paper first. Then a Toy language model.
Write one word. Then we tally the room.
Which words appeared often? Which appeared once?
A distribution shows how the choices are spread.
Count repeats. Treat the period as a token.
Never count across a line break.
a?Find every occurrence. Keep the repeats.
| Next word | Tally | Count |
|---|---|---|
| moon | ||
| map | ||
| star |
| Current word | Every next word, repeats kept |
|---|---|
| the | robot, robot, robot, fox, fox, robot |
| robot | paints, paints, folds, paints |
| fox | paints, folds |
| paints | a, a, a, a |
| folds | a, a |
| a | moon, star, map, moon, map, moon |
| moon · star · map | . (always) |
Start at the. Read its row.
Write the most common next word. Repeat.
Stop at the period. Record your sentence.
Same book. Same start. Same selection rule.
What if another next word gets a chance?
The computer can count the patterns.
We can investigate how it chooses among them.
Machine Learning for Kids · generating text · Toy
| BookBot | On screen |
|---|---|
| Our book | Corpus: the text we supply |
| Current word | Smallest context setting |
| Always take the most common | Temperature: change how strongly common options are favored |
Open the activity and copy the corpus
Use the smallest context setting.
Keep top-p at the high end.
The Toy model continues this text.
Restore the same start before every run.
Temperature toward low.
Generate 3 times.
Temperature toward high.
Generate 3 times.
Keep corpus, starting text, context and top-p fixed.
Predict first. Screenshot the settings for A and B.
Screenshot every run.
Copy the first 10 generated words from A1 and B1.
Loops and identical outputs are evidence too.
Imagine an artwork: a robot’s nighttime garden.
Choose a phrase you generated.
Keep it, edit it, or replace it as the artwork’s title.
My final title is ______.
I chose / changed ______ because ______.
Keep the original phrase beside your final version.
I changed ______ and kept ______ fixed.
I observed ______.
My evidence is ______.
This does not establish ______.
What did you decide that the model did not?
Filename: Lastname-U1D3
Today: one model, different settings.
Next: the same task, different models.
Bring a task you care about.
Decide what a useful result would look like.