Use this script for a 60–90 minute pilot focused on the sequence token → vector → next-token probabilities → human feedback, and the difference between fluent output and understanding.
Core question. When a language model writes, what is it actually doing?
Materials
- Tokenizer + Temperature Visualizer
- Contextual Vector Lab for one predict → reveal contextual-neighborhood comparison
- Whose Preference? Lab for an individual panel-composition and aggregation comparison
- ELIZA Simulator
- Text Experiment Board
- Optional: A/B/C Comparison Board for pilot evidence
- AI Use + Consent Checklist
- Next-Token Prediction Game (paste the Zoom chat block of guesses)
- Optional: Embedding Projector as a real-project reference for static embedding neighborhoods
- Optional: 3–5 pre-written sentence stems for the prediction game
Run of show
| Time | Segment | Facilitator move | Participant action |
|---|---|---|---|
| 0–5 | Welcome & norms | Name the question. Remind everyone direct AI use is optional. | Choose a participation pathway. |
| 5–15 | Unplugged prediction | Show a sentence stem; collect guesses in Zoom chat, then paste the chat block into the Next-Token Prediction Game to tally the room beside the model's top-k. | Predict next words; compare the room's distribution with the model's and notice how context shapes guesses. |
| 15–35 | Tokens, vectors + Temperature | Show token chunks, then use one Contextual Vector Lab predict → reveal cycle before comparing probability distributions at low and high temperature. | Separate token ID, starting vector, and contextual representation; ask what “nearby” means; then compare greedy and sampled output. |
| 35–55 | ELIZA vs. frozen comparison text | Inspect matched rule, pattern, and response template. | Compare visible rule-matching with frozen LLM-style text. Name its incomplete provenance; do not attribute it to a particular model. |
| 55–75 | Investigation | Assign one Text Experiment Board section. Offer the Whose Preference? Lab for people investigating how rater-panel and aggregation choices become an authored preference signal. | Run or analyse a default test, prompt variation, ELIZA comparison, or preference-panel change. |
| 75–90 | Debrief | Use the frame: human, machine, system, ethics, pedagogy. Ask whose definition of “better” entered the signal. | Share one evidence-based claim and one question. |
Optional pilot-evidence move: after the investigation, ask one group to enter its baseline, temperature change, and greedy/sample comparison in the A/B/C Comparison Board, then export Markdown for facilitator notes.
Facilitator prompts
- "What made your next-word guess feel obvious?"
- "What does the tokenizer split that you would not have split as a human reader?"
- "Are two nearby words close because they mean the same thing, occur in similar contexts, or reflect a learned cultural association?"
- "At low temperature, what becomes more predictable?"
- "At high temperature, what becomes more surprising or unstable?"
- "What does ELIZA make visible that a modern LLM hides?"
- "Whose definition of ‘better’ entered the preference signal, and whose was excluded?"
- "Where could fluency be mistaken for understanding?"
Investigation prompt
Run the same starting phrase at low and high temperature. Then compare greedy decoding with sampling. What changed in the output, and what did not change about the mechanism?
Low-AI / No-AI pathway
Participants can complete the session using only the visualizer, ELIZA, and frozen comparison examples. They do not need to log into or prompt a live LLM. The comparison text has incomplete model provenance and should not support claims about a named model.
Fallback plan
- If the Tokenizer tool feels too dense, use only the example buttons and temperature slider.
- If the Contextual Vector Lab stalls, keep the illustrative vector on the deck and run the neighborhood prediction verbally; name any proposed position as an authored example, not a model result.
- If projection space is limited, focus on the probability chart and generated text stream.
- If ELIZA examples feel sensitive, use low-stakes prompts about school, projects, or planning rather than emotional disclosures.
- If the preference lab stalls, rank the three apology responses aloud and compare what changes when a learner, teacher, editor, accessibility advocate, or policy team gets the deciding vote. Name this as an authored role-play, not real rater data.
- If a participant does not want to use AI, ask them to design a classroom version of the prediction game.
Pilot QA notes
During the pilot, note:
- Which controls participants found without explanation.
- Whether "temperature" became understandable through the tool.
- Whether participants distinguished a fixed token ID and starting vector from a context-dependent representation without treating the 2D map as literal model space.
- Whether ELIZA's rule inspector was legible on screen.
- Any moment where participants confused probability with truth.
- Any ethical discomfort, especially around chatbot intimacy or trust.