How machines imagine

Watch the recording

Open the Session 2 recording on YouTube

Start here

Session overview

The core question was: when an image model generates a picture, what is it actually deciding? We moved from tokens to pixels — an image is a two-dimensional grid of pixel data rather than a one-directional token stream, which is one reason image generation costs so much more compute than text.

The tour ran in order: the human pre-work underneath everything (WordNet's map of English words, ImageNet's millions of human-labeled images), then diffusion — generation as machine-learning-aided denoising, chiseling an image out of a field of gray noise the way a sculptor works. During training, known noise is added to examples and the network learns to predict and remove it. From there: how the whole prompt conditions each denoising step, why the same prompt can produce many plausible images, why uncontrolled edits often redraw more than intended, and where defaults come from when a prompt leaves details unspecified.

Who was in the room

Around fifteen people contributed in chat and discussion. Participants are identified here by first name or chosen display name only:

Session 1, revisited

We opened with what landed and what's still fuzzy from the text session. The lingering questions were sharp enough to shape the rest of the camp:

• Ryan wanted more on how a model processes input, not just how it generates output — how does it determine what the user intends?
• Cha asked how the training corpus is compiled, and what the smallest useful LLM would even look like.
• Randy asked what actually drives model improvement — better data, better algorithms, or something else?
• Shelly brought the classic doctor riddle: swap the genders in the riddle and models still give the statistically popular answer rather than the logical one. The most probable answer is not the reasoned answer — which set up this session's bias discussion perfectly.
• Meghan went down a rabbit hole on tokenization across languages and dialects — if a model is trained in Spanish, how much better is it at Italian than English? She's building this into a new class on AI and Social Impact, and shared a paper on token alignment across languages.
• Shane showed his LLM-built dev kit for microcontrollers (ESP32-P4/S3) that treats actuators and LEDs as first-class objects — built entirely through conversation, no hand-written code, inspired by Digimon-era handheld toys.

The Session 1 survey (five participant responses) matched the discussion: temperature, attention, and alignment clicked; "how it all comes together" is the piece people still want. Text conditioning is central to text-to-image systems, so many of these questions carry forward.

Pixels & the human label layer

An image model never sees a cat, a tree, or a hallway in the human sense. It processes numerical representations of pixels or latent features. We walked a blurry image up through increasing detail and asked the human version of the machine's question: at what point do features snap into place? Earlier work in classification, edge detection, and recognizing text inside images helped establish foundations that later generative systems build on.

Underneath modern image systems sits decades of human pre-work. WordNet (Princeton, 1980s) mapped English words and their relationships; ImageNet (Stanford) organized millions of human-labeled images with that hierarchy. Their human-designed categories and labels helped shape major lines of computer vision; modern generative systems also draw on large-scale captions and, in some cases, later preference ratings.

Diffusion: chiseling from noise

Generation starts as a field of noise, and the model denoises it under prompt conditioning — like the sculptor's line about the statue already being in the marble, you chisel away noise until an image is left. During training, known amounts of noise are added to examples and the network learns to predict that noise, supplying the denoising step used during generation.

Image-processing techniques such as denoising and anti-aliasing are old; learned versions now use the same broad GPU ecosystem that accelerated modern ML. Shane asked whether classic dithering techniques from games are folded into ML denoising. The room treated that as an open question and pointed to DLSS as a nearby example of learned reconstruction. Gaming graphics and ML both benefited from GPUs, while cryptocurrency mining drove additional demand for the hardware; they are not the same computation.

The Sudoku frame: a teaching analogy

Ryan asked whether each clause in a prompt adds a new layer of diffusion. Adding words does not add layers or new embedding dimensions; it changes the prompt representation that conditions the denoising process. The Sudoku frame is useful because the prompt's conditions interact and must cohere, but the analogy has a limit: unlike a completed Sudoku, a prompt has no unique solution. Repeated generations can produce many different images that all plausibly satisfy the words.

That helps explain why editing can be hard without making it universal. A base generated output is not a conventional editable scene graph, so plain re-prompting may redraw more than the requested detail. Masks, cross-attention controls, locked noise, and structured workflows can preserve or localize parts of an edit. Shane connected the broader consistency problem to why much of Hollywood cooled on image generation and to workflows that add control back: ComfyUI's model chains and Corridor Crew locking the initial noise to get consistent frames (Ryan shared their video). Repeated generation — especially for video — still raises a real compute and sustainability question.

The doctor default

Prompt an image model with "a doctor" and nothing else, and the output still supplies gender, age, race, setting, camera angle, and language on the wall. Learned training patterns are one influence; tuning, filtering, interface defaults, prompt rewriting, and sampling may also shape what appears. We ran it live in ChatGPT and read the visible choices off the result together.

A follow-up experiment failed in an instructive way: Shane asked for a half-diffused version of the image, and the model just rendered a doctor behind blurry glass. The words in a prompt are treated as content to depict, not as controls over the generation process — you'd need access to the model itself to stop diffusion halfway.

Human Diffusion Canvas: being the model

The centerpiece: participants opened the Human Diffusion Canvas and performed a simplified image-model role. The prompt arrived in pieces — pale figure → emerging from shadows → a slow scratching sound → echoed through the hallway — and each phrase meant committing a pass of drawing, coarse to fine, like steps of denoising. (Full prompt: "A pale figure emerged from the flickering shadows as a slow scratching sound echoed through the hallway.")

The results across participants were strikingly similar: a figure centered in a receding hallway, creepy horror palette. When we generated the same prompt in ChatGPT, its composition matched the room's — hallway perspective, small figure in the distance. The convergence revealed culturally shared visual conventions, not one correct image; the defaults live in shared culture as much as in training data. Judy thought the hallway image resembled imagery from the French film Diabolique.

Two honest caveats we named: revealing the prompt piecemeal mimics thinking word-by-word, but a real model receives all the words at once; and the room's convergence partly reflects our shared genre knowledge — which is exactly the point about where defaults come from. A "slow scratching sound" also forced the question of what you draw for words with no visual referent.

Tool friction, logged: the brush color wouldn't change between commit steps for participants, and the brush size could be manually overridden (defeating the big-to-small progression). Fixes are on the host's list below. Exporting PNG/GIF worked, and the GIF replays of everyone's drawing process were a highlight.

Guest spotlight

Teaching notes

Action items

• Participants: make one Session 2 artifact — a tool, webpage, activity, or set of questions about image generation that would be useful for your students or your own learning — and share it before Session 3 (six routes on the assignment page)
• Participants: try the session tools and report back — what works, what doesn't, what's missing
• Participants: the GitHub repo is open — feed a tool into an LLM and generate your own customized version (yes, pull requests welcome)
• Host: fix the Human Diffusion Canvas — brush color change between commits, lock the brush-size progression
• Host: build in-person classroom versions of the Zoom-specific activities
• Host (long-term): the digital workbook goal — interactive tools that build ML intuition the way physics and biology sims do

Links & resources from chat

This summary was drafted and revised with AI assistance. Quotations and attributions were checked against the session transcript and chat export; final facilitator review is recommended.