An interactive app that walks through every stage of the LLM pipeline — from pattern matching to training a transformer from scratch — with working code you can run locally.
Requires pnpm and Node.js 20+.
pnpm install
pnpm devIf pnpm reports ignored build scripts, run:
pnpm approve-buildsSelect esbuild with Space, press Enter, then rerun pnpm install.
The simplest possible "AI": an ELIZA-style chatbot that responds with if-statements, streamed word by word via SSE. Same plumbing as ChatGPT, zero intelligence.
Trains a neural network live. A single-layer perceptron fails on XOR (proving Minsky/Papert right). A multi-layer network succeeds via backpropagation — the same algorithm every neural network uses today.
- Route:
src/routes/neural-net/ - Hook:
src/client/hooks/use-neural-net-chat.tsx - Component:
src/client/components/neural-net-result/
A from-scratch BPE implementation that trains on your input text. Watch merge steps animate as the algorithm builds a vocabulary from characters to words.
- Route:
src/routes/bpe-tokenize.ts - Hook:
src/client/hooks/use-bpe-tokenize-chat.tsx - Component:
src/client/components/bpe-tokenize-result/
Trains word embeddings from scratch using Word2Vec skip-gram with negative sampling. Watch vectors learn that words used in similar contexts should cluster together.
- Route:
src/routes/train-embed/ - Hook:
src/client/hooks/use-train-embed-chat.tsx - Component:
src/client/components/train-embed-result/
Trains a decoder-only transformer entirely from scratch — no ML libraries. Every operation is implemented by hand: multi-head causal self-attention, layer normalization, feed-forward layers, backpropagation, and Adam optimization. Uses multi-threaded data parallelism via SharedArrayBuffer for training speed.
- Route:
src/routes/train-transformer/ - Hook:
src/client/hooks/use-train-transformer-chat.tsx - Component:
src/client/components/train-transformer-result/
Every section follows the same pattern:
- Server route (
src/routes/) — Hono POST handler that processes input and streams SSE events - Client hook (
src/client/hooks/) — manages state and connects SSE events to UI updates viauseSSEChat - Result component (
src/client/components/) — renders the streamed data as a visualization
Core infrastructure:
- SSE streaming:
src/server/lib/sse.ts(server) /src/client/lib/sse.ts(client) - BPE tokenizer:
src/server/lib/bpe.ts(shared by Basic Tokenizer, Train Embeddings, and Train Transformer) - Generic chat hook:
src/client/hooks/use-sse-chat.ts - Message types:
src/shared/types/message.ts
Papers referenced in the codebase, in the order the concepts appear across the demos:
-
Weizenbaum (1966) — "ELIZA — A Computer Program for the Study of Natural Language Communication Between Man and Machine" dl.acm.org/doi/10.1145/365153.365168 Section 1 — the original pattern-matching chatbot that inspired the Simple Chat demo.
-
Rumelhart, Hinton & Williams (1986) — "Learning Representations by Back-Propagating Errors" nature.com/articles/323533a0 Section 2 — the backpropagation algorithm that made multi-layer neural networks trainable.
-
Glorot & Bengio (2010) — "Understanding the Difficulty of Training Deep Feedforward Neural Networks" proceedings.mlr.press/v9/glorot10a.html Section 5 — Xavier/Glorot initialization, used to set initial transformer weights.
-
Mikolov et al. (2013a) — "Efficient Estimation of Word Representations in Vector Space" arxiv.org/abs/1301.3781 Section 4 — introduces Word2Vec and the Skip-gram architecture used in the embedding trainer.
-
Mikolov et al. (2013b) — "Distributed Representations of Words and Phrases and their Compositionality" arxiv.org/abs/1310.4546 Section 4 — introduces negative sampling, the training trick that makes Skip-gram practical.
-
Kingma & Ba (2014) — "Adam: A Method for Stochastic Optimization" arxiv.org/abs/1412.6980 Section 5 — the Adam optimizer used to train the transformer.
-
Sennrich, Haddow & Birch (2016) — "Neural Machine Translation of Rare Words with Subword Units" arxiv.org/abs/1508.07909 Sections 3, 4, 5 — Byte Pair Encoding (BPE), the tokenization algorithm used throughout.
-
Ba, Kiros & Hinton (2016) — "Layer Normalization" arxiv.org/abs/1607.06450 Section 5 — layer normalization, applied before attention and feed-forward layers in the transformer.
-
Vaswani et al. (2017) — "Attention Is All You Need" arxiv.org/abs/1706.03762 Section 5 — the transformer architecture implemented from scratch.
-
Radford et al. (2018) — "Improving Language Understanding by Generative Pre-Training" cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf Section 5 — GPT-1, the decoder-only transformer pre-training approach this demo follows.
-
Holtzman et al. (2019) — "The Curious Case of Neural Text Degeneration" arxiv.org/abs/1904.09751 Section 5 — nucleus (top-p) sampling, used for text generation after training.