rzen
9fd56b6063
Add degradation experiments: one-parameter-at-a-time ablations
...
Controlled study of how the model degrades. Four variants, each = v1 with a
single field replaced (dataclasses.replace), so differences are attributable:
- small: capacity down (2L/64d, ~0.38M params)
- short: training time down (max_iters 20k->2k)
- ctx: context window down (256->64)
- data: data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice)
- scripts/compare.py: sample same prompt across all trained configs with a
shared seed, print side by side, write reports/compare.md
- tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields
differ from v1) + small param count (3 tests). Full suite: 29 passing.
2026-07-12 12:58:48 -04:00
rzen
8915d274f5
Phase 5: autoregressive sampler (temperature + top-k)
...
- src/sample.py: load checkpoint + tokenizer, generate token-by-token with
temperature/top-k (or greedy), stop at <|endoftext|>, crop to context window.
--suite samples a fixed prompt suite into reports/samples-<config>.md.
- tests/test_sample.py: length, context-window safety, early stop, greedy
determinism, str output (5 tests, tiny untrained model)
2026-07-12 11:34:45 -04:00
rzen
234a37f8fd
Phase 4: training loop (code only — for user to run)
...
- src/train.py: AdamW (decay on matrices only), cosine LR w/ warmup, grad
clipping, periodic train/val loss, CSV logging, resumable checkpoints.
--overfit mode runs the overfit-one-batch sanity check. MPS, fp32.
- configs/v1.py: TrainConfig (batch 64, peak LR 6e-4, 20k iters, etc.)
- tests/test_train.py: LR schedule + optimizer grouping (3 tests, no run)
2026-07-12 10:51:58 -04:00
rzen
e708a5b886
Phase 3: decoder-only transformer (~4.3M params)
...
- src/model.py: GPT written module by module — RMSNorm, explicit causal
self-attention (scores/mask/softmax/weighted-sum), GELU MLP, pre-norm
residual blocks, learned positions, weight-tied head. GPT-2 style init.
- configs/v1.py: frozen ModelConfig dataclass (4L/256d/4h/256ctx/4096vocab)
- tests/test_model.py: exact param count (4,262,144), forward shapes, init
loss ~= ln(vocab), weight tying, causal-masking check (5 tests, forward-only)
- Verified forward pass runs on MPS.
2026-07-12 10:48:55 -04:00
rzen
c202bada6a
Phase 2: token dataset + corpus encoder (code only, not yet run)
...
- src/dataset.py: memmap uint16 token loader + random-crop batch sampler
(x, y shifted-by-one, reproducible via generator)
- scripts/encode_corpus.py: stream stories -> encode -> uint16 .bin, one EOT
token appended per story, flat memory. encode_file() factored out for testing.
- tests/test_dataset.py: batch shapes/shift/bounds/reproducibility +
encoder round-trip/separators/uint16 on synthetic data (5 tests)
2026-07-12 10:38:20 -04:00
rzen
3a489fef0a
Phase 1: trained 4096-vocab tokenizer + per-word encode cache
...
- artifacts/tokenizer.json: 3839 merges, trained on 50MB sample in 87s.
3.97 bytes/token on held-out text.
- encode() caches word->ids so encoding the full corpus is mostly dict
lookups (TinyStories has a small vocabulary).
2026-07-12 10:33:24 -04:00
rzen
2c5fc29592
Phase 1: from-scratch byte-level BPE tokenizer
...
- src/tokenizer.py: byte-level BPE with GPT-2-style word pre-tokenization,
protected <|endoftext|> special token, save/load as inspectable JSON.
Simple recount-per-merge trainer over unique words.
- tests/test_tokenizer.py: round-trip, special-token integrity, unicode,
lossless pre-tokenization, save/load, determinism (8 tests, no pytest dep)
- scripts/train_tokenizer.py: train on a corpus sample, report bytes/token
2026-07-12 10:31:16 -04:00
rzen