- reports/degradation-analysis.md: interpretation of the one-parameter-at-a-time
ablations (ctx/short/small/data vs v1), grounded in val loss + sample text.
Key findings: held-out loss tracks quality for generalizing models; different
degradations give qualitatively different failure text; data-starvation
overfits (train ppl 1.1 / val ppl 322) with samples that hide the damage.
- reports/compare.md: side-by-side samples across all configs
- reports/loss-{small,short,ctx,data}.csv: variant training curves
Controlled study of how the model degrades. Four variants, each = v1 with a
single field replaced (dataclasses.replace), so differences are attributable:
- small: capacity down (2L/64d, ~0.38M params)
- short: training time down (max_iters 20k->2k)
- ctx: context window down (256->64)
- data: data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice)
- scripts/compare.py: sample same prompt across all trained configs with a
shared seed, print side by side, write reports/compare.md
- tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields
differ from v1) + small param count (3 tests). Full suite: 29 passing.
- reports/loss-v1.csv: full train/val loss log over 20k steps
- reports/loss-v1.txt: ASCII loss curve (steep descent then flattening tail)
Training ran ~58 min on M1 MPS at 5.7 it/s. No overfitting (train~=val).
- scripts/plot_loss.py: reads reports/loss-<config>.csv, renders train/val
loss as a terminal ASCII chart, saves it to reports/loss-<config>.txt.
Verified on a synthetic curve. Keeps deps at torch + numpy.
- README: documented plotting choice and the Phase 4 run commands.
- artifacts/tokenizer.json: 3839 merges, trained on 50MB sample in 87s.
3.97 bytes/token on held-out text.
- encode() caches word->ids so encoding the full corpus is mostly dict
lookups (TinyStories has a small vocabulary).
rzen