3 Commits

Include renames

Author	SHA1	Message	Date
rzen	9fd56b6063	Add degradation experiments: one-parameter-at-a-time ablations ... Controlled study of how the model degrades. Four variants, each = v1 with a single field replaced (dataclasses.replace), so differences are attributable: - small: capacity down (2L/64d, ~0.38M params) - short: training time down (max_iters 20k->2k) - ctx: context window down (256->64) - data: data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice) - scripts/compare.py: sample same prompt across all trained configs with a shared seed, print side by side, write reports/compare.md - tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields differ from v1) + small param count (3 tests). Full suite: 29 passing.	2026-07-12 12:58:48 -04:00
rzen	234a37f8fd	Phase 4: training loop (code only — for user to run) ... - src/train.py: AdamW (decay on matrices only), cosine LR w/ warmup, grad clipping, periodic train/val loss, CSV logging, resumable checkpoints. --overfit mode runs the overfit-one-batch sanity check. MPS, fp32. - configs/v1.py: TrainConfig (batch 64, peak LR 6e-4, 20k iters, etc.) - tests/test_train.py: LR schedule + optimizer grouping (3 tests, no run)	2026-07-12 10:51:58 -04:00
rzen	e708a5b886	Phase 3: decoder-only transformer (~4.3M params) ... - src/model.py: GPT written module by module — RMSNorm, explicit causal self-attention (scores/mask/softmax/weighted-sum), GELU MLP, pre-norm residual blocks, learned positions, weight-tied head. GPT-2 style init. - configs/v1.py: frozen ModelConfig dataclass (4L/256d/4h/256ctx/4096vocab) - tests/test_model.py: exact param count (4,262,144), forward shapes, init loss ~= ln(vocab), weight tying, causal-masking check (5 tests, forward-only) - Verified forward pass runs on MPS.	2026-07-12 10:48:55 -04:00