 rzen
|
234a37f8fd
|
Phase 4: training loop (code only — for user to run)
- src/train.py: AdamW (decay on matrices only), cosine LR w/ warmup, grad
clipping, periodic train/val loss, CSV logging, resumable checkpoints.
--overfit mode runs the overfit-one-batch sanity check. MPS, fp32.
- configs/v1.py: TrainConfig (batch 64, peak LR 6e-4, 20k iters, etc.)
- tests/test_train.py: LR schedule + optimizer grouping (3 tests, no run)
|
2026-07-12 10:51:58 -04:00 |
|
|
rzen
|
e708a5b886
|
Phase 3: decoder-only transformer (~4.3M params)
- src/model.py: GPT written module by module — RMSNorm, explicit causal
self-attention (scores/mask/softmax/weighted-sum), GELU MLP, pre-norm
residual blocks, learned positions, weight-tied head. GPT-2 style init.
- configs/v1.py: frozen ModelConfig dataclass (4L/256d/4h/256ctx/4096vocab)
- tests/test_model.py: exact param count (4,262,144), forward shapes, init
loss ~= ln(vocab), weight tying, causal-masking check (5 tests, forward-only)
- Verified forward pass runs on MPS.
|
2026-07-12 10:48:55 -04:00 |
|