Add degradation experiments: one-parameter-at-a-time ablations

Controlled study of how the model degrades. Four variants, each = v1 with a
single field replaced (dataclasses.replace), so differences are attributable:
- small: capacity down (2L/64d, ~0.38M params)
- short: training time down (max_iters 20k->2k)
- ctx:   context window down (256->64)
- data:  data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice)

- scripts/compare.py: sample same prompt across all trained configs with a
  shared seed, print side by side, write reports/compare.md
- tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields
  differ from v1) + small param count (3 tests). Full suite: 29 passing.
This commit is contained in:
2026-07-12 12:58:48 -04:00
parent 84c41bc612
commit 9fd56b6063
9 changed files with 260 additions and 0 deletions
+1
View File
@@ -25,6 +25,7 @@ class ModelConfig:
class TrainConfig:
# Data / batching. tokens per step = batch_size * context_length = 16,384.
batch_size: int = 64
max_train_tokens: int | None = None # None = whole train set; set to starve data
# AdamW optimiser.
learning_rate: float = 6e-4 # peak LR (reached after warmup)