Add degradation experiments: one-parameter-at-a-time ablations

Controlled study of how the model degrades. Four variants, each = v1 with a
single field replaced (dataclasses.replace), so differences are attributable:
- small: capacity down (2L/64d, ~0.38M params)
- short: training time down (max_iters 20k->2k)
- ctx:   context window down (256->64)
- data:  data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice)

- scripts/compare.py: sample same prompt across all trained configs with a
  shared seed, print side by side, write reports/compare.md
- tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields
  differ from v1) + small param count (3 tests). Full suite: 29 passing.
This commit is contained in:
2026-07-12 12:58:48 -04:00
parent 84c41bc612
commit 9fd56b6063
9 changed files with 260 additions and 0 deletions
+4
View File
@@ -116,6 +116,10 @@ def main() -> None:
tokens_dir = ROOT / "data" / "tokens"
train_data = load_tokens(tokens_dir / "train.bin")
val_data = load_tokens(tokens_dir / "val.bin")
# Data-starvation experiments restrict training to a slice; val stays full so
# the generalization gap (overfitting) is visible.
if tc.max_train_tokens is not None:
train_data = train_data[: tc.max_train_tokens]
print(f"train tokens: {len(train_data):,} | val tokens: {len(val_data):,}")
model = GPT(mc).to(device)