Add degradation experiments: one-parameter-at-a-time ablations

Controlled study of how the model degrades. Four variants, each = v1 with a
single field replaced (dataclasses.replace), so differences are attributable:
- small: capacity down (2L/64d, ~0.38M params)
- short: training time down (max_iters 20k->2k)
- ctx:   context window down (256->64)
- data:  data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice)

- scripts/compare.py: sample same prompt across all trained configs with a
  shared seed, print side by side, write reports/compare.md
- tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields
  differ from v1) + small param count (3 tests). Full suite: 29 passing.
This commit is contained in:
2026-07-12 12:58:48 -04:00
parent 84c41bc612
commit 9fd56b6063
9 changed files with 260 additions and 0 deletions
+32
View File
@@ -166,6 +166,38 @@ experiment-llm/
(v1: grammatical, story-arc coherent; plot logic wobbles as expected)
- [x] Every phase re-runnable from a clean clone with documented commands
## Degradation experiments (iteration 1.5)
A controlled study of *how* a model gets worse: start from the v1 baseline and
change exactly **one** variable per run, so any difference in the output is
attributable to that variable. Each variant config is literally `v1` with one
field replaced (see `configs/*.py`); `tests/test_configs.py` enforces that only
the intended field changes.
| config | one change from v1 | isolates | expected failure mode |
|---|---|---|---|
| `v1` | — (baseline) | — | grammatical, coherent story arcs |
| `small` | 2L/64d, ~0.38M params | capacity | generic, repetitive, forgets rare words |
| `short` | `max_iters` 20k→2k | training time | half-learned grammar, weaker structure |
| `ctx` | `context_length` 256→64 | long-range coherence | fine locally, plot threads break sooner |
| `data` | `max_train_tokens`→1M | data quantity | overfits: memorizes, novel prompts collapse |
Run each (they all train faster than v1; `short` is ~minutes):
```sh
python src/train.py --config small # then short, ctx, data
python scripts/plot_loss.py --config small
```
Then compare generated text across every trained variant, side by side (same
prompt, same RNG seed, so differences reflect the model):
```sh
python scripts/compare.py # default prompt, all configs
python scripts/compare.py --suite # every suite prompt -> reports/compare.md
python scripts/compare.py --configs v1,small # a subset
```
## Iteration 2 candidates (out of scope for now)
- Scale toward the coherence sweet spot (~1530M params, longer training)