Add degradation experiments: one-parameter-at-a-time ablations
Controlled study of how the model degrades. Four variants, each = v1 with a single field replaced (dataclasses.replace), so differences are attributable: - small: capacity down (2L/64d, ~0.38M params) - short: training time down (max_iters 20k->2k) - ctx: context window down (256->64) - data: data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice) - scripts/compare.py: sample same prompt across all trained configs with a shared seed, print side by side, write reports/compare.md - tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields differ from v1) + small param count (3 tests). Full suite: 29 passing.
This commit is contained in:
@@ -166,6 +166,38 @@ experiment-llm/
|
||||
(v1: grammatical, story-arc coherent; plot logic wobbles as expected)
|
||||
- [x] Every phase re-runnable from a clean clone with documented commands
|
||||
|
||||
## Degradation experiments (iteration 1.5)
|
||||
|
||||
A controlled study of *how* a model gets worse: start from the v1 baseline and
|
||||
change exactly **one** variable per run, so any difference in the output is
|
||||
attributable to that variable. Each variant config is literally `v1` with one
|
||||
field replaced (see `configs/*.py`); `tests/test_configs.py` enforces that only
|
||||
the intended field changes.
|
||||
|
||||
| config | one change from v1 | isolates | expected failure mode |
|
||||
|---|---|---|---|
|
||||
| `v1` | — (baseline) | — | grammatical, coherent story arcs |
|
||||
| `small` | 2L/64d, ~0.38M params | capacity | generic, repetitive, forgets rare words |
|
||||
| `short` | `max_iters` 20k→2k | training time | half-learned grammar, weaker structure |
|
||||
| `ctx` | `context_length` 256→64 | long-range coherence | fine locally, plot threads break sooner |
|
||||
| `data` | `max_train_tokens`→1M | data quantity | overfits: memorizes, novel prompts collapse |
|
||||
|
||||
Run each (they all train faster than v1; `short` is ~minutes):
|
||||
|
||||
```sh
|
||||
python src/train.py --config small # then short, ctx, data
|
||||
python scripts/plot_loss.py --config small
|
||||
```
|
||||
|
||||
Then compare generated text across every trained variant, side by side (same
|
||||
prompt, same RNG seed, so differences reflect the model):
|
||||
|
||||
```sh
|
||||
python scripts/compare.py # default prompt, all configs
|
||||
python scripts/compare.py --suite # every suite prompt -> reports/compare.md
|
||||
python scripts/compare.py --configs v1,small # a subset
|
||||
```
|
||||
|
||||
## Iteration 2 candidates (out of scope for now)
|
||||
|
||||
- Scale toward the coherence sweet spot (~15–30M params, longer training)
|
||||
|
||||
Reference in New Issue
Block a user