Add degradation experiments: one-parameter-at-a-time ablations
Controlled study of how the model degrades. Four variants, each = v1 with a single field replaced (dataclasses.replace), so differences are attributable: - small: capacity down (2L/64d, ~0.38M params) - short: training time down (max_iters 20k->2k) - ctx: context window down (256->64) - data: data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice) - scripts/compare.py: sample same prompt across all trained configs with a shared seed, print side by side, write reports/compare.md - tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields differ from v1) + small param count (3 tests). Full suite: 29 passing.
This commit is contained in:
@@ -0,0 +1,17 @@
|
||||
"""Degradation experiment: same as v1, but trained on a tiny slice of data.
|
||||
|
||||
Isolates *data quantity*. Only max_train_tokens changes from v1: the model sees
|
||||
~1M training tokens instead of 555M. Expect overfitting — train loss falls while
|
||||
val loss (measured on the full held-out set) climbs, and novel prompts collapse
|
||||
as the model regurgitates memorized stories.
|
||||
|
||||
Tip: this overfits within a few thousand steps. Watch the val loss turn upward
|
||||
in the log and stop early (Ctrl-C) rather than running all 20,000 steps.
|
||||
"""
|
||||
|
||||
from dataclasses import replace
|
||||
|
||||
import v1
|
||||
|
||||
model = v1.model
|
||||
train = replace(v1.train, max_train_tokens=1_000_000)
|
||||
Reference in New Issue
Block a user