Add degradation experiments: one-parameter-at-a-time ablations
Controlled study of how the model degrades. Four variants, each = v1 with a single field replaced (dataclasses.replace), so differences are attributable: - small: capacity down (2L/64d, ~0.38M params) - short: training time down (max_iters 20k->2k) - ctx: context window down (256->64) - data: data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice) - scripts/compare.py: sample same prompt across all trained configs with a shared seed, print side by side, write reports/compare.md - tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields differ from v1) + small param count (3 tests). Full suite: 29 passing.
This commit is contained in:
@@ -116,6 +116,10 @@ def main() -> None:
|
||||
tokens_dir = ROOT / "data" / "tokens"
|
||||
train_data = load_tokens(tokens_dir / "train.bin")
|
||||
val_data = load_tokens(tokens_dir / "val.bin")
|
||||
# Data-starvation experiments restrict training to a slice; val stays full so
|
||||
# the generalization gap (overfitting) is visible.
|
||||
if tc.max_train_tokens is not None:
|
||||
train_data = train_data[: tc.max_train_tokens]
|
||||
print(f"train tokens: {len(train_data):,} | val tokens: {len(val_data):,}")
|
||||
|
||||
model = GPT(mc).to(device)
|
||||
|
||||
Reference in New Issue
Block a user