Add degradation experiments: one-parameter-at-a-time ablations
Controlled study of how the model degrades. Four variants, each = v1 with a single field replaced (dataclasses.replace), so differences are attributable: - small: capacity down (2L/64d, ~0.38M params) - short: training time down (max_iters 20k->2k) - ctx: context window down (256->64) - data: data quantity down (max_train_tokens->1M; new TrainConfig knob + train.py slice) - scripts/compare.py: sample same prompt across all trained configs with a shared seed, print side by side, write reports/compare.md - tests/test_configs.py: enforces one-parameter-at-a-time (only intended fields differ from v1) + small param count (3 tests). Full suite: 29 passing.
This commit is contained in:
@@ -25,6 +25,7 @@ class ModelConfig:
|
||||
class TrainConfig:
|
||||
# Data / batching. tokens per step = batch_size * context_length = 16,384.
|
||||
batch_size: int = 64
|
||||
max_train_tokens: int | None = None # None = whole train set; set to starve data
|
||||
|
||||
# AdamW optimiser.
|
||||
learning_rate: float = 6e-4 # peak LR (reached after warmup)
|
||||
|
||||
Reference in New Issue
Block a user