rzen c202bada6a Phase 2: token dataset + corpus encoder (code only, not yet run) ...

- src/dataset.py: memmap uint16 token loader + random-crop batch sampler
  (x, y shifted-by-one, reproducible via generator)
- scripts/encode_corpus.py: stream stories -> encode -> uint16 .bin, one EOT
  token appended per story, flat memory. encode_file() factored out for testing.
- tests/test_dataset.py: batch shapes/shift/bounds/reproducibility +
  encoder round-trip/separators/uint16 on synthetic data (5 tests)

2026-07-12 10:38:20 -04:00

..

check_mps.py

Phase 0: project plan, pinned environment, data download + MPS smoke test

2026-07-12 10:24:52 -04:00

download_data.py

Phase 0: project plan, pinned environment, data download + MPS smoke test

2026-07-12 10:24:52 -04:00

encode_corpus.py

Phase 2: token dataset + corpus encoder (code only, not yet run)

2026-07-12 10:38:20 -04:00

train_tokenizer.py

Phase 1: from-scratch byte-level BPE tokenizer

2026-07-12 10:31:16 -04:00