Phase 1: from-scratch byte-level BPE tokenizer

- src/tokenizer.py: byte-level BPE with GPT-2-style word pre-tokenization,
  protected <|endoftext|> special token, save/load as inspectable JSON.
  Simple recount-per-merge trainer over unique words.
- tests/test_tokenizer.py: round-trip, special-token integrity, unicode,
  lossless pre-tokenization, save/load, determinism (8 tests, no pytest dep)
- scripts/train_tokenizer.py: train on a corpus sample, report bytes/token
This commit is contained in:
2026-07-12 10:31:16 -04:00
parent e760c00043
commit 2c5fc29592
4 changed files with 359 additions and 0 deletions
+1
View File
@@ -130,6 +130,7 @@ experiment-llm/
├── requirements.txt
├── configs/ # one file per experiment (v1, v2, ...)
├── scripts/ # one-shot entry points: download, encode, plots
├── artifacts/ # trained tokenizer.json (small, committed)
├── src/
│ ├── tokenizer.py # BPE: train / encode / decode
│ ├── dataset.py # memmapped token files, batch sampling