rzen 2c5fc29592 Phase 1: from-scratch byte-level BPE tokenizer ...

- src/tokenizer.py: byte-level BPE with GPT-2-style word pre-tokenization,
  protected <|endoftext|> special token, save/load as inspectable JSON.
  Simple recount-per-merge trainer over unique words.
- tests/test_tokenizer.py: round-trip, special-token integrity, unicode,
  lossless pre-tokenization, save/load, determinism (8 tests, no pytest dep)
- scripts/train_tokenizer.py: train on a corpus sample, report bytes/token

2026-07-12 10:31:16 -04:00

..

test_tokenizer.py

Phase 1: from-scratch byte-level BPE tokenizer

2026-07-12 10:31:16 -04:00