Phase 1: from-scratch byte-level BPE tokenizer
- src/tokenizer.py: byte-level BPE with GPT-2-style word pre-tokenization, protected <|endoftext|> special token, save/load as inspectable JSON. Simple recount-per-merge trainer over unique words. - tests/test_tokenizer.py: round-trip, special-token integrity, unicode, lossless pre-tokenization, save/load, determinism (8 tests, no pytest dep) - scripts/train_tokenizer.py: train on a corpus sample, report bytes/token
This commit is contained in:
@@ -130,6 +130,7 @@ experiment-llm/
|
||||
├── requirements.txt
|
||||
├── configs/ # one file per experiment (v1, v2, ...)
|
||||
├── scripts/ # one-shot entry points: download, encode, plots
|
||||
├── artifacts/ # trained tokenizer.json (small, committed)
|
||||
├── src/
|
||||
│ ├── tokenizer.py # BPE: train / encode / decode
|
||||
│ ├── dataset.py # memmapped token files, batch sampling
|
||||
|
||||
Reference in New Issue
Block a user