rzen/experiment-llm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5 Commits 1 Branch 0 Tags
3a489fef0a417077270fcb333e191ad21cd270d9
Commit Graph
5 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
rzen 3a489fef0a Phase 1: trained 4096-vocab tokenizer + per-word encode cache 
- artifacts/tokenizer.json: 3839 merges, trained on 50MB sample in 87s.
  3.97 bytes/token on held-out text.
- encode() caches word->ids so encoding the full corpus is mostly dict
  lookups (TinyStories has a small vocabulary).
 2026-07-12 10:33:24 -04:00
rzen 2c5fc29592 Phase 1: from-scratch byte-level BPE tokenizer 
- src/tokenizer.py: byte-level BPE with GPT-2-style word pre-tokenization,
  protected <|endoftext|> special token, save/load as inspectable JSON.
  Simple recount-per-merge trainer over unique words.
- tests/test_tokenizer.py: round-trip, special-token integrity, unicode,
  lossless pre-tokenization, save/load, determinism (8 tests, no pytest dep)
- scripts/train_tokenizer.py: train on a corpus sample, report bytes/token
 2026-07-12 10:31:16 -04:00
rzen e760c00043 Mark Phase 0 environment + dataset item done 2026-07-12 10:26:03 -04:00
rzen 0c26cd1f1f Record canonical TinyStoriesV2 SHA-256 hashes for reproducibility 2026-07-12 10:25:53 -04:00
rzen 5a7f2177bc Phase 0: project plan, pinned environment, data download + MPS smoke test 
- README with full pipeline plan (tokenizer -> data -> model -> training -> eval)
- pip/venv setup pinned to torch 2.13.0, numpy 2.5.1 (Python 3.13)
- scripts/download_data.py: one-time TinyStoriesV2 fetch with SHA-256 recording
- scripts/check_mps.py: verifies MPS backend, backward pass, GPU speedup
 2026-07-12 10:24:52 -04:00