moshferatu 0d79fa2009 Add initial implementation of tokenizer which merges tokens according to the byte pair encoding algorithm		2024-02-24 07:34:27 -08:00
.gitignore	Add .gitignore	2024-02-24 06:54:59 -08:00
text.py	Add example text to tokenize	2024-02-24 07:33:58 -08:00
tokenizer.py	Add initial implementation of tokenizer which merges tokens according to the byte pair encoding algorithm	2024-02-24 07:34:27 -08:00