Commit Graph

Select branches

Hide Pull Requests

main

Mono Color

• ecd9c7eeee Add the ability to encode and decode text to and from tokens main moshferatu 2024-02-25 06:40:08 -0800
• 0d79fa2009 Add initial implementation of tokenizer which merges tokens according to the byte pair encoding algorithm moshferatu 2024-02-24 07:34:27 -0800
• 29c234efa8 Add example text to tokenize moshferatu 2024-02-24 07:33:58 -0800
• faca122a7a Add .gitignore moshferatu 2024-02-24 06:54:59 -0800