moshferatu/gpt-tokenizer
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3 Commits 1 Branch 0 Tags 35 KiB
0d79fa2009
Commit Graph

1 Commits

Author SHA1 Message Date
moshferatu
0d79fa2009 Add initial implementation of tokenizer which merges tokens according to the byte pair encoding algorithm 2024-02-24 07:34:27 -08:00