Building a new tokenizer

Learn how to use the 🤗 Tokenizers library to build your own tokenizer, train it, then how to use it in the 🤗 Transformers library.
This video is part of the Hugging Face course: huggingface.co/...
Open in colab to run the code samples:
colab.research...
Related videos:
Training a new tokenizer: • Training a new tokenizer
Byte Pair Encoding Tokenization: • Byte Pair Encoding Tok...
Unigram Tokenization: • Unigram Tokenization
WordPiece Tokenization: • WordPiece Tokenization
Don't have a Hugging Face account? Join now: huggingface.co/...
Have a question? Checkout the forums: discuss.huggin...
Subscribe to our newsletter: huggingface.cu...