LLM Tokenizers Explained: BPE Encoding, WordPiece and SentencePiece

[@datamlistic]

If you enjoy learning about LLMs, make sure to also watch my tutorial on prompt engineering: kzbin.info/www/bejne/Y3Olpp99gpurfJI

[@boredcrow7285]

straight to the point pretty great! I have doubt in sentencepeice does the model split the corpus into character level and do the same as BPE or word peice instead of splitting it on the basis of spaces in case of english??

[@datamlistic]

Thanks! Yes, sentence piece considers the space as a stand-alone character. No pre-tokenization based on space is done there.

[@sagartamang0000]

Wow, that was amazing!

[@datamlistic]

Thanks! Happy to hear you think that! :)

[@a7med7x7]

Amazing explanation ❤

[@datamlistic]

Glad you think so! :)

[@snehotoshbanerjee1938]

Best explanation!!

[@datamlistic]

Thanks x2! :)

[@snehotoshbanerjee1938]

Best Explanation!!

[@datamlistic]

Thanks! :)