ChatGPT has Never Seen a SINGLE Word (Despite Reading Most of The Internet). Meet LLM Tokenizers.

[@shabiehsaeed8633]

Hi Jay, I love the work you have done! Ever since I read the Illustrated Transformer, I was blown away by your explanations and illustrations. You really explain advance concepts with such clarity and simplicity. I am very grateful to you for that! I really look forward to reading and learning from your book! Thank you so much!!

[@devashishsoni9371]

Can you please share the link of that?

[@lazycomedy9358]

yeah. Same here, shout out for that.

[@jwilber92]

Great content as always, Jay!

[@ranjancse26]

This is incredible. Great work! Keep it up :)

[@WhatsAI]

Great video as always Jay! :)

[@arp_ai]

Thank you Louis!

[@123arskas]

Good one

[@goelnikhils]

Great Work

[@KumR]

Hi Jay - Great video.. Wondering if this is similar to computers doing everything in 0s and 1s although from the OS level the abstraction is different . At least conceptually. Coming to the book , I am not able to find it anywhere... Is there a link ?

[@prabaj84]

Hi Jay, thanks again for explaining a complex topic in simple way, if I may ask, what tool do you use to generate graphics for your blogs? Thanks in advance

[@kidsfungaming6756]

Hi Jay, I love your presentation, it is so inspiring and you make the hard concepts simple and clearer. Regarding the tokenizer, if every word is one token and the same is mapped over a single vector (embeddings) then how do LLMs clearly understand the meaning of the same word in different contexts? I will appreciate your answer and I am sorry if my question is too naive. Thank you

[@gama3181]

And how the people know wich tokenizer is the best way to split the vocab? This follow a math rule or statistical pattern? or it depend on the computing budget?

[@TheAero]

I would love if you could go into the following: RLHF. PPO. PEFT. LORA etc. Adapters. soft-prompting. scaling transformers.

[@arp_ai]

Delicious topics indeed

[@mohamadbebah8416]

Great!! Thank you very much

[@khaledsrrr]

❤ very nice

[@boonkiathan]

Neither have our neurones

[@arp_ai]

Aha! But which neurons though!

[@amittripathi6664]

Hi Jay, thanks for the video. Could you also please share the code?

[@ashisranjanlahiri]

Hi Sir, your video always amazed me. Need more videos for sure. Can you please share the notebook link.

[@arp_ai]

Thank you! Haven't published the notebook yet, but that's a good idea

[@Patapom3]

Great! But how does the tokenizer works now? 😅

[@arp_ai]

Wonderful! If you feel comfortable to tackle this now, then this video has done its job. We'll address it more in the book (and possible a subsequent video). But if you wanna get into training tokenizers now, this is a great guide: huggingface.co/learn/nlp-course/chapter6/5?fw=pt

[@tanmoy.mazumder]

could you perhaps do an even deeper dive about how these models exactly produce the output vectors and then how those get turned into these tokens and stuff?

[@arp_ai]

Not much has changed since my videos on GPT3, honestly. Check those out.