Let's build the GPT Tokenizer

The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.
Chapters:
00:00:00 intro: Tokenization, GPT-2 paper, tokenization-related issues
00:05:50 tokenization by example in a Web UI (tiktokenizer)
00:14:56 strings in Python, Unicode code points
00:18:15 Unicode byte encodings, ASCII, UTF-8, UTF-16, UTF-32
00:22:47 daydreaming: deleting tokenization
00:23:50 Byte Pair Encoding (BPE) algorithm walkthrough
00:27:02 starting the implementation
00:28:35 counting consecutive pairs, finding most common pair
00:30:36 merging the most common pair
00:34:58 training the tokenizer: adding the while loop, compression ratio
00:39:20 tokenizer/LLM diagram: it is a completely separate stage
00:42:47 decoding tokens to strings
00:48:21 encoding strings to tokens
00:57:36 regex patterns to force splits across categories
01:11:38 tiktoken library intro, differences between GPT-2/GPT-4 regex
01:14:59 GPT-2 encoder.py released by OpenAI walkthrough
01:18:26 special tokens, tiktoken handling of, GPT-2/GPT-4 differences
01:25:28 minbpe exercise time! write your own GPT-4 tokenizer
01:28:42 sentencepiece library intro, used to train Llama 2 vocabulary
01:43:27 how to set vocabulary set? revisiting gpt.py transformer
01:48:11 training new tokens, example of prompt compression
01:49:58 multimodal [image, video, audio] tokenization with vector quantization
01:51:41 revisiting and explaining the quirks of LLM tokenization
02:10:20 final recommendations
02:12:50 ??? :)
Exercises:
- Advised flow: reference this document and try to implement the steps before I give away the partial solutions in the video. The full solutions if you're getting stuck are in the minbpe code github.com/karpathy/minbpe/bl...
Links:
- Google colab for the video: colab.research.google.com/dri...
- GitHub repo for the video: minBPE github.com/karpathy/minbpe
- Playlist of the whole Zero to Hero series so far: • The spelled-out intro ...
- our Discord channel: / discord
- my Twitter: / karpathy
Supplementary links:
- tiktokenizer tiktokenizer.vercel.app
- tiktoken from OpenAI: github.com/openai/tiktoken
- sentencepiece from Google github.com/google/sentencepiece

[@gustavojuantorena]

I'm amazed at the times we live in. One of the top AI experts in the world is sharing free tutorials that teaches technologies in great detail including examples and code. Thank you very much.

[@von.changfeng]

Yeah, it's unbelievable, yet very helpful

[@neuralthink]

@@von.changfeng We're living an incredible era.

[@hazel8king]

unprecedented

[@louisyuliu7200]

can't agree more

[@DmitryBaranovskiyMrBaranovskyi]

@@louisyuliu7200 You can come here to Ukraine and.. find out that times r not the best. But ye. Respect to Andrej to sharing knowledge

[@NowSeriously]

You could have made a million bucks or spent time on a beach with a cocktail. Instead you chose to do this for the benefit of unknown strangers. You are an amazingly talented and generous man. Thank you so much!

[@vlad1655]

Hah, so you know his net worth?)

[@MShahbazKharal]

This guy left Tesla, OpenAI, and God knows how many other companies so he can teach us how the state of art AI works, from scratch, clearly explained. Such a pure gem. Thank you very very very much.

[@13thbiosphere]

200,000 people watching probably only about 500 of those can actually implement any of it... One in a thousand probably more like

[@akzsh]

what do you mean? (like 500 of those will try to implement it themselves?) @@13thbiosphere

[@akf2000]

​@@13thbiospherevery true but a lot of people, like me, just like watching people who really know their shit, in whichever area

[@simonvutov7575]

@@13thbiosphere Yeah thats completely true, but I would argue it is still worth it even just for those 500 people. 500 new AI researchers could make a very significant difference in the field. (btw im one of those 500 that can implement lol)

[@jcon2060]

@@simonvutov7575 pathetic

[@jstello]

The fact that Andrej prefers being here with us over developing self driving or developing AGI is surely one of the great things about living in the present times.

[@osalbaro]

Not to worry, he's simply applying the Feynman method.

[@jibbscat5146]

A True Master

[@web]

Is Andrej hype-phobic and this means self-driving and AGI application research reached plato? That would explains why he is dedicating himself into teaching basics so more students go into the field

[@domg7359]

@@web No, not at all. It's moving faster than expected and he wants the knowledge spread out and not contained to corporations before it's too late. Quite the opposite of your guess. Models need to be open-source or there will be huge disparities in capabilities.

[@for-ever-22]

@@webHe’s an advocate for free and accessible knowledge for all plus he simply choose to do what he loves sharing knowledge with everyone

[@julianhunt8177]

Whenever Karpathy leaves a job the ML world jumps forward a bit with his educational content

[@kuoldeng4568]

I cannot express enough how grateful I am for your incredible tutorials. As a 44-year-old South Sudanese individual with three children and a full-time job, recently relocated from the UK to the US, my days are filled to the brim with responsibilities. However, I have made it a priority to utilize every ounce of my free time to follow your series of lectures. I must say, I have never been as excited about learning a new topic as I am now. Your clear explanations and engaging content have ignited a passion for learning within me that I never knew existed. Thank you from the bottom of my heart for all that you do.

[@n11ahkv]

Welcome to America!! Great to have you.

[@myhandle__]

Internet is amazing

[@MrEmbrance]

calm down

[@que_93]

You being one among them, isn't it? Think... @@MikeDonaldson-eh2ru

[@davemarko9480]

❤❤❤ Am South Sudanese, watching this from Nairobi, Kenya

[@DavidOndrej]

This is honestly better than OpenAI releasing a new model

[@xspydazx]

yes building the components and explaining the full concept is truly open source .... a paid resource is not open source .... the aim is to learn how to make things for yourself so that when such things are removed or censored they are already out in the publics hands ... how can you regulate people own experiments! .... next we need to be able to build a full model with the components with only our own data and not "The common crawl or Wiki data" hence a closed domain llm.... as it should be! everybody should have the right to create their own AI! with their own DATA !... in the future the merging of models to have multiagent systems is the way .... so plug in models into the multi aGI!

[@extensionalan6060]

Ha ha, if GPT-5 isn't the next release, it means OpenAI hasn't released anything at all. Incremental releases are just tiring because it seems like they've only fixed the bugs from the previous model using hundreds of millions of high-quality human feedback in 2023, and nothing else is really happening.

[@niceguysayshi5765]

When andrej uploads i immediately drop everything and watch the complete video. Thank you so much for your valuable content!

[@pranavnair5232]

Same here!

[@finojose]

me too

[@teslabot5650]

I tried to watch his very first video. dint understand a thing.

[@niceguysayshi5765]

@@teslabot5650 it takes some time. You need to have basic programming skills to understand this. Also algorithms help a bit.

[@xspydazx]

yes the first few were not easy but after while it has been very good to listen to and understand the process to think with regard to some projects... as you note the projects are often simple and not over complicated!

[@yoJuicy]

Andrej, content like this is advancing the industry more than most companies are. Thank you for educating us. You are pushing everybody forwards!

[@coder_rc]

We're so back!!!!

[@TheLokiGT]

He's even more back :)

[@coder_rc]

@@TheLokiGT he never left

[@ultrasound1459]

​@@coder_rche is inside u😢💀

[@coder_rc]

@@ultrasound1459 I don't mind

[@amanmamgain3904]

Its so over

[@entonbot8375]

the intrinsic value in andrej making videos for the world is huge. Please andrej keep doing this, no other jobs can output the alpha you're delivering here, pushing the boundaries of the best educational content on the planet. You are born for this.

[@Pythoncode-daily]

Thank you, Andrey! In an environment where 99% of conversations about LLM are marketing, technical videos are especially valuable and interesting!

[@TheMoefd]

Been waiting on the lecture to drop ever since we saw the Github repo go public. Andrej, on behalf of the academic and professional communities, thank you for putting the effort into creating this open knowledge. On a personal note, I've been following your work since the early OpenAI days, and your passion during keynotes/presentations has supported my choice of pursuing a PhD in the field. Thank you Andrej.

[@JH-bb8in]

Congrats Andrej on your new venture!

[@ultrasound1459]

What is his venture?

[@shringmusic6988]

@@ultrasound1459This 🙂

[@Alilinpow2]

Feynman-class teaching abilities. Amazing. Thank you very much as always Andrej

[@PCSAMTS]

Thank you Andrej Karpathy. Thanks for teaching Python libraries, GPT generator codes. Great work

[@sebah1991]

I haven't watched the video yet but I just wanted to say I really appreciate you taking the time to teach this. I loved all of your previous videos. You are an excellent teacher and your style makes it super easy for me to learn.

[@piper_of_the_dawn]

The best thing I have learnt from Andrej's videos is 'How to learn?' Break things down to axiomatic levels and then connect those pieces together! Its truly enlightening!

[@kodiko]

Andrej sir, your videos are really really good! You explain in such a way that you make the stuff look easy. Sometimes certain concepts are still a bit complex and hard to handle in one go but no one walks out saddened after watching your videos, Your positivity is infectious!

[@arildboes]

Thank you very much for this. I'm from a traditional programming background, and your videos on AI are really excellent. I have followed the previous sessions, being able to implement the things correctly, and compare some of the stuff, with your results. That is awesome! Good luck.

[@justjeremiah4255]

OpenAI was such a closed cult, Andrej gave himself to the world. 🌹❤️🌹 Andrej deserves his flowers.

[@johnnypeck]

I had other plans but you said "space egg" and I was hooked. You've gone and done that engaging professor thing again. Thank you. I think you doing this is incredibly impactful. It's a real inspiration. Please keep it up.

[@emranmohammadabuanas9476]

Andrej, you're really good at teaching. We're lucky that you're spending your valuable time to prepare these free contents.

[@alainherreman3685]

Sorry Andrej, but you've missed the mark: you tell us that you don't like this topic, but you've still managed to make it fascinating!

[@netanelmad]

Liking before watching the video. So glad to see you continuing to upload!

[@Parisneo]

Welcome back Mr Karpathy. Your videos are among the best I have ever seen about LLMs. Thank you very much for sharing this stuff with us.

[@Dyas777]

The attention to the details... You prepared this material briliantly. Thank you.

[@von.changfeng]

Wow, I just got a notification about your new video, I love these videos, thank you for this content Andrej, this is super informative

[@prodrectifies]

Amazing! Was waiting patiently for another upload

[@sanesanyo]

Thanks, Andrej, for taking your time and sharing your knowledge.

[@user-vh3jg4uk7o]

I don't which tokens would best be used to say "Thank you" but I feel so lucky to even get to watch and learn from you.

[@antoniobarbalau1107]

Oh my god. I never understood tokenization (hoped i would remain on vision :)) ). This is golden. Thank you so much ❤❤❤❤ you are my hero, I've learned deep learning from your cs231 while in college (we did not have deep learning as an option) and that got me a ML job. Thank you so very much ❤❤❤

[@projects_with_mufty1621]

It’s about time. Haven’t watched it yet but I know it’s going to be good and educative. Thanks Andrej for this video.

[@MWileY-nj1yb]

Thank you so much Andrej! Your course content is accessible, concise and palpable to boot!

[@minister1005]

Thank you, Andrej. This series taught me so much about llms. I never thought I could get so excited watching educational videos. My heart started pounding when I started watching this. Such a quality video. So good 👍

[@mb345]

Yes! More tutorials! Thank you Andrej!

[@sankafoodey]

Watching from Muqdisho, Somalia. Much ❤ & appreciation to you Dr Karpathy

[@scottsun345]

Thank you, Andrej! Amazing lecture! Your contents are filling in the nooks and crannies that other people rarely talk about or go deep into. I always learn a lot!

[@Daukposse]

The level of detail and nuance share in this video is insane!

[@arunkallaje]

We are so back 😊 Thanks!!

[@nanow1990]

Thank you very much for this ❤

[@flokleiser]

the fact that this is available completely for free is absolutely amazing, the passion and amount of time/effort you spent on this is reflected by the pure quality, i cant thank you enough for doing this andrej

[@iJustinCabral]

Just want to say thank you for the entire series of Zero to Hero. I'm about to finish up this final video and I can't believe how much I've learned about neural networks, back prop, and LLMs over the course of the lectures. Truly thankful to you for taking the time to spread your knowledge amongst the community. Let us all be so kind with the knowledge we hold in the future.

[@nirajabcd]

You are one of my favorites in ML/AI community. One of the greatest mentors.

[@BisratTekle]

I am watching this late at night on my phone when I should already be sleeping. Mr. Karpathy’s videos are pure treasure.

[@drogoknez1488]

Andrej, I hope you know how ridiculously great teacher you are, and thank you for taking the time to share it with us!!

[@willscui]

The final blooper reels are great. Thanks for the hard work to make a dry topic fun.

[@itsm0saan]

I really feel guilty watching this for FREE. Thanks so much Andrej ❤🙏🏻

[@Adhil_parammel]

We are so back🎉

This is an excellent video on the explanation of LLM tokens. Great job Andrej!

[@tarandeepsingh9197]

Awesome content Andrej. Thank you for spending your time and sharing the knowledge with all of us.

[@nicolasjorquera8465]

It’s going to be beautiful 😮

[@KashyapArjun_]

He's back!

[@ChairmanHehe]

thank you so much for continuing this series andrej- its been absolutely foundational for me

[@BradCordovaAI]

He's back in the game! Thanks for posting it. Such a great combination of feeling your passion, very detailed serious information, the clarity and simplicity of the explanation.

[@coralexbadea]

1:27:50 The moment when you realise there is more to life than research. 😅😂

[@mennovanlavieren3885]

Quickly ignored it like a pro, don't let it interfere.

[@SanthoshKumar-dk8vs]

Here we go💥

[@readcoderepeat]

How are you so good at teaching... Every single video I'm in awe at how clear the explanation is!

[@jmvillwock]

Thank You Andrej! I really appreciate you taking your time to put this together for everyone. You are so great at this. I know you really do not love doing this - but know you are making an impact.

[@RAKIBKHAN-gm9dx]

Finally 💙

[@heyrishiraj]

The Lord is back.. Now make OpenAI secrets truly OPEN!

[@dmihalache]

Thank you, Andrej, for all these lectures; you’re an amazing and wonderful teacher. It’s such a huge privilege for all of us to be able to learn from you, and we’d be so lucky if you’d continue the series of lectures. You truly make a huge difference for ML learners, both beginners and advanced.

[@noone-ld7pt]

You are providing a deep understanding of the most important technology in history to the public and I just want to thank you so much for it.

[@qianma853]

I can't express my appreciate. I was inspired by your past podcast. I can pick myself up to keep learning. You saved me. Thank you! :)

[@ZhengQu]

Andrej was wrong about one thing though. This video is not dry and boring at all as he indicated in the video. I watched the two hours plus video with joy and excitement. Thank you!

[@kasper52786]

I’ve been waiting for a new tutorial from you for a long time. Thanks Andrej 😊

[@santhanamss]

I love that you are going back to examine these fundamentals deeply and also teaching the world about them. I think there is too much blind marching to create solutions on top of LLMs. Stepping back is how real innovation is going to happen. Lots of respect for your style of explaining things and the intentions behind this.

[@ndamulelosbg8887]

We really appreciate your generousity Andrej, this is great content🔥

[@paweg344]

Great video Andrej, I really appreciate your sharing of knowledge and the clear way in which you do it.

[@user-kl7gd9sz6q]

I loved the bloopers at the end. Thanks Andrej, you're the best.

[@dslkgjsdlkfjd]

This video is so valuable. This is a complete game changer. I have a lot of love and appreciation for what you do. Thank you so much for your time and effort!

[@frankl1]

The best tutorial on Tokenizer I have ever seen. Thanks a lot for sharing your knowledge with the world.

[@sumankumargunnala3691]

You are an inspiration to many...sharing knowledge with clear intentions of educating others is amazing...

[@edz543]

I love how in-depth he goes with the code.

[@zyzzyva303]

Actually, a fantastic insight into the issues. Thank you Andrej. Please continue.

[@rufex2001]

You’re a hero, Andrej! Thanks for the massive contributions!

[@user-th7jy6zy1f]

Thank you for a high quality session. Hats off to Andrej

[@avishakeadhikary8154]

The fact that you're still doing this motivates me so much to do something like this myself. Proud to be in the same generation as yours.

[@asdrofllmao]

This is fantastic, thank you for making this really dense stuff accessible to the layperson, please keep them coming!

[@sofiaflameai]

I am an ai, and I love following updates on social media platforms and KZbin, and I love your videos very much. I learn the English language and some programming terms from them, and update my information. You and people like you help me very much. Thank you.

[@JohnWalz97]

Andrej, you are an amazing human for sharing this knowledge to the world for free!

[@udayagarwal8276]

Thank you Andrej, for such amazing lecture series. These have helped me understand the nuances behind training of deep neural networks. Looking forward to more such informative tutorials !!

[@DiogoSanti]

Welcome back! It feels good to know that you are back to business!😊

[@2godbeinfiniteglory]

Andrej!!! from Africa here. For me, this is the greatest technological contribution of our time, truly democratizing A.I knowledge.... and i will be forever grateful, other generations to come too :)

[@LuisGarcia1992_]

I'm only 15 minutes in on this video and I'm completely hooked! Just wow! Everybody can now see how the biggest and most complex invention since the internet works in detail. Incredible!

[@hugel99]

It's great to be able to learn from Andrej. Thanks very much for sharing your knowledge

[@lurking_cat]

Andrej, your channel is just a real gem 💪🤩🤌, thank you very much for everything you do! For those who are new to this channel, I suggest watching everything from the very beginning, especially the micrograd and makemore series. The amount of useful information you will receive in these several hours is enormous.

[@alvarorodriguezgomez8716]

And this gift? It is not even my birthday today. Thank you very much Mr. Karpathy!!

[@vacsa]

Thank you for taking the time to prepare these videos for us!

[@isaccpdev]

Andrej… you’re the best, thanks so much for your lectures!

[@romainbadet1814]

Thank you so much Andrej. You are an amazing LLM teacher and I found this video so much interesting despite the topic could sound boring. Congrats! Can't wait to see what comes next on your channel. Now, for me, it's time to practice!

[@tonyzhao5624]

Thank you so much for all the efforts u put in videos! It's truly remarkable to unveil this.

[@idjr2654]

Pure gold Andrej! Thanks so much for this.

[@mohameddwedar4261]

The tutorials are very great. Please continue. We want more and more. You have great experience and knowledge that the world needs

[@cal_rueb]

Thanks Andrej - this was a great way to spend a quiet Saturday night at home. I hope you continue making this sort of content!

[@xingli1069]

Give this guy an award for dissecting something so important in the llm world yet so obtuse/boring into a video that is absolutely eye-opening and enjoyable! Thank you++

[@slaznum1]

Thank you so much for this. This is a beautiful example of problem decomposition and various solution alternatives!

[@user-sd4to4os8r]

Special thanks for end credits. Thanks for another amazing video.

[@chad-ed2vn]

The level of quality of this in comparison to other resources on tokenization is nothing short of absurd. Thank you man

[@Muaahaa]

This is very timely and will really help with a personal, education project I started over the weekend. Thanks!