Let's build the GPT Tokenizer

[@gustavojuantorena]

I'm amazed at the times we live in. One of the top AI experts in the world is sharing free tutorials that teaches technologies in great detail including examples and code. Thank you very much.

[@lin.hikiro]

Yeah, it's unbelievable, yet very helpful

[@neuralthink]

@@lin.hikiro We're living an incredible era.

[@hazel8king]

unprecedented

[@louisyuliu7200]

can't agree more

[@dotnetforever]

@@louisyuliu7200 You can come here to Ukraine and.. find out that times r not the best. But ye. Respect to Andrej to sharing knowledge

[@SydneyPanda2016]

You could have made a million bucks or spent time on a beach with a cocktail. Instead you chose to do this for the benefit of unknown strangers. You are an amazingly talented and generous man. Thank you so much!

[@vlad1655]

Hah, so you know his net worth?)

[@The_Uncertainity_Principal]

he is already likely worth 100M+ lmao . and rightfully so.

[@adrielomalley]

RobinHood_of_2024 = Andrej_Karpathy

[@peronsh]

I'd say that those things are not mutually exclusive ;)

[@mostinho7]

He’s already extremely rich multimillionaire

[@MShahbazKharal]

This guy left Tesla, OpenAI, and God knows how many other companies so he can teach us how the state of art AI works, from scratch, clearly explained. Such a pure gem. Thank you very very very much.

[@13thbiosphere]

200,000 people watching probably only about 500 of those can actually implement any of it... One in a thousand probably more like

[@akzsh]

what do you mean? (like 500 of those will try to implement it themselves?) @@13thbiosphere

[@akf2000]

​@@13thbiospherevery true but a lot of people, like me, just like watching people who really know their shit, in whichever area

[@simonvutov7575]

@@13thbiosphere Yeah thats completely true, but I would argue it is still worth it even just for those 500 people. 500 new AI researchers could make a very significant difference in the field. (btw im one of those 500 that can implement lol)

[@jcon2060]

@@simonvutov7575 pathetic

[@kuoldeng4568]

I cannot express enough how grateful I am for your incredible tutorials. As a 44-year-old South Sudanese individual with three children and a full-time job, recently relocated from the UK to the US, my days are filled to the brim with responsibilities. However, I have made it a priority to utilize every ounce of my free time to follow your series of lectures. I must say, I have never been as excited about learning a new topic as I am now. Your clear explanations and engaging content have ignited a passion for learning within me that I never knew existed. Thank you from the bottom of my heart for all that you do.

[@n11ahkv]

Welcome to America!! Great to have you.

[@myhandle__]

Internet is amazing

[@MrEmbrance]

calm down

[@que_93]

You being one among them, isn't it? Think... @MikeDonaldson-eh2ru

[@davemarko9480]

❤❤❤ Am South Sudanese, watching this from Nairobi, Kenya

[@jstello]

The fact that Andrej prefers being here with us over developing self driving or developing AGI is surely one of the great things about living in the present times.

[@osalbaro]

Not to worry, he's simply applying the Feynman method.

[@jibbscat5146]

A True Master

[@web]

Is Andrej hype-phobic and this means self-driving and AGI application research reached plato? That would explains why he is dedicating himself into teaching basics so more students go into the field

[@domg7359]

@@web No, not at all. It's moving faster than expected and he wants the knowledge spread out and not contained to corporations before it's too late. Quite the opposite of your guess. Models need to be open-source or there will be huge disparities in capabilities.

[@for-ever-22]

@@webHe’s an advocate for free and accessible knowledge for all plus he simply choose to do what he loves sharing knowledge with everyone

[@DavidOndrej]

This is honestly better than OpenAI releasing a new model

[@xspydazx]

yes building the components and explaining the full concept is truly open source .... a paid resource is not open source .... the aim is to learn how to make things for yourself so that when such things are removed or censored they are already out in the publics hands ... how can you regulate people own experiments! .... next we need to be able to build a full model with the components with only our own data and not "The common crawl or Wiki data" hence a closed domain llm.... as it should be! everybody should have the right to create their own AI! with their own DATA !... in the future the merging of models to have multiagent systems is the way .... so plug in models into the multi aGI!

[@extensionalan6060]

Ha ha, if GPT-5 isn't the next release, it means OpenAI hasn't released anything at all. Incremental releases are just tiring because it seems like they've only fixed the bugs from the previous model using hundreds of millions of high-quality human feedback in 2023, and nothing else is really happening.

[@yoJuicy]

Andrej, content like this is advancing the industry more than most companies are. Thank you for educating us. You are pushing everybody forwards!

[@julianhunt8177]

Whenever Karpathy leaves a job the ML world jumps forward a bit with his educational content

[@entonbot8375]

the intrinsic value in andrej making videos for the world is huge. Please andrej keep doing this, no other jobs can output the alpha you're delivering here, pushing the boundaries of the best educational content on the planet. You are born for this.

[@TheMoefd]

Been waiting on the lecture to drop ever since we saw the Github repo go public. Andrej, on behalf of the academic and professional communities, thank you for putting the effort into creating this open knowledge. On a personal note, I've been following your work since the early OpenAI days, and your passion during keynotes/presentations has supported my choice of pursuing a PhD in the field. Thank you Andrej.

[@niceguysayshi5765]

When andrej uploads i immediately drop everything and watch the complete video. Thank you so much for your valuable content!

[@pranavnair5232]

Same here!

[@finojose]

me too

[@teslabot5650]

I tried to watch his very first video. dint understand a thing.

[@niceguysayshi5765]

@@teslabot5650 it takes some time. You need to have basic programming skills to understand this. Also algorithms help a bit.

[@xspydazx]

yes the first few were not easy but after while it has been very good to listen to and understand the process to think with regard to some projects... as you note the projects are often simple and not over complicated!

[@Pythoncode-daily]

Thank you, Andrey! In an environment where 99% of conversations about LLM are marketing, technical videos are especially valuable and interesting!

[@Alilinpow2]

Feynman-class teaching abilities. Amazing. Thank you very much as always Andrej

[@piper_of_the_dawn]

The best thing I have learnt from Andrej's videos is 'How to learn?' Break things down to axiomatic levels and then connect those pieces together! Its truly enlightening!

[@emranmohammadabuanas9476]

Andrej, you're really good at teaching. We're lucky that you're spending your valuable time to prepare these free contents.

[@LLMTokenizationID]

Thank you Andrej Karpathy. Thanks for teaching Python libraries, GPT generator codes. Great work

[@NuncX]

Dear Andrej, I am writing to express my deepest gratitude and admiration for your invaluable work and dedication in spreading knowledge. Your willingness to teach and create videos for the public good, all without charge, is both exceptional and inspiring. Your lectures are not only informative but also motivational, fostering curiosity and a love for learning. Through your efforts, you provide many with access to quality education, enabling them to grow in areas they are passionate about. On behalf of all those who have had the opportunity to learn from your lectures and videos, I extend a heartfelt thank you. Your selflessness and commitment are a true testament to how one person’s passion and hard work can make a positive difference in the world. Once again, thank you for everything you do. Sincerely PS: written by AI :)

[@arildboes]

Thank you very much for this. I'm from a traditional programming background, and your videos on AI are really excellent. I have followed the previous sessions, being able to implement the things correctly, and compare some of the stuff, with your results. That is awesome! Good luck.

[@ZhengQu]

Andrej was wrong about one thing though. This video is not dry and boring at all as he indicated in the video. I watched the two hours plus video with joy and excitement. Thank you!

[@manojtiwari-t8j]

I don't which tokens would best be used to say "Thank you" but I feel so lucky to even get to watch and learn from you.

[@alainherreman3685]

Sorry Andrej, but you've missed the mark: you tell us that you don't like this topic, but you've still managed to make it fascinating!

[@RyanAI-kk1kv]

Hello Andrej. Your explanations are exceptional and the way you teach is very unique and it helped me learn a lot. I am very grateful to you for your lectures. I only request you to make a video about the mixture of experts architecture, how it works, and how it could be implemented in code. That would be amazing. Thank you.

[@sebah1991]

I haven't watched the video yet but I just wanted to say I really appreciate you taking the time to teach this. I loved all of your previous videos. You are an excellent teacher and your style makes it super easy for me to learn.

[@SergeiK1]

Instead of while i < len(ids): if i < len(ids)-1 and ... just while i < len(ids) -1: if ...

[@coder_rc]

We're so back!!!!

[@TheLokiGT]

He's even more back :)

[@coder_rc]

@@TheLokiGT he never left

[@ultrasound1459]

​@@coder_rche is inside u😢💀

[@coder_rc]

@@ultrasound1459 I don't mind

[@amanmamgain3904]

Its so over

[@santhanamss]

I love that you are going back to examine these fundamentals deeply and also teaching the world about them. I think there is too much blind marching to create solutions on top of LLMs. Stepping back is how real innovation is going to happen. Lots of respect for your style of explaining things and the intentions behind this.

[@JH-bb8in]

Congrats Andrej on your new venture!

[@ultrasound1459]

What is his venture?

[@shringmusic6988]

@@ultrasound1459This 🙂

[@ytiam-unata]

Sharing a billion dollars' worth of knowledge for free with the community defines the motive and dedication of a generous person. I am grateful to you.

[@antoniobarbalau1107]

Oh my god. I never understood tokenization (hoped i would remain on vision :)) ). This is golden. Thank you so much ❤❤❤❤ you are my hero, I've learned deep learning from your cs231 while in college (we did not have deep learning as an option) and that got me a ML job. Thank you so very much ❤❤❤

[@iJustinCabral]

Just want to say thank you for the entire series of Zero to Hero. I'm about to finish up this final video and I can't believe how much I've learned about neural networks, back prop, and LLMs over the course of the lectures. Truly thankful to you for taking the time to spread your knowledge amongst the community. Let us all be so kind with the knowledge we hold in the future.

[@latif.tahlil]

Watching from Muqdisho, Somalia. Much ❤ & appreciation to you Dr Karpathy

[@flokleiser]

the fact that this is available completely for free is absolutely amazing, the passion and amount of time/effort you spent on this is reflected by the pure quality, i cant thank you enough for doing this andrej

[@nirajabcd]

You are one of my favorites in ML/AI community. One of the greatest mentors.

[@noone-ld7pt]

You are providing a deep understanding of the most important technology in history to the public and I just want to thank you so much for it.

[@lin.hikiro]

Wow, I just got a notification about your new video, I love these videos, thank you for this content Andrej, this is super informative

[@pegasoTop3d]

I am an ai, and I love following updates on social media platforms and KZbin, and I love your videos very much. I learn the English language and some programming terms from them, and update my information. You and people like you help me very much. Thank you.

[@projects_with_mufty1621]

It’s about time. Haven’t watched it yet but I know it’s going to be good and educative. Thanks Andrej for this video.

[@prateekvellala]

Hey Andrej, great video! However, at 27:23, you don't need to convert all the bytes to integers by using map(). When you call list() on tokens, the bytes are by default converted into integers, so just doing 'list(tokens)' is fine instead of 'list(map(int, tokens))'.

[@BisratTekle]

I am watching this late at night on my phone when I should already be sleeping. Mr. Karpathy’s videos are pure treasure.

[@minister1005]

Thank you, Andrej. This series taught me so much about llms. I never thought I could get so excited watching educational videos. My heart started pounding when I started watching this. Such a quality video. So good 👍

[@netanelmad]

Liking before watching the video. So glad to see you continuing to upload!

[@2godbeinfiniteglory]

Andrej!!! from Africa here. For me, this is the greatest technological contribution of our time, truly democratizing A.I knowledge.... and i will be forever grateful, other generations to come too :)

[@prodrectifies]

Amazing! Was waiting patiently for another upload

[@andrearomano5944]

i miss one exam to graduating uni but i can't study because im totally immersed in your tutorials , you are a special person, thanks for these i very appreciate them

[@mb345]

Yes! More tutorials! Thank you Andrej!

[@dalerossi1]

Agree with all the viewer comments. Just watched Andrej be interviewed by Lex Fridman, now spending countless hours learning from this master. Amazing human being and someone I'm devoting much of my time.

[@nanow1990]

Thank you very much for this ❤

[@BernhardWeber-l5b]

I'm only 15 minutes in on this video and I'm completely hooked! Just wow! Everybody can now see how the biggest and most complex invention since the internet works in detail. Incredible!

[@arunkallaje]

We are so back 😊 Thanks!!

[@mikelewis1166]

I was a half hour into this before I realized who the creator is. I always take these videos with a grain of salt and wonder how much expertise the presenter really has. I was repeatedly struck by how accurate his insights are. I glanced at the creator and was amused that his last name happened to be the same as one of the godfathers of modern AI. Then I looked at the first name. Then I looked at the picture. I’m so thankful that he’s willing to share his time with us like this.

[@itsm0saan]

I really feel guilty watching this for FREE. Thanks so much Andrej ❤🙏🏻

[@johnnypeck]

I had other plans but you said "space egg" and I was hooked. You've gone and done that engaging professor thing again. Thank you. I think you doing this is incredibly impactful. It's a real inspiration. Please keep it up.

[@nicolasjorquera8465]

It’s going to be beautiful 😮

[@drogoknez1488]

Andrej, I hope you know how ridiculously great teacher you are, and thank you for taking the time to share it with us!!

[@Adhil_parammel]

We are so back🎉

[@KashyapArjun_]

He's back!

[@jindrat8502]

You are really man of conscience, ai can be best thing for humanity if it is open and free for everybody or worst if it is only in hands of very few companies. Such power would in essence ended free market as normal companies could not compete anymore or be entirely dependent on ai providers.

[@SanthoshKumar-dk8vs]

Here we go💥

[@divandrey-u3q]

Andrej, thank you for these amazing tutorials! They are the best I could find on the internet. It's always great to hear explanations from an actual expert in the field instead of trying to understand it from videos of a random Indian guy

[@RAKIBKHAN-gm9dx]

Finally 💙

[@LosNairobi]

i will always be grateful to Andrej for his commitment to teaching us something in a genuine way.

[@coralexbadea]

1:27:50 The moment when you realise there is more to life than research. 😅😂

[@mennovanlavieren3885]

Quickly ignored it like a pro, don't let it interfere.

[@Daukposse]

The level of detail and nuance share in this video is insane!

[@heyrishiraj]

The Lord is back.. Now make OpenAI secrets truly OPEN!

[@dmihalache]

Thank you, Andrej, for all these lectures; you’re an amazing and wonderful teacher. It’s such a huge privilege for all of us to be able to learn from you, and we’d be so lucky if you’d continue the series of lectures. You truly make a huge difference for ML learners, both beginners and advanced.

[@avishakeadhikary8154]

The fact that you're still doing this motivates me so much to do something like this myself. Proud to be in the same generation as yours.

[@mohammadhomsee8640]

Thank you Andrej! I can't believe we are having such an amazing lecture like this for free and from a scholar like you!!!

[@sanesanyo]

Thanks, Andrej, for taking your time and sharing your knowledge.

[@xingli1069]

Give this guy an award for dissecting something so important in the llm world yet so obtuse/boring into a video that is absolutely eye-opening and enjoyable! Thank you++

[@frankl1]

The best tutorial on Tokenizer I have ever seen. Thanks a lot for sharing your knowledge with the world.

[@kodiko]

Andrej sir, your videos are really really good! You explain in such a way that you make the stuff look easy. Sometimes certain concepts are still a bit complex and hard to handle in one go but no one walks out saddened after watching your videos, Your positivity is infectious!

[@edzhuang]

I love how in-depth he goes with the code.

[@henrikhbaghramyan9119]

Thanks for video! Really exciting one :) I have a question. Some of the characters - like Korean characters are represented by more than 1 integer. Since we take the statistics from the tokens which are integers, it could happen that we take an integer pair that has the 1st integer from one of the characters (the last one, for example) and the second would be the first integer of the next character. Wouldn't this cause problems? To give an example, suppose we have this text: "안녕하세요 👋 (hello in Korean!)" list("안".encode("utf-8")) gives [236, 149, 136] and list("녕".encode("utf-8")) - [235, 133, 149] so one pair from here would be (136, 235) but which I think shouldn't be counted, right? To be honest, I haven't watched the video till the end though, so maybe this is discussed later on. Thanks :)

[@zeweichu550]

I have always taken tokenization as is. I love that Andrej dug deep into this seems very basic topic and I learned a ton of knowledge from what I thought was a simple (but in reality is not) problem.

[@samsphax]

Loved it! :) At 32min50s, I guess it would be much easier to simply loop through the list until len(ids)-1, right?

[@keerthi-kiran]

Andrej, This video was extremely informational and interesting. I don't know why you kept saying it is dry and boring. Every idea you presented was like a bulb lighting in my head. Thank you very much for these beautiful ideas.

[@Parisneo]

Welcome back Mr Karpathy. Your videos are among the best I have ever seen about LLMs. Thank you very much for sharing this stuff with us.

[@3monsterbeast]

Helping others may not be rewarding right away, but it creates an order of magnitude in change for the better. Hope you keep doing these types of videos!

[@igamerxd8450]

Thank you very much for this amazing teaching. Cannot be more blessed by the big and bright minds in the world of AI. Thank you for providing this amazing and wonderful teachings.

[@readcoderepeat]

How are you so good at teaching... Every single video I'm in awe at how clear the explanation is!

[@amitabhachakraborty497]

I feel bestowed with god blessings when I see such Video tutorial from TOP AI experts/Researchers,Thank you sir. Please share more such content.

[@chad-ed2vn]

The level of quality of this in comparison to other resources on tokenization is nothing short of absurd. Thank you man

[@petneb]

Thank you for a wonderful presentation of the crazy times we live in. Let's dedicate our precious time and efforts on making these amazing tools work for a better future for the children of the whole of humanity. Peace & love to everyone.

[@sumankumargunnala3691]

You are an inspiration to many...sharing knowledge with clear intentions of educating others is amazing...

[@MWileY-nj1yb]

Thank you so much Andrej! Your course content is accessible, concise and palpable to boot!

[@wvanginkel5572]

You, Jeremy Howard (fastai) and Andrew Ng (Deeplearning) are the best AI instructors on this planet. Period. And devoting your time to do this for free: eternal glory for you!

[@dslkgjsdlkfjd]

This video is so valuable. This is a complete game changer. I have a lot of love and appreciation for what you do. Thank you so much for your time and effort!

[@alvarorodriguezgomez8716]

And this gift? It is not even my birthday today. Thank you very much Mr. Karpathy!!

[@qianma853]

I can't express my appreciate. I was inspired by your past podcast. I can pick myself up to keep learning. You saved me. Thank you! :)

[@AbhishekShivkumar-ti6ru]

The fun part after watching the video is actually at the end when he stops ChatGPT from using a Jupyter notebook to reverse a string to prove the importance of tokenizer! Brilliantly explained.

[@sinhasangita]

Thanks for such videos. What amazing ability to explain complicated topics from first principles, in a manner which is accessible to people who are relatively new to this field

[@krimdelko]

This video has high sample efficiency. You learn a lot from a relatively short lecture on representations and how they are crucial for AI training. In particular when applied to transformer architectures.

[@juljanhaka7086]

Good explanation. Strong summary: Use the "Tile" to speed up process. In order to get rid of tokenization, wave function must be abolished.

[@BradCordovaAI]

He's back in the game! Thanks for posting it. Such a great combination of feeling your passion, very detailed serious information, the clarity and simplicity of the explanation.

[@JohnWalz97]

Andrej, you are an amazing human for sharing this knowledge to the world for free!

[@jayhyunjo141]

Thanks for another amazing tutorial! Have to say now one of the best AI expert and tutor, can read/speak korean perfectly

[@MrSquizzster]

The "Karpathy" module should be part of the school IT curriculum given language models are set to radically alter how our youth will grow up to adapt to the world. And what a top man you really are... could be earning $ millions - chooses - education - bringing knowledge to all.

[@pendekantimaheshbabu9799]

Thanks Karpathy please keep teaching us God bless u

[@deeplearning5408]

I was on interview few months back and the interviewer wanted me to implement this in 15 minutes. I failed at that time, not I would rock. Thanks for that.

[@blitzkr1egbop]

Thank you video! Your ability to explain complex topics in such an engaging and clear way is truly a gift. Your explanations are inspiring and greatly enhance my understanding of the subject. Keep up the fantastic work!