Let's reproduce GPT-2 (124M)

[@kiw2535]

It’s rare to find such high-quality, free resources that make complex topics accessible and engaging!

[@xspydazx]

i think you will find out that some people know that they wil be taking this technology away from the public ....... same as google glass .... as soon as america find out they cannot sread dissemination of knowledge they will cuts all internet cables !

[@Handlebrake2]

Jesus - that's some tip!

[@bbcc2960]

❤

[@unclecode]

@@Handlebrake2 :)) Jesus - That's some 4hrs of brilliant content!

[@fallingexistence]

Dude this guy's net worth is $50M, you could've bought yourself 10 burritos at chipotle.

[@carlosgermosen1103]

Do not ever look at how long your videos are. Your content is perfect and you should keep explaining things step by step. You are doing a great job. I believe you will be remembered in history as one of the pillars of AI.

[@pin65371]

I think one cool thing with videos like this is once Google implements their AI into youtube anyone will be able to watch this and just start asking questions. I've been learning a lot by watching videos like this and just copying parts of the transcript into ChatGPT to ask questions when I dont understand something.

[@sohambit9393]

I was excited by how long it was instead 😂😂

[@quackwilliams5933]

@@pin65371 jesus christ what a thought...and Andrej just starts talking back to you, answering your exact questions...

[@barackobama4552]

@@pin65371 What other videos do you recommend that have helped you?

[@xspydazx]

Yes a great sharer of the important information and implementation. As well as to let you know it's in your hands to make your own before the internet is closed down or restricted in your areas or your cable gets cut !! Great work ❤

[@Doggi2dog]

My life is simple; Andrej drops GPT-2 The Movie, I watch.

[@AndrejKarpathy]

"GPT-2 The Movie" 😅

[@DamianReloaded]

The movie and the sequel. I had to force myself to stop watching after I realized an hour had passed.

[@asatorftw]

@@AndrejKarpathy The sequel "GPT The Movie" will be a old Hong Kong style "martial arts" movie about GPT getting beaten up by the Loss Function, then entering his training phase with Gradient Descent Sensei and the final showdown vs the big evaluation boss.

[@georgiandanciu3482]

You know you gotta bring the popcorn

[@AbhigyanKeshav169]

Andrej poster video on your biography​@@AndrejKarpathy

[@mohamedalansary2542]

The fact this video is free is incredible.

[@manojr440]

Thanks for spreading the knowledge! Happy to see a 4hr workout session 😅

[@AndrejKarpathy]

:) Workout 🏋️‍♂️ is very much the right way to think about it imo!

[@jeffrey5602]

Heavy weight training

[@MsLe2016]

more like a 3-day session to me, but I'm happy :)

[@neilamrathod122]

Sorry , i love your videos and what you doing for me. I couldn't attend Stanford or get into openai but learning from you is blessing to me.. i would pay you back 100times in coming years. And i was watching your git repository last two months, i could see many git code push in private ,but i was confused what he is working on.. this is he was working on. To provide quality pratical knowledge to us all on youtube.

[@shutup1209]

Hey, the webside for the GPT2 is down, is there anyway to dowload it ?

[@neilamrathod122]

Sorry i will have look up to it ..i will do today night will reply to you after that ​@@shutup1209

[@DingLi-hw4ul]

hf​@@shutup1209

[@hobbytan6841]

@@shutup1209 you should follow the video and make it from scratch! 😀

[@dinorossi6611]

Those are generous tips :). I wanna learn basics so I can understand this first.

[@C0D3633K]

Andrej is doing himself what OpenAi was supposed to do in the early days - make AI open. Thank you, Andrej!

[@zachyamaoka7916]

Thanks Andrej! You have taught me everything I know about the theory and practice of neural networks, starting with CS231n till now. I love how you explain things starting with simple examples to build intuitions (template matching for CV, bigram/table look up for sequence modelling), and then build to state of the art. Your lessons have had a profound impact on my learning, and I can imagine there are 1000s of engineers out there just like me.

[@charlesm835]

You're a legend! Andrej will light up when he see's this 🤍

[@AndrejKarpathy]

you're too thankful ty! :)

[@anthonycho6344]

My anterior mid-cingulate cortex is getting bigger just watching this video because it’s hard! Thank you for your lessons, master Kaparthy.

[@user-oq7ju6vp7j]

It's such a privilege to watch high-quality content from leading experts for free

[@biboyog]

true

[@unclecode]

Thanks! 4 hours of decoding a "Decoder-Transformer", Kudos and appreciate your existence in this field.

[@sabareesh_42]

Thanks!

[@broccoli322]

Perfect to watch while on a plane

[@Ishaheennabi]

and then running gpt 2 on planes computer

[@waytolegacy]

This guy is "the one" in the industry, who has helped me understand the LLMs. I respectfully love this man. Hats off.

[@marcotuc-ilmarinaio924]

Thank you Andrej! from zero to hero boosted my professional career!

[@tanaysood]

Thanks AK, appreciate you sharing your knowledge with the world!

[@chenmarkson7413]

I am an undergraduate student. This is the lost lecture that professors never touched upon but absolutely crucial, thank you!! I especially love how you start from the basics for so many notions, and I really learned a lot.

[@ppyogesh7394]

Which year in you are and which country

[@chenmarkson7413]

@@ppyogesh7394 I am at the University of Toronto, going to the third year this September

[@Khobalt664]

You are the Excalibur of cutting through the hype. Thank you so much. Your ethics are inspiring, and your educational materials priceless.

[@niclaswustenbecker8902]

I think your teaching projects may be the most impactful ever existed. First CS231n the course that inspired a generation of students to pursue Deep Learning, then micrograd and nanogpt that gives us the power to recreate billion dollar research for basically free. Not to mention that many companies like tiny grad and suno were inspired by your projects. Thanks for sharing your knowledge in such a clean and elegant way!

[@JT-mr3db]

The intellectual generosity of this man is of the highest standard.

[@tothespace2122]

These kinds of videos is what the world needs more. Long form content with real time thinking. You see the master at his craft in real time. So much to learn from this!

[@rainwang77]

Hello Andrej, thank you so much for the sharing and effort! Really appreciate it!

[@dylan_curious]

I like when you add comments/metaphors about your intuition for how and why it works. Thanks you.

[@pxbroccoli]

Checkout this man here, he got the best Ai news

[@generallifing]

We have one of the best people on the planet to walk you through it step by step (I haven't seen it yet, but I believe so). I am eager to learn this and want to master it. Thank you, thank you, and thank you very much!

[@somdubey5436]

The longer your videos are, the better it is for humanity. I think you are such a wonderful person and providing this stuff for everyone for free, can't thank you enough.

[@Themojii]

I've learned a lot from your Neural Network video playlist. Thank you

[@themenon]

Many Thanks to Andrej for making this tutorial available to everyone! I have never seen a clearer explanation of a nn before stumbling upon this zero to hero series. This will help all the people articulate the inner workings of neural net and help people understand deeper concepts, that is hard to understand. Looking forward to learning more with Andrej!

[@gromeronaranjo]

I rarely comment on videos, but I had to here I had to. Your in-depth high-quaity resources are something to talk about. You make very complicated topics easy and engaging, your provide the knowledge for anyone to learn these highly-regarded concepts. Fruthermore, you are truly advancing the general knowledge of the public by providing these powerful videos. I would just like to express my gratitude for your videos, and how they really are making a positive impact. Thank you for dedicating many hours of work to upload these videos.

[@bmatichuk]

Your guidance is inspiring.

[@JC-ys2ch]

Super useful! Looking forward to any in-depth "anatomy" video on Mamba Architecture as well.

[@divandrey-u3q]

Yeah, mamba would be interesting to see... I hope Andrej hears us

[@forrestye2194]

Finally, finished watching such a long video. Thank Andrej for sharing so many details of your knowledge. Like your teaching style so much since Tesla AI day. You are the best AI teacher!

[@qiuchenguo2788]

Simply the best deep learning and LLM series online! Please keep making more videos and I'd love to be part of the journey!

[@souravzzz]

🤗What an absolutely fantastic explanation! Every minute is filled with nuggets of deep insights!

[@IgorTsvetkov]

Thanks for your Zero-to-hero series!

[@AndrejKarpathy]

wow you're very thankful ty! :)

[@Ibbysz]

Andrej in a few years: Lets reproduce GPT-5 (124T)

[@Alconno]

we can only hope

[@Person-hb3dv]

and it's gonna cost 3$ to reproduce

[@Katatonya]

I wonder how long will it take for even GPT4 to be trainable on our own rigs. Nvidia said that in 8 years, computation will be reduced 350x times. (i.e. a gpu you will buy in 8 years, will be 350x better at training, if I understood correctly). Though is this enough? And 8 years is a loooong time in the present AI world.

[@Person-hb3dv]

@@Katatonya I think it's going to happen even faster. with the current rate of advancements, in 8 years training something like GPT-4 locally would be trivial to say the least. but i'm just guessing. who knows what will happen.

[@Katatonya]

@@Person-hb3dv I think it purely depends on if the models will get much much more efficient and cheaper to run. They most likely will. Hardware-wise though, Nvidia didn't guess what will happen in 8 years, they know for sure as they have to plan in advance.

[@palimondo]

Tieto videá sú naozaj skvelé. Snažil som sa hlbšie pochopiť ako fungujú LLM čítaním vedeckých publikácií. Keďže som vyštudoval odbor softvérové inžinierstvo a nie umelá inteligencia a tiež som neabsolvoval postgraduálne štúdium, “papers” sa mi, kvôli medzerám v mojich znalostiach, čítajú veľmi ťažko. Keď mi ukazuješ kód, krok za krokom, všetko do dnes zapadá a dáva mi to zmyslel. Asi nebudem na LambdaLabs node robiť pre-training vlastného modelu, ale ten pocit, že s tým čo si ma tu naučil by som to teoreticky dokázal je neskutočne silný. Si perfektný učiteľ. Ďakujem ti, Andrej!

[@AndrejKarpathy]

super :) skvele je to pocut a dakujem!

[@Alex-qz4nk]

That’s cool how Andrej explains right after releasing code

[@wvanginkel5572]

What an amazing video! Inspiring and so learningful. You (together with Jeremy Howard and Andrew Ng) are true gems for the AI community and master educators! Please keep the great videos coming and more than happy to pay!

[@rachadlakis1]

What an incredible journey you've shared in reproducing the GPT-2 (124M) from scratch! Your dedication and attention to detail are truly inspiring. Thank you for taking us through the entire process with such clarity and enthusiasm. Your commitment to sharing knowledge and resources is invaluable. Keep up the amazing work! 🌟

[@realhuman2545]

Least AI-generated comment

[@anandteerthrparvatikar5359]

You are doing great job and teaching which many top 50 universities combined couldn't manage in years

[@SpenserFL]

Thanks very much Andrej! Your videos are real gifts to the whole world.

[@ch0j468]

Seeing an Andrej upload in my recommended is like a mini holiday.

[@jstello]

Haven't been this excited about a KZbin video since makemore! Your videos are like an antidepressant. Such a joy to watch and follow and completely send contained. It's like having Mozart explain his art note by note

[@mjmrozek]

Hi Andrej, your content is incredibly inspiring and motivating. Watching you build something as complex as GPT-2 from scratch pushes me to improve my own tutorials, even if they're on simpler topics. Thanks for sharing your knowledge and for being such a positive influence on the community. Keep up the amazing work!

[@johnini]

Thanks to your previous videos, I watched this one at 1.5x speed. During the nearly 3-hour runtime, I found myself clapping alone in my room and even crying because of how amazing this content is! We are so lucky to have your next-level expertise captured in a KZbin video!

[@KapilSharma-lt4gm]

Thanks for this incredible resource. For anyone wondering about the transposes in the parameter copying from HF GPT2 model to implemented one. HF model uses nn.Conv1d for qkv projection while Andrej uses nn.Linear. The weights dimensions in Conv1d are transposed. Hence, we need to transpose some of these weights before copying them over to Andrej's model.

[@coolarun283]

To anyone looking for the possible cause of the error in the parameter count: It is due to the vocabulary size. In GPT-1 it was around 40000, whereas in GPT-2 the vocab_size is around 50000. So, with 40K we will get 117M and with 50K we will get 124M.

[@jimmy21584]

These videos are the best resource on modern neural networks I’ve found. Based on earlier videos, I built my own GPT with PyTorch. Now I’m doing a bunch of big projects based on what I’ve learnt. Thank you!

[@bycloudAI]

4hrs while being 4k quality is chef kiss

[@roeniss]

you see 4k quality option? I don't see it :(

[@zeweichu550]

This is an unbelievably high quality lecture! I always learn a ton of new things from Andrej Karpathy. Actually I believe if I have to rank the amount of knowledge I learned from a single person, Andrej would easily rank as #1.

[@sumanthmurthy1642]

The best part of Dr. Karpathy’s videos is that he explains “WHY” than just “HOW”. Moreover, he has the humility to say “I don’t know why” or “This is too long to read” (as rare as they are). I’m curious why you don’t use “assert” statements? The #1 thing I learnt from my mentor is the use of assert statements (makes me more cocky and confident) 😁🤣

[@frodo114]

Hi Andrej, just wanted to thank you. You are a truly inspiration. Thanks for all the effort you put in this videos and all the tremendous value they offer when being publicly spread

[@Ip_man22]

Thanks! Really appreciate the effort you put into making these high quality educational videos!

[@Jonathan-ru9zl]

We are living in great times, where geniuses like Karpathy offers their invaluable knowledge for free, and people are rewarding him with the sum of money they can afford 🎉

[@saurabhchalke]

Thank you ser, this is priceless. Felt sad that it had to end at some point. Please cover more topics like mech interp, fine-tuning, mixture models, etc.

[@hengry2]

You are the reason I got interested in neural networks, thank you for being a great teacher.

[@veluvishwa6915]

Hii bro, can i get roadmap for ML an deep learning please

[@jayhu6075]

He shares his knowledge with humanity without focusing on profit. My deep respect.

[@forrestye2194]

The spelled-out intro to neural networks and backpropagation: building micrograd -> Iron Man The spelled-out intro to language modeling: building makemore -> The Avengers Building makemore Part 2: MLP -> Avengers: Age of Ultron Building makemore Part 3: Activations & Gradients, BatchNorm -> Captain America: Civil War Building makemore Part 4: Becoming a Backprop Ninja -> Doctor Strange Building makemore Part 5: Building a WaveNet -> Guardians of the Galaxy Let's build GPT: from scratch, in code, spelled out -> Thor: Ragnarok State of GPT | BRK216HFS -> Avengers: Infinity War Let's build the GPT Tokenizer -> Ant-Man and the Wasp Let's reproduce GPT-2 (124M) -> Avengers: Endgame Long movies, series of consecutive movies, requires multiple viewings to grasp all the details. Thank you for enriching my weekend.👏

[@ManuelAlbarracin-sn3dp]

Truly amazing. Many thanks for the generosity with which you share your deep knowledge. I personally struggle following the code with its many details and idiosyncracies, but the "high-level intuition" comes across perfectly and it's deeply satisfying to get a glimpse of the nature of a technology that seems "indistinguishable from magic". Bravo Andrej.

[@PrabhjotSinghDhillo]

Thanks Andrej!

[@CarlosReyes-ku6ub]

Kind remainder that GOOD videos are NEVER too long

[@nickbrooks5684]

Thank you for contributing to Open Source models! And not just open weights!

[@user-rs4sg2tz6k]

I've never seen and experienced like you teaching me making me think i can learn everything with your teaching

[@ainnovation6967]

Thanks You Andrej!

[@StevenBBryantAuthor]

I thoroughly appreciate how you continue to give back to the community. This helps raise the water level for everyone on the way to building mastery! Thank you!

[@kazmi401]

I found GPT4 Here!

[@nchahine]

Having to work when you just want to watch Andrej's videos is like being invited to an open buffet but you're on a diet :)

[@africanbuffalo]

Thank you Andre for all these amazing in-depth, high quality tutorials!!!

[@hipotures]

Thanks for sharing your knowledge!

[@andreyashgaliev9372]

Currently, I'm just watching your videos. They makes me calm and happy. Hope to continue studying later this year.

[@mdrzazga]

Thanks for all these videos Andrej!

[@ir0nt0ad]

Thank you so much Andrej! Would be great to see you implement a modern DL recommendation system and/or cover the theory behind different regularization methods.

[@harshvardhanbansode685]

That would be so much fun and interesting

one of the best teachers I ever had

[@tijm6140]

Thanks for the video. I like your intuition for weight decay. Since the decay is proportional to the value, it encourages the contributions to the residual stream to be spread over more neurons.

[@Issam0hm]

Another piece of art 🔥

[@siddhanthbhattacharyya4206]

you're probably the best teacher for ML I've had, none of my professors had your level of clarity, or ability to express concepts with simplicity, I'm still watching your cs231n course. One day when I make it as a successful guy in ML/DL/AI I'd love to have an opportunity to meet you. thanks man.

[@barni_7762]

you are such an amazing teacher... it took me quite a while to acquire all the knowledge you communicated so concisely and understandably in this video from other sources

[@colinzhou9560]

OMG a 4hr movie!

[@kevinyang3298]

Andrej explains everything so clear, so logical and knows the "why" to every choice. You can't find a better tutorial than this. Brillant video!

[@tempestuousfabe]

Love your content, thanks!

[@user-mj2lm5fh1j]

I was about to implement the GPT2 starting today but had no idea where to start. This video is made for me. Thank you so much 🙌

[@NestorEscoto]

I can't believe we have access to this content for free from one of the brightest minds in the field! What a privilege; thanks, Andrej.

[@davidlyng2485]

This video is absolutely brilliant! Thank you so much Andrej for taking the time to share your knowledge with us!

[@mohammedjaddoa9783]

your explanation is really amazing, please keep fulfilling the gap >>>> build things from scratch

[@user-yw5me7pb2x]

the GOAT has returned!

[@vincentc1784]

Andrej you are doing so much for the community! Really want to express my gratitude here.

[@rolandrobertsons3069]

Thank you andrej! I have watch all your videos about gpt and learn a lot! As a poor college student, It's your videos that leading me to the road of llm.

[@yoloswaggins2161]

Better trilogy than lord of the rings

[@hassi007]

It’s very rare to find such high-quality, free resources!

[@zendr0]

Huge respect for Andrej🤗. Sharing knowledge for free is incredible.

[@przadka]

Thanks, Andrej! After watching all your videos, learning with you, and laughing at your jokes, I feel like we’re friends. Consider this tip my way of inviting you out 😊

[@webgpu]

YOU are Awesome, Andrej!! 🥂🤖

[@user-sz1iw4zi4y]

This is one of the best overviews I've seen not just on LLMs, but on the entire Deep Learning process. Thank you for going into so much detail, you're expertise really shows through your explanations. Would I watch another 4 hour video from you? Absolutely, any day!

[@mandilquioxtenlp1202]

Yayyyy thank you Andrej

[@fraserl]

Andrej I cannot thank you enough for these videos. Your ability to explain deep learning concepts in a simple manner is unparalleled. I’ve always been hugely interested in ML since my early teens. Now I’m currently doing my Masters project comparing Transformers to Mamba and xLSTMs and doing a PhD in deep learning next year. I’ve been following your work since I first heard about PixelCNN++ and have been inspired ever since. Keep up the great work!

[@user-yx8rn1ov1x]

Hi Andrej, what's the difference between this one and your "Let's build GPT" video? Which one should one learn first/which one is preferred?

[@muhammadharris4470]

Was wondering the same 😅

[@hengry2]

Use the "lets build" first, then this one; it goes over the understanding of it first, like the tokenization one as well.

[@natebrake4114]

Thank you Andrej for the lecture, enjoyed every minute of it! I especially found the discussion about torch compile to be helpful and interesting. I had been doing some experiments on how to speed up Mistral 7B inference in huggingface and was not seeing any improvement from torch compile. This is motivating for me to go back and try to understand what might be going wrong 😅. Thanks!

[@nitinnilesh]

The whole optimisation part in this video is something incredible. It is just impossible to find out these optimisation techniques on internet for DL models. Andrej doesn't have much research papers, but I believe that each one his videos is equivalent to a research paper having equal impact as of the original transformer paper.

[@mikestaub]

He is our modern-day Richard Feynman.

[@DailySFY]

Andrej thanks a lot!! Please don't mind the video length at all. It is really educational. Even if it is 100hrs or more I would watch it. Depth is much more important and is what gives me the joy of learning. Please continue doing the awesome work. Poor rn, but will def contribute in the future.

[@MichaelKleyn]

Legend