It’s rare to find such high-quality, free resources that make complex topics accessible and engaging!
@xspydazx2 ай бұрын
i think you will find out that some people know that they wil be taking this technology away from the public ....... same as google glass .... as soon as america find out they cannot sread dissemination of knowledge they will cuts all internet cables !
@Handlebrake22 ай бұрын
Jesus - that's some tip!
@bbcc29602 ай бұрын
❤
@unclecode2 ай бұрын
@@Handlebrake2 :)) Jesus - That's some 4hrs of brilliant content!
@fallingexistence2 ай бұрын
Dude this guy's net worth is $50M, you could've bought yourself 10 burritos at chipotle.
@carlosgermosen11032 ай бұрын
Do not ever look at how long your videos are. Your content is perfect and you should keep explaining things step by step. You are doing a great job. I believe you will be remembered in history as one of the pillars of AI.
@pin653712 ай бұрын
I think one cool thing with videos like this is once Google implements their AI into youtube anyone will be able to watch this and just start asking questions. I've been learning a lot by watching videos like this and just copying parts of the transcript into ChatGPT to ask questions when I dont understand something.
@sohambit93932 ай бұрын
I was excited by how long it was instead 😂😂
@quackwilliams59332 ай бұрын
@@pin65371 jesus christ what a thought...and Andrej just starts talking back to you, answering your exact questions...
@barackobama45522 ай бұрын
@@pin65371 What other videos do you recommend that have helped you?
@xspydazx2 ай бұрын
Yes a great sharer of the important information and implementation. As well as to let you know it's in your hands to make your own before the internet is closed down or restricted in your areas or your cable gets cut !! Great work ❤
@Doggi2dog2 ай бұрын
My life is simple; Andrej drops GPT-2 The Movie, I watch.
@AndrejKarpathy2 ай бұрын
"GPT-2 The Movie" 😅
@DamianReloaded2 ай бұрын
The movie and the sequel. I had to force myself to stop watching after I realized an hour had passed.
@asatorftw2 ай бұрын
@@AndrejKarpathy The sequel "GPT The Movie" will be a old Hong Kong style "martial arts" movie about GPT getting beaten up by the Loss Function, then entering his training phase with Gradient Descent Sensei and the final showdown vs the big evaluation boss.
@georgiandanciu34822 ай бұрын
You know you gotta bring the popcorn
@AbhigyanKeshav1692 ай бұрын
Andrej poster video on your biography@@AndrejKarpathy
@mohamedalansary25422 ай бұрын
The fact this video is free is incredible.
@manojr4402 ай бұрын
Thanks for spreading the knowledge! Happy to see a 4hr workout session 😅
@AndrejKarpathy2 ай бұрын
:) Workout 🏋️♂️ is very much the right way to think about it imo!
@jeffrey56022 ай бұрын
Heavy weight training
@MsLe20162 ай бұрын
more like a 3-day session to me, but I'm happy :)
@neilamrathod1222 ай бұрын
Sorry , i love your videos and what you doing for me. I couldn't attend Stanford or get into openai but learning from you is blessing to me.. i would pay you back 100times in coming years. And i was watching your git repository last two months, i could see many git code push in private ,but i was confused what he is working on.. this is he was working on. To provide quality pratical knowledge to us all on youtube.
@shutup12092 ай бұрын
Hey, the webside for the GPT2 is down, is there anyway to dowload it ?
@neilamrathod1222 ай бұрын
Sorry i will have look up to it ..i will do today night will reply to you after that @@shutup1209
@DingLi-hw4ul2 ай бұрын
hf@@shutup1209
@hobbytan68412 ай бұрын
@@shutup1209 you should follow the video and make it from scratch! 😀
@dinorossi66112 ай бұрын
Those are generous tips :). I wanna learn basics so I can understand this first.
@C0D3633K2 ай бұрын
Andrej is doing himself what OpenAi was supposed to do in the early days - make AI open. Thank you, Andrej!
@zachyamaoka79162 ай бұрын
Thanks Andrej! You have taught me everything I know about the theory and practice of neural networks, starting with CS231n till now. I love how you explain things starting with simple examples to build intuitions (template matching for CV, bigram/table look up for sequence modelling), and then build to state of the art. Your lessons have had a profound impact on my learning, and I can imagine there are 1000s of engineers out there just like me.
@charlesm8352 ай бұрын
You're a legend! Andrej will light up when he see's this 🤍
@AndrejKarpathyАй бұрын
you're too thankful ty! :)
@anthonycho63442 ай бұрын
My anterior mid-cingulate cortex is getting bigger just watching this video because it’s hard! Thank you for your lessons, master Kaparthy.
@user-oq7ju6vp7j2 ай бұрын
It's such a privilege to watch high-quality content from leading experts for free
@biboyog2 ай бұрын
true
@unclecode2 ай бұрын
Thanks! 4 hours of decoding a "Decoder-Transformer", Kudos and appreciate your existence in this field.
@sabareesh_422 ай бұрын
Thanks!
@broccoli3222 ай бұрын
Perfect to watch while on a plane
@Ishaheennabi2 ай бұрын
and then running gpt 2 on planes computer
@waytolegacy2 ай бұрын
This guy is "the one" in the industry, who has helped me understand the LLMs. I respectfully love this man. Hats off.
@marcotuc-ilmarinaio9242 ай бұрын
Thank you Andrej! from zero to hero boosted my professional career!
@tanaysood2 ай бұрын
Thanks AK, appreciate you sharing your knowledge with the world!
@chenmarkson74132 ай бұрын
I am an undergraduate student. This is the lost lecture that professors never touched upon but absolutely crucial, thank you!! I especially love how you start from the basics for so many notions, and I really learned a lot.
@ppyogesh73942 ай бұрын
Which year in you are and which country
@chenmarkson74132 ай бұрын
@@ppyogesh7394 I am at the University of Toronto, going to the third year this September
@Khobalt6642 ай бұрын
You are the Excalibur of cutting through the hype. Thank you so much. Your ethics are inspiring, and your educational materials priceless.
@niclaswustenbecker89022 ай бұрын
I think your teaching projects may be the most impactful ever existed. First CS231n the course that inspired a generation of students to pursue Deep Learning, then micrograd and nanogpt that gives us the power to recreate billion dollar research for basically free. Not to mention that many companies like tiny grad and suno were inspired by your projects. Thanks for sharing your knowledge in such a clean and elegant way!
@JT-mr3db2 ай бұрын
The intellectual generosity of this man is of the highest standard.
@tothespace21222 ай бұрын
These kinds of videos is what the world needs more. Long form content with real time thinking. You see the master at his craft in real time. So much to learn from this!
@rainwang772 ай бұрын
Hello Andrej, thank you so much for the sharing and effort! Really appreciate it!
@dylan_curious2 ай бұрын
I like when you add comments/metaphors about your intuition for how and why it works. Thanks you.
@pxbroccoli2 ай бұрын
Checkout this man here, he got the best Ai news
@generallifing2 ай бұрын
We have one of the best people on the planet to walk you through it step by step (I haven't seen it yet, but I believe so). I am eager to learn this and want to master it. Thank you, thank you, and thank you very much!
@somdubey54362 ай бұрын
The longer your videos are, the better it is for humanity. I think you are such a wonderful person and providing this stuff for everyone for free, can't thank you enough.
@Themojii2 ай бұрын
I've learned a lot from your Neural Network video playlist. Thank you
@themenon2 ай бұрын
Many Thanks to Andrej for making this tutorial available to everyone! I have never seen a clearer explanation of a nn before stumbling upon this zero to hero series. This will help all the people articulate the inner workings of neural net and help people understand deeper concepts, that is hard to understand. Looking forward to learning more with Andrej!
@gromeronaranjo2 күн бұрын
I rarely comment on videos, but I had to here I had to. Your in-depth high-quaity resources are something to talk about. You make very complicated topics easy and engaging, your provide the knowledge for anyone to learn these highly-regarded concepts. Fruthermore, you are truly advancing the general knowledge of the public by providing these powerful videos. I would just like to express my gratitude for your videos, and how they really are making a positive impact. Thank you for dedicating many hours of work to upload these videos.
@bmatichuk2 ай бұрын
Your guidance is inspiring.
@JC-ys2ch2 ай бұрын
Super useful! Looking forward to any in-depth "anatomy" video on Mamba Architecture as well.
@divandrey-u3q2 ай бұрын
Yeah, mamba would be interesting to see... I hope Andrej hears us
@forrestye21942 ай бұрын
Finally, finished watching such a long video. Thank Andrej for sharing so many details of your knowledge. Like your teaching style so much since Tesla AI day. You are the best AI teacher!
@qiuchenguo27882 ай бұрын
Simply the best deep learning and LLM series online! Please keep making more videos and I'd love to be part of the journey!
@souravzzz2 ай бұрын
🤗What an absolutely fantastic explanation! Every minute is filled with nuggets of deep insights!
@IgorTsvetkov2 ай бұрын
Thanks for your Zero-to-hero series!
@AndrejKarpathyАй бұрын
wow you're very thankful ty! :)
@Ibbysz2 ай бұрын
Andrej in a few years: Lets reproduce GPT-5 (124T)
@Alconno2 ай бұрын
we can only hope
@Person-hb3dv2 ай бұрын
and it's gonna cost 3$ to reproduce
@Katatonya2 ай бұрын
I wonder how long will it take for even GPT4 to be trainable on our own rigs. Nvidia said that in 8 years, computation will be reduced 350x times. (i.e. a gpu you will buy in 8 years, will be 350x better at training, if I understood correctly). Though is this enough? And 8 years is a loooong time in the present AI world.
@Person-hb3dv2 ай бұрын
@@Katatonya I think it's going to happen even faster. with the current rate of advancements, in 8 years training something like GPT-4 locally would be trivial to say the least. but i'm just guessing. who knows what will happen.
@Katatonya2 ай бұрын
@@Person-hb3dv I think it purely depends on if the models will get much much more efficient and cheaper to run. They most likely will. Hardware-wise though, Nvidia didn't guess what will happen in 8 years, they know for sure as they have to plan in advance.
@palimondo2 ай бұрын
Tieto videá sú naozaj skvelé. Snažil som sa hlbšie pochopiť ako fungujú LLM čítaním vedeckých publikácií. Keďže som vyštudoval odbor softvérové inžinierstvo a nie umelá inteligencia a tiež som neabsolvoval postgraduálne štúdium, “papers” sa mi, kvôli medzerám v mojich znalostiach, čítajú veľmi ťažko. Keď mi ukazuješ kód, krok za krokom, všetko do dnes zapadá a dáva mi to zmyslel. Asi nebudem na LambdaLabs node robiť pre-training vlastného modelu, ale ten pocit, že s tým čo si ma tu naučil by som to teoreticky dokázal je neskutočne silný. Si perfektný učiteľ. Ďakujem ti, Andrej!
@AndrejKarpathy2 ай бұрын
super :) skvele je to pocut a dakujem!
@Alex-qz4nk2 ай бұрын
That’s cool how Andrej explains right after releasing code
@wvanginkel55722 ай бұрын
What an amazing video! Inspiring and so learningful. You (together with Jeremy Howard and Andrew Ng) are true gems for the AI community and master educators! Please keep the great videos coming and more than happy to pay!
@rachadlakis12 ай бұрын
What an incredible journey you've shared in reproducing the GPT-2 (124M) from scratch! Your dedication and attention to detail are truly inspiring. Thank you for taking us through the entire process with such clarity and enthusiasm. Your commitment to sharing knowledge and resources is invaluable. Keep up the amazing work! 🌟
@realhuman25452 ай бұрын
Least AI-generated comment
@anandteerthrparvatikar53592 ай бұрын
You are doing great job and teaching which many top 50 universities combined couldn't manage in years
@SpenserFL2 ай бұрын
Thanks very much Andrej! Your videos are real gifts to the whole world.
@ch0j4682 ай бұрын
Seeing an Andrej upload in my recommended is like a mini holiday.
@jstello2 ай бұрын
Haven't been this excited about a KZbin video since makemore! Your videos are like an antidepressant. Such a joy to watch and follow and completely send contained. It's like having Mozart explain his art note by note
@mjmrozek4 күн бұрын
Hi Andrej, your content is incredibly inspiring and motivating. Watching you build something as complex as GPT-2 from scratch pushes me to improve my own tutorials, even if they're on simpler topics. Thanks for sharing your knowledge and for being such a positive influence on the community. Keep up the amazing work!
@johnini2 ай бұрын
Thanks to your previous videos, I watched this one at 1.5x speed. During the nearly 3-hour runtime, I found myself clapping alone in my room and even crying because of how amazing this content is! We are so lucky to have your next-level expertise captured in a KZbin video!
@KapilSharma-lt4gmАй бұрын
Thanks for this incredible resource. For anyone wondering about the transposes in the parameter copying from HF GPT2 model to implemented one. HF model uses nn.Conv1d for qkv projection while Andrej uses nn.Linear. The weights dimensions in Conv1d are transposed. Hence, we need to transpose some of these weights before copying them over to Andrej's model.
@coolarun2832 ай бұрын
To anyone looking for the possible cause of the error in the parameter count: It is due to the vocabulary size. In GPT-1 it was around 40000, whereas in GPT-2 the vocab_size is around 50000. So, with 40K we will get 117M and with 50K we will get 124M.
@jimmy215842 ай бұрын
These videos are the best resource on modern neural networks I’ve found. Based on earlier videos, I built my own GPT with PyTorch. Now I’m doing a bunch of big projects based on what I’ve learnt. Thank you!
@bycloudAI2 ай бұрын
4hrs while being 4k quality is chef kiss
@roeniss2 ай бұрын
you see 4k quality option? I don't see it :(
@zeweichu550Ай бұрын
This is an unbelievably high quality lecture! I always learn a ton of new things from Andrej Karpathy. Actually I believe if I have to rank the amount of knowledge I learned from a single person, Andrej would easily rank as #1.
@sumanthmurthy1642Ай бұрын
The best part of Dr. Karpathy’s videos is that he explains “WHY” than just “HOW”. Moreover, he has the humility to say “I don’t know why” or “This is too long to read” (as rare as they are). I’m curious why you don’t use “assert” statements? The #1 thing I learnt from my mentor is the use of assert statements (makes me more cocky and confident) 😁🤣
@frodo1142 ай бұрын
Hi Andrej, just wanted to thank you. You are a truly inspiration. Thanks for all the effort you put in this videos and all the tremendous value they offer when being publicly spread
@Ip_man22Ай бұрын
Thanks! Really appreciate the effort you put into making these high quality educational videos!
@Jonathan-ru9zl2 ай бұрын
We are living in great times, where geniuses like Karpathy offers their invaluable knowledge for free, and people are rewarding him with the sum of money they can afford 🎉
@saurabhchalke2 ай бұрын
Thank you ser, this is priceless. Felt sad that it had to end at some point. Please cover more topics like mech interp, fine-tuning, mixture models, etc.
@hengry22 ай бұрын
You are the reason I got interested in neural networks, thank you for being a great teacher.
@veluvishwa69152 ай бұрын
Hii bro, can i get roadmap for ML an deep learning please
@jayhu60752 ай бұрын
He shares his knowledge with humanity without focusing on profit. My deep respect.
@forrestye21942 ай бұрын
The spelled-out intro to neural networks and backpropagation: building micrograd -> Iron Man The spelled-out intro to language modeling: building makemore -> The Avengers Building makemore Part 2: MLP -> Avengers: Age of Ultron Building makemore Part 3: Activations & Gradients, BatchNorm -> Captain America: Civil War Building makemore Part 4: Becoming a Backprop Ninja -> Doctor Strange Building makemore Part 5: Building a WaveNet -> Guardians of the Galaxy Let's build GPT: from scratch, in code, spelled out -> Thor: Ragnarok State of GPT | BRK216HFS -> Avengers: Infinity War Let's build the GPT Tokenizer -> Ant-Man and the Wasp Let's reproduce GPT-2 (124M) -> Avengers: Endgame Long movies, series of consecutive movies, requires multiple viewings to grasp all the details. Thank you for enriching my weekend.👏
@ManuelAlbarracin-sn3dp2 ай бұрын
Truly amazing. Many thanks for the generosity with which you share your deep knowledge. I personally struggle following the code with its many details and idiosyncracies, but the "high-level intuition" comes across perfectly and it's deeply satisfying to get a glimpse of the nature of a technology that seems "indistinguishable from magic". Bravo Andrej.
@PrabhjotSinghDhillo2 ай бұрын
Thanks Andrej!
@CarlosReyes-ku6ub2 ай бұрын
Kind remainder that GOOD videos are NEVER too long
@nickbrooks56842 ай бұрын
Thank you for contributing to Open Source models! And not just open weights!
@user-rs4sg2tz6kАй бұрын
I've never seen and experienced like you teaching me making me think i can learn everything with your teaching
@ainnovation69672 ай бұрын
Thanks You Andrej!
@StevenBBryantAuthor2 ай бұрын
I thoroughly appreciate how you continue to give back to the community. This helps raise the water level for everyone on the way to building mastery! Thank you!
@kazmi4012 ай бұрын
I found GPT4 Here!
@nchahine2 ай бұрын
Having to work when you just want to watch Andrej's videos is like being invited to an open buffet but you're on a diet :)
@africanbuffalo2 ай бұрын
Thank you Andre for all these amazing in-depth, high quality tutorials!!!
@hipotures2 ай бұрын
Thanks for sharing your knowledge!
@andreyashgaliev93722 ай бұрын
Currently, I'm just watching your videos. They makes me calm and happy. Hope to continue studying later this year.
@mdrzazga2 ай бұрын
Thanks for all these videos Andrej!
@ir0nt0ad2 ай бұрын
Thank you so much Andrej! Would be great to see you implement a modern DL recommendation system and/or cover the theory behind different regularization methods.
@harshvardhanbansode685Ай бұрын
That would be so much fun and interesting
27 күн бұрын
one of the best teachers I ever had
@tijm61402 ай бұрын
Thanks for the video. I like your intuition for weight decay. Since the decay is proportional to the value, it encourages the contributions to the residual stream to be spread over more neurons.
@Issam0hm2 ай бұрын
Another piece of art 🔥
@siddhanthbhattacharyya4206Ай бұрын
you're probably the best teacher for ML I've had, none of my professors had your level of clarity, or ability to express concepts with simplicity, I'm still watching your cs231n course. One day when I make it as a successful guy in ML/DL/AI I'd love to have an opportunity to meet you. thanks man.
@barni_7762Ай бұрын
you are such an amazing teacher... it took me quite a while to acquire all the knowledge you communicated so concisely and understandably in this video from other sources
@colinzhou95602 ай бұрын
OMG a 4hr movie!
@kevinyang32982 ай бұрын
Andrej explains everything so clear, so logical and knows the "why" to every choice. You can't find a better tutorial than this. Brillant video!
@tempestuousfabe2 ай бұрын
Love your content, thanks!
@user-mj2lm5fh1j2 ай бұрын
I was about to implement the GPT2 starting today but had no idea where to start. This video is made for me. Thank you so much 🙌
@NestorEscoto2 ай бұрын
I can't believe we have access to this content for free from one of the brightest minds in the field! What a privilege; thanks, Andrej.
@davidlyng2485Ай бұрын
This video is absolutely brilliant! Thank you so much Andrej for taking the time to share your knowledge with us!
@mohammedjaddoa97832 ай бұрын
your explanation is really amazing, please keep fulfilling the gap >>>> build things from scratch
@user-yw5me7pb2x2 ай бұрын
the GOAT has returned!
@vincentc17842 ай бұрын
Andrej you are doing so much for the community! Really want to express my gratitude here.
@rolandrobertsons30699 күн бұрын
Thank you andrej! I have watch all your videos about gpt and learn a lot! As a poor college student, It's your videos that leading me to the road of llm.
@yoloswaggins21612 ай бұрын
Better trilogy than lord of the rings
@hassi007Ай бұрын
It’s very rare to find such high-quality, free resources!
@zendr02 ай бұрын
Huge respect for Andrej🤗. Sharing knowledge for free is incredible.
@przadka8 күн бұрын
Thanks, Andrej! After watching all your videos, learning with you, and laughing at your jokes, I feel like we’re friends. Consider this tip my way of inviting you out 😊
@webgpu2 ай бұрын
YOU are Awesome, Andrej!! 🥂🤖
@user-sz1iw4zi4y2 ай бұрын
This is one of the best overviews I've seen not just on LLMs, but on the entire Deep Learning process. Thank you for going into so much detail, you're expertise really shows through your explanations. Would I watch another 4 hour video from you? Absolutely, any day!
@mandilquioxtenlp12022 ай бұрын
Yayyyy thank you Andrej
@fraserl2 ай бұрын
Andrej I cannot thank you enough for these videos. Your ability to explain deep learning concepts in a simple manner is unparalleled. I’ve always been hugely interested in ML since my early teens. Now I’m currently doing my Masters project comparing Transformers to Mamba and xLSTMs and doing a PhD in deep learning next year. I’ve been following your work since I first heard about PixelCNN++ and have been inspired ever since. Keep up the great work!
@user-yx8rn1ov1x2 ай бұрын
Hi Andrej, what's the difference between this one and your "Let's build GPT" video? Which one should one learn first/which one is preferred?
@muhammadharris44702 ай бұрын
Was wondering the same 😅
@hengry22 ай бұрын
Use the "lets build" first, then this one; it goes over the understanding of it first, like the tokenization one as well.
@natebrake41142 ай бұрын
Thank you Andrej for the lecture, enjoyed every minute of it! I especially found the discussion about torch compile to be helpful and interesting. I had been doing some experiments on how to speed up Mistral 7B inference in huggingface and was not seeing any improvement from torch compile. This is motivating for me to go back and try to understand what might be going wrong 😅. Thanks!
@nitinnilesh2 ай бұрын
The whole optimisation part in this video is something incredible. It is just impossible to find out these optimisation techniques on internet for DL models. Andrej doesn't have much research papers, but I believe that each one his videos is equivalent to a research paper having equal impact as of the original transformer paper.
@mikestaub2 ай бұрын
He is our modern-day Richard Feynman.
@DailySFY2 ай бұрын
Andrej thanks a lot!! Please don't mind the video length at all. It is really educational. Even if it is 100hrs or more I would watch it. Depth is much more important and is what gives me the joy of learning. Please continue doing the awesome work. Poor rn, but will def contribute in the future.