How to Build an LLM from Scratch | An Overview

👉 Need help with AI? Reach out: shawhintalebi.com/
This is the 6th video in a series on using large language models (LLMs) in practice. Here, I review key aspects of developing a foundation LLM based on the development of models such as GPT-3, Llama, Falcon, and beyond.
More Resources:
👉 Series Playlist: • Large Language Models ...
📰 Read more: towardsdatascience.com/how-to...
[1] BloombergGPT: arxiv.org/pdf/2303.17564.pdf
[2] Llama 2: ai.meta.com/research/publicat...
[3] LLM Energy Costs: www.statista.com/statistics/1...
[4] arXiv:2005.14165 [cs.CL]
[5] Falcon 180b Blog: huggingface.co/blog/falcon-180b
[6] arXiv:2101.00027 [cs.CL]
[7] Alpaca Repo: github.com/gururise/AlpacaDat...
[8] arXiv:2303.18223 [cs.CL]
[9] arXiv:2112.11446 [cs.CL]
[10] arXiv:1508.07909 [cs.CL]
[11] SentencePience: github.com/google/sentencepie...
[12] Tokenizers Doc: huggingface.co/docs/tokenizer...
[13] arXiv:1706.03762 [cs.CL]
[14] Andrej Karpathy Lecture: • Let's build GPT: from ...
[15] Hugging Face NLP Course: huggingface.co/learn/nlp-cour...
[16] arXiv:1810.04805 [cs.CL]
[17] arXiv:1910.13461 [cs.CL]
[18] arXiv:1603.05027 [cs.CV]
[19] arXiv:1607.06450 [stat.ML]
[20] arXiv:1803.02155 [cs.CL]
[21] arXiv:2203.15556 [cs.CL]
[22] Trained with Mixed Precision Nvidia: docs.nvidia.com/deeplearning/...
[23] DeepSpeed Doc: www.deepspeed.ai/training/
[24] paperswithcode.com/method/wei...
[25] towardsdatascience.com/what-i...
[26] arXiv:2001.08361 [cs.LG]
[27] arXiv:1803.05457 [cs.AI]
[28] arXiv:1905.07830 [cs.CL]
[29] arXiv:2009.03300 [cs.CY]
[30] arXiv:2109.07958 [cs.CL]
[31] huggingface.co/blog/evaluatin...
[32] www.cs.toronto.edu/~hinton/ab...
--
Book a call: calendly.com/shawhintalebi
Socials
/ shawhin
/ shawhintalebi
/ shawhint
/ shawhintalebi
The Data Entrepreneurs
🎥 KZbin: / @thedataentrepreneurs
👉 Discord: / discord
📰 Medium: / the-data
📅 Events: lu.ma/tde
🗞️ Newsletter: the-data-entrepreneurs.ck.pag...
Support ❤️
www.buymeacoffee.com/shawhint
Intro - 0:00
How much does it cost? - 1:30
4 Key Steps - 3:55
Step 1: Data Curation - 4:19
1.1: Data Sources - 5:31
1.2: Data Diversity - 7:45
1.3: Data Preparation - 9:06
Step 2: Model Architecture (Transformers) - 13:17
2.1: 3 Types of Transformers - 15:13
2.2: Other Design Choices - 18:27
2.3: How big do I make it? - 22:45
Step 3: Training at Scale - 24:20
3.1: Training Stability - 26:52
3.2: Hyperparameters - 28:06
Step 4: Evaluation - 29:14
4.1: Multiple-choice Tasks - 30:22
4.2: Open-ended Tasks - 32:59
What's next? - 34:31

[@ShawhinTalebi]

[Correction at 15:00]: words on vertical axis are backward. It should go "I hit ball with baseball bat" from top to bottom not bottom to top. 👉Need help with AI? Reach out: shawhintalebi.com/ 🎥Series playlist: kzbin.info/aero/PLz-ep5RbHosU2hnz5ejezwaYpdMutMVB0 -- References [1] BloombergGPT: arxiv.org/pdf/2303.17564.pdf [2] Llama 2: ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/ [3] LLM Energy Costs: www.statista.com/statistics/1384401/energy-use-when-training-llm-models/ [4] arXiv:2005.14165 [cs.CL] [5] Falcon 180b Blog: huggingface.co/blog/falcon-180b [6] arXiv:2101.00027 [cs.CL] [7] Alpaca Repo: github.com/gururise/AlpacaDataCleaned [8] arXiv:2303.18223 [cs.CL] [9] arXiv:2112.11446 [cs.CL] [10] arXiv:1508.07909 [cs.CL] [11] SentencePience: github.com/google/sentencepiece/tree/master [12] Tokenizers Doc: huggingface.co/docs/tokenizers/quicktour [13] arXiv:1706.03762 [cs.CL] [14] Andrej Karpathy Lecture: kzbin.info/www/bejne/oXTGaXmjesdkpLs [15] Hugging Face NLP Course: huggingface.co/learn/nlp-course/chapter1/7?fw=pt [16] arXiv:1810.04805 [cs.CL] [17] arXiv:1910.13461 [cs.CL] [18] arXiv:1603.05027 [cs.CV] [19] arXiv:1607.06450 [stat.ML] [20] arXiv:1803.02155 [cs.CL] [21] arXiv:2203.15556 [cs.CL] [22] Trained with Mixed Precision Nvidia: docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html [23] DeepSpeed Doc: www.deepspeed.ai/training/ [24] paperswithcode.com/method/weight-decay [25] towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48 [26] arXiv:2001.08361 [cs.LG] [27] arXiv:1803.05457 [cs.AI] [28] arXiv:1905.07830 [cs.CL] [29] arXiv:2009.03300 [cs.CY] [30] arXiv:2109.07958 [cs.CL] [31] huggingface.co/blog/evaluating-mmlu-leaderboard [32] www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf

[@amortalbeing]

thanks a lot for the refs , Shahin Jan ❤ keep up the great job 👍

[@seanwilner]

This is a about as perfect a coverage of this topic as I could imagine. I'm a researcher with a PhD in NLP who trains LLMs from scratch for a living and often find myself in need of communicating the process in a way that's digestible to a broad audience without back and forth question answering, so I'm thrilled to have found your piece! As an aside, I think the token order on the y-axis of the attention mask for decoders on slide 10 is reversed

[@ShawhinTalebi]

Thanks Sean! It's always a challenge to convey technical information in a way that both the researcher and general audience can get value from. So your approval means a lot :) Thanks for pointing the out. The blog article has a corrected version: medium.com/towards-data-science/how-to-build-an-llm-from-scratch-8c477768f1f9?sk=18c351c5cae9ac89df682dd14736a9f3

[@AritraDutta-tz4je]

Sir can you tell me how are you training your llms?

[@xxcusme]

most of people watching this video is through certain prompt of how to build LLM and these people is the rest 10% by your logic, the makers & inventors

[@shilpyjain6147]

Hey Shaw - Thank you for coming up with this extensive video on building LLM from Scratch, it certainly gives a fair idea on, how some of the existing LLMs were created !

[@dauntlessRx]

This is literally the perfect explanation for this topic. Thank you so much.

[@barclayiversen376]

Pretty rare that I actually sit through an entire 30+ minute video on youtube. Well done.

[@racunars]

All the series on using large language models (LLMs) are really very helpful. This 6th article, really helps me to understand in a nutshell the transformer architecture. Thank you. 👏

[@ShawhinTalebi]

Glad it was helpful!

[@GBangalore]

Thank you so much for putting these videos together and this one in particular. This is such a broad and complex topic and you have managed to make it as thorough as possible in 30ish minute😮 timeframe which I thought was almost impossible.

[@ShawhinTalebi]

My pleasure, glad it was informative yet concise :)

[@tehreemsyed8621]

This is such a fantastic video on building LLMs from scratch. I'll watch it repeatedly to implement it for a time-series use case. Thank you so much!!

[@LudovicCarceles]

"Garbage in, garbage out" is also applicable to our brain. Your videos are certainly high quality inputs.

[@ShawhinTalebi]

Glad it was helpful :)

[@asha328]

One of the best videos explaining the process and cost to build LLM🎉.

[@theunconventionalenglishman]

This is excellent - thanks for putting this together and taking the time to explain things so clearly!

[@ethanchong1026]

Thanks for putting together this short video. I enjoy learning this subject from you.

[@ShawhinTalebi]

Thanks Ethan, glad you enjoyed it!

[@bradstudio]

This was a very thorough introduction to LLMs and answered many questions I had. Thank you.

[@ShawhinTalebi]

Great to hear, glad it was helpful :)

[@goldholder8131]

This is the most comprehensive and well rounded presentation I've ever seen in my life, topic aside. xD Bravo, good Sir.

[@ShawhinTalebi]

Thanks so much! Glad you liked it :)

[@mater5930]

I became interested in creating an LLM and this is the first video I opened. I am so greatful for it because I see I will never be able to do it on my own. I don't jave the money of resources. Thank you for the high level overview.

[@qicao7769]

Best and most efficient video about the basic of LLM!!!! I think I have saved 10h for reading. Thanks!

[@ShawhinTalebi]

Love to hear it! Glad it helped

[@EigenA]

Great channel, 3rd video in. You earned a sub. Thank you!

[@lihanou]

clicked with low expectation, but wow what a gem. Great clarity with just the right amount of depth for beginners and intermediate learners.

[@ShawhinTalebi]

Glad it was helpful :)

[@robwarner1858]

Amazing video. Lost me through a fair bit, but I came away understanding more than I ever have on the subject. Thank you.

[@ShawhinTalebi]

Glad it was helpful :)

[@aldotanca9430]

Thoroughly researched and referenced, clear explanations inclusive of examples. I will watch it again to take notes. Thanks so much!

[@ShawhinTalebi]

Great to hear! Feel free to reach out with any questions or suggestions for future content :)

[@aldotanca9430]

Thanks! I would have plenty of questions actually, but they are probably a bit too specific to make for a generally relevant video. I am exploring options for a few non-profit projects related to musical education and research. They need to integrate large bodies of text and produce precise referencing to what comes from where, so I was naively toying with the idea to perhaps produce a base model partially trained on the actual text in question. Which, I understood from the video, is a non-starter. So I will look into fine-tuning, RAG and prompt engineering. I suspect I will spend quite a lot of time watching your convent, given you covered quite a lot. I also learned quite a bit more from this specific video. Right now I am studying the basics, including a bit of the math involved, and it is a bit slow going, so I am quite grateful :)

[@ShawhinTalebi]

@@aldotanca9430 That sounds like a really cool use case (I've been a musician for over 14 years)! If you want to chat about more specific questions feel free to set up some office hours: calendly.com/shawhintalebi/office-hours

[@aldotanca9430]

@@ShawhinTalebi That's very generous of you! I will book a slot, would love to chat, I think it would help me immensely to rule out blind alleys and at least get a well informed idea of what is feasible to attempt. I did notice the congas, piano and Hanon lurking in the background, so I suspected the topic will be interesting to you. It is about historical research, but it is also very applicable and creative for improvvisation. Perhaps I can compile a very short list of interesting resources, in case you want to check it out at some point for musical reasons :)

[@Hello_kitty_34892]

Your voice is relaxing.. I love that you don't speak super fast like most tech bros... And you seem relaxed about the content rather than having this "in a rush" energy. def would watch you explain most things LLM and AI! Thanks for the content.

[@ShawhinTalebi]

Thanks for the feedback. More AI/LLM content to come!

[@chrstfer2452]

That was simply incredible, how the heck does it have under 5k views. Literal in-script citations, not even cards but vocal mentions!! Holy shit im gonna share this channel with all my LLM enamored buddies

[@ShawhinTalebi]

Thanks, I'm glad it was helpful. You're referrals are greatly appreciated 😁

[@sinan325]

I am not a programmer or now anything about programming or LLMs but I find this topic fascinating. Thank you for your videos and sharing your knowledge.

[@ShawhinTalebi]

Happy to help! I hope they were valuable.

[@mujeebrahman5282]

I am typing this after watching half of the video as I am already amazed with the clarity of explanation. exceptional.

[@ShawhinTalebi]

Thanks, hope the 2nd half didn't disappoint!

[@joedigiovanni8758]

Great job demystifying what is happening under the hood of these LLMs

[@ShawhinTalebi]

Happy to help!

[@ifycadeau]

Love these videos! Keep it up Shaw!

[@ShawhinTalebi]

Thank you 🙏

[@shih-shengchang19]

Thanks for your video; it's awesome. You explain everything very clearly and with good examples.

[@ShawhinTalebi]

Thanks for the feedback, glad it was clear :)

[@aalaptube]

Very good information! Answered a lot of core questions I had.

[@ShawhinTalebi]

Glad it was helpful!

[@randomforest_dev]

Awesome Video! Thanks.

[@rezNezami]

excellent job Shawhin. Merci.

[@ShawhinTalebi]

Thanks Reza, glad you liked it!

[@funnymono]

Exceptional material

[@ares106]

thank you, this is infinitely more enjoyable for me than reading a paper.

[@ShawhinTalebi]

😂😂 I’m glad you liked it!

[@fab_spaceinvaders]

second this, keep the good work flowing all around 🎉 🙏

[@romantolstykh7488]

Great video!

[@Kiririn]

thank you for making this

[@ShawhinTalebi]

Happy to help!

[@MegaBenschannel]

Thanks for the great and pack expose. 😀

[@ShawhinTalebi]

Glad it was helpful 😁

[@LezzGoPlaces]

Brilliant!

[@user-cu7vl1jj9v]

Very very good!!

[@DigsWigs2022]

Great explanation. I will have to watch it a few times to have a basic understanding 😂

[@YohannesAssefa-wk5oo]

thankyou bro for your help

[@ShawhinTalebi]

Happy to help!

[@SamChughtai]

Thanks, Shaw!! Great video and excellent data, would love to be your mentee, sir!!

[@ShawhinTalebi]

Thank you for your generosity! I don't currently do any formal mentorship, but I try to give away all my secrets on KZbin and Medium :) Feel free to share any suggestions for future content.

[@akramsystems]

This is Gold

[@ShawhinTalebi]

Glad it's helpful :)

[@arpadbrooks5317]

very informative thx

[@ShawhinTalebi]

Happy to help!

[@wilfredomartel7781]

🎉❤❤❤amazing video

[@ShawhinTalebi]

Thank you 🙏

[@philtoa334]

Nice.

[@gRosh08]

Cool.

[@echofloripa]

Wow, what a great content, thanks for that!! In LLM Fine-tuning, is there also a suggestion table between number of trainable parameters and tokens used (dataset size)?

[@ShawhinTalebi]

That’s a great question. While I haven’t come across such a table, good rule of thumb is 1k-10k examples depending on the use case.

[@echofloripa]

@@ShawhinTalebi thanks for the quick reply! What about the number of trainable parameter, should we worry about that? What if my number of examples is smaller than that let's say a 100 to 200?

[@ShawhinTalebi]

​@@echofloripa IMO you've got to work with what you've got. I've heard some people get sufficient performance from just 100-200 examples, but it ultimately comes down to what is acceptable for that particular use case. It might be worth a try. Hope that helps!

[@siyufan1084]

Arrived right on time! The quality of the video is consistently excellent as always

[@ShawhinTalebi]

Great to hear! I'm glad they are helpful :)

[@PorterHarris]

Great content Shaw! Next step Im having troubles figuring out, is there a way to run locally an existing GPT and do prompt engineering or model fine-tuning on it with my own training data?

[@ShawhinTalebi]

Thanks! While this depends on your local machine specs, the short answer is yes! My next video will actually walk through how to do this using an approach called QLoRA.

[@vijayakashallenki7275]

Waiting for the complete AI-ML playlist! sir please

[@mitdasondi2171]

watching this right before my interview.

[@ShawhinTalebi]

Good luck!

[@mitdasondi2171]

@@ShawhinTalebi cleared 1st round, now its on Thursday, i hope your luck brings me my dream job ❤️

[@rajez.s7157]

Can Ray clusters be used here for mutiple GPUs training of LLMs?

[@ShawhinTalebi]

I haven't used Ray clusters before, but skimming their website it seems like it was specifically made for ML workloads.

[@abcoflife6420]

Thank you so much for rich information, my target is to DIY one from scratch .. 😢 for sure it wont be billions of tokens, I want to make it practical for example for home management, or school reporting system ... instead of static reports . to enable it to create and run its own sql queries and run it .. 😅

[@ShawhinTalebi]

Happy to help! To make something practical I'd recommend using an existing model fine-tuned to generate SQL queries e.g. huggingface.co/defog/sqlcoder

[@Nobody2310]

what is the most basic technical artifact that is used/required to build any LLM? Is that an existing LLM such as Llama 2?

[@ShawhinTalebi]

I am not quite sure of the meaning of "most basic technical artifact," but here's one perspective. There are two ways to build an LLM: from scratch and fine-tuning. When training from scratch, the essential piece is the training data used to develop the model. When fine-tuning, the essential piece is the pre-trained model you start from (e.g., Llama2). Hope that helps!

[@ronakbhatt4880]

@17:08 isnt weight of decoders are wrong if 0 is the weight of token to the future token to it?

[@ShawhinTalebi]

Sorry I didn't understand your question. Could you rephrase?

[@vsudbdk5363]

Any resources on enrichment of prompt template, I feel in my case difficult one to understand and implement as an LLM returns response based on how we define the template overcoming unecessary context...

[@vsudbdk5363]

Recently begun exploring Generative AI need like proper guidance on where to learn and do the code part, ik it will be a long journey understanding the math behind it, learning concept and code, staying all night for checkpointing metrics, performance and all.. thank you

[@ShawhinTalebi]

Great question. The video on prompt engineering might be helpful: kzbin.info/www/bejne/ZpTJaKmwgsSXkJI

[@ShawhinTalebi]

That's a good mindset to have. AI is an ocean, with endless things one can learn. This playlist could be a good starting place: kzbin.info/aero/PLz-ep5RbHosU2hnz5ejezwaYpdMutMVB0

[@vsudbdk5363]

@@ShawhinTalebi thank you very much

[@nobafan7515]

Thank you for the video! I was wondering if you can help me. Lets say i ask gpt if romeo and juliet was a comedy or a tragedy, and the only data it has was put in by people that didnt have time to fact check the data, and i wanted my own gpt (lets say this is one of the tiny ones that can easily run on my laptop) so it can explain the history of it so it can explain to me the facts of it. Do i need to dive in the llm model and find that specific data to correct it? Can i fine tune it to improve it (lets say i have a gpu big enough to train this llm)? Is the model fine, but i need a different gpt?

[@ShawhinTalebi]

If I understood correctly, the question is on how to ensure the LLM gives accurate response. While there are several ways one can do this, the most effective way to give a model specialized and accurate information is via a RAG system. This consists of providing the model specific information from a knowledge base depending on the user prompt.

[@user-oo2co6xb8u]

Hi, i have domain specific pdf files . How do i train using transfer learning? Please advise

[@ShawhinTalebi]

Depends on what you mean by transfer learning. If you simply want to extract knowledge from a PDF I'd recommend exploring RAG or using off-the-shelf solutions like OpenAI Assistants interface. Happy to clarify, if I misinterpreted the question.

[@inishkohli273]

Just completed the whole video . Took me 10 days, . It is a good idea to just provide surface knowledge and not overwhelming the students but instead letting them to research and further read it on their own by giving tons of references. I have a suggestion, why not create a open notebook allow student to edit and fillup more information/learning materials because there were some point in the video where it feels like you could have elaborated more or scratched and summarized even a small portion of that subject more. Thanks

[@ShawhinTalebi]

That's a great suggestion! I've always been a fan of "open-source" textbooks and the like. Feel free to share any points you'd like me to discuss further in future videos of this series :)

[@abhishekfnu7455]

Is there a way to use Data Dictionary to train LLM model to generate SQL queries later on?

[@ShawhinTalebi]

Yes, but you will likely need to transform the data a bit before it can be used for fine-tuning. I give a concrete example of this here: kzbin.info/www/bejne/aoOkp32qaMuKpas

[@Nursultan_karazhigit]

Hello , Thanks . Do you know is it possible to create an own LLM for own startup?

[@ShawhinTalebi]

Of course this is possible. However, it is rarely necessary. I'd suggest seeking simpler (and cheaper) solutions before jumping to training an LLM from scratch.

[@Nursultan_karazhigit]

@@ShawhinTalebi thanks

[@amparoconsuelo9451]

Can a fine-tuned LLM be repurposed and re-fine-tuned for more than one task?

[@ShawhinTalebi]

Yes it can! In fact, that is what OpenAI did with their RLHF technique to create their InstructGPT models

[@techdiyer5290]

What if you could make a small language model, that maybe only understand english, can understand code, and is easy to run?

[@ShawhinTalebi]

That is a compelling notion. If we can get there, then it would make this technology even more accessible and impactful.

[@shrinik1969]

Size = accuracy...small may not give u what u want

[@Decapodd]

NanoChatGPT

[@jackflash6377]

Just now asking GPT4.0 to help me with training text. It is not allowed to assist in training any LLMs and would not give me anything.

[@ShawhinTalebi]

I believe it’s now against OpenAI’s policy to use their models to train other models. You may need to look to open-source solutions eg Llama2, Mistral

[@petevenuti7355]

​@@ShawhinTalebi . How could it possibly stop it ? If the model being trained fed the prompt and used the response for reenforcment and alignment?

[@hashifvs519]

Can you post a video onu continual pretraining of llms like Llama

[@ShawhinTalebi]

Thanks for the great suggestion. I’ll be doing more content on fine-tuning so that will be a good topic to cover there.

[@hari_madh]

bro i want to build an LLM.. does this video help me learn myself and build LLM myself? possible? (i did not see it till now)

[@ShawhinTalebi]

While this video may be a helpful first step, more resources will be necessary. Here are a few additional resources I recommend. - kzbin.info/www/bejne/oXTGaXmjesdkpLs&ab_channel=AndrejKarpathy - huggingface.co/learn/nlp-course/chapter1/1?fw=pt

[@dohua_ai]

So my dreams about own LLM are broken(( So as i understood the only way to build some personal LLM is FineTuning? Atleast while cheap ways of training not appeared yet...

[@ShawhinTalebi]

I wouldn't give up on it! My (optimistic) conjecture is as we better understand how these models actually work we will be able to develop ones that are much more computationally efficient.

[@TheIronMason]

When it comes to transformers. Are you saying they're more than meets the eye?

[@ShawhinTalebi]

That's a good way to put it 😂

[@issair-man2449]

Hi, hoping that my comment will be seen and responded... I FAIL to understand: If a simple model learns/predicts, couldn't we prompt it to delete the trash data and train itself by itself autonomously until the model becomes super intelligent?

[@ShawhinTalebi]

LLMs alone only do token prediction, as discussed in the first video of this series: kzbin.info/www/bejne/qnerloiJf6aMmKc While an AI system could in principle train itself, it would require much than just LLM to pull that off.

[@lyonspeterson1094]

Good contents. But when I watch the video, there are so many ads. I;m even confused what I am supposed to watch.

[@catulopsae]

What does it mean the amount of parameters???

[@ShawhinTalebi]

Good question. A model is something that takes an input (say a sequence of words) and produces an output (e.g. the next most likely word). Parameters are numbers which define how the model takes inputs and translates them into outputs.

[@catulopsae]

@@ShawhinTalebi thank you

[@Sunnyangusyoung]

What if I don’t want to build my model but work for someone who is building one.

[@varadacharya2802]

Can you make a series on Data Science and Artificial Intelligence Topics

[@ShawhinTalebi]

Anything in particular you'd like to see?

[@varadacharya2802]

@@ShawhinTalebi I would be nice if you made on AI for begineers who do not know any algorithms of AI like DFS , BFS etc

[@guerbyduval4104]

Do you have a course on how to do it as a programmer instead of *like a chat gpt talker* ?

[@ShawhinTalebi]

I don't have a from scratch coding tutorial yet. But I am a fan of the one from Andrej Karpathy: kzbin.info/www/bejne/oXTGaXmjesdkpLs

[@jamesmurdza]

The matrices at 16:40 don't look right to me. I think the words labelling the rows should go from top to bottom, not bottom to top.

[@ShawhinTalebi]

Good catch! Yes, the word labels are inverted on the Y axis. A corrected visualization is provided in the blog: medium.com/towards-data-science/how-to-build-an-llm-from-scratch-8c477768f1f9?sk=18c351c5cae9ac89df682dd14736a9f3

[@hypercoder-gaming]

When you were calculating the cost, you estimates that a 10b model would take 100k GPU hours but Llama 2 took 180k GPU hours and that was 7b. These estimates are way off. How is it that 100b costs less than 70b?

[@ShawhinTalebi]

The numbers from Llama 2 were only meant to give an idea of scale. More precise estimates will depend on the details of the use case.

[@crosstalk125]

Hi, I like your content. But I want to point out that what you are calling tokenization is vectorization. Tokenization breaks documents/sentences/words into subpart and vectorization converts tokens into numbers. Thanks

[@ShawhinTalebi]

Thanks for raising that point. Here I'm lumping the two together, but these are 2 separate steps.

[@Joooooooooooosh]

Wait how did we get from $180K for a 7B model to $100K for a 10B model...

[@ShawhinTalebi]

This is what we Physicists call an "order-of-magnitude estimate"

[@C_Hart]

I would like to train a LLM to give driving advice and give it lots of statistical data over and over again on the 8 women and 0 men that have ever crashed into me. Would be hilarious to have a LLM that gave driving advice relevant to the things Ive encountered on the road driving for 10 years of on-the-road work. You know what I'm saying?

[@imaspacecreature]

Some of you need to look into your own "internal libraries", this video is someone attempting to teach you where to get the fish. Some of you get so hungry for the fish, but won't even understand the water it resides in.

[@Vermino]

Just my prediction on data curation & copyright, what is going to happen is the companies of LLM's will do what Google did back in the day with scraping websites. Then once legislation passes, they will say "if you don't want your data crawled, opt-out with robots.txt". Right now it's a grey area and companies are building their data as quickly as possible to get in before the regulation. "Better to ask for forgiveness, rather than permission."

[@ShawhinTalebi]

I can see that. It seems to be a hot-button topic these days.

[@PabloPernambuco]

Now, I am discovering my low QI... 0,001% of learning...😂

[@MaxQ10001]

Is it just me, or does the math on the "how much does it cost" not make sense? 7b uses 180,000 hours, so 10b used 100,000 🤔 hours

[@ShawhinTalebi]

This is only meant to give a sense of the cost's scale, so I round to the nearest order of magnitude :)

[@Decapodd]

​@@ShawhinTalebi if you read his message again, I think he made a mistake

[@Decapodd]

10b is 1million hours not less than 180 thousand

[@MrSpikegee]

The cost estimation does not seem correct, you have to take into account the time taken to train to estimate the hardware buying scheme, here you estimate as if we wanted to train the model in one hour - if you accept for eg. 4 days approx. 100h, you need only ten A100 which would be 100k€, so less than with the renting option.

[@ShawhinTalebi]

I may be missing something here. Based on the numbers in the Llama paper, training 10B and 100B parameter models with 10 A100s, would take about 10,000 hr (1.1 years) and 100,000 hr (11 years), respectively.

[@harekrishna_3]

Chatgpt suggest me this channel 🎉

[@ShawhinTalebi]

LOL, hope it was a good rec :)

[@DYLOGaming]

Well this was a video that explains things conceptually but does 0 building from scratch

[@ShawhinTalebi]

Sorry for the lack of code in this one, maybe other videos in the series will be more helpful. An end-to-end technical guide on building an LLM isn't something I can fit in one video, but I hope to share more videos toward this end in the future.

[@gamea9g349]

why cant they use algorithims insted of this huge data sets

[@ShawhinTalebi]

This is the big paradigm shift with machine learning (ML). Traditionally programmers implement logic into computers line by line. However, there are some tasks that are difficult (if not impossible) to explicitly program computers to do (e.g. driving, responding to open-ended prompts, etc). For these tasks ML allows us to program computers with data.

[@alexandrumittermuller4430]

nothning good i not learn how to programe and make llm

[@ShawhinTalebi]

Sorry to hear, if I may recommend this video: kzbin.info/www/bejne/omnTg4arip1khrcsi=Hyq1GB5jFbC6khXA

[@bittertruthnavin]

90 % of the people out there are just trying learn how to drive a car, not how to build a car. All these youtubers are showing is how to build a car which further complicates the process of new AI adoption and demotivate people. I just saw video by this lady somewhere and she showed she trained mistral 7b LLM in less than 1 hours for $1. Folks keep looking for better and simple information, remember we don't need to know how to build a car in order drive one.

[@saisriramakarthikeyakollur3989]

I'm curious to know more about mistral 7b. Any chance you could share your source? All I found was the trained model on huggingface

[@ShawhinTalebi]

Excellent point. Using these models to solve problems is much more impactful than knowing how they work. Other videos in this series are more hands-on than this one. Suggestions for future content are also appreciated :) kzbin.info/aero/PLz-ep5RbHosU2hnz5ejezwaYpdMutMVB0

[@Drdozzer]

What if what I want to do is build a car 😅

[@scotthjackson5651]

I literally want to know about how to build an LLM.

[@imaspacecreature]

This comment is extremely concerning. Do what you will, but a LOT of humans are PROGRAMMED CONSUMERS. Aka, you are like a Tool, you only do what you are given, like a task. Don't uplift or look down on other for wanting to know how things work. And also, some people drive cars and don't even know *how to drive, and they kill people every minute on a roadway. You should rethink your own logic on this, before you even entertain a "artificial intelligence"..

[@guerbyduval4104]

3:23 What is the point of talking gpu hours and money spent on cloud. I have and Nvidia RTX 3070 with 5120 cuda cores. I don't care spending money to train. I just can train on my laptop.

[@ShawhinTalebi]

That was to give context on training an LLM from scratch. This is negligible for smaller models.

[@ButchCassidyAndSundanceKid]

A bit disappointed with this video, there is no coding !

[@ShawhinTalebi]

Thanks for the feedback. Yes this one stays relatively high-level since a coding video would be (much) longer. For a hands-on guide I'd recommend Andrej Karpathy's video: kzbin.info/www/bejne/oXTGaXmjesdkpLs

[@genkidama7385]

internet is full of trash to be honest. train llm on books would be a great leap for mankind. code datasets are full of "oh its too complex" and only produces comments instead of code. Machine speaking as a human and complaining about things beeing too complicated to do. thats how lame datasets are. computer is a computing device and refuses to work because the dataset says things are too complex. jesus. unbelievable levels of trash.

[@ShawhinTalebi]

The greatest challenge is that their aren't enough (high-quality) books to keep scaling these models. How we solve this problem is going to define the next wave of LLMs. (Maybe we'll switch to lectures and other videos)

[@handleking1]

Bro doesn't even know how to spell NVDIA correctly.

[@ShawhinTalebi]

Spelling was never my strong suit XD

[@thechoosen4240]

Good job bro, JESUS IS COMING BACK VERY SOON;WATCH AND PREPARE

[@ShawhinTalebi]

🙏

[@nick066hu]

Thank you for putting together this video, helped me a lot to understand LLM training. One question: with the advent of trillion token models and beyond, I wonder where will we get all that training input data from. I guess we already consumed what all humanity has produced in the last 5000 years, and by adding another 10M digitized cat videos, the models will not be smarter.

[@ShawhinTalebi]

Good question! I suspect there is still much content out there that hasn't been touched by LLMs i.e. non-digital text and proprietary data. Nevertheless, this content is still finite and the "just make a bigger model" approach will eventually hit a limit.

[@gameplaychulangado494]

i am from brazil and my english is not so nice but i did understand you almost well. I think that biggest challenge will be math and matrix 3D and more. 🥲

[@ShawhinTalebi]

Glad it made sense :)