How ChatGPT is Trained

This short tutorial explains the training objectives used to develop ChatGPT, the new chatbot language model from OpenAI.
Timestamps:
0:00 - Non-intro
0:24 - Training overview
1:33 - Generative pretraining (the raw language model)
4:18 - The alignment problem
6:26 - Supervised fine-tuning
7:19 - Limitations of supervision: distributional shift
8:50 - Reward learning based on preferences
10:39 - Reinforcement learning from human feedback
13:02 - Room for improvement
ChatGPT: openai.com/blog/chatgpt
Relevant papers for learning more:
InstructGPT: Ouyang et al., 2022 - arxiv.org/abs/2203.02155
GPT-3: Brown et al., 2020 - arxiv.org/abs/2005.14165
PaLM: Chowdhery et al., 2022 - arxiv.org/abs/2204.02311
Efficient reductions for imitation learning: Ross & Bagnell, 2010 - proceedings.mlr.press/v9/ross...
Deep reinforcement learning from human preferences: Christiano et al., 2017 - arxiv.org/abs/1706.03741
Learning to summarize from human feedback: Stiennon et al., 2020 - arxiv.org/abs/2009.01325
Scaling laws for reward model overoptimization: Gao et al., 2022 - arxiv.org/abs/2210.10760
Proximal policy optimization algorithms: Schulman et al., 2017 - arxiv.org/abs/1707.06347
Special thanks to Elmira Amirloo for feedback on this video.
Links:
KZbin: / ariseffai
Twitter: / ari_seff
Homepage: www.ariseff.com
If you'd like to help support the channel (completely optional), you can donate a cup of coffee via the following:
Venmo: venmo.com/ariseff
PayPal: www.paypal.me/ariseff

[@Mutual_Information]

Very insightful. Following Dall-e, it seems OpenAI was a little bit more protective of their training IP (only a blog on ChatGPT - no paper). You have enough familiarity with the surrounding papers and tech to paint a clear picture of what their doing. Excellent work and again, very insightful!

[@ariseffai]

Thanks DJ, appreciate the kind words :)

[@laurenpinschannels]

by the way mutual information, I would love to see you make your subscription lists public

[@b0nce]

For real, DJ, on every ML/DL/Math YT channel I like, I've seen your comment at least once :D

[@Mutual_Information]

@@laurenpinschannels ha I didn't realize it was private. Switched! Enjoy :)

[@danielhenderson7050]

Agreed, thank you for sharing

[@joshelguapo5563]

Since chatgpt blew up it's been tough to find technical content on chatgpt so thanks for pulling this up!

[@lucasjackson7647]

Just chatgpt it lol

[@technophobian2962]

One of the reasons for that is openAI not being very open.

[@AkolytosCreations]

+1 to this. I have spent hours trying to find technical content like this. Videos either assume you know everything about AI and jump straight into in depth things (and even these videos are rare), or are so superficial it doesn’t really say anything. This was that perfect inbetween.

[@dkarkada]

one of those elusive youtube gems. Wish there was more content out there for the serious nonexpert. Thanks!!

[@kumakumako]

Thank you for making the video. Great balance of technical content and accessibility for people (like me) who aren't in the field.

[@abhishekpatil6071]

A really fun video to watch, kudos to you for making such an esoteric topic easy to understand (at least in broad terms) for a layman as well.

[@charlesje1966]

Thankyou. I've been learning chatgpt to program microcontrollers and this video clear up a lot of questions and helps explain the common problems I get from the chatgpt bot output. I'm finding that it takes a lot of work on the part of the user to establish context, provide training examples, and to find the best wording to achieve your goal.

[@mertozlutiras]

You are doing an amazing job explaining the complex concepts in a simple way. Keep up the good work!

[@ChocolateMilkCultLeader]

One of the only useful videos on ChatGPT on this platform. Great work

[@nedyalkokarabadzhakov5405]

That we need people on youtube that provide actual useful easy to comprehend knowledge, based on their leanring experience. Basicaly any human that have signigicant leanign expience and knowledge in one or more domains is a human chatgpt. Thanks for the content.

[@BlueBirdgg]

Best video Ive watched describing ChatGPT! (and watched more than 20+) You have great insights!

[@user-gg7vb8te9v]

Great work, Ari! Thank you very much for crafting the content, it's really easy to digest.

[@whatamievendoing]

Amazing video. Thanks for publishing this. Going to dig through the rest of your videos too

[@curumo_curunir]

Very simple and effective explanation. Thank you.

[@Etcher]

Excellent video, thank you - definitely one of the best technical explanations of what is going on under the hood of ChatGPT I have found on YT to-date.

[@TasteTheStory]

On my KZbin channel, I tested how good ChatGPT is at writing movie scripts! I found the results to be interesting.

[@minsohee]

Thank you so much for your efforts, this video was by far the most helpful for my project!

[@thegooddoctor6719]

Brilliant. On aspect of Intelligence is a measure of one's ability to describe a complex topic into simplistic terms everyone can understand. My friend - you have that ability in spades. Congrats and Thank You !!!!!

[@PrafulKava]

Best step-by-setp explanation !

[@miguelalba2106]

Technical, concrete and easy to follow explanation, good video 🔥

[@DanielTorres-gd2uf]

Hey, just found your channel. Awesome stuff (currently studying for a masters in ML, it's crazy to see topics I've covered in class come up here)!

[@billvvoods]

@Daniel Torres, Congratulations. Just curious but what was your bachelors in?

[@DanielTorres-gd2uf]

@@billvvoods Mechanical Engineering!

[@billvvoods]

@@DanielTorres-gd2uf very nice! I wish you the best in your studies. I’m now inspired 😉

[@DanielTorres-gd2uf]

@@billvvoods Thanks, you as well! :)

[@sdsd5450]

Thank you so much! It is such a great video even for beginners!

[@Francis-gg4rn]

amazing, please make more!

[@FiEnD749]

Dude, your content is incredible!

[@VaibhavShewale]

good insight to how it works learned something new!

[@albertkwan4261]

This is the best explanation of ChatGPT!

[@lij3900]

Hi Ari, really appreciate you made the video! It is great learning experience. Do you mind sharing the transcript on your website as well? For tech stuff, people like me learned better by reading than by watching videos. I tried use the extension to get the video script, but it is not 100% accurate so some tech words are not correct.

[@Billionaire-Odyssey]

Very much valuable content explained with clarity I wonder why you channel haven't still exploded you earned a new sub and continue making videos on such topics

[@jeffhayes8543]

Very well presented. Thanks!

[@bogdanpatedakislitvinov2549]

Very well-made presentation, please make more! Subscribed

[@johnchange5691]

Thank you! Well explained.

[@dwt6273]

Thank you! Very informative!

[@alonsamuel7106]

Great explanation and naration...! Thanks!

[@Doggieluv25]

This was so helpful thank you!!

[@arijitdas4504]

Absolute gem ❤

[@Bianchi77]

Cool video shot, well done, thanks for sharing :)

[@narendiranchembu5893]

This is a very nice explanation, thanks! What tools do you use to make your videos?

[@ariseffai]

Thanks! For this one I used a combination of keynote & FCP

[@mdzeeshansiddique8185]

On the same boat here, after minutes of going through click baits, finally a worthy explainer. Thank you.

[@jaymehta5886]

Nice explaination. Thanks

[@panashifzco3311]

Well- explained video. So cool!

[@jadenlorenc2577]

the clearest ai expert on youtube

[@masoncdyer]

Excellent review

[@aethermass]

Great explanation.

[@pw7225]

Dang, this is a GOOD video. So many crap videos have been published on the topic. Hard to find one that has substance. THANK YOU!

[@gpt-jcommentbot4759]

lol i hear this on every ai video.

[@vijayanandpalaniswamy2240]

Excellent insight dude! Awesome work. I need some help on time series algorithms? dataset with multiple parameters. can you help?

[@user-fj9bh7kt7t]

Very good presentation!

[@nebrasothman1817]

thank you great video, great detailed explanation

[@ariseffai]

Thanks Nebras!

[@sethjchandler]

Great job. Going to show this to my class (Large Language Models for Lawyers, University of Houston Law Center)

[@rl6382]

Just wanted to thank you for these videos.

[@ConceptsWithCode]

Nicely done. Thanks for creating this video. Few quick questions/clarifications. (1) Given the reward model rates an entire response as opposed to each partially complete sentence as tokens are emitted, isn't the final stage also rating the reward for an entire possible sentence (that is terminated by a stop token?). (2) Also was the use of the SFT also in the third stage for KL divergence calculation omitted in the figure because it seemed like too much detail? (3) You mention the upper limit is 3000 words. Is this an approximation for tokenized words that would a maximum sequence length of 8k? (4) Lastly, any idea if the parameters of the model is 16bit float or 32 bit float? Thanks in advance!

[@SirajRaval]

this is so good, subscribed.

[@yugen3968]

Scammer pos

[@user-wr4yl7tx3w]

Given that the scores used to train the reward function is small, compared to the universe of potential questions and answers, it's hard to see how a small training set can possibly be sufficient to train adequately. Still amazes me.

[@user-wr4yl7tx3w]

Great content, Thanks

[@MyMmmd]

I'd love to know more about those "expert" conversations. Do you need to be an expert in the conversation matter or is it just used to make sure it's good at conversing (rather than getting the facts right)? How many of these expert conversations are useful? Is it a case of diminishing returns beyond a certain point? I'm guessing this isn't freely available information but it's fascinating to me.

[@dr.mikeybee]

This is a coherent nicely structured explanation of ChatGPT's architecture. Thank you for sharing this. BTW, how likely is it that OpenAI will create a new model with primarily supervised learning? I assume they are curating a new training set from both human responses and model-generated responses. It seems to me that a smallish self-supervised transformer model, trained in an autoregressive fashion from a well-curated knowledge base like Wikipedia and the Encyclopedia Britannica, etc., would be a great start for transfer learning from a curated supervised training set. Your video seemed to suggest this possibility. Moreover, it would be very interesting to run this side-by-side with a different architecture based on a vector database and semantic search for knowledge collection, retrieval, and context building. The results of this could be passed through an LLM for human readability and probabilistic generation. This should result in some sort of fuzzy-verified responses.

[@tejshah7258]

Legend has returned - pls make more videos!

[@AR-iu7tf]

Nicely done. Thanks for creating this video. Few quick questions/clarifications. (1) Given the reward model rates an entire response as opposed to each partially complete sentence as tokens are emitted, isn't the final stage also rating the reward for an entire possible sentence (that is terminated by a stop token?). Or do you believe the output sentence is rated for each token emitted until stop token? (2) Also was the use of the SFT also in the third stage for KL divergence calculation omitted in the figure because it was too much detail? (3) You mention 3000 words max limit. Is this an approximation for the max sequence length of 8k tokenized length (4) Lastly, do we know if the model parameters are 16bit floats or 32 bit? Thanks again for making this informative video.

[@ariseffai]

Thanks! 1) Yes, the reward model rates an entire completed response. So an "action" here is a full response (sequence of tokens w/ ) emitted by the model. 2) Are you referring to the plot at 12:06? 3) The 3K words is an approximation to 4K max tokens, as described here: help.openai.com/en/articles/6787051 4) Great question! I'm not sure of the precision of the production model. Do post if you find it :)

[@AR-iu7tf]

@@ariseffai thank you for your response. Regarding question 2- yes exactly. From the openai blog picture and the instructgpt paper I assumed three models were used in the final RL training - a copy of the SFT that became final production model(RL model ) with updated weights , the reward model and a frozen SFT model for the KL divergence computation that constraints the RL model to generate original sentence but not too far off from SFT. Is that your understanding too ? Regarding 3- certainly will post in case I find it . Thanks again !

[@MrLazini]

Very informative

[@juanmanuel8464]

Great content!

[@alimansourey2076]

Well done !

[@gorgolyt]

Great summary. I didn't follow when you said "we need to model to act during training" as a way of mitigating distributional shift... can you explain some more?

[@ariseffai]

So basically, if the model takes zero actions during training, this means we'll have a big difference between the deployment distribution of states (when the model selects actions itself) and the training distribution (when the model merely observes the human's actions). There are different ways to have the model select actions during training. One is by using a standard reinforcement learning setup, as mentioned in the video. In that case, the policy model is directly rewarded for actions it itself executes. But another possibility comes from "on-policy" imitation learning, such as the DAgger algorithm. We iteratively execute the current policy to gather new training states, but then have an expert provide the correct action labels -- see arxiv.org/abs/1011.0686

[@muidhasan9498]

please make more videos like this

[@wenderse]

May I ask what technology you used to create such nice explanatory videos? Did you use 3b1b's manim engine? thanks.

[@LSS94]

Very much looks like it!

[@ariseffai]

Not for this one - just keynote and FCP. But I have used manim in a couple other videos :)

[@hoangviet1381]

nice video, thanks !!!

[@lazycompounder3724]

that was awesome

[@EducatedButton]

Thanks a lot for the explaination. How does it work during inference time to keep a conversation back and forth?Is the user's current chat session provided to the model as input along with a new user prompt?

[@ariseffai]

That's right. There's a certain context window of previous text to which the model can attend (on the order of thousands of tokens). This will include both previous user inputs and model responses from the current conversation.

[@karihotakainen5210]

How does the reward model score a single action, when it is trained to choose between two actions? Or does the policy model actually generate k actions that the reward model can then score and then choose a reward knowing which action the policy model saw as the most probable one? I'd really appreciate an answer, thanks.

[@ddystopia8091]

Hello, I want to work in this field. Now I'm a first year student studying informatics, how should I move towards it? Thank you!

[@user-mh9up1mw3r]

What is the architecture of the policy model and how large is it? How does it use the pretrained LLM?

[@mentor1013]

Can you please make a video on Midjourney as well?

[@HH-mf8qz]

Very good video Can you maybe make an updated version now that chatgpt 4 is released and the new googel gemeni is about to come out for mixel input AIs

[@epeeypen]

i used chatgpt to help we write a love letter and it went really well.

[@brandonojalvo9775]

Your love is a lie

[@sjakievankooten]

Love the explanation!! Also thanks for making the video darkmode 😊

[@sjakievankooten]

@@TasteTheStory good videos mate, but no need to spam it here :)

[@TasteTheStory]

@@sjakievankooten Not spaming just trying to connect with people who share the same interest. thanks for your note.

[@notgabby604]

That's all very high level usage of neural networks. While some people think the basic foundations haven't set yet. Like for example 2 Siding ReLU.

[@yugen3968]

Hey, where could I approach you to clear a few things out about this...?

[@wladefant]

Are you saying that the 3.000 words can not be increased by just for example more ram usage per chat (chatgpt)?

[@posthocprior]

Thanks so much for posting a clear explanation. After watching this, I feel like I do after I've been explained how a magic trick works: disappointed.

[@nchristensen3309]

Is the operations from us as users part of the reward system ?

[@baohq]

What is the platform that OpenAI uses to build chatgpt. Like pytorch, tensorflow or something ?

[@satishkumar-ir9wy]

Hi, can you make a small video to build ChatGPT with NLP based classification Model.

[@sanchi3944]

Lmao this literally what i asked GPT today since I'm making a chatbot on Rasa. Looks like the algos are pointing me in the right direction for once!

[@roromaniac8]

This was a wonderful explanation! Wouldn't it be expensive to have that much human capital evaluating and simulating chatbot responses? Seems especially so when you consider the wide amount of domains ChatGPT is able to provide reasonably correct responses to.

[@CyberDork34]

Yes it is expensive. OpenAI outsources these tasks to countries like Kenya to save on these costs. It's kind of dubiously ethical but yeah

[@roromaniac8]

@@CyberDork34 do you have a source that I could read about this? I haven’t been able to find something online.

[@anthonydemattos432]

It is possible to do most of this process with just the fine tuning api?

[@Mike-vj8do]

amazing video Ari. Where is the name from? Israeli?

[@AstroPinion]

Thanks!

[@ariseffai]

Thanks Randall!

[@siw504]

Nice Video

[@juliarose2133]

anyone know what the equation is at 4:08 , where i can find more on it?

[@wladefant]

13:12 The new bing (sydney) is able to link sources perfectly now

[@regCode]

I'm having trouble understanding supervised fine-tuning in this context. What are the labels? What is the task?

[@icyou8496]

good explanation!! i just wandering how / what chatgpt threshold for displaying no results? i have observe something like this for example : me : (example of 1 non professional gamer) chat gpt : i dont have enough data for him me : (example of 1 professional gamer in same game) chat gpt : *explain professional player* me :(asking the first player) chat gpt : *explain about that non professional gamer*

[@DrJanpha]

Codes as training data are only briefly mentioned?

[@shivangitomar5557]

BEST!

[@serioussrs9349]

Cool bro

[@MainTeknoID]

Kerenn

[@peccavius]

Thanks for the talk! You mention that the reward model is trained using cross-entropy loss as a binary classifier. I don't think that's accurate since you don't have a ground truth label for, say, response A (since the score is relative to others). The openAI paper just uses the negative log difference in scores between the higher and lower ranked response as the loss.

[@ariseffai]

You're welcome! That's not quite correct. The classifier is trained to predict which of two responses is ranked higher by the human contractors. Then, the scalar logit output by the trained classifier for an individual response can be used as a reward signal.

[@romeopeter1922]

the Math and some of the logic went over my head so I'm going to tray and summarise what I think I understood: ChatGPT is built on top it's predecessor InstructGPT which is mathematically trained with large data set to give spit to instructions given. However for ChatGPT, just spitting answers from instructions isn't enough and needs to be retrained over a method called 'Reinforcement Learning' which uses a 'reward model' to rank the next favourable answer. Did I get it? if not, then please tell me what I'm missing but in plain language because I know the above is flawed.

[@BetaTester704]

According to ChatGPT it's memory is limited to only one prior message in the same conversation, beyond that it can't remember anything.

[@rickylehr9284]

Why does it care about the reward in reward reinforcement?

[@stephenthumb2912]

a bit amazing how the hallucinations begin, so similar to a human caught in a lie or imagination, the lies built on lies get progressively more absurd in the same way that an untruth from a human where it gets more and more difficult and outlandish to make up a reason based on a stack of false premises.

[@Black-ww6lj]

Plot twist : Content of this video was generated by chatGPT

[@andramalexh]

PPO= operant conditioning?

[@StephenGillie]

ChatGPT is like texting someone using only autosuggest, with most/all of the internet as the database. The real innovation in ChatGPT is compressing most/all of the public internet to just 350 GB, and an open source project has this down to 1.62 GB.

[@D_Winds]

I would like to have this reservoir of data.

[@uestcpandou]

Regard Chatgpt or other LLMs as database is a huge misunderstanding. Same as human brain, LLM is not so good at memorizing, compared to its ability of reasoning and fabricating new things. Combine traditional db and LLM to make them do what they are good at is the only way in the long run.

[@nixedgaming]

Calling transformers with self attention and multiheaded attention “autosuggest” is wildly reductive and borderline disingenuous, even if it’s technically correct

[@StephenGillie]

@@nixedgaming The best kind of correct! Others remind that calling this compression is also wildly reductive and borderline disingenuous, so at least I'm consistent.

[@BooleanDisorder]

Calling it autosuggest is like calling humans autosuggest. "Humans are just autosuggest with all of their lives as the database." People misunderstand what's what. You need to separate the task from what it actually is. The task is next word prediction. How it works is definitely not like autosuggest. The same way my task right now is next word "prediction" while writing this.