How ChatGPT is Trained

[@joshelguapo5563]

Since chatgpt blew up it's been tough to find technical content on chatgpt so thanks for pulling this up!

[@lucasjackson7647]

Just chatgpt it lol

[@technophobian2962]

One of the reasons for that is openAI not being very open.

[@AkolytosCreations]

+1 to this. I have spent hours trying to find technical content like this. Videos either assume you know everything about AI and jump straight into in depth things (and even these videos are rare), or are so superficial it doesn’t really say anything. This was that perfect inbetween.

[@Mutual_Information]

Very insightful. Following Dall-e, it seems OpenAI was a little bit more protective of their training IP (only a blog on ChatGPT - no paper). You have enough familiarity with the surrounding papers and tech to paint a clear picture of what their doing. Excellent work and again, very insightful!

[@ariseffai]

Thanks DJ, appreciate the kind words :)

[@laurenpinschannels]

by the way mutual information, I would love to see you make your subscription lists public

[@b0nce]

For real, DJ, on every ML/DL/Math YT channel I like, I've seen your comment at least once :D

[@Mutual_Information]

@@laurenpinschannels ha I didn't realize it was private. Switched! Enjoy :)

[@danielhenderson7050]

Agreed, thank you for sharing

[@dkarkada]

one of those elusive youtube gems. Wish there was more content out there for the serious nonexpert. Thanks!!

[@kumakumako]

Thank you for making the video. Great balance of technical content and accessibility for people (like me) who aren't in the field.

[@nedyalkokarabadzhakov5405]

That we need people on youtube that provide actual useful easy to comprehend knowledge, based on their leanring experience. Basicaly any human that have signigicant leanign expience and knowledge in one or more domains is a human chatgpt. Thanks for the content.

[@prachi-k2y]

I appreciate how SmythOS prioritizes security without compromising speed. The frequent updates keep the system running efficiently. The customizable desktop environment is a great touch for users who like to personalize their setup. A solid choice for everyday use!

[@ChocolateMilkCultLeader]

One of the only useful videos on ChatGPT on this platform. Great work

[@BlueBirdgg]

Best video Ive watched describing ChatGPT! (and watched more than 20+) You have great insights!

[@Billionaire-Odyssey]

Very much valuable content explained with clarity I wonder why you channel haven't still exploded you earned a new sub and continue making videos on such topics

[@abhishekpatil6071]

A really fun video to watch, kudos to you for making such an esoteric topic easy to understand (at least in broad terms) for a layman as well.

[@charlesje1966]

Thankyou. I've been learning chatgpt to program microcontrollers and this video clear up a lot of questions and helps explain the common problems I get from the chatgpt bot output. I'm finding that it takes a lot of work on the part of the user to establish context, provide training examples, and to find the best wording to achieve your goal.

[@mdzeeshansiddique8185]

On the same boat here, after minutes of going through click baits, finally a worthy explainer. Thank you.

[@Quynh-AnhTran]

Great work, Ari! Thank you very much for crafting the content, it's really easy to digest.

[@thegooddoctor6719]

Brilliant. On aspect of Intelligence is a measure of one's ability to describe a complex topic into simplistic terms everyone can understand. My friend - you have that ability in spades. Congrats and Thank You !!!!!

[@pw7225]

Dang, this is a GOOD video. So many crap videos have been published on the topic. Hard to find one that has substance. THANK YOU!

[@gpt-jcommentbot4759]

lol i hear this on every ai video.

[@Etcher]

Excellent video, thank you - definitely one of the best technical explanations of what is going on under the hood of ChatGPT I have found on YT to-date.

[@TasteTheStory]

On my KZbin channel, I tested how good ChatGPT is at writing movie scripts! I found the results to be interesting.

[@curumo_curunir]

Very simple and effective explanation. Thank you.

[@mertozlutiras]

You are doing an amazing job explaining the complex concepts in a simple way. Keep up the good work!

[@whatamievendoing]

Amazing video. Thanks for publishing this. Going to dig through the rest of your videos too

[@miguelalba2106]

Technical, concrete and easy to follow explanation, good video 🔥

[@nebrasothman1817]

thank you great video, great detailed explanation

[@ariseffai]

Thanks Nebras!

[@jadenlorenc2577]

the clearest ai expert on youtube

[@wladefant]

Are you saying that the 3.000 words can not be increased by just for example more ram usage per chat (chatgpt)?

[@minsohee]

Thank you so much for your efforts, this video was by far the most helpful for my project!

[@DanielTorres-gd2uf]

Hey, just found your channel. Awesome stuff (currently studying for a masters in ML, it's crazy to see topics I've covered in class come up here)!

[@btrees]

@Daniel Torres, Congratulations. Just curious but what was your bachelors in?

[@DanielTorres-gd2uf]

@@btrees Mechanical Engineering!

[@btrees]

@@DanielTorres-gd2uf very nice! I wish you the best in your studies. I’m now inspired 😉

[@DanielTorres-gd2uf]

@@btrees Thanks, you as well! :)

[@karihotakainen5210]

How does the reward model score a single action, when it is trained to choose between two actions? Or does the policy model actually generate k actions that the reward model can then score and then choose a reward knowing which action the policy model saw as the most probable one? I'd really appreciate an answer, thanks.

[@albertkwan4261]

This is the best explanation of ChatGPT!

[@DanielJacobsen-s2o]

What is the architecture of the policy model and how large is it? How does it use the pretrained LLM?

[@adosar7261]

@Ari Seff Fantastic video! I have one question for the supervised fine-tuning. Training is essentially next-token prediction just like in the pre-training but with a dataset more specific to the task we want (i.e. chat)?

[@ariseffai]

That's exactly right! The training objective is still next-token prediction (minimizing NLL) in SFT

[@tejshah7258]

Legend has returned - pls make more videos!

[@user-wr4yl7tx3w]

Given that the scores used to train the reward function is small, compared to the universe of potential questions and answers, it's hard to see how a small training set can possibly be sufficient to train adequately. Still amazes me.

[@sethjchandler]

Great job. Going to show this to my class (Large Language Models for Lawyers, University of Houston Law Center)

[@regCode]

I'm having trouble understanding supervised fine-tuning in this context. What are the labels? What is the task?

[@wladefant]

13:12 The new bing (sydney) is able to link sources perfectly now

[@PrafulKava]

Best step-by-setp explanation !

[@bogdanpatedakislitvinov2549]

Very well-made presentation, please make more! Subscribed

[@Bero134-v8i]

Nice clear explanation. Thanks!

[@Francis-gg4rn]

amazing, please make more!

[@EducatedButton]

Thanks a lot for the explaination. How does it work during inference time to keep a conversation back and forth?Is the user's current chat session provided to the model as input along with a new user prompt?

[@ariseffai]

That's right. There's a certain context window of previous text to which the model can attend (on the order of thousands of tokens). This will include both previous user inputs and model responses from the current conversation.

[@SirajRaval]

this is so good, subscribed.

[@yugen3968]

Scammer pos

[@MyMmmd]

I'd love to know more about those "expert" conversations. Do you need to be an expert in the conversation matter or is it just used to make sure it's good at conversing (rather than getting the facts right)? How many of these expert conversations are useful? Is it a case of diminishing returns beyond a certain point? I'm guessing this isn't freely available information but it's fascinating to me.

[@sdsd5450]

Thank you so much! It is such a great video even for beginners!

[@StephenGillie]

ChatGPT is like texting someone using only autosuggest, with most/all of the internet as the database. The real innovation in ChatGPT is compressing most/all of the public internet to just 350 GB, and an open source project has this down to 1.62 GB.

[@D_Winds]

I would like to have this reservoir of data.

[@uestcpandou]

Regard Chatgpt or other LLMs as database is a huge misunderstanding. Same as human brain, LLM is not so good at memorizing, compared to its ability of reasoning and fabricating new things. Combine traditional db and LLM to make them do what they are good at is the only way in the long run.

[@nixedgaming]

Calling transformers with self attention and multiheaded attention “autosuggest” is wildly reductive and borderline disingenuous, even if it’s technically correct

[@StephenGillie]

@@nixedgaming The best kind of correct! Others remind that calling this compression is also wildly reductive and borderline disingenuous, so at least I'm consistent.

[@BooleanDisorder]

Calling it autosuggest is like calling humans autosuggest. "Humans are just autosuggest with all of their lives as the database." People misunderstand what's what. You need to separate the task from what it actually is. The task is next word prediction. How it works is definitely not like autosuggest. The same way my task right now is next word "prediction" while writing this.

[@narendiranchembu5893]

This is a very nice explanation, thanks! What tools do you use to make your videos?

[@ariseffai]

Thanks! For this one I used a combination of keynote & FCP

[@notgabby604]

That's all very high level usage of neural networks. While some people think the basic foundations haven't set yet. Like for example 2 Siding ReLU.

[@DrJanpha]

Codes as training data are only briefly mentioned?

[@HH-mf8qz]

Very good video Can you maybe make an updated version now that chatgpt 4 is released and the new googel gemeni is about to come out for mixel input AIs

[@yugen3968]

Hey, where could I approach you to clear a few things out about this...?

[@baohq]

What is the platform that OpenAI uses to build chatgpt. Like pytorch, tensorflow or something ?

[@FiEnD749]

Dude, your content is incredible!

[@ConceptsWithCode]

Nicely done. Thanks for creating this video. Few quick questions/clarifications. (1) Given the reward model rates an entire response as opposed to each partially complete sentence as tokens are emitted, isn't the final stage also rating the reward for an entire possible sentence (that is terminated by a stop token?). (2) Also was the use of the SFT also in the third stage for KL divergence calculation omitted in the figure because it seemed like too much detail? (3) You mention the upper limit is 3000 words. Is this an approximation for tokenized words that would a maximum sequence length of 8k? (4) Lastly, any idea if the parameters of the model is 16bit float or 32 bit float? Thanks in advance!

[@RixtronixLAB]

Cool video shot, well done, thanks for sharing :)

[@VaibhavShewale]

good insight to how it works learned something new!

[@epeeypen]

i used chatgpt to help we write a love letter and it went really well.

[@brandonojalvo9775]

Your love is a lie

[@anthonydemattos432]

It is possible to do most of this process with just the fine tuning api?

[@juliarose2133]

anyone know what the equation is at 4:08 , where i can find more on it?

[@wiktorm9858]

Ignore this equation - it just means that llm is a function giving xt+1 based on all previous wirds, x1,x2,...,xt. There is no deeper philosophy in it.

[@alonsamuel7106]

Great explanation and naration...! Thanks!

[@dr.mikeybee]

This is a coherent nicely structured explanation of ChatGPT's architecture. Thank you for sharing this. BTW, how likely is it that OpenAI will create a new model with primarily supervised learning? I assume they are curating a new training set from both human responses and model-generated responses. It seems to me that a smallish self-supervised transformer model, trained in an autoregressive fashion from a well-curated knowledge base like Wikipedia and the Encyclopedia Britannica, etc., would be a great start for transfer learning from a curated supervised training set. Your video seemed to suggest this possibility. Moreover, it would be very interesting to run this side-by-side with a different architecture based on a vector database and semantic search for knowledge collection, retrieval, and context building. The results of this could be passed through an LLM for human readability and probabilistic generation. This should result in some sort of fuzzy-verified responses.

[@rickylehr9284]

Why does it care about the reward in reward reinforcement?

[@satishkumar-ir9wy]

Hi, can you make a small video to build ChatGPT with NLP based classification Model.

[@ddystopia8091]

Hello, I want to work in this field. Now I'm a first year student studying informatics, how should I move towards it? Thank you!

[@nchristensen3309]

Is the operations from us as users part of the reward system ?

[@ConceptsWithCode]

Nicely done. Thanks for creating this video. Few quick questions/clarifications. (1) Given the reward model rates an entire response as opposed to each partially complete sentence as tokens are emitted, isn't the final stage also rating the reward for an entire possible sentence (that is terminated by a stop token?). Or do you believe the output sentence is rated for each token emitted until stop token? (2) Also was the use of the SFT also in the third stage for KL divergence calculation omitted in the figure because it was too much detail? (3) You mention 3000 words max limit. Is this an approximation for the max sequence length of 8k tokenized length (4) Lastly, do we know if the model parameters are 16bit floats or 32 bit? Thanks again for making this informative video.

[@AstroPinion]

Thanks!

[@ariseffai]

Thanks Randall!

[@vijayanandpalaniswamy2240]

Excellent insight dude! Awesome work. I need some help on time series algorithms? dataset with multiple parameters. can you help?

[@lij3900]

Hi Ari, really appreciate you made the video! It is great learning experience. Do you mind sharing the transcript on your website as well? For tech stuff, people like me learned better by reading than by watching videos. I tried use the extension to get the video script, but it is not 100% accurate so some tech words are not correct.

[@jaymehta5886]

Nice explaination. Thanks

[@Mike-vj8do]

amazing video Ari. Where is the name from? Israeli?

[@wenderse]

May I ask what technology you used to create such nice explanatory videos? Did you use 3b1b's manim engine? thanks.

[@LSS94]

Very much looks like it!

[@ariseffai]

Not for this one - just keynote and FCP. But I have used manim in a couple other videos :)

[@jeffhayes8543]

Very well presented. Thanks!

[@panashifzco3311]

Well- explained video. So cool!

[@gorgolyt]

Great summary. I didn't follow when you said "we need to model to act during training" as a way of mitigating distributional shift... can you explain some more?

[@ariseffai]

So basically, if the model takes zero actions during training, this means we'll have a big difference between the deployment distribution of states (when the model selects actions itself) and the training distribution (when the model merely observes the human's actions). There are different ways to have the model select actions during training. One is by using a standard reinforcement learning setup, as mentioned in the video. In that case, the policy model is directly rewarded for actions it itself executes. But another possibility comes from "on-policy" imitation learning, such as the DAgger algorithm. We iteratively execute the current policy to gather new training states, but then have an expert provide the correct action labels -- see arxiv.org/abs/1011.0686

[@masoncdyer]

Excellent review

[@Doggieluv25]

This was so helpful thank you!!

[@johnchange5691]

Thank you! Well explained.

[@mentor1013]

Can you please make a video on Midjourney as well?

[@_romeopeter]

the Math and some of the logic went over my head so I'm going to tray and summarise what I think I understood: ChatGPT is built on top it's predecessor InstructGPT which is mathematically trained with large data set to give spit to instructions given. However for ChatGPT, just spitting answers from instructions isn't enough and needs to be retrained over a method called 'Reinforcement Learning' which uses a 'reward model' to rank the next favourable answer. Did I get it? if not, then please tell me what I'm missing but in plain language because I know the above is flawed.

[@sanchi3944]

Lmao this literally what i asked GPT today since I'm making a chatbot on Rasa. Looks like the algos are pointing me in the right direction for once!

[@andramalexh]

PPO= operant conditioning?

[@user-fj9bh7kt7t]

Very good presentation!

[@dwt6273]

Thank you! Very informative!

[@rl6382]

Just wanted to thank you for these videos.

[@posthocprior]

Thanks so much for posting a clear explanation. After watching this, I feel like I do after I've been explained how a magic trick works: disappointed.

[@sjakievankooten]

Love the explanation!! Also thanks for making the video darkmode 😊

[@sjakievankooten]

@@TasteTheStory good videos mate, but no need to spam it here :)

[@TasteTheStory]

@@sjakievankooten Not spaming just trying to connect with people who share the same interest. thanks for your note.

[@peccavius]

Thanks for the talk! You mention that the reward model is trained using cross-entropy loss as a binary classifier. I don't think that's accurate since you don't have a ground truth label for, say, response A (since the score is relative to others). The openAI paper just uses the negative log difference in scores between the higher and lower ranked response as the loss.

[@ariseffai]

You're welcome! That's not quite correct. The classifier is trained to predict which of two responses is ranked higher by the human contractors. Then, the scalar logit output by the trained classifier for an individual response can be used as a reward signal.

[@arijitdas4504]

Absolute gem ❤

[@Black-ww6lj]

Plot twist : Content of this video was generated by chatGPT

[@muidhasan9498]

please make more videos like this

[@roromaniac8]

This was a wonderful explanation! Wouldn't it be expensive to have that much human capital evaluating and simulating chatbot responses? Seems especially so when you consider the wide amount of domains ChatGPT is able to provide reasonably correct responses to.

[@CyberDork34]

Yes it is expensive. OpenAI outsources these tasks to countries like Kenya to save on these costs. It's kind of dubiously ethical but yeah

[@roromaniac8]

@@CyberDork34 do you have a source that I could read about this? I haven’t been able to find something online.

[@jogoeire]

Tip: As a KZbinr, buy yourself a decent recommended direction microphone. Your mic picks up background noise and echos. Only saying this cos your content is so good.

[@someguyfromafrica5158]

I believe this since its very clear that ChatGPT is left-leaning. The dumbed-down model (175B parameters to 1.3B) was also preferred which says everything I need to know.

[@user-wr4yl7tx3w]

Great content, Thanks

[@aethermass]

Great explanation.

[@stephenthumb2912]

a bit amazing how the hallucinations begin, so similar to a human caught in a lie or imagination, the lies built on lies get progressively more absurd in the same way that an untruth from a human where it gets more and more difficult and outlandish to make up a reason based on a stack of false premises.

[@icyou8496]

good explanation!! i just wandering how / what chatgpt threshold for displaying no results? i have observe something like this for example : me : (example of 1 non professional gamer) chat gpt : i dont have enough data for him me : (example of 1 professional gamer in same game) chat gpt : *explain professional player* me :(asking the first player) chat gpt : *explain about that non professional gamer*

[@AnthatiKhasim-i1e]

"It's fascinating to see how AI optimizes Facebook interactions! From personalized recommendations to automated moderation, AI is changing the game. Have you explored AI features like chatbots yet?"

[@juanmanuel8464]

Great content!

[@mr.rndmguy]

I'm learning about it's trained models and it's inner functions, just to create a perfect Jailbreak. thanks

[@BetaTester704]

According to ChatGPT it's memory is limited to only one prior message in the same conversation, beyond that it can't remember anything.

[@moheuddin_sehab]

that was awesome