Train your own language model with nanoGPT | Let’s build a songwriter

[@EkShunya]

i love the ml community. Everyone is sharing information :)

[@foreignconta]

You know there is a new optimizer in your name. :) I wrote my nanoChatGPT from scratch and had to clip its context size to stop ooming my GPU memory. The only thing it does with 8 decoder layers is to output nonsense. Thinking about training it on a better instruct dataset. I am using Sentencepiece tokenizer. Subscribed.

[@SophiaYangDS]

Yes the Sophia optimizer! And thanks for the great insight 🙏

[@foreignconta]

@@SophiaYangDS You're welcome. I am looking forward to your videos.🙏

[@xspydazx]

Also thanks ! As after watching this video I also re-watch kaparthy and recreated everything and it worked ok for a text generation model .

[@TerrorByteTW]

Totally going to download by Twitter and Facebook data to train a model that talks like me lol. Let's see how this goes!

[@SophiaYangDS]

Sounds fun! Good luck!

[@TheSterg98]

How did it work out?

[@F336]

Yea his video was like a missing link explained perfectly...This one is nice too...:)

[@Avra_b]

Wow Sophia , happy to see quick concise 15 mins on “how to implement nano-GPT” . Super helpful . 🎉😊 Btw , by any chance , did you try the lyrics / or any other datasets with more iterations ? How was the result ?

[@kilianlindberg]

I concur, kudos for the impressive work showcased in this video! On a side note, I experimented with Nano-GPT a few weeks back and found it quite user-friendly, thanks to the clear instructions provided on its GitHub page. I successfully ran it on both a 2018 iMac (running Big Sur) and a 2019 MacBook Pro (equipped with Ventura), each with 16GBs of RAM. The MacBook Pro's GPU offered a slight speed advantage. Undoubtedly, Nano-GPT conveys potential for more specialized/dedicated models running on a local machine.

[@SophiaYangDS]

Thanks so much Avra 🤗! Yes I did try it a little more and it looks better after 1k iterations. I should have trained more to show in the video 😅

[@SophiaYangDS]

​@@kilianlindberg Thanks so much Kilian! Yes nano-GPT is so user-friendly. Andrej Karpathy is a legend! I'm so glad I finally watched his video and tried out nano-GPT.

[@adamstewarton]

​@@SophiaYangDS how long roughly did it take?

[@fab_spaceinvaders]

came from its video and learned more this morning than in one week hands on 13b stuff 😂

[@johnnyjax6444]

Can we train this Model on any thing , eg analyze the our project Code base by same way feed our source code

[@iDineshKrishnan]

Great! Thanks for doing this video, very informative.

[@SophiaYangDS]

Thanks so much for your support 🙏

[@AllenLSY]

6:40 on m3 max, each iter runs 14ms. Sophie, ask your company to give you a better laptop :)

[@curioussoul5151]

Thank you so much for this video! It was very helpful

[@SophiaYangDS]

thanks for the kind words 🙏

[@seattledude2022]

Andrej just told me that he likes your video :)

[@Ashton351]

Fantastic, thank you so much.

[@宋保兴]

wow thx, i learn a lot from this video

[@imai_official]

--start is promt? also Sophia please tell, will it work for other language?

[@amparoconsuelo9451]

If I will order from Amazon a GPT assembly kit, what would it deliver me? How much would the kit cost?

[@venkatreddy-px8fm]

Please, can you let me know if you connected your GPU to MAC M1 laptop or other source ?

[@danasugu1767]

Hi, Sophia, thanks for this tutorial. Instead of a txt file could we used a pdf file?

[@PotatoMagnet]

PDFs are not very machine readable text. TXT is best. Maybe use a pdf parser?

[@DaveisLearningtoPlay]

i had this on in the background and all of the sudden heard the applause track when it generated some content. that was very funny. ;)

[@xspydazx]

Sorry but how to save the final model produced for deployment ? .. then how to fine tune this model for a downstream task such as question and answer ?

[@mshonle]

The “3” in the list of tokens was conspicuous. Turns out it’s a left over page heading.

[@jmisc]

can this be done in a chatGPT way, where we say, here is the lyric, revise it, or here is a song about potato, now write a lyric about tomato in the same style?

[@datahacker1405]

It can be done with langchain prompting and custom llm wrappers

[@AryanSingh-py7sm]

i want to ask one doubt. cant we get an output from a query we write and not all the decoded data?

[@SunGlassesBlue]

Thank you ! You Saved my day ~~

[@sw-ln1hh]

Thank you for your video

[@lutfiikbalmajid]

Can i train this in Mac Air M1?

[@bogdanbarbu363]

At 8:50 you say that you can pass the data type as a parameter. How do you do that?

[@og_23yg54]

What if your watching this video you should know that before h came here go learn python then come back

[@colinmaharaj]

Can this be done in C++?

[@yusufcan1304]

Edited. thanks

[@fractal3588]

can this be a totally offline process?

[@SophiaYangDS]

Yes

[@spencerfunk6697]

so this is using smaller lang models? could you train something even smaller like koala lol?

[@RedCloudServices]

Sophia if we can load lots of text into langchain and vectordb and have chatbot why do we still need to fine tune a LLM?

[@SophiaYangDS]

Hi great question! Check out my latest video interviewing Lewis Tunstall. He talked about zero-shot, few-shot learning vs fine-tuning.

[@rockstreamlive]

is nano gpt offline or need internet for working?

[@SophiaYangDS]

Offline. That's the beauty of nanoGPT : )

[@sonkyung3297]

a tutorial of a tutorial -- lol -- much thanks!! 🙏

[@HANKTHETANKAD11]

Do ai Poe chat how generation image and lyric on there can you do review on ai Poe chat

[@axystorm5269]

does anyone have any idea how to get this working with windows?

[@shihuacheng5429]

WSL2

[@pythonfamily]

Dollar sign hahahaha