The End of Finetuning - with Jeremy Howard of Fast.ai

Fast.ai’s “Practical Deep Learning” courses been watched by over 6,000,000 people, and the fastai library has over 25,000 stars on Github. Jeremy Howard, one of the creators of Fast, is now one of the most prominent and respected voices in the machine learning industry; but that wasn’t always the case... Read the full show notes here: www.latent.space/p/fastai
0:00:00 Introduction
0:01:14 Jeremy’s background
0:02:53 Founding FastMail and Optimal Decisions
0:04:05 Starting Fast.ai with Rachel Thomas
0:05:28 Developing the ULMFit natural language processing model
0:10:11 Jeremy’s goal of making AI more accessible
0:14:30 Fine-tuning language models - issues with memorization and catastrophic forgetting
0:18:09 The development of GPT and other language models around the same time as ULMFit
0:20:00 Issues with validation loss metrics when fine-tuning language models
0:22:16 Jeremy’s motivation to do valuable work with AI that helps society
0:26:39 Starting fast.ai to spread AI capabilities more widely
0:29:27 Overview of fast.ai - courses, library, research
0:34:20 Using progressive resizing and other techniques to win the DAWNBench competition
0:38:42 Discovering the single-shot memorization phenomenon in language model fine-tuning
0:43:13 Why fine tuning is simply continued pre-training
0:46:47 Chris Lattner and Modular AI
0:48:38 Issues with incentives and citations limiting innovation in research
0:52:49 Joining AI research communities through Discord servers
0:55:23 Mojo
1:03:08 Most exciting areas - continued focus on transfer learning and small models
1:06:56 Pushing capabilities of small models through transfer learning
1:10:58 Opening up coding through AI to more people
1:13:51 Current state of AI capabilities compared to computer vision in 2013 - lots of basic research needed
1:17:08 Lightning Round

[@vaibhavnakrani2983]

I see Jeremy I hit like! Beautiful soul. We are getting ready so that we can take the baton from you for accessible learning.😄

[@WilliamHuster]

Fantastic. I learned a lot. Thanks everyone.

[@ds920]

Agree, absolutely!❤ I love that description of the state of things. But I did believe it would work, I just haven’t got enough dedication to try it actually for several years in 2017-18. And I’m not sure still we’ve cracked it, but if we did it’s pretty sweet to think now, who the hell we are😂😂😂

[@KevinKreger]

Fire and ice! Plus great content 🙂 Jeremy is inspiring.

[@ds920]

Brave statement in the end, it should go to title.

[@abhimanyuyadav2685]

Jeremy’s videos are doing the best on the channel

[@paulgaddis4329]

I'll make the analogy that we are making "swords". The tuning is the forging, the data, the materials of the sword. The prompting is the wielding of the sword. Give a swordsman of 50 years and a bronze sword, they are still a force to be reckoned with. Give him a titanium blade made to his measure and form, and all can tell at a glance the dance that would be dealt called death as soon as his hands grip the hilt.

[@Dis-Trackted]

Great title. You got me. I subscribed and I’ll keep an eye on the channel.

[@zacharydaniels3186]

Fantastic interview guys. Let's get him and George Hotz in the same room somehow...

[@LatentSpaceTV]

I know right? The Latent Space Summit with all speakers in person would be fun :)

[@mshonle]

It’s a good strategy for professors to allow students who ask for extensions to just grant them, because so often the student never follows through, so it turns out to be very little work for the professor to seem reasonable/accommodating.

[@rufex2001]

Suuuch an awesome dude! Inspiring!

[@kirubeltadesse4364]

I learned a lot from the talk thank you :)

[@mungojelly]

it's amazing what it turns out you can do w/ tiny models, but uh, i don't see how it helps very much as far as distributing power,, small models being powerful doesn't only help people who that's the only model they can afford to run, it's also super cool if you've got a zillion dollars to run models, makes your agent swarms way more efficient and effective, very synergistic w/ the benefits of scale, gives your big models lots of little friends to think fast and wide w/ & you've still got the big models that can go deep,, in comparison someone at home can do one little thing fast, which is just a cool toy, especially in a world where more powerful people can summon swarms of trillions of models at once & their abilities set a standard for what people know ai is capable of

[@okkrish]

He mentioned about some Discord channels, is it possible to list them on the description?

[@LatentSpaceTV]

here! github.com/swyxio/ai-notes#communities

[@widezyp2228]

Nice interview. But the chapter timings are all off by quite some time. Always have to go back 1-2min.

[@LatentSpaceTV]

I built github.com/FanaHOVA/smol-podcaster for chapters but sometimes it picks up the END of the question's timestamp for the chapter. I'll update the prompt to make sure it doesn't!

[@TheZXspeed974]

Great episode like always. Maybe it needs a cut at 4:07

[@LatentSpaceTV]

Jeremy and Rachel are actually married, so he was referring to her!

[@sherwyndaman]

Have you seen NEFTune

[@JumpDiffusion]

ништяк

[@brytonkalyi277]

★ I believe we are meant to be like Jesus in our hearts and not in our flesh. But be careful of AI, for it is just our flesh and that is it. It knows only things of the flesh (our fleshly desires) and cannot comprehend things of the spirit such as peace of heart (which comes from obeying God's Word). Whereas we are a spirit and we have a soul but live in the body (in the flesh). When you go to bed it is your flesh that sleeps but your spirit never sleeps (otherwise you have died physically) that is why you have dreams. More so, true love that endures and last is a thing of the heart (when I say 'heart', I mean 'spirit'). But fake love, pretentious love, love with expectations, love for classic reasons, love for material reasons and love for selfish reasons that is a thing of our flesh. In the beginning God said let us make man in our own image, according to our likeness. Take note, God is Spirit and God is Love. As Love He is the source of it. We also know that God is Omnipotent, for He creates out of nothing and He has no beginning and has no end. That means, our love is but a shadow of God's Love. True love looks around to see who is in need of your help, your smile, your possessions, your money, your strength, your quality time. Love forgives and forgets. Love wants for others what it wants for itself. Take note, love works in conjunction with other spiritual forces such as faith and patience. We should let the Word of God be the standard of our lives not AI. If not, God will let us face AI on our own and it will cast the truth down to the ground, it will be the cause of so much destruction like never seen before, it will deceive many and take many captive in order to enslave them into worshipping it and abiding in lawlessness. We can only destroy ourselves but with God all things are possible. God knows us better because He is our Creater and He knows our beginning and our end. Our prove text is taken from the book of John 5:31-44, 2 Thessalonians 2:1-12, Daniel 7-9, Revelation 13-15, Matthew 24-25 and Luke 21. Let us watch and pray... God bless you as you share this message to others.

[@paul_pierre]

F1rst