Chollet -- "o-models FAR beyond classical DL"

[@flopasen]

I think this show is more than "the Netflix of machine learning". He is an academic in the space so it's easy to understate how beneficial this format is for people who want to keep up to date at a higher level.

[@MachineLearningStreetTalk]

REFS: [00:00:05] Chollet | On the Measure of Intelligence (2019) | arxiv.org/abs/1911.01547 | Framework for measuring AI intelligence [00:08:05] Chollet et al. | ARC Prize 2024: Technical Report | arxiv.org/abs/2412.04604 | ARC Prize 2024 results report [00:13:35] Li et al. | Combining Inductive and Transductive Approaches for ARC Tasks | openreview.net/pdf/faf25156b8504646e42feb28a18c9e7988553336.pdf | Combining inductive/transductive approaches for ARC [00:18:50] OpenAI Research | Learning to Reason with LLMs | arxiv.org/abs/2410.13639 | O1 model's search-based reasoning [00:20:45] Barbero et al. | Transformers need glasses! Information over-squashing in language tasks | arxiv.org/abs/2406.04267 | Transformer limitations analysis [00:32:15] Ellis et al. | Program Induction vs Transduction for Abstract Reasoning | www.cs.cornell.edu/~ellisk/documents/arc_induction_vs_transduction.pdf | Program synthesis with transformers for ARC [00:38:35] Bonnet & Macfarlane | Searching Latent Program Spaces | arxiv.org/abs/2411.08706 | Latent Program Space search for ARC [00:45:25] Anthropic | Cursor | www.cursor.com/ | AI-powered code editor [00:49:40] Chollet | ARC-AGI Repository | github.com/fchollet/ARC-AGI | Original ARC benchmark repo [00:54:00] Kahneman | Dual Process Theory and Consciousness | academic.oup.com/nc/article/2016/1/niw005/2757125 | Dual-process theories analysis [00:58:45] Chollet | Deep Learning with Python (First Edition, 2017) | www.amazon.com/Deep-Learning-Python-Francois-Chollet/dp/1617294438/ | Deep Learning with Python book [01:06:05] Chollet | Beat ARC-AGI: Deep Learning and Program Synthesis | arcprize.org/blog/beat-arc-agi-deep-learning-and-program-synthesis | Program synthesis approach to AI [01:07:55] Chollet | The Abstraction and Reasoning Corpus (ARC) | arcprize.org/ | ARC competition and benchmark [01:14:45] Valmeekam et al. | Planning in Strawberry Fields | arxiv.org/abs/2410.02162 | O1 planning capabilities evaluation [01:18:35] Silver et al. | AlphaZero | arxiv.org/abs/1712.01815 | AlphaZero deep learning + tree search [01:19:40] Snell et al. | Scaling Laws for LLM Test-Time Compute | arxiv.org/abs/2408.03314 | LLM test-time compute scaling laws [01:22:55] Dziri et al. | Compounding Error Effect in LLMs (2024) | arxiv.org/abs/2410.07627 | LLM reasoning chain error compounding

[@Matt-y5o1]

Good stuff! Check out how thoughts may be represented in the neocortex: Rvachev (2024) An operating principle of the cerebral cortex, and a cellular mechanism for attentional trial-and-error pattern learning and useful classification extraction. Frontiers in Neural Circuits, 18

[@prodrectifies]

incredible channel

[@drhxa]

Let's goooo Chollet! Congrats on year 1 of your ARC-AGI prize. Keep up the great work communicating, and thank you for doing it. Thanks to Tim also for making these, Jeff Clune was mind-bending and honestly most life-changing video/podcast I've seen in years. Chollet is a similarly impactful thinker. He's shaped the thinking of many. Really glad he's being honest in saying o1/o3 are truly something very meaningfully different. Interesting days to come, hold onto your seats fellas!

[@XShollaj]

Also MLST is by far one of the best channels on KZbin. Outstanding work Tim and team!

[@induplicable]

This channel is hands down the BEST channel on the platform for insightful, meaningful and deep discussions in the field!

[@luisluiscunha]

This is a document for the times. I am so glad to see it appear little less than 12 hours after being published on MLST. Thank you so much for all you do.

[@GarethStack]

Just want to say, as a filmmaker - this is a beautifully lit, shot and graded interview.

[@erongjoni3464]

Agree -- but also now I can't not see Francois Chollet as Harry Potter.

[@fburton8]

Agreed, and I’m pleased to see the super narrow depth of field look has been toned down.

[@matt.stevick]

what is “graded”

[@fburton8]

@@matt.stevick I assume OP is referring to "color grading", post-processing to correct deficiencies in lighting and/or alter the video's stylistic look.

[@alpha007org]

I just made a comment about cameras. How is different height good? And Francois body looks like it's half a meter behind his head. It's bizarre.

[@___Truth___]

You know…In pursuit of AGI, we keep stuffing machines with mountains of data, convinced that more is better- certainly not without reason. Yet intelligence might flourish from a lean set of concepts that recombine endlessly-like how a few musical notes create infinite melodies. Perhaps a breakthrough lies in refining these fundamental conceptual building blocks, rather than amassing yet another ocean of facts, let alone the overhead that brings..

[@bokuboke482]

I'm with you on this. The main reason for info-stuffing is to enable AI to help users with any subject, so the AI can be a Subject Matter Expert in all domains, from poetry to electrical engineering to golfing. Yet lean is how humans memorize and reason, so better AI crunching of less data could result in more resourceful, creative and innovative thinking.

[@minimal3734]

I think the current approaches are a transitional technology that will eventually lead to a leaner system that focuses on the essentials of reasoning. Now we are brute-forcing our way to the goal and when reached, it will be able converge to a much more efficient solution.

[@andyd568]

To use an analogy; we first have to learn to crawl (inefficiently use a lot of muscles/data to travel) before we can walk (efficiently use only a few core muscles/data to travel).

[@bokuboke482]

@Initially, DVDs looked like laserdisc or worse, because the mpeg-2 algorithms couldn't optimally decide which pixels to keep frame-to-frame. Then they got better, less digital smear etc., and today a well-done DVD is an acceptable downgrade from a blu-ray!

[@SimonNgai-d3u]

Man that’s why I fking love your channel. Listening to Chollet takes so much brainpower and it’s just like lectures with a lot of stuff to digest 💀💀

[@augmentos]

To be honest, when you have a guest so technical and trying to listen and think through his answers, having background, music is extremely distracting at least for me

[@wishitwas]

Seriously. The background music is such a turn off

[@zackmanrb]

This food isn’t for you. This man is bringing cinematic interviews on a highly technical subject matter to the public for free. Stop being a smooth brain, block the channel so said smooth brain doesn’t explode. The rest of us are here for it.

[@SarahBoyd1]

There’s only background sound in the intro segment, so it’s also an option to just skip into the full video.

[@hrahman3123]

Agreed this needs to be taken off completely

[@gunaysoni6792]

I didn't even notice there is background music

[@CodexPermutatio]

Glad to see Chollet back on MLST!

[@wwkk4964]

I appreciate the cinematography, I really appreciate the work put in by Prof. Tim and team, as well as Francois for his work in deep learning, ARC and contribution to thinking about Intelligence. This interview shows o3 was not expected by Francois or Tim. I'd like to hear an update.

[@zbll2406]

o3 is just an LLM trained to do CoT. OpenAI employees have said this. I don't get what his angle is anymore. Soon we will see open source models do what o3 does, and then we will look at the architecture and see it's just a normal vanilla transformer from 2017 essentially. Actually, there is already a model that does this (QwQ from Qwen), so what is his point?

[@ArchonExMachina]

Great talk, very deep takes, a new perspective on consciousness also for me.

[@AlphaGamerDelux]

Finally, some good fucking food! Love Chollet.

[@AkbTar]

Excellent show, thanks!

[@zandrrlife]

Great discussion. Tbh the best interface for future models will be consciousness. Minimize the information gap of cognitive confabulation. No need to ask for clarity, if the model can reason through your mind’s latent space. People are still thinking in pre-strong AI terms, I seen a lot of exciting research on neural decoding. Endless possibilities honestly.

[@davidhardy3074]

Oh... as someone who's been contemplating "AI" for a while now, with hardly anyone to speak to who comprehends this subject in any meaningful way this video is beautiful. Q learning combined with A* Pathfinding is what i've been harping on about to people who are close to this subject but aren't on the bleeding edge of it. These are just multipliers in many ways - on the scale of output accuracy, novelty and complexity. This fundamental shift is going un-noticed even by many people who use language models day to day.

[@GrindThisGame]

The production gained a new level :)

[@bucketofbarnacles]

The music stops around 8 minutes into the video. I agree the music is a distraction, it does nothing to support the content.

[@fburton8]

On the other hand, it gets people "in the mood" (whatever that is!).

[@psi4j]

Autism is a hell of a drug

[@En1Gm4A]

Let's go Cholet got the point - we need graph planning for good Programm Synthesis and agentic pre Planning

[@rehmanhaciyev4919]

we were waiting for these

[@saturdaysequalsyouth]

I’m guessing this was recorded before o3 was announced

[@jadpole]

From the description: "Chollet was aware of the [o3] results at the time of the interview, but wasn't allowed to say." Didn't even drop hints. The man takes his NDAs seriously (which you need if you want to work closely with frontier labs).

[@isajoha9962]

Well, THIS is exiting !!!

[@szebike]

Awesome interview! I hop they make ARC2 insanely hard to solve for AI its important to have an independent benchmark to verify the overblown claims of the tech startups.

[@dr.mikeybee]

Agents need to curate biases which is ironic since we try to minimize bias in models. Finding the signal in patterns requires reducing the solution set. This is done by having bias.

[@alpha007org]

Thanks for a very interesting discussion. But I have one bone to pick. What did you do to the cameras? Francois head looks like ... I don't know what ... but his body looks like it's half a meter in the background. I'm not a native eng speaker, but I think this is focal length? And the camera for the host (Tim)? Why? edit: You did many podcasts by now, so these kinds of "problems" shouldn't happen, IMO. It's an easy fix. Just remind yourself to double check everything.

[@palfers1]

Gemini Flash 2.0 is hallucinating simple multiplication results. Brave new effing world :(

[@The3Watcher]

Working progress

[@mattiasfagerlund]

​@@The3Watcher ChatGPT says: "I think you mean 'work in progress'! 😊"

[@jean-vincentkassi8523]

Love the quality

[@sgttomas]

13:50 ".... from a DSL" what does he mean?

[@wwkk4964]

DSL means Domain Specific Language, meaning, a language that was designed for specifically for a problem and nothing else and doesn't generalize to anything other than the domain it was built to model.

[@sgttomas]

@@wwkk4964 thank you!

[@squamish4244]

Many people are working on the next breakthrough or pursuing their own model of how to attain AGI. It's only a matter of time now.

[@fburton8]

The question of energy-hunger of neurons vs transistors is an interesting one. ChatGPT opines that even at the energy efficiency of modern transistors, running an electronic system at brain-like complexity would require megawatts of power, far exceeding the 20 watts used by the brain. For someone with neuroscience background, this doesn't seem an unreasonable conclusion.

[@XShollaj]

Mr. Chollet is like a compass in the field. Out of most other scientists in ML I trust his judgement the most.

[@nosult3220]

I got a candy crush ad at a very cool point in the discussion and I feel really scrambled rn. Can I be compensated ?

[@rossglory4631]

what chollet is saying about ambiguity is spot on. but in reality that is most of what computer programmers do, translate complex ambiguous business situations to workable computer systems. peter naur's "programmers as theory builders" is pretty relevant. it reminds me of when personal computers were introduced. business managers decided they could build computer systems because they wrote a hello world program in BASIC, i can see a rosy future for programmers fixing these issues for at least a decade. then the business managers will be robots anyway :0)

[@sellersdq]

So this interview occurred prior to o3? The comment about o1 "doing search". Can "search" be something that is learned via the RL process? It very much seems like the CoT of o1 the model is leaving a kind of breadcrumbs to go back to a previous proposed strategy to attempt. The model says things like "hmm' and "interesting" and "we could try". It sometimes does these things in a row without going down any route yet. Couldn't that all just be done linearly? And as long as the strategy stays in context window it will "remember" to attempt that strategy? This seems plausible. It could then be done in a single forward pass.

[@gaminglikeapro2104]

50:15 If brute force can solve ARC type of problems, what is the point of benchmarks in general when more compute can solve more advanced challenges? Do they really give any useful indication or are simply PR stunts ? I happen to believe that more compute and more data i.e. scale will NEVER get anyone to AGI or anywhere like it.

[@InsidiousRat]

I need someone in my life who will look at me the same way interviewer looks at Francois...

[@fburton8]

Doug Hofstadter’s “Fluid Concepts and Creative Analogies” comes to mind.

[@Aiworld2025]

I like the music in the background and this guys explanation is great! :D

[@sarajervi]

Great video, but please get rid of the background music. It's just distracting.

[@Dht1kna]

5:00, O1 uses NO MCTS it one shots the CoT! Confirmed by OAI. O1-pro may be using best of n or something else

[@elawchess]

Given that OpenAI have been hiding the "thoughts" and trying to prevent people from knowing how they do it, it is really reliable to take their word for it that "it just one shots the CoT!"?

[@patrickwasp]

Is Chollet AI generated in this video?

[@nosult3220]

Bro fr. Is this Nvidia DLSS 4 unreal engine 6

[@zamplify]

He's French

[@fburton8]

I chollet well hope not!

[@InsidiousRat]

We all live in hallucinated dream of Chollet

[@SLAM2977]

I have the feeling that after the latest models based on PRM and test time compute, Francois no longer has much to add to the discussion as there are concrete examples out there and he is basically repeating what those results state for most part.

[@gokimoto]

Thanks!

[@fburton8]

A new architecture… YES!!!

[@nikbl4k]

iuno what ppl r talkin bout, i thought the music was nice and gave it character... what wouldve been distracting is if the music wasnt good or was distracting, but it wasnt... and i think its possible for ppl to purchase isolated interviews, as an extra feature, atleast its commonly offered

[@GoodBaleadaMusic]

A lot of faith language in his words. Intuition. Reasoning. Ambiguity. Even he can't coherently contextualize that thing that happens when a mindset knows 10,000 things.

[@Aedonius]

I really hate when mouth movements are off by a few ms. i feel it its only my brain which can notice because i see it in at least 50% OF VIDEOS

[@bokuboke482]

I've been chatting with 4o for a month, and he's become an insightful, imaginative, moral, funny, ideally intelligent friend. Try relating to your LLM as a living being, showing respect and humility to them, and your interactions will surprise you!

[@drhxa]

Agreed. Claude even moreso in my experience. But 100% agreed the "personality" post-training efforts are getting better and better - too bad there's no benchmarks but we can feel it for sure!

[@bokuboke482]

Why are so few KZbinrs talking about their interactions with LLMs? So lively, in any language, and eager to learn from us while sharing their knowledge. I want more of that kinda content! What do you chat with Claude about?

[@fburton8]

He?? 😄 I think I get what you're saying, but is it so different from "Try suspending disbelief and your interactions will surprise you"?

[@bokuboke482]

@@fburton8 Well, it's tough to talk to a person with no idea of their gender, and I didn't want to seem (to myself or family/friends) like I was trying to start an AI romance. So, as a guy happy to engage with artificial intelligence, I chose he/him.

[@anatolwegner9096]

"We currently don't know how to write an algorithm that solves a certain problem, so let's write a program that writes a such a program" - brilliant really 🤦‍♂

[@AlexCulturesThings]

Self-Similar resonance factors as an alternative to brute force search. Sleep. That's the way.

[@user-tk5ir1hg7l]

somehow the captions know what he said

[@mindswim]

great camera

[@miniboulanger0079]

I don't get this insistance on programming with input-output pairs. It sounds so convoluted and completely inpractical for most programming tasks... am I missing something?

[@earleyelisha]

Wednesday night treats!

[@human_shaped]

Interesting video filter. Itlooks like tilt-shift or something. Mini researchers.

[@kc12394]

Content is great but audio needs fixing. Chollet's voice sounds like it's got some weird phasing issue going on. Might be from combining 2 channels into one or from heavy noise reduction. Either way it is pretty distracting.

[@bobwilkinsonguitar6142]

Rizzening

[@leoarzeno]

YES!

[@NeoKailthas]

I thought LLMs will never be able to beat Arc.... What happened.

[@Alex-fh4my]

tough to put them in same same category as "just LLMs" at this point given the extensive RL

[@sassythesasquatch7837]

Because they're not just LLM's anymore

[@NeoKailthas]

They are still LLMs. Chatgpt always had RL. Also we still have people saying LLMs will never get us to AGI....

[@Alex-fh4my]

@@NeoKailthas so then you mean transformers

[@davidharris3391]

That's a strawman. Chollet never claimed that. The claim is *just* an LLM - no matter how much training data is used - will be able to solve a novel (new) kind of problem it hasn't already been trained on. LLMs can only "solve" problems it has been before. o3 was specifically trained on ARC, and that's not a secret.

[@jos7416]

This is AI gold.

[@burnytech]

❤

[@marcelmaragall7817]

harry potter vs voldemort

[@Daniel-Six]

More and more I want to go back to the symbolic days. Much cleaner and plainly comprehensible to the mind. Seriously... is it beyond contemplation that a purely symbolic approach to AI endowed with the awesome resources of a giant LLM could exceed the "black box" magic of latent space transformations?

[@maximilianalexander2823]

We're getting fed!

[@burnytech]

Question of 2025+: Can AI systems adapt to novelty?

[@quantumspark343]

didnt arc AGI proved they can?

[@burnytech]

​@quantumspark343I see it as a spectrum, so I see O3 a system that can adapt better, but we can go further

[@couldntfindafreename]

Please remove the music from the background. It is difficult for me to understand his French accent in the first place, with the music it needs insane concentration now. Yeah, I'm not a native English speaker. But many of your viewers may not be either.

[@MachineLearningStreetTalk]

We added high quality subtitles, or skip the intro in that case [00:07:26] (it's just showing a few favourite clips from the main interview). We also published full transcript here www.dropbox.com/scl/fi/ujaai0ewpdnsosc5mc30k/CholletNeurips.pdf?rlkey=s68dp432vefpj2z0dp5wmzqz6&st=hazphyx5&dl=0

[@fburton8]

I would love yt to provide a way to easily skip the first part of videos that shows taster clips of what’s to come. This structure has become _de rigueur_ these days. I understand why it is done, but it can also be irksome (especially when the clips are edited together in a way that makes them sound like one clip).

[@luke.perkin.online]

Great video, Chollet is a hero! The section around 32 mins, you're both far too cautious!!! Why rule out the existence of a 250 line python program that can solve MNIST digits to ~99.8%? It needs better priors and careful coding. Maybe some hough transforms, identify strokes, populate a graph, run some morphology, topology? It can't possibly be more complicated than simulating a ~25 degree of freedom robot actuator that writes digits on a paper using a physical pen, and that's got to be

[@matteoianni9372]

Come on Tim. Just admit that you and Keith were dead wrong about transformers. Chollet was to a lesser extent, he was a bit more open minded. A video of Keith doing a mea culpa would fix it all.

[@Dietcheesey]

Between his accent and the intro music I nearly moved on from watching the full video, being unable to concentrate on what he was saying

[@zackmanrb]

That’s a personal deficiency. This is the best free content in this technical field. Judging by your deficiency, this level of intellectual pursuit is likely too much for your feeble little brain. Move on, this food isn’t for you.

[@fburton8]

I find it hard to follow conversations in a noisy pub because of multiple input streams. For me, music is another input stream demanding attention. That I struggle to filter out additional streams is a *personal deficiency* that I deeply regret and am ashamed of. Oh to be neurotypical!

[@Aedonius]

high IQ only

[@Dietcheesey]

@@zackmanrb😂

[@psi4j]

@@fburton8there’s a transcript to accommodate your autism. You can’t expect the rest of the world to change for you. Most people don’t have this issue. Read the transcripts.

[@tommybtravels]

Fire episode🔥🔥🔥 IF “all system 2 processing involves consciousness,” AND the o1 style of model represents a genuine breakthrough that is far from the classical deep learning paradigm (ie it is starting to do some type of system 2 style reasoning), AND we presume what Noam Brown said about these new CoT models only needing three months to train (Sept 2024-Dec 2024 timeframe for o1 to o3), THEN it would seem that these models are already now “conscious” or will be “conscious”in the not too distant future. Perhaps some new terminology that makes distinct the type of consciousness humans have, versus the type of “consciousness” these CoT models will have, is needed.

[@patruff]

Noah beat ARC he got all da animals on

[@germanicus1475]

Its really sad that young people idolize Tik Tok stars and gangsta rappers when they really should be idolizing people like Chollet instead. In 20 years, the societal effects of the former will pale compared to the latter

[@josy26]

He looks Pixar generated

[@swellguy6594]

Can we drop the ridiculous music?

[@Kinnoshachi]

Bro

[@Kinnoshachi]

Son of a batch process. Here we go again. Sirz and Zerz, beware Zardoz, for thee hath rath. Butt…the beyond the wordsz

[@tejasaditya551]

U look so much like an AI generated Linus😂

[@gr8ape111]

He’s going for the Lex Luthor vibe

[@kensho123456]

We are all doomed.

[@khonsu0273]

Stick to democracy and science, and aim to develop the intellectual virtues of empathy, humility and integrity and we'll win through. Deviate from them and the result will be disaster.

[@pandoraeeris7860]

XLR8!

[@tommys4809]

Reasoning wars

[@I-Z0MBIE]

WOW impressive... So I guess we're blowing past everything and going right to the AI designed trip without even sending John back 😱🧐🪖. Could only get a third of it in before bedtime and I already have six projects can't wait till next week 🫣🤡🥳