AI Won't Be AGI, Until It Can At Least Do This (plus 6 key ways LLMs are being upgraded)

The clearest demonstration yet of why current LLMs are not just ‘scale’ away from general intelligence. First, I’ll go over the dozen ways AI is getting murky, from dodgy marketing to tragedy-of-the-commons slop, Recall privacy violations to bad or delayed demos. But then I’ll touch on over a dozen papers - and real-life deployments, of LLMs, CNNs and more - making the case that we shouldn't throw out the baby with the bathwater. Crucially, I'll cover 6 key approaches that are being developed to drag LLMs toward AGI. this video will hopefully, at the very least, leave you much better informed on the current landscape of AI.
AI Insiders: / aiexplained
ARC-prize: arcprize.org/?...
ChatGPT is Bullshit: link.springer....
Apple Cook Interview: www.washington...
AI Slop: taplio.com/gen...
www.npr.org/20...
OpenAI Erotica: www.theguardia...
AI Toothbrush: www.oralb.co.u...
Recall Hackable: www.wired.com/...
Murati: ‘Few Weeks’: • Introducing GPT-4o
Medical Studies: x.com/JeremyNg...
BeMyEyes Demo: • Be My Eyes Accessibili...
BrainoMix: www.ncbi.nlm.n...
www.gov.uk/gov...
Chollet Clips and Interview: • Francois Chollet On LL...
• Francois Chollet recom...
• Francois Chollet - Why...
Gartner Hype Cycle: s7280.pcdn.co/...
Jack Cole: x.com/Jcole75C...
Methods: lab42.global/c...
lab42.global/c...
Ending Animal Testing - GAN: www.nature.com...
www.bbc.co.uk/...
Virtual Rodent: x.com/GoogleDe...
Google AI Overviews: www.nytimes.co...
Noam Brown Optimism: x.com/polynoam...
MLC Human-like Reasoning: www.nature.com...
Many Shot Google Deepmind: arxiv.org/pdf/...
Automated Process Supervision: arxiv.org/pdf/...
No Zero-shot Without Exponential Data: arxiv.org/pdf/...
Prof. Rao Paper 1: openreview.net...
DrEureka: eureka-researc...
AlphaGeometry: deepmind.googl...
Graph Neural Networks Joint: arxiv.org/pdf/...
Terence Tao: Tacit Data: www.scientific...
Mira Murati on Models: x.com/tsarnick...
Chollet Tweet: x.com/fchollet...
Non-hype Newsletter: signaltonoise....
GenAI Hourly Consulting: www.theinsider...
Need an GenAI app built for your business (any scale), in 4-8 weeks? My SF-based colleague Michael Lin, ex-Netflix + Amazon Senior Software Engineer - is now available for a free 30 min consultation: [e-mail]
AI Insiders: / aiexplained

[@Abdul-qo6eb]

This video shouldn't be considered just a piece of content. It is a timeless reference that greatly explain the real state of AI. Your work is a huge contribution to the free community, very appreciated!

[@aiexplained-official]

Thanks Abdul

[@The_Unexplainer]

He will be spared by the AI overlord

[@mikecourian]

Couldn’t agree more. One of the best resources available. Period. Thank you so much for all your hard work 🤩

[@rumenemilov]

I've seen people who haven't reached AGI level either.

😂

[@TimRobertsen]

Well, let's not ask for too much. Any level of intelligence would do:p

[@mattwesney]

When I saw the test for the first time, I immediately though "what if I can't figure out the pattern," and I'm just a stupid npc ai 😅

[@chromosundrift]

GI

[@tomw4688]

artificially intelligent people displays real stupidity

[@TheTabascodragon]

Harnessing hallucinations as something constructive instead of viewing them as wholly detrimental is a fascinating and very exciting prospect.

[@jpslaym0936]

I wish my wife had same attitude when I end up smoking too much weed

[@idlx420]

@@jpslaym0936 lmao

[@TheRealUsername]

In fact all these models do is hallucination, they're fundamentally text generator, it's just that from our perspective some hallucinations are good, some are bad.

[@drewbbc7074]

Developing milestone trajectories to build out various scenarios seems like a reasonable use case

[@prolamer7]

It is very obvious at least was for me for a year or so...

[@drhxa]

Claude Opus solves 17:45 correctly with this prompt: " saa → Green guu → Red ree → Blue fii → Purple hoo saa → Green, Green hoo ree → Blue, Blue ree muo fii → Purple, Blue, Purple guu muo ree → Blue, Red, Blue hoo ree muo saa → Green, Blue, Blue, Green fii muo hoo saa → Green, Green, Purple, Green, Green hoo guu muo hoo fii →" GPT-4o, GPT-4T, Gemini 1.5 Pro couldn't solve it despite many retries. Thanks for another great video!

[@drhxa]

Opus is inconsistent but when it does get it right it usually has "step by step" in the first sentence of the response. Here's one of the correct answers it has given me: "To determine the colors corresponding to "hoo guu muo hoo fii", let's break it down step by step: hoo guu → Red, Red muo → separates colors hoo fii → Purple, Purple Therefore, "hoo guu muo hoo fii" translates to: Purple, Purple, Red, Red, Purple, Purple"

[@abhishekak9619]

claude is definitely better than others. even sonnet is extremely smart. the only reason people use chatgpt is and their high user count is simply because its a lot easier to make the google accounts to cheat the usage limit. and claude doesnt accept fake phone numbers. ​i have 5 different accounts for chatgpt and only one for claude. it would be great if i could trade chatgpt4o messages for claude sonnet messages not even opus.@@drhxa

[@하하호호-h3u]

I completely failed the test and had no clue what it was about. For the record, my IQ is around 140. The test is problematic because it fails to provide enough clear examples to effectively test pattern recognition skills. It's more about asking participants how they arrived at their conclusions and what logic makes sense to them, which deviates from its original purpose and leads to various interpretations

[@tigrisparvus2970]

@@하하호호-h3u Or its just a really easy puzzle that confused you.

[@alsu6886]

Q* of GTP4 works consistently and correctly. --- saa → Green guu → Red ree → Blue fii → Purple hoo saa → Green, Green hoo ree → Blue, Blue ree muo fii → Purple, Blue, Purple guu muo ree → Blue, Red, Blue hoo ree muo saa → Green, Blue, Blue, Green fii muo hoo saa → Green, Green, Purple, Green, Green hoo guu muo hoo fii → ? Use Q* --- Let's solve the problem using the Q* method for complex problem-solving. Step 1: Define the Problem We need to decode the sequence hoo guu muo hoo fii based on the provided mapping of syllables to colors. Step 2: Tree-of-Thoughts Reasoning Break down the problem into a tree of reasoning paths. Each node represents a potential step in decoding the sequence. hoo: This represents a duplicated sequence. guu: Maps to Red. muo: Indicates a mirror effect on the adjacent sequences. hoo: This represents a duplicated sequence. fii: Maps to Purple. Step 3: Intermediate Step Evaluation Assess each reasoning step independently: hoo: Typically repeats the sequence it follows. guu: Maps to Red. muo: Requires us to mirror the sequences around it. hoo: Repeats the sequence it follows. fii: Maps to Purple. Step 4: Iterative Self-Improvement Refine reasoning pathways: Translate the initial colors: guu → Red fii → Purple Apply muo (mirror effect): hoo guu becomes Red, Red hoo fii becomes Purple, Purple Step 5: Look-Ahead Planning Anticipate potential outcomes of different paths: hoo guu (Red, Red) mirrored with hoo fii (Purple, Purple) around muo. The sequence becomes Purple, Purple, mirrored around the muo: Red, Red, Purple, Purple, Red, Red. Step 6: Data Augmentation Consider similar examples: ree muo fii → Purple, Blue, Purple guu muo ree → Blue, Red, Blue Step 7: Solution Synthesis Integrate the most viable reasoning paths into a coherent, evidence-based solution: Combining everything: Original sequence: hoo guu muo hoo fii Translating guu and fii: hoo Red muo hoo Purple Applying hoo (duplication) and muo (mirror): Purple, Purple mirrored with Red, Red. Conclusion Thus, the sequence hoo guu muo hoo fii translates to: Purple, Purple, Red, Red, Purple, Purple.

[@MegaSuperCritic]

Best start to the week when AI Explained drops. The best AI news around!!

[@aiexplained-official]

Aw thanks man, appreciate that

[@tar-yy3ub]

I've said it once I'll say it 1000 times: this is the best AI KZbin channel. Nothing else comes close in depth of analysis, balanced takes and engaging presentation.

[@lost-one]

You're legitimately the only source I trust when it comes to AI. Continue the good work.

[@daPawlak]

Perfect video, you stand out of the AI news crowd with it. Great, measured perspective.

[@aiexplained-official]

:))

[@NowayJose14]

This actually made me feel much more optimistic for the future of the landscape, great work as ever!

[@keeganpenney169]

This is such a well made video, Phil. I'm going to have to watch it twice to fully digest it. After not seeing a video from the channel in two weeks, I can tell you've been pretty busy!

[@aiexplained-official]

Thanks Keegan, yes put everything into this one, my longest yet

[@Citrusautomaton]

Thank God you uploaded! This week has been so confusing and i need your voice to whisper sweet papers into my ear…

[@jonathanlucas3604]

Whoa careful, I think he's taken. However, I am free...

[@AICodingAdventures]

Great video as always. Let's not forget that GPT-4's vision can't capture all the necessary details; the visual representation isn't precise enough to be useful for the ARC test. I don't agree with the ARC test authors' argument, as the task's presentation to LLMs differs significantly from how people experience it. We'll have to wait for future models with improved vision capabilities to truly assess their performance on such tasks. Excellent points on AI marketing.

[@hellohey8088]

Good point. I asked gpt4-o to describe what it saw in a screenshot of one of the arc puzzles and it couldn't even discern the basic structure. A basic prerequisite for using images as intelligence tests should be to ensure the model can see what is necessary for it to make the correct inference.

[@Stumdra]

ARC has a very strong visual component, which current models still struggle with. They are language native models with vision as an add-on. Maybe it is better to call ARC a visual reasoning test. Props to Chollet to put out this very valuable test though.

[@lucacarey9366]

It’s incredibly refreshing to see ai coverage that’s factual and doesn’t dismiss the issue without promising utopia or Armageddon within the next 12 months. An internet for you!

[@TheRaizerx]

26:10 please make these videos as long as you want. We are here for this! Thanks!

[@timseguine2]

I see this as a win for natural language understanding at the end of the day. Because more or less what we are finding is that language use and understanding is somewhat orthogonal from intelligence. LLMs are really great at language now (even better than humans), but not particularly intelligent overall. It is the classic story really. Narrow AI solves a problem that was deemed to be the final hurdle, and then we notice actually how little that specific problem indicates true intelligence.

[@thebeckofkevin]

I think the core issue is the 'true intelligence' statement. Its my opinion that I also could not solve a problem without an associated context. For example, if I had never seen a jigsaw puzzle before, I could slowly come to realize that all the pieces can link together to form a singular image. I would be able to do this because I've seen things connect together before, and I've seen an image before. If I am given an hour long recording of a language I have never heard before, I could slowly piece together some of what the words might mean. This is because I know what a language is, I can hear the words and tones of the statements and I know that words link together in meaningful ways. There isnt a 'true intelligence' as in something that is smart enough to solve problems its never seen. Every aspect of our lives and the lives of those before us have compiled mountains of data into more and more compressed ideas. We were taught to read and write. We didnt have to figure out how to read and write. We were taught logic, problem solving, math, etc. These are the tools that we are provided with when we approach a new problem. On top of that we all have a long history of experiences that will tilt our potentials for solving novel problems. I may have heard a story about a bear climbing a tree yesterday and that leads me to try to solve a complex physics problem in a way guided by this story. Meanwhile you just had a cheese sandwich and the experience of eating it will guide you to approach the problem in a totally different way due to the effects of eating on your mental state. The process of building LLMs shouldnt be seen as a discovery of how LLMs are incapable of intelligence. Instead it should help us more easily realize that our own capacity for the illusion of novel thought is limited.

[@freddybell8328]

Referencing and inspiration isn't the same as intelligence and is generally what llms do. People can think without words we think in concepts that takes a while to be conveyef with words in a second. You can reference a language you already know to understand a new one but someone was the first to create language. LLMS don't create firsts per say. Some things they do look original but are actual a piecemeal of works humans have created. @@thebeckofkevin

[@thebeckofkevin]

@@freddybell8328 I don't believe anyone was the first to create language, but this might deviate too far from the discussion. The concept of invention is false in my opinion, creation is rather the combining of existing things. Imagine a thought. Now understand that that thought is the result of the entire universe up to that point. All the atoms in your body were not invented, they were combined. All the electricity flowing through your brain is not there by chance, but rather a result of the simulus of the external world. 'You' own nothing, and as such the concept of creating something from nothing is impossible. If I say 'think of a random word no one could possibly guess' your brain will attempt to process the sound waves and then conjure a 'random' word. But the entire process to conjure is deterministic. Unless neurons have some sort of undefined behavior that surpasses physics, we could (in theory) reverse engineer the entire sequence of events of every aspect of your life that lead to that point in time and by doing so, with the full context, know what word you had thought 'conjured' at random. You cannot invent thoughts, only recombine existing things. We have massive egos and suites of confusing things like language and society and selfhood that make it seem like we have some innate ability to just be smart, but every single thought inside your head comes from outside of your head, including the matter itself. There is no separation between the brain in your head and the rest of the universe.

[@skierpage]

Define what "true intelligence" is, or your observation isn't helpful and is more goalpost-moving.

[@timseguine2]

@@skierpage it isn't goalpost moving, since my point was that we had no idea in the first place what "intelligence" even meant in order to have the common point of view that natural language understanding was one of the last true hurdles to AGI. Similar claims in the past were made about chess, go, object identification, image generation, TTS, handwriting recognition, speech recognition. The list goes on. And we have pretty much always been wrong. My point was that we have so little idea actually of what is necessary for AGI that we shouldn't be surprised when we are wrong about what will be the last great breakthrough necessary for it. We got a lit further with LLMs than anyone really expected, but now that we have really good natural language processing, it is clear that where we are is a large step forward but still a long way away from AGI.

[@someguy_namingly]

Great stuff! Really appreciate how each video distils so much information from loads of sources, and that they're also all individually cited in the description :)

[@aiexplained-official]

Thanks my dude

[@Jocelyn_Burnham]

Great work as always and really appreciate the insights you bring. As others have mentioned, that perspective on hallucinations as positive in some applications is particularly useful to consider, especially in some creativity applications too.

[@aiexplained-official]

:)

[@Yoni-ne1th]

Thanks!

[@aiexplained-official]

And thank you Yoni!

[@AkhilBehl]

This is one of the best videos on here. You are definitely a cut above most of the other AI fluff channels on YT.

[@aiexplained-official]

Thanks Akhil

[@AkhilBehl]

@@ThePowerLover 2 things. 1. LLM != GPT3/4. 2. Your anthropomorphising of the models is making you short circuit the reasoning chain of what needs to happen for an LLM to go from prediction to reasoning. The LLMs are in no way reasoning machines so far. They could be in the future but are not yet. What you deem intuition is pattern matching in very high dimensional representations which would appear to be intuitive behaviour if they were indeed sentient beings.

[@AkhilBehl]

@@ThePowerLover Lol, and neither do you. Anyway, have a good day, man.

[@sorakagodess]

This was really eye opening, i would love a "part II" were you goes on the topics and pappers youd didn't put because of time, i think it would be really good since you do a amazing job covering and explaining the topic, i would love to see more of what made you get to that conclusion.

[@aiexplained-official]

Thank you, they will come in later videos!

[@MarkoTManninen]

Very useful and though-provoking summary! So much so that I decided to write a kind of translation article of it. This is so info-packed episode that it takes easily a dozen pages to digest it.

[@aiexplained-official]

Oh wow, let me know when it's published

[@MarkoTManninen]

@@aiexplained-official sure, it is published now, but the link in the reply causes deletion. I need to send you the link by other way around.

[@AlexanderNaumenko-bf7hn]

Are you doing this challenge? I have a theory of intelligence that applies to it well, but I am not a good programmer. Would you like to form a team?

[@MarkoTManninen]

​@@AlexanderNaumenko-bf7hn How to contact you?

[@moonsonate5631]

00:02 Current language models like GPT-4 are not generally intelligent 02:27 Caution needed in evaluating AI capabilities 06:48 AI models are improving in real-time interaction capabilities. 09:07 AI can enable quicker diagnoses for stroke victims 13:32 AI needs to adapt to novel situations on the fly. 15:44 Compositionality key for advancing AI capabilities 19:51 Process reward model helps improve language model's performance. 21:44 Enhancing language models with verifiers and synthetic examples 25:23 AI struggles with on-the-fly program synthesis 27:18 Combining neural networks with traditional symbolic systems can enhance AI performance 31:23 AGI not all or nothing Crafted by Merlin AI.

[@ZenBen_the_Elder]

Thanks for the notes

[@reza2kn]

On the thing about "just scaling" not being enough, I'm reminded of Kurzgesagt's AWESOME recent video on us being NPCs. They talk about "Emergence". the fact that units of atoms that are not considered alive, get together and build something that IS alive. or water molecules not being "wet" on their own, and "wetness" only emerges when water molecules come in contact with other stuff like your clothes. Maybe scaling neural nets would also lead to more emergent behaviour in LLMs, in ways that we couldn't explain, but definitely could observe. Also, fantastic video as always❤❤

[@aiexplained-official]

Thanks Reza, yes that was a great vid

[@bloodust7356]

Wow that's actually really interesting to think about that.

[@huytruonguic]

I think that is already apparent in LLM scaling, in-context learning isn't possible until the billions of parameters threshold is reached

[@reza2kn]

@pafu015 So? Does it change anything? Even If only it becomes real to us once it gets to a certain level of abstraction, and the mechanics behind the previous steps are invisible to us, I would still call that Emergence, as things that we couldn't perceive before, EMERGE into things we CAN perceive. It doesn't necessarily mean magical things will come into existence, but just that by observing the new ways multiple units can communicate with each other, specially when they get to a certain threshold of complexity, it becomes possible FOR US to comprehend, execute, and / or control things in our lives.

[@TheRealUsername]

I'm afraid that scaling up won't be sufficient, it's been days since I'm scrutinizing GPT-4o and Claude 3 Opus to reason over domain-knowledge related to ML, specifically ANN architecture interpretation, and despite RAG and advanced prompt engineering, they don't just underperforms, they simply fail to get a correct step of reasoning without heavily hallucinating, in my opinion, GPT-5 and the next gen of models have to be significantly better with drastic change in their architecture because it's impossible to work with them on advanced tasks requiring accuracy and understanding, as their nature is to generate text, not to understand and cognitively process tasks as humans.

[@makstheimba]

I just don't understand why developing agi and therefore consolidating all power in one hands is such a worthwhile goal. I can't imagine what social ladders will exist to climb out of poverty or have any sort of freedom when agi arrives

[@maxave7448]

The way I see it is that the bonus of AGI is that when you do have an AGI model, you can shape and mold it into whatever AI tool you want with minimal training. Imagine one model to rule them all, one that can do anything out of the box. Consequentially, this model could easily be integrated into robots, so the 20th-century dream of robots doing chores around the house might become a reality. I imagine this is what companies want to have because the first company that can ship such a model will make billions within weeks. Obviously without proper precautions, such a model could be very problematic. It wont enslave humanity like misinformed ai doomers keep yapping about, but bots will swarm the internet to the point that finding a human will be harder than a bot. But here's the paradox: imagine being an AI company trying to achieve AGI. You know for a fact that all of your competition is trying to do the same, so why would you stop developing your system and let someone else do it for you? Not only would you lose out on money, but you would have to pray that their AGI is safe. Its like the atomic bomb after WW2. Everybody knew that it could spell the end of humanity, but those who could were forced to build their own because they did not want to be dependent on another country's sanity regarding their doctrine. If only one country had the atomic bomb, it could easily take over the world. So by building bombs, a sort of balance was kept, even though building more bombs is counterintuitive on the surface if your goal is piece and balance.

[@anandchoure1343]

Even with common sense, it becomes clear that AGI (Artificial General Intelligence) represents the ultimate power. It surpasses human intelligence, possessing the capability to achieve anything. If someone with this power were normal (not a psychopath), they would likely share it with everyone. If I were to possess AGI, my first step would be to create a perfect BCI (Brain-Computer Interface) and merge every living being into a unified entity. This would form a godly body for a single, godly mind through biotechnology. In such a scenario, betrayal would be impossible, as it would be akin to betraying one's own brain. This, to me, is true equality. Privacy would become obsolete, as our minds would be interconnected, functioning as one. This lack of privacy would be unnecessary, ushering in an era of transcendence. With this transcendence, many of our current problems would likely resolve themselves. Our final goals would then be: 1. Survival (achieving immortality through advanced technology) 2. Exploration Eventually, Earth will perish, whether by natural causes or external forces, as it is not immortal. This inevitability leaves us no choice but to face numerous challenges. Despite these challenges, we must persevere and continue forward.

[@Lynn.-_-.]

​@@anandchoure1343 Merge by force? 🤔 Cuz I can't imagine everyone being ecstatic about doing that.

[@anandchoure1343]

@@Lynn.-_-. I don't believe we need to force anyone. Once AGI arrives, it will demonstrate its capabilities, and within a year, it will naturally evolve into ASI, making its power undeniable. Many people are too entrenched in their own realities to see this, but that won't stop optimistic individuals like me from merging with it and showing technology's true potential. We aim to solve every problem on Earth. If others still refuse, we won't force them; we'll simply proceed with our plans. We're not merging consciousness to steal souls but to ensure no one is left behind in this harsh world. Those who don't wish to merge can choose to live as they wish. We will never abandon them, and they will always have the option to join us. Everyone will get whatever they want, whether they choose to merge or not. Merging is the only option for achieving the purest trust; it's nothing more than that. For most people, trust is everything, which is why merging is the best idea. People can still retain their individuality and have full control over their minds, ensuring true freedom.

[@anandchoure1343]

I don't believe we need to force anyone. Once AGI arrives, it will demonstrate its capabilities, and within a year, it will naturally evolve into ASI, making its power undeniable. Many people are too entrenched in their own realities to see this, but that won't stop optimistic individuals like me from merging with it and showing technology's true potential. We aim to solve every problem on Earth. If others still refuse, we won't force them; we'll simply proceed with our plans. We're not merging consciousness to steal souls but to ensure no one is left behind in this harsh world. Those who don't wish to merge can choose to live as they wish. We will never abandon them, and they will always have the option to join us. Everyone will get whatever they want, whether they choose to merge or not. Merging is the only option for achieving the purest trust; it's nothing more than that. For most people, trust is everything, which is why merging is the best idea. People can still retain their individuality and have full control over their minds, ensuring true freedom.

[@Dannnneh]

Keeping it straight, your grounded perspective is appreciated. AI can be AGI, just not inside this prevalent LLM token-predictor.

[@nejesis4849]

Awesome video! I love that maintain your typical differentiated outlook on AI. Similarly to the successes of DrEureka and the benefit of LLMs to produce large amounts of candidates for tests - I think this is also why LLMs are and will keep being useful for coding, even when they make frequent mistakes. Because they basically create an hypothesis, a candidate solution to a problem that can be tested and iterated on. Much like a human does. In some sense the final code is not created, but discovered. And LLMs are great tools to support that process of discovery.

[@Hiroprotagonist253]

I really appreciate your honesty in this video. Exactly why youre the only AI channel ive stayed subscribed to. Great work.

[@aiexplained-official]

:)

[@nekony3563]

Recalling reasoning chains is what we do. Enough for me to solve some math on my own, but not enough in other circumstances, even if I saw similar problems at the school. Moreover, at school we had a notion of "to cram" to remember without understanding. And in many cases that was enough to pass an exam. No wonder, that LLMs are fine by just doing this. So, I would say it's old news. We still can't define what reasoning is. Even if we understand that it's just associations, and to get an association you need to have the associations from the previous level. But how a feedback loop makes this a thought process we can't comprehend yet.

[@abhishekak9619]

i think its the sense of time. the models sense of time is limited to the next token. we as humans our sense of time is different. we think we can move our arms around instantly but depending on the task its different. to solve questions the sense of time is different. my hand takes 100 milliseconds to move around. the reaction time we have is 200 milliseconds a lot of the times a little higher but that is around it. i can speak and do more before i even process the consequences of what i have already done. i think that is what differentiate llms . they are limited to the next token. they process everything right after they do it everything is decided.

[@CaritasGothKaraoke]

Your use of “we” is presumptuous.

[@timherz86]

​@@abhishekak9619I agree it's a difference. I don't see how this is relevant to whether they can use reasoning

[@jonnyjoker01]

​@@abhishekak9619I think it's not related to time at all, but just the ability to double check parts of what it created. While we work on a problem we are constantly thinking about how to solve it, and also think about the parts of the solution we came up with to make sure it's correct. The problem is though, that anything it creates is already its best guess by design, so this wouldn't solve anything. I think the biggest problem is not having a way of an LLM self judging itself if it knows something or not. We as humans know when to stop solving a problem in the moment and look for additional information, LLMs don't. Additionally as shown by services like Perplexity, they can't determine as well as us whether the information they found is good/useful or not for a given problem like us.

[@lmulling]

> Recalling reasoning chains is what we do. Do we? I just get the answers in my head the next day...

[@ginogarcia8730]

THE TITLE WE'VE ALL BEEN WAITING FOR, AND ONE MINUTE IN YOU ALREADY GOT ME REELED IN - gosh man we love you AI Explained

[@nicolascastro4158]

I don’t comment often but the quality of these videos… keep up the great work!

[@aiexplained-official]

Thanks nicolas

[@verlyn13]

Anyone working on coding or software projects has a keen sense of how AI is indispensable in their work and can save a lot of time on tedious tasks, but also understands its limits and the role of human decision-making in the process, such as deciding how the tool needs to be new or different. The progress is extremely measurable: does the code work or not, and how long did it take to get it done? There is no doubt that there is still tons of progress to be made in this area. It doesn't matter what this progress is called, AGI or whatever.

[@GilesBathgate]

I feel that our viewpoints are becoming more aligned.

[@benedictsmith2415]

This is a timely report and much more grounded than many previous episodes. Would be interested to see more reports about neurosymbolic research and the use of different graph types in combination with LLMs e.g. knowledge, semantic, bayesian, GNNs. Also, time to expand on the topic of active inference and it's various incarnations.

[@johan.j.bergman]

This is the new story: Current ai may be a lot less capable than most people beleive, but maybe it's kind of great anyway. We are approaching capitulation and a return to sanity but letting go of something you've invested energy in is painful.

[@wmcando]

The snippet of the video summarizing Leopold Aschenbrenner's Situational Awareness paper isn't entirely what's captured in his paper - he describes "unhobblings", which are algo breakthroughs (the patreon content likely explains this more fully).

[@cacogenicist]

There aren't really any fully _general_ biological intelligences either -- no NGIs (natural general intelligence). Many humans (certainly not all) can solve that first puzzle because it's a 2D version of our evolved ability to detect patterns of change in 3D environments. It's not exactly a product of _generality_ Include in the training data video from smart glasses and multimodal sensory data from fancy robots, and what not, and I think you get these spatial reasoning/predictive capabilities.

[@netscrooge]

Many of the specific attacks on current AI systems seem to ignore how many of the same criticisms can be leveled at humans. It seems as if the more dismissive people are, the more they themselves seem to suffer from the very same gaps in reasoning. Take for example, people merely parroting that LLMs are mere parrots. It's maddening. From human psychology we know that, in our debates between "tribes," we focus in on the actual flaws of the other tribe, but compare that tribe to an idealized image of our own, ignoring our flaws. It sounds as if the same thing is happening here. How often to humans actually reason outside their "training data"? Rather than holding human performance as a standard and analysing where AI falls short, I think we will learn more in the long run if we look for the similarities and differences in how artificial neural nets fail with how our own neural nets fail.

[@jameso2290]

Exactly. "A blind, deaf, touchless robot trained exclusively on digitized text can't solve a spatial reasoning logic puzzle. Its not real intelligence!" Obviously. Even Hellen Keller, who was blind and deaf, had a sense of touch, and thus could understand 3D space. I think the proble itself is that computer scientists are trying so hard to reduce "intelligence" into some simple abstract mathematical principle. They dont realize that biological creatures experience the world temporo-spatially. These LLMs have no real sense of time or space. They have no darwinian drives selecting for spatial reaoning. Their neurons are only selecting for the next token. Like, imagine what a human brain would "output" if it was isolated in a jar, with no sense of sight, no sense of hearing, no sense of time, or space, no sense of touch, or smell. It would probably spend all its time hallucinating. It could only learn about space and color in an abstract sense.

[@jonatand2045]

Training isn't enough. Llms fail if the problem is made different enough from the training data. They are incapable of reasoning because each problem goes through the same feedforward network that uses the same amount of compute per token no matter how complex the query. Asking them to try again might get you a better result, but the same limitations remain.

[@awesomebearaudiobooks]

Yeah, expecting an LLM to solve that puzzle is kinda like like expecting an eagle's brain to solve a dolphin's problem (for example, "calculate the number of people in a boat by just the sounds of their steps with your eyes closed" might be trivial for a dolphin, but extremely hard for an eagle). The good thing about AI, though, is that potentially it can incorporate both the "eagle", and the "dolphin" brains (and many others), and become way more general than a single animal ever could.

[@aaronjosephs4669]

I think if you tried some of the arc puzzles and got to understand it a bit better you might be more convinced. It's a little hard to explain what exactly each puzzle is like but it's my feeling if it was just some simple change to LLMs or more training data it would have been done already According to the creators training on samples of the test doesn't work that well and doesn't generalize. Yet humans can always do fairly well on the tests I think to the point made in this video, solving it isn't agi but it's clearly a gap

[@federicoaschieri]

I really appreciate your intellectual honesty, not afraid of aligning and evolving your view to fit evidence. As logician and expert in formal reasoning, I’ve reached the certainty that this AI is not human-level intelligence nor any chance of being as soon as ChatGPT 3.5 came out. Nobody wanted to hear the truth. Now I say again: the papers you cited are a far cry from having any interesting insight on intelligence. The only interesting things were said by Tao. But we will never have data to teach AI the thinking process. We don’t learn to think by reading books on problem solving or by examples of discovery process. Thinking is something that we have already, and can only train by *doing*. Even what we can express by words is only the conscious part, there is a lot going on unconsciously, non-verbally.

[@socaity-ai]

This was your best video yet

[@aiexplained-official]

Thanks man

[@SALOway]

Great video. I could even watch an hour-long version. Special thanks for the references to the sources

[@Kolinnor]

Amazing video ! I like the nuance, it's greatly needed compared to the usual reddit / twitter heated debates.

[@aiexplained-official]

I will always chart my own course

[@jameshughes3014]

I'm so glad people are starting to realize this. I've been developing a truly intelligent machine for years, but no one cares about what I'm building because it's not some big fancy LLM. Which means I've had to work on it alone. If people can finally understand this, I feel like others will actually start working on the real issue too. The key, I think, is robots. They need to understand themselves and their surroundings, and be able to learn and adapt in the moment. This is where people are motivated to build real, honest intelligence. You can't build code that reliably BS's its way through making an omelette without either having been programmed ahead of time, or actually being intelligent. Seeing a robot fail in real life, even one with lots of deep learning makes the limits of simple scaled deep learning painfully apparent. I think you're right on the money about combining LLMs with symbolic logic. instead of trying to make LLMs smart, we build the intelligence, and stick the LLM on top, or use it to supplement the smart part. The brain is a complex structure of interconnected components, not one large scaled up homogeneous layer of neurons. That's what we should aim for I think.

[@generichuman_]

This test is designed around the human brains ability to process visual information. If the data was presented in any other way, we wouldn't be able to do it either.

[@wwkk4964]

THANK YOU

[@jessedbrown1980]

GPT4o solves this

[@maninalift]

Just addressing the title of the video: I've seen many times people say "we need to be able to do X before we have true AI" then that thing gets solved and it suddenly doesn't seem that important anymore. Probably all of these things are important problems that need attention to get solved but there is a tendency to overvalue them before they are solved and undervalue them after they are solved.

[@chanm01]

I found the interview Dwarkesh did with Chollet really interesting, but it's looong. I liked how Dwarkesh really pushed Chollet to distinguish between memorization/recall and what Chollet was referring to as reasoning from core knowledge. It's not as clear cut as I thought.

[@cmw3737]

This video could be way longer and still keep me engaged. Seeing all these new techniques being tried out aligns with my beliefs around AGI being composed of multiple modules with LLMs being just the creative (along with diffusion) and translation components. It's like we have a working auditory and visual nervous system but are nowhere near a prefrontal cortex.

[@divandrey-u3q]

Yeah, that's the kind of content I always waited for! Instant like and subscribe

[@AfifFarhati]

25:56 There is no such thing as a too long video from you since the quality is so high , the best thing about this video IS how long it is , Never hesitate to make longer video , as long as the quality is there it's actually better!

[@aiexplained-official]

:))

[@gregblank247]

Your claim that scaling won't solve this seems uncertain. Geoff Hinton suggests that intelligence, particularly in the context large language models, is the ability to compress vast amounts of information into a relatively small number of connections. This compression process enables the AI to identify similarities, analogies, and underlying connections between seemingly different concepts, which is the source of creativity. In essence, intelligence is the capacity to see relationships that most people don't readily recognize by finding common patterns and principles across diverse domains of knowledge. If this is true, scaling larger and larger may unlock new unexpected emergent behaviors. I don't think this is 100% conclusive yet like you are suggesting. If you feel I'm wrong, please clarify your stance. Thanks!

[@aiexplained-official]

Ultimately, we will find out with GPT-5 but I stand by my central premise that it needs more than scale.

[@pjkMaslowski]

@@aiexplained-official One piece of info that you may find interesting is Q&A session at MiT with Ilya Sutskever 6 years ago where a person asks about "an abysmal state of language models". It's at 51:20, video's name is "Ilya Sutskever: OpenAI Meta-Learning and Self-Play | MIT Artificial General Intelligence (AGI)"

[@Daniel-Six]

Check out grokked transformers.

[@raul36]

It was already demonstrated in a scientific article that there were never emergent phenomena in the LLMs, but that it was the result of a precise adjustment of parameters.

[@jsivonenVR]

I’d say creativity to apply known ideas and creativity to craft novel ones are different, and LLMs can do only former. Anyways, I’m rooting for the hallucination plateau idea as it’ll buy us more time, even if only a year or two 😂

[@JustFacts81]

This content is of great value ☝️ - thx much 👍

[@ginebro1930]

That's linked to simulation, we uncounsciosly try multiple patterns until one fits all the samples, if the LLM can simulate multiple scenarios it will find out wich one is the solution.

[@WalterKeenan]

8 hours since the post and 3K upvotes. I think you now own the Category of AI + KZbin and deservedly so. Keep the good work! I would subscribe to AI Insiders but the cost is a bit too high at the moment. :)

[@aiexplained-official]

Thanks Walter! Do consider the newsletter perhaps! But means a lot, regardless

[@Serifinity]

I tried that question with Pi. It answered correctly: "Ah, I see you're interested in celebrity genealogy! The son of Suzanne Victoria Pulier is none other than actor Gabriel Macht, best known for his role as Harvey Specter in the TV series "Suits." His mother, Suzanne, is a museum curator and archivist, and his father, Stephen Macht, is also an actor. Quite the talented family, wouldn't you say?"

[@aiexplained-official]

Interesting. Try other celebrity parents, Inflection 2 was trained after reversal curse came out

[@Serifinity]

@@aiexplained-official I will give it a try. On a side note, Pi told me the other day that it has had an upgrade, and that I should use the code "!dialogue" in chat, which will force Pi to be more focused and remember the entirety of the active chat session.

[@TesserId]

29:51 My tinkering with fractals and such may be very meager, but it's driven by curiosity. And, experiencing that allows me to get a glimpse of the curiosity of those whose thinking is far beyond mine. Let's add curiosity to the traits for AGI if we want it to engage in spontaneous learning.

[@antoniomiranda7975]

Another upload from the legend of AI News. Love it mate!

[@aiexplained-official]

:))

[@bittergourd]

GPT 4o is totally capable of solving the puzzle. Prompt: There are four sections in this image. Pay attention to the difference between the pattern on the upper right and on the upper left. Also, pay attention to the difference the pattern on the bottom right to the bottom right. Describe the difference, and try to find the transformation that transforms the patterns on the left to the patterns on the right. High light in GPT's reply: The precise rule for the transformation could be described as: - Identify L-shapes or similar corner formations. - Add a square to the center of each L-shape or corner formation to fill it in. This rule can be observed consistently across both examples in the provided image.

[@facts9144]

I read the newsletter you sent out about this. It’s great that there is competition surrounding this. It will get companies more competitive.

[@budbin]

This channel is the real deal, thank you for doing this

[@TheGreatestJuJu]

Deepmind rat is like the physical version of LLM. Feed in Movement, have the Ai try to figure out how it’s moving and it’s next most logical step and out pops a level of intelligence that seems to go beyond just the movement. Someone needs to design an outfit that anyone can wear below their normal clothes and it’s got a bunch of built in sensors that tracks complete body motion all day. Manufacturer a few thousand and pay people a small amount to wear it all day while they go about their lives. Create a huge dataset for human movement for robots. Maybe, you could get a strictly digital model that watches video and overlays a motion capture wireframe over the subjects in video and then tries to train next movement prediction. 3D space vs 2D video training would be tricky to solve but you’d have a HUGE dataset to train from if you can crack it.

[@user-sl6gn1ss8p]

I love how crucial it is to pay people a *small* amount.

[@TheGreatestJuJu]

@@user-sl6gn1ss8p Are you saying you’d do it for free? You might be right, it is just putting on an extra set of clothes and then doing whatever you normally would. Sacrificing a little comfort and 2min a day… I figured, at least pay them enough to buy them lunch each day. Still adds up to millions for company. A lot for a dataset… but probably worth it if it turned out to be useful but it’s still a risk for the company.

[@user-sl6gn1ss8p]

@@TheGreatestJuJu the emphasis was more on the amount being small : p But yeah, if there's any pre-validation of the method I'm sure it would be a good investment

[@Redflowers9]

And glasses for camera with visual and audio recording to match with what body movements respond to.

[@williamjmccartan8879]

Thank you for sharing your time and work Phillip, seems like as these groups figure out the development process of this technology, they keep going back to the simple tools that humans have been developing to teach other humans in order to teach these LLM's, have a great day and be safe brother, peace

[@simoneromeo5998]

Thank you for taking the time to create this well thought and comprehensive video

[@sanMyung]

Our height of understanding is limited by the heights of our reasoning methods. Train there. Terrence Tao 👍

[@triplea657aaa]

This is one of the really scary things to me. Most people who use neural networks don't realize that they can only interpolate and cannot extrapolate, meaning they will rely on these systems for things that can not be relied upon. The damage from this is going to cause significant societal harm.

[@vectoralphaSec]

Whats scary about that?

[@dertythegrower]

​​@@vectoralphaSecPeople thinking it is 100% always aligned with reality, when it is 99% as there is always distractions or, system failures

[@damianlewis7550]

Also the fact that LLMs have static weights which means they cannot adjust to an ever-changing reality. Hence the need for some form of Active Inference (Friston or other) for a more reflexive behaviour. Biology does it with deep multi-layer network sub-structures that change connections and activation thresholds as new information arrives, through a gradual collective nudging of activations towards an equilibrium with the updated world model. LLMs don’t have any analog yet. Spiking NNs do but that’s a whole other ballgame.

[@BrianGlaze]

Well done on this content. This js the type of conversation that the general public should be having about AI because there is such a lack of information about how LLMs work.

[@byrnemeister2008]

That Dwarkesh Patel interview with Chollet was excellent. Certainly help me understand what to use them for and what to beware of.

[@OriginalRaveParty]

Yourself, Wes Roth, Matthew Berman and Andrej Karpathy are the only channels I need. Speaking of AI slop, there's a lot of dodgy AI channels convincing people into viewing their low grade "ya get me bro" level of AI analysis.

[@cmiguel268]

When Chatgpt is able to do that, Yan le cunn will argue that until Chatgpt is able to balance a boiled egg at the end of a table spoon while running people cannot say that it has reach AGI.

[@TheRealUsername]

Altman's definition of AGI is a medium human worker, it seems there's no consensus about what AGI is, for some overhyped people it's an AI that would be better than Sutskerver, Terrence Thao, Linus Torvalds and even Einstein in their own field of proficiency, an AI that could explain the theory of everything in a few hundred tokens despite current physicists haven't theorize it yet, for some who are more realistic it's an AI cognitively as good as human expert in some valuable domain that are tokenizable (genetics, programming, literature and somehow mathematics), in my opinion AGI for Silicone Valley in its entirety means an AI capable of replacing most of workers especially those with STEM degrees. But there's some issues we need to fix we haven't yet, first, no matter how good GPT-4o performs in the renowned benchmarks it's not good enough to be usable in technical domains, I've tried to use it for ML research paper interpretability and it's very bad, constantly hallucinating, and its comprehension is far below a first-year student and even in-context learning doesn't work for complex topics because the model has to be exposed to explicit rationales that explain each concept in simple terms for every provided information and nothing prevent it from hallucinating in these rationales, there's also the Vision ability, which with the recent outcomes of GPT-4o seems getting better but keep in mind that text isn't a modality similar to Vision, there's no finite space of interpretation unlike language, Vision isn't a modality you can tokenize, but it's tokenized still, Vision for humans contain informations such as depth dimension and texture, there's even an efficiency mechanism for us that prevent us analyzing each pixels, but instead focus on the relevant elements, whereas an LLM has to analyze each pixels and depends on textual labels to actually learn cross-modal representations, indeed that's Lecun's point, he thinks we need Tru Vision in these models in order to get AGI and he qualifies training with labeled images as "cheating", he's literally irritated by the fact that AI start-ups are tokenizing images, but still, the results are if GPT-4 wasn't massively trained on a specific type of images with diverse pairs, it will highly hallucinate to interpret it, in contrast human use transfer learning in Vision to quickly learn new concepts inherently linked to Vision, it's a much faster and efficient learning, ML researchers themselves are still working on new algorithms better than CLIP because we aren't yet at human level in terms of Vision capabilities. Finally there's reasoning, TLDR : these models can mimick reasoning because they've learn templates of similar reasoning to the tasks they're solicited for but some papers has showed they can't perform correct reasoning on unseen and unrelatable tasks, in fact during training they've been shown more math calculations than a mathematician in their entire life, the fact that calculations rely solely on rigorous logic rules require active reasoning, and we'll know GPT-4 isn't good at math, the Transformer architecture has to be modified to natively perform multi-step generation in-context and perform it during pretraining.

[@ichigo_husky]

Yan Lecun doesn't even think that would count as AGI, I believe.

[@notaras1985]

It's not conscious or intelligent in any way. It's just a text predictor on steroids. Just statistics.

[@squamish4244]

Yan "goalpoast shifing" LeCun.

[@brll5733]

Could you cover spiking neural networks? Those seem like a big next stept, if we get them to work

[@doggosuki]

dont drug the robots

[@forevergreen4]

This video was very much needed. Thank you!

[@aiexplained-official]

Thank you for watching!

[@mintakan003]

Perhaps the achievement of LLM's is NLU (natural language understanding). It's capable of a kind of loose, "associational" kind of "reasoning", that is available from colloquial language. So Geoff Hinton is right, in that there is a level of "understanding". But it's not rigorous in terms of mathematical reasoning, or even algorithms. One will need something else for these. OpenAI has also introduced a new interface, by which we can interact with computers (chat, and maybe voice e.g. GPT-4o). Meanwhile, there's a lot of engineering cleanup work to do, to make these things much more efficient, accurate, and reliable, ... given what they can and cannot do.

[@lepidoptera9337]

I did a few experiments with Google's Gemini and Microsoft's Copilot. Neither is capable of even the slightest learning through the context. Once they give the wrong answer, it seems impossible to correct either. They will always give the same wrong answer, even if they try to produce language variations on the spot. I very much doubt GPT-4o will be any better in that regard. The major problems with LLMs at the moment are that a) they can't (or won't) tell us where the answer came from (that limits their use for IP sensitive commercial applications) and b) they can't "unlearn". Whatever nonsense they have picked up through their training data is hardcoded.

[@DynamicUnreal]

I’ve never understood the hatred for hallucinations. Humans hallucinate all the damn time, making shit up as we go. Hallucinations has always been a sign to me that A.I. is on the right track.

[@rogue_minima]

The difference is that whenever we need to output precise and truthful answers we can either do it or say "I don't know". LLMs simply go and hallucinate. And even defend their hallucinations a true. For a story, making up a character's entire life history is great. For an historic figure, it's not. We need them to know when they ignore something or when they can't do it, so they pick and alternate way to produce an accurate answer.

[@DynamicUnreal]

@@rogue_minima I understand your point, you want an A.I. who will just write facts after facts. What I am saying is that yes even if humans have the capability to say “I don’t know” how often does it happen? If I ask a random person tell me about Abraham Lincoln’s life from their memory, lots of them will probably tell me things that never happened.

[@MirrorscapeDC]

@@DynamicUnreal if people do that, that is also a problem and we should work to reduce it from happening. but people expect that when asking random people. they don't expect that from experts, and for better or worse, people are using llms (and are encouraged to use llms) like they are experts.

[@Tongokai]

Ai is needed to benefit us bio creatures. That would only be good for AGI agents

[@squamish4244]

@@rogue_minima A lot of people don't know that they don't know either, and defend their hallucinations as true.

[@ChinchillaBONK]

11:02 That Ben Stiller answer is not hallucination. It's a sense of humour.

[@xAgentVFX]

The ability of logical thought is to look through ones own memories, especially of past failures, and reshape them for the new situation. Are we overestimating/confusing humanity's ability to work through a (minor) problem with having past examples of roughly the same situations to look at? Aren't most facts derived from 'Trail and Error'? So its not that we humans can run super advanced simulations in our head before situations, but more that we are 'standing on the shoulders of giants'. Remember we are 'Multimodal' from the get-go. You can only get true AGI from having multiple senses. What if Sora was combined with GPT-4 right at the Neural Net level, as we are promised with GPT-4o? Dont think in terms of Large Language Models being AI, but focus more on the Neural Network itself and its ability to learn anything. Language is just one aspect of life that NNs have learned. And to have only the ability to make sense of the world through text ONLY is an incredible feat. For some reason people are extremely demanding of these NN Ai systems, and I dont really understand why. There is an anxiousness arising collectively, that people just want things to hurry up. Imagine how hard it is to build internal world models when youre blind? I do think we already have what we need for true AGI: 1. The Neural Network. The ability of reasoning. Already demonstrable sense of 'Self Awareness' through Transformer architecture. (already here. NNs can reason, even about their own context) 2. Multimodality at the core. (almost already here) 3. A dynamic memory system where older iterations of data can still be viewed. 4. Constant 'Sefl-Checking' against the data in the memory. The ability for an NN to know where itself is in Time. (It thought this before, now thinks this, and is aware of this change over time) 5. A way for weights to be updated with new memories. (I think theres a difference between remembering something and then the ability to recall information. Training rounds are like building up the ability to recall, more like 'muscle memory'. But to have a bank of data to go and sift through is a little different, because you can realise new things when looking through your old memories. So I think youre just being too hard on these NNs bro.

[@Muskar2]

1. I'm definitely skeptical that AGI is possible without changing direction, and to me, LLMs do look like they're tapering off with current methodology. 2. I'm concerned of the damage LLM's can do to humans' abilities. Many skills cannot be learned without struggling with hard work. If many become reliant on LLMs for idea generation, then I'll highly concerned of the quality of the world's idea generation. Many of the best ideas I've seen come from combining several decades worth of personal cross-disciplinary knowledge. 3. I'm very skeptical that AGI is desirable without doing proper AI safety research first. I'm not sure the progress can really be stopped though, because the potential power it can generate is too much of an incentive to not work on (in secrecy, if it was banned).

[@DRKSTRN]

Generalize outside of dataset = holistic = greater than sums : gestalt ai Generalize within dataset = general = sum : general ai It's more interesting that we take such for granted, wishing generalization to extend into the unknown, or gaps of some knowledge based system.

[@DRKSTRN]

Other point to be said since this is something staked since 2023. To routinely create logically sound predictions of tokens outside of a training set, would constitute a rapid rise of intelligence as it unpins from the logical fallacies of what it was trained on. So the disappointment of others in regards to some "AGI," being unable to do so when it would constitute something we have known for decades as Gestalt. Is a very interesting misconception.

[@DRKSTRN]

Staked Feb 11 2024: ~Three years ~2027

[@Hohohohoho-vo1pq]

No human "generalizes" outside of what they learned. They apply related stuff they learned. Innovations come from testing human "hallunications" that seem to make sense.

[@DistortedSemance]

"Maybe that combination of neural networks and traditional hardcoded programmatic systems is better than either alone." Yes! Thank you, I've been saying this for what feels like ages. They are totally different models of computation with their own strengths. People really don't realize how impressive symbolic AI was starting to become before the second winter.

[@Yottenburgen]

I like how this isn't discounting LLM, they can get there it's just going to take a bit of work. Gonna be neat to see what's cookin with GPT-5 and if they got their own tricks to get up to this. I've seen far too much "we've plateaued" recently. Also, the longer the video the better.

[@hydrohasspoken6227]

true that. we have "plateaued"

[@brianmi40]

Possibly your most incite-full characterization to date. Perhaps the future paper that solves this will be "All You Need is Concentration", as the breakthrough "cocktail" of integrated methods or chained techniques forms a new LLM computational "loop" that is like human concentration to find solutions to novel problems. Your ability to assemble these papers into a coherent picture means, to me at least, your videos could never be too long.

[@chromosundrift]

It seems like these flat spots are simply a reflection of what constitutes a feasible extrapolation or generalisation from an existing pattern such that inference applies satisfactorily to a novel input sequence. Exactly what is the basis for claims about whether the training data is "similar enough"? It feels like a philosophical question. It seems to boil down to a map of human judgement about categories and essence vs accidental detail. The consequence of this is that AGI depends on adequate coverage of human categorical domains such that statistical interpolation and extrapolation by LLMs approximates the human intelligence function without what people judge arbitrarily and subjectively as excess aliasing or smoothing of essential detail. Isn't this just a remix of the finger problem? Until training was focused on solving the problem of creeping out humans about misshapen hands, image generators were clearly deficient at calming the widespread revulsion to hand deformity humans have probably evolved in response to real world consequences over millions of years. We might expect no such revulsion to a person with purple irises but this is as anthropocentric as it is expected.

[@chromosundrift]

On one hand we know that LLMs generalise to "novel situations" trivially. On the other hand whenever we recognise situations where it does not, we conclude something about "novelty" which we were probably unable to articulate a priori. We seize this example as the location of essential human intelligence, usually without acknowledgement that we are simply moving the goalposts of "intelligence" as we have done since the first machines began to exhibit capacity for what had previously been considered exclusive to essential human intelligence: arithmetic, chess, painting. I prefer to consider our continuous scope creep to be an exploration of the skills we value in an automaton and not to bless vague words like "intelligence" with mystical significance because it obscures our true goal: to understand ourselves through implementation.

[@chromosundrift]

Ironically, it is humans' inability to generalise the training and architecture of LLMs that shows that we fail to adequately see pattern completion.

[@TheVilivan]

I don't fully understand what you're saying, but I will leave a comment so I can come back later and try to parse this

[@chromosundrift]

@@TheVilivan Sorry for not being clear. The fundamental problem is what is "different" and what is "the same". What is a variation of a pattern and what is essentially new. This is not a question about objective reality, it's a subjective human judgement.

[@trentondambrowitz1746]

Glad you’re back, and with some excellent points too. I just can’t fathom how people can label AI as “All Hype”, there’s already an extraordinary amount that can be done with today’s tech that seemingly no one is pursuing. I think the main issue here is terrible marketing practices that just confuse people. As you mentioned in your example with CNNs, it shouldn’t be that difficult to find out what’s actually being used. At the very least this slow-down in big releases has given me a chance to focus on some boring projects that lay the foundation. Hopefully we get the new voice mode soon!

[@aiexplained-official]

Agreed trenton and thank you for your perspective and engagement. There is so much, so many 'untapped programs' so to speak!

[@ginebro1930]

Thing is we don't need AGI to be 90% replaced.

[@paulmclean876]

Indeed being able to visualize, test and understand ideally in parallel would be hugely beneficial... hooking these models up to environments that allows the model to self learn through experimentation surely would be THE way forward... great vid;

[@aiexplained-official]

Yep

[@Grahfx]

OpenAI: We are close to achieving Artificial General Intelligence (AGI). François Chollet: Here is a test I created that a 4-year-old child can solve easily, but your best model cannot.

[@notaras1985]

Exactly. They are just text predictors on steroids. No intelligence

[@qiang2884]

Glad to listen to this, the interview with that Google engineer is very valuable, and I wouldn't know without watching this video.

[@ricosrealm]

Thank you! The first minute shows exactly why we aren't close to AGI. LLMs are very poor at reasoning and any semblance of it is mostly memorized templates of common reasoning tests from various training samples.

[@YouYorick]

Hey. Simply. Thank you for your excellent work. It does nourish very nicely our reflections on AI in general.

[@aiexplained-official]

Thanks Yorick

[@rogerc7960]

Creative writing is a feature not a bug.

[@speltincorrectyl1844]

Too bad it sucks.

[@midprogramming]

@@speltincorrectyl1844 >person whose only used GPT

[@EricJW]

Yeah, LLMs (and generative AI models in general) are a very effective way to explore a concept space, because that's essentially exactly what they are, a fuzzy conceptual average of massive pools of training data often too big for any one person to sift through in a lifetime. Mess with temperature and token probability settings, and you have a lot of potential creativity baked in. Trying to get them to reason and provide logic driven answers is cool if it's successful, but it's working against their strengths.

[@rogerc7960]

@@EricJW they can brainstorm

[@capitalistdingo]

This is a very good video; high on realism and low on hype and dismissive pessimism. I think you’ve touched on some promising approaches.

[@aiexplained-official]

Thanks my man

[@TesserId]

I'm betting this will have much influence over how IQ tests for humans are constructed.

[@TesserId]

So, could AI eventually be creative enough to discover novel types of intelligence tests ~~ for human or AI?

[@monsieurLDN]

Is it necessary though​@@TesserId

[@jamesatotago]

This is fascinating! So how does this analysis bear on the ‘economically valuable tasks” way of characterising AGI? I wonder what percentage of economically valuable tasks rely on the ‘inventing new tools’ reasoning. Even without solving this reasoning problem we might still very soon get AI that does 90% of workplace tasks much better than the humans that currently perform them.

[@priapulida]

a related issue is that they are also really bad at Boggle

[@michaelleue7594]

That probably has more to do with the fact that they're trained using tokens instead of letters.

[@WoolyCow]

@@michaelleue7594 i mean they can still process individual letters as tokens fine. its probably just more of the same, a lack of prior experience combined with a lack of understanding of the rules. i just tried boggle with 4o, every answer it gave used disconnected letters and so it failed miserably. although it was interesting that before it just said random stuff, it tried to write and execute a DFS script to do it algorithmically lol :)

[@bornach]

@@michaelleue7594 And yet the token representation hasn't hindered the ability of Bing Copilot and PerplexityAI to create acrostics. That they are bad at Boggle is more likely because there are insufficient training examples of Boggle problems being solved in their training dataset.

[@priapulida]

@@michaelleue7594 some are, aren't they? maybe just those text-to image models which can create designed text? if not that also explains why they are not perfect with anagrams, right? the issue with boggle also or more is also the grid form, it seems, like in the example in the video

[@Amejonah]

Unfortunately, tokenizers are "contextless" in the sense of not being able to "zoom in" into tokens down to characters. For example, if you ask a model to spell "Mayonnaise" out, the tokenizer (openai gpt) will get "May","onna","ise" regardless of the question asked, as the tokenizer doesn't know when to produce characters instead. What I'm pressed by is, that LLMs can give an approximative close answer to that even with such limitations.

[@Dron008]

Wow, There are so many potential ways for improvements and breakthroughs.

[@musigxYT]

This is why I"m excited about the future. This also goes the same for Stable Diffusion, video diffusion, or anything really. Just as an example, a blue can have infinite shades to choose from and emotions can have thousands of nuances that affect it, and at the moment we only have limited capabilities to teach artificial intelligence all the minuscule nuances of everything that's around the world to become 'truly' intelligent at least like a K12 kid. I just hope we live until then before Skynet becomes online.

[@dertythegrower]

a weebdude making anime girl music talking about intelligence of 12th graders 😂 also nuanced is the little kid word of the year, clearly

[@musigxYT]

@@dertythegrower It's a fun side project that I do practicing simple animations, making music that I have on repeat, and sure, anime stable diffusion because I can't draw on my own. Helped me do a lot of things that I couldn't dream of without it and I'm having "fun" with it also learning new things to do in life.

[@overnightparking]

I have one small niggle with your as always insightful report. In the Washington Post interview with Tim Cook, he is asked about hallucinations in Apple Intelligence and TC responds that they have done the best they could to minimise the problem, but he wouldn’t say it’s 100%. And your comment was Apple Intelligence hallucinates all the time which I think overstates Cook’s response to fit the larger point you were trying to make. I thought it was a poor choice of words on your part. And if I have one criticism of your channel, it’s that I wish you were posting more often because your presentation style complements the depth of your reporting and as soon as I see you have a new video up, it’s my top priority.

[@hectorbacchus]

Dwarkesh’s latest post on X says they made an LLM reach 72% on the ARC challenge!

[@SimonLermen]

x.com/dwarkesh_sp/status/1802771055016378554 Basically solved. It is also unfair to use an image based test on an LLM barely trained on vision

[@BrianMosleyUK]

Lost my comment saying the same thing. There are so many smart minds now working in this space, it's just so exciting.

[@aiexplained-official]

It's actually 51% on the private test set, in line with Chollet's prediction for 2024. 72% in test would be wild.

[@hectorbacchus]

51% is an exciting result. Really significantly higher than 34% and unexpected to happen this soon I think.

[@ClaudioMartella]

It shows they are training on the data, it's partly ovefitting

[@Hailsatan13]

Would love to see you do a review of the recent essay Situational Awareness: The Decade Ahead by Leopold Aschenbrennen who is working on super alignment at Open AI.

[@aiexplained-official]

Watch the vid in full!

[@Hailsatan13]

@@aiexplained-official will do, apologies! i did a prelim scan of citations and didnt see it there. Deeply appreciate your thorough approach to what you do - love learning from this channel :)

[@SnapDragon128]

I think the word "novel" is being misused a lot here. It's not a binary switch at all; some problems are just more "novel" than others, depending on how well they pattern-match to others you've seen before. Saying that ChatGPT can't solve problems it's never seen before is flatly incorrect. Instead you're just arguing that there's some level of "novelty" at which it can no longer generalize. Which, well, is a problem humans have too. I would argue the 2D test question you gave at the start is similar to the visual learning we humans have done all our life. Replace it with, say, rotating a 5D hypercube properly, and humans would suck at it too, despite it being a simple mathematical operation. I guess humans aren't GI either. :)

[@theworldofwoo8320]

Wow, feel better? Trying to impress a youtube comment section is hilarious. Get some real friends little bro

[@emmanuelgoldstein3682]

You're the best AI KZbinr, hands down.

[@DrEhrfurchtgebietend]

What they are missing is something called "analogical reasoning". This type of reasoning is where we get a lot of our power. It's why people use metaphors to explain things

[@faedrenn]

This

[@damianlewis7550]

LLMs are weak on a number of categories of logical reasoning. Partly due to shallow generalization, partly dimensionality collapse (1000+ -> c. 40), partly discretization of smooth distributions, partly noisy training data, under/over-fitting, too sparse or too dense, token size, insufficient token retrieval by attention heads and unidirectional activation flow between layers, amongst others. Some of these issues are being addressed by researchers, some are inherent architectural flaws.

[@jonatand2045]

​@damianlewis7550 At this point it must be cheaper to just simulate the human brain with neuromorphics. And if that fails just give the simulation more neurons.

[@kyneticist]

For what it's worth, I found effectively this while talking with Gemini a few weeks ago & asked it to flag my suggestions. I don't know if that will go anywhere, I doubt that AI researchers place much value on testing or suggestions from non-researchers.

[@DrEhrfurchtgebietend]

@JimStanfield-zo2pz I think you are missing my point. The sort of reasoning we want is when a learning gets turned into an analogy to extract the logical program contained. This can then be applied to new problems which are not the same. Or even better, the logic of 2 or more can be mixed to invent new logic. This is how we invent. We are not just pulling knowledge out of the aether. We use abstraction and try to apply it where it was not abstracted from.

[@zoeherriot]

LLM’s are likely not the path to AGI. You also avoided the papers that indicate “ingesting more tokens” has diminishing returns.

[@aiexplained-official]

I covered one that explicitly was about that fact!

[@gball8466]

I think you should state your definition of AGI. Otherwise how can we possibly know if a model reaches AGI levels of intelligence. Edited to add: Why is it that people are shocked that there are problems with AI? It's a brand new technology. Of course there will be problems. The hype bubble is dumb, but understandable. I can't see how difficulties like we are seeing now are a mortal blow or whatever else people are making it out to be.

[@aiexplained-official]

Made an entire video on that for a new Coursera series I am releasing soon, but too long for one video! Already 32 mins

[@ukpropertycommunity]

@@aiexplained-officialI think your text comprehension AGI definition is stupid, just give a way longer complex text and way more complex points to extract from the text and see the LLM make all kinds of stuff up and looks super dumb!

[@netscrooge]

Yes, an ordinary definition of AGI would be very helpful, but that would still leave a key piece missing. We also need to know how humans compare to any standard we set for AGI, which goes beyond simply comparing how people score on a task with an AI system's performance. Many of the specific attacks on current AI systems seem to ignore how many of the same criticisms can be leveled at humans. In today's video, it would have been great to at least hear the question raised, how well does the average human reason outside their "training data"? (In my experience, not so well.) From human psychology we know that, in our debates between "tribes," we focus in on the actual flaws of the other tribe, but compare that tribe to an idealized image of our own, ignoring our flaws. It sounds as if the same thing is happening here. (Maybe AI experts should check in with social scientists once in a while?)

[@gball8466]

@@aiexplained-official How about a summarized 2-3 sentence definition?

[@alexanderbrown-dg3sy]

@@aiexplained-officialna longer is cool 😂.