AI isn't gonna keep improving

[@splunge2222]

Gordon Moore was not a “Dev”. He was the co-founder of Fairchild Semiconductor and Intel (and former CEO of the Intel).

[@jonnyeh]

He was an engineer first, so that’s probably where the confusion was from. Definitely not some rando.

[@splunge2222]

@@jonnyeh yes, but he made the “law” as CEO of Intel.

[@monad_tcp]

No, he's the only dev, he actually developed something. Because photolithography is developing a chemical to produce a pattern.

[@MasamuneX]

yeah some random guy lmaooo just made a guess lmaooo just an uninformed shot in the dark just lucky

[@LawJolla]

I like Theo. But that was one of the most boneheaded things I’ve heard him say.

[@iantimmis651]

Calling Gordon Moore a "dev and hardware enthusiast" is hilarious. Dude literally founded Intel

[@diamondlion47]

Yeah, this guy is completely clueless. Like saying the internet won't get better in 1997.

[@warrenarnoldmusic]

​@@diamondlion47Thats why i keep on saying frontend devs are not devs💀🌚😅

[@warrenarnoldmusic]

Down right disrespectful😅, my lord🤦

[@miasmator]

Moore might have been able to design and build Intel 14900k with his bare hands and a piece of paper, but he was not able to learn the latest piece of shit frontend framework in 3 days after it was released in 4chan

[@monstardev7189]

@@warrenarnoldmusic stay mad

[@DelkorYT]

"Moore - a dev and hardware enthusiast" -- Theo That is... technically correct

[@KManAbout]

The best type of correct 😂

[@billywhite1403]

@@DelkorYT I thought, if we factor in multicore chips, that Moore's law is holding up pretty well, is this not true?

[@mirakekkle9476]

@@billywhite1403 Yep. The main problem with Moore's law is clock speeds, power consumption, and die/process manufacturing size (excluding design problems like clock timing amongst other things). All those 3 things combined will slowly kill transistor count.

[@billywhite1403]

@@mirakekkle9476 I'm not saying you're wrong, but I do think they were saying the same sort of thing thing in like 1987 (Just to pick a random year before things got really digitized, aka before a paradigm shift in chip design, aka we could be standing on the threshold and not know it. I'm sure there is a limit on physical transistors. But there are so many other ways to indicate 0 and 1, especially on the particle scale, I can imagine we figure that out within the next few decades

[@gameguy301]

@@mirakekkle9476 those issues are explained by the breakdown of Denard scaling which happened in the mid 2000s.

[@snats]

The problem as always is when you have a 99% reliable system and you want a 99.9% reliable model. The .9% difference is 10x more than anything else

[@LuisOrtiz-zu3dq]

Yeah, because that's a perception problem. We won't know the actual percentage until we reajust the scale once our knowledge is increased.

[@SahilP2648]

What are you on? Benchmarks should never be in percentages. Do you see CPU and GPU benchmarks in percentages? No because we don't know the upper limit so we can't logically create percentages. Same with LLMs and the like. For whatever reason people are using percentages, but that doesn't mean it's correct. For example, we have an IQ system with a max of 200. But we decided that with some parameters in mind. But there could be conscious entities, non-humans, which could have an IQ of 1 billion. They could do all the computation of humanity till date in 1 sec. We just don't know.

[@johndank2209]

You saw NeetCode's vid too huh?

[@milanpospisil8024]

@@SahilP2648 True, but those percentages are according to benchmarks. Also benchmarks should be adding more complex tests while the LLMs are improving.

[@blisphul8084]

Yup. Gemma 2 2b (2.6b actually) is really good at translation, but when it struggles, I go straight to Mistral Large 2, which is more that 47x its size.

[@GraveUypo]

8:50 that is absolutely the wrong way to look at those graphics. that isn't a chart with uncapped ceiling, it tops at 100. so if we invert the data to show "mistakes" (just a way of saying) instead of the score, you'll notice that jumping from gpt-4 turbo1 to gpt-4o, which in the graph is roughly 74 to 80 (or a "8% improvement), is actually a 23% reduction in mistakes if you notice that it went from failing 26% of the time to just 20. and reliability in a system is not something that is valued linearly. a system with only 50% accuracy is as good as useless, while something approaching 90% starts to get actually useful. According to that graph, gpt-3.5 is probably around 57 score. compared to gpt-4's score of roughly 67, that was a reduction of only 15%. so gpt4t to gpt4o was a bigger jump in reliability.

[@snailedlt]

This makes a whole lot sense! Great explanation!

[@ericka.montanez6821]

Same as uptime. Going from 99% availability to 99.9% is not a 0.9% improvement, but rather a 90% reduction of downtime.

[@ZeerakImran]

not to mention the massive speed improvements and cost reduction for using these models within a year's time. Not sure how much of that can be placed on other factors (infrastructure, funding..) but Microsoft has been backing openAI for a while now so I'll put it down on the models improving (though it could be the increasing competition..). I think it is the models that have been made significantly faster and cheaper than before.

[@othercryptoaccount]

yeah this dude started with a conclusion and tried to find arguments to support it. I'm definitely "worried" about a slowdown, but this video actually made me slightly less worried lol

[@dg223-p1r]

This claim of 'smaller and smaller wins' also assumes that these benchmarks scale linearly, which likely isn't true. It's possible that while 70% of the questions are easy to answer, each subsequent 5% is significantly harder, than the last and require more training. So a 'small' n% gain in performance GPT-4o gains over GPT-4 may actually measure a significant jump over a subset of very difficult questions.

[@_nononono_]

Calling Gordon Moore a dev/hardware enthusiast would’ve been funny, if it was intended as a joke

[@NeostormXLMAX]

i mean its that mean like marines vs stick, where they dumb down some ones credicials for meme

[@mattolivier1835]

He was a dev. Deal wit it fool.

[@fischi9129]

small factual correction: "One of the crazy things apple invented, was the idea of having different cores with different roles"... No they didn't, it was actually ARM. Hetereogeneous Computing strategies and Big/Little architectures were not invented by apple :)

[@heidgunther4060]

they also did not invent the ideo of doing video encoding in hardware, idk where he got that from.

[@celloninja]

@@heidgunther4060 Apple users thinking Apple implementing a feature = Apple invented the feature is a very common thing.

[@autohmae]

Yeah, I don't know if ARM was first, but ARM started doing this in 2011.

[@LiveType]

@@celloninja While that may be a thing, apple does "perfect implementation" of ideas more often than not. They have top notch engineers working there.

[@johnrperry5897]

​@@celloninjayou absolutely nailed it hahaha

[@yahm0n]

As we reach higher benchmark scores, you have to flip your understanding of them. If accuracy goes from 80% to 90%, that feels like a 10% improvement, but in reality the error rate has gone down by half, which is basically a 100% improvement.

[@mAny_oThERSs]

You also forgot to mention how extremly flawed, early-stage and biased the ai tests are that are supposed to measure their performance. It is way too abstract to just slap a number on it. You dont say "this guy has 76 intelligence" or "this guy is 13.2% smarter than the other guy". It's artifical intelligence, intelligence, the thing we have. So as hard as it is to accurately measure our own true intelligence as difficult it will be to measure ai intelligence, but that doesn't it plateaued and it doesnt mean the improvements wont be noticeable. Its honestly laughable to think that the top-notch scientific masterminds of this generation and the most valueable companies in the world are all wrong and wasting money, because a guy on youtube read a graph and said "see? It's plateauing!"

[@antman7673]

Thank you, I would have needed to write that comment instead, if it were not for you.

[@NeovanGoth]

I totally understand that a lot of people don't like "AI" (what they actually mean is generative AI), because it became a symbol for everything that is wrong with big tech, just as Glyphosate became a symbol for everything that is wrong with industrial farming. Yet I'm baffled how even tech-savvy people only focus on the results not being completely perfect, while ignoring how much of an achievement it already is to get almost-perfect results. We have computers that easily pass the Turing test.

[@atanas-nikolov]

@@mAny_oThERSs You might want to look into survivorship bias. Plenty of big companies have invested mind-numbing amounts of money into things that never panned out. It's the nature of the game.

[@GraveUypo]

a much better way to say the same thing i tried to. nice.

[@anonymouscommentator]

corrections: 3:00 compares SINGLE thread with an ever growing number of gpu cores lol. i find this graphic highly misleading. peak operations per second continue to follow moores law even on CPUs. people said moores law is plateau-ing for decades now yet it hardly slows down. the reason why you dont notice a difference between the 10700 and the 14900 is because both are "good enough" for normal every day tasks. however in productivity workloads like cinebench 2024 the 10700k scores around 740 points while the 14900k scores 2180. thats almost 3 times as much. 6:04 no, you can run, look at and modify any llama model code. its on their public github repo. you still havent learned that after your bad rant about meta. mistral on the other hand only provides the executable, not the code. difference between open weight (mistral) and open source (meta). 7:30 apple certainly didnt invent hardware accelerators or the the big-little architecture. also neither of that wouldnt show up on the funky graph you showed anyway because it only includes single core performance. 8:10 that is probably one of the worst charts you could have picked to show the slowing down of progress though i do somewhat agree with you.

[@monad_tcp]

He's a web developer, come on , they live inside a browser, which sucks all the gains in performance by being bad tech.

[@NerdyWasTaken]

​@@monad_tcpbro is cooking, even smallest Nextjs(which Theo loves) app uses 400MiB memory.

[@vedshah8784]

I'm actually getting pretty sick of how under-researched his videos are. I noticed this with the Llama 3.1 video where he absolutely butchered what the model size means.

[@Winnetou17]

The graph at 3:00 is classic NVidia shenanigans. These guys are totally incapable to ever produce a graph that's both correct and not misleading. And useful. Other than that, yeah, good corrections. There's even more minor corrections, if someone would be pedantic. I think that Theo deep down knows he's not accurate / properly correct, but overall he gets the big idea pretty well. Edit: one thing that I can give Theo is that he does listen to the suggestions and corrections, well, as much as he can take in a limited amount of time. You can see in this video that when he showed the fast inverse square root he didn't say that John Carmack invented it. That's an improvement on his part.

[@paw565]

Underrated comment

[@marinepower]

It feels less like LLMs have plateaued and more like the benchmarks are all being gamed and optimized for. Claude 3.5 sonnet, for example, is a cut above all other models.

[@PraiseYeezus]

it's not so drastically better that it doesn't belong in the same conversation. The amount of "I apologize, you're absolutely right" I get from Claude doing the typical LLM mistakes almost reads like a parody sometimes.

[@marinepower]

@@PraiseYeezus I think what shows this more is what happened with chatgpt. GPT-4o is drastically dumber than GPT-4, yet it's somehow considered a giant upgrade according to the published metrics.

[@StupidusMaximusTheFirst]

I don't understand what you mean by benchmarks being "gamed" and optimized for? For what? AI/ML will plateau if it hasn't already. This is unavoidable. Maybe there will be small improvements in the same sense hardware plateaued a long time ago and there are only tiny improvements. I realize manufacturers advertise their new hardware as some massive upgrade, it's not. Maybe they'll do the same for AI models. Case in point, I still run a 2012 pc. Long gone are the days where a 286 was more like 1000 (that is one thousand) times faster than a 8086 and the difference was unreal compared to an not so older say 6510 or say a Z80. Now you might get point clock increments and maybe some more cores to compensate. The same thing will happen with AI programs, those who envision singularities are fools. Whether those singularities are AI or black holes, they all be fools imnsho. i.imgflip.com/3pvz1p.jpg

[@backkslashhh]

​@@marinepower Absolutly perfect example of goodharts law!

[@merefield2585]

@@marinepower yeah, even GPT 4 Turbo is clearly better

[@pushpindersingh785]

I disagree with the interoperation of graph at 8:39 , Its a benchmark score out of 100. It will always asymptote and isn't analogous to a TOPS or Transistor count graphs. To see a real asymptote we would want a harder benchmark where we start at lik 0-20% performance, go up to like 50-60% with newer models but stop improving there and well away from a human expert score on the same benchmark

[@MarekMuzyka]

And also time of the release on x axis dont measure how much data were those models trained on nor how much computing power they needed. If there was any plateau it wouldn't be on potential growth of GenAI with bigger models

[@LucyAGI]

I don't know what people expect from an exponential growth, 100% then 200% and so on ?

[@jsrjsr]

​@@LucyAGI we expect all the things tech bros promised when they were hyping chat gpt when it first appeared. Now that it does not deliver , you all cope and become wordcels to justify the failure .

[@neon9165]

@@LucyAGI Goal is, make an AI smart and general enough to where it can perform the tasks of a Machine learning expert and work on ai itself, if you can inference a lot of them, then you have 1000 experts (or more) making ai smarter, more efficient so on, thats where the exponential would kick in (theoretically, we dont know where the ceiling on intelligence is so the "exponential" could be short-lived) When we'll get there is questionable. *maybe* we'll get there with LLM based systems, i believe we'll see llms being used as a weak reasoning engines in a system of tools, verifiers and agents within the next year, possible this falls through though and that llms scaling doesnt make them any better meaning we hit a hard stop ceiling and need to find another architecture alltogether but imo, thats unlikely as of right now / too early to say (as we havent made the aforementioned tools yet)

[@jonp3674]

Yeah came here to post this, agree completely that all benchmarks would look like the scores are flattening as they reach the top. Also with benchmarks like MMLU they've been through carefully and found a bunch of the questions have issues with them such as no right answer, multiple right answers, missing information in the question etc, which means that no system will get above a certain level.

[@jwickerszh]

Aren't the LLM benchmark score here in %? Thus of course there was always going to be a plateau at 100% anyway ... 🤔

[@jmd448]

Yes, I was thinking the same. Although, the issue of not having a better metric (measuring stick) is still true.

[@SahilP2648]

Ask yourself why percentages lmao. It doesn't make any sense.

[@LiveType]

It's the difference between 99% and 99.99% being a 100x improvement despite only being " about 1%".

[@jwickerszh]

@@jmd448 Probably better benchmarks will come out as both performance and expectations evolve. The real issue here is that there are no cold hard metrics to compare, even comparing numbers of parameters isn't exactly equivalent to number of transistors in a chip...

[@Schlafen-wx1kx]

@@LiveType Bullseye. Its a 10000% improvement if my math is right better stated as 100x though

[@RishavKundu]

Just to note, LLMs are the current focus and are unparalleled for natural language processing, but even if LLMs do plateau, I really do think there is further research + neural net architectures that will give us another boost forward in AI progress, over time I can def see multiple 'model architectures' working in tandem to complete complex work. So basically, I think even if hardware compute advancements are slowing, progress and research into the fundamental technology is accelerating, and I hope we will discover breakthroughs which allow us to derive more intelligence from less raw compute. Yes, etched and others are working on altering the hardware architecture to fit neural nets, but there is much to be said to iterating neural net architectures to utilize raw compute orders of magnitude more efficiently.

[@merefield2585]

well they kind of have already .... Llama 3 8B is really very good considering the amount of compute required. GPT 4o-mini is exceptional considering it is likely a tiny fraction of the size of its big brother. But the entire "scale to improve" thing seems like a hiding to nothing and incredibly inefficient. We need another architecture level eureka.

[@cherubin7th]

No all large models convert to be the same.

[@ReedoTV]

This happened with re-enforcement learning too. The models had so many nodes that back propagation had virtually no effect, meaning they became too big to train any more and even got worse with more training.

[@RogueAI]

Our brains are proof that computing density and efficiency have a LONG way to go before they hit a wall

[@charlieyoutube5792]

Good point

[@AB-ms7my]

7:34 Isn't it big-little idea from ARM? What about prior ARM?

[@Garcia98]

It is but we all know Theo's biases towards Apple...

[@monad_tcp]

​@@Garcia98 Apple doesn't even make CPUs , they only optimize the design of the ARM.

[@Winnetou17]

It is. Though it is from 2011, and Apple was using ARM for the iPhones since 2009, right ? They could've had some input on that, though the wikipedia article about big.LITTLE doesn't say anything about Apple in the initial development.

[@alles_moegliche73]

i thought he was rage-baiting, but hes just biased

[@bitwize]

Apple's M series are completely custom designs that use the ARM ISA. But they're significantly different from, say, a Cortex, and contain significant extensions like an option to use a stricter x86 style memory model to make Rosetta2 work at near native x86 speed.

[@ryanquinn1257]

The problem with Moore's Law is it became Moore's Yearly milestone. I'd argue once they realized the trend they would withhold performance knowing putting too much into a gen will guarantee a very small bump in next gen because we are reaching physics barriers where transistors are getting too close to isolate their signals / have them behave properly and hold up over time.

[@monad_tcp]

Intel did that with their 14nm . That's now why they have competition from TSMC and Samsung. They're going to eat that and have to sell all their factories. Their production methods are still behind others. They thought they had a monopoly and were able to keep the next steps and amortize it for the next Gen . Didn't work, their competitors went ahead to the next nodes. But even them are now struggling with their fake 3nm. There's no real 3nm. It's over. CMOS is done, that mine won't put out more diamonds. And any other method ironically wasn't researched because it would be a step down to performance levels of 1990 until 5 generations later, finally it out paces CMOS, that would be 2030. They had to have stopped putting everything on CMOS tech in 2015. It's too late to keep the Moore's law now. Its dead.

[@ryanquinn1257]

@@monad_tcp Yep I can't say I'm surprised what's happening to Intel is happening. You want to be the master of the clock, but it's becoming a honed samauri that's recently fading vs a machine gun. Intel just kept pushing smaller gains for bigger costs. I moved desktops where I could to AMD. The power efficiency savings especially heavy load over time made it easy. I started using macbooks once apple had the mac silica ARM. Late Intel chips were toasters in that aluminum body == recipe for early death. It was also stupid for mobile-use. I it to last long and not worry about "oh no turn off bluetooth, don't play spotify, dont have youtube going". I borrowed a friend's X1 carbon and it lasted like 2.5-3 hrs of real use with its Intel i9 vs easy 8 hrs from the mac. I'll want to see a few good gens in a row to consider Intel again unless really really really is best use-case.

[@SahilP2648]

We would get photonics-based processors and maybe IGZO as the material change. Shunpei Yoshida the original creator of the LCD said IGZO is a really good candidate, offering upto 100x speeds with the same manufacturing and architecture.

[@LiveType]

@@monad_tcp Intel's mistake was thinking they could do 10nm without EUV. They sort of did but 10nm only shipped thanks to EUV.

[@Poctyk]

@@monad_tcp Aren't we shifting to GAA transistors right about now? >Moore's law is dead (again) advanced packaging says "lol"

[@sayakpaul3152]

Some rebuttals here. First, I am an Engineer at Hugging Face, working on generative modeling, which, to me, is not JUST language models. It also include {image,video,speech}-generation models, too. I am mentioning it at the beginning of my comment to let others know that I am not a podcaster nor an influencer who doesn't know his shit. 1> Too bad that you only cared about compute and that too only from a single provider i.e., NVIDIA. TPUs have been there for a long time and has a moderate share of the market. They don't have the problems that NVIDIA GPUs have such as shared memory, availability, etc. It's a completely different stack and when done correctly, can be faster than GPUs. TPUs are, of course, primarily used by Google to train and serve their flagship models. But other companies such as Apple, Midjourney, etc. have been using them as well. 2> You only showcased public benchmarks but in reality, a business that is remotely serious about integrating "AI stuff" will have internal and private benchmarks which will keep on evolving. Also FWIW, in the domain of {image,video}-generation none of the models exhibit the performance ceiling yet. Let's not discard those models as they are quite business-friendly. 3> Model architectures and modalities exhibit different properties and hence, their dynamics are different. For example, Transformers lack inductive priors as CNNs, so, they usually need more data to counter that. Then, you have LLMs which tend to be memory-bound, whereas diffusion models tend to be both compute and memory bound. This leads to different engineering practices. 4> 3. brings me to this point. We have seen technological advancements across all facets -- data, architecture, pre-training, optimization, etc. Not all these axes have been fully exhausted. As long as there is a good dynamic in any one of them, things will keep improving is what I believe.

[@dirremoire]

All your points are very much true, but you might be too hard on him. He's just a KZbinr.

[@damancia_rajongo_1]

@@dirremoire A KZbinr with a huge audience, which leads to influence and power over his audience. Every content creator should be aware that the mistakes they make in their videos are going to be parroted from then on until someone with a similar audience size can disprove them - but by then, the damage was already done.

[@jeff__w]

I’m just a regular person, not a tech or engineering guy-I had to look up to see what TPUs were, for example-and your comment was very helpful and informative. Thanks!

[@JohnKruse]

Clearly the things that you cite are why Kurzweil doesn't use Moore's Law as the basis for his projections. He uses "computations per second per dollar", which is tech agnostic. I work with LLM APIs regularly and am convinced that they are great with Kahneman's System 1 thinking, and once System 2 is brought online, the floodgates will open for widespread use.

[@LanceBryantGrigg]

He is right on a few things, that is, Theo. We consistently over reach our expectations whenever technology moves forward and pretend that this time is different. Your response here is clearly another example of that as you have not really rebutted anything so much as chosen cavoites. It does make sense from a point of view for sure, that is; you are in the thick of things working on it every day, your likelihood to be overly optimistic about what you work on is high, meanwhile a guy like Theo is dealing with a double whammy of obsolescence, not only his ability to be an influencer is under threat by content creation, his primary skill as a developer is also under threat. My point here is that you both have an extreme bias towards the situation which is part and parcel why you responded at all. For Number 1; Not saying that TPU's aren't a better platform but this is probably more akin to comparing ARM To x86 architectures. ARM has been "better for a decade". But why isn't it fully adopted still? Exactly. You want to just ignore that entirely. I feel like number 2 is really just changing the discussion. 3 and 4, I guess time will tell. I personally have noticed that the expectations have gone up on me dramatically as a programmer, and as I use these tools, I often find that they still are unable to do basic anything and it requires so much work to create a final solution that I just would have been best to type it out myself and mentally break down the problem myself. AI is good as a text processor to help me manage and maintain lists and do mass text manipulation, its great at helping inspire me towards choosing a direction, but its ultimately terrible at doing anything really intelligent. For sure we are slowing down, I think even you are agreeing with that, but there is no reason we are going to slow down forever, it may speed up again; trying to predict this is a bit insane.

[@djayjp]

Lol Apple didn't invent (or first use in a chip) efficiency cores or video decoder cores.

[@TheGoncas2]

The message of the video is correct, but I can tell you have no idea of what you're talking about, you're just making wrong connections and interpretations of what's in front of you.

[@ChrisHillASMR]

All scientism is like that

[@truthmatters7573]

You really misread that graph around 17:00... it's not a progression of a single technology across the hype cycle with every point being a further refinement that brings it further along the cycle, but rather each point is an independent technology that currently exists at some point of its own hype cycle.

[@ericjmorey]

Yeah, he skipped over the key legend at the bottom and missed the point of that graphic

[@cem_kaya]

i get your point, and you are correct. LLM's might hit a plato however, this does not mean it will stop the development of ai products. The product innovation and foundation models do not move at the same speed.

[@incription]

you have no idea what you are talking about, turbo is a smaller faster version of gpt 4, of course it wouldnt be better, it was a leap for it to have clkose to the same performance, and 4o was an even smaller model that is multimodel, so it can input and output text, image and audio all at once, which has never been done before at that performance level

[@EDM-Jigger]

Bro there are tons of experts saying that we are plateauing, and I have been saying Ai was going to plateau. like, literally, if you understood, how computers work. it would be really hard for you. to try to rationalize. how a. I wasn't going to plateau.

[@incription]

@@EDM-Jigger there is a relationship between loss and compute, basically we are bottlenecked by compute, bear in mind current AI models are less than 0.1% of the size of one human brain, and our architectures arent as efficient as the brain. We are still very early on in AI and it wont plateau because we will keep finding ways to improve models

[@EDM-Jigger]

@@incription Yes, and or no, I'm going to get any stronger underneath a classical compute logic. How would it if it's 0.1 percent the size of the human in. a 4.7 gigahertz. processing core is only capable. of completing 4.7 billion calculations a second. and we can't cramt transistor zone any smaller. without having an electrical current jump to the next transistor creating an ecc error. in the L-3 cache of your CPU. How on Earth do you think we're going to get that 4 billion into the trillions? underneath this classical compute logic. of true and false which stands for one. or zero.

[@relaxandlearn7996]

So many bad takes and half-knowledge. Every fucking Technology that uses a computer chip, evolves over many models in a S-curve, and every fucking time we get to the and of one S-curve we have already a new tech that performs worse but than exceeds the old s-curve.. this happen over the last 40 years over thousand times, and this will not stop. this has nothing to do with economics, nothing with bad software or even with society, technology is at a point where we see every 4-6 month a new "thing" that is in its low 10% performance but beats the old "thing" that is already in its 90% performance. Every fucking time. Only Wars,Diseases and market crashes stop this for a fev months.

[@kurt7020]

LLM problem: The set of all plausible (but incorrect) outputs grows exponentially faster than the set of all correct outputs.

[@KalaniMakutu]

Even amidst authoritative data sources used during training, not all factually accurate data is equally true or helpful given context. And context is something that LLMs struggle with and sometimes hyper fixate upon. For example: “The sky is blue.” True, but… is it as true as “The sky is sky blue”? What about “the cloudless sky at mid-day has the optical wavelength: 470nm”?

[@therainman7777]

That’s not actually a problem. That was a silly little concept that Yann LeCun postulated and which has been proven completely wrong; this does not cause problems for LLMs at all.

[@kurt7020]

@@therainman7777 Citation?

[@jimmyn8574]

@@kurt7020 simple solution: feed correct data only

[@kurt7020]

@@jimmyn8574 Generally if the solution is simple and obvious - you don't understand the problem. I have found this to be true of almost everything in code and life in general. Nothing is every easy. Learn. Try hard. Be kind. Best of luck!

[@MoeShlomo]

The AI model release chart doesn't look like it's plateaued yet as Theo stated. It still looks like it's still linearly progressing in terms of benchmark scores relative to time for each of the three models.

[@exponentialXP]

if you give a gpt double the compute, the improvement is the same forever due to scaling laws

[@entropiceffect]

A language model can read all the websites and books it wants to about how to ride a bike, but unless it actually has to control a bike and perform specific tasks it will suck at them, it will never be an expert on riding a bike. This is the fundamental flaw with current LLM models, they're fancy search engines (they have a use - but also a limit)

[@exponentialXP]

@@entropiceffect No if an LLM is big enough according to transformer scaling laws, its outputs will be undistinguishable from its inputs, so yeah a 405B model wont be able to but a 405T model will be able to. For example, if an LLM did have to ride a bike you'd give it a picture of the bike your using and than use an LLM to predict body movements, right? Well, since it has learned so much about bikes in its training data and because it is so large and can generalize so well it could do the task with ease. We never though an LLM could reason or simulate the world but when we scaled it up it could do it. This video literally stated that transformers wouldn't get better anymore even though all the newest papers show completely linear improvements eg llama 3 paper, this video was just complete misinformation

[@therainman7777]

@@exponentialXPIt’s not the same forever; scaling all plateau eventually. But, we’re not there yet and he’s wrong to say we’ve plateaued. He made a number of mistakes in this video.

[@therainman7777]

@@entropiceffectNo, that’s simply not true. You’re talking about tacit knowledge as opposed to explicit knowledge, but you don’t need tacit knowledge to be an expert at something; you only need tacit knowledge to actually _do_ that thing. Since we don’t need LLMs to actually ride around on bikes, they don’t need a that tacit knowledge. But LLMs can easily be experts on how to ride bikes (and many already are).

[@GoldenTV3]

I think the whole point of singularity isn't that each specific component will get infinitely better, it's that that component will help us discover new components in which those components will help us discover new ones. And the rate of speed between accelerating quicker and quicker. Just as agriculture was a component that lead to the industrial revolution Then the industrial revolution to electricity Electricity to the computer age Computer age to internet Internet to AI AI to ... And all of these things had sub branches. Computers helping with simulations. Electricity to new forms of motors etc...

[@CHROMIUMHEROmusic]

Yeah big things are always built on iteration. There will be more advancements, we (obviously) just don't know what/how yet. as it always has been

[@matt.stevick]

“we are in Moore’s Law squared” - Jensen ai is beginning to synthesize data and creating its own ai models that will create its own synthesized data and ai models based on what taught it, etc. in a positive loop 🔁. (This is September 22, 2024 today). Eventually this will grow at such an automated pace that we will be living with AGI then ASI. it’s kinda hard to imagine what things will be like even by September 2025.

[@samhaskell8868]

Thing AI advocates always miss, is that the growth of these models is rapidly constrained by compute and energy resources, those exponential curves assume society decides that AI is a good place to direct all those resources

[@thegrumpydeveloper]

And nvidias ceiling (money grab) on vram. My hope is Mozilla’s llamafile gets use closer to at least being able to offload more to cpu and ram.

[@3_smh_3]

Hubris. I tried to say to some of these enthusiasts how the electricity consumption by these models is not very viable. The only response I got, 'hur dur, it will get more efficient'. As someone from a third-world, I understand there far more important things electricity is needed for than text and image generations.

[@monad_tcp]

​@@thegrumpydeveloperit always make me angry. Why don't we have an ATX standard for GPUs to decouple everything , why do we need to buy RAM from Nvidia at extortion prices for the same shit everyone uses in their main memory. VRAM is RAM, the only difference is that your CPU bus is 512bits (4 channels of 64bits) and GPUs are 4096bits and they have a couple of extra lines for synchronization as their access is less random. But it's roughly the same shit, witn small changes in the memory controller.

[@Babbler49]

Always remember that exponential curves IRL always turn into logistics curves once they hit some kind of limiting factor.

[@monad_tcp]

​​@@3_smh_3 they don't care it cost $1/h to use GPUs and when the fake marketing idea of AI is to replace humans with minimum wage of $15/h. Capitalists salivate at that thought. No one is looking at countries that earn less than that, or where energy is expensive. Also third world countries are corrupt, their energy cost is usually way, way less than rich countries, it's all taxes on energy, that's why they're poor. The energy cost is actually not higher because there's little demand. Which is ironical as the data labeling used to make AI is probably $0.2/h for human actually doing the work. AI is basically just outsourcing, but in time instead of space.

[@mstr293]

15:50 Hype Cycles and how it connects to the stock, forex, crypto market.

[@Penrose707]

Takes like these are going to age like such horrid milk lmao

[@mojajojajo]

Gordon Moore “Some Dev” 😂

[@potato-yx5te]

Gordon Ramsay "some cook"

[@YadraVoat]

@@potato-yx5te That likewise is a terribly inadequate parallel...but at least they are both named Gordon. 😜

[@potato-yx5te]

@@YadraVoat I agree haha, im sorry as well.

[@mattolivier1835]

He was a dev. Deal wit it fool. Don't cry.

[@mojajojajo]

@@mattolivier1835 don’t be a clown here

[@steve-k3b]

The reason performance is plateuing is because the scores are getting close to the maximum value of 100%. You obviously can't increase by 30%+ once you get to 90%. This is the S-curve phenomenon.

[@steve_jabz]

Turbo and Omni models aren't about scaling up. They already said after GPT-4 was trained that their plan was to first make them efficient and multimodal before scaling up again, since that was far more important and sustainable. I'll wait for actual data on the next model up before I make assertions that require it.

[@exponentialXP]

the whole point of the architecture behind chatgpt is that it scales completely log(compute) linearly, it will never plateau

[@exponentialXP]

Give a gpt double the compute as much as you want and th3 improvem3nt is the same

[@RasmusSchultz]

regarding improvements over time - as Nate B Jones pointed out on his channel recently, we're measuring progress incorrectly for where we're at on that curve. while going from 90% to 95% in a benchmark looks like a small improvement, this is actually a whopping 2x improvement *in error rate* which is what we're trying to reduce now. when you're measuring progress towards a limited goal of 100%, progress is going to look slow towards the end, but that probably just means we need new benchmarks and different ways to evaluate the results. with regards to the ARC-AGI test: I don't understand how or why this is a test of language models - it's not a language problem, as far as I can tell? and I'd expect, if there's an AI (not an LLM) that can teach itself to play Go, the same AI would be able to figure out the simple rules of this "game"? so this just looks like a simple case of "wrong tool for the job"? I wouldn't expect a language model to be good at this, why would it be?

[@SahilP2648]

Yet if we are 10-5% away from 100% and if 100% is supposed to be AGI, we are nowhere close yet. Using percentages in benchmarks is so stupid.

[@RasmusSchultz]

@@SahilP2648 there's no real agreement on the exact definition of AGI, as far as I'm aware. but I don't think it's merely about perfecting current benchmarks.

[@autohmae]

The most 'general thinking' AI systems we have are LLMs I think the idea of the ARC-AGI test is to try and get the LLMs to do logic and maybe math, because that's the part the LLMs are bad at. Or maybe they want to have a good way for LLMs to outsource it to a module which can do it correctly. Any solution would do.

[@SahilP2648]

@@RasmusSchultz when a model can derive new maths and physics equations like 20th century scientists, we have achieved AGI. That's a hallmark test. Doing this currently requires human intelligence and machines can't help, only experimentally. Theoretical physics is bound by human intelligence.

[@RasmusSchultz]

@@SahilP2648 this just doesn't sound like something a language model can do, at least not in their current form, being essentially just next word prediction. as you said, maybe through tool use, but what kind of tools? we already let them run Python code. I don't know. we'll see I guess. I'm a lot more skeptical than I was 3 months ago, because I don't feel like we're seeing the same breakthroughs, or even promises of breakthroughs. but we'll see 🙂

[@frankjohannessen6383]

GPT-4o is obviously much better than GPT-4 turbo. And Claude sonnet 3.5 is better than GPT-4o. And the smaller models like Phi3, Gemma2 and Llama 3.1 are rivaling the much,much bigger legacy ones like GPT-4. As for the benchmarks you can't just look at the increment in absolute value. You have to consider that it gets harder and harder to increase: going from 96->97 will probably require the same amount of "improvement" as going from 70->85. And the benchmarks are also gameable. There are plenty of fine-tunes with significantly higher values than the original models without them seemingly being better in general.

[@magnanova]

Apple M1 single thread geekbench 6 score = ~2400 Apple M4 single thread geekbench 6 score = ~3700 So 54% improvement in 3 years. ~15% per year. MultiThreaded performance can increase as much as you want to pay for extra cores.

[@DanSolowastaken]

Since you asked, A lot of it is the co-piloting that makes it so remarkable. Sure LLM's on their own won't replace 100% of jobs. However what we are learning is that "good enough" showed up last year and we are learning how to make better use of what we've got. As you mentioned the hardware will be a big part of it, but like you said we have diminishing returns there. What we are quite likely to see is an app or likely brand new kind of software that shifts gears through different kinds of AI. When the first steam engines were pumping water out of coal mines, they had this perspective. We need to realize that we don't need more efficient engines initially, what we need to learn how to adapt what we've got in new ways. *That* is the way we'll find under-the-hood AGI.

[@TheManinBlack9054]

I think you are confusing AI with LLMs. LLMs are a subfield of ML and ML is a subfield of AI. AI is a big field with many things in it.

[@Arthur-jg4ji]

yup that is infuriating bro he forgot all the other field that progress extremely fast like the image recognition or voice ect

[@p4rzy123]

Yeah thats excactly what I also thought

[@ckq]

But the LLM paradigm is what is considered the future of AI. Almost everything can be viewed the way LLMs do (states, next token prediction) and so all these other AI models would be obsolete if the hype about LLMs and transformers are real.

[@DamoOne]

Great video. Makes sense. I would actually say that the amount of sense this video makes will double every year.

[@I-AMSelf]

Aged poorly

[@DanLuna-z5w]

how so?

[@codeChuck]

Good point! It is funny how ai is responding to the 'count the r in strawberry'. Always amuses me :D

[@noggyatthebar]

The current models need to be scaled up. We're nowhere close to hitting the "physics" equivalent for AI

[@timsell8751]

Except they're getting smaller as time goes on now. Smaller yet more efficient. If rumors end up being true, gpt-4o mini is something like 8 billion parameters.

[@prestonrasmussen1758]

@@timsell8751it’s both. The largest models are getting larger and the smaller models are catching up quick to the previous generation of the larger models

[@noggyatthebar]

@@timsell8751 that isn't an argument against, if the newer traning techniques are applied to the larger models we'll see a jump. They haven't been applied coz it's probably expensive to retrain the larger ones. If anything shows that the more efficient techniques will create an even bigger step up for gptv5

[@googleyoutubechannel8554]

Imagine how good this video could have been if this guy had remembered the word 'asymptotic' from his college classes...

[@pe3094]

The entire history of human technological advancement has been filled with plateaus and eventual moments of overcoming them.

[@Tom-rt2]

Sure, but the question is, how long will the plateau last? A few years? Or are we talking about decades before we get another breakthrough?

[@pe3094]

@@Tom-rt2 I'd give elementary level rudimentary agi 5-10 years to emerge, with notable marked improvement every 5 to 10 years thereafter.

[@Yavin4]

@@Tom-rt2 Even at the current plateau, AI renders vast business practices obsolete. We haven't even begun adapting it. I just went to the DMV to renew my license and was given paper form to fill out.

[@Katatonya]

@@Tom-rt2 There's no current plateau in my opinion. We're FAR from one. Companies are releasing smaller and smaller improvements to their models instead of waiting many months to release big ones. We didn't get GPT 1, 2, 3, 4 one after another, there was significant time between their releases, and this time Sam himself said they'll instead release smaller, more frequent models, hence they're called 4turbo, 4o, etc. If the strawberry leaks are to be believed, GPT5 that they have internally is insanely powerful, it has a much bigger jump than 3 to 4. Except they can't figure out safety, so they won't release it any time soon. If for any reason they're going to slow down, isn't because of a plateau in performance, but a plateau in safety.

[@AFellowCyberman]

Yeah this channel and the commentators here are just coping because they will only be able to date AI gfs in 30 years.

[@darelblack4666]

The graph you are showing is for a single-threaded cpu, but we actually started to have multiple cpus working in parallel so the moores law kinda still works

[@bagofmanytricks]

Moore's Law is not about it being 2 times "faster" or 2 times more dense, just that we can make working chips with double the amount of transistors (they can still be bigger, more cores and so on, the biggest chips now are usually GPU's). It still works that way although it's moore (pun) like every 2.5 years or so now, and instead of just making things smaller we do more layers and have better heat transfer. And the LLM's were always at a plateau, it's just that we can put in words what the limitations are now. All improvements now is just adding abstractions on top that make better prompts for you, a kind of transpilation and db layer that's not directly related to LLM's at all.

[@asdfwtghwagvb]

No shit a score out of 100 plateaus rapidly. You're completely wrong about OOM increases in computational load on these models, GPT4o was CHEAPER to train than GPT4, there was not a comparable OOM change as there was from 3.5 to 4. There are very simple reasons to argue that LLM performance might be plateauing and that it certainly will pretty soon, but this chart is not it.

[@koma7778]

You are quite smart. I give you the props. Usually i spend my time criticizing content online but this stuff here is solid. No bs.

[@plarion12]

"everything that can be invented has been invented." Charles H. Duell, 1981

[@brandonjade2146]

I hope no one took him seriously after that second again

[@gil6875]

AI will not stop Improving. But the models of computers themselves and the way scientists train and bulit AI .might have to significantly change

[@itsdakideli755]

Very bold title coming from someone who has no experience in this field.

[@therainman7777]

Yeah this guy’s a clown for this video. Too many mistakes, misunderstandings, and unwarranted assumptions to count.

[@Phal-nz3xp]

Yes this youtuber is an absolute clueless clown

[@freeottis]

Ad hominem

[@therainman7777]

@@freeottis It’s not an ad hominem. If anything it’s an appeal to authority, but OP wasn’t claiming that the guy is wrong simply because he doesn’t have experience. He was commenting on the extreme confidence this guy makes his claims with, when even experts in the field of AI with 20+ years of experience tend not to speak in such certainties when making predictions, because they’re educated enough to know that it’s a very difficult field to predict. Especially when it’s moving as rapidly as it is now.

[@freeottis]

@@therainman7777 why is it not an ad hominem?

[@TheGoodContent37]

Dude, when someone completely develops a "video to video" AI that would be PLENTY to change the entertainment industry for decades to come! All AI art efforts should focus exclusively on that. It's been years since Ebsynth started to develop that.

[@hank9th]

Highly recommend the book "The Alignment Problem". Does a great job taking you through the history of AI progress, and one of my big takeaways was that AI advances in a far more "stair step" way than most other forms of digital technology.

[@samhindhaugh696]

from a lot of peoples perspective there was nothing then suddenly GPT-3 not realising the decades of research and the newly developed transformer model that went into that and are now expecting that same growth. Its finding a new break through, fully optimising that new model, then waiting for another Phd thesis to come out again.

[@kraldada6557]

​@@samhindhaugh696 well there were some revolutionary thesis last week. where a model competed in mathematics on a level of 17 years old genius. solving international mathematics olympiad and achieving silver medal. Yes it might take couple years to include this in a day-to-day LLM model but the thesis is here. so it's probably about the alignment problem more or less

[@deeplearningexplained]

The actual next step here will be similar to what happened in early 2015 with vision models when we realized most weights were useless and wasting compute, we'll optimize. It's a very natural cycle in AI research that has been repeating forever now. However, one thing that is super weird with the current scale of models we have is the emerging ability that some of the models seems to be exhibiting. Like very very large models aren't behaving the same way as their "smaller" counterpart (who still have billions of parameters). Some of the largest models out there behave like they are able to be meta-learner, even without changing their internal weights.[1] What I think will happens now is we'll converge into a semi-universal way of encoding the data, which will force a change and optimization of architectures (which already is kinda happening across the different provider). If you look at the massive success that Meta had with Segment Anything family of model it's pretty clear that's the flow. [2] These two force combined will give us a first truly multi-modal AI that will be less wasteful than current mega models, while keeping the interesting emerging abilities. That + more optimized compute will give way to the next wave of AI advancement for sure. [1] research.google/blog/larger-language-models-do-in-context-learning-differently/ [2] ai.meta.com/sam2/

[@supercheapme]

The platou in performance may have something to do with the benchmarks used in the graph, models are achieving 95%+ on MMLU and MATH so there isn't much room to see improvement. Hugging face revamped a bunch of the benchmarks for that reason

[@exponentialXP]

no its because they have less difference in compute, and you can scale llms infinitely btw theres no plateau according to transformer scaling laws

[@scottpatrick801]

I had this realization recently. I've always felt closely tied to technology, especially when you're constantly having dreams and visions about the "future". What's interesting, while playing Atari 2600 40+ years ago, I had a vision of a future computer/game console that was a crystal or diamond cube that had "Mind-Fi", a literal program that played in your mind that mimicked reality. No wires, plugs, controllers, or physical storage medium. Just a 3" clear cube lol

[@SearchingForSounds]

I agree that raw LLM intelligence is slowing. Claude Sonnet 3.5 's big leap for me is the artifact and project knowledge base features they built into the product... not the actual llm itself. The LLM being trained to work smarter with tools is the next incremental improvement. For example, can I fine tune llama 3.1 to work with a custom vscode extension that can search, replace and delete inside my codebase? That would smart tool use. Not AGI. Anyone else thinking like this?

[@monad_tcp]

I trained one to predict commands in my command line based on my history. I would never trust any cloud service with that data, I had to do it myself on my own hardware.

[@R1L1.]

Claude 3.5 the model itself is also a big improvement, why? Cuz the small improvements it made over lets say gtp4, ppl were saying it would be very hard to and would take years do but they managed. But yeah artifacts is insane, read somewhere its a system prompt so that's even more insane.

[@exponentialXP]

no llms scale log(compute) linearly infinitely with no plateau, the reason it is slowing down is because there is less compute difference between model v2 and v3 eg

[@antiderrida2117]

“Single threaded” was never a qualifier for moores law

[@jaysonrees738]

One of the biggest things that will change how fast things can be is the semiconductor material. Everyone keeps trying to make silicon faster, but we've already found better options.

[@emmanuelgoldstein3682]

Imagine rage baiting data scientists while wearing your mom's shirt.

[@GeatMasta]

8:35 i feel like this is missing that each of the models on the GPT4 line are getting smaller and cheeper; they are not intended to be smarter they just happen to be, the intent is to get the thing to run on a phone.

[@Douchebagus]

Why do people keep saying AI isn't going to keep improving when everyone who stays current in the field sees the opposite? The last 8 months have been insane and it's getting even more insane over time.

[@korozsitamas]

because most benchmarks are flawed, and some pretend they aren't

[@exponentialXP]

if you give an llm double the compute its improvement is the same, it doesnt plateau.

[@joseijosei]

A lot of people don't want AI to do good, because of X reasons. This means they are biased. Isn't console gaming still kinda popular? Why are there so many people saying it's better than PC gaming? Those two completely different topics have the same answer: people want to feel like they made the right choice. Like if they took the right career, choose the right gaming platform, got the right job, etc. Btw, if you are programmer, and you are reading this thinking that I'm talking about your job, because that's the popular thing to say: no, AI will not replace your job. I know it. You know it. Everyone knows it. Again, people LOVE to say that because they are biased. They love to believe that they were "saved" by dropping out of college, or that they avoided a huge waste of time by not studying, wanting their time partying or in videogames. Again, when people want to believe something, they will. Let them talk. Reality hits hard, and not only programming is among the safest career now that AI is a thing, but also, chances are they are doing one of those jobs that can easily get replaced by AI, actually do their job, and they aren't even paying attention to it.

[@realkyunu]

Everyone in the field just wants to see the opposite. Obviously, they need to overhype the tech, otherwise investors just flee.

[@rsingh2083]

Because username : check

[@ChRW123]

I think apart from microchip architecture and other technical issues the main wall AI is going to face will be energy consumption. At a certain point it will drain too much energy from other fields in society making people question if it is worth it.

[@AjinkyaBhushan]

What I can see from this problem is we are not able to scale up because of: 1. Hardware Limitations 2. Algorithm Limitations 3. Poor quality data.

[@blackstratmx]

So we cannot scale, thats it

[@SahilP2648]

We need a quantum supercomputer, and we need a new algorithm designed from the ground up for quantum computers. Research Orch OR theory by Sir Roger Penrose. If this theory is to be believed, then consciousness is because of quantum entanglement which means quantum mechanics plays a curious role. This would also mean all these benchmarks are futile since we can't even predict yet how exponentially superior quantum computers can potentially be, and by extension the AGI systems developed on quantum computers.

[@AjinkyaBhushan]

@@SahilP2648 Quantum computers aren't a new tech. Its like more that 12 years when it was successfully. Actually, quantum computers aren't traditional computers. They big, expensive and critical science project. I think quantum computers are really hard to work with. We all have to see what future is going to be like. LLM's are great but again they hitting their limitation not on the hardware end I would say (not at the current moment) but dealing with algorithm issue, we need to research more new algorithms. And we don't have quality data, new legislation are going to be another challenge.

[@autohmae]

I'm always amazed when someone brings up quantum computers, because quantum computers just solve 1 class of math problem we can't solve with traditional computers. Pretty certain the GPUs we use for LLMs/AI, don't depend on this math problem class or aren't limited by this specific math problem class. What I do know the specific class of math problem, GPUs obviously are the most efficient for it (when not using quantum computing), because they are the best for math. Their are obviously people working on trying to use quantum computing for AI, but that's a big research topic with no easy solution.

[@MrXelaim]

4. Inbreeding

[@spazneria]

Ahhhh it is so validating to see o1 drop. You're all misunderstanding. AI progress is not about progress in one regime, it is a uniquely synergetic field. Looking at progress of one part of the whole does not give you an appropriate perspective on what's happening.

[@JazzyMaxine]

3:48 I got a secondhand M1 (with the minimum memory and storage) from FB marketplace for £200, not including shipping >:) It has a small aesthetic defect but works completely fine! I felt so proud of that deal.

[@SahilP2648]

Not bad but considering the recent need for memory, you would need to upgrade soon. Enjoy it while you can though.

[@ren.oooooo]

You back the hypothesis with data and I respect you for that

[@kintustis]

comments are: -criticizing the benchmarks -attacking his credentials -nitpicking semantics without arguing the point -"wait for -...." -accusing people of having no vision while providing no concrete examples of practical uses -comparing their ai girlfriends to the invention of the wheel

[@jameskamotho7513]

They are quite emotional. As a layman, I expected to learn from them but all I got was people just being salty...

[@SimonHuggins]

Aha. Until the next leap forwards. I don’t get how people don’t get that innovation comes in leaps and then increments that optimize the leap, then the next leap etc. With some leaps actually being dead ends long-term. We are so used to the fast moving increments we miss that leaps are happening all the time but take time to become integrated and then incremented upon.

[@plebbit]

this video is gonna age like milk

[@Zeekee2]

🍓

[@nawabifaissal9625]

already has lol

[@F0XRunner]

I refurbished my PC around 6 years ago with a 1080 NVIDIA graphics card. I have yet to need to upgrade it for any reason. Not remotely surprised how quickly AI is plateauing given how much money is being poured into its development.

[@thomassynths]

Meanwhile Groq is running faster than the latest Nvidia chips, all the while running on ancient 14nm printouts.

[@SahilP2648]

We still need Nvidia hardware for training thanks to CUDA, but that might change because of the ZLUDA project, allowing AMD GPUs to use CUDA code. But I only have surface level knowledge about all this, so I am not sure. And Groq's chips are specially designed to have very fast access to memory with faster bus or something, allowing faster processing.

[@timsell8751]

Safe to say, investing in Nvidia right now will turn out to be profitable for some time to come. That is, if you believe that AI is just getting started, as I do. Everything I'm reading points to NVIDIA retaining their lead for 2 - 3 years at least. Fabs ain't easy to make. I did not know that about AMD and Cuda though, gonna have to look that one up. That would be huge if true - and it would be great for everyone going forward, as I'm worried about this insane monopoly that NVIDIA could gain/already has. But even then, AMD is 10-15% of market share compared to 75%-85% with Nvidia. Gonna take quite a lot to significantly impact those numbers.

[@HybridHumaan]

"Moores law is an old law from the programming world" alright im outta here lmfao.

[@Hanzimann1]

You are looking at a one-year window in AI abilities, and even if we disregard the fact that moving closer to a 100 percent correctness in tests will require an exponential effort, there is absolutely no grounds that support that LLMs won't change in structure and training material, and that the hardware will not move from GPUs to something more specialized and intelligent. You cant really compare LLMs to something with physical limitations like transistors/CPUs. It is a bit like taking a handful of neurons and mashing them all together expecting to get a brain out of it.

[@photonboy999]

*let me summarize...* 1) language models that "train" using scraped data will massively slow down (overhyped and lack of new data to learn from), whereas 2) Machine Learning that involves setting up a problem to solve and having the computer learn to solve it will accelerate (i.e. NVidia DLSS upscaling... make low res image look like high-res image... and eventually make UNHEALTHY cell look like healthy cell. i.e. devise a treatment)

[@joschomo1010]

Yeeee, i love it when random swe make videos about AI. Unless you are a data scientist, ml engineer, or researcher, your opinions mean less than nothing

[@zedzed3533]

Lmao

[@TheAero]

super narrow perspective. CPUs single core performance has not been increasing rapidly, but energy efficiency/usage has gone down fully. CPUs no longer are used to full capacity, and you can always parallelize to more coress. When we got to 1nm, 0.5nm we will have fully efficient CPUs, that are becoming cheaper every year due to optimization in manufacturing and reduced marketing. Then it will be all about stacking more cores, utilizing better cooling and probably switching to ARM, which can give huge boosts to compute. Also most things are going to the GPU side of things, so we may soon see a shortage of need for CPUs. Finally till we plato, which will happen in 5-6 years, quantum and AI will be the main pieces. The modern computer will probably be out of usage in 10 years time tops. It's probably gonna be server based connection with a subscription, and you just having a screen, keyboard, mouse to access using satellite or 6G outside, or wifi through home.

[@lucastononrodrigues1069]

kind of dumb at minute 9 to use that graph as an argument, if you have a max score benchmark of course it is going to plateau, thats when you create harder benchmarks.. if you are looking for arguments for whether we are or not in a AI bubble you are going to find very convincing and plausible conclusions in both sides - not very productive

[@exponentialXP]

i dont find any convincing arguments for it fo stop improving, because it just doesnt plateau, it scales infinitely according to transformer scaling laws (this is why transformers are used btw), qnd doubling the compute always gives the same improvement.

[@exponentialXP]

its not even rarely disputed that if you give an llm enough compute it will have 100% accuracy and be undistiguashable from its training dataset, and since scaling laws are linear we know this for sure. now we have transformers instead of cnns for image generation and audio generation so theyll be indistinguishable from the training set aswell. The reason Sora had such good quality was because they had a much larger transformer than anyone else.

[@exponentialXP]

the only reasons i see ai not becoming human level is if we run out of data and augmented/completely new artificial data is not an option or we cant figure out a way for llms to be autonomous and if they turn out not to be creative enough (in that case we need a loss function). im not sure of this youtubers point

[@brambasiel]

Didn't expect Theo of all people bringing up Moore's Law. 😉

[@Jayc5001]

“AI isn't gonna keep improving” he talks about AI reaching a plateau. That makes no sense in this context. It's funny to me that his own starting example proves why his statement is nonsense. AI labs aren't locked in to current designs. They already know its limitations well. And if they find out it won't lead to AGI they will switch directions without hesitation. They Don’t care about trends or predictions based on what already exists because they have always been at the limit of what is known to be possible. The top AI labs goal has always been to exceed what we know to be possible to achieve their goal. We know AGI is physically possible. We know it's a thing that CAN be done. We have existing examples. But we don’t know HOW it can be done. If we just cared about making one AGI we would just have a kid. Or adopt a pet. We already have intelligent systems we can raise and use. The quest he asks that makes me think he's dumb is if we can improve. I can bet my life on “yes” without hesitation. The goal is achievable. The question in the air that he should have been asking is can we achieve the goal in a timely and cost effective manner. That is what the problem is. In real life right now, if labs hit a roadblock it's their job to keep going. And that's how we make progress.

[@Jayc5001]

“Oh no we hit a roadblock so I guess we should just give up.” Said no one ever. They will keep going until they get to their goal or run out of money trying.

[@Jayc5001]

The reason computers keep getting faster isn't because of some natural force. It got faster because people made it faster. People for various reasons wanted computers to be better at computing, and they have continued to make the breakthroughs necessary to complete that goal. The reason computer architecture continues to change is because the architecture doesn't matter, the goal does. If current computer architecture does not fit our goals then we change it. If current AI systems don’t work, we will simply change it. It wont plateau not because of the existing technology but because of the effort of those working towards their goal.

[@InanimateObject123]

He's obviously talking about LLMs specifically, and he's correct. LLM and AI have erroneously become synonymous at this point

[@jaehparrk]

nah bruh your nvda stock is gon go down once this overrated hype is done it already is goin down

[@Jayc5001]

@@jaehparrk I don't own any Nvidia stock btw.

[@MarekMuzyka]

Was there any graph that showed how GenAI improved over used computing power to train models? I feel like this is the only thing we should look at if we were to believe GenAI can be improved without guiding it in one direction. There were benchmarks scores over release date but that shows like noting

[@KAZVorpal]

The reason for the AI plateau is how bad the major AI Developers are. For example, when it comes to pre-trained Transformer LLMS, there haven't been any real, major advancements since GPT-2. They just keep throwing more resources, and additional side gimmicks, at the existing concept. These create an illusion of progress, but there is no new technology or theory involved. Open AI is the worst thing in machine learning technology, but unfortunately is also the richest, because they've sold out everything they used to pretend to stand for. They are secretive, proprietary, and lazy. They don't produce new ideas or technology, they just keep shoveling money at old technologies.

[@dg223-p1r]

"For example, when it comes to pre-trained Transformer LLMS, there haven't been any real, major advancements since GPT-2. They just keep throwing more resources, and additional side gimmicks, at the existing concept." That just isn't true. RAG, for example, is a significant innovation, not a gimmick. The capacity scale context windows of LMs from 512/1024 (SOTA GPT-2 at the time) to 4096 on every LM and 1M+ on new big ones is a major innovation (likely involving transformer-alternative architectures). Multimodal LMs are an incredible advancement. And last but certainly not least, the ability to make LMs 500-1000x the size of the largest GPT-2 version, that took innovation in compute resources and model training efficiency. That was all in 5 years. There's no illusion in that progress.

[@KAZVorpal]

@@dg223-p1r Wow, they are SO corrupt that they just deleted my rebuttal, which is all facts about how every single part of what you described is nothing but a plugin or expanded resources on what is, essentially, GPT 2. They are criminals.

[@KAZVorpal]

@@dg223-p1r Wow, the criminals keep deleting my entirely-EULA-compliant response. Everything you list is exactly the kind of gimmicks I am talking about. None of them are an advancement of the model.

[@dg223-p1r]

@@KAZVorpal I'm curious what innovation you feel took place in 2019 with GPT-2. The transformer came out 14 months earlier, Dec. 2017. What was so great about GPT-2 that it counts as an innovation on an architecture but nothing since does?

[@KAZVorpal]

@@dg223-p1r You've got me there. GPT 2 was the first implementation that worked well, but that's more a Turing Test thing than an innovation in architecture. I was aware of this, but trying to start with the one that fundamentally behaved like the modern version. So, really, OpenAI has created nothing at all, beyond being the most successful implementation of Vaswani's Attention is All You Need. But, again, I felt that is a separate discussion, what I was focusing on is GPT 2 being what feels like the same thing as the modern product.

[@orphica5739]

Interesting take, but I think the chart's a bit misleading. It cuts off before showing GPT-2 (Feb 2019) and GPT-3 (June 2020), where we saw gradual improvements before a big leap. The increased milestones on the right might look like diminishing returns, but they really reflect growing competition and innovation in the space. We’re seeing more players in the game and continuous advancements rather than a true plateau. The competition is pushing the boundaries, not signaling a slowdown.

[@CharleswoodSpudzyofficial]

Honestly, this would be an absolutely wonderful place for AI to stop

[@Loggerr]

I agree hahaha

[@pablogtzgileta]

💯

[@Wentris71]

Yeah, stops before it takes over our world😂

[@szharychev]

You mean unreliable bs of tech which should’ve transformed how every single job is done? Definitely nice place.

[@Rolyataylor2]

The only reason we are reaching a plateau is because they're being optimized for a set of benchmarks. We need more benchmarks so the models can get good at everything

[@PristinePerceptions]

Theo, you don't know who Gordon Moore is, do you? 😂😂😂

[@davivify]

First, Moore's Law had nothing to do with speed. It was referring to transistor density - or the number of transister that could fit on a die. It was a very nice _side effect,_ for the longest time, that smaller transistors also happened to be faster. As you point out, that's no longer the case. Yet transistors continue to shrink as we begin to explore feature sizes measured in Angstroms. At some point, of course, we'll hit a wall but we're not quite there yet.

[@TripeDemo]

This video has aged like milk lol new video title: "okay now i'm scared" referring to the new o1 modell

[@SouhailEntertainment]

Introduction: AI Plateau Hypothesis (00:00:00) Moore's Law Explained (00:00:29) Decline of Moore's Law (00:01:59) GPU vs CPU Performance Trends (00:04:28) Analog AI Chips and Specialized Compute (00:05:24) Rise of Open-Source AI Models (00:06:18) Potential Plateau in AI Model Improvements (00:07:20) The Bitter Lesson: Focus on General Methods (00:09:53) Importance of Algorithmic Innovations (00:12:33) Future of AI: New Architectures and Methods (00:14:33) Hype Cycle for Artificial Intelligence (00:15:30) The ARC Prize and AGI Benchmarking (00:18:56) Conclusion: Future AI Beyond LLMs (00:21:42)

[@SuperChaoticus]

Cell phones are a great example of tech reaching a plateau. When was the last time we had anything but incremental improvements to our phones? AI is running out of information to train on.

[@jichaelmorgan3796]

Video game graphics also

[@calmhorizons]

And cars. More and more doo-dads, still a box on four wheels.

[@digital_down]

Physics laws in transistors is not the same thing as software architecture in AI. Researchers, such as myself, are well aware of the limitations of current neural networks, but just how the architecture evolved in transistors, neural networks can have obvious improvements than the traditional latent space that we use now. Everyone seems to ignore the entire pipeline and focus on the bigger abstraction of the current architecture slowing down. I have heard the term AI winter year over year since 2018 from the general public, but not so much from researchers. Once again, this is not a physics limitation, it is a software one… one that has a proclivity to change drastically with further research.

[@YanikTheYak]

Autonomous cars are NOT far from functioning, they are driving people around San Francisco. They are here now, hence they are on their way up the second bump in the hype cycle (the realisation phase)

[@iso_2013]

Do they work well with Inclement weather yet?

[@AFellowCyberman]

​@@iso_2013 Not yet but they will. Give it a decade or so.

[@iso_2013]

@@AFellowCyberman A decade or so is not "here now"

[@zukodude487987]

I love that you take the hype cycle chart as fact, based on what data did people determine that AI's hype and decline looks like that. What data points were used to construct the chart at 15:46? Looks like its made up.

[@tehvvisard]

To quote a wise man: "The ability to speak does not make you intelligent". We've only learned the AI to speak not to recognize what to say

[@TheManinBlack9054]

I think it does make someone intelligent, he was wrong

[@mikicerise6250]

That makes it as intelligent as the average person.

[@luckylanno]

It's like we've invented the speech part of a synthetic brain, but we still need to invent the intelligence part.

[@prestonrasmussen1758]

You talking about Theo or the AI?

[@LyricsQuest]

@@luckylanno The problem solving aspect. Ability to figure out effective solutions to novel problems (Not seen anywhere before). I think a good place to start for this is the Millennium Prize Problems. AI is allowed to research, learn appropriate problem-solving techniques on its own, but it must solve the problem at hand.

[@TorBarstad]

Will be interesting to see how things progress. One open question from my perspective is how much further LLM-video-stuff has yet to go (in the short/intermediate term).

[@GalaKrond-b7k]

He posted this a day after Flux released publicly in order to avoid looking like a fool... and this video still ended up aging like milk within a month lmfaoooo 🤣🤣🤣🤣

[@KarlOlofsson]

Not to mention there is a finite amount of documented human knowledge and art to train AIs with. Increasing the capabilities of AIs will hit a wall very soon when all of Wikipedia etc has been used.

[@merefield2585]

I'm pretty sure it's consumed all of Wikipedia already, or do you have a source to the contrary?

[@KarlOlofsson]

@@merefield2585 that's my point, all models probably already have all of Wikipedia and are stuck waiting for humans to make changes and add more pages. AIs can't really come up with their own ideas (and judge their value) yet so they are forced to wait on humans telling them what is correct and not (and we can't even agree most of the time lol)

[@merefield2585]

@@KarlOlofsson thanks for clarifying. Have a good day 👍

[@waltercapa5265]

@@KarlOlofsson We wouldn't want that tho. AI is a tool and we are the ones taking the decisions. Why would training with Wikipedia already be a problem? If you ask a human to learn something, and he reads all the possible knowledge about it, why would that be a problem?

[@KarlOlofsson]

@@waltercapa5265 because you can't call AI intelligent if it can't figure things out for itself, then it's just a glorified search engine or robot. I call them ML-bots. They have to become self learning to not be dependent on human bottlenecks but that will likely never happen. AIs will likely become very simple assistans but nothing more.

[@mattdoestechstuff]

I could be wrong, but I dont think apple had much to do with big little cpu architecture. ARM have had that in their designs for over a decade now. Additonally, the idea of having specific chips that handle specific tasks has been in use by CPU manufacturers for a long time. For example Intel released their Quick Sync technology in 2011 to improve video related operations. So though Apple has this too, its not something Apple has made mainstream or had to "bet" on, its very proven and very common in all cpus from desktops to phones, they are just more abundant in ARM devices due to their reduced instruction set compared to x86

[@gauthierquercia5540]

if you take Yann LeCun as your main "authority", there is a great interview he made just DAYS before Sora was announced. Then you watch Sora trailer by OpenAI, and it's tech papers next to Yann LeCun interview (especially his analysis on transformer capability). You will notice that EVERY SINGLE POINT he made was invalidated by a Trailer and a tech demo. spoiler : Everything he said that LLM and transformer where unable to do, the very next day open AI prove that they could. his whole sch-ting is that transformer a incapable to do real thinking, SORA proved him wrong, Sora and it's "World Simulation Tech" is the ultimate proof that he was wrong. Proof that Transformer tech is Morphing at scale and dependent upon Magnitude of Compute power, that why X, open AI, Microsoft and Guess who ? META are buying insane amount OF GPU to train LLM. Moore Law is completely under the dirt as parallel computing aka "GPU" emerged. those are not impaired by physics, you can stack them to infinity. At that point , you can see that he was flat wrong. even if I like meta for realising LlaMA, they are catching up. he is not a leader in this technology, he deal with scraps, he is bitter to not be a the tip of the spear, it's a pattern for him. to make you point stronger use Geoffrey Hinton, now that a better authority. not a washed up professor that play catch up. if he wasn't for meta nobody will listen to him.