What a day in AI! (Sora, Gemini 1.5, V-JEPA, and lots of news)

[@YannicKilcher]

W&B Course on LLM Structured Outputs: wandb.me/course-yannic OUTLINE: 0:00 - OpenAI Sora 3:25 - Gemini 1.5 with 1 Million Tokens context window 4:50 - V-JEPA 6:50 - Sam Altman raises 7 TRILLION dollars for AI chips 9:30 - Sponsor: Weights & Biases course on Structure Output from LLMs 11:30 - Bard becomes Gemini 13:55 - GOODY-2: The world's most responsible model 16:05 - miqu-1-70b leaked from Mistral 18:25 - Zuckerberg on Meta's open approach to AI models 21:40 - 1X advances robotics 23:30 - Questions around Bard's arena leaderboard position 27:00 - Various other news References: gist.github.com/yk/65fe3d582a43540a61718b9e4b0706d0 (they were too long for this description)

[@andrewm4894]

Clicked and enrolled 🫡

[@grdfhrghrggrtwqqu]

Absolutely no one cares about Sora until we can use it. People that put their sunk cost fallacy into a college degree and need AI to be 'productive'? Give me a break.

[@BoominGame]

Some human will be worst than a 6 nodes network that spews random stuff.

[@skierpage]

Welcome back! We missed you and ML News. David Shapiro has left the building to live in his mind in a future he can taste; Two Minute Papers continues to lazily edit video clips off announcement websites into tired old eye candy while providing zero insight; the veddy British AI-Explained is good but he only covers a few topics. You are da man. A few minor suggestions coming from a place of love and respect: Your OUTLINE in the video description conks out at 27 minutes with 56 minutes still to go! What a massive shot of news. Your gist of references is great but... a) has no timestamps b) if it was a .md Markdown file instead of text, the links in it would be clickable c) you could cut off all the tracking info in the query string part of the URLs (if it doesn't benefit you). 6:52 the curved screen view of each story is cool, but you missed a trick by sliding it off to the left and replacing it with... the same screenshot. Instead the curved screen view should de-warp to full-screen. 13:27 why The Verge hate? Their journalists are better than Business Insider (a low bar), and Nilay Patel's "Apple Vision Pro review: magic, until it's not" review is the best early AVP review I've watched. 39:54 it's pronounced "HerkyuLAYNEium" in English. There are even KZbin videos for it; the most popular one from Pronuncation Guide is computer-generated text-to-speech, and _wrong_! 50:09 the "BUD-E by LAION" model is followed 15 seconds later by the "Bunny" family of lightweight multimodel models", I had to stop and rewind to figure out if buddy and bunny were the same thing. And Bunny is based on LLaVA, not LLaMA or LaMDA or Alpaca. We're going to need LLMs just to come up with new model names. (You didn't credit Bunny's authors, it's from BAAI-DCAI, the "Data-centric AI Group @ Beijing Academy of Artificial Intelligence"; not enough time to say that in 15 seconds.) Thank you!

[@BoominGame]

@@skierpage Yes he is up there with the best, I hope he doesn't give up. Maybe he was a bit disappointed that his crowd-sourced fine-tuning didn't go viral and was put into the shadow with all the new alternative OS model put out there.

[@Timotheeee1]

to put into perspective what 340000 H100s looks like, the entire llama 2 + code llama series required an amount of compute equivalent to about 1000 A100s for 3 months, or about 333 H100s for 3 months. Meta now has 1000x that!

[@IgorGarbuz]

Agree, OpenAI is fully committing to being a product company, but this video model could still serve as a crucial component in the broader architecture of general AI systems they develop. I may think about it as working toward the AGI and commercializing bricks on the go.

[@memegazer]

I agree I they have an ability to produce data, compute, and storage though, in a way that would bottleneck most researchers. To my view these results look like synthetic data generated rendered assets, from either nerf or guass splat assets generated from irl vid sequences/images, and perhaps even AI mocap methods as well (I am not sure how to feel about what was the primary source of data but I suspect strongly that at the very least photorealistic rendering played some role as synthetic data in the training set). The demo vids have an unncanny valley feel reminiscent of photorealism rendering imo. This should lead to some internalization for the model about 3d spatial translation operations and to some extent an uncanny/incoherent physics modeling (judging by the dreamstate like artifacts in some examples). Also it should not go unnoted the level of temporal consestency on display, which could indicate time lapse/object permances is being explored within the problem space by data structuring.

[@mykhailyna1]

Thanks for the episode!!!! We have been missing you!! A while 1 hour of editing must have been tough for you

[@vslaykovsky]

I believe that modeling video sequences IS the path to AGI, because it's the most straightforward way to 'ground' the model into reality.

[@СоюзниксОкинавы]

How do you make it do any useful work then?

[@TheMarcusrobbins]

@@СоюзниксОкинавы Whenever you have something that can generate you can use it to predict. When you can predict you can predict forward what the result of your actions will be. That's half the battle, prediction is a huge part of AGI.

[@MrBillythefisherman]

​​​@@СоюзниксОкинавы simple you make it output from its latent space, text, code, mouse movement, arm movement, leg movement etc etc This is really just around the corner if not possible already by some video-to-text model simply bolted on the end although I think we can do much much better than that.

[@theneuralbit]

Generative - implies hallucinations. JEPA seems more intuitive to me into video understanding

[@jyjjy7]

Most of what consciousness does is literally modelling reality like these videos are doing. Shapiro is more likely correct than not, AGI by the end of the year.

[@MonkeySimius]

I don't think I've seen your channel before, but if most of your videos are of this quality then I'm going to keep coming back. I like that you aren't over hyping the newest and greatest thing while still discussing it.

[@zeev]

I watched this whole video and I LOVED it , thank Yannic. i know there's a lot of AI youtube blogs but most really suck and yours is a way better.

[@andriypashynnyk4278]

very good digest! please keep making those!

[@YvesQuemener]

Happy to see you back!

[@videowatching9576]

Great points about Gemini naming being confusing - i didn’t think of that

[@zzzzzzz8473]

incredibly packed news ! thank you so much for these , there was so much going on that i missed a lot of the various news developments . that study on Escalation of Risks in LLMs , where it chooses to use nukes , is such a flawed premise because its limited to the prompt stating the scenario like a game with limited movesets and simplified outcomes . if you at all stated the possibility of other techniques , diplomacy , inventing alternative weapons/tech , or included the consideration of higher utility function , then the LLMs consider the damage both radiation and political such an action would have . it equivalent to presenting the LLM the trolley problem , and then being appalled it didnt even consider stopping the trolley .

[@TomM-p3o]

Did Gemini 1.5 just kill RAG(vector databases specifically)?

[@elawchess]

I believe that Bard (Gemini Pro) that's in the chatbot arena leader board is likely Ultra, unless it's an early version of the Gemini 1.5 Pro. I used it on there way before Ultra came out and it's definitely not that bard pro we all had. The quality was very GPT-4 like and even the speed at which it was generating felt like that of a bigger model. It could have been Ultra but maybe google didn't want to say it was that, in case it did poorly. It's still on there labelled as that, and you can go and see for yourself that it feels more of like an ultra than what's actually coming out of what was bard gemini pro.

[@matteo-pu7ev]

Thank you Yannic! I really appreciate your content and unpredictable humor. Keep on keeping!! Oh and please don't switch up your dope glasses for ai ones or any other kind for that matter.

[@KolTregaskes]

2:10 I'm sure you've read by now but this *is* actually a step towards AGI. They are building a world engine/model/simulation that can be used for embodiment and thus robotics.

[@natecodesai]

What you said around 2:00 mark is now evidenced by Andrej Karpathy's leaving openAI even though he said there was no drama around it and he wanted to focus on personal stuff.... but might not be the direction he wants to go with it, and wants to keep pushing toward AGI, etc.

[@zbigniewchlebicki478]

I think it would be appropriate to have 2 different leaderboards for evaluation of LLMs - one including external search and one self contained.

[@elawchess]

Following on from my earlier comment about this issue, I really doubt the ranking is anything to do with the internet access. Many questions people ask don't even need very current affairs. The gemini model that is beating GPT-4 on the chatbot arena really is outputting GPT-4 like content when you use it from the chatbot arena. It might have been a case of deliberate mis-labelling from google in case it didn't do well, and it's very likely Ultra in disguise or an early version of 1.5 Pro.

[@winsomehax]

I was sittibg quietly minding ny own business when yannick pops up a notification. Now i have to watch. Sigh. Lol

[@GabrieldeOliveira86]

same here😂

[@Blacky372]

Great to have a new ML News episode on a rainy sunday!

[@igorpalecek3176]

So haooy to see antoher Kilcher News!

[@memegazer]

Imo the way that Gemini performs well over 1 mil tokens is not unrelated to how sora can maintain unprocedented temporal visual consistency.

[@memegazer]

Object permanence is what I am facinated about

[@iambinarymind]

Much thanks, Yannic

[@Dron008]

Great! Very informative with a good sense of humor.

[@KevinAshcraft]

Fantastic! Thank you for the updates

[@theneuralbit]

Was waiting for this!!

[@AiveanZ]

Stable Cascade is not mentioned?

[@lexer_]

Around 49:40 the background music is THE music habie147 also uses in every one of his videos. Very strage moment of overlap.

[@DucNguyen-qx1pq]

Fantasic, thank you Yannic

[@zandrrlife]

Why you're my favorite researcher bro 😂. Since you're in the mud, you're aware of the reality, thus no openai D riding. There are tons of labs innovating. Tbh we have something as good as Sora and LM based. Just a lack of compute. I'm sure lots of labs are an investment away from becoming meaningful players in the space. Openai really just repurpose what others create in my personal opinion. Respect the minds other there though. I really don't believe openai will be top in 5 years. It will be a collection of companies. Not mention. Now I'm sure we can generatively create model weights. Rapid prototyping. Things will quickly accelerate for everyone and compute will be less of a pure advantage.

[@miedzinshs]

Ability to generate video is directly relevant for AGI. SORA appears to be able to generate a coherent continuation form a single image. Now imagine putting the model in a robot body and giving it live video input. Now you have a robot that can make reasonable predictions on what will happen next in its environment - including physics, interactions between pets, people and objects. This is going to be dramatically useful for robotics and a way bigger deal than producing bland stock footage videos.

[@СоюзниксОкинавы]

No, SORA in the robot would not "make reasonable predictions" about the environment, it would just generate video output, which have nothing to do with robot's actuators. And yeah, physics is still somewhat problematic for GenAIs, even to SORA.

[@miedzinshs]

I mean OBVIOUSLY it will take much work. SORA is not even released. The point is that models that can create a coherent video continuation from a single image will surely be able to do that also from a video input. And that is a critical conceptual breakthrough for creating agents that can make predictions in broad range of environments. Watch this happen in next 16 months.

[@elawchess]

@@СоюзниксОкинавы Well nothing that your interlocutor said implied that SORA was the finished product with very accurate Physics. It's work in progress but the point is that in order for it to continue videos well is has to understand Physics to some level and that can contribute towards AGI, even more so as the video predictions (and indirectly the Physics knowledge) improves.

[@СоюзниксОкинавы]

​@@elawchess It doesn't "have to understand", It is enough for it to just be a function that happened to map prompts to videos with some success. If gradient descent find out some set of weights that do exactly what it supposed to do - generate videos, doesn't mean that it would be any useful outside of video generating. Like look, image labeling, image segmentation and image generation models are in general have very different architecture, trained on a very different data and can't be replaced by each other. Progress in one of this fields often have no effects on other fields.

[@elawchess]

​@@СоюзниксОкинавы I am using that understand not in a human sense. I mean it would need to track. It's a similar debate to when people just say about the text GPT that "it's just predicting the next word so that means it's dumb and just doing statistical analysis. But the idea is that in order to actually predict the next word well such that you can't easily distinguish it's output from human output it's got to track what's actually going on pretty well and in that sense it does "understand". That's the sense I mean it. I think it's a bit useless to go into the Qualia of things and whether it's understanding is genuine understanding or the same as a human's and blahblah. E.g Look at the issue they are having with the corner cases e.g that treadmill video from open AI that went wrong and it has the person running on the wrong direction on the treadmill and the whole is jolting around in an un- natural way. That would mean the current model would be bad at predicting what might happen if say a cup is dropped on a treadmill. But say it's improved and when we ask for it to generate someone running on the treadmill it does it so that it's not distinguishable from a real videa and when we ask it to generate what happens when we drop a cup it does it well too, then you are moving towards AGI because it's output gets more and more indistinguishable from say a human that you put there to monitor things. It tracks better is all I mean by understand. So don't let that anthropomorphic word throw you off. Whether it's really really genuinely understanding like what we might ascribe to humans is really irrelevant frankly.

[@ekstrapolatoraproksymujacy412]

I think that "temporal smoothness" of Sora may be from vqvae in the same sense that vqvae causes smoothness in 2d space so it could be the same in temporal dimension if one token spans multiple frames

[@robbiero368]

Folding bikes are allowed on the London tube, non folding are not. Half folding ones are an edge case 😅 Electric unicycles are also banned, probably easier to detect those...

[@el-_-grando-_-_-scabandri]

41:53 Brian Barlic from the new atlas did a good vid analyzing this. as a slightly Skeptical man as you're shown to us here ... i highly recommend that vid to you to understand the nature of startups in the U.S vs China&Russia.

[@LysergicKids]

Research aside, every other week OAI comes out with a statement that can roughly be translated to: "Heavens! AI is oh so very dangerous.. if only a big strong, daddy government with bulging muscles could.. regulate my competition." *Bats eyelashes*

[@tiagotiagot]

Understanding the world to the point of recreating it does fit in the scope of AGI....

[@JosephCardwell]

this was awesome

[@teshbek]

The problem with model that make people wear clothes, is that there is lots of people that don't want their photo altered in any way

[@hasko_not_the_pirate]

26:40 obviously humans would be LLM augmented

[@Wobbothe3rd]

The argument that training a model on copyrighted data makes anything generated by the model derivative work is absurd, no way will that argument hold up in court. All of us learn from copyrighted data, unless a model actually is used to distribute copyrighted content it isn't violating the law. But the problem you point to is still valid - by distributing the training data that includes copyrighted content they are distributing copyrighted content. But even if a court were to rule against that the model itself would still be perfectly legal to distribute.

[@perer005]

If you listen to OpenAI, the main thing is the ability to allow models to train on video data. Sora is more of a nice bonus created in the process towards going towards AGI.

[@jaredgreen2363]

58:18 finally something with technically unlimited context…

[@Experternas]

the difference between folding bikes and non folding bikes is that folding bikes such as electric scooters are allowed on the public transport while the regular bikes are not. not strange.

[@pengchengwu447]

7 Trillion is like Japan & Germany's GDP combined.

[@mgostIH]

You know the course for structured LLM outputs is gonna be good when the dude teaching it is called JSON.

[@KolTregaskes]

1:18:30 All I can say is you can generate this response yourself.

[@AmeshinART]

Yay this is awesome :D

[@SurrogateActivities]

13:50 "emotional labor" is so milquetoast 😂

[@ericadar]

"sora has nothing to do with AGI"? isn't it possible the model has an internal world/physics model to be able to predict the video sequence? sure, it's not reasoning but still seems like an important step forward

[@JohnLewis-old]

This has everything to do with AGI. How could you say this has nothing to do with AGI?

[@makhalid1999]

ML News is back ❗️❗️❗️

[@videowatching9576]

Big fan of your content! While you technically have caveats that are accepting of say an AI open source company having a business model, I’m making an assumption that you haven’t studied business in-depth. Would recommend “Poor Charlie’s Almanac” about Charlie Munger - I think this would help build understanding for the challenges of a business to drive innovation and sustain a business!

[@tantzer6113]

Those robots seem destined for warehouses.

[@andrewm4894]

It's Monday!

[@winsomehax]

This has nothing to do with agi? Surely the abiliity to ensure consistency across time with those images of objects means a big move in the direction of agi and the real world

[@СоюзниксОкинавы]

No, it is not supposed to work like that. Like nowadays anything to do with statistics/computers now being labeled "iS a aGi PaThWaY, eXpErTs ArE wOrRiEd!!!11".

[@thenoblerot]

Stay hydrated!

[@Kram1032]

39:40 - I think the issue is that "the bad thing" isn't really about the presence or lack of clothes. The bad thing is a disregard for autonomy. And that *is* a form of disrespect and *can* be harassment, regardless of what direction the change happens in.

[@jameshughes3014]

wait... if a person doesn't show symptoms, are they really psychotic? That sounds a little fishy.

[@volkerengels5298]

"What A Time To Be Alive" - every 2min a new hurray-paper-study . *"What A Time To Be Alive"* -> the most cynical euphemism after 'Hallelujah'.

[@YEASTY_COMMIE]

I don't see how these smart glasses aren't a scam. There's a reason why VR headsets are thick and need stacks of advanced lenses. Try reading text 1cm away from your eyes, it's impossible. Putting a transparent OLED screen on transparent glasses is very easy. You could even do something similar with LCD screens. We've had the technology for a decade. Why has no one done it before ?

[@Hexanitrobenzene]

Yeah, I've noticed that, too. Way too thin. If Apple could make it this thin, they would have done that already...

[@fontenbleau]

some received copyright strikes for that Ai generated clips

[@Timotheeee1]

46:40 GPT-4 turbo API is still lazy for coding

[@jaredgreen2363]

They stopped publicly claiming to be on track to making agi a long time ago.

[@alexgonzo5508]

GOODY-2 Referring to advancements and new releases in AI as "a day in AI" downplays the significant research, development, and collaboration that goes into creating such technologies. It is important to recognize the hard work and dedication of individuals and teams behind these innovations, rather than trivializing their efforts in a single phrase.

[@BerntGranbacke]

Thank you!!! You really put your thumb in the eye of the woke things, really really beautifully.

[@memegazer]

Also disagree with your framing of the marketing The marketing isn't "we will let you prepare" it is more like we are testing the waters with the classic early adoption is a status game to gauge how much resources we might commit to make this level of our product a commercially viable thing through false scarcity. I mean remember when you had to be invited to even sign up to facebook...people ate that shit up (well most I got an invite and still don't have a meta account bc I was so replused by how effective those tactics where to the exclusion of superior efforts like myspace that was concentrated on feature rich free options for rapid growth)

[@bzqp2]

Is it just me or is that Gemini Advanced (the one you have to pay but it's free) much worse than the GPT4 in Bing? I can't get it to write even the simplest code for me, whereas Bing is just my personal programming assistant nowadays.

[@jaredgreen2363]

37:22 imagine forcing clothes onto people….

[@aneeshprasobhan]

may he they're already using AGI to develop their AI tools like SORA ?

[@mallow610]

He should have a bake sale for the $7T

[@peterkeyson8232]

"stay hypertuned" :P

[@shadfurman]

The problem with these weapons, and automated security, is the institutions in charge of them lack natural meritocracy, which means those in charge of the institutions will bias towards their own interests and away from the interest of the people. You have to have a naturally selecting meritocracy, that incentivizes alignment to the interests of those paying for it, or you end up with a military industrial complex that blows people up as a reason to buy weapons with other people's money. If you think about it, it really can't get much more disgusting. They could just give them the money and leave out blowing people up, but this is democracy and you have to convince people you're doing something, so they have to make up a reason to murder people, sell the public on it, use all their weapons, so they can buy more, so they get a relatively tiny kickback, a job in the private sector. Trillions of dollars and millions of lives wasted, for a fairly paltry paycheck. Sociopathic anyone? This is the problem with weapons, we all feel it in our gut, but it's not the weapons. People don't like the solution. They hear the solution and IMAGINE Walmart acting like government, and kinda freak out. But Walmart doesn't have the incentive to act like government, if they did, they would just be government, and people would write their mass murder of as just ... Just like government. Even if they acted like a mafia, they only reason they'd have to do that is if the government made their business illegal. But it's more ironic, cause they're freaked out about Walmart acting like government, and they think government is necessary to prevent Walmart from acting like government, but they're completely fine with government acting like government. If you just called government Walmart, it would be exactly like Walmart acting like government. "The beginning of wisdom is to call things by their true name."

[@XOPOIIIO]

Even primitive AIs with simple integer overflow mistake would force Gandi to nuke everyone.

[@lucamatteobarbieri2493]

《Who knows if OpenAI will still be relevant in five years[...] these things can go away before you can say "Open source"》😂

[@gulllars4620]

Take-away for today: AI Hurcalean script kiddies and DignifAI :)

[@andybaldman]

Lots of technology most of the world doesn’t need.

[@twobob]

Stability lost their way

[@thetechnocrack]

Anytime someone says the word "Emotional labor" i am out @13:44 kzbin.info/www/bejne/aIXPeo2hqb52nJY 😂

[@sean_vikoren]

To understand google, just go there and have lunch. You will find a sea smiling, happy, self congratualting, know-nothing, everyone gets a prize, idiots. After that, it's just; Oh yeah. Of course.

[@Zantorc]

Goody 2 shoes 🤣

[@TomM-p3o]

Sora needs 7 trillion dollars worth of chips to run at scale 😂😂😂

[@stefanobraghettocatoni1464]

Obviously you tried to make it looks like the models name are complicated. Plus there are plenty of products that are free for a couple of months. I think you personally have something against google.

[@adambarker3130]

When did Yannic catch this bad case of vocal fry? Either he's getting worse, or I'm getting more sensitive...

[@geobot9k]

He could be getting over a cold. Wdym by getting more sensitive?

[@float32]

@@geobot9kannoyance at things like vocal fry, chewing, etc is *entirely* psychological. There is literature showing that focusing on an annoyance makes you more sensitive to it.

[@scientist30]

How do you know it is just bunch of data? You know nothing about their model

[@myon9431]

When I heard everyone talking about Sora I thought they were talking about Kingdom Hearts at first

[@andrewdunbar828]

her-kyool-ay-nee-um

[@memegazer]

Disagree "this has nothing to do with AGI" This is a step towards teaching ML object permanence. So your scoffing it is trivial lacks proper imagination to my view.

[@lucilaci]

does yannic have vocal fry?

[@TJ-hs1qm]

if "emotional labour" is woke, what is "intellectual property" ?

[@andybaldman]

More stuff to make fake things that will take the place of real things, and have less meaning.

[@lizardrain]

lol buddy wish you'd make shorter videos. you can just split them up, it's not hard.

[@ChairmanHehe]

such a ridiculous bs reason for not releasing

[@Ori-lp2fm]

Drugs

[@TJ-hs1qm]

... go home 😂

[@ArianeQube]

"emotional labor' :)))) jesus christ snowflakes took it to a new level.