Actually GOOD Open Source AI Video! (And More!)

Get caught up on the latest in AI with MattVidPro!
▼ Link(s) From Today’s Video:
Story Diffusion: / 1786312166964105609
Story Diffusion Demo: huggingface.co/spaces/YupengZ...
Story Diffusion Github: github.com/HVision-NKU/StoryD...
AI Town 1 click launcher: / 1784599385877176405
1 million context llama 3 8b: / 1785036209468907796
AlphaSignal OAI Search: / 1786025388902097077
Jimmy Apples AI Search: / 1785436609095328067
Github Copilot Workspace: / 1785324101734707664
Udio Updates: / 1785658654571602000
► MattVidPro Discord: / discord
► Follow Me on Twitter: / mattvidpro
-------------------------------------------------
▼ Extra Links of Interest:
✩ AI LINKS MASTER LIST: www.futurepedia.io/
✩ General AI Playlist: • General MattVidPro AI ...
✩ AI I use to edit videos: www.descript.com/?lmref=nA4fDg
✩ Instagram: mattvidpro
✩ Tiktok: tiktok.com/@mattvidpro
✩ Second Channel: / @matt_pie
-------------------------------------------------
Thanks for watching Matt Video Productions! I make all sorts of videos here on KZbin! Technology, Tutorials, and Reviews! Enjoy Your stay here, and subscribe!
All Suggestions, Thoughts And Comments Are Greatly Appreciated… Because I Actually Read Them.
-------------------------------------------------
► Business Contact: [e-mail]

[@balogunlikwid]

Openai waits, while open source catches up.

[@kolosso305]

To us it appears like "waiting", but for all we know we could be weeks away from a major release

[@__D10S__]

i mean open source is great and all, but this is incredible cope. Open source is only catching up to stuff that OpenAI has been done with for over a year

[@_SimpleSam]

@@__D10S__ Never bet against the degens. We are talking about an internet technology that is being absolutely mobbed. That much brain power on target and even just the tertiary advances will produce significant value. Unless you think there is some secret sauce unlikely leap trade secret - the haxors will win out. The ONLY way this doesn't occur is government intervention: which they are trying HARD for.

[@cesarsantos854]

​To catch up with a multibillion, MS backed set of the best in the field... In a year. Quite impressive.

[@harnageaa]

Dumbest move I ever saw. Not releasing models cuz they don't want to shock the world. e.e Intensifying AI's arms race is the best thing that happened to humanity since internet. These dudes are just stalling for no real reason smh.

[@MojaveHigh]

The reference photo for the spaceship comic is of Yann LeCun, who is not the developer of Story Diffusion; he is none other than one of the Godfathers of AI and is the head of AI at Meta.

[@vincentvoillot6365]

Just tried it, Story Diffusion work "fine". Plug and play with any SD 1.5 & SDXL models, imagine what the community will do with this ^^

[@Faded_Fragger]

how do i use it on a1111?

[@vincentvoillot6365]

@@Faded_Fragger No plugin ATM, but it use Hugging Face pipeline with a custom attention layer, so it shouldn't be long before arriving in A1111

[@KlimovArtem1]

I love this channel! There is so much going on in the AI sphere. Even though I am a software engineer it’s really hard to be up to all the news, and this channel helps a lot. Thank you!

[@WesTheWizard]

If you have Windows then you also can have Linux! You can set it up in a virtual machine, but I would just install it as a second OS on your machine and set up dual-boot. That way you get native access to all of your hardware and can play with all of these cool new AI projects with full performance!

[@CrystalBreakfast]

Yeah that bit confused me, I was like "uh but EVERYONE with a PC has a Linux PC if they want one." Then again, a lot of young people like Matt aren't actually as tech-savvy as you would think, due to the dumbing down of technology in the name of being "user friendly." As a result many of younger adults can't figure out software or troubleshoot when things go wrong. :/

[@peaolo]

Maybe you're right, actually I think younger adults could be tech-savvy but in a different way and on different topic than us (I'm 43). Of course struggling with rudimental technology and limited internet access hardened our skin for daily PC use and in my case having 25+years of experience both in personal PCs and professional enterprise systems let me look at a lot of IT management tasks as "basic stuffs", but couriousity and perseverance are the keys and IMO they both belong to Matt or a bunch of other teenagers/young persons. Time is on their side ;-)

[@memesandfun1]

Definitely want that Udio live stream!

[@laughingoutloud3713]

lol just try it out yourself its free

[@shaunralston]

Excellent reporting, MVP🍋 . . . always on the cutting edge!

[@MattVidPro]

Thank you shaun!!!

[@Pepius_Julius_Magnus_Maximu...]

Awesome, inspiring video as always Matt, thank you for your research and putting all this together man. I absolutely agree on the window context size part, I work with AI developing apps and oh boy if we could have it solve a bigger problem in one go without having to break it down into smaller problems it would boost everything up.

[@markmuller7962]

Nice catch Matt! I'll definitely play with this conics/video tool, looks sweet af

[@yorth8154]

that udio song kinda gave me cyberpunk edgerunner vibes. Love it!

[@vi6ddarkking]

By the end of the Year? With six and a half months of improvements plus, SD3 Fine Tunes, plus ControlNet. Oh Yes. Most Definitely.

[@dcpuzzles2990]

The comic strip creation part is available as an install via Pinokio and that worked for me. The video part isn't included as yet.

[@isaacbegue]

I think the real solution for consistent characters is that, with the output, the very same AI gives you a file with the exact data that it needs to generate the exact same character, something like a lora maybe?, but for every single character it makes. Then be able to use that same output in the next inputs you need to generate.

[@denisgabriel4645]

This is very good because I want open source AI video generators with Sora-like quality. By the way, it's weird that you haven't made a video about Vidu.

[@SebSenseGreen]

14:19 Wow, that's pretty good! I'm impressed.

[@johnwilson7680]

I am confused why the Sora video of the lady with the sunglasses is in this. Does that mean the other videos are based off pre-existing videos? Is it like image to image but video to video?

[@jonathanberry1111]

No, they just sed it as a reference, but you could just use text. They just wanted to show how it can generate based on something else, like you could put in a home video or photo as reference.

[@googleSux]

Netflix Video creation on demand within 3 years.

[@runningwithSaul]

2 years

[@Arthur-jg4ji]

@@runningwithSaul 2 years and 1 month

[@DataJuggler]

@@Arthur-jg4ji Like the Price Is Right backstab '2 Years 1 Month And 1 day'. Actually, it is probably 9 hours after AI learns how to write its own code and ignores us.

[@LoFiChillandBeatsVibe]

NetflAIx 😎

[@emanuelec2704]

try with novels first. good luck.

[@SSLCLIPS-TV]

"Open AI" is the end of the beginning

[@DisturbedNeo]

Unless there's been a breakthrough recently that I'm unaware of, the memory requirements for LLM context windows scale quadratically. Basically, you need to store the way each token interacts with each other token for the attention mechanism in the transformer architecture, so you multiply the number of tokens by itself, then multiply by 2 (the number of bytes needed to store the info for each token pair) to get the amount of memory you need to allocate for the full context window. So a 1M context window would be: 1,048,576 x 1,048,576 x 2 bytes = 2,199,023,255,552 bytes of memory, which is roughly 2TB I barely have that much in my SSD, let alone as RAM, so I don't think we'll be having 1M context windows locally any time soon, but a PC with 64GB of RAM could handle the current GPT-4 context window size of 128K alongside the memory needed for the model itself and whatever application is actually running it, no problem.

[@cesarsantos854]

Very expensive but not impossible.

[@marinepower]

There have indeed been advancements, primarily around flash attention, which reduces memory costs for transformers during training by 2-3x and during inference by around 5-10x. It's possible that at higher context lengths the savings are even higher.

[@lewisgriffiths5901]

It is still super expensive but these days seeing vram never increases but you can now have 256gb ram if u can afford it.

[@peaolo]

Matt, if you want to try Linux you can even use a live distribution and boot it from an USB drive... If it includes proprietary GPU drivers (are you on Nvidia side, right?) there's a chance you could test the AI game even without installing anything (with the exception of some disk space on a windows partition to store the game files/folders needed). Distros coming with preinstalled proprietary drivers should be Ubuntu, Fedora, Mint, Manjaro, Garuda, ... (I use the latter two)

[@AmandaFessler]

Started on Udio a few weeks back. Got an interesting start for a Jpop song. But seeing the short context at the time, I decided to wait until it improved. And now here we are. Got it working in just a few tries. It's not perfect, but I like it.

[@Zefrem23]

I'm still only seeing 30s extensions, how do you get the 2 minutes?

[@AmandaFessler]

@@Zefrem23 The extensions are still only 30s long, but it can remember back as far as 2 minutes. :)

[@Peter.Wirdemo]

Cool. Btw Yann LeCun is kind of .. known.

[@MattVidPro]

Ive def heard the name!

[@Brax1982]

Aside from how odd it is that an AI influencer supposedly never heard of that man...if that AI video tool is Chinese, how could that be the creator?

[@hangry3102]

The Udio stream sounds like fun!

[@iresineherb7]

it was a month or two since I didn't hear about new ai is there anything new interesting? is sora out yet for public? what about sd3

[@LouisGedo]

*Udio is such impressive technology!!* 😲

[@googleSux]

I created a couple AMAZING Vivaldi style violin concertos. Udio is much more than just contemporary stuff.

[@Liminal-Mystic]

I find it is actually better at stuff like this.

[@Will-kt5jk]

The more specific you get on genre/style tags, the better it seems to get

[@TimeTalesTT]

Your channel were the first to introduce me to the world of stable diffution...

[@kolosso305]

Technically you can still run linux apps using WSL, but it may not run as efficiently

[@freke80]

Yes, came here to say this as well. Not too difficult to install

[@GaryJr530]

Thats funny, cuz matt wolfe ACTUALLY pronounces 'Ideogram' wrong, and YOU say it right! 😅

[@GaryJr530]

(eye-deogram)

[@HikingWithCooper]

By the time Sora is finally released, it will be obsolete. Yes, it's the best today. But it's not very good. I'm confounded as to why they released the generation used in this video. Her left and right legs swap several times. No point on the screen is consistent for the whole 5 seconds.

[@adarmawan1977]

mmm,the music sounds like synth pop with M2M sounding vocals.. cool !

[@agnesslovehealz]

Udio rocks and lots more control setting etc like I had a song with best beat was able to just start from that beat and repeat it so dope

[@vi6ddarkking]

You know. Last december I made a prediction that we would be able to turn your Silly Tavern AI chats into Comic Strips by the end of this year. It would appear I was being pessimistic.

[@joeyhandles]

BURN THE LUDDITE

[@MadAFDuck]

Yeah i love Udio and it would be great to enjoy it with you live.

[@JuanGabrielOyolaCardona]

Thanks for sharing 😀 this is amazing.

[@ScottzPlaylists]

About Github Copilot Workspace: 🤯🤯 Wow, now Github, owner Microsoft can get thousands of free Ideas to jump on (and outperform the little App creators. (if it's a good Idea) and an AI can evaluate the Income potential)) Great Idea MS ‼‼‼‼

[@4mb127]

Will quality suffer from a longer context tho?

[@Jack-ft6qm]

How to up this app

[@jsnavely76]

StoryDiffusion looks great. One of the things I was wanting to do with AI art from the early days of SD 1.5 was make comic books, but the inconsistency makes it difficult. Hopefully, this will work better than my previous attempts. I wonder how successful AI-based search engines will be given the extreme censorship imposed on most AI models. The top search terms tend to be NSFW, so I imagine lots of people trying to search for porn or guns or even simply searches for celebrity names and getting mad when the AI search engine blocks their searches and gives them a lecture. The AI music sounds good, but the vocals are still very robotic sounding. Then again, a lot of music today sounds just as bad with all the auto-tune crap they put on the vocal fix the voices of bad singers.

[@Ovyron]

It's gonna be their end when you search for something and they output a dog that barks at you "unsafe content detected"!

[@krunkle5136]

It just might be able to make videos like this one day, making its own prompts.

[@CaidicusProductions]

I can't wait to try this at home. I'd like to make music videos for my songs.

[@CaidicusProductions]

My biggest issue with currently available video generators is the 2 second limit.

[@HumanGamer]

You might be able to run the AI Town thing via WSL on windows

[@nemonomen3340]

I can trim the track in Udio now? _LET’s GOOO!!!_ That’s all I needed before making the switch.

[@hipjoeroflmto4764]

Suno nerfing the quality was all I needed lol

[@nemonomen3340]

@@hipjoeroflmto4764 They lowered the quality? When?

[@joeyhandles]

When these long context bots are able to replace sparknotes, then and only then, i will be impressed.

[@DataJuggler]

14:40 That sounds like any other bad song that's popular now days. Disney would love it.

[@ackerwoman16]

😂

[@MrJaggy123]

Converting a video into XML to edit it? Are you talking about generating a descriptive transcript and editing the transcript and generating a new, different video? That wouldn't really be editing...

[@YeeLeeHaw]

It's super easy to install Linux, Matt. Join the cool team.

[@DjGolucky]

getting fed up. all these next gen generators. BUT WERE ARE THEY. Review after review. I want to use them

[@dreamphoenix]

Wow, thank you.

[@mattfarmerai]

Cocktail Peanut is a legend.

[@ozerune]

You should be able to run the town AI through WSL. Just run wsl --install in windows terminal.

[@drlordbasil]

Once its live on windows I'll be making offices and building for AI agents to go to work for a company. :D That's my goal.

[@marwangs686]

Love open source

[@kolosso305]

pretty good for open source

[@pklemming]

I am surprised you don't run linux too. If you wanted to just play around, you can run it from usb, but it is not too bad to dual boot. You can always start with something easy like ubuntu, then move on from that when you are ready.

[@AprilEclipse2024]

BEST KZbinr ❤❤❤❤❤❤❤❤

[@maxieroo629]

For 1m tokens is a few-100 GB of ram. 260k is ~64 GB I think

[@TheBuildstar]

Live stream defo. Give us some notice though I was nearly an hour late the other day 🤦🏽‍♀️

[@danver8433]

I just wish there was a local model that was good for translation.

[@ronilevarez901]

? Im using Marian MT and so far it has done it's job flawlessly in English-Sspanish and back, so there are free, open source models that can run locally even on phones.

[@ryglitheegg]

Nice

[@TheoIftimie-gj4fv]

Imagine remaking Megamind 2 with this XD

[@jamilmalas]

I tried to install it locally also didn't work

[@wielandsmith]

I'm glad Udio keeps working on improvements because I still end up going back to Suno.

[@mickelodiansurname9578]

@MarrVidPro 2:40 That's Yann LeCun Matt, you know, Metas Chief scientist, the creator of GNN's, like global superstar data scientist guy! [rolls eyes]

[@markmuller7962]

Suno already had long start generations of 1 to 2 minutes, maybe that's why it sounded so much better than Udio

[@HUEHUEUHEPony]

when google said openai has no moat,they were not joking

[@LoFiChillandBeatsVibe]

Yeah, Matt, every time I get to about 19.6 seconds, it errors out. I'd say their demo videos are about on par with Vidu, but not quite as good as Sora.

[@markmuller7962]

We need real AI to be integrated into Dwarf Fortress, imagine what a colossal game would come out of it

[@skiroviepcvr7792]

1:19 Do not count on a Victory yet because Alejavi is coming behind with free to use Ai tutorials.😮🤐

[@lchantry]

🤯 Can you imagine Dwarf Fortress with AI? The world may burn...

[@jamesclow108]

"Actually GOOD Open Source AI Video!". I didn't think the code has been released yet, so it's currently closed source.

[@miguelsureda9762]

GIRLS. Did you hear this ? “OPEN A.I. MY SUGAR PAPI !”. A flashing mulatta PEACHY da Whuupi seems to know more. Heard on Spotify and You Tube music.

[@darkman237]

Tried the local install, over 10 GB of space and couldn't run it SIGH!

[@sneedtube]

At least WSL2 Ubuntu is mandatory in your position man, no shaming but it's true.

[@drgoldenpants]

Hey thats my song they used as example

[@MrDonCoyote]

I love the future!

[@ronilevarez901]

Wait until you actually live it. There's always darkness along the light.

[@kuromiLayfe]

bytedance shares lots of seemingly high quality results but the actual models tend to be lackluster or require tons of extra work .. never see anyone able to get the same results as their demo examples even after 1000 iterations

[@freakdeath1020]

AI powered RPGM? bet me on that!

[@hangry3102]

Getting Linux as a secondary to Windows isn't too hard of a task, it is free and the community has made it pretty easy. If you do want to try that simulated village.

[@studiophantomanimation]

Long context is more useful than reasoning

[@MaximilianFeichtinger]

I want a Matrix MMORPG where some characters are ai and some are real people - and you have to do quests with both and so on - and you never know who is ai and who isn't as the ai tries to learn to become more and more like the players around them :)

[@ronilevarez901]

Since the time in which The Matrix movie happens the AIs are already indistinguishable from humans, the story you propose could fit more during Humanity's fall, right after they lose the war against the machines and they're studying humans to create the first simulation. Or maybe even before the war started. Could be like the start of machines plot: learn to be like humans to infiltrate and gather intelligence before war begins. And about the gameplay, I imagine it could be like Among us: you receive intel about a crime and you have to investigate the suspects and find out who is the perpetrator. Sometimes it will be a fact that it was an AI and other times it will be defintitedly done by a human. And sometimes you switch places and you're the perpetrator. All in an open world.

[@DiceDecides]

Sora hype slowly fading lol

[@Cingku]

8:36 This is insane guys...I cannot get to 1 mil token because I got 32 GB RAM only and 12 GB VRAM (RTX 3060)...but I can set it to 100k context windows in Ollama modelfile and it works and local in my computer! Localll! Now I can summarize long text without paying for chatgpt. That is insane. It said I need 64G RAM for 256k context windows and I probably will add another 32 GB RAM later..it's worth it..but its speed like couple of tokens per second but does not matter it works and I want summary after all and it didn't take a lot of time to wait.

[@ronilevarez901]

Sure. I can't even run gpt2 without slowdown so...

[@lewisgriffiths5901]

If your using ddr5 based system and have big focus on going past 256k get a board which supports over 128gb ram

[@nickgirdwood3082]

Couldn't you run that game thing on a Linux VM? Windows has a Linux VM built it, I think. Something like that.

[@Brax1982]

That song sounds like Chvrches, if they had cheap anime lyrics and Lauren Mayberry didn't sing with her Scottish accent.

[@hamidmohamadzade1920]

OH MY GOD Can open ai cope with open projects :)

[@YeeLeeHaw]

Come on Matt, you out of everyone should know who Yann LeCun is.

You know that linux is free and runs almost on any pc right? :D

[@creed4788]

Ai isekai games coming soon

[@CallMeThyme]

huh?

[@henrischomacker6097]

This is a kind of a mass complaint to at least... * Then two Matts * TheAIGRID * Wes Roth * Matthew Berman ...and who else is out there publishing "tested" AI news. Seems that you all lie to us when you show us your "live tests" of the new AI services suited to multimedia publishers. Every video you show us shows how you test a new service, tell us how easy and fast it is to use it and even how cheap it is... But when it comes to your own videos, nothing of that seems to be true any more. You yourself are the exact target group of customers for whom this services are mad but you tested them all as excellent... ... but don't use them! I have a good friend here in germany who is a real tech nerd, but unfortunately does not speak good enough english to follow your videos. And I would love to show him this or that vidoe of yours... ...but I simply can't because after two sentenses he will klick ict away and say "... who unserstands that crap? Show me that in german..." but I can't. What's the consequence of that? It seems that you all are simply lying. - The tools are not so easy to use as you show it, not so cheap, are they even good? Are you just paid for tellin us that? Obviously YOU as the main target customers for that tools should be very inrerested in expanding your target group to get a world wide audience and millions of subscribers. But you rather go without them? * What about really simple to do Text-Subtitles for KZbin? * What about alternative videos in the most common 10 languages? * What about alternative videos in the most common 10 languages with lipsync? I am really disapointed! You all showed us how simple it seems to be to "click" together an agent workflow to achieve even the last point. While your viewers really try to program all that for personal local services themselves with ollama and 4090s you do not seem to be able to use even those API tools to automate your workflows so your audience has a benefit. I am really disapointed! so disapointed! ...oh now, how am I disapointed! Take a break, make it happen, Point 3! What about the next live session? - Finally make it real!!!!!!!!!!!!!! I still believe in you all, but now you all also have to deliver... ...finally! - That's what you owe your audience! Sincerely yours, Henri Schomäcker

[@scrutch666]

in 10 years... "its bot bad. its not bad " i cant hear that anymore. It sucks

[@dwainmorris7854]

Everytime you guys talk about consistency you always refer to the characters face . Thats no big deal all you have to do is create a LORA for that . The problem is consistant clothing in different scenes and camera angles .Thats the real pain

[@nemonomen3340]

LORAs are used to help create a specific trained feature (could be a specific face, or a shirt, or something by else). My understanding is that they are used for images rather than video and even if you could apply them to videos, they wouldn’t do anything to solve the temporal consistency issue. You could get a video full of frames that each look like John Cena, but watch it as a whole and his face is shifting and flickering around like a demon’s possessed him.

[@dwainmorris7854]

@@nemonomen3340 True but still it is esay to get a consistent face but we need cossistant clothing to make comics .also we need AI that will not censor fight scenes and figures holding weapons

[@ronilevarez901]

@@dwainmorris7854 1) creating a Lora is already a big deal. 2) want an uncensored AI? Train it yourself. It is possible. Kinda.

[@dwainmorris7854]

@@ronilevarez901 No Lora is old news, been there done that . I have used Lora over and over again from every app on the market and none of them will let you do consistant comics. LORA is good if you want to put your favorite celebrity in a swimsuit. but if you want to put that same person in the same clothing in different scenes you can forget it .

[@ronilevarez901]

@@dwainmorris7854 Wait, wasn't Ip adapter used for that exactly? Sorry, I'm a little uninformed because I haven't being able to use SD since Colab banned Automatic.

[@harlycorner]

Wtf? You don't know who Yann LeCun is? Wow... Awkward.

[@KrafterMC]

ok

[@jonmichaelgalindo]

They have NOT published the video code. They haven't published the video weights. They don't have a working demo. The whole video thing could be 100% fake. I'll believe it if / when I see it. Like Ella SDXL just never dropped. Testing the comics thing now if I can. Edit: Nevermind this is rubbish. The "consistent characters" in the comic strips are literally just IPAdapter. There is no special sauce here at all. IF AND ONLY IF they publish the video code, I'll look into this. Guess what? They won't.

[@dikanev]

Yann LeCun developer haha. MattVidPro news channel about AI. facepalm

[@havocthehobbit]

This dude doesnt know who Yann LeCun is as an AI enthusiast , is like a soccer fan not knowing who any of the Renaldo's are , might want change the channel name to MatVidAmeture AI commentator or pay more attention beyond just the popular apps that come out . Sorry Im critical I know but I think a little more respect should be payed to a passion and some of the biggest living innovators who have put in decades of their lives to give you a platform of topics, especially when they are featured in your own vid , wouldnt even take 4 minutes seconds to research a piece of each topic youre talking about instead of referring to it as , I think he's a programmer .

[@driftedsun]

cringe

[@rodentrider2]

4

[@tombradford7035]

It is _dreadful_ - keep getting errors. It'll fall at the wayside when Sora lands.

[@rock808]

Matt! Please don't be a jerk and call a piece of software something different than the authors / creators endorse. This saddens me to hear you're going to keep going against their name just because YOU want to. :(