NVIDIA’s Neuralangelo AI: Gaming Anywhere on Earth!

[@TheAkdzyn]

This is very exciting. The two technologies have finally been combined. NeRFs have always been exciting to me because of how accurately they model the world around them but being able to create high resolution 3d assets from them is a massive improvement. The magic system is going to improve random character generation in video games for npcs etc. This is amazing! Thank you so much!

[@shreyass5756]

2023 is definitely a year to be remembered in the future. Not necessarily as a year where everything started but a time to be remembered never the less

[@seanmurphy6481]

This is pretty cool. Is there anything similar to this that lets you do text-to-motion and have a character model perform a movement that you can control by typing different actions into a prompt?

[@LarryPanozzo]

Wow, really good surface reconstruction. 👏🏼 And video-to-motion is so cool

[@IcyLucario]

"A world where anything-to-anything will be possible." That's the dream! What a time to be alive! :D

[@PandaJerk007]

And it all becomes meaningless. What good is a impressive game or picture when there will be millions of equally good ones made everyday. Nothing will stand out.

[@apophenic_]

​@PandaJerk007 value lies in the individual. not supply or demand.

[@PandaJerk007]

@@apophenic_ that is a good thought, I hope you are. I just know in our modern world, heavily-atheist, our minds & lifestyles have been strategically sabotaged so that we just want to consume meaningless products endlessly. But I agree we should think more deeply and consider where true value comes from.

[@DaSmerte1337]

@@PandaJerk007 Do you think being heavily-atheist is a bad thing for us humans?

[@PandaJerk007]

@@DaSmerte1337 I do think there are some negative side effects of atheism that people don't really acknowledge. for example in a city running out of resources or without the rule of law, greed becomes logical and stealing or violence can become logical. --- Christianity & other faiths have a core message that you should love others and sacrifice your own comfort for the benefit of others.

[@zerorusher]

Everything to Everything seems precisely what I can do when I close my eyes and imagine stuff. Imagine where we are gonna be in 2025!

[@noneofyourbusiness8625]

Hold me not the papers, I'm scared

[@fynnjackson2298]

Thats like a blink of an eye away, exciting times. Startreck holodeck nano simulation in 5 years perhaps. What a time to be alive!

[@MementoTurtle]

Imagine in a decade, what about what the little ones are growing up with

[@Fermion.]

​@@MementoTurtle In 2050: Granddad, tell us again, more stories of the ancient times before AI.

[@syrus3k]

@@noneofyourbusiness8625 Indeed. The rate of things improving seems exponential now.. We're really not far off these models being able to just improve themselves - then what?? Basically you'll be able to ask for what you want and it'll be able to find a way to do it. I'm most excited about the non-software answers to long standing problems.. I'm fairly sure AI will have the power to save humanity.. but also destroy it... more likely save it though. I'm always the optimist.

[@avdain]

Did anyone notice the text at the bottom left that looks like a "Shutterstock" watermark? Time: 2:22 😉

[@哈比人-y1g]

Ummm, I've been using Neuralangelo the first day they released and unfortunately the input video require very high precision landmark in every frame it extract, therefore what you mentioned in the video isn't achievable with the latest model of Neuralangelo and in most cases the result is worse than traditional photogrammetry. It's good to always imagine two more papers ahead but currently it's more like ten papers in the middle.

[@spaceghostcqc2137]

As a photogrammetry enthusiast I feel like we will need more sophisticated data collection tools to really scan the world efficiently. That said I don't see any reason why AI can't eventually be used to identify and replace objects in scans with known matches (basically comparing against a database of models encompassing) which would show up with collisions, rigging, shaders, etc. There are so many things that have many copies (as they were mass produced) and those could be modeled to precision to improve yields. As an artist I'd like a lidar/photogrammetry/lightfield wand/HMD combo tool that would allow me to do quick and reliable field scans but that's a ways off.

[@哈比人-y1g]

@@mnomadvfx That's really not the point, they use COLMAP to detect landmarks which is basically the same technique used by most photogrammetry software. If you're interested you can check out this paper COLMAP: Structure-from-Motion Revisited(2016) and you'll know why I said there's 10 papers in the middle.

[@Occulator2]

#2:22 These Shutterstock watermarks are hilarious - makes the training set pretty apparent

[@shadowskullG]

Is like 3D Gaussian splattering? I didn't hear what technique the paper uses

[@哈比人-y1g]

Neuralangelo generated meshes that can be edited and used in games, cinematics, and everything related. While Gaussian splattering generate volumes which currently unable to edit which basically useless in current VFX or game pipeline.

[@pfizerpricehike9747]

@@哈比人-y1gfree Open source software exists to turn nerfs into mesh, nvidia literally just stole code from more than one source and people without further knowledge will praise them for doing so And yes this is basically 3d Gaussian nerfs then put through free open source converter, and literally another two free open source projects controlnet specifically the openpose preprocessor for it for the motion tracking in humans Look into any of them and you‘ll see this is in no way magic but just an accumulation of stolen free Open source code, put together in one packet to be sold to those stupid enough to not do any research in the field themselves but still be interested in it. It’s a cash grab for the average joe that has no idea who the original creators where that have been robbed of their products by some of the biggest tech giants in the world. Adobe firefly is also almost completely based solely on controlnet, a students sparetime project, which then got stolen by tencent, Microsoft, adobe. And Qualcomm is building Chips for embeddeded systems that do hw Accel for Controlnet If the guy made his code proprietary he’d be one of the richest humans in the blink of an eye, given how much adobe alone makes with firefly which is almost a carbon copy with a slightly nicer interface and atm still inferior to CN.

[@alhassanbah]

What a time to be alive ❤❤❤

[@DerdOn0ner]

This is going to be amazing for archeological documentation

[@PandaJerk007]

That is true. One of the few good outcomes from this technology.

[@Th3_Native]

This is insane and truly the first steps into true virtual worlds like ready player one.

[@XtremBizZzar]

This, is rly, cool, thanks, for, the video. Make, more, of, these, pls!

[@brednbudder]

priceless youtube channel; never stop. these breakdowns are amazing for my layman brain.

[@Lell19862010]

I can't wait to be able to play SWAT in augmented reality in my own house, automatically reconstructed by this

[@fynnjackson2298]

Ad in AI video upscaling on top of this to refine the details and 'imagine' a higher definition reality as a third layer.

[@哈比人-y1g]

Not currently possible, neuralangelo still highly rely on landmarks just like traditional photogrammetry and upscale will mess it up.

[@tuseroni6085]

only issue i see is that all the parts are stuck together, in that scene, correct me if i'm wrong, i don't think you could pick up one of those tools. you could likely make it so that you can but it means removing the tools first, then generating the environment, then modeling the tools and placing them in the environment. still faster than making the models by hand...unless you want the models to also FUNCTION, like turning a screw or what have you.

[@manoskor1979]

Great video! May I ask why you removed the intro?

[@Rearendoftrain]

That’s incredible, instead of simulating physical attributes, we just ask the AI to give it its best guess after being trained in real simulations. This is exactly what artists do when they attempt to recreate the effect.

[@RealmsOfThePossible]

What a time to live in a simulation!

[@rojnx9]

I wonder if you could use all the 360 street view photos you find on google maps combined together to form complete 3d models of cities.

[@googlebackup-ll8fr]

ai car training goes brrrrrrrrrrrrr driverless car racing dow n ew york with less than 0.00001% accident rate

[@williamlowry8809]

I guess you'd only a 3D model of the streets, but if you manage to combine that with traditional LIDAR and drone imagery you could get something nice

[@MeNoOther]

Let's hope a neural brain interface is invented soon. Sword Art Online

[@TeamUnpro]

That means that in the next update, those scissors along with other objects will be automatically separated by the AI and made into individual objects

[@alansmithee419]

This isn't really related but for some reason this got me thinking that a werewolf-like social deduction game where instead of two werewolves two of the players are AIs powered by LLMs and everyone has to discuss who they think the AIs are, while the AIs try to get everyone else kicked out. Would probably be pretty easy currently but it could be fun to see how they gradually get better over the coming years.

[@himan12345678]

They have already had LLMs playing a werewolf game. There is a research paper on it. Tho to my knowledge there has not been that combined with Turing test, which is more what you are dreaming of.

[@DJ-Illuminate]

Any chance on talking about 4D G-Splatting and when or is anyone working on creating moving meshes from the splat data?

[@irwansutanto5202]

Bro Voice is more AI than AI itself😁

[@DanFrederiksen]

It needs to recognize shapes, not just blob a matrix over it. Recognize sharp lines as sharp lines, not seaweed. They had one job :)

[@cl8lf1wm8b]

we will get there. look at image creation by ai 10 years ago and now

[@movieidiots5542]

What a time to be alive...

[@inesis]

Gaming Anywhere on Earth!* *Only available for NVIDIA RTX8090 Ti users. Get your card now for only US$ 7999 !!

[@daslolo]

"we can place ourselves in virtual worlds" - getting deeper in illusion

[@BurstNibbler]

Awesome, thanks man!

[@elroneantefavorito8314]

amazing! love your vids!

[@krasome6440]

photorealistic gaming seems closer to be available for all in many more games, not now but somewhere in the future

[@ratatouillepg3135]

WHAT A TIME TO BE ALIVE!

[@houghwhite411]

Never thought Lidar would have competitor this early

[@Jacobk-g7r]

Grounded is a game that is like honey i shrunk the kids, using this we can make maps and use ai to populate the map with shaders and interactables and all that.

[@stephenrodwell]

Thanks! Great content. 🙏🏼

[@Jacobk-g7r]

3:07 Mocap for everyone and anywhere.

[@Beck-Gaël-CS-Ph.D]

Thank you a lot for the beautiful piece of informations you resume elegantly. And yeah this paper applications are just mind blowing !!!!

[@crowlsyong]

1:10 woah! hahah you make me smile.

[@gameofpj3286]

Can't wait to put an AI me into a virtual version of my dorm room :D

[@mateegyed1139]

Could this benefit from the new paper describing DynIBaR?

[@moahammad1mohammad]

Yay! We finally get a photogammetry method that's better than NeRF

[@SaumonDuLundi]

What a time to be alive!

[@jimj2683]

This will enable Google Earth to become much more detailed. Maybe even enough to use it in a global GTA type game. GTA 7 Earth?

[@jimj2683]

@@mnomadvfx I agree. AI is the only way to do it. Doing it manually with people would be impossible on a global scale.

[@alexandrepv]

Hi! What is the paper you mentioned for text-to-motion?

[@Herman_HMS]

what was used to create those motion to video videos?

[@James-wd9ib]

3:09 his hands go in his pockets every time he swings his arms

[@kazkas000]

His speech is like a broken npc's

[@Thedeepseanomad]

Whooww! What a time to be....

[@theocachet6496]

Can you give the reference to the Hassan et al 2023 paper please?

[@nanopi]

Is it possible to do a Bad Apple silhouette to animation reconstruction with this?

[@哈比人-y1g]

Nope

[@andropar3353]

Is this already like Gaussian splatting or is it already outdated?

[@Qubot]

Basically AnmateDiff + Controlnet

[@stoatystoat174]

soon we can have ourselves replacing Jackie Chan in Police Story video etc

[@noneofyourbusiness8625]

Hold me not the papers, im scared but excited.

[@jdmeesey]

My dream magic system is almost complete...

[@stevemata8350]

I love it. Thanks 😊

[@TheUnknownD]

I wonder If an AI is talking to us

[@AntonioAponte00]

even if you get to do a text to 3D at the same lvl as midjourney, there's still the problem of no human cognition involved in the result and also they have to solve functionality of the 3D model, a one piece sccisor needing ai after to make it usable in a 3D world is yet to come, and even is it does early, we can fathom the computational requirements to process layers and layers of this for a complete simulation. Getting this into gaming is far from us, but I do see a near future where getting faster concept to accurate scan data possible, that almost accounts for a 1/3 of the time to get a 3d model ready for production.

[@halfexcat1822]

he speaks like the high elves in Oblivion

[@filipewnunes]

OK, this and some GPS control points and my architectural surveys could reach a new high on delivery speed and quality. Can we do it?

[@trollenz]

No way ! Neuralangelo code published as well 🤯

[@哈比人-y1g]

Try it and you'll know it's limit

[@trollenz]

@@哈比人-y1g what do you mean ? It's not as good as it looks?

[@romannavratilid]

When is this comming to profesional programs end users can actually use (like Metashape etc.)...? anybody any idea?

[@margoyle6742]

... and experience all this, while sitting in bad posture, staring at a colorful plastic canvas called 'screen', in a closed room, that distracts us from all the damage we've done to the 'real world'. What a time to be alive... indeed.

[@noneofyourbusiness8625]

For every optimistic thought in my head about the near future a very pessimistic thought follows it.

[@MementoTurtle]

It's the lens you use to look at everything, I say it is what it is

[@Panzer_the_Merganser]

Yeah that was my takeaway too. It's beautiful outside today, think I'll go enjoy that instead of staring at screen.

[@hombacom]

It’s still not finished, but better than last year and perfect some papers down the line! Stop your time consuming learning and just stay tuned for more chocking updates

[@aeonbreak4728]

how can I use neurangelo with my phone?

[@johnsports_iii]

I'd love to see a model that can denoise the final surface using information about the types of objects in the scene.

[@himan12345678]

I think such a thing would be best by an AI model that understands physical creation. So like for the scissors it would understand manufacturing processes and how that would effect the final output 3d model. Or one for a rock would understand geological formation processes. Organic models would need understanding of organic growth.

[@lasyx9786]

Maybe you will be able to create a racing game that takes place in your city.

[@Greedygoblingames]

Incredible! I've said it before but it seems we're in the process of basically creating The Matrix. Once we combine these techniques with Neurolink that process will almost be complete, for better or worse!

[@stefanschuchardt5734]

Seems like it. It gets more plausible every day. Its like the concept that something like matrix is possible is already proven.

[@shreyanshdarshan3199]

Neuralangelo can't reconstruct the 3D surface from a single viewpoint. It needs multiple viewpoints of the same object to create the 3D reconstruction. So, I am a little confused about what Prof. Karoly said in the video about obtaining 3D reconstructions from AI generated videos, as generating multiple consistent views of an object from different viewpoints from generated videos should be a challenge. Can someone please explain?

[@哈比人-y1g]

No, I think he just saw two papers laying around and try to stick them together without really reading the papers.

[@steveosteen]

WATTBA ! 🤩

[@aresaurelian]

The stick figures looks exactly like the ones on Ghost Adventures tv-show. Perhaps we can reify the ghosts using this technique in real time.

[@swordofkings128]

You think Rufus will make 3d princess Jane with this?

[@m1dii]

Was the WANDB segment AI-generated?

[@RadialSeeker113]

Now, It's, not, about, understanding it, buuUUT..., it EEEZ, about, it being hard, to listen to. Aaand...., an AYY EIII, could, probably, juuust, read out ,his transcript, AAnd..... ; It would, sound, much more natural. Your videos would be much more digestable to most non native(eng is my second language) and native speakers if you just fed your transcript into a free ai voice generator. I love what you have to say. Its that that the cadence, emphasis on completely random syllables, constant, commas, when, speaking AAAND , over extending elrandom words makes it super hard to pay attention to and listen to. I suggest, doing some, reasearch, into, natural language models, for, English, aand, understanding, how, they synthesize speech. Once again I mean no harm or ill will. I just wish to see your channel grow and for people to be able to listen to it normally. I wrote the first paragraph of this comment as an example to show you how unnatural it sounds compared to a normal paragraph like this one

[@TKs3DPrints]

but when will the average user get to play with the nvidia and build are own photogrammetry worlds. i so want to use this program i have used Photogrammetry for years to create 3d models. and so looking forward to this.

[@哈比人-y1g]

Currently traditional photogrammetry is still doing better job than Neuralangelo with less landmarks required, but two more paper after, we don't know.

[@BadAverageGamer-BAG]

Cool stuff

[@Sekir80]

Motion to video is still needs two more papers, the temporal coherence isn't really there, yet.

[@哈比人-y1g]

What he mentioned in the video isn't possible with neuralangelo even if you use a real life video. Károly apparently jump way too far this time.

[@Sekir80]

@@哈比人-y1g Thanks for clearing this up! And you spell Károly correctly!

[@youdontneedmyrealname]

parabolic curve coming into view around the planet sir! THAT'S NO MOON!!!

[@eSKAone-]

Take all the photos of all the people and let the AI recreate all the major cities for a GTA World game 💟🌌☮️

[@ristopaasivirta9770]

Soon it will be "Two Papers Per Minute" 👀💦

[@containedhurricane]

It will increase the requirements and the standards of 3D modelling jobs

[@pandoraeeris7860]

So, can I use this right now? How do I get this on my laptop?

[@哈比人-y1g]

No, you can run Neuralangelo and text to motion two model individually, but Károly already jump ten papers ahead so it's a NO to what he mentioned in the video

[@djayjp]

Wonder how Neuralangelo compares to Gaussian Splatting...?

[@哈比人-y1g]

Different usage, Neuralangelo generate meshes while Gaussian Splatting generate volumatic clouds.

[@djayjp]

@@哈比人-y1g But they both produce real-time 3d rendered images from photographs/video. Which is better for that purpose?

[@JuanHerrera1]

The evil potential of this tech is crazy

[@Jagerbomber]

FMV Games happened decades ago 🙃

[@Soba64]

Are you using AI for your voice? It sounds different now.

[@piotrwojcik9112]

Exciting indeed. Deepfakes production at scale. What a time to be alive.

[@GL-GildedLining]

3:57 What's mind-boggling to me is that the AI-trained cloth simulations actually look more "real life" authentic to my eye than the human-engineered ones. Such a rush!

[@chyrek.ambient]

Are you lifting your weights and biases?

[@aziskgarion378]

This might even actually make miniaturists back in the job market. Let them make mini models of cities or places for games, and then 3D represent them back in a virtual environment.

[@PandaJerk007]

I like your thinking, but sadly I can guess AI will generate thousands or millions of miniatures of whatever you want, so the human artists will suffer because to the outside observer nothing will stand out.

[@ytubeanon]

we can put our face on thé lead actor in a movie and watch ourselves and friends or family as onscreen characters

[@jacobfoster]

Maybe in the future, digital environments and characters will be represented as an abstract collection of ideas rather than polygon geometry and textures. Characters would be rendered as and when certain levels of details are necessary.

[@DamianReloaded]

The impact of this in robotics is gonna be huge

[@Baraborn]

Very Cool.

[@pandoraeeris7860]

The proto-Matrix is here.

[@zrakonthekrakon494]

Within the next decade I predict with a good degree of certainty Gen Z and Gen Alpha will comfortably spend 50% or more of their waking hours inside 3 dimensional virtual worlds with AR or VR headsets either local or online

[@Legendarior]

Dude I know it's taking time to record audio, but the AI for your text to speech you are using is not smooth, it's not enjoyable to listen to the videos at higher speed. There are a bunch of useless pauses that sounds unnatural.

[@zandrrlife]

This is actually his real voice. Stop bitching bro.

[@HarveyShankar]

Dude it's how he's talked forever

[@USBEN.]

Nvidia not releasing these things even as demos is kinda disappointing.

[@pedxing]

Full neural rendering by 2030? 2040?