Can You Put All This In a Photo? 🤳

[@ginojansen2708]

The hand motion tracking algorithm is beyond stunning. You can see the skin stretching!

[@pepre7594]

@@DavidMulderOne however, 2 more papers down the line!

[@autodidact7127]

@@DavidMulderOne yeah but they WILL

[@jonasgrill1155]

Currently the best we have in terms of real-time hand tracking for VR without the use of something like hand tracking gloves, is the hand tracking that Oculus has developed for the Quest and Quest 2. And honestly, Quest hand tracking isn't _terrible,_ but it's not great either.

[@SpaytX]

Thank you Dr. Károly Zsolnai-Fehér, I'm writing my BA Thesis in Computer Graphics and your Contributions and Videos inspire me very much!

[@Kyss007]

I wonder when ai will be able to process that name. I mean most humans cant

[@neillunavat]

I am preparing for my Diploma in Computer Science as I am about to get out of high school 😁

[@theencore398]

You act so friendly with us, that we just forget to respect you as a doctor that your are. Lots of love and respect for you sir.

[@larion2336]

Why does that even matter.

[@LanceThumping]

Hopefully this will eventually lead us much closer to a daydream of mine: Take something like the Orbi Prime or strap a regular 360 camera to yourself in some convenient way and use that on family trips. Then when you take the footage home, you should be able to import it so that you get a full 3D VR experience of that you can dive back into. You could walk around the Grand Canyon (for example) again by yourself (from the map) or follow your family (in or out of body) to re-experience what happened again. I don't think there is a real hardware limitation right now preventing this, the only thing we are missing is the software that can put together the absolute massive amount of data a multi-hour 360 video contains into a proper 3D VR experience.

[@MrMarty77]

That would indeed be an incredibe application of VR technology!

[@thephoenix6651]

This would be amazing.

[@atishayjain8631]

Hey Two Minute Papers, Have been following ya from a long time and I am loving it! The papers are awesome. Looking forward to some great content from ya

[@Stellar-Cowboy]

What are papers in the first place

[@atishayjain8631]

@@Stellar-Cowboy The papers are what the videos are based on

[@martiddy]

@@Stellar-Cowboy is the academic studies made by scientists/researchers

[@dontaskme1625]

Just imagine getting hit in the face with a selfie stick someone flings around their head at a tourist attraction because of this technique. What a time to be alive ;)

[@Robert_McGarry_Poems]

Imagine swinging around with a selfie stick and then seemingly at random getting punched in the face.

[@Morningstar_37]

@@Robert_McGarry_Poems Or you swing your selfiestick, suddenly the camera gets off and just flies into the water because you flung too hard

[@Sashazur]

You can already get hit in the face with a selfie stick at a tourist attraction, but I guess this technique might make that risk even higher.

[@LincolnWorld]

What a time to get a concussion! Hahaha!

[@onesocialunited]

Tourists? Who can travel these days? What a time to be alive in dystopia- but we can look forward to all this VR and brain reality simulation.

[@hindigente]

I often get lost in streets across the world in Google street view. I can only imagine how awesome it would be to do that in 3D with a V.R. like that. Really interesting stuff.

[@beskamir5977]

@@goos6005 Pretty sure it'd have to move really quickly or the car would have to drive really slowly/stay still for it to work. I doubt this solution would work if the flailing traced a spiral.

[@brucebaker8844]

Street view is available in VR already with one of the best free VR applications In my opinion ‘Google Earth VR.’

[@antman7673]

Washing hands in a virtual world will take over real life washing. -Imagine the water savings!

[@kellykeefer5485]

Imagine the food poisoning!

[@markmuir7338]

I love the stereo views - I can just cross my eyes over slightly and see the actual 3D scene directly from my phone screen - poor man's VR! These scenes look amazing - the authors did excellent work.

[@theaninova]

I've been accumulating VR panoramas somewhat by accident for a few years now as the Google Camera defaults to VR panoramas, and only a few months ago have I been able to properly view them, and now I'm taking even more of them, even without parallax it captures the feeling of being there quite well.

[@AngryApple]

And now imagine removin the hassle of taking 30+ photos, standing still and sometimes bad image allignment and getting better results in somewhat 3D in just seconds. I hope someone will take the code and makes a drag and drop solution out of this work. Maybe insta360 should do it :D

[@AdrianMark]

Thank you Dr, this is an excellent exposition to the current state of VR

[@jackfrosterton2530]

This is just amazing! I thought it could be years and before we were here. They're beautiful

[@ItsaDigitalHamster]

Kudos to the authors for making their source code as well as demos available online. Let's go, open source!

[@6kwecky6]

Interesting! I imaging that if we put the camera on a mechanical stand to flail for us, that would reduce noise by a lot as well. In addition, if we later apply machine learning to remove/generate things that move by time like waves, people and such. We got an amazing technology.

[@nahCmeR]

It's funny, I still think back to when VR was in its infancy and practically everyone wrote it off as a gimmick that wouldn't ever be a thing. What a time to be alive.

[@zetahurley294]

I think it's misleading to keep calling these "360 photos" at least call them "360 3D photos"!! Its much harder to get the depth and movement and even just good looking stereoscopic vision than just a simple 360 photo!! You're underselling how impressive this is!

[@DanIel-fl1vc]

It has no use, how are you gonna turn this into a product? This is what frustrates me most about these AI technologies, they seem aimless. This can't compete with video of a tourist attraction. I think even being able to walk around in a virtual representation of the world is overrated, look at google maps. You can walk down almost every road in the world. This is at best another photo filter you can play around with for 5 minutes because you're bored out of your mind. Not impressive, pointless. AI programmers need to find out what tools would actually be useful to people.

[@zetahurley294]

​@@DanIel-fl1vc Google maps street view IS just 360 photos, this actually allows looking around at your own discretion and has much higher data density. You wanna get a better look at a specific sign? This is much easier than trying to pause the video at just the right time. "another photo filter"? If this becomes popular or used by google it could mean COMPLETE google maps street view and similar viewing of normally unmapped areas. You looking to move somewhere, or getting a new job? why not seamlessly traverse the whole street! This makes the ability to capture that kinda data MUCH more accessible. Also if you even READ THE TITLE you'd know this doesn't even use AI! This is actually programmed by people. Also fun fact about computer science (and most science in general) it builds on itself!!! Even useless applications can help develop technology that later gets built upon in very useful ways! A new technique for getting data image reformatted in a way to connect several different images in insightful ways? Ah yes there's NO way that ANY of the code used to make this could possibly be used to improve medical scans or make more human-friendly renderings of other useful information.

[@DanIel-fl1vc]

@@zetahurley294 It's a similar thing to AI technology that ages people, upscales video, composes music, reads back text to you. What people who work to produce quality art or entertainment do is. They cast older people, they shoot the video at a high resolution, they hire an actual composer of music or a narrator to read you back the text with feeling. This has no use to anyone who is in the business of producing something of worth. Maybe this could lead to something nice, but for that nice thing to be made they can't aimlessly try random stuff and hope something is marketable. They have to ASK the professionals what they need. Until they do that this is just another photo filter for bored people to play with.

[@zetahurley294]

@@DanIel-fl1vc except for the fact that computer science is built off of things that no one knew to even ask for, revolutions in fields don't come from asking what it needs, it comes from asking "what if we gave it this" asking what something needs will generally just create better versions of existing things, not new things that can revolutionize fields. Also this isn't a photo filter???? This is a program for consolidating 360 photos into a VR capable environment? Super useful for getting quick scans of locations you want in VR! The fact that they bothered to make it means they probably want something specific from it. Personally, I can imagine being out in the world and finding a room or a location I'd love to use as my workspace, but can't spend an hour mapping it out for whatever reason, so a 10 second strange act resulting in this would be perfect! If people want something, if there are use cases of it, it's not pointless. What programs that make the world better have you made recently? are you saying these people making this is bad because they did something closer to benefitting society in the way you think they should, but not actually doing it exactly? If you're thinking "well clearly they have the skills to write the programs, they're wasting their talent" I hate to break it to you, but computer programming isn't one field, it's many, and image processing doesn't correlate necessarily to whatever field you're thinking of, and if it does, congratulations!!! That means that this paper is useful because the code in it could be applied there as well!! If the ability to code in the fields is similar enough, then the code created for the fields is also similar enough.

[@DanIel-fl1vc]

@@zetahurley294 Well this is image manipulation technology. So I assume it's meant to entertain us? How does this compare to similar technologies that entertain people. Answer, about as entertaining as using a photo filter to change your hair color. What use is capturing real life and experiencing it virtually, of no use at all because we already got it. It's called real life. What they should focus on instead is how to allow the person to move around in the virtual world while remaining stationary in the real world. Or maybe how to render 3d worlds realistically by converting them to photo real images. That's maybe something the AI could do, recognize objects in the scene by what they're labeled as and convert them into something that appears real. I wonder if that's something any of the AI people even considered working on? That might actually lead to something, but instead they're swinging a camera around their head so they can look at reality from the comfort of their chair at home. What use is that when you got video or heck you could just take a walk. I think a lot of them are like the computer dorks in the 70s, they never imagined a mouse or a desktop computer for navigating through folders. It took someone like Steve Jobs to get them to work on that. They need to be told by people what is actually useful instead of aimlessly playing around.

[@kennedya7848]

in 20 years, this will be the dominant form games

[@lista_devideos_cursoscompl1338]

in 10 years o 6.5 year

[@DavidBerglund]

I think these techniques will see widespread use in all kinds of places but for gaming I think you're looking for a world that feels alive and needs lot's of simulations of physical phenoma. And realistic feeling characters. Though the progress on that isn't really slow either

[@TheNicoFX]

Always a pleasure to get news around here, thanks

[@brettcameratraveler]

If you filmed something handheld with with your phone in the way we all do > then decided you wanted to also visit the memory in VR (360 3D) you could take one of these Omniphotos 360 photos afterwards standing in the same position. A smart difference map could compare the two files and then automatically cut out and insert any moving elements from the first video file into the 360 still omniphoto. The end result would be a very quick and easy way of capturing a 360 6DoF video with audio. (Granted that the only things that would be moving would be those that were of interest from the first video. Ps. This would work even better if they simply added a 2nd lens into phones so we can capture 3D video)

[@willhart2188]

Hi. I made a video showcasing this in VR, using my Valve Index headset to record. The effect was OK for showing the 3D space, though it was kinda blurry. Also a lot of the examples were monitor only.

[@StephenRansom47]

Congrats to OmniPhotos... Location Scouts and Studio Film Work will really love this. With the new Big Screen Set Technology and this... You could film anywhere. Great stuff.

[@OrangeC7]

Once, I tried viewing Google Street View in VR, and because all the pictures were flat it was unfortunately very off-putting. Hopefully, if these kinds of techniques can eventually be implemented in Google's systems, we might one day be able to hop into Street View with a headset and feel that we really are right there!

[@pricklypear5717]

The good thing about it is that it does not require new equipment. Anyone with a camera can do it.

[@frosecold]

Two Minute Papers >6 mins, GREAT!

[@PMX]

The depth / parallax information is in indeed incredibly important when seen in a vr headset. I took tons of regular 360 photos with an Insta 360 One X on my last vacation (way back when travel was possible 😞 ) and while they look awesome when seen on a phone or desktop, in vr the lack of parallax makes the scale of everything look wrong. Proper vr 360 pictures contain not just a single 360 degree picture but multiple slightly different viewpoints so that each eye sees a slightly different point of view, allowing your brain to reconstruct proper depth and size information. I wish I had taken several 360 shots taking this into account, then I would have been able to use something like what is mentioned in this paper to build proper vr scenes.

[@ntwadumela_jadu9747]

Ready Player One? I wonder who will own the rights to virtual simulations of real-world landmarks. Could this even be copywritten? I can just see someone charging a virtual tourist fee. Virtual property broker will be a career? The world is moving so fast. Looking forward to the next video. Thank you for posting .

[@laky2k866]

Agreed. I could see tourism companies in 5-10 years starting VR tourist experiences for cheaper than real world tourism. Interesting world we'll live in not very soon.

[@LanceThumping]

At most I think someone could own the rights to a specific capture/experience but anyone could make their own.

[@tomthroffle]

This has been aready Happening in Videogames for the last 20 years, where Developers either License Buildings / Racetracks / Stadiums, or they make a 'Good Enough' approximation of it.

[@ntwadumela_jadu9747]

@@tomthroffle was unaware, thank you for pointing that out.

[@AceDeclan]

All of these questions sounded dumb af.

[@MrPinguinzz]

If this hand tracking algorithm works as well as demonstrated there it completely breaks the vr industry Cant wait for a vr headset with no controllers and a hand tracking like that

[@Andreadel96]

After watching your video I immediately tried this out on my VR headset. Considering that this can be done just with a consumer 360° camera I'm quite impressed. Stationary objects work really well, however, moving objects like people going through the scene don't look quite right (especially when not closing one eye). Overall, truly amazing paper, and I can't wait to see what the next ones bring!

[@PlanetXtreme]

This would be extremely useful for 3D animation software! If you could export your hand movements to an animation, that'd be so much less work for the animators.. speeding up films. I love it!

[@Stelassin]

Tourist locations are gonna look like a collection of helicopters with everyone spinning cameras over their heads

[@heinrichdertote149]

I love the *hand crafted* part...

[@lista_devideos_cursoscompl1338]

true , i need link

[@heinrichdertote149]

@@lista_devideos_cursoscompl1338 you are chrome. How do you not have link??

[@caffeinum]

Wow, this is just amazing! I have about 100+ photos collection over ~2 years travel in the Google Cardboard Camera app, which they recently deleted from the stores. I didn't find anything to replace it, so I just keep the app on my phone and continue taking 3D panoramas. But now, I hope I can finally move my collection elsewhere!

[@bob2859]

Warped video - brilliant! Here I was expecting a more traditional photogrammetry approach. Maybe we will see lightfield equipment in the consumer space after all...

[@Samurook]

This is far from a braindance, but it already makes me imagine investigations in VR.

[@beskamir5977]

I wonder if a longer selfie stick would further improve the quality? It'd have more data from a greater distance apart to work with so in theory it should but swinging anything at a tourist attraction is kind of a bad idea so making it even larger won't help in that regard.

[@AngryApple]

Seriously Insta360 should hire these people, this is a impressive USP for them. Of course it can be done with other 360 cams (what others even are there) but its based on the insta360 X

[@smileyp4535]

What a time to be alive indeed, I only hope that we can actually get to the point of being able to truely enjoy and experience all these amazing innovative designs and AI without being held back by the profit above all else motive that hamstrings real progress and democratization 😣

[@Solizeus]

I think the flailing could do some improvement, like, you could have a more stable device with a staff in a tripod and the flailing could be done by spinning a disk with a counter balance to the camera

[@kiachi470]

This is definitely amazing Paper...Super exciting to, And a another open door for the Industry to

[@evanrudefx]

NGL I thought it was photogrammetry it looks so good like its an actual scan when the camera changed.

[@gerry._.y]

i got motion sickness watching this

[@thesral96]

This is like Google Lighfield for everyone! What's the process of creating this? Can i do this?

[@marco_foco]

Too bad there wasn't footage of you playing NVIDIA FunHouse in the NVIDIA Zurich office! :D

[@pillantechvideo]

Wow, very nice result, it's seem like a lightfield by Debevec, but absolutely cheaper than another options. Thanks you for sharing us!

[@will2see]

0:21 - I really can't imagine how you would simulate zero gravity with VR.

[@MrJaggy123]

The thing I'm most hoping for is that we can get to a point of using techniques like this to make a "Google Jump" system available to everyone without needing tens of thousands of dollars (or more)

[@teddy7746]

As soon as VR sight seeing is a thing I'm getting my mom a headset, using a photoscan I have of my dad and making him into a character for her to go sight seeing with again since he passed away a few months ago. Maybe I can use GPT3 and make him interact with her or something 🤔

[@arodic]

Don't you get a ton of motion blur when spinning a camera like that?

[@AngryApple]

Yes an no, because its a 360 camera. The way it captures everything is "slow" moving. Dont ask me why, physics and stuff :D You still get motion blur from like moving people, but the camera itself doenst produce any motion blur

[@SWinxyTheCat]

I tried the VR demo, and it was pretty good I would say. I do wish we could tell the algorithm which parts of the image are flat and solid, and where they are.

[@jackfrosterton2530]

I just can't bring myself to believe that that is a real name.

[@larsfeed]

I love this channel and talking about the papers so I was wondering whether you could create an official discord channel so that people can discuss and talk about the papers in ordered chats?

[@thephoenix6651]

There is already an official discord server. Go to explore and type "two minute papers" and you'll find it.

[@timothywilliams8530]

what a clever idea!

[@dandiey]

What a time to be alive!

[@NiteSeek]

What a Time to be alive

[@TommyLikeTom]

flash forward to the future, tourists are flailing around the world, there is a flailing gold rush, with people racing to flail in famous landmarks around the world

[@willhart2188]

VR is always cool.

[@MarinusMakesStuff]

I am so looking for programs that help me make better 3D scans from video, I hope AI can do this better soon, and I hope this will be available to the public too. I really love your channel and it's great that some of the examples you share can actually already be used and sometimes I'm so jealous to see the access you sometimes have to certain work! It must be great to be in the middle of it all. Thanks for sharing.

[@thesral96]

0:12 Yeah but only if the connect their Facebook account so Facebook can analyze their environment, fitness and generally every movement in VR.

[@Kavriel]

Holy mother of papers !

[@trantung2013]

Could you give me the link to the part that you mentioned in the first half, a virtual hand creation technique?

[@Mudstersrocks]

What VR headset would you recommend for this if someone were to buy one today for use with Windows?

[@getsideways7257]

Speaking of "better flight simulations"... Real time Navier-Stokes solver based flight models when?

[@a51mj12]

The Big Question™ is CAN YO MAKE GAMEZZ WIT DIS???

[@Thats_Cool_Jack]

i have one of those cameras i want make these so bad

[@SamuelLoy]

what this looks like is just 360 photogammetry

[@alio2269]

Two minute papers uploaded two minutes ago :)

[@zombiekiller7101]

Same

[@TheGoodContent37]

So, this is how robots will be able to find us in the future...

[@Stellar-Cowboy]

Okay so I got a few questions, even after watching this channel for a year: What are papers? How can we use the technology that is presented in these vids (ex how can I use the image stabilisation presented in previous vids)?

[@zetahurley294]

Research papers, that's what he's discussing most of the time. Some of the technologies are publicly available and you can find github pages in the descriptions, however because this is active research not all the content shown is available for the public.

[@Stellar-Cowboy]

@@zetahurley294 oh ok thanks a lot

[@WhozBrokenHD]

would this technique work even better if the circle u spin the 360 camera in would be greater?

[@TheTwick]

When the first people go to Mars - we will go along?

[@pillantechvideo]

What about open vr movement?

[@RobertHildebrandt]

So this is how we will take our pictures once Light-Field-Displays are the norm.

[@haydensimpson02]

This was hand crafted??! 😧

[@malumy]

a lot of mobile devices have more than 1GB of memory

[@jackfrosterton2530]

Can anyone find the link to those web based VR photos? I downloaded a tiny zip with 2 or 3 pictures but having trouble running the windows based version. He shows a whole library of photos in the video.

[@fluffy-cat]

It eats 8 gigs of RAM, it doesn't even give any memory for SteamVR which causes it to crash, which makes this "app" impossible to use in VR with 8GB RAM.

[@designfu9444]

I cannot find the paper about hand motion tracking [smith et al.2020] Anyone knows the title of this paper? Many thanks

[@atease9087]

Could change this data to touch Data ( imagine going through the feeling and touching different shapes insert that I could be translates to a blind person) so people are blind could see it how does this idea 15 years

[@StephenRansom47]

...a continuation of that list @0:60 or so... VR control of robot arms for hostile environment work Unconscious tactile reception of VR user - Research. Assembly work with VR controlled hands at the Factory. Robot coaching with real-time VR trainer. Rescue work with VR control. Full body prosthesis. Research - will a non responsive subject use VR? ETC...... Fantastic!

[@OhNoRh1no]

Is the logical evolution to track your steps as you walk around taking an omni Photo? So now you can replicate walking around in a wider space in VR? :D

[@lorinenecro8555]

will this work for hdri's?

[@euden_yt]

That would be a great idea, now add eye light adaptation on VR, this can be done in games already. Just that a single photo is gonna be gigabytes in size :/

[@sammikinsderp]

Wonder if you can use multiple cameras to reduce the time required for "Flailing". Perhaps use several 360 cameras on a motor, attach it to a car, drive around, build a VR map of the world. Get it done Google!

[@thesral96]

They already got a similar technology. Check out Google Lightfields.

[@ranjaxwolf9725]

Free hand tracking is cool and all but it removes the ability to provide haptic feedback and feedback is at least 50% of the experience of touch interacting

[@Sl4yerkid]

by everyone you mean everyone with vr.

[@guy_th18]

It always hurts to see an idea you've had for years done before you...

[@Brazil-loves-you]

No AI?🥺🤧

[@semle346]

i was waiting for this one

[@ronnetgrazer362]

AI could have hand-crafted this method.

[@lista_devideos_cursoscompl1338]

Soon

[@nonamenoname1942]

Ehhh... First selfie-sticks and now this...

[@mullukkullum4929]

4 je Alcorythm ❤️

[@ryanscott5317]

train pilots and surgeons? maybe replace them too.

[@atomspalter2090]

jo ths is great

[@luisescobarleonardo8688]

Si

[@thatpacho]

It uses only one gig of ram? Dude windows 10 alone uses 4gb lol this algorithm is insane Edit: phones now adays that are $1000+ are starting to have 8gb of ram or more so even mobile it should be fine

[@chubbymoth5810]

Ehm,.. you may want to read some information about computers. 1 GB of Ram is pretty massive for a single object. WIn10 is 4 GB in memory running an insane amount of bloatware and snooping processes. A few hundred at least. This single object can be replicated a few times before filling up your RAM. Compare it with juggling package of sugar, 1 is easy, 4 starts getting pretty tiring and 100 would probably over exert you. Same rule applies to pushing data around for your computer. Now juggle the 100 KG package.

[@thatpacho]

@@chubbymoth5810 if it were vram I might agree with you, but this ain't vram. I think. But even considering consumer products nowadays 1gb vram isn't even a lot. Unless your poor.

[@thatpacho]

@@chubbymoth5810 01000110 01110101 01100011 01101011 00100000 01111001 01101111 01110101 00101110 00100000 00001101 00001010

[@steveman1982]

No AI required.... Booooo! ;)

[@LinChio]

it's no sense if you have a 360 photo....I don't understand....

[@euden_yt]

Unlike 360 photos and 360+3D photos (it exists too) This one lets you move your head around the space, search for Google Lightfields for an example. This paper describes how to take that type of VR photos without using the array of cameras that Lightfields used an instead only one 360 camera can be used.

[@LinChio]

@@euden_yt yes but so much data...can he take normal video or a series of 360° still photos and calculate that too? it seems to me a bit overcomplicated...but still interesting if some one can improve it with minor data I'm on it.

[@tomarty2103]

1:11 This is why it will be a long time before surgeons trust software like this.

[@zetahurley294]

Did... did you see the newer paper? Shown literally 2 seconds after

[@tomarty2103]

@@zetahurley294 The new method looks really promising, but new technologies need to prove reliability before being used for high risk applications. As a game developer / software engineer I think it has the potential to make VR more accessible, but it really needs haptics.

[@zetahurley294]

@@tomarty2103 I agree - but pointing to an old versions problems that have now been resolved isn't really the source of concern. Also though for training purposes you don't need the level of trust you need for actual performing surgeries, that's kinda the whole point of how it's being planned to be implemented - a low-risk training situation. My old highschool actually taught kids how to use the power tools for making sets for the plays in VR before they used the real tools (someone donated a headset or something) Better haptics will really make the difference though, on that I can agree, although the lack can also help train steady hands, as you gotta be keeping it steady all on your own!

[@handrez.]

📸🐵

[@AceDeclan]

35th!

[@almostcertainlynotapotato6528]

23 minutes ago!