What Is 3D Photography? 🎑

[@code4chaosmobile]

One step closer to the Harry Potter grade stuff

[@aakarshan4644]

i was about to say that lol

[@luis96xd]

I thought the same xd

[@matthieuovp8654]

This company prints 3D cards like In Harry Potter : pops.co using similar tech I believe

[@CosmiaNebula]

Here's an idea. Feed it a photo of a something standing in the center, and continuously rotate around it. See what the algorithm would hallucinate for the backside! Because if it can imagine moving the view angle by 5 degrees, I could take the photo with a 5-degree rotation, and apply the algorithm again, and again, until I get a full rotation. And we get a kind of "exquisite cadaver" video. I tried the Weights and Biases colab notebook, but got stuck. If you can do it, please try. Some ideas: Holding a globe blog.blogthings.com/wp-content/uploads/2013/01/woman-holding-globe.jpg Random person standing in a field. blog.theretreat.org/hubfs/images/Man-Standing-in-Field.jpg?t=1473880856264 Bottle of cola (can't find a good picture) Hot air balloon cdn.britannica.com/84/158184-050-1D7ADEB5/balloon.jpg

[@NukeMarine]

Well, we have AI that animates photos already and AI that can simulate a person's voice from a script. Basically, this technique can transform text messages from friend into video using their photo avatar. Such a technique w/ VR can transform a discord chat room into a group of people that look around and talk each other.

[@Mark73]

1:23 There's no content there. It's all blurry behind the kid.

[@Spax_]

I know

[@hellnawnaw]

Yeah but it's good enough really

[@evertchin]

@@hellnawnaw it is not... really... dont fool yourself.

[@thethoeby]

@@evertchin yeah, but thats not an issue after all. Look at other papers - filling the missing parts is good enough to solve those problems. I guess it was done this way for another reason (maybe to save compute power since this paper is from facebook)

[@NEELAM_SINGH8066]

It was just a beta sample ...do you think filling that gap is a hardwork for A.I

[@tommeakin1732]

When it was scrolling through that feed with all the 3D pictures I felt like I was in an elevator with a window looking in on every floor ; - ;

[@thalescarl1589]

Or looking through the doors from the Monsters inc movie.

[@SirusStarTV]

Anxiety kicks in?

[@Klaster_1]

How is this different from previous similar results? Why no comparisons were made? This video feels off.

[@danielsilva9502]

I agree. I see a lot wrong with these 3D pictures. It looks exactly like an effect that already exists on FaceBook.

[@edwardecl]

Facebook app requires additional hardware like depth sensors, this does not. I bet this technique can be adapted so a depth sensor is not required for depth of field effects as well. Should make phones cheaper to make, and like said in this video should work on any photos not just the ones taken on newer phones.

[@ristopaasivirta9770]

I think the idea of this was to take the previous methods and make them into a phone app that runs super fast at the cost of some quality. I've seen better approximations before but they were made on a desktop in a specific software. This could be a nice real-world application and seems to be working nicely.

[@Johanns0r]

@@edwardecl This paper _is_ about the Facebook tech :) We upgraded the 3D photos algo a while ago, so it doesn't require a depth map as input anymore.

[@edwardecl]

@@Archimedes.5000 My bad then, I was going by an old post from a couple of years ago that said it required a depth sensor or some other requirements, maybe they changed that since then. I don't use Facebook so I have no idea. Either way if they open the software up to running on the phone itself you wont need Facebooks servers or whatever to do it for you.

[@shrammstorm]

holy mother of papers!

[@capsey_]

Holy sugar cane!

[@rowboat10]

Holy tree bark!

[@wallacesantos0]

I was holding on to my paper when he said that. Made me laugh so much! xD

[@BsktImp]

Be careful how you hold on to your mother [of papers]...

[@brexitgreens]

While at the same time not neglecting your glasses.

[@JoshFlorii]

hasn't this been possible for years? ive definitely seen this for years

[@finneganblack2038]

Are you talking about photogrammetry because for that you usually need to rotate around the object and get a lot of angles

[@lucyf6516]

I’ve definitely seen stuff like this on FB but I think they have to use at least a depth map for that.

[@xthesayuri5756]

To create pictures that have 3D effects? sure. To create them with a NN from a raw image as input? No.

[@kiraacorsac]

I swear there is older episode of two minute papers that presents the almost the same idea (just a different paper).

[@hetsmiecht1029]

@@kiraacorsac yeah, I think so.

[@vibhupande]

Should've contrasted it to the existing 3D photo techniques (that few phone manufacturers offer), and explained how it does something new/better from that understanding. Can't imagine any real use of this, just a fancy visual effect.

[@chrisbanach3425]

I don’t get it, this feature has been on Facebook for at least 2-3 years, no? So what do you mean by “I can’t wait till it will be available?”. Also the “filling” lacks for now, often ending up with just some blur.

[@Helvanic]

It is available if you take a 3D photo in the first place. With this tech you can do it with any 2D photo.

[@agiverreviga4592]

@@Helvanic Turning 2D photo into 3D (without a second camera or a sensor to estimate depth) has also existed for months, if not 1-2 years. The mobile app "Focos" does exactly that and it works quite good. It's on iOS, not sure about Android.

[@chrisbanach3425]

@@Helvanic nah, phones that can do real 3D stereoscopic photos don’t need AI to fill in the gaps, because the information is (mostly) already there. And you can tell the difference by how blurry it gets around the foreground character when you move around your phone, at least with the algorithms used by Facebook. Some apps like LucidPix do a much better job. Regardless, unless I’m mistaken this is nothing new.

[@martinmazanek5192]

I thought that there was already a similar video done on basically the same topic discussing even more advanced methods 🤔🤔🤔

[@clochard4074]

I love the content but I already knew each answer from the tone of your voice! You are not good at surprises when you are that happy, professor!

[@DigitalicaEG]

Is it just me or is anyone else already using this feature on Facebook for a while now?

[@AshT8524]

Yea, I thought these were available from quite sometime.

[@Johanns0r]

Yeah, this is the paper where we describe that tech. We should've probably made it more obvious in the paper, but then SIGGRAPH wants their submissions to be anonymous.

[@tobiaszstanford]

​@@Archimedes.5000 the article in the link you uploaded states that: "Facebook says the tool works best if you avoid images with narrow objects in the foreground, or with lots of reflections. It's also a good idea to pick images with objects at various depths for the best effect." "the app can't create new data that isn't already in the image, so it will extrapolate what the area behind the picture's subject will look like based on nearby pixels." Meaning that Facebook's algorithm is inferior to the one presented in this video, although as the article was posted on March 03 2020, it could of been significantly improved by now.

[@samyaks]

As the author of the paper said above (J Kopf), this paper was published by Facebook Research themselves and it's probably the same technique as the one used in the FB app. Source: github.com/facebookresearch/one_shot_3d_photography

[@brexitgreens]

@@Archimedes.5000 Also [this Android application][1] has been on Google Play for two years. [1]:play.google.com/store/apps/details?id=ru.roadar.dpth

[@graealex]

Not only have we seen the AI depth algorithm in several videos of yours, but in those the AI background filling was much smarter, at least regarding the sample at 0:42 where the background clearly has artifacts.

[@CadetGriffin]

Artifact where a dark portion of the surface on the horizon is missing: *exists* Károly Zsolnai-Fehér: Let's pretend it isn't there until the inpainting improves.

[@woddenhorse]

🔥🔥🔥 Where's the line : "What a time to be Alive!!"

[@danielrodrigues4903]

I think he saves that for the really amazing papers!

[@opendstudio7141]

His boss showed up and caught him playing with his phone; that is the reason he was whispering. 😉

[@wmpowell8]

What happens when we confuse the algorithm with, say, an image of a Penrose triangle?

[@lucyf6516]

Would love to see this

[@1992jamo]

I was curious enough to run it: imgur.com/a/LaVv51x

[@andrewlees3946]

@@1992jamo it really doesn't like that

[@elijahtommy777]

@@1992jamo hey how do you access the software to "3D-ify" your photos?

[@davidwuhrer6704]

@@1992jamo That looks pretty cool.

[@ThomasAndersonPhD]

4:27 That shot of the astronaut on the moon, with this effect, really gave me a wonderfully dizzying existential feeling. This tiny speck of a human being on this huge bland rock of a moon, no atmosphere above, just unfathomably vast cosmic dark.

[@macScsgo]

This could seriously change the entire way social media is displayed to us

[@rowboat10]

Imagine looking through a social media page in vr, and seeing all of these photos move as you move yourself

[@macScsgo]

@@rowboat10 Since Facebook owns Instagram, I could see this bringing back image sharing in a completely new way

[@brexitgreens]

@Bronze HD I would take it one step further and make it possible to enter the photographs, participate in the scenes and change the past.

[@Sashazur]

All it can do is make all your still photos tilt a little bit. It looks cool but it can’t substantially change the user experience - you can’t rotate the photos very far and the picture information behind the foreground objects is invented, not real.

[@EternalDensity]

"That content was not part of the original photo." You mean they grey blur?

[@pigboiii]

That timeline with 3D pictures is super trippy

[@Kyrelel]

Isn't this just the "living pictures" technique that has been round for a few years? Also, it is a bit silly to show us, twice, the technique working in around 1 second and then ask us to guess how long it takes :/

[@Max24871]

Well no, live pictures just include video from shortly before to shortly after the photo itself was taken

[@brag0001]

@@Max24871 yes, but the lighting and the position of the kid on the ground didn't change at all. Try to keep a kid this age standing still for a minute with a phone in his hand, and you know why we knew it was barely a second 😉

[@MattCruikshank]

I've seen applications where a camera did face tracking in order to provide a parallax rendered view. This is yet another piece of content that would look great through that viewer.

[@MORTYCJA]

this can have so much potential within creative photography o_o

[@RobertHildebrandt]

Ok, all we need is a lightfield display and combined with this paper we will never look back.

[@SpisUchmich]

Hm... I think the image inpainting at 0:56 looks rather week... you can clearly see that the horizon line gets blurred behind the kid..

[@Supreme_Lobster]

I agree

[@Hyperboid]

With a head tracker you could pull off a really cool effect where you could simply move around and it would look like a portal. No wii remote required.

[@drdesten]

Honestly, Depth map estimation was possible long ago. (Maybe not aaaaas good as now but still) And the impainting is pretty bad to (way worse than some other AI's that have been made) Maybe I'm not getting something here, but this doesnt seem too impressive to me...

[@cskhard]

I can't wait this gets popularized !

[@IronFreee]

Reminds me the old Westwood Blade Runner game when you had to analyse 3D pictures to find clues

[@simo7466]

thank you for posting these high quality videos, appreciate it!

[@applebanana6149]

I think the first time I saw this effect was with the James Cameron Titanic Imax film. They made old black/white photos pop out. Very cool that this now works on your phone!

[@bob_._.]

I'd call it maybe 2.3D. It's a digital equivalent of the multiplane cameras used by animators since the 1920's; just flat image layers moving relative to each other, but with the missing information filled in by blurring/smudging.

[@JorgetePanete]

It's cool , i hope the background redrawings get better

[@Monsterplod]

This would be good for 360 images too, even just for the parallax when looking around.

[@Vennotius]

Feels like an iPhone ad.

[@ArthurRTK]

When will we be able to download the app?

[@woutervanzon4620]

please reply to me too with the answer kind strangers

[@realfangplays]

Apparently, the paper is by Facebook and the effect has been available in Facebook for a while

[@SpirosKography]

So, that thing that Facebook already offers then?

[@atlas4733]

this would look fricking amazing on a 3ds

[@RacingTIR0]

imagine how different the world will be in 10 years, after a few iterations of the stuff you show on this channel 🤯

[@AngelLestat2]

2:00 Why so much suspense if you have review several times Depth Field masks made by AI apps before. About how to refill the missing data, that is something that photoshop does since 7 years ago.

[@rowboat10]

What if this is someone's first video on two minute papers?

[@averygoodfantasticname4206]

Bruh

[@brexitgreens]

Not only Photoshop but also libré Resynthesizer (for GIMP) and "Repair → Inpainting" in G'MIC (also for GIMP).

[@scienceshiritai5604]

This is a video that can actually be done in 2 minutes. I've been watching this channel for a year and have been enjoying it. I just can't help to notice that this channel's quality is going down, sadly :( These last videos are just footage released by research teams and re-edited with a bit of "wow"s added and a sponsor message. Did the researchers use a new type of DNN? How large was the dataset this was trained on? Etc. It looks like these videos are made without even taking a look at the paper.

[@henoktilaye3152]

Holy mother of papers!!! Loved that!

[@JoelGustafsson]

How is this different from what Facebook is doing right now?

[@helper_bot]

it IS facebook's, check out the description. and in fact this tech has been showcased by Two Minute Papers 5 months ago, but its a different paper i think, so i expect better results (the last one has some artifacting)

[@agusmigoyo]

Awesome man! Where can i try it?

[@WoLpH]

The amount of seconds it takes to process the image is only a small part of that proposition. Modern phones have multiple processors and most of those are constantly idling in a very low power mode so taking a full second could mean it will drain your battery in about 2 hours instead of 12 hours without this process. Without knowing further it's hard to say if it's using a single processor, all of them or a completely different process. After all, the modern iphones already have depth cameras so it is likely that some part could be off-loaded.

[@odw32]

My guess was actually a bit too optimistic at 100-200ms, but you know what... A few papers down the line, someone will probably have found an optimized depth mapping or in-painting algorithm, and chips also get more and more specialized features for these kinds of AI tasks. So I wouldn't be surprised to see near real-time post-processing of photos like this in a few years.

[@AlanAztec]

Holy mother of mothers of papers. Instagram coming soon.

[@timguo6858]

Wait I feel like 2minute papers covered this topic before and it was very similar, like look behind the photos. Is this just de javu?

[@mahmudulhasan443]

Holy mother of papers ❤️

[@jarredeagley1748]

Colorized 3D Saturn V I need this in my life

[@pyanek]

After 22%, it went super fast in real time, so it's pretty much instant, or is it that the software stays on 22% for a long time?

[@elijahizere]

"Holy mother of papers!" that's a new one

[@SnowTerebi]

The background blocked by the original front object is blurry tho...

[@satibel]

guess: 5-10 second edit: well, on my phone that'd be the case :p

[@recklessroges]

I guessed an average of 60 seconds per 4Mpx photo.

[@brag0001]

@@recklessroges I knew it was barely a second, because of the kid holding it in almost the same position without any change of the lighting nor of the kids position on the ground 😉

[@indexMemories]

2d2Depth algorithm was around for a while. But this is pretty impressive.

[@MikeKoss]

On Facebook, post any picture and choose the 3D effect. Same thing as this paper.

[@samyaks]

This paper is in fact from Facebook Research. The Facebook app uses the same technique. Source: github.com/facebookresearch/one_shot_3d_photography

[@AIFuturist]

I understand how difficult it can be to choose the interesting paper for a review when you already covered over 500 papers.

[@defdac]

The example @01:05 have a very obvious ghost/blur effect instead of a seamless AI generated background?

[@TracksWithDax]

For the average consumer, this seems more like just a cool little gimmick. I could definitely see this being very useful in multimedia production though

[@Apollyion]

You can actually already do in Adobe Photoshop and After Effects, but an automatic program that does this in 1 second surely does cut down the time quite a bit. And the AI for depth mask is neat, painting depth masks by hand tends to be bothersome on more complex photos!

[@JoshTenMusic]

Imagine this applied to every frame of a video

[@Otis151]

Is this already in the wild? I feel like I saw 3D photos like this on my Facebook feed from my friends with iPhones.

[@YitzharVered]

I'm waiting for the day that ai makes an entire world based off of a photo

[@edenatlas7440]

What a time to be Alive! And I mean it this time.

[@Szylepiel]

My Facebook App has this or similar option, it takes second or two, less than five for sure. Cool stuff.

[@luis96xd]

Wow, This is so amazing!

[@superafins]

Facebook has been doing this for a few years now.

[@hanseldsilva2393]

From what I can see, I wouldn't call the results, although having parallax depth, reality 3D but more like those pop-out greeting cards which seems kinda gimmicky to me. The novelty wears off before you realise. But like you always say, two more papers down the line, it'll be much better, and that's what I'm looking forward to. 😄

[@blasthardcheese4336]

This is insane!

[@Bob78]

More than insane really...

[@LetsTakeWalk]

Tbh, it’s more like 2.5D. You can’t rotate around the subject. 3D complete 360 degrees would be the next step. Now that would be a paper.

[@badhrihari1705]

Still quite good, don't you agree?

[@LetsTakeWalk]

@@badhrihari1705 Oh yes, it is still brilliant.

[@openroomxyz]

Is this not already implemented inside Facebook app for some time?

[@highvis_supply]

Has anyone had any success in getting any decent results from this? I tried running it with several of my own photos and literally only their example photos generate something coherent

[@clankill3r]

Can someone help me? I'm looking for an application that can create frames for an animation based on a start frame and an end frame. In the example it had a cat like animal that was rotating his head. I thought Two Minute Papers did a video about it but I can't find it. Hope someone knows cause my google skills are not good enough for this...

[@istiblasko1]

Does it require depth sensor in the camera?

[@traianima]

i've seen these kind of photos on facebook for about a year now

[@NilsWesthoff]

Check out Looking Glass 3D displays and then remember this paper. Very exciting! I expect 3D photography/videography is going to become it's own industry in the near future.

[@ElRak123]

Hi What is the diffrance to the App "Lucidpix" and why is that new? Even on facebook you could see that for a while now. Else i am a big fan. thanks for the videos

[@RoboticusMusic]

In the next paper temporal coherence to convert 2D video into 3D VR video in real time.

[@jackk3094]

What if you took a standard face mesh and some facial recognition. Could we have really detailed face mapping?

[@Ch50304]

Is there a way for someone to us it like in an app format.

[@bunshine]

did you not notice the blur behind the kid at the start or..?

[@MrVivec90]

Is it an opensource or commercial? Can we try the source code on python, or something?

[@joeygenna4801]

It’s really cool, and I think it has a lot of potential, but to me the person (or object of focus) still looks flat. Like a 2d cardboard cutout in a 3D environment. I don’t know if it’s just me, but I still think this is really awsome Edit: I remember not to long ago you covered an ai that could create a 3D model of a person or object. I think if that was implemented, it would make the final image that much better

[@bird_eagle]

Could they do this with an entire video?

[@AlexanderBourke]

Is it possible to demo/test this technique with my own photos?

[@ehsan_kia]

Coincidence that Google just announced today they're bringing this very feature to Photos soon?

[@andreasv9472]

12 seconds. I loose, making me happy and sad at the same time

[@Lugmillord]

I can sense a billion dollar startup company.

[@rdcdt6302]

Do you know if it s possible to make it work on video?

[@papalouis9111]

It's been on Facebook for quite a while now?

[@Backup439]

Yes..

[@tri-aayam]

How does it differs from 3d ken burns effect ?

[@BBorn223]

I really want to incorporate wandb for my project, but tensorboard seems to be able to gives the information i need Is there real difference between using tensorboard and wandb?

[@Here24]

I tried out the mentioned Colab and it is amazing. Only problem is it doesn't generate heatmaps. So I used some Github Deep3D rep. If you want to see the results, check out this video: kzbin.info/www/bejne/p6WcloRsd9x0eM0

[@imjody]

Outstanding! 😎

[@Hypotemused]

“Holy Mother of Papers” 🤣🤣

[@SirDragonClaw]

This is actually one of the least impressive things I have seen on this channel, we have had the ability to estimate depth, and do infill painting for many many years.

[@rysea9855]

Then whats the point of having multiple cameras in your phone?

[@NeilMohammed]

cant wait to see this on my instagram feed

[@AzogDefilerFromMordor]

Again I am lost in those links. I cant find that demo or how to use it. Either I am dumb or its really for people with some scientific background only.

[@Agrover112]

Isn't this live at Facebook ?

[@smhily]

Somebody, please, tell me how I can use this 😭

[@davidwuhrer6704]

Does it have to be in colour?