Consistent Video Depth Estimation

Our SIGGRAPH 2020 paper about reconstructing dense, geometrically consistent depth for all pixels in a monocular video. The input is just a hand-held captured cell phone video.
Paper and more results available at our project page:
roxanneluo.git...
Code:
github.com/fac...

[@jayperalta7104]

This instantly slapped the depression off my face. What a time to be alive.

[@x0ty]

this is game changing stuff. very impressive stuff guys, cant wait to play around with this

[@Johanns0r]

github.com/facebookresearch/consistent_depth

[@MattSeremet]

0:45 love the static pov shot here. I would love to play with this process, fingers crossed on the code coming soon. Cheers on the awesome results!

[@swancollective]

True, but how would you go about extracting a right eye view from the created depth map?

[@user-cw3nb8rc9e]

3:10 Here you could extract depth of the scene from perspective, by comparing how fast pixels and certain features move in the foreground and how slow in the background. I am surprised that river and everything came out so weak, almost invisible in the depth map. With the technique I described it should work very good to improve this.

[@NerdyRodent]

Nice work and MIT licence too. What a time to be alive!

[@gauravchaudhari4572]

I just saw the code.. This is so cool.. really helped me understand this implementation... I didn't know you also used Flownet, i got kinda excited that your project has created smooth video without its use, this is cool nonetheless

[@MichaelSavidgeStoryteller]

Wow, this would be great for converting 2D video to Stereoscopic 3D! Thank you so much for all the hardwork you and your team has put into this code!

[@cg.man_aka_kevin]

How to do this???

[@decibelfilm]

Can't wait to play around with this extremely promising code, could you possibly post an alert or announcement on KZbin when you make the code available? Great work, guys.

[@Johanns0r]

Thanks :) Not sure how to post an alert here, but it should be in about a week or so. I'll update the description, then!

[@decibelfilm]

@@Johanns0r thanks Johannes, will keep an eye on it.

[@bboynick3460]

@@Johanns0r Hello there. It's been 3 weeks since you uploaded the video. Im sure many people, including me myself, are eagerly waiting for the souce code to be released. Will it be available any time soon?

[@playboicarti.]

any update when the code will be available?

[@luna010]

@@Johanns0r code still says coming soon.

[@yolomuffins1437]

impressive, this is the same technology used in the skydio 2 drone for obstacle avoidance

[@shawn3d]

Hi Johannes, is there a way to use this to extract depth maps from video from a consumer standpoint? I want to be able to create depth maps from video to and depth of field in After Effects. Is there a way I can work with this software. Your help will be greatly appreciated :)

[@aeverless]

man this is absolutely INSANE waiting for github to try it out

[@dragonmares59110]

Would be curious to see the results on something with complexes reflective surfaces.

[@ruman2494]

Yes, like image on mobile screen

[@myelinsheathxd]

This is the most crucial TECH for AI learning, we give a game from any game store to train, then they learn VISUAL WORLD without leaving silicon

[@TheSamucacs]

can u make a software with all bundled? its hard to set all things up

[@shumshersubash]

Does the repo have code for injecting 3D objects in the video as in 0:50?

[@Kopsirtube]

how can I use the resulting depth map for creating blur effect? davinci, sony vegas, premiere.. etc ?

[@WhiteDragon103]

I find it interesting that simply trying to minimize error to ground truth isn't enough to have consistency, and that explicitly minimizing consistency is required.

[@kwea123]

I'm more interested in how the video effect is added, it's already applicable with one image only!

[@wiiu7640]

I'm having problems getting it to work. It says that I am missing dependencies but I have made sure to follow the instructions carefully. Could someone post a setup video so I can make sure I'm doing it correctly? I run Windows 10 on my machine.

[@reddcube]

Impressive. Would there be a benefit to using dual capture of a Wide camera and the Main camera?

[@Johanns0r]

Yes, this would make the problem a lot easier, because the scene is static across the two views and you get a consistent scale because of the constant camera baseline. However, not many people capture dual-videos :)

[@ge2719]

@@Johanns0r would it be as simple as just feeding both sets of footage to the code you have released though or would you need to modify the code for it to use two sets of footage?

[@RANDOM-em6bv]

This cat is legendary

[@lanik8163]

OMFG I want this now >~< I don't care what hoops I have to jump through to get this running! It's fucking amazing! OwO

[@BlakeEdwards333]

Thanks for sharing!

[@ajay6225]

Can we do in one cameras

[@rameezshamalik6735]

Can you share the dataset

[@cocohand781]

when will you share your code

[@dudegalen]

how do i use this to create an alpha channel mask

[@user-cw3nb8rc9e]

Hi. Could you please make a walkthrough on youtube, how to use your github code on PC with Geforce card? I am sure lots of people would love to experiment with this 3D depth.

[@nils2868]

Does this work with video where the camera isn't moving?

[@xanthirudha]

Is this using Light Field Synthesis?

[@chahatdeepsingh8473]

Amazing work! FYI: @2:47, 'Input' and 'Consistent Depth' images are exactly the same.

[@Johanns0r]

Ooops :)

[@tanmay8639]

I am excited to play with this

[@antoniocottone5382]

this method works with static images too?

[@importon]

Very nice! Will there be a google collab notebook available to try this out for noobs like me? That would be amazing!

[@Johanns0r]

Yep, we just released the code, and the is a Colab Notebook, too.

[@importon]

@@Johanns0r Thanks so much for doing this! I'm confused about 1 step however. What do I input for "camera model" and "camera params" If I want colmap to calibrate my camera????

[@luna010]

when will you release code?

[@Johanns0r]

github.com/facebookresearch/consistent_depth

[@MaxLohMusic]

Code still says "coming soon"

[@Johanns0r]

github.com/facebookresearch/consistent_depth

[@serhatkuk4180]

Well done.

[@3n19ma]

make this public

[@Johanns0r]

github.com/facebookresearch/consistent_depth

[@NicoVuignier]

Mind=blown

[@Aeonhem]

Very exciting!

[@BorisNovikov1989]

Realtime? How performant it is? Pretty impressive!

[@ohiasdxfcghbljokasdjhnfvaw4ehr]

no

[@kwea123]

40 minutes for a video of 244 frames and 708 sampled flow pairs as said here syncedreview.com/2020/05/04/consistent-video-depth-estimation-generating-hq-depth-maps-from-single-video-input/

[@AceHardy]

👑

[@GBREAL01]

Will this be able to be used on single images or is it for video only? Thanks

[@Johanns0r]

The extra stuff that our method does only makes sense with video. For a single image you could use a method like Midas, however.

[@zaxsif]

Can I run this for a single image?

[@Johanns0r]

No, the extra stuff that our method does only makes sense with video. For a single image you could use a method like Midas, however.

[@user-cw3nb8rc9e]

By the way - are you able to extract more depth from faces and human bodies, generally objects? It seems faces and human bodies are rather flat here. Am I right? Many solutions make things work by extracting moving elements from the scene, but these still look like 2D posters on 3D background. Your stuff here is very good, but still I notice there is no great detail in depth of the scene. Can you, will you improve this?

[@TheCatherineCC]

What's that song? :)

[@Pronobozo]

brilliant

[@sherifhany386]

Insane

[@987654321zer0]

This is so cool

[@jaiv]

That is so good haha!

[@mattizzle81]

Too bad it is not real-time or near real-time. Other than video effects I could see this maybe being used to create training data for a faster supervised algorithm, I wonder what the results would be if you used this to generate training data for something fast and simple which can be run in realtime, like pix2pix for example.

[@ProfessionalTycoons]

amazing

[@mostafamohsen250]

wait so can you do this in real time or do you have to process the whole video first before performing this?

[@Johanns0r]

You have to process the video, and it's pretty slow.... Once you've computed the depth, though, you can play around with effects in real timel

[@Hwangssss]

I want more cute cat images.

[@michaelhartjen3214]

Is this Realtime?

[@Fapppful]

Is there something like a scientific notation cheatsheet for uneducated programmers anyone is aware of?

[@MrAlextorex]

check fastAI Practical Deep Learning for Coders course. I think they mention there a cheatsheet.

[@Fapppful]

@@MrAlextorex Awesome! Thanks for sharing

[@Alpha17x]

This is awesome. too bad it won't be something I can use on a creative basis for like 15 years probably. (because cool stuff takes forever to get integrated into anything.)

[@felix-dk9tr]

Be the change you want to see

[@FordTransitvan]

Can't understand a word