TimeLens: Event-based Video Frame Interpolation (CVPR 2021)

State-of-the-art frame interpolation methods generate intermediate frames by inferring object motions in the image from consecutive key-frames. In the absence of additional information, first-order approximations, i.e. optical flow, must be used, but this choice restricts the types of motions that can be modeled, leading to errors in highly dynamic scenarios. Event cameras are novel sensors that address this limitation by providing auxiliary visual information in the blind-time between frames. They asynchronously measure per-pixel brightness changes and do this with high temporal resolution and low latency. Event-based frame interpolation methods typically adopt a synthesis-based approach, where predicted frame residuals are directly applied to the key-frames. However, while these approaches can capture non-linear motions they suffer from ghosting and perform poorly in low-texture regions with few events. Thus, synthesis-based and flow-based approaches are complementary. In this work, we introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both. We extensively evaluate our method on three synthetic and two real benchmarks where we show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods. Finally, we release a new large-scale dataset in highly dynamic scenarios, aimed at pushing the limits of existing methods.
Reference:
Stepan Tulyakov*, Daniel Gehrig*, Stamatios Georgoulis, Julius Erbach, Mathias Gehrig, Yuanyou Li, Davide Scaramuzza.
TimeLens: Event-based Video Frame Interpolation
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, 2021
PDF: rpg.ifi.uzh.ch/docs/CVPR21_Geh...
Webpage: rpg.ifi.uzh.ch/timelens
Code and Datasets: github.com/uzh-rpg/rpg_timelens
Slides: rpg.ifi.uzh.ch/timelens/slides...
Our research page on event based vision: rpg.ifi.uzh.ch/research_dvs.html
For event-camera datasets, see here:
rpg.ifi.uzh.ch/davis_data.html
and here: github.com/uzh-rpg/event-base...
For an event camera simulator: rpg.ifi.uzh.ch/esim/index.html
For a survey paper on event cameras, see here:
rpg.ifi.uzh.ch/docs/EventVisio...
Other resources on event cameras (publications, software, drivers, where to buy, etc.):
github.com/uzh-rpg/event-base...
Affiliation:
D. Gehrig, M. Gehrig, and D. Scaramuzza are with the Robotics and Perception Group, Dept. of Informatics, University of Zurich, and Dept. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland rpg.ifi.uzh.ch/
Stepan Tulyakov, Stamatios Georgoulis, Julius Erbach and Yuanyou Li are with Huawei Zurich Research Center.

[@theslowmoguys]

Well we had a good innings, lads. 😂

[@bcmason1911]

lmaooooooooo

[@SirWrender]

omg hahaha Gavin! I was wondering if you'd see this

[@danielgehrig9009]

Don't worry, there are always more work to to :)

[@turgor127]

This is gonna be in 2 minute papers.

[@BUDA20]

is now ;)

[@FloraSora]

Nice.

[@James-wd9ib]

what a time to be alive

[@irlshrek]

Man. Obviously this could be turned into an application that you feed a video into and it spits out a high refresh rate version. Incredible

[@silvahe]

i was wondering that how can you let the normal camera and the event camera get the same sight.

[@FloraSora]

I need an in-depth tutorial for this so badly!!! I'm dying to use this.

[@detroxlp1]

same for me, English is not my native language, I do not understand how to use my own videos.

[@bruce_luo]

Hope we can have event-based cameras as easy as webcams someday.

[@wayoftradingmalayalam2624]

Bro you find 2. Put that in Google collapse, any tutorial out there in the youtube ?

[@Upscaled]

Great video, looks awesome

[@kylebowles9820]

nice work!

[@cahydra]

it would be nice if this was released to the public, or if it already is then make it more user friendly since it would be amazing to try this

[@dmalyavin]

There is already github repo for it, and you can request access to their training data and training code. However just to note that you cant run this against a normal video as far as I can see. Their process requires an additional data feed that captured by their event camera. If anyone knows how to ether generate or capture this type of interframe event information (as far as I understood its just per pixel brightness changes) would be cool to find out. Thanks!

[@FloraSora]

@@dmalyavin Waait so I can't feed in my videos even if I jumped through all the hoops to get the github code to run? Aaaaaagghhh

[@NilasEdits]

​@@dmalyavin So this won't be a replacement to Twixtor.. and here I was getting my hopes up :'(

[@Ardeact]

I thought dain was impressive but the fact that there's basically no artifacts on the moving image is amazing

[@reel60frames45]

My videos are the reference to study the issues.( artefacts by high dynamic motion)

[@skk6811]

This is optical flow on steroids. Should write this to run on the Apple M1 chip.. It would rock.

[@bob2859]

Amazing! Now I just need a Prophesee Gen4M to fall off a truck...

[@Kaapalkeens]

oh man, really a great technology. But apparently its only on special hardware, Great, nontheless!

[@roidroid]

It would be interesting to gradually reduce the framerate of the RGB camera, to find the limits of *how much* of the missing data your system is capable of constructing. I'm curious how FEW keyframes are truly required to get reliable results. Or maybe a system can automatically request the RGB/keyframe only as required, and only for the nessesary part of the image rather than the entire frame.

[@danielgehrig9009]

Thats a really cool idea! In our paper (linked in the description) we go down to 5 FPS and are still able to interpolate the video reasonably well. However, I would not recommend to go any lower than that. When using events you can indeed do something like "adaptive slow-mo" where you only request keyframes when there is motion (enough events). This will make the approach much more efficient!

[@danielgehrig9009]

@Haneesh Allu i guess that you can alway collect a fixed number of events and then you would have an adaptive framerate. Since the fastest moving object will trigger the most events it should always adapt its framerate to this object. For slower moving objects it should be easier then.

[@megazenn22]

uploaded at 25fps 🤣🤣🤣🤣🤣