This AI Learned To See In The Dark! 👀

[@charliecolyer5965]

Just as a thought, if this network could ever be made fast enough to create these images in real time we could have night vision goggles that let you see as if it was the middle of the day.

[@eliaslay7699]

Charlie I could imagine that AR headsets like hololens could come with such features in the future

[@MPeti02]

I wonder though what those scenes actually looked like to the human eye? I don't really know how bad most cameras are in low light, but I can imagine the human eye could be closer to the improved results than the initial pictures without any goggles.

[@charliecolyer5965]

There is for sure some decrease in quality between the real life scene and the picture being taken, naturally. But a lot of these pictures were taken at night, the difference would be literally night and day between the AI generated version and what a human would see.

[@004307ec]

By using FPGA, I think it can run in real time (almost).

[@PandoraMakesGames]

Should be able to get an approximation quite quickly.

[@OrangeApocalypse]

This will be handy when SkyNet is locating the last Humans huddled together in the dark

[@raunak51299]

don't give it ideas

[@nibirananda9381]

Skyence 🤣

[@pacificnorthwet]

I'm glad we're considering Skynet's feelings here, not enough people consider just how hard it is, or how hard it can be on a poor superintelligence, to completely eliminate their creator's species.

[@cheesuscheetos4076]

If we get killed by AI, it's because we deserve it.

[@magicmulder]

That could be achieved with simple heat detection already.

[@nurbsenvi]

This is insane... we will be living in the world where nothing is real but accurately guessed lol

[@jonathanwalther]

RandomTechS@#T And this is very near to the common idea how our brain works. There no such thing like real, everything is interpreted and (re)constructed.

[@idjles]

That’s exactly what your brain has been doing with data from your eyes and ears all your life - faking and interpolating. We’re now seeing how fake everything is that we perceive.

[@curlyfryactual]

This is actually a problem that has plagued AI in the past, what is the "correct" way to see

[@jackedelic9188]

If u treat ur brain as a much more advanced version of such vision AI, then in a sense we're living in a world where nothing is real.

[@nurbsenvi]

Jack Wong Actually that’s even more insane as we are interpreting an interpreted image.

[@Xartab]

I was holding onto my papers, but I still got whiplash from this technique. The amount on information that can be correctly extrapolated from the samples is staggering!

[@MoonMoonStarBass]

"computer, enhance!!!" "computer, zoom the image"... I'm glad we can bring these things closer to reality

[@Zoonofski]

"Give me a 360-degree extrapolation from the reflection on his belt buckle"

[@B----------------------------D]

@@Zoonofski lmao

[@jasonchiu272]

Everyone gangsta till you zoom in and it says "send help"

[@brennan123]

Wonder if the same technique can be applied to audio. Have 2 mics, 1 high end mic close to the speaker for ground truth, and 1 cheap mic further away. Learn to produce better audio.

[@starrychloe]

Brennan Cheung - it can. Google demonstrated sound isolating AI that can pick out one conversation in a crowd. Search KZbin.

[@NolePTR]

It could also be applied to make cheap headphones sound far better like studio ones. Figure out how cheap headphones distort the sound, so after the distortions, it will produce perfect sound.

[@rachelslur8729]

@@NolePTR 👍

[@xxportalxx.]

@@NolePTR The problem with that is that we could already do it with simpler programs, we just don't lol Nearly all speakers, amps, etc. have their frequency response graphed in their datasheets, all you'd need to do is overlap all of these for a headset, and then send it to a company who makes an eq app that can then adjust the volume for each frequency spectrum. No fancy ai training neccessary

[@secondsandthings]

​@@xxportalxx. AI could still help remove some subtle noise and "fill-in" sound information that was probably not captured. Adjusting based on frequency spectrum can be a further mix/mastering on top of that

[@alexlamson]

Those before-after comparisons are incredible. Considering how much color is lost in the shadows of a low-light image, I imagine there's a lot of recoloring going on to get the output images. It kind of reminds me of the black & white -> color paper from a while ago. Really amazing work.

[@xxportalxx.]

I would imagine that to the computer the color isn't actually lost, considering unlike our eyes color cameras only record color intensity. All it has to do is add to the color values it has, and tune them I'd imagine, the real magic is getting the weights and the sharp edges right

[@andyowens5494]

At 2:40, the window frame gets a distinct blue hue post-processing: that suggests its extrapolating from a few pixels of more blue data, rather than actually getting the colour right. As XPORTALXx says, its more about working out where the edges are, and knowing which artefacts are meant to be smooth and which have “texture”. Still, very impressive results, albeit an odd way to publish results (on KZbin, rather than journals, but hey, welcome to this millennium I suppose).

[@jojoposter]

@@andyowens5494 This video was not created by the original researcher. The paper was published traditionally, you are just watching the abstract

[@ChaotikmindSrc]

And bam one year later, my google pixel 3 take photo without flash in full night like it is daytime !!!

[@renerpho]

Night Sight is great, but it is mostly using high ISO and long exposures, and to a lesser degree image processing. At least nothing nearly as sophisticated as what is shown in the video. Note that the software you are using is available since the fall of 2018.

[@ChaotikmindSrc]

@@renerpho It is obviously stacking frames to increase the light received, for sure. I don't know if you own one, but i was very surprised by the results, which are certainly on par with what we can se in that video. "Note that the software you are using is available since the fall of 2018." What's your point here?

[@renerpho]

@@ChaotikmindSrc I do not own one, but I have used one. Its Night Sight feature is quite powerful, and it gives good results in many situations. There is no doubt about that! It does not solve the issue of increased noise though. In dark scenes, it still applies very high ISO to increase its sensitivity, which is different from what the algorithm in the video does. As a result, Night Sight's images can get quite noisy. "Note that the software you are using is available since the fall of 2018." My point is that the software was released too early to have the algorithm from the video implemented. Your original comment about "one year later" seemed to imply that Google had one year to get this included, which is not the case. Maybe that's not what you wanted to say. Google has so much experience with AI that I wouldn't be surprised to see something like this in their next version of the software/phone.

[@rohitghumare7515]

It's actually like using the pro mode with long exposure and low iso , My budget phone camera can take better night shots than pixel 3 with pro mode 100 ISO and 32 sec exposure , I think I'll rather wait 32 seconds to take a nice night shot than spend 4 times more to get same result in less time

[@10shatrunjay]

@@renerpho No dude there is no increased noise. I think it does apply some version of this algorithm to enhance the image. You must have used an older version of the Google Camera or something.

[@DeSinc]

I *always* thought about this, even if the data is limited in low light why can't you just use multiple shots to work out what the image "probably" should look like? I've been waiting for this to happen ever since I bought my first phone that boasted good low light picture quality but ended up being total trash. like you said I hope this stuff gets put onto phones soon. I wonder if you could process it on the phone, or if it would send to the cloud to process there.

[@Xeller_real]

omg, ok.. what are you doing here? :D

[@marioisawesome8218]

@@Xeller_real he does not need an entry ticket my boy.

[@Xeller_real]

@@marioisawesome8218 ok my boy

[@marioisawesome8218]

@@Xeller_real no i am the king you can be zelda

[@DewFlip]

Thank you for making these videos. Its easier to keep in touch with the changing ML stuffs with simplified videos like yours. Cheers!

[@lm1338]

I assume it takes raw sensor data, so there's a lot of information in the dark images we can't see. Even with Photoshop you can boost the exposure of the raw file after it's taken because the darker shades aren't compressed together, so this is mainly a denoising algorithm. Mentioning it because some people seem to think it extrapolates what objects the bright images would contain, or what shape or colour they are.

[@adamrath7095]

For me that's the crazy part- the necessary data and equipment for insanely amazing low-light camera sensitivity WAS ALREADY THERE, but we feeble humans couldn't leverage it.

[@IIStaffyII]

@Adam Rath Your missing the point we knew that the data existed. However instead of amplifying and guessing with educated math out way to how the image would look with more light we just expose our sensor for a longer time. Thus leaving the image 100% represented to how it should look in low light conditions. This does tho bring a new possibility that is taking low light images without potential motion blur.

[@rachelslur8729]

@@IIStaffyII 👍

[@xxportalxx.]

Exactly, most of the color info would probably be already there, it just has to tune the weights and clean the edges with some sort of anti-aliasing like program

[@ikannunaplays]

@@xxportalxx. All the AI is doing is determining what is noise and what is not and then removing the noise.

[@MobyMotion]

Now it makes sense why Pixel phones have such great image quality, even with fewer lenses or smaller sensor sizes than the competition. I didn't realise how much of a difference a little ML could make. I wonder if you could synthesize the dark images, to get a much larger dataset? You could get millions of creative commons / public domain images and just make them darker and add tons of noise.

[@TwoMinutePapers]

If the characteristics of the noise match the camera's noise patterns, I can imagine that. This is not an easy problem. There are still other roadblocks though, the darkening also has to match, etc. Cool idea. :)

[@vladkostin7557]

the networks are also probably best trained for each particular camera specifically for real-time use cases. I think the training data is not hard to obtain anyway with a camera rig that is an array of cameras.

[@odw32]

Interesting, because you have ML "painting" a better photo based on trained recognition within the noise. It makes you wonder how to define "real" vs "fake" photos, because it takes a lossy picture and adds detail. What does this mean for evidence... At what point could "dreamed up artifacts" start to pop up? I think that's an important question, as these techniques might be used on CCTVs to solve crimes.

[@kushcabbage7505]

make darkening AI to train undarkening AI

[@vincent10kd]

Orian de Wit zoom and enhance!

[@gajop]

Enhance. Enhance. Enhance. Kids of tomorrow will be wondering what flash is.

[@rachelslur8729]

👍

[@andyowens5494]

Kids of tomorrow wont be able to get creative either :(. “Seeing” the image is not the same as seeing the beauty; back lighting, side flash etc.

[@MrCmon113]

I think you are suffering from a misapprehension. No kind of data processing can ever replace lighting. Just like the bat needs to shout in order to get information about it's surroundings, our surroundings need to be lit up for us to see. And the better they are lit the more information there is.

[@Q_QQ_Q]

@Taxtro its right since there is no such thing as completely dark neighbourhood . Its just that camera tech cant see but now with AI we can improve it to a greater length .

[@rileyguy5892]

@Taxtro Ah ah ah, you're pulling a bill gates "We'll never need more than 400kb of ram" or something like that. AI has already demonstrated the ability to understand lighting, check out KaryKH's video on generating celebrities. The AI can understand light direction and shading to a pretty substantial degree. Who's to say we *won't* have digitized lighting in the future?

[@curlyfryactual]

I was simply unable to hold on to my paper this time

[@Romulusmap]

I think there's no other channel on KZbin that blows my freakin mind with each video. Wow...

[@Navhkrin]

"Hold on to your papers" 10/10

[@bananalord8575]

"I LOVE IT"

[@dizzyaaron]

As a photographer, I always wondered why it was that we couldn't have an algorithm that would do just this! Shut me up. LOL!! This is insane! I am sure they will also teach the AI how to keep the image quality post lightening! Can't wait for this update to my phone in the future!

[@nipunasudha]

*This channel is a treasure that we must protect* ❤️

[@adamkrasuski4743]

I wonder if their dataset consisted of pictures taken from just one camera. If so, then it is quite possible that they all have a specific noise pattern, unique to this model or even to this specific camera, so the trained network may not generalize. Even if that's true, the result is still useful, since cameras could be in principle "calibrated" in the factory, with neural model fixing noise trained specifically for each one.

[@zjohnson1632]

akrasuski1 Maybe but according to the onscreen text each image was taken with a different camera.

[@rednafi]

Generalization isn't that necessary here. If you are a camera manufacturer, you can train and customize the model only for your camera hardware and upscale the image.

[@nononono3421]

They mention that in the paper at the end as an area of research for the future, as they currently assume it's trained for a given sensor.

[@NolePTR]

Trained for a specific sensor is actually best. Quicker training and the ability to keep an edge on competition.

[@Israelpwn]

Judging by the researcher's video, they practised with several camera models, so it seems to be a pretty general NN kzbin.info/www/bejne/p4iuhnmBbLKKrMk

[@glibjibb]

You know it's gonna be a crazy one when he goes "Hold onto your papers"

[@dsp4392]

Great coverage as usual. This will certainly be used in future iterations of phone camera software. That said, the power of long exposure photography is vastly underappreciated here. Having a sensor exposed for long periods creates spectacular results, it doesn't just "maybe help a tiny bit". Also, long exposure shots are made out of actual captured signals rather than extrapolated data. The downside is obviously that they do require the phone to remain stable for a few seconds.

[@RGPankO]

"And now, hold down to your papers" - educational and entertaining, the best way to learn :) thank you!

[@todabsolute]

This is what we REALLY need in our phones and not 4 cameras or other shit

[@ianerixon]

This will be great for all of those security lights in cities. Can turn them off and still get crystal clear image of the perp... although light still functions as a deterrent and allows for guards etc to easily spot someone trespassing irl. But we should see some level of light pollution reduction which is much needed

[@TheBaconWizard]

And as soon as we can get the processing time down to milliseconds using something portable, we have perfect night vision.

[@sifer0]

I love it when he calls me a scholar.

[@beepboopgpt1439]

I always get serotonin bursts from your Ai videos.

[@id104335409]

Next up: an AI was found looking at humans while they sleep.

[@elektriksheep]

At 2:08 he uses the word "aperture" where he actually means shutter speed. A camera or lens's aperture is always open. It is the shutter that opens and closes. If we're talking about long exposures, which is he, he means a longer shutter speed in order to brighten the image or increase the exposure of the image.

[@insert_creative_line_here1516]

I want this in glasses so you could see in the dark

[@kim15742]

I have to say, the photo at 2:45 looks really cool with ISO grain

[@jerekabi8480]

AI learned to walk, now AI can improve low light photography..this stuff is super amazing

[@przemekkobel4874]

For some reason my gut feeling tells me that with a reference image (black one) you can denoise like that without any AI. Not to mention that many neural networks of today shouldn't be described with words like 'intelligence' (as was shown with wolf/husky fiasco or those self-driving terminators pretending to be cars).

[@yosealdo1]

For smartphone, you can use Google Pixel 2, 3, or 4, you can use feature on its camera aplication calls "night sight".

[@jeongheonlee4556]

thanks to you i had an opportunity to learn about U-net(the architecture used in the paper). a nice paper. thanks!

[@damir1234567890]

This is amazing, I will try to see if I can train a neural net with old and remastered startrek to apply it to old episodes I'd like to see remastered too.

[@RelatedGiraffe]

I wonder why they didn't just take long-exposure photographs and decrease the light intensity of them and add noise to create short-exposure photographs when they created the dataset? Seems like that would have saved them a lot of work, as well as the risk of accidentally slightly changing the camera angle or the scene itself between the photographs. Appart from that, this is a really interesting application of image denoising. And the results are just astounding!

[@quentinretourne8502]

This paper is awesome. It's really astonishing. And also, look at the background in some photos: the trees look kind of weird 😛. Nothing surprising since they were not even distinguishable on the dark photo, but still fun to see !

[@Jacob_Crowthorne]

It's AMAZING!! Thanks a lot for posting it!

[@Kingkhan-qk2vk]

Now we won't even be able to hide in dark when ai comes to get us

[@WangleLine]

This is freaking incredible! Can't wait to see this in video editing software or blender or similar programs o:

[@paperabsorbdotcom656]

Amazing !!! great time to be alive !!!

[@silberlinie]

Ganz außerordentlich. Den allergrößten Respekt.

[@michaelemouse1]

How effective would it be instead to take 2+ quick images and then crosscheck them to figure out what they don't have in common, which will mainly be noise? Don't radars already do that with, for example, moving target indicator mode?

[@Sisyphus56]

Cameras basically take a ton of pictures and average them out already. The problem is if you move, even slightly, while the picture is being taken it will be extremely blurry.

[@michaelemouse1]

The average camera on a phone or sub-500$ camera does that? I thought they took 2 pictures then picked the best one.

[@KindOfyeah]

@@michaelemouse1 lol what is this 2001

[@bernhardtrian7471]

will this technique be adopted to smartphone users with A.I. cameras with updates ? Like my own current - the google pixel 2 ? Or only in the next Gen upcoming smartphones

[@franciscogtome]

Simply brilliant! 👏

[@QasimWani69]

Just for clarification, the AI-based system produced those images through convolutional neural networks without the use of flash? Wow!

[@PennyHerbst]

?

[@programaths]

With the Note 10, I took a photo during late afternoon with high ISO and tweaked the white balance then showed to my colleagues. They were a bit puzzled because it was a daytime photo, but that looked exactly as if I took it few minutes ago. If you do not fear to explore "pro" settings, you can already do fabulous things! Even a picture of the floor with low ISO and big sun can look very good! And with that kind of AI, well, it's freedom. Maybe "bye-bye" golden hour ^^

[@RaoulEvilD]

Holly smokes!!! Always ready to hold on to my papers! :-D Your videos are always so delightful, thank you for all!

[@95TurboSol]

This is mind blowing

[@nononono3421]

I wonder if it can perform well on images that aren't so dark? One would assume so, but for the few examples I looked at (takes forever to load) they are all very under-exposed and then far more exposed. In most cases, we usually want to preserve the brightness as we see it with our eyes, but without noise, not make the image much brighter. Of course that depends on the use case, when you actually want to see much clearer it can be great, such as for night-time surveillance, conservation efforts, and many more.

[@albingrahn5576]

the title reads like a post apocalyptic journal entry

Man, love your channel!

[@DOGMA1138]

There is major detail loss due to denoising and reconstruction; from the looks of it much higher detail loss than traditional patch/nearest neighbor based denoising techniques on high iso photography. You can especially see it in the loss of the depth details in the flower petals, text details is also lost compared to high ISO photography.

[@KyranFindlater]

it looks nicer for humans, but yes there seems to be a smearing and loss of information. but really, it is quite a nice set of results.

[@DOGMA1138]

@@KyranFindlater It really doesn't look nicer, it looks brighter it's the same problem as some people perceive louder audio as higher quality despite the fact that loudness often comes at the expense of dynamic range. Comparing an image reconstruction technique against raw high ISO images is pointless because there are a lot of post processing techniques on high ISO images that do not cause loss of information and produces very good result as far as denoising, removal of posterization artifacts, ABL normalization etc. go all while not raping the dynamic range of the image and not introducing any detail loss.

[@eldiableon]

This is amazing! Great work!

[@ImGonnaShout2000]

The uses for this lends itself immediately to the imagination. I hope we see this implemented in phones and editing software soon. Amazing

[@tarmac5482]

At first it blows my mind but as I think more about how a human eye can visualize things in lowest of the lights, I began to appreciate how closer we are in mimicking our ownselves. Btw human eye resolution ~576 MP and color spectrum of ~7 Million colors

[@oswaldkit]

incredible! fantastic! extraordinary!

[@jaydeepvipradas8606]

Great work! As neural network is not learning at run time, i.e. it is already trained using samples, you can convert neural network into a mathematical transform matrix. Spacial domain transform are common practice. Using weights and activation functions, a special domain transform can be defined. This will allow other mathematicians to further develop techniques for image processing, including frequency domain.

[@simovihinen875]

Thanks for watching and for your generous support and I'll see you... in low light, durr!

[@kabirbroadcasting]

For a layman this is a good example of what AI can do.

[@CabrioDriving]

Only works with RAW of a specific pre-trained sensor. You won't fix your JPGs this way. This is good for camera manufacturers though.

[@kenivia9476]

great video as always! really hope more people see this

[@MrMysticphantom]

If the performance of this is made fast enough and resource utilization small enough, maybe this can do live video too, on the phone. Oh man, I can only hope. Heck even if it reduces the FPS quality in the live video feed, it might still be amazing.

[@Rotem_S]

he said it currently is 1 sec per picture, so either you'll have 25 cores and a 1 second lag, or use a special algorithm using the fact that videos are continuos

[@NolePTR]

The fact videos are continuous can be used, definitely. There are compression algorithms that can handle this sort of thing.

[@christopherludlow684]

>Beating the top tier machine The winning move is running away.

[@junogregoire5257]

Very impressive results. I wounder though if we could get a lot more detail by shooting few pictures in rapid succession, and then having the AI stack them.

[@YGODueltainer]

Does Google's cam nightsight considered using machine learning to achieve bright as day light photo at night?

[@alexharvey9721]

Holy smokes! I'm glad I was holding onto my papers!

[@letMeSayThatInIrish]

Amazing. Also very useful in real-life applications.

[@syedabuthahirkaz]

Oh This AI thing ! Nothing seems impossible. I love this one.

[@Christian-zv2em]

This may be useful for self driving cars which also have to see in the dark. Processing time only has to improve...

[@venkate5hgunda]

Does anyone know of an open source dataset resource for training this?

[@johneygd]

But it can only work with matching pixels, if a scene has no matching pixels ,the system cannot make a brighter version of ir, so it’s dependend on recorded scenes in the morning or afternoon,also i can’t imagine that it could even work with video’s.

[@makethisday7349]

This is just like movie VFX breakdown

[@yotraxx]

Impressive, as always !

[@247_sirazulmonir9]

just wanted to know if google camera uses this AI

[@daniel_960_]

Well smartphones have some insane night photography now. Was blown away how usable a photo was from an environment where I myself could barely see anything. The amount of detail in darker scenes is astonishing. Was iPhone 11.

[@fenixgrey7419]

Sorry if out of topic, but if this is true, can machine learning "guess/reconstruct" faces with a lot of noise like from low end CCTV? Or, with machine learning, can we make digital zoom become "analog"?...

[@A1egz]

Infrared and thermal cameras: *_am I a joke to you?_*

[@mariovelez578]

but we DID get a smartphone implementation of this! it's in the new Pixel 4! I love it when you review an AI and it actually gets implemented somewhere!

[@spankymebottom]

that is nuts! it looks like the closer images suffer from "soft lens" effect but the distant ones are amazing. pretty sure they will perfect the close up shots as well. nuts I say! how does this compare to real time night vision? a one second delay is acceptable for most use cases

[@crb2222]

Incredible stuff.

[@MrYerak5]

Can it do a CSI las vagas zoom on the reflection of a spoon that a killer used to get his mog shoot?

[@snaplemouton]

*Holding onto my paper intensify*

[@illogicmath]

Amazing but scary at the same time

[@alexhutchins6161]

Cant believe it's not even removing shadows to do this. Really impressive

[@nobocks]

Has photographer I'm so hype on this. Need an ai for overexposed to for her one file

[@bilobolygregsmith270]

So... if I knew how to code... Would this be a program for editing a photo after it is taken, or does it need to be an app for a phone for this to work. I want it. It's cool

[@johnjelatis2033]

He said that it takes raw input data, so it’d have to be an app, though an implementation could be written to use from camera roll, I will try to make a webpage to do this soon (reply to this to remind me in a week).

[@bilobolygregsmith270]

@@johnjelatis2033 really cool. Let me know if it works out

[@easternwind4435]

The Picture at 02:28 actually looks better without the new tech

[@easternwind4435]

@Cole Park it doesn't look like night anymore more like twilight and the contours seem blury

[@the-selfish-meme7585]

So with GPT3, + GANs generating avatars + voice simulation from seonds of audio + full body deepfakes from a photo... reality is going to be an even more slippery business than it always was... yikes! 2 papers away from utter mayhem ... what a time to be alive... :) Love the channel, by the way - keep it up!

[@Flyboard12345]

2:08 not aperture but shutter

[@appa609]

how well does it perform on photos different from the training set?

[@lorforlinux]

That was breathtaking!! super duper cool boss

[@wojciechszmyt3360]

This is amazing... I need this!

[@BohumirZamecnik]

Wow. That's remarkable! It reminds me another recent paper which denoised outputs from a global illumination image synthesis algorithm. If it works there it's quite likely to work also for photos.

[@satortenet]

It's over a year now... Why am I not seeing this in my smartphone yet?

[@slaveNo-4028]

so even the darkness wont protect me from a person taking unwanted pictures, ouff

[@cebasVT]

Yes, on phone camera not unthinkable anymore. If you check in cebas.com/finalRender, AI denoiser was already implemented for high quality, photorealistic rendering. Thanks for sharing.

[@vodouch17]

This is actually now implemented in to ai denoiser but it instead uses the very grainy image to enhance the image using ai

[@voltavidTony]

Don't google phones already do this?

[@ElDesvanDeDan]

This is 2 years old

[@voltavidTony]

@@ElDesvanDeDan Google was able to do this with the Pixel 2, which came out in 2017. They had a dedicated chip for processing images to make it run faster btw