Video Game & Complex Bokeh Blurs - Computerphile

How do Madden, FIFA, PGA Tour get that lovely shallow depth of field in real time? Dr Mike Pound explains how Complex Gaussian Blurs can be separable.
Mike's code: github.com/mik...
EXTRA BITS: • EXTRA BITS : More on B...
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottsco...
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com

[@ShinoSarna]

Fun fact: original 1998 Half-Life was going to have an effect where an enemy can hit you so hard your glasses fall off and your vision is blurred, but it was rejected because it was impossible to do at the time.

[@gordonfreeman5958]

That is a fun fact indeed!

[@Scotty-vs4lf]

wow thats actually pretty interesting. shows how quickly computers evolved

[@n7565j]

Outstanding explanation!!! My son was home last weekend trying to explain this to me and I just couldn't picture it... (he works for Ubisoft) Your explanation was superb sir, thank you!!! :-)

[@richardtickler8555]

the goal was to make the picture less clear and your son achieved that

[@n7565j]

@@richardtickler8555 I see what you did there... Well played sir :-)

[@raykent3211]

Useless titbit: I live up the road from La Gacilly, Brittany, original home of Ubisoft. Many years ago I bought a high quality pc sound card, branded "Guillemot". Same family business. The same little town is also the home of Yves Rocher cosmetics. Amazing.

[@theguythatcoment]

Damn, I wish my dad would do something like that. I just gave up on trying to explain to my family what I do all day long, it makes me feel all awkward and missplaced when everyone stops paying attention because they don't know what I'm talking about. Your son must be so lucky to have you.

[@jojolafrite90]

Ubisoft... Could you tell him to stop making shitty games? Thanks.

[@TorgieMadison]

0:35 "Gaussian blur, the thing we focused on..." hhehhehheh

[@rachelslur8729]

🤣

[@HarleyAMV]

*Game Devs:* **This Video** *Gamers:* **Turns Depth of Field off**

[@tetramaximum]

Because FPS matters

[@giganooz]

@@tetramaximum Because details matter

[@lizardweedzard4495]

Because not feeling motion sick matters

[@retrodragon2249]

@@lizardweedzard4495 If seeing DOF makes you sick then I'm truly sorry for you.

[@lizardweedzard4495]

@@retrodragon2249 it's very pretty but having it constantly update and refocus wasn't very pleasant.

[@phillipharryt]

Thank you for posting Dr Pound on Valentine's day. In my array of favourites he occupies index 0.

[@phillipharryt]

@Sassy The Sasquatch well I'm not in the UK

[@JR-mk6ow]

That's such a nice thing to say

[@supersonicgamerguru]

Arrays can be in arbitrary order, you should use a max heap to ensure that your favorite person is at index 0.

[@Xilefian]

Doom 2016 does a fast real-time Bokeh where the half-resolution framebuffer is blurred with a circular average (tapped fragments in a circular pattern, no Gaussian used) and then a second blur is performed where the colours are blended with a colour "max" operation (brightest RGB components kept).

[@peterjohnson9438]

it's brilliant in its simplicity and still produces really nice results

[@totlyepic]

id's always had amazing graphics programmers (headed by Carmack until only a few years ago, obviously). It's awesome to see that they still support that culture.

[@garm0nb0z1a]

Would've thought it would be smaller than half-resolution. I use much smaller resolutions and don't see much of a delta in the image, but a massive difference in processing time and storage.

[@TavishMcEwen]

That's what i was thinking would make a lot more sense

[@TunnelDragon44]

Damn, every time I hear something about how doom 2016 was made I'm extremely impressed.

[@robisnowtired]

Real game developers who are supposed to know what they are doing: *furiously taking notes*

[@SillyMakesVids]

Imaginary game developers: *this is too complex*

[@invention64]

@@SillyMakesVids I would bet this might be complex for some game devs since not necessarily all of them are full stack.

[@DarkOracleOfDeath]

Usually game developers use existing game engines like Frostbite, Unity, etc. Those are already written and it's a little bit easier. Not to say developing a game is easy by any means, I have no idea what it's like. I don't work in the field. But what I said in the initial sentence is pretty much what I understood from the video. That's the real backend working at kernel level.

[@stormsurge1]

@@DarkOracleOfDeath I agree. Bigger game development companies have people who work on this, but indie devs almost certainly don't make their own engines. Even big companies use an engine in multiple games after it has been developed.

[@ozzieggg]

@@SillyMakesVids dad joke math joke mix, i see what you did there

[@peteckone]

I love how Mike always is on the defensive with the symmetry of his drawings :D

[@qwertyasdf66]

I'm from an audio background and the moment he drew the overlaid sine waves it immediately looked to me like sine sound waves of different frequencies overlaid to create square waves. That made me immediately understand what he's talking about.

[@Nick-bh5uk]

Fourier transforms are everywhere!

[@Nagria2112]

stop everything. best host of computerphile is here. i could listen to his explainations the whole day

[@nilspin]

I did this at my first job, after separating it into two cheap draw calls, code sped up 80% lol I was so happy

[@777thejesusfreak777]

I see mike. I click like.

[@Wolves2314]

Technology! mike is my fav

[@franzusgutlus54]

You are a simple man...

[@GijsvanDam]

My brain melted when you showed the bokeh blur of the stars. It went into an infinite loop of trying to focus.

[@geoffsmith8172]

Man, it's enjoyable just to hear an academic not be dismissive of game design.

[@SergeyArhipenko]

It has surprisingly much resemblance with a drill that can drill square holes. Rotation is an equivalent of a sinus wave - and if you combine right set of waves (in Fourier series), you can get virtually any shape. Consequently if you then translate sinus to rotation, you can drill a hole of any shape - the only limit is a resistance and strength of materials involved.

[@niwasox3]

There is also a brilliant separable hexagonal blur to simulate specific lenses that essentially does 3 line passes at 120° to each other. And the technique shown here can be used for much more difficult filters, for my thesis I implemented a realtime fullscreen inverse-square blur to simulate global illumination.

[@georganatoly6646]

Something like a given object is dynamically defined a z depth, 1/z^2 based blur value? The interesting bit would be how you defined which objects should have a given z value in real time.

[@kushagrakhare6995]

Niel... Could you share your thesis?.... I am working on inverse lighting and your thesis could help me....

[@TheSandvichTrials]

Ditto, can you share that thesis? A realtime fullscreen inverse-square blur seems very interesting to me!

[@mdtrx]

It's always fun to know how much goes into games and people don't even realize it

[@MrSkinkarde]

People realize it. Maybe you don’t

[@shaunhutchinson4707]

Would love to see more videos of Mike talking about the implementation of maths in video game development. Would be nice if he/you were able to do a collab with Digital Foundry.

[@SteelSkin667]

That is a fascinating subject, I've always been, fascinated by camera effects in video games. You can tell in an instant when a game is using some other method of blur to simulate depth of speed rather than a proper bokeh approximation.

[@spookylilghost]

A lot of people in the comments objecting to motion blur, which I can understand but it's important to realize that the tech developed for motion blur is also used in per-object motion blur which is typically a lot more tame and doesn't get the motion sickness complaint. DoF similarly can be used really well in cutscenes, but can be distracting during gameplay. On slower paced games, though, it can work with gameplay as well (turn based RPGs especially).

[@Bbonno]

Multi-camera phones must be doing some version of this to let you play with the bokeh.

[@Jokker88]

yes they blur the image using similar software but they generate the depth map approximation from the parallax of the two cameras by using photogrammetry. The depth map is used to separate foreground from background and to adjust the amount of blur in relation to the depth.

[@Pretagonist]

Well the phone would have a lot longer than 1/60 of a second to do its magic so it might not need filters optimized for speed like these ones.

[@ElectricWheelz]

Jerry Rapp there is potential if they want to start applying it to video at 60 FPS

[@ES-qy2ju]

This is even more easy with ToF sensors

[@azertazert98]

@@ES-qy2ju but they are pricier than a camera😥.

[@ChrisConnett]

Those are the most beautiful freehand Gaussians I've ever seen.

[@gm4984]

I think that using the same image to compare both codes is really simple and clever!

[@JBP.3D]

To give an idea of the sort of improvements you can get with separable kernels, I optimized an image processing algorithm with that technique that would run at 4.5FPS at VGA resolution. After I was done, I could run it at 21FPS at 720p resolution. This involves some other tweaks like moving sampling point to reduce and weigh at the same time but a lot of the performance improvement comes from kernel optimization.

[@Vaasref]

I think an other possibility would be to detect edges to determine where the effect would be best used as it would be wasted to use that blur on a plain surfaces.

[@Rudxain]

I saw a video about the trigonometric sinc function, and it looks very similar to the waveforms that cancel each other to create an approximation of the rectangular function. I guess this is one of the reasons why sin(x) / x is so important in signal processing

[@dreammfyre]

Digital Foundry collab, when?

[@maxemore]

I will never be as awesome as this guy

[@chkone007]

Very interesting, I already work on this, to add something on the explaination, what we usually do, is changing the size of the blur size based on the distance to the focal plan by reading the Z-Buffer, what a real lens do. Or the separation between foreground and background is more background, focal region, foreground and blur the front & back separately to not mix color of back & front blury region. And in game we directly work with HDR/Raw buffer where the values are not clamped to 0-1 (aka 0-255) but work with much precise buffer 10 or 16 bits per channel RGBA.

[@aleph_eight]

sir I have a doubt that if we create a square wave even with a complex gaussian curves wont it create a square in 2d when we try to do create a 2d kernel using the 1d square ware?

[@charstringetje]

The donut shaped bokeh is what you get when you use a mirror telelens. It uses a combination of concave mirrors to do the magnification instead of lenses.

[@Hamachingo]

5:14 the accuracy of that hand drawn Gauß curve 😍

[@Vextrove]

Like adding sine waves together, that converge into a square wave?

[@rachelslur8729]

Exactly.

[@unfa00]

I thought about Monty Montgomery's "Digital Show & Tell" video, where he shows how that works in audio :)

[@OlliNiemitalo]

OP here. Yes, I did exactly that as a proof of concept when developing the algorithm, and attenuated the next period of the square wave by applying the Gaussian envelope. But the results were improved further by using also a different Gaussian width for the different frequencies, and by allowing any frequencies, not just harmonics as in Fourier theory, and just optimizing numerically.

[@flaguser4196]

nice real time bokeh reflecting from the whiteboard in the background!

[@bryan69087]

MORE MIKE POUND also what model lenovo is that?

[@vananandreas3070]

I think it's a thinkpad model

[@Edward70891]

An L480, maybe?

[@EnigmacTheFirst]

Ah, yes. The Frostbite Engine. Known for its amazing sports games.

[@Veptis]

Apart from video games, it's also a topic for rendering photorealistic CGI. One of the best plugins to do this in post (on a 2D image) is called Frischluft. If you have a 3D scene and a render engine you can go way crazier. Disney for example scanned real cine lenses to reproduce optical results (including filters and lens effects like adding a split diopter).

[@wixic111]

Mike Pound ❤️ My favourite lecturer

[@TylerMatthewHarris]

Thank you! I hope for much more of this type of video. Image processing is so important

[@dhoyt902]

One has to bring e^(-x^2) to the complex plane to reach the Gaussian Integral of sqrt(Pi). The area under the bokeh curve is Pi, from that derivation.

[@noa670]

It's just in the game

[@noa670]

EA sports

[@steinardarri]

Their tagline

[@mikakorhonen5715]

@Sassy The Sasquatch Joke went over your head mile high. :D

[@umeng2002]

beautiful bespoke bokeh blur depth of field

[@joho0]

EA Tiburon in Maitland, Florida has been producing Madden NFL since 1989. One of Orlando's hidden gems.

[@sarthakshah1058]

Using a whiteboard marker on paper? ABSOLUTE MADLAD

[@raglanheuser1162]

i wish i were british just so i could "have a go" at things

[@leonnis84545]

Props for the little lego man and his car.

[@mapmemer1897]

Great video! Please do more of these game related videos!

[@younghsiang2509]

Love this!!! Please make more about Game Rendering Techniques!!!

[@adamspihlman9665]

I would willingly give this man all of my passwords

[@Finicky9]

I always think that game engine design problems are the best combination of maths and programming that there is and on top of that it has to do in real-time. Even robotics gets shortcuts that games engines don't have the luxury of having.

[@jagardina]

I learned imaginary numbers but have completely forgotten what they are and what operations are useful with them. I'd appreciate a refresher video.

[@jojolafrite90]

Lots of game engines use Bokeh based depth of field. And they will use it more and more. Even Dark Souls had a mod that added this type of blur (you could even choose a Gaussian based one)

[@msolomonbush]

Mike, could you run an inverse of this operation to try to "de-blur" images? Or is this specifically a one way function?

[@daniellee6912]

I always wanted to know this

[@acbthr3840]

Michael Bush I think the main issue is that blurred shapes in an image overlap and that overlap gets interpolated together. I imagine it'd be quite difficult to reverse.

[@DeathBean89]

Assuming that the kernels that you're using are square (N by N pixels) and invertible (they have a non-zero determinant), then you should be able to define an inverse for the kernel and de-blur the blurred image.

[@mikakorhonen5715]

Google lightfield camera. It can do that with real world photos, but is too limited for everyday use.

[@smoscar]

If you think about the way light gets propagated when out of focus (visualized in minute 2:05), you’ll notice that the information about those areas is lost. I’d imagine it would require a very sophisticated artificial intelligence model to derive/guess what information was there to generate the blurred area.

[@jellyboy00]

I don't know/understand the difference between adding two complex filtered image together or adding four real filtered image together. 9:20 is adding real part and imaginary part same as addition done on two real value matrix?

[@dream_emulator]

This is so awesome.

[@Lucas72928]

How would a blur using a depth pass work? In the video they showed the depth pass as a mask, but I don't fully get how you would weight the convolution to make it less pronounced in a range of gray values from the depth pass

[@Blackreaper777]

With stuff like this, if it's done properly, you don't even notice, because it just looks normal. You start noticing when it's not done well.

[@artursz1993]

Thanks, Mikey! Was easy to understand, how it works! I see Mike, I like :D

[@rcfanatic101]

Blew my mind... as always!

[@Byt3me21]

Great explanation, thanks!

[@daft_punker]

Mike Pound!!!!! What a guy

[@ericsbuds]

fantastic. it amazes me how many calculations are happening every second when I play a game on my computer. I play competitively so I generally turn off all the fancy graphics and try to get 200 -300 frames per second.

[@kylebowles9820]

What kind of monitor do you have that can do that framerate justice?

[@ericsbuds]

@@kylebowles9820 believe me I need a new screen, but every frame makes a difference.

[@Pianet]

Dr. Mike works with making DSP realtime efficient. I never knew he was truly a man of culture.

[@vix86]

In Computer Vision, you often run a Gaussian blur pass on an image to reduce the amount of data that you might pass into your algorithm (ex: Object Recognition). Seeing this video, I wonder if a Bokeh blur would be better or worse in most CV tasks.

[@totallynotabot151]

Blurs in CV typically act as low-pass filters, removing noise so you don't have stray pixels to deal with. I don't think the type of filter shown here would help with that task; however it looks like the Bokeh filter needs 2x the computations (and possibly 2x the memory).

[@Jonas-mg5wx]

Wow! More please!

[@Toschez]

Bokeh is interesting to a Japanese in two ways: why there is h at the end (we don't romanise as such, it's just boke. Maybe to make it look more "oriental"?), and how the word that just means blur or haze in Japanese became the word for specifically this type of lens blur.

[@Hwyadylaw]

I think the H was added because otherwise the E would be silent in many languages. (Like the word Joke in English)

[@juantelle1]

Thank you! I’ve always wondered how to they achieve cinematic depth of field (because Gaussian blur always looked awful). Is this the same thechnique used in ray tracing renderers?(blender, cinema 4s) also in portrait mode on smartphones

[@Mythricia1988]

Ray tracing renderers don't need to fake depth of field, because they simulate how light actually behaves. So it occurs as a natural effect, if the focal length and aperture is set up for it.

[@karlkastor]

Does this work for other shapes? If you put a piece of paper with e.g. a heart-shaped hole directly on your camera lens, the bokeh will have the shape of a heart.

[@DrewPlusPluss]

Karl Kastor yes, if you store a bunch of points in a heart shape in an array, then use that array as the pixel offset value! You’d have to store a large amount of offsets to get a decent quality. Although this method would not be separable. You could do any shape, but... lots of offsets.

[@arnavrawat9864]

Love this guy

[@kevinocta9716]

What's interesting to me is then in a telescope when you have a star in perfect focus, even then there is that same sort of 2d wave-disk (like your 1 or 2 part approximation) that I'm thinking is connected to that same fourier analysis kind of thing that you're doing here- it's called a diffraction pattern. Interesting. It's similar to how Pi ends up popping out of seemingly nowhere in some math problems...

[@atimholt]

I want to see someone try to do an organic bokeh. I had an epiphany once where I realized that the “frozen sparks” shapes surrounding bright point sources at night must be this kind of thing, but with the color part of your eye (your aperture) not perfectly stopping light in a circular shape (and maybe there’s some diffraction going on, as well). Gets harder to calculate with our spherically arranged photoreceptors, and human irises are as unique as fingerprints. I imagine it’s also more pronounced for people with blue eyes.

[@SenorQuichotte]

Use bessel functions as basis instead of gaussian. In the games i think they would pre render the background if you already have a priori knowledge and not waste the GPU

[@dhvalden]

yes! dr. mike pound!!

[@turolretar]

We need a Pound Grid vs Parker Square showdown

[@mohamedmoatyhassan5125]

I love this prof

[@vananandreas3070]

Nice explanation.

[@kamon242]

The only thing missing is a real life comparison of the number of operation / increase in time efficiency between the twin 1D vs the 2D disc convolution methods.

[@wedusk]

I love this guy!

[@benjaminbrady2385]

3:47 It's not the best kind of correct!?

[@serger3302]

How does this compare to the technique where a vertex/fragment shader is used to create actual circle geometry around high intensity areas?

[@tempdrive2342]

sir I have doubt, if the faking is possible why mobiles have separate lenses for these,due to more clear effect ?or can we expect a single lens doing all these in future?

[@alphanimal]

That was really interesting!

[@fuksrskrzqlw1213]

No Bob and Alice this time... :( Lovely blur filter explanation Dr !

[@richardtickler8555]

i think theyre somewhere in the backgrounds but its too blurry to properly see them

[@fuksrskrzqlw1213]

@@richardtickler8555 indeeed , for that i launch a Broadcast Ping since it s too blurry to see... Hopefully they are alive to respond.

[@apratim4859]

is this supposed to be like additive audio generation where bunch of sine waves of different frequency come together to form a sawtooth or square waveform ???

[@Friedolin-qz9id]

its cool to do it that way but I dont quite understand why the output of the gaussian curve could be checked and then rounded? I guess devs dont want to do that because it doesnt look as blurry anymore? but couldnt the check have some kind of threshold then? like round up everything above 0.8 and otherwise multiply it by 1.25 eg

[@kevinowenburress2435]

*sees thinkpad* what thinkpad is that? Can you use the inverse of the gaussian on the outside and subtract the blur from the not sharp edges?

[@SimonBuchanNz]

That's basically exactly what this is!

[@TheThetobias1997]

With talking about imaginary numbers it is important to note: i is not the square root of -1, rather i squared is -1. You would think that's equivalent, but with the square root thing you can proof -1=1 which is not... Ideal

[@BurakBagdatli]

This is a really neat solution. I wonder what it would take to get this to run on a GPU and parallelize it like crazy.

[@andtfoot]

As soon as I saw the diagram at 4:20, I thought of approximating square waves using a sine wave and its odd harmonics (with more harmonics = better result). This seems kind of similar...?

[@willhendrix86]

I wonder if RTX Ray Tracing is capable of physical based and realistic lens bokeh rendering? I would like to see such an application vs a calculated approximation.

[@theRPGmaster]

Yes, this can be done with raytracing

[@Janokins]

Anyone got a link to the original presentation / paper on this?

[@drawapretzel6003]

just wait for ray tracing, when we can render our games with actual light, and then do actual lens physics with them.

[@uhrguhrguhrg]

@MichaelKingsfordGray it is called "not being a condescending asshole". You must try it some day.

[@DasAntiNaziBroetchen]

@@uhrguhrguhrg Ray tracing doesn't use "actual light" but a crude approximation of it (rays). And if you want to have DoF using ray tracing, you have to massively increase the number of rays used, which will still make it impractical for quite some time (unless we find some amazing noise reduction algorithm that works for out of focus areas).

[@robertidonotsharemyfullnam496]

would it be hard to implement this as a photoshop filter with an alpha channel as z-depth source ?

[@TheBabalaba]

Waiting for FFTs!

[@Sh-hg8kf]

I am struggling to understand this, how do people evem know adding imaginary numbers lets you somehow get a disk like blur? The very concept of imaginary numbers sorta confuse me even after I read up on them a lot :(

[@BADC0FFEE]

Great explanation :)

[@eratzleretour1027]

Cool! Like for JPEG2000, are wavelets used in video games nowadays?

[@finnaustin4002]

Compiterphile: interviews of computer scientists who don't know how to take a screenshot

[@Y0uTubeCommentPoster]

This is neat, but I feel like compute shaders have made these sorts of techniques outdated. All of this separable stuff is ultimately to reduce the number of samples you have to take from the source texture for each pixel that you're calculating. But with a compute shader, you can simply have each thread sample the texture, and then share that sample with other threads in the same group using groupshared memory. Other features such as atomics and unordered access view textures open up even more doors.

[@WindBendsSteel]

the most complex algorithm to create a disc

[@hikari_no_yume]

Do games actually use Gaussian blur much at all? The easiest blur to do on a GPU is to linearly scale down the image, and cheap depth-of-field effects do just that.

[@NJBoyd]

Fantastic explanation, but rather heavy on the Frostbyte engine / Madden plugging. Given the uncanny timing of both this and Stephen Colbert's new video having a heavy focus on the game, I'm a little dubious that there aren't some "sponsored content" labels missing in this or that video.

[@corvineberlein]

Or, you know, Frostbyte might just be a really good engine and it's mechanics are interesting and worth talking about. EA has their massive flaws but not every positive thing said about them has to be sponsored.