Gaussian Splatting explorations

Рет қаралды 24,588

9 ай бұрын

Let's dive into Gaussian Splatting: what is it, how are scenes represented, and what fun things can we do with it?
This is a fairly informal and code-heavy video - let me know if you like this format!
GS website (with links to paper):
My lesson on optimizing things with fun losses such as CLIP: johnowhitaker.github.io/tglco...
My Twitter, if you want updates on this as I go: / johnowhitaker
As mentioned in the video, please do let me know if you have any further questions or suggestions :)

Пікірлер: 58

@KP-if2rm 8 ай бұрын

This is awesome! Could you do like a detailed explanation of the math in the paper? There is tons of videos on gaussian splitting, but yours is the one that really helps to explain the details. But... it could be even more detailed!

@ajamoffatt 9 ай бұрын

Jonathan thank you for the great walk through. The video structure is excellent. It is also really cool work you’re doing to stylize the base results!!

@basiliotornado 9 ай бұрын

I never would've thought style transfer possible like that! Nice video!

@dyllanusher1379 9 ай бұрын

I love the pace, the comedic timing, the content and animations!

@camtrik3686 9 ай бұрын

Very nice video, thank you! Can you also share the notebook code you use in the video?

@Mr_NeRF 9 ай бұрын

Very nice video with very good explanation of the key concepts! Especially the sh are nicely explained.

@Wilqposnb 9 ай бұрын

i like this guy's personality

@MikeTheAnomaly 8 ай бұрын

Wonderful explanation! Thank you!

@georhodiumgeo9827 9 ай бұрын

Bro your ChatGPT history looks a lot like mine, just tons of insane neerdy stuff. How fast is the retraining process? Like could you retrain a scene during gameplay with a good gpu? Especially if you had some trained gaussians already in place. I've been wanting to make a project like this for a while but I can't figure out a way to do it with mesh objects. What I want to do is have an AI NPC on level 1 start talking to you adout something, anything really and let the player lead the conversation. Lets say you mention clouds then it starts googling clouds and starts training based on that as a prompt. By the time you get to level 2 its ready and level 2 is based on some structure but everything has morphed into clouds. That would be nuts when you realize you are just manifesting your own game play with what you are talking with the NPC about. I was thinking about having some primative sphere for most objects then using picture to depth field from stable diffusion help reshape the mesh then wrap a new texture on it. I think it could work on a high end computer but this method seems like it would be almost a dreamy hasey way to do it. If you could buy some time in cut scenes and dialog interactions it might be possible. Anyway this whole process for rendering is just nuts. Trying to wrap my brain around it.

@grantbaxter554 6 ай бұрын

Very cool, very well explained, thank you

@datasciencecastnet 9 ай бұрын

Ha, guess who forgot to add the visual aids during the intro..... hand gestures will have to suffice ;) Paper and code show up from 6 minutes in.

@user-ni3iq6xv3u 7 ай бұрын

Great video and explanation Jonathan! I really enjoyed your approach with CLIP to be able to modify gaussians given a text prompt. Do you have a link to the jupyter notebook you used in the video?

@itsm0saan 9 ай бұрын

thanks for the video ❤! As a suggestion, Would like to see something about peft (LoRa, Quantization …..)

@MattCruikshank 9 ай бұрын

Can you say the final size of the scene, especially compared to the size of the input images and the number of iterations? Would it be viable to embed the scene representation into a video game, for instance, or are they enormous? (Or would we need to use something like Wavelets to represent the scene, and transform back into these harmonic values before rendering?)

@monstercameron 9 ай бұрын

This is crazy interesting but no one is talking about it

@rmt3589 9 ай бұрын

People just started talking about it. This is the second video on it I've watched today.

@hdslave 9 ай бұрын

All github videos are like that. No one cares until it's integrated into some kind of consumer app. Ai stuff was the same way. Gan videos got no plays and no one was talking about ai until dalle and midjourny came out

@Instant_Nerf 8 ай бұрын

We are busy working with it

@nicholassmit6875 7 ай бұрын

One month later, my feed is literally filled with it 😁

@user-uu5ml5dc6n 5 ай бұрын

It is starting to blow up

@dl569 5 ай бұрын

thanks, very clear!

@philtoa334 9 ай бұрын

Very Good.

@Nik-dz1yc 9 ай бұрын

I'm curious though; Weren't there differential renderers with spherical harmonics already a thing since like 2008 ?

@lion87563 4 ай бұрын

Guy with worst image quality ever explains technique that produces best image quality ever. Just some geek joke, thank you for such a nice presentation

@Ali-wf9ef 3 ай бұрын

It would've been cool if you could visualize which point in the scene you are showing the spherical harmonics for

@er-wl9sy 8 ай бұрын

Thanks. Could you go over the CUDA code as well

@adityakompella9203 9 ай бұрын

Would it be possible to post the notebook that you walk through in the video?

@user-td2sz3yh9b 9 ай бұрын

nice

@BradleySmith1985 8 ай бұрын

this will be the new live AR option.

@naninano8813 8 ай бұрын

well spherical harmonics (thanks for the explanation btw) are cool and all but they are not adequate way of modeling light caustics. like if you have something refractive in your scene, like a half full glass of water, I wonder if splats can represent how objects behind it would warp. and re-lighting a splat scene (adding a light source, post capture) is tricky IMO

@HaiweiShi 6 күн бұрын

Hi, really good video. Could you share the jupyter notebook you showed in this video? It would be so grateful!

@pouljensen2789 6 ай бұрын

Can splats emit light (instead of just reflecting)? If not, how difficult would that be to implement? I'd like to try modelling aurora, which would correspond to fully transparent splats emitting light.

@aintgonhappen 9 ай бұрын

the description is missing the link to the paper website 😥

@WhiteDragon103 9 ай бұрын

Do the gaussians model view direction dependent effects? Or are they best suited to only represent lambertian materials? E.g. can specular highlights, refractions, non-planar reflections, fresnel effects, etc. be modelled using these? If so, how? As far as my understanding goes, these gaussians are basically single-colored blobs, similar to billboards (as used by particle effects for games).

@datasciencecastnet 9 ай бұрын

They model directional effects using something called spherical harmonics, where the color of a gaussian depends on the viewing angle. It isn't perfect, but it let's you get the appearance of reflections and shine.

@WhiteDragon103 9 ай бұрын

@@datasciencecastnet I see. SH is efficient for representing lighting as used by diffuse (e.g. lambertian) surfaces, as you only need 4 parameters per color channel. However, I've seen their technique produce sharp reflections (the red sports car has a sharp reflection on the hood). For this to be possible using vanilla SH, you'd need an impractical number of parameters per gaussian. Are they perhaps passing the output color returned by the SH calculation through a clamp or sigmoid, so that sharp edges can form when using very high magnitude SH coefficients?

@datasciencecastnet 9 ай бұрын

@@WhiteDragon103 they use 3rd degree SH (so 16 coefficients in total), as far as I know no extra clamping or activation functions.

@quyet-65cs3buivan8 20 күн бұрын

very interesting, thank you! Can you also share the notebook code of your on this video?

@gridvid 9 ай бұрын

Can you also model, texture, light and animate than render with this tech?

@mort_brain 9 ай бұрын

So, but it doesn't use any poligonal meshes so you can't use it with other techniques, or can you?

@pajeetsingh 9 ай бұрын

High Fidelity? Fidelity?

@specyfickRC 4 ай бұрын

Can you share this code you have made for this video ?

@Mitobu1 9 ай бұрын

I wonder if this would work with a laplacian distribution rather than a gaussian 🤔

@laurenpinschannels 9 ай бұрын

what's your reasoning for why to do that?

@Mitobu1 9 ай бұрын

@@laurenpinschannels well laplacian distributions are like really narrow gaussians with long skinny tails, and is a powerful tool used in source localization. It is also prevalent in how neurons organize themselves to support sparse coding, which can be applied in pruning the neural network.