3D Gaussian Splatting! - Computerphile

A new technique to turn pictures of a scene into a 3D model is quick, easy and doesn't require that much compute power! Dr Mike Pound and PhD student Lewis Stuart demo and explain.
Lewis used this Particle simulation in Unity: GitHub - keijiro/SplatVFX: github.com/keijiro/SplatVFX
NeRFStudio is here : docs.nerf.studio/index.html
Previous (nerf) video: • NERFs (No, not that ki...
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharanblog.com
Thank you to Jane Street for their support of this channel. Learn more: www.janestreet.com

[@yyanri]

I love this two person interview format. It’s so dynamic.

[@crack64]

Flows much better than the cameraman - guest format. Having the cameraman intervene here every now and again also worked very well!

[@teekanne15]

Also gives credit to the young scientific staff, that is often the power house in academia.

[@jones1618]

"Posh Point Cloud" is the best best short description of Gaussian Splatting I've heard.

[@MichaelOfRohan]

Posh, pish... whats a few percent in taxes between friends?

[@Cl0udWolf]

What I like most about Gaussian splattering is that it is basically a non neural network approach that is competitive if not better than neural network approaches.

[@a_soulspark]

CloudWolf! you should try making a gaussian splat rendering datapack for Minecraft

[@DanieleCapellini]

I mean, it still uses gradient descent for optimization, which makes it pretty close to neural networks

[@Cl0udWolf]

@@DanieleCapellini gradient descent does not feature back propagation or connected layers and it doesn’t learn it just is a method for solving a problem that is not solvable analytically.

[@andrepascoa6687]

But it does do backpropagation with the differentiable rasterizer@@Cl0udWolf

[@DanieleCapellini]

@@Cl0udWolf I wonder what method neural networks use to "learn" optimal changes of parameters based on the gradients they've calculated during backpropagation.

[@enijar]

Loved the Pound particles haha

[@HereticB]

I love how in traditional computer science nerd fashion whenever either of them speaks the other one just looks extremely awkward, like they don't know what to do with themselves. love the new format btw seriously, do more of these please

[@luicecifer]

Ah, great. Now I have "I don't know what to do with myself" of the White Stripes playing on loop in my head.

[@codycast]

lol. Settle down man.

[@Eagle3302PL]

It's not just awkward nerd waiting, when smart people like this talk shop they are constantly thinking through what's being said, so it's also a display of intent listening and conceiving follow up statements or questions. E.g. in your typical talk show with charismatic people, no one actually listens to what the other person says, they just pretend to pay attention while waiting for their turn to say something pre baked or to latch on to a fragment of a sentence said by the other party. But here we see both of them mulling over what the other says and then trying to fill in any gaps they think need filling. It's a much deeper level of engagement in conversation, that's why they focus their gaze on e.g. their hands or the table, to keep themselves free from distraction and focus on the words.

[@NimbusAbi]

Loool

[@sirfer6969]

@@luicecifer The Hardest Button to Button...you're welcome

[@wallyhall]

Eye contact makes a lot greater difference to the video than I would have guessed. Really great content.

[@OldShatterham]

love how enthusiastic Lewis and Mike are about these techniques!

[@zzzaphod8507]

0:23 and later, Humble Pi in the background of a video released on Pi Day, well done

[@vesk4000]

This is so cool! I've been looking for a neat explanation on what actually Gaussian Splatting is for a long while now. And who better to explain it. Perfect!

[@maxmusterman3371]

Yes ❤ please more videos about techniques like this. Also reeeaally enjoyed the 'old' computer vision and neural net videos. They inspired me to pursue a carreer in this direction! thank you

[@sirfer6969]

Creating a universe of mini Mike Pounds...how unintentionally sweet

[@makebreakrepeat]

I've been trying to understand this for so long 😅 Thank you for the clear explanation!

[@cazino4]

I'm always pleasantly surprised when I see a computerfile video with Mike popup on my recommendations.

[@lpbaybee4942]

This is such a cool topic, and such a great explanation. Thank you!

[@Tony-Omega]

Gotta love that MGS clip at 6:24.

[@petergerdes1094]

Is there a reason for using gaussians rather than, say, some kind of wavelets to allow automatic compression by rounding high frequency information?

[@tripplefives1402]

The gaussians are rendered in real time as textured polygons by retail graphics hardware.

[@danzmachinz2269]

Nice! SMERFs next! Or maybe how to generate a 3dgs point cloud from only 2 or 3 cameras?

[@bnolsen]

We used this splatting to do Ortho rectification for line sensors 20 years ago.

[@Ceelvain]

I've got two questions: When we split a gaussian, how do we avoid having them both converging to the same place and size? Is there a repulsion term in the loss function? Like, electrons repelling each other? And how do we avoid gaussians just fully transparent? Also, I don't think it would handle specular reflections or anything that depend on the position of the camera.

[@stefanoscolapasta]

In GS they optimize and adaptively densifiy/control the Gaussians in order to tackle over-reconstructed area and under-reconstructed ones. In the case of under-reconstruction, meaning that space needs to be filled, a new Gaussian is created as a clone of one in the under-reconstructed area and placed along the direction of the positional gradient. In over-recontructed areas instead, the target Gaussian is simply split and divided by a factor of 1.6 (determined experimentally). To determine the position of the obtained Gaussians from the splitting process, they perform a normal sampling based on the scaling values of the gaussian (mean centered at the Gaussian mean and Standard Dev based on the Gaussian's scaling parameters). They do matrix multiplication on these samples with the original rotation matrix of the GS on which splitting is been performed, in order to align them with the current spatial orientation of the initial one. Subsequently, they use these sampled values as an offset from the original mean to determine the 2 new means for each couple of newly generated Gaussian obtained from the splitting process on the initial Gaussians it was applied to.

[@stefanoscolapasta]

Fully transparent Gaussians are simply pruned as they do not contribute to the final rasterized image (more recent papers adopt smarter pruning techniques). Regarding specular reflections, in this video they did not go into detail, but the 3D Gaussians are basically 3D ellipses, and on the surface of and Ellipse you can define Sferical Harmonic functions, these are used to represent color on the surface of the Gaussians as a function of the viewing direction.

[@deltamico]

​​@@stefanoscolapastathat's neat, so the 3d gaussians fill out the same-collored borders of the objects and then each gaussian is given a view dependant function of color?

[@vibaj16]

@@stefanoscolapasta they look like 2D ellipses (as they have to be for traditional rasterization)

[@GameDevGeeks]

I don't understand this, but from examples I've seen GS can handle reflections pretty well

[@rich1051414]

There is a certain fuzziness and artifacting that is unique to splatting which begins to be a bit of an eyesore the more you work with them. Almost like you are wearing googles with vasoline smeared in random spots. I hope we discover some way to overcome that. Maybe combining traditional and splatting and doing boolean operations to cull the splatting errors?

[@drdca8263]

Maybe applying something like a de-noising model as a post-processing layer?

[@Pixelarter]

I think most of the artifacts are due to insufficient camera angles for training. It ends up positioning some splats in mid air, in a way that work for the training angles, but no when you move around. Also, the more splats you use (and the smaller their minimal size), the more detail you can capture, and sharper the details can become. However at the expense of performance. An alternative would be to use asymmetric splats, with adjustable degrees of blur on each side of the ellipse (like angled spotlight projections). That could represent sharper edges with fewer splats. But depending how costly more complex splats are to render, it may not be worth it and be slower than rendering a larger number of symmetric gaussians.

[@rich1051414]

@@Pixelarter If the splat positioning were restrained in 3d space for each object being represented, I think it would automatically reign in the artifacting. Like a low poly textureless version of what is being rendered used as a guide to restrain the splatting being placed to within those bounds.

[@vibaj16]

That's just due to the splats being large, which is due to not enough camera angles. The more of the scene you give the algorithm (especially giving close up shots of things) the more detail (i.e. smaller splats) it can produce, but that means more effort for both you and the computer. It's probable best to use this technique for relatively small objects, and removing large splats in the sky to be replaced with a more traditional skybox technique.

[@trickster721]

It's just the opposite drawback of triangles. With triangles, a lack of detail appears as overly sharp points and edges, while in gaussians low detail creates blur instead.

[@MikeTwohey]

I want to see this as a video with sports events. I imagine flying through a sports scene in 3D while the action is in motion. How many camera angles would be needed? Would the speed be quick? I guess each frame would need to be trained and stored (weeks of training). I think each frame is tens of megabytes, so it might be too much to actually be executed. Your thoughts?

[@trickster721]

A research team has already made some short 3D videos using animated gaussians, recorded inside a dome covered in video cameras. Google "Dynamic 3D Gaussians".

[@romajimamulo]

How well does it do with perspective correct drawings and recreating their look? How many are needed?

[@stare4539]

The first game with gaussian splatting will be revolutionary.

[@TheJacklikesvideos]

you two drawing pictures of pine tree renders reminds me of the brothers drawing the trees in Myst by smearing big green cones.

[@paszTube]

Haha, there’s a NERF gun on the desk 😂

[@Exaskryz]

I have to give this a rewatch, but I feel a segment is missing. How are the gaussians generated? I acknowledge the parts of the video discussing what they are, the benefits they confer to render time, flexibility and application. But I missed how to actually produce my own gaussian burst as recommend to do of my house at the end of the video

[@trickster721]

There's still a pre-rendering step where the gaussians are generated, like in conventional photogrammetry. You feed the images into a simple neural network that builds the gaussians. It's just adding a second layer of data to the point cloud.

[@stefanoscolapasta]

Initially the images are passed through a classical Structure From Motion algorithm in COLMAP which estimates a coarse point cloud of the scene. Those points are used as initial 3D positions to place the first 3D gaussians.

[@syjwg]

What a nice explanation! We've come a long way from the ray tracing in the early, but at that time, mind-boggeling 'Wolfenstein' or 'Doom' engines.

[@TheJacklikesvideos]

ray casting*

[@vermeul1]

First time I laughed out loud on computerphile 😂

[@grggrgrgg]

0:40 running of a web server and sending the images? The data is loaded from a server yes, but it's being rendered completely locally, in the browser, no? This would not at all affect the performance

[@skf957]

Superb stuff, as always. Not going to pretend I understand everything, but that's down to my neural network not being all that I'd like it to be.

[@AlexTuduran]

Great, but the rendering of splats does not use a depth buffer. All the splats are gaussian - it means that even at 100% opacity, the edges will be transparent, eventually fading to 0 opacity. What's usually done, is that you sort all the splats and render them from back to front. That's the current pinnacle of the technique. Some opt in for sorting once every Nth frame, which greatly improves performance, but other than that, there's a high degree of overdraw. And they're not like circles - their projection may look like a fluffy cloud, but they're volumetric by all accounta. Aside the color, opacity and spherical harmonics, they have a position, a rotation and a scale - all in 3D, so they look differently depending on the angle you're looking at them.

[@nicolas.predella]

Maybe a set of depth buffers (where each iteration traverses one transparent object)? Let's say that after traversing the first object the accumulated alpha in an area is equivalent to 1, everything behind could then be skipped

[@patrickreece7323]

That requires sorting!@@nicolas.predella

[@MarcAyouni]

I was always a bit put out with NERFs, since a scene is a NN and all the problems that this entails. GS has such incredible potential.

[@nickbensema3045]

so my main question is, is this going to give us a higher quality version of the 8800 Blue Lick Road house in VR?

[@ElTrolldego]

How does Gaussian Splatting handle specular surfaces?

[@Joseph843]

It renders what is in the reflection with another set of gaussians. It's cool because it actually creates a mirror world. I saw someone zoom in through the reflection in a pool of water which was pretty trippy.

[@trickster721]

The lighting is just whatever was captured by the camera. In a sense there are no "surfaces", the only conventional 3D simulation used is the original point cloud. Adding simulated lighting will require additional developments. (Maybe photographing the subject in multiple lighting conditions, for a start. Could be as simple as abusing the camera's flash, or waving a light around as you record video.)

[@ElTrolldego]

@@trickster721advantage NeRF then, if I understood that correctly, since they can handle view dependent reflection and refraction.

[@trickster721]

​@@ElTrolldego I looked into it, and NeRF works the same way, it's just capturing existing highlights and reflections as an artifact. Versions of NeRF with dynamic lighting are either using a specialized photography setup, like I figured, or generating a conventional 3D model and lighting that. Neither technique can get correct physical material information from arbitrary photos, that data has to come from somewhere. I think the original NeRF presenters were slightly hyping up its ability to "handle" specular and reflections, it's not deliberate like that implies, it's just a side-effect. Both techniques just reproduce whatever is shown in the source images, they don't actually know the difference between a window, a mirror, and a swimming pool.

[@Chewychaca]

So basically objects become collections of colored fuzz balls instead of meshes. I feel like this solves problems in rendering foliage. I wonder if it's possible to render both kinds at the same time.

[@GuagoFruit]

Is there potential for animated/moving scenes with Gaussian splatting?

[@hannah42069]

yes, but as of now it's only for pre-recorded movement. there's a video on youtube somewhere, i think "4d gaussian splatting"

[@GuagoFruit]

@@hannah42069 oh yeah I've seen that one. It appears that it can only be recorded in a fixed space using multiple cameras, so it's not too useful in the way Gaussian splatting was intended to capture arbitrary still scenes. Not quite what I had in mind.

[@SyntheticFuture]

With static object and stiff objects it shouldn't be too hard as the gaussians can keep their shape and size. But I image it becomes really complicated once every gaussian itself has to animate as well.

[@deltamico]

One could try and find a transformation of connected subset of gaussians that when aplied and rendered minimizes the difference to an image where the transformation is applied. You have now found a joint in the object and its corresponding transformation

[@maxmusterman3371]

Like a gaussian mixture model?

[@mytelevisionisdead]

Dying of laughter here with the millions of Mike Pounds :') :')

[@euunul]

Noob question, how is this functionality different from fotogrametry meshing?

[@trickster721]

Using fuzzy colored blobs instead of triangles lets you create images using less information, in the same way that a painting can look more "correct" to our perception than a photograph. They're underselling how performant this is, the tree scene they show here probably runs about as fast as something like Minecraft on a cheap modern GPU. I'm guessing the office PC they use in the demo has integrated graphics.

[@Marco-xz7rf]

Could you please try and do this with pictures / videos of any of the moon landings? that would be so awesome. I already tried with A11 but it never worked, but there weren't that many pictures.

[@vibaj16]

probably not enough pictures for a decent reconstruction

[@Marco-xz7rf]

@@vibaj16 yeah ap11 hasn't but maybe others. Or do you know if there is a possibility to give some kind of positional and angle information with the pictures? wouldn't be easy to extract those but it should be possible to estimate some.

[@iamavataraang]

Is gaussian splatting what polycam uses?

[@wdyahnke]

If you want to play with Gaussian Splatting the iOS app Scaniverse will scan rooms and objects using this technique

[@TESRG35]

So it's basically a 3D gaussian mixture model where the number of gaussians is learned?

[@ThereIsNoRoot]

I read the original paper and am familiar with the technique. Was excited to see y'all do a video on it but this video could definitely have used better production to explain Gaussians. For example, t was 13 minutes before you showed an image of a gaussian blob.

[@xarisfil58]

There are algorithms that can extract the 3D scene from the NERF to a ply or other 3D format. Marching Cubes for example.Also Gaussian Splatting doesn't capture Phong Illumination or difficult illumination in general except you overfit the model and make really small gaussians. So photorealism with gaussian splatting is something debatable. Last but not least there are new models like F2-Nerf that are way faster that the NeRF you are using implementing 3D Multiresolution Hashencoding that helps the model converge faster. After seeing the video I tend to believe that gaussian splatting is like patching 3D shape with small bell shaped Plasticine objects to create a 3D scene that looks like the one in the image. This even defies bias that is made from camera i.e. different illumination from different angles due to light captured from camera

[@trickster721]

This is like complaining about jpeg or mp4. The point isn't photorealism, it's that it's a viable realtime technique that performs like conventional rendering on cheap hardware. Realtime optimization is all about doing things "worse".

[@francoismorin6806]

Is there a link where we can download a sample scene to navigate inside ?

[@naturallyinterested7569]

The Multiverse of Mikeness

[@DonckIT]

Impressionism in 3d 😱

[@John-pn4rt]

Has Matt Parker paid up for the product placement?

[@isi1044]

Whats the previous video?

[@sidd065]

NERFs (No, not that kind) - Computerphile Can't post a link but you can look it up.

[@Computerphile]

kzbin.info/www/bejne/rXzWoHqferN1frc - now in description as well -Sean

[@petergerdes1094]

Just saying it's doing gradient descent doesn't tell us its particularly cheap. I mean backpropogation in a nueral net is just a form of gradient descent.

[@vodiak]

I'd love to see the particle explosion but with the original gaussians instead of replacing them with a sprite.

[@RPG_ash]

Oh the excitement 😅

[@oldspammer]

In the mid 1990s, I was part of a project to create the instrumentation for an autonomous mining vehicle. It used a distance measuring instrument. This comment was made not publically visible by the "algorithm" until I made it exclude the technicalities involved. Curious that?

[@oldspammer]

Referenced things must have negative connotations?

[@oldspammer]

My user name likely has me on some blacklist?

[@oldspammer]

Maybe if I translate the original comment into Greek?

[@oldspammer]

Στα μέσα της δεκαετίας του 1990, ήμουν μέρος ενός έργου για τη δημιουργία των οργάνων για ένα αυτόνομο όχημα εξόρυξης που πήγαινε από σταθμό GPS σε σταθμό με το φορτίο ορυκτών για να τα απορρίψει. Χρησιμοποίησε ένα ορατό κόκκινο σύστημα μέτρησης απόστασης λέιζερ μέσης ισχύος που παρήγαγε χιλιάδες μετρήσεις απόστασης ανά δευτερόλεπτο ενώ τις σάρωνε χρησιμοποιώντας έναν κινητήρα άξονα με ελεγχόμενη ταχύτητα βρόχου κλειδώματος φάσης που είχε έναν καθρέφτη τοποθετημένο στον κινητήρα για να παράγει σταθερή ταχύτητα μέτρησης απόστασης λέιζερ που θα σάρωνε η σκηνή πάνω από το όχημα και σε κάθε σταθμό που επισκέπτεστε θα υπάρχει ένας γραμμικός κώδικας μέτριου μεγέθους σε μια ορισμένη απόσταση πάνω από το όχημα. Η ταχύτητα των υπολογιστών τότε δεν ήταν υπερβολικά γρήγορη, αλλά τώρα οι υπολογιστές είναι πολύ πιο γρήγοροι, ώστε όλα τα δείγματα απόστασης να μπορούν να καταγραφούν όμορφα σε μια συσκευή αποθήκευσης υπολογιστή. Το θέμα θα ήταν ότι όλα αυτά τα δεδομένα σύντομα θα γέμιζαν τις περισσότερες συσκευές αποθήκευσης, εκτός εάν χρησιμοποιήθηκε μια καλή μέθοδος συμπίεσης για την εξοικονόμηση χώρου μέσα σε τέτοιες συσκευές αποθήκευσης. Αυτός θα ήταν ένας πολύ γρήγορος τρόπος για να δημιουργήσετε έναν τρισδιάστατο χάρτη επιπέδου παιχνιδιού για χρήση σε παιχνίδια. Εάν μια περιοχή αποφευχθεί η σάρωση, θα φαινόταν στο παιχνίδι σας να έχει μια τρύπα στον χάρτη του παιχνιδιού σας σε ορισμένα σημεία. Πιθανότατα, ορισμένοι παίκτες παιχνιδιών χάκερ θα έβρισκαν έναν τρόπο να βγουν έξω από τον χάρτη σας και να πυροβολήσουν παίκτες που ήταν ακόμα μέσα στον χάρτη, ανίκανοι να ανταποκριθούν στους αντιπάλους τους που εξαπατούν. Η πρώτη μου μέθοδος ήταν να χρησιμοποιήσω έναν ψηφιακό βρόχο κλειδώματος φάσης που ήταν ελεγχόμενος από κρυστάλλους, αλλά το όχημα εξόρυξης θα υπόκειται σε πάρα πολλές προσκρούσεις στην επιφάνεια του δρόμου υψηλής ταχύτητας που τα επίπεδα κραδασμών για το κύκλωμα θα προκαλούσαν ζημιά στην κρυσταλλική αναφορά συχνότητας που είναι τοποθετημένη στην πλακέτα κυκλώματος. ο μηχανικός βρήκε μια άλλη ακριβή μέθοδο αναφοράς που βασίζεται σε αναλογικές μεθόδους που δεν έχουν ευάλωτα εξαρτήματα κυκλώματος που δεν θα καταστραφούν από το όχημα που κινείται σε ανομοιόμορφες συνθήκες οδοστρώματος σε σχετικά υψηλές ταχύτητες. Ο κινητήρας του άξονα ήταν ένας τριφασικός κινητήρας συνεχούς ρεύματος χωρίς ψήκτρες που κινούνταν από μια γέφυρα H MOSFET σε κάθε μία από τις τρεις φάσεις του. Ο κινητήρας θα παράγει τους δικούς του παλμούς μέσω των αισθητήρων φαινομένου του χωλ εντός του κινητήρα που θα μπορούσαν να χρησιμοποιηθούν ως ανάδραση στο κύκλωμα ρυθμιστή ταχύτητας κινητήρα βρόχου κλειδώματος φάσης που έλεγχε την αλληλουχία φάσης της γέφυρας H. Το μόνο που θα ήταν απαραίτητο για τη χρήση ενός τέτοιου οργάνου λέιζερ μέτρησης απόστασης θα ήταν να γείρετε το κουτί σάρωσης με λέιζερ μια μικρή σταθερή γωνία αφού λήφθηκαν αρκετά δείγματα από το παράθυρο που είχαν τοποθετήσει κύκλωμα μετριασμού της ομίχλης στο τζάμι επειδή αυτό το πράγμα επρόκειτο να λειτουργούν σε διάφορες καιρικές συνθήκες. Ο ηλικιωμένος που εργαζόταν μαζί μου ήταν ήδη 45 έως 55 ετών εκείνη την εποχή, επομένως μπορεί κάλλιστα να πέθανε στο ενδιάμεσο διάστημα, και εγώ ήμουν μια ή δύο δεκαετίες νεότερος από αυτόν. Ο χρόνος, οι πόλεμοι και οι διάφορες ασθένειες συνεχίζουν να είναι όλοι εχθροί μας. Αναφορά Wikipedia 256χρονος Κινέζος βοτανολόγος πεθαίνει το 1933 Ένα βίντεο σχετικά με την ιατρική δικαίως ισχυριζόταν ότι ΔΕΝ ήταν δουλειά του να αυξήσει σε μεγάλο βαθμό τη διάρκεια της ζωής μας, αλλά το αντίθετο και να προεδρεύει σε ένα πρόγραμμα για να διασφαλίσει ότι ο καθένας μας πεθάνει αφού περάσουν τα χρήσιμα χρόνια μας. Προφανώς αυτός ο τύπος ήταν άνθρωπος, αλλά αν διαβάσετε την περιγραφή του, ήταν περίπου 7 πόδια ύψος. Κατέληξε να έχει 20 ή περισσότερες συζύγους και πολλά παιδιά με την καθεμία, τα οποία τα έζησε όλα πολύ εύκολα. Μέχρι σήμερα, πιθανότατα έχει πολλούς απογόνους.

[@oldspammer]

The technical details were able to be translated into the target Greek language by Google translation tools. There is nothing in there of an offensive nature. Why do they do this?

[@nutzeeer]

Arent there negative gaussians? Could cut out the crescent of a crescent moon.

[@wizard7314]

Is it neural network based or not? Mentioned "training" multiple times in reference to Gaussian splatting.

[@drdca8263]

It isn’t neural network based. The training is gradient descent based.

[@halfsourlizard9319]

... it's almost like ANNs aren't the only way to create machine-learnt models! ;P

[@ZarahTaylor08]

coooorrr that lewis guy is sexy

[@jannik323]

so these are basically little particles instead of vertices that form faces

[@tripplefives1402]

Still verticies. Just a bunch of 2 triangle squares sent to the GPU where the shader draws a transparent circle on them.

[@el_es]

Oooh what if this could be coupled with the Apollonian Gasket algorithm ? Let the gaussians be circles always, and let them fill the shape?... and then in 3d...?

[@ehtishamullah7534]

Not rendering something that isn't there, makes it more useful / reliable in some applications.

[@gokusaiyan1128]

like medical ? but where can it be useful

[@barefootfred1479]

Have computer scientists reinvented the atom?

[@antivanti]

So basically they're like 3D brushstrokes or fluffy voxels... Kind of reminds me of how the graphics in "Dreams" for Playstation 4 are rendered

[@sinom]

I learned about gaussuan splatting at uni last year so I thought it was an older technique. Didn't know the paper on it was published basically only a few months before that. It definitely seems way too simple to be something only developed last year.

[@stefanoscolapasta]

It actually takes a lot of inspiration from EWA Volume Splatting from 2001.

[@Satscape]

Posts into suggestion box: "Meshtastic". Geek level 11. It's becoming quite popular in the UK, but quite quiet (English is awesome!) in Nottingham. It's geeky, it involves ESP32 computers and Chirp modulation and witchcraft?. I still don't understand it, so need an aduilt to explain it to me. Thank you.

[@jounalansman1769]

Ending is gold haa

[@__meo__]

Matt Parkers book in a research office... Sure... :)

[@wouldntyaliktono]

For those of you wondering how people end up at companies like Meta or Spotify, it's oftentimes dudes like this. The handful of people out of a PhD program in computer science who just LOVE writing splatting APIs in Rust or Go.

[@erickmarin6147]

This will easily get to the limit if perceivable graphics once you have hardware made for representing gaussians in a screen

[@charlieangkor8649]

I don't think overfitting means that.

[@georgesos]

That was an introduction, but i d like to see some rendered examples.

[@jondo7680]

100fps? Well wait until consumers demand 1024 times the density of blobs.

[@busterdafydd3096]

8:22 Lewis should of drawn bigger, and used more pages.

[@kenziemckenzie-bennett5399]

humble pi!

[@cetyl2626]

Humble pi

[@sirMishka828]

I was interested in photogrammetry while ago, amazing how neural networks are doing it

[@Aeduo]

How long until someone brings this in to Garry's mod?

[@xizar0rg]

This seems like fuzzy voxels.

[@NotSlaya49]

Even if the tech tech behind them is really, REALLY cool, this ultimately boils down to just the 3D equivalent of pre-rendered backgrounds like the old Resident Evils and point & click games used-- Almost completely static and unreactive to change. Even worse, you're limited to what you can get pictures or video of, which would be pretty artistically limiting. So, I wouldn't really count on Gaussian Splats making their way into games in any significant way.

[@trickster721]

That's just a limitation of the way the gaussians are currently generated, there's no reason they can't have more lighting information. But yeah, it's for rendering 3D photos, not simulated worlds. Maybe someday.

[@NotSlaya49]

Yeah, you could always include more information to support dynamic lighting, but I meant static in the sense that nothing moves or can be meaningfully interacted with. Granted, that's kind of just a problem with *a lot* of fancy graphics innovations (tech demos of unmoving environments always looking 100x better than anything that actually gets shipped), but still. The 3D photos are cool either way.

[@stefanoscolapasta]

@@NotSlaya49 Research on 3DGS is booming right now, recent papers bind 3D Gaussians to the surface of a mesh which is used as a reference for dynamic scenes. I see the future as being a hybrid classic triangle meshes + 3D gaussians together.

[@MrBluelightzero]

I feel like you glossed over the details a bit too much.

[@mcmormus]

Men I don’t understand much but by the excitement of them I got very excited too 🤣

[@krz9000]

I prefer meshes

[@BrutalStrike2]

Put Mike to the background to get more views 😂

[@erickmarin6147]

A grandient can be descended in any way but backpropagation is what makes ai

[@erickmarin6147]

It doesn't learn because it's not deep

[@deltamico]

There is a community at r/GaussianSplatting, come and discuss ^-^

[@Henrix1998]

I feel like NERF has more potential, this will just hit "pixel" count limit. Pairing NERF with general image generator could work nicely.

[@deltamico]

what is the pixel count limit?

[@SteveMacSticky]

Due to my huge daily consumption of blitter chips, gaussian splatting occurs frequently whenever I use the toilet

[@skeleton_craftGaming]

Smh

[@Ceelvain]

It's so simple! I could probably code it in an afternoon.

[@D-K-C]

Ъ.

[@paulsutherland3813]

I feel like you cloned your Mike Pound gaussian

[@paulsutherland3813]

Oh i wrote that just before all the Mike Pound gaussians appeared

[@BigJonYT]

I think you can think of nickel plated steel as three resistors in parallel. You have a pure nickel resistor on top, then a steel resistor in parallel, and then another nickel resistor on the bottom. So, the total resistance will be a combination of the resistance of the plating and the core material. Some current will run through the surface plate and some through the core material.

[@phutureproof]

lots of freedee objects, about free years ago

[@marsovac]

Even if you increase the number of gaussians by two, or even three orders of magnitude, this would still look like crap compared to traditional rendering. And at that point it would also be unusably slower. Ofc it is not used for rendering, but for generating from images, but still...

[@Aeduo]

Maybe could be used for skyboxes or distant/LOD stuff or maybe specific objects in the world that might look better rendered this way. But yeah probably not a replacement. This kinda reminds me of demos (of mostly vaporware) we've been seeing for years though, of these world models that you can zoom in to and it's made of ... something. I wonder if this is that or some similar technology.

[@Pixelarter]

I particularly think the result of gaussian splatting is better than traditional render. It breaks down more elegantly with blur, while with traditional rendering you get unnatural polygons and pixelation.

[@stefanoscolapasta]

I believe 3DGS is more in line with what reality is in the actual physical world. It is matter of time before we'll see dynamic 3DGS LODs.

[@mikelenox7999]

This is the dumbest I've ever felt.

[@GCAGATGAGTTAGCAAGA]

I'd rather compare gaussian splatter with voxels, than triangles. But in general - cool video.

[@etilworg]

So, this is how they spy inside buildings

[@gsestream]

so why do you need a neural network, you could just use a linear algorithm. cant you figure out how to make it. photogrammetry with depth. its called a 3d model, not a neural network. k.i.s.s.

[@UltimatePerfection]

Too bad this won't really be useful in games, only in stuff like archvis. I would love to have tech that allows for hyperrealistic levels, but that ain't it sadly.

[@stefanoscolapasta]

Why do you think it won't be useful in games?

[@UltimatePerfection]

@@stefanoscolapasta Too big storage/processing requirements for too little gain (realistic look can be achieved much cheaper with stuff like Unreal's Nanite).

[@casperghst42]

Raytracing, it was a rally big thing back in the 90s and early 00s.

[@browut6666]

Woosh

[@TheJacklikesvideos]

soon we'll be designing procedurally generated open world games rendered with this technique and stable diffusion, with live light and particle and physics simulations. and people will still choose to play ugly hand crafted DOOM wads and The Elder Scrolls 3. edit: it's me, i'm people.

[@Lion_McLionhead]

Looks horrendous compared to openai sora

[@drdca8263]

It isn’t for the same task. Sora produces videos from a prompt and some other guidance, and presumably takes a lot of compute. This is used to render in real time motion through actual recorded scenes. The use-cases are nearly disjoint.