3D Gaussian Splatting for Real-Time Radiance Field Rendering

SIGGRAPH 2023
(ACM Transactions on Graphics)
----------------------------------------------------
repo-sam.inria.fr/fungraph/3d...
Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates.
We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (≥ 100 fps) novel-view synthesis at 1080p resolution.
First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

[@sitkinator]

Now this is epic.

[@shottysteve]

What a time to be alive!

[@bilyyj]

Oh my God. I'm shaking right now

[@jajajinks1569]

Wtf this is like pure magic. Amazing work, can’t wait to see it in action!!

[@NiallHorn]

Really impressive, congrats.

[@artispeedy]

This is insane. Good work.

[@chucktrier]

Wow Awesome work looks stunning! I would love to do a Vulkan implementation.

[@oliviercampagne1726]

can we say that is just amazing !!

[@ShaharHarshuv]

I don't understand any of the technical terms but that looks like dark magic to me. This is like, good enough to use in high end films now.

[@Instant_Nerf]

The future will be representation. I wonder if there is enough viewing angles on smartphones for real time representation.. or maybe a sort of setup with multiple cameras? Or maybe the LiDAR on an iPhone has enough depth for representation ?

[@zodchiy3d]

Awesome! I want to see those scenes in VR! 🥽

[@c016smith52]

Me too, I hope this is something that could port into something on Quest, via Unity/VR or otherwise!

[@MonsterJuiced]

What an incredible achievement, how do I get my hands on this? :o

[@hubertwojniak5541]

That's amazing! Good job : O

[@deadbeat_genius_daydreamer]

C'est magnifique

[@samsoulee]

The future of photography 😱

[@mr.critic]

Mind blowing

[@saltmuffinLGDPS]

every damn day we get closer to a hype realistic optimized vr game

[@akiray1999]

amazing

[@WhiteDragon103]

How are non-lambertian (i.e. viewpoint dependent) effects, like fresnel, specular, etc. achieved with this approach? It appears as though the technique struggles to represent glass (e.g. the reflections and transparency of the windscreen as shown in the project page). I suppose one approach may be to optimize spherical harmonics coefficients which are later pruned after training, though this may result in really bad ringing. Or you could have a "cone of influence" (cone orientation quaternion, anisotropic cone fov) that defines the opacity of a given gaussian depending on the viewing angle. Unfortunately this would likely massively increase the number of gaussians you'd need for certain surfaces, and you may need to know how many to allocate for these surfaces ahead of time for initialization. Third, you could have a very sparse voxel grid of learned functions that converts the viewing angle from the gaussian center position, and the gaussian center position within the voxel cell, to rgba. This may be the fastest, as you'd only need to evaluate this once per gaussian per frame, not per pixel.

[@arsenurano4164]

It's written in the paper, sh coefficients is optimized during training too

[@tehwayniac]

This is amazing! Since it's able to render orders of magnitude faster, is it also able to capture moving objects in the camera view?

[@dan_obie]

I'm interested in this as well. I imagine that the moving object would ultimately be considered noise and be culled. What happens to leaves and tree branches as they move during image capture?

[@AlexTuduran]

The future of gaming just got set.

[@Fiwiipe]

Waiting for a tutorial for how to setup 😍

[@gridvid]

Waiting for this rendering tech to be available in Blender, Unreal and Unity 😊

[@trollenz]

I've seen plugins for unreal and unity

[@theishiopian68]

Every day we get closer to the holodeck

[@greg.skvortsov]

Just imagine the future: an instant 3D models mrom a mobile phone camera!

[@CornuDev]

Amazing

[@naninano8813]

finally something that is easy to interpret and to work with. no idea how come they are able tor render novel views that fast. if every scene has >100k splats, I would pre-render them into bilboard sprites, but how come reflections are so good?

[@LeePenkman]

The colours are stored with some extra view dependant properties which I think helps

[@naninano8813]

@@LeePenkman I since found out that one of splat's parameters (also learned) are spherical harmonic coefficients. and there is a marvelous breakdown of all the math in "Spherical Harmonic Lighting: The Gritty Details". easy read. still reading it.

[@adhirajdeshmukh6813]

@@naninano8813 Thanks for referring this.

[@TESRG35]

1. Wouldn't it be more accurate to compare 3dgs to nerf when both are either initialized at random or both with sfm points? 2. Since this method is based on rendering and not ray tracing, how can it do such a good job at modeling the sun reflecting of a shiny object like the countertop only at certain angles?

[@v000000000000v]

any comparisons with mobilenerf?

can we generate the point cloud instead of capture it from real life? i'm thinking of game development use cases

[@TedHolmwood]

Will this be available for public/professional use soon?

[@VitaliyHAN]

wow

[@lunaletuna]

I wonder if you could say train an ai on Gaussian splatters of animals that come from the input of dna sequences if you could get it to output mammoths

[@Universaa]

it will certainly be intriguing.

[@SonicEclipse456]

Cyberpunk 2077 brain dance irl

[@ElQaheryProductions]

what gets optimized during backprop?

[@elfferich1212]

gaussians

[@andybrice2711]

Essentially: A cloud of blurry ellipses. Their dimensions and rotations, and what colour they appear from different viewing angles.

[@ElQaheryProductions]

@@andybrice2711 thanks!

[@kelligwamtwa3958]

🤯

[@homarslamson]

in the 90s

[@Dragonblood94]

Streetview is about to become the biggest game map ever

[@SexualChocolate]

Can anyone say unlimited detail?

[@FrankHarwald]

Problem: neither PSNR nor plain SSIM, while very easy to compute, are particular good measures for subjective image similarities - someone better run these tests with more meaningful metrics like MS-SSIM & VMAF (which on top of being more close to human perception of visual differences are also used in many existing image quality, restoration & compression tests).

[@Seercho]

Deckard: Enhance...

[@ewerybody]

What's a gaussian? 😐

[@andybrice2711]

In this context: Basically a blurry egg-shape floating in space.

[@drdca8263]

A Gaussian is the bell curve shape (or a 2D or 3D or etc. version of this)

[@JibbSmartGD]

What a time to be alive!