How to View 3D Gaussian Splatting Scenes in Unity

In this tutorial, I show you how to import 3D Gaussian Splatting scenes in to Unity and view them in real time. From there, you can add post processing, effects, VR support, or wherever your imagination and skill take you. The first part of the video covers how to import the 3D Gaussian Splatting model. In the second half of the video, I show you how to add post processing effects like color grading.
Episode Links
GitHub Repo: github.com/aras-p/UnityGaussi...
Blog post comparing detail levels: aras-p.info/blog/2023/09/13/M...
Getting started with 3D Gaussian Splatting (learn to make your first scene):
• Getting Started With 3...
Please follow my channel for advanced tips and more informational videos on computer vision!
Follow me on LinkedIn: / jonathans. .
Follow me on Twitter: / jonstephens85

[@raddles]

got this working in VR and scanned a few of the rooms in my house! really impressive how good some of the models can look that way!

[@thenerfguru]

I'd love to see some videos of your VR captures!

[@raddles]

​@@thenerfguru my personal models are pretty spotty due to still having an 8gb gpu but I'm planning to post a vid soon with it also in the premade models

[@Because_Reasons]

Can you create a tutorial please? :) Definitely excited to try in VR.

[@smtabatabaie]

That's great! did you run it in Quest? And can I ask which plug-in did you use? you used Oculus or OpenXR for this? because Oculus doesn't allow Graphics API Direct3d12 and the 3dgs project doesn't run in Direct3d11, so I tried OpenXR but got a bunch of errors.

[@raddles]

@@smtabatabaie i used openxr in unity, their default vr toolkit setup and just made sure to follow a tutorial for getting a vr environment working. Had to sort through a few things but it was working pretty easily, just very laggy on my setup

[@Pohakoo]

literally needed this exact tutorial 5 hours ago. This was uploaded 3 hours ago. Amazing timing, great tutorial

[@thenerfguru]

Dang! I wished I had a Delorean so I could go back in time and post sooner!

[@charse09]

This is so awesome!! Thanks for introducing this researches and making this video! It is a great help for me :D

[@thenerfguru]

Glad it was helpful! Follow Aras for more amazing work.

[@c016smith52]

DUDE I'm so excited to watch this!! Thank yoU!

[@thenerfguru]

Yes!!! I just hope people can make cooler things than me. You can now mix meshes with Splats in 1 scene.

[@shoobidyboop8634]

Great content, keep up the great work!

[@thenerfguru]

Thanks!

[@jag24x]

Great video again!

[@thenerfguru]

Thank you! I felt like I botched this one when I messed up the post processing volume.

[@felixgeen6543]

Thank you for this Splat Guru

[@PhilippePerruchot]

Great Video and good lesson to use Unity.

[@user-yw1lj5et9p]

thank you very much

[@thenerfguru]

You’re welcome

[@unitycoder]

nice! btw. usually Post Processing layer is added into camera gameobject (but that scene didn't seem to have any camera yet, so it created camera on that gaussplat. gameobject itself) so the easy way would be adding Camera gameobject into hierarcy first and assign post process layer there. the post process volume can be added to any gameobject when that global checkbox is enabled. And one more note, this is using Standard Rendering Pipeline. (in case someone creates project with Universal Render Pipeline or High Def. Render Pipeline, this project might not work and post processing is built-in there)

[@thenerfguru]

Thank you! Aras Pranckevičius, the plugin creator, also reached out telling me essentially the same thing. This just shows that I am not all that familiar with Unity.

Thanks Jonathan for another great video, I am more into UE stuff. I am looking forward to see the upcoming tutorials about integrating gaussian splatting into UE5! 🚀

[@thenerfguru]

I see there is a $99 plugin on the marketplace now. I am holding out, I am sure someone will hack a free solution.

Thanks for the heads up @@thenerfguru I will also hold out! :) Hopefully not for long...

[@r.d.x7403]

Hey, Johnathan. Just wanted to send my thanks. I’m doing research on integration of Radiance fields in BIM/CIM softwares. Needless to say, your videos have helped me quite a bit. (Btw, screw unity)

[@thenerfguru]

Ha! Literally, the Unity announcement came at the WRONG time. Oh well. Would live to hear more about how you feel this fits in the BIM/CIM stack. I talked about NeRFs and Geospatial at GeoWeek last year.

[@visaginas360]

Maybe you have come across information on how you can transfer footage from a 360 camera to a point cloud with a 360 camera

[@chrisclark9203]

Thanks a bunch for the workflow. Everyone will love you for a UE5 workflow though :D

[@thenerfguru]

There is now a $99 plugin for UE5. I haven't tried it yet.

[@AnonymousMF961]

Thank you for the video, what is your recommendation for the best drone for 3D Gaussian Splatting?

[@davidsudd]

thanks so much for this vid! I'm looking to go down the 3DGS rabbit hole (have done a ton of photogrammetry and some nerfs) and have a test scan of an interior I'm looking to get into VR using 3DGS - I have more experience with UE but could go with Unity if it's the better path for this - in your experience is on better than the other with current implementation (UE vs Unity). I know the UE plugin is using niagra with particles but from what I've seen looks good. Also I scanned with Lidar plus bracketed photos so I have a great camera calibration and alignment (in Reality Capture) and would love to keep those intrinsics/extrinsics which have already solved instead of relying on colmap. I've run through some conversions for nerf from RC but have you played with similar workflows for GS? Thanks - awesome work!

[@liomiglio2674]

Hey, Jonathan. I have an important question. I was training images for a few hours, however because I was doing another render at the same it crashed. Is it possible to start training from where it last trained or does it have to train from the very beginning?

[@feetpics]

This was very helpful thank you, I got it running on my quest 3, it was very laggy but it worked. Is it possible to build it onto the headset because I encounter many errors when I attempt it?

[@cosmicfissure924]

I want to express my gratitude to the KZbin channel for its fantastic tutorials. Is there any way to reduce the size of the generated .ply files? Additionally, I'd like to better understand the difference between .ply and .splat files, as, following the tutorial, we mostly end up with .ply files. Thanks in advance for your help!

[@MaxKamrani]

good work. thanks. keep it up. I am following your progress. one question for you : do you know if we can merge multiple 3D gaussian scene to make a bigger one ? ( and same question for Nerf)

[@thenerfguru]

Possible, I’m not sure how to do it thought. Block NeRF stitched together multiple NeRFs.

[@sihlemakalima]

Great tutorial! I ant to know how can I import gaussian splatts into another unity project that I'm already working on...

[@S41L0R]

i heard about this plugin! i can't wait till theres an unreal one too

[@thenerfguru]

Hold onto your papers. It exists!

[@thenerfguru]

Also, I know how to use UE5 way easier than Unity

[@grilledwaffle]

Im sure its possible but me being unexperienced with programming/coding and an amature in Unity, I wish I could upload one of my Gaussian Splats into VRChat but Vrchat's World SDK is only for Unity version 2019.4.31f1, I tried to get the gaussian splat unity asset to work in it, but the window doesnt show up in the tools tab, and trying to get the vrcsdk working in Unity version used in this video, I cant get the VRChat world splash screen opened to upload it...

[@attilagergely3646]

Hey im pretty new to this. Is it also possible to do it in a 3D VR project?

[@justindaniels5923]

How many pictures does it take to create the GaussianSplattingScene? Also, great videos, thank you!

[@user-dq2xn9fu7y]

Do you think using GS models of trees and plants in architectural rendering is possible of you have a whole garden design?

[@user-zl9gl9cy3l]

hi bro. How much data is needed for the model to perform well like bicycle ? and i have a 2 minute video from which i need to choose what fps per second. please give me your advice

[@Pananl2three]

Thanks for your tutorial!! Could 3D Gaussian splatting interact with light in the scene, and could it be captured by a reflection probe?

[@alteredcarbon3853]

I have a question about these gaussian splatting scenes. Can you do modular environments with them? By that I mean can you isolate a part of the scene like a door or a car and reuse it or recombine it in a different way in another scene just like we do with photgrammetry scenes if you get what I mean? Or is the result totaly static with no creative freedom?

[@thenerfguru]

You could mix and match as long as it's all in the same coordinate space. You could achieve it in this Unity environment.

[@DREAMSJPEG]

Hey Nerf Guru, thank you for the detailed tutorials that you've provided. Do you know if there is a method / tutorial demonstrating the creation of gaussian splatting from point cloud data like .las?

[@thenerfguru]

I do not. In fact, the main reason for a point cloud is for an initial set of points in space to start optimizing the gaussians. The whole process can be achieved without a point cloud too. What you really need is a sparse set of images with their known camera poses.

[@DREAMSJPEG]

@@thenerfguru thanks for clarifying - makes sense. It's a shame bc the company I work for has lots of survey / point cloud data

[@RiccaDiego]

Fantastic tutorial! But when I try to import the Gaussian splat in the tools menu, there is an error that I can't find anywhere on the internet: "the output folder must be within the project, it was "C:/User...". No matter what I do, this error keeps appearing. I'm using Unity version 2022.3.21f1.

[@mariomaffiol]

Que componentes de PC tienes para estos renders ?

[@dylancullen1604]

we didn't see the assests folder in your git repo. can you help us?

[@basilaldwin]

it doesn't seem to work what version of the release are you using?

[@fabianr9394]

How does it convert images to spatial data such that it creates an "navigable enclosed world"? Is it also possible with some python plugin by any chance?

[@thenerfguru]

Do you mean this? kzbin.info/www/bejne/i4nXppydrsSqj8U

[@Thats_Cool_Jack]

Curious if this is possible with unity 2019 and vrchat...

[@thenerfguru]

Not sure. I guess you can give it a try?

[@elvinshoolbraid84]

I'm trying to use models from Luma, but the tool is saying "PLY vertex size mismatch, expected 248 but file has 27" and I'm not sure what's up. Do you know what might be the issue?

[@thenerfguru]

Not sure. I don’t think you can actually use the PLY file from Luma. Download the Gaussian Splat from the Unreal Engine section and that should work.

[@visaginas360]

Maybe you have come across information on how you can transfer footage from a 360 camera to a point cloud with a 360 camera

[@carlosnavarro-cn]

Any thoughts on how to get this working with URP or HDRP?

[@thenerfguru]

No idea. I wish I knew more about how Unity works.

[@edwardvalleramon3631]

hola mencionas que en el min 3:44 que hay que convertir en formato gaussiano, o si no ver tu tutorial para principiante podrias dejar el link, asimismo no se si lo que intentado esta bien intente de mi nube de puntos trabajada de una via llevarla con formato de extencion .ply para cargarla en unity pero me sale error supongo que asi sea en extencion .ply aun debe de transfromarse a gaussiano, espero puedas apoyar por favor , saludos desde peru

[@Zvezdan88]

Good tutorial! I need to check the UE5 plugin tho

[@thenerfguru]

I’ll try to bust out a video tomorrow night.

[@Zvezdan88]

I found one plugin on the marketplace, but it's $100, which sounds unreasonably expensive. @@thenerfguru

[@OriginRow]

@@thenerfguru 🕺🕺

[@marverickbin]

The plugin on tools -> gaussian splatting -> create gaussian splatting assets, has not the option input folder, only input ply. And just by importing the ply is not working... Just renders a blank scene.

[@Because_Reasons]

I imagine exporting into Unreal 5 would be epic too.

[@b4rtmod]

Sadly for me for some reason, when i try to import any ply (by obviously using input ), I got this error PLY Vertex size mismatch, expected 248 but the fas has 27 I tried with a few PLY files that are available on GitHub and all of them give the same error, mostly showing different numbers!

[@alejandrocapra8654]

same error

[@b4rtmod]

@@alejandrocapra8654 My problem was solved; I was trying to import the input.PLY, and in fact, you need to open the point_cloud folder and then choose the file you want to open inside the folders (iteration_7000 or iteration_3000, that will load the model correctly without that error

[@2aalanantunes]

In my case, when I open the tools tab in Unity, instead of asking to import the folder, it asks to import the .ply file and I opened the first input.ply is an error, in fact the qrfile that should be opened, is in the point_cloud\iteration_30000 folder point_cloud.ply file - the manual Next up, create some GaussianSplat assets: open Tools -> Gaussian Splats -> Create GaussianSplatAsset menu within Unity. In the dialog, point Input PLY File to your Gaussian Splat file (note that it has to be a gaussian splat PLY file, not some other PLY file. E.g. in the official paper models, the correct files are under point_cloud/iteration_*/point_cloud.ply).

[@feetpics]

I found a fix, if its in a zip folder but you have the right file type, you have to extract it first.

[@alpaykasal2902]

Maybe someone answered about your camera problems already... but I see your main camera was a child of the splat model... that means your main camera was inheriting all transforms from the splat's axis (which may have been wonky). If you keep the main camera at the top level, and set the position and rotation transforms to 0,0,0 things should behave a lot better.

[@neilbradley9035]

Can this be done in Unreal as well???

[@thenerfguru]

There is now a $99 plugin for UE5. I haven't tried it yet.

[@Josh-hl8jl]

haha cool to see my uni in here

[@visaginas360]

Can you try walking into a scene with virtual reality goggles on?

[@psaicon0]

please add your pc configs somewhere in your channel or videos! It's really frustrating trying something only to realize it's not possible on my machine

[@thenerfguru]

Sorry to hear that! I’ll follow your recommendation and do that.

[@davide.ercolano]

Can the scene be built and exported as stand alone software?

[@thenerfguru]

I assume so. I believe someone did this as a Gaussian Splatting VR Viewer.

[@cheya3758]

Hi, isn't that the .ply model? Is it possible to convert it to usd/usdz?

[@thenerfguru]

The ply point cloud is just a starting point for where the Gaussian splats render. I foresee in the future, this visualization gets enveloped into the USD umbrella. It’s too early though.

[@adanmontelongo8014]

excelent video, UE5 please

[@thenerfguru]

There is now a $99 plugin for UE5. I haven't tried it yet.

[@prodhab_]

Is this possible in Blender?

[@thenerfguru]

Yes, it looks like Alex Carlier is working on it: x.com/alexcarliera/status/1698769174950985787?s=20

[@alvydasjokubauskas2587]

Original author removed assets from GitHub, sadly I cannot reproduce steps mentioned

[@alvydasjokubauskas2587]

Oh I found he moved assets inside project folder

[@thenerfguru]

Good to know!

[@EconaelGaming]

Bad timing, lol But it's exciting to see that tech in engines. Maybe Unreal Engine though, or Godot

[@thenerfguru]

It’s now available on UE5 too.

[@EconaelGaming]

@@thenerfguruThat was quick!!

[@EconaelGaming]

Can you please link it?

[@torusx8564]

On my way to create VRCHAT world with this level of graphics wish me luck

[@basilaldwin]

if you figured it out lmk

[@28Negrov]

Hi! I've been following your progress in the Nerf topic for several months now. You are Elon Musk! But no matter how hard I try, I can't find an answer to my question about the interaction of 3d programs with Nerf. I know that I can unload a grid or a point cloud from Nerf, but I can't unload the dynamics of reflections, because these are obviously different environments.. I want to upload scenes from Nerf to a blender, for example, while maintaining the dynamics of reflections when moving the camera inside the blender engine. Tell me please, do you think it will be possible in the near future? Now I have to match cameras inside Nerf and inside blender, working with masks in Nuke to achieve approximate results

[@28Negrov]

Is it working here? In Unity? Oo

[@thenerfguru]

Most likely you should look into TurboNeRF if you are interested in Blender. Nerfstudio and Luma AI both have Unreal Engine formats too.

[@visualstoryteller6158]

If unity, then fr unreal too i m hopeful ❤.. please try a solution for unreal engine.

[@thenerfguru]

I saw a plugin released today for UE5, I’ll check it out.

[@visualstoryteller6158]

@@thenerfguru waiting 🙌

[@randall.chamberlain]

This video just cost you $0.20 per click. Sincerely, the Unity Gestapo

[@thenerfguru]

Ha! Sad day... I really hope this all gets resolved in a way that is beneficial for the developers.

[@Relaxing137]

we should all switch to unreal

[@thenerfguru]

There is a $99 plugin for Unreal.

[@lordofthe6string]

I use Unity every day, and I'll just say you run into stupid rendering problems like you did with the camera flipping all the time. I recently had it so that the camera would only render 1 single column of pixels but smear them horizontally UNLESS the resolution was perfectly 6:9... only in the editor though, if I built the game it worked perfectly.

[@thenerfguru]

Ugh! Maddening! I wished I knew Unity better. Seems fickle at times.

[@impactsuit9871]

Bad timing...

[@thenerfguru]

I know 🙄

[@importon]

Unity is a 4 letter word these days

[@thenerfguru]

The timing couldn’t be any worse for this video. Haha

[@OriginRow]

Yeah. I will be blissful if it is available for Unreal Engine

[@MistereXMachina]

@@OriginRow it is through LumaAI. Not gaussian splattering, but NeRF at least

[@user-ik5kv3fd8k]

Shit crap cash grab