Sound Propagation With Bidirectional Path Tracing | Two Minute Papers

[@syk0saje]

"This is ice cream for my ears." *immediately imagines Karoly smooshing two cones of ice cream into his ears*

[@livingcorpse5664]

I hope not with the sharp point ends first.

[@SomeRandomPiggo]

i know this is an old video but this is absolutely incredible

[@tamasperci4168]

My guess is that the audio samples themselves should be as raw as possible for this to achieve optimal effect. Recordings usually don't isolate the effects of the environment from the sound. If there is a way to record the sounds in a room with walls that simply absorb the sound, I bet that they would fit very well into these engines, even if the raw recordings would sound very unnatural. I wonder how much the microphones themselves affect the recording, though.

[@L.L.2045]

interesting point

[@luiztomikawa]

The Microsoft thingy about sounds might be boosting this vídeo on the algorithm! What a time to be alive!

[@Kram1032]

nice! Although that first test scene was a bit too busy to my ears. Hard to judge the quality with all of that going on at the same time. I thought I might have heard some background noise in it but that may just have been the source playing back water sounds. Really the main problem with such techniques, if you want to add them to games, is that those might seem like rather quick times but that neglects the potentially many other things that have to be computed at the same time for a game to work. Still, I'd love games to have such proper full 3D sound.

[@ciarantaaffe5259]

I downloaded the demo, only to discover that the audio is compressed to AAC @ 317kbps. This youtube video has audio encoded with opus(251). This demo does not demonstrate the true potential of this technology. Despite this problem with the demo, the idea is probably excellent, and I can't wait for an improvement in the mostly neglected audio aspect of games and media.

[@Kram1032]

​@@ciarantaaffe5259 You're talking about something that's four years old. (Dang time flies) I really hope we're gonna get stuff like this but there would've been lots of opportunities by now

[@KillahMate]

@@Kram1032 As best as I can tell, Sony's Tempest audio engine developed for Playstation 5 (when used alongside the PS5's ray tracing capabilities) is designed to accelerate exactly this kind of audio processing. I know that Mark Cerny, the designer of the PS5, has certainly been talking about ray traced audio for years now - in fact he talks about that more than he does about ray traced graphics. So it looks like we're only just now starting to utilize these techniques properly in the commercial space. It's not too surprising, after all ray tracing has been a thing for decades but real time ray tracing has been a thing for just two years.

[@Kram1032]

@@KillahMate as far as I understand it, even lowish sample raytracing works pretty well with audio. Of course the more samples the better the results, but compared to the adhoc methods people otherwise use, even low sample stuff already sounds like a huge improvement. That said I'm sure RTX has changed the equation there and I'm very glad that this finally gets some traction

[@007LvB]

​ @Kram1032 What if you calculate the sound contribution every 4th frame and then linearly interpolate between them, or something?

[@Napoleonic_S]

what games uses all these real time sound propagation tech? I feel audio is too neglected in games... come to think about it, these tech should be able to be fully implemented for walking simulators.

[@jamegumb7298]

Halflife 1 already had something like it, if a bit rudimentary. AMD also has it on their gpu's but no game so far uses it, also the amount of soubnd sources at 1 time is a bit limited so far which does not help.

[@backtoklondike]

@@jamegumb7298 You are talking about trueaudio if I'm not mistaken. It was actually used in some games like Thief 2014, Murdered Soul suspect and Lichdom: Battlemage. But after that, the support was dropped. I've tried out how it worked in Thief and if was very poorly implemented, like it had a very weird reverb and didn't sound natural at all. Not like how it was in this video.

[@jamegumb7298]

@@backtoklondike Halflife had a completely software solution, not AMD dependent. Games after that had TrueAudio at times, but it was "gpu accelerated" using a certain not too userfriendly plugin and dedicated dsp on the gpu. It ddi not work great. AMD knew. So they came with TrueAudio Next what I was referring to, which is not dependent upon a dedicated dsp and engine plugin, instead it is a devkit on gpuopen and it is 100% gpu accelerated, no seperate dsp. It came out in either 2016 or 2017, so not the one you tested. Now that works properly if used properly. Problem is with many sources the latency goes up (not too much but still) and even though Nvidia can use it they prefer their own solution, as always which does not aid adoption. Second issue is a 100% loaded gpu leaves little room for TrueAudio Next, which is why an integrated gpu should be standard, to offload positional audio in games, in addition to other stuff such as ssh on gpu, rsa on gpu, encoding for a stream while gaming, and a load of other stuff. That or a 2nd graphics card.

[@PersonausdemAll]

​@@backtoklondikeThief 1, 2 and 3 where better 🤫

[@jamesking2439]

This would be amazing in a creepy horror game.

[@jetso2000]

It's in alien isolation I think

[@AceDeclan]

@@jetso2000 can you confirm this?

[@jetso2000]

@@AceDeclan bruh you can literally google it

[@teroblepuns]

This is why I'm here

[@PersonausdemAll]

​@@jetso2000Google it for us pleeeeaaasee

[@vladkostin7557]

I was pretty sure you can pre-compute this same way you precompute dynamic GI

[@DasAntiNaziBroetchen]

I remember there being an old paper about precomputing this. The problem likely is that you won't be able to move sound sources but just turn them on and off.

[@hellboy6507]

Man I’ve been trying to find examples of this for weeks. All I’ve found was a Half Life 2 mod. Wish there were more examples, I would love to try this out myself.

[@ViRiXDreamcore]

You can do it in Unity and Unreal with Steam Audio.

[@identiticrisis]

​@@ViRiXDreamcorethanks for this, I wasn't aware of any accessible third party API for these techniques. It could be fun to try some effects and finally implement some project ideas!

[@NeoShameMan]

Want to see voxel cone tracing and baking adapted to that lol and put in unity3d ;)

[@TwoMinutePapers]

Right - and there should definitely be an episode on VCT soon! :)

[@johnsherfey3675]

Two Minute Papers what about voxle cone sound?

[@bwiebertram]

We need this for minecraft cave update

[@livingcorpse5664]

I wanna see ray tracing (both visual and sound) used for a Godzilla game. A game based off of a film series known for it's visual and sound effects deserves better. Heck I can see this changing the music, anime and movie industry as well, not just games.

[@theomann1534]

How would this have anything to do with the movie industry?

[@peterlewis2178]

​@@theomann1534 Well, for animated movies it could potentially work, depending on how immersive you actually want it to be though. Though for regular movies, to get a similar effect, I think the technique of using two microphones to simulate both ears is critically underused, and it could be interesting (like in this video: kzbin.info/www/bejne/f4anhZ-slsydgKM)

[@GreenMagic0]

What is the castle 3D model in this video? Is it available for free somewhere??? Anything?

[@dorfischer]

Has this research field advanced yet since then? Has anyone ever combined this with an air-turbulence simulator to add high-frequency richness to the sound?

[@Rhodochrone]

I love this kind of stuff, though I'm curious about the rattly sound of the audio. It's akin to when sound is slowed down by adding tiny repeats to retain pitch. Is it the audio equivalent of raytracer noise? Or maybe it's the reverb?

[@armenmichaeli6704]

Vision is said to be the primary human sense, or so I have heard. I personally find sense of hearing to be very important in computer reality simulations like many games aspire to do, and I am sure there are many others who feel the same way. Yet, sound has often been a second grade citizen in at least games (which can be said to primarily drive evolution of rendering solutions). Probably because if we start to dedicate as much effort to calculate sound correctly, development teams will have to spend a far larger budget of their time and money for that, meaning that someone will have to do the R&D upfront and give it or sell it afterwards in the least. Then there is also the factor of having enough computational resources -- strangely and in contrast with what happened to at least the PC graphics rendering hardware since the VGA adapter, the modern sound card is a rather cheap and simple piece of silicone that has not seen much love since the days of Sound Blaster, Gravis Ultrasound and Aureal Vortex sound cards which aspired to do for sound what cards like Nvidia GeForce and AMD Radeon do for graphics. And it's a shame because the market ought to be there, or at least get there soon. Which brings me to another interesting point which has to do with this video of course: sound "ray" tracing (because sound isn't really rays). Now, what is typically referred to with "ray tracing" is special application of wave propagation calculations to light "wave-particles" (because light behaves as both, as we know from school). Meaning that sound behaves in a similar way -- waves of sound originate from sources (typically called light sources with light, unsurprisingly), carry multiple frequencies (color with light rays/waves), refract, reflect, diffuse and do pretty much all the things waves, including light waves, do. The point is that much of what we have learned about ray tracing and ways to optimize it with cards like Nvidia RTX etc, can be applied to sound calculation, with equally impressive results, I believe. We should totally be looking into it. Sound is good :)

[@tylergubala7355]

Not to discredit your statement, but I believe "graphics" processors are becoming general function enough to augment traditional sound cards in this way; no need for dedicated hardware for this.

[@Aqua_1014]

This is fantastic stuff!

[@DobryakDobreyshiy]

Wonder why RTX still not got such kind of 3D Audio extension. This could change gaming even more dramatically.

[@michaelemouse1]

So, it would sound like the Vault 1080 mod for Fallout 4 with hopefully less overdone reverberations?

[@Hinge45]

have searched for this thing for months... can't believe i found this on two minutes of paper

[@FireOccator]

Are we going back to buying separate sound cards again?

[@iviecarp]

fun fact, I think the engine used in the simulation you showed is Godot - might be wrong but that particular castle hallway is the same they use for their engine feature demonstrations

[@satoshinakamoto171]

where do u find these papers ?

[@sopwafel]

I just found and very like your videos

[@anarki777]

Eventually this technology will be used in video gaming. We have the theoretical knowledge, we just need the hardware capabilities.

[@dominicstocker5144]

Check out John Lin‘s stuff, you might like it

[@noxim_]

And we never get this in games

[@Kram1032]

"high echo" isn't really what this is about. You can fake that in principle quite easily. Though having walls properly occlude or funnel full 3D sound is tough. For instance, in many games, if you use headphone, you can position yourself such that any given sound source is only fully audible on one ear, because they simply take the direct sound direction into account, not reflections. It really sucks when that happens. In this work, and others like it, you will not have such problems.

[@Kram1032]

I'd say it's not so much your hearing as it is a bad example - in particular that first one is way to busy. The second example is clearer with respect that's actually happening though. But there have been better examples given to other works.

[@aliazzam626]

We'll get it in PlayStation 5

[@ksonsouchou]

@@aliazzam626 nope its in XSX. kzbin.info/www/bejne/pnrdqKJjgt12eZo PS5 uses its own tech

[@DasAntiNaziBroetchen]

Look at Mr. Game Engineer over here being all smart and shit.

[@Andytlp]

audio is so behind graphics its not funny. Sound can set a better atmosphere than graphics.

[@kcsi1]

The diffraction and the nonlinearity are neglected. Maybe these are not so important.

[@jnfran92]

They are pretty important, actually. The problem is that a wave-based simulation needs a huge computing power and memory capacity in order to simulate large spaces at high frequecy!

[@GETNAKD]

So, this is what Mark Cerny told on the interview with Wired.

[@arcadealchemist]

this is one of the MOST important parts of most games, SOME games have AMAZING audio without this because the staff they have on audio know how to SHAPE sound properly but it's the same as bad lighting in games bad audio is the most common issue in games like espcially when it's the ONE thing that ruins a game, you can have stunning graphics then the audio is half baked erven tho their "Audio team" claim to be pushing amazing audio, ACCORDING TO WHO? i bet you guys have gaming headphones and a budget DAC for your home PC.

[@mortarbackmusic8511]

Rainbow 6 Siege does realistic sound propagation similar to this system.

[@sabrango]

dam this is what expensive cinema softs performs, even better

[@DIO.S_STANDAH]

Shape of sound. - without that , it usless.

[@DIO.S_STANDAH]

Listen a bee around ear and feel how air touching all space . Angle of sound can be isomorphic

[@techpriest4787]

Welp. Now we have RT hardware to run it faster though.

[@Luka1180]

How far away are we from achieving such sound synthesis to the point where it would be possible to 3D model any animal’s vocal organs, send a sound through them and have the resulting sound that is produced at the other end be the accurate phoneme that would result in the real world when sound+air goes through such vocal organs in a certain configuration? Say I 3D model an accurate model of the human vocal organ, or a chimpanzee’s perhaps, and assign the accurate materials too to it. And it is a model with movable parts like in real life. I move the parts to make an “O” sound. Sending the audio through the geometrical shapes, I’d then get the O sound. How far away do you think we are from something like this? It is my dream to utilise such technology to actually produce alien vocal organs and mouths for the worldbuilding in my science fiction books. To figure out what sounds they could make so that I can make constructed languages that fit with that for them. As well as figuring out what kind of organs they’d need, of what shapes, if I wanted them to make certain pre-chosen sounds.

[@SethizFTW]

Rainbow six siege needs this tech. Amirite

[@muckily]

was just looking at a different one from Microsoft www.microsoft.com/en-us/research/uploads/prod/2016/10/PrecomputedWaveSimSoundPropagation.pdf www.microsoft.com/en-us/research/project/project-triton/#!publications Pretty interesting

[@warsin8641]

RAY TRACING :O

[@vanessavenkov6668]

Immersive as fuck

[@32BitJunkie]

Whoa it's weird to hear him like this... i do like the added depth on the topic and less forced enthusiasm, but this isn't a very time efficient video compared to his newer stuff

[@Caleb-fv5fp]

My aureal vortex is good enough

[@grenoblica]

A3D sounds better still

[@ADRIANO782]

great vid but "I'll see you next time" was not said correctly. 0/10