[SIGGRAPH 2018] Toward Wave-based Sound Synthesis for Computer Animation

NEW 08/02/2018: We have a highlight version of the video. See here: • [SIGGRAPH 2018] [Highl...
graphics.stanford.edu/projects...
Jui-Hsien Wang, Ante Qu, Timothy R. Langlois, and Doug L. James. 2018. Toward Wave-based Sound Synthesis for Computer Animation. ACM Trans. Graph. 37, 4, Article 109 (August 2018), 16 pages. doi.org/10.1145/3197517.3201318
Abstract:
We explore an integrated approach to sound generation that supports a wide variety of physics-based simulation models and computer-animated phenomena. Targeting high-quality offline sound synthesis, we seek to resolve animation-driven sound radiation with near-field scattering and diffraction effects. The core of our approach is a sharp-interface finite-difference time-domain (FDTD) wavesolver, with a series of supporting algorithms to handle rapidly deforming and vibrating embedded interfaces arising in physics-based animation sound. Once the solver rasterizes these interfaces, it must evaluate acceleration boundary conditions (BCs) that involve mode- and phenomena-specific computations. We introduce acoustic shaders as a mechanism to abstract away these complexities, and describe a variety of implementations for computer animation: near-rigid objects with ringing and acceleration noise, deformable (finite element) models such as thin shells, bubble-based water, and virtual characters. Since time-domain wave synthesis is expensive,we only simulate pressure waves in a small region about each sound source, then estimate a far-field pressure signal. To further improve scalability beyond multi-threading, we propose a fully time-parallel sound synthesis method that is demonstrated on commodity cloud computing resources. In addition to presenting results for multiple animation phenomena (water, rigid, shells, kinematic deformers, etc.) we also propose 3D automatic dialogue replacement (3DADR) for virtual characters so that pre-recorded dialogue can include character movement, and near-field shadowing and
scattering sound effects.

[@slademcbride3225]

im going to start calling cymbals non linear thin shells from now on

[@Kombi-1]

heheheheeeeeeee

[@bruce_luo]

Luke, I am your fawawawawawawawa therrrrrrrrrrrrr.

[@ThaMentalGod2003]

Bruce Luo darth vader had a really high ping

[@Frautcres]

This is by far the most convincing solver so far. Amazing work!

[@coma-body-stilllife]

I've been waiting for someone to piece this together. All the parts of virtual sound synthesis have existed for a while. Binural spatializers, physical gas simulators and tools to interpret wave patterns as sound. These are very good results!!

[@Peacepov]

I hope this gets integrated to a game engine or a 3d software soon. Thank you all for your hard work, This's amazing!

[@totalermist]

This is *not* real-time, though. That dripping tap took almost 19 hours to render on 32 CPU cores...

[@Peacepov]

No, I mean the code/algorithm that generates the sound, it would mean you'd have to create ui for the artist to set element type and other attributes, it'll be quite technical no doubt, but so worth it.

[@BD12]

things like this NEVER make their way out, don't kid yourself hahaha. Every SIGGRAPH or MIT demonstration I've ever seen was just for these guys to wank over while they do their thesis

[@Reversed82]

it seems more like it's meant to accentuate foley work on animated movies or something similar, however it might be possible to pre-render convolution impulses for a game and use that in real time applications instead, at least for some use-cases

[@tempname8263]

Someday it will be. But not in this decade.

[@mada1241]

Cant wait for this to be implemented into gaming. But I will wait.

[@Nerdule]

Woah, this really blows all the previous sound-synthesis work I've seen out of the water. Congratulations!

[@8BitEggplant3]

These videos are cool and all and I'm amazed by the work that's gone in to all of these techniques but this is the first time a siggraph demonstration has really made me question my grasp on reality.

[@UghZan11]

3:33 Is that a cat on the reflection on trumpet?

[@peteblac1]

Brilliantly conceived and executed. Where science meets art requiring indepth of visual and auditory modalities. Kudos.

[@1ucasvb]

REALLY excellent! Great work!

[@maulcs]

I've imagined something like this for awhile now, crazy to see it for real

[@Sl4yerkid]

3:33 When the plunger went in front of the trumpet the first time (just to show the animation) my brain was automatically changing the sound I heard... When I watched the clip again without looking at the screen this time, I heard it as it should sound. very interesting

[@stanleyyyyyyyyyyy]

This is what I call excellent understanding of world around us. Great job guys!

[@thecanadianwombat8486]

This kind of stuff is really cool, it gets even crazier when you think about it in the context of things like video game application.

[@hypersonicmonkeybrains3418]

This is really awesome! we need the bucket over head sound fx for the next elder scrolls game.

[@muzikermammoth3995]

Acoustic shaders sounds incredible!

[@tomshepperd3535]

Simulating compression waves in a virtual space to generate real-world organic sounds? Incredible.

[@gamecity7265]

What an amazing future you are buildind

[@MsJeffreyF]

This is incredible, great job

[@RmaNYouTube]

Why the hell this is not available for sound designers/Visual Artists/Musicians to use?!!!! The World Needs It.

[@totty2524]

Oh my god, this is amazing!

[@alejandroz1606]

Outstanding work!

[@Malakyte-Studio]

Very interesting. Great work. I wish to see the results of this development applied to sound for automotive (loudspeakers playing music in a complex cockpit).

[@MelloCello7]

Absolutely incredible!!

[@JasonSmithDuhmeister]

Really great work. Keep it up!

[@Collinoeight]

Wow. Excellent work.

[@lucabluewaterfall]

I've been wondering whether this is possible with current technology for ages!! Amazing

[@unlogik6895]

Wow this technology is awesome. I have the vision in 20 years its normal to use it in videogames and interactive video game movies.

[@risist4502]

Oh god... I was watching another siggraph video while at the same time doing something else. It was late at night, that said it was quite quiet. And suddenly i hear those sounds. I was sure that something is happening with my stomach. Really realistic sounds.

[@bananartista]

I want this in my DAW

[@selftransforming5768]

Woah amazing!

[@maychan26]

This is extraordinary...!!!

[@JonesCrimson]

For anyone unfamiliar with latin or how professional study papers are written, "Et Al" means "And Others." So, it is researcher Langlois and others being cited, implying that more than one person was deeply involved in or helped write the paper.

[@boriswilsoncreations]

Thanks. I was wondering about that for a moment

[@MooseY17]

Love the space odyssey trumpet :D

[@SpaghettiToaster]

It's called "also sprach Zarathustra".

[@Yizak]

Okay that is amazing

[@brainsanitation]

I Noticed that the fan blocks the voices and maybe reflects it but doesn't seem to "chop" the breath carrying the voice like it would in this dimension.

[@rudnfehdgus]

This is amazing....

[@Serij92]

Amazing!

[@0hate9]

Damn, I REALLY want this.

[@xanthirudha]

AMAZING

[@francisconascimento7447]

This is just perfection. Why is this not implemented in games? Or is it?

[@XIIF]

we need this technology.. integrated sound with 3d applications.

[@coma-body-stilllife]

Creating VSTi instruments will be a sure way to monetize this research when render times reach near RealTime.

[@gloverelaxis]

You absolutely don't need real-time rendering for this to be totally revolutionary for recorded music.

[@coma-body-stilllife]

​@@gloverelaxis ok

[@VideosBySimon]

man these 3d research papers are the most surreal shit ive ever seen

[@BarnacleButtock]

Does this system have any accounting or calculations based on position of the microphone

[@xirustam]

I knew this is possible, but probably takes too many resources for being useful nowadays. However, it's good to know that the algo already exists.

[@olivecool]

wait how do they make the examples and is the software free

[@DanielShealey]

This is amazing. I've been wondering when we would be able to truly "render" sound for a while now. How long does it typically take to output some of these demonstration files?

[@totalermist]

Selected numbers: • Dripping Faucet: duration 8.5s; 18.6 hours render time on 32 CPU cores • Bowl and Speaker: duration 9s; 45 min on 320 CPU cores • Trumpet: duration 11s; 33 min render time on 640 CPU cores Source: graphics.stanford.edu/projects/wavesolver/assets/wavesolver2018_opt.pdf pg.10 Table 1

[@RussianPunchProductions]

just to make sure I get this: a) this synthesizes sound in real time from thin air depending on materials, physics & force or b) this takes 3d sound sources (mono sound) and propagates them physically correct according to nearby objects with set materials?

[@HowardCShawIII]

Well, sort of. It simultaneously performs the sound synthesis and simulates the *effects* of a 3D environment on the vibration of air in its volume, which kind of incorporates a and b at the same time, but more. Hence the comments on the pitch shift of the spalling bowl being due to near field effects - that part was not a result of synthesizing the sound of the bowl, but of the synthesized sound *interacting with itself* due to reflections off the bowl and the floor. Adding the 3D sound sources to that is as simple as simulating a speaker cone vibrating in response to that data (exactly as happens in the real world - speaker just works by wiggling a cone back and forth in response to the data). Very cool stuff.

[@AraiKay]

Can someone make a music out the the sounds in this video?

[@alexhein7583]

THIS WILL BE REVOLUTIONARY FOR MUSIC PRODUCTION. I am imagining a VST synthesizer where you can model sounds from real life objects!! Or create virtual objects in a 3d space, than generate sounds produced by hitting them, blowing on them, etc.

[@alexhein7583]

Could even lead eventually to an accurate guitar emulator. Guitar sample instruments dont come close to the real thing but this could change that

[@Patrick73787]

IS THIS THE AUDIO VERSION OF RAY TRACING???!

[@sharonpakk]

insaaaneee

[@Berniebud]

We need this shit in games

[@Noone-of-your-Business]

So... the processed voice and trumpet are... what? Recorded sounds or completely synthetic?

[@porksmash]

They were both pre-recorded sounds processed by this system

[@gloverelaxis]

This is absolutely fucking groundbreaking.

[@boriswilsoncreations]

How do I get this? Seriously xD

[@DanielShealey]

It makes me wonder where this tech could take us in the worlds of simulation. Material sciences, engineering, product design, medical research even?

[@coma-body-stilllife]

You could literally say that about any novel technology. Why even say something like that. ugh....

[@1.4142]

Relatable

[@junipiter4689]

4:34 inspiration for psycho pass by Xavier wulf

[@user-on9nc9bt5n]

すげー魔法みたい

[@moth.monster]

yall this shit sounds moist

[@pianojay5146]

acoustic and asthetic

[@draeath]

"This DOI cannot be found in the DOI System"

[@sandersmcmillan5388]

Wowwww

[@AMR-bf8nx]

Maybe Nvidia can create a new soundcard using this technology with advanced AI for producing near real time synthesized sound, like today they are doing with raytracing with the RTX series. That would open a whole new world of opportunities in the music industry.

[@user-yt1co4yt6h]

And after all of this, we have among us

[@sabrango]

DAMM

[@17MetaRidley]

Any chance of coming to softwares like blender? Is it possible that it is already applied in games of the 9th generation?

[@Dr.W.Krueger]

This isn't for games, blendlet.

[@17MetaRidley]

​@@Dr.W.KruegerHum... Not yet. But, How longe? 😅

[@userou-ig1ze]

sad this is targeted at offline synthesis. Next step would be an ANN approach to do this in f*ing-seconds? @FellowScalars: is this published yet?! Link?!?

[@AnteQu]

See the project webpage in the video description: graphics.stanford.edu/projects/wavesolver/ . The page contains a low-res and a high-res paper that you can download.

[@userou-ig1ze]

Ante Qu i meant the link to the paper for the online method...

[@wigwagstudios2474]

1:31

[@user-to2tt3qx5w]

fucking wow, im baffled

[@ldbpictures7212]

500th like

[@jayjoonprod]

WTF you guys created a world following our physics inside a computer Only if someday computers get really really fast

[@pencrows]

The legos sound kinda soft

[@AnityEx]

now simulate two drums and a cymbal falling from a cliff

[@nic12344]

It's not : "Luke, I am your father" But rather : "No, I am your father"

[@mattmexor2882]

Nicholas R.M. We don't need no stinkin' badges

[@nic12344]

MattMexor2 You're gonna need a bigger boat!

[@SpaghettiToaster]

@@nic12344 That's a lot of fish!

[@boriswilsoncreations]

Mandela effect

[@Unreissued]

fuck im high

[@idot3331]

2:59 *_B_*

[@twister5752]

🅱️

[@red9317]

Archeologists they like bones and ancient civilisations, archaeologists!

[@HarmoniChris]

My man. World Doctors is hilarious.

[@red9317]

@@HarmoniChris I just noticed the character model in the video hahah.

[@daanhoek1818]

Now i totally believe we could be living inside a simulation

[@Quaz-jinx]

Reddit?

[@Veptis]

The metal sheet and bowl were great. Cymbals not at all.

[@d3tach3d]

this is mind blowing. Raytracing for Light and now this for sound!? holy shit

[@vodkacannon]

😄

[@OrangeC7]

1:40 Because this is how physics works

[@littlesnowflakepunk855]

It actually kinda is. It's animated rigidly to demonstrate the change in pitch and timbre when bending a vibrating sheet of metal.

[@AshLordCurry]

wOw

[@explosu]

Wat.

[@HarmoniChris]

2:30 Arch-ae-ol-o-gists they like bones, and Ancient civilizations arch-ae-ol-o-gists (And one of them's gay)

[@Slvrbuu]

No! I am your father.

[@CariagaXIII]

i wish i can ABCD in a barrel

[@blueberry1c2]

a *B* c (d)

[@tmcchamp8200]

I can imagine a time where video games will have real time sound simualtions This would require a lot of computing but would save companies a lot of money cuz they save on sound recording and stuff… Maybe idk

[@guy3nder529]

well the cymbal was kinda disappointing.

[@twister5752]

🅱️

[@iLikeTheUDK]

Bye bye foley people?

[@totalermist]

Unlikely - it requires longer to render the sounds than for a Foley artist to create the sounds and a sound engineer to mix it.

[@DanielShealey]

totalermist ... "For now" I really think it won't take that long for this sort of thing to become commonplace in production (5-7 yrs) Not as a replacement but at least as an aide. CAD doesn't replace engineers and architects. Surely as with any other tech, the processing time will drop significantly the more people use it.

[@totalermist]

Daniel Shealey - I wasn't too sure about "Surely as with any other tech, the processing time will drop significantly"-remark so I went to check what happened in terms of processing time during the last 5 years. I took the Intel Xeon E5-2640, a mid-range 6 to 8 core 90ish W data centre CPU as an example to estimate what happened to processing power in the past 5 years. The model went from 6 cores at 2.5 GHz in its first incarnation to 8 cores at 2.4 GHz in its current version. Performance went up from 9500 Pts [1] to 15331 Pts [2] for an increase of *62%* in about 5 years (there is no direct successor and the somewhat similar Xeon Gold 5115 yields no significant performance gains). If we just take these 62% and round them up to 100% we get from 24 hours using 36 CPU cores *down to 12 hours* processing time for the _10 second metal sheet shake_ sound simulation in the next years [3]. Now I don't know about you, but I'd like to see a production company that saves time and money by having a pretty beefy server system render for half a day instead of just letting a Foley guy/gal shake a metal sheet for half a minute and have a sound engineer mix it... [1] bit.ly/2GUTqnL [2] bit.ly/2IKQHTM [3] graphics.stanford.edu/projects/wavesolver/assets/wavesolver2018_opt.pdf

[@DanielShealey]

totalermist Sorry, I wasn't clear. I meant this type of rendering will get faster in the future. Software will get more efficient. Hopefully they'll find a way to change over to something like a GPU based. Othwise it would have been a pretty useless venture to develop in the first place. For making sounds simulate water... Yeah, come on. That's nonsense for now. It will probably be first used in product design. 3D modeling high end speaker systems and simulated binaural coustic design for expensive vehicles. One I could see right away is acoustically engineering auditory "dead" spaces into architecture. Placing and testing baffles to create quiet spaces. Sound effects for movies is still a long way away. Civil engineering for putting buildings next to highways with solutions other than "a giant concrete wall"

[@DanielShealey]

totalermist but I agree. Rendering the sound of metal sheets alone would be kind of silly. But the render time of a 3D square as a proof of concept was probbaly met with the same eyerolls until Pixar came along. The people making this seem like they have a little more in mind that these few proof of concepts.