With the upcoming release of the Oculus Rift and overall increased interest in virtual reality (VR) in gaming came also the development of 3D game audio. Finally, sound starts to be recognized as a vital part of the immersion in games, especially those using virtual reality technology. Still, major focus is being put on graphics – this is where 95% of the VR development is happening but at the same time there is an avalanche of free and commercial spatialization engine plugins/libraries that implement 3d (HRTF) panning supported by real-time reverberation with simple room modelling. Examples are: 3Dception by Two Big Ears, Oculus Audio SDK by Oculus VR, Phonom by Impulsonic…you can find many more but all are very similar when it comes to features. The details lay in sound quality, performance and price. Almost all of them allow integration with major game/audio engines: Unity, Unreal (only recently), Wwise, Fmod, which brings me to the topic of this post: are such 3rd-party plugins the future of 3D audio in games?
Currently, there are two dominant game engines on the market: Unity and Unreal Engine 4. Both have very basic built-in audio engines and offer no built-in 3D audio/spatialization (note that stereo panning and distance attenuation is not 3d audio). Neither does any of the two most popular audio middleware: Wwise and Fmod (Both offer 3D audio only via paid plugins). The only free solution are the libraries/plugins i mentioned above.
Let’s look at Unity. The way all spatialization plugins for Unity work is simple: they grab audio samples in OnAudioFilterRead method in a script attached to the sound source to spatialize, process it on the native (C++) side via .dll that resides in the Plugins folder and return it back to Unity (managed code) to mix. This, however is not ideal and induces a significant overhead. However, the guys behind Unity seem to have recognized the need for 3D audio and the problem with current approach and included the Audio Spatialization SDK on their recently announced roadmap (full link here):
The feature, however, went from “on track” to “at risk”. Nevertheless it clearly means that Unity is developing a way to support those spatialization plugins instead of working on expanding their own audio engine capabilities in that area. Seems like a smart move. Why invent the wheel again? Let’s see how it looks in the case of Unreal Engine.
While Unity almost from it’s inception provides an easy way to access raw samples, Unreal didn’t offer any way to do it up until few months ago other than modifying the engine itself (it’s open source). In their release that came out few months ago they integrated the Oculus Audio SDK into the engine itself (available as a plugin) although only in the lower quality version and only on Windows 64-bit. What’s important is that while doing that, they created a way to get to raw audio samples by implementing your own audio spatialization plugin. This, however, is no where documented. It basically comes down to implementing the IAudioSpatializationPlugin and IAudioSpatializationAlgorithm interfaces, the latter one being where raw audio samples can be accessed. This is what the original source code comment says about it:
/** * IAudioSpatializationAlgorithm * * This class represents instances of a plugin that will process spatialization for a stream of audio. * Currently used to process a mono-stream through an HRTF spatialization algorithm into a stereo stream. * This algorithm contains an audio effect assigned to every VoiceId (playing sound instance). It assumes * the effect is updated in the audio engine update loop with new position information. * */
While implementing that it’s also necessary to implement own XAPO (XAudio2 custom effect). Other than that it’s an ordinary Unreal plugin (module) so it’s possible to load own .dll and process the audio there, just like in case of Unity plugin. However, the whole process seem hacky and is full of pitfails and the lack of documentation doesn’t help. Most importantly, it’s Windows 64-bit only. It seems like it’s just a byproduct of adding support for Oculus Audio library that is yet to be fully implemented and officially released, but it’s there. Also, Unreal 4 audio side is undergoing heavy redesign at this moment because of the fact that audio in the engine is implemented differently for every platform to the point where they admitted that “every platform has almost it’s own audio engine”. After they sort that out it seems very likely they will provide a way to create custom spatialization effects in a cross-platform manner the way Unity plans to and the aforementioned interfaces are a big hint.
Since Unity and Unreal are the two giants in the game industry and most of the games in the near future will be created with their use, it seems like the future of 3D audio depends on those specialized in spatialization libraries/plugins/SDKs. Hopefully, this time, the audio development won’t get halted by some artificial patents and copyrights and we won’t end up stuck with the 3D counterpart of the Dolby Digital. Most importantly, there is a chance that due to the availability of good 3d audio solutions we will finally start catching up to graphics.