Soundscapes – The Future of Virtual Surround

This is part three of a three part series, earlier we looked at the future of sound and music.

Understanding the subtleties of how the brain perceives sound, and the real world concerns, such as what speakers or headphones you’re using,  are the obstacles we have to overcome before we can achieve perfect virtual surround. Given that this is a field where research and development is still being done I don’t expect much if anything to show up on the console side that players and game developers could use. It will have to be done on a game-per-game basis as technology evolves.

Perfect surround over headphones

The most seamless surround experience can be had over headphones. It makes sense since headphones are basically speakers covering your ears. The only thing you need to do is send a signal which emulates what sound should sound like when it comes from different angles. This is called binaural techniques or HRTF (Head Related Transfer Functions). You can experience perfect surround today via binaural recordings which are field recordings done with a dummy head with two microphones in the ears. The effectiveness of binaural recordings varies based on the shape of your head, ears and headphones you use, but when it works it creates a very convincing and compelling effect. Whether you’re using in-ear, on ear or around the ear headphones has an effect on how good “imaging” you’ll experience. Around the ear headphones play sound that interacts with your outer ear, while in-ear headphones bypasses that part of your hearing entirely. This is why around the ear headphones can produce a larger more spacious sound than in-ear headphones do. The clip below is a binaural recording and requires headphones to sound as intended. You should notice that the piano will sound as if it being played far away from you in a way you might not have experienced from any music recording before. How realistic the effect appears to you will depend on how similar your head is to the person doing the binaural recording, as they are done with two microphones put into the ears of a head, and on the headphones you use.

One big difference between headphones and speakers is the “super stereo effect” with headphones. If you listen to speakers both your ears are hearing what’s coming out of both speakers. With headphones this doesn’t happen, so if something plays only in the right channel your left ear will not hear it at all. This effect can quickly cause listener fatigue as your brain isn’t used to this effect. In the real world you don’t hear sounds like these and, in order for virtual headphone surround to work, the effect of the sound arriving at the other ear has to be simulated. Depending on how far apart your ears are, for example based on the size of your head the delay of the sound first hitting your right ear and then arriving at the left needs to be different. The sound arriving at the other ear needs to be equalized to remove the frequencies that your head attenuates.

There are scientists working on solving all these issues right now, so the bigger problem will be how to find out what kind of head the player has so the effect can be calibrated effectively. Either the player could simply tell the game just how large the diameter of his/her head is, or there could be 3 or 5 preset head sizes to pick from in the headphone mode options. That would probably be enough to create a compelling surround effect for 99% of players and it would make the playing experience more comfortable since the super stereo effect would be reduced or removed entirely. The German audio company SPL created a video showing off the functions of their Phonitor headphone amplifier which tries to solve some of these issues for audio professionals. I haven’t seen any other video which demonstrates the problems with headphone listening, and the solution to it, as clearly as they do.

Apart from SPL, other companies such as Dolby, DTS, SRS, and Sonic Emotion are working on ways to improve headphone listening. These are just a few of the ones I know that are actively working on new technology right now. Other, such as Blue Ripple Audio, have well working solutions you can use in games today, but it’s unclear what kind of development we can expect from them in the future. Either way headphone surround is a field that is getting a lot of attention from many players so it’s inevitable that we will see great advanced made in the next few years.

Rapture3D, which is used by all Codemasters games on PC since Dirt 2, has different headphone surround pre-sets based on scientific measurements of heads. That they are categorized by color however isn’t very helpful as you have no idea if the head is similar to your own.

Creating phantom speakers

Virtual surround over speakers is different from headphones. There are standards for speaker placement, but many people aren’t aware of them. Placing speakers correctly is critical for good virtual surround and current technologies such as Dolby Virtual Speaker assume the listener has placed them according to the standards. But if the user isn’t aware of the standards or isn’t able to place the speakers in their perfect place the algorithms fall apart. If the virtual surround was handled by the game itself there should be a calibration wizard with which the game learns from the player just how the speakers are placed. This way the virtual surround technology would make correct assumptions of how to best create the illusion of a 360 degree sound field around him/her. The calibration sequence would have the player place the speakers in a 3D space representing his/her room and at the same time sound effects would fly around the sound field so the player could in real time hear how well the effect is working. It should also appear during the installation of the game so to not be overly intrusive and it would have an interactive element to it so the player could move around sounds manually to play with the 3D sound effects.

Virtual speaker surround might not sound as interesting to most players as headphone surround does, but when done correctly speaker surround can be very convincing. It’s possible to do without adding odd artifacts to the sound that would make you want to shut it off. The best virtual speaker technology I’ve heard is by Creative Labs with their CMSS Virtual Speaker technology. The next video was intended to be a comparison between EAX on and off in Mafia, but the person who made it had CMSS Virtual Speaker enabled so it’s a good demonstration of that as well. When EAX and the virtual speaker technology kicks in halfway through notice how the car drives off into the right and behind the listener. When the player also turns slightly to the right before entering the lobby the gramophone pans from left to rear left.

Combining the sound tech, new creative music and a huge virtual sound field will make games of today look hopelessly old fashioned and stale in comparison. Photo realistic graphics will require orders of magnitude more computing power than our gaming machines have now. Being able to create sound indistinguishable from real life is much easier and at least as important to the overall experience as the visuals are. Right now games still have things to learn from movies when it comes to dynamic range and music, but the potential for game audio is to be better than anything movies could ever hope to accomplish. The point at which games are able to sound more compelling than any movie could, or even more compelling than the real world itself, might happen sooner than we think.

Remember to read the earlier parts of the series looking at the future of sound and music.