Duality audio upgrade

[ChristianGreiner]

Here are some ideas (brainstorming) how we could improve the audio features of Duality.

1. Support Multiple SoundListeners at Once
See #482

2. Adjust the volume via Gizmo
Set the min. / max. distance of the audio emitter via scene editor like scaling an object:

3. Cone Angles

the cone angles (inner and outer) specify how "sharp" the direction is. if inner and outer are 180, the sound is the same from all directions. if inner is 45, and outer is like 60, you'd have the full audio when the direction points at the listener within 45°, no (or minimum) audio at above 60°, and fading between both inbetween those angles
by Adam

4. Audio Effects
Add audio effects to a sound emitter (or sound instance ?)

• Low Pass
• High Pass
• Echo
• Distortion
• Reverb

5. Audio Areas
Define audio areas (like shape of rigidbody ?) so the audio emitter gets the same volume everywhere in this area.

So do u have further proposals and ideas?

[mfep]

Had a look into OpenTK regards the audio system. Points 3 and 4 can be solved and implemented with ease in the current system. Let's see what they have:

• OpenTK has support for cone angles for audio sources. This functionality can be exposed to AudioInstance and AudioEmitter as well.
• There seems to be placeholders for highpass and bandpass filters, but according to the comments, these are not implemented. Perhaps a newer OpenTK/OpelAL version has them.

There are a number of effects supported by OpenTK. The full list seems to be:

• reverb
• chorus
• distortion
• echo
• flanger
• frequency shifter
• vocal morpher (a formant filter I guess)
• pitch shifter
• ring modulator
• autowah
• compressor
• equalizer

From this list, the ones @ChristianGreiner mentioned are more useful for gamedev purposes.
On the front end, I'd propose a struct owned by the SoundEmitter and SoundInstance objects along the following lines (horrible pseudocode). Obviously any of the sub-structs being null'd mean that the particular effect is off.

struct AudioEffectSettings {
  struct ReverbSettings = null;
  struct EchoSettings = { DelayTime = 5.0f, /* other parameters of echo */ }
  // other effect types
}

There is yet another consideration: if my understanding is correct, OpenTK can only apply these effects/filters on its internal sound sources. At the moment each of the Duality sound sources are mapped to a OpenTK sound source. If #482 was implemented somehow, these per-source effects wouldn't be available anymore.

[mfep]

Here is a proposal of a possible course of implementation:

1. Idea 3. (cone angles) and 4. (effects) can be implemented in an additive manner (not breaking compatibility) exposing the corresponding OpenTk functionality. As described above.
2. Idea 2. (volume gizmo) is more of a cosmetic/editor feature, and thus quite a separate business. It would make a nice standalone Good-First-Issue entry as well.
3. Idea 1. (multiple sound listeners) and 5. (audio area) are more advanced audio features, and are more difficult to implement without breaking the API. Also, if I get it right, it's not a very urgent issue, thus maybe the better way of doing this would be an external plugin / experiment based on community effort. Also, with that, this advanced audio functionality would be completely opt-in.

Details of the audio plugin:

• Many game engines have the concept of audio channels and mixers. Each audio source can be assigned to a mixer channel, and effects can be applied per-channel. Volume and pan can be directly controlled per-source, is indirectly controlled by the source's world position and also affected by the settings of the channel the source is assigned to.
• Currently in Duality, every sound source is assigned to one native audio instance. The plugin would work differently, implementing it's own SoundInstance, SoundEmitter (SoundInstance wrapped in a Component) and SoundListener classes. Each of the SoundInstances would be assigned to a mixer channel, and each mixer channel would be assigned to a native audio instance.
• The implication of the former is that the distance-based volume calculation and intensity falloff needed to be implemented plugin-side, because the OpenTk instances would be used for a different purpose. This is quite a task, also it's a question whether managed code is performant enough for this sort of number-crunching.

Any input on this?

[ilexp]

Idea 3. (cone angles) and 4. (effects) can be implemented in an additive manner (not breaking compatibility) exposing the corresponding OpenTk functionality. As described above.
Idea 2. (volume gizmo) is more of a cosmetic/editor feature, and thus quite a separate business. It would make a nice standalone Good-First-Issue entry as well.

Agreed on both points. However, implementing the whole effects range seems like something to be used primarily in scenarios where you'd also require or use a more complex channel / mixer setup - so I'd defer this part for now.

Details of the audio plugin:

Sounds very neat, and yes, that would likely be a software implementation that really works on the actual audio samples manually - or at least I'm unaware of any matching OpenAL functionality. As such, this would be quite a bit of work, and should be treated separately from the other points.

One thought I'd like to add here as well: Right now, we have good control over rendering sources and targets - it's just a few lines to render a camera onto a texture, or scene A into texture A and scene B into texture B. When approaching channels and mixers, it would be nice to have the same functionality for audio as well, the ability to "render audio into a texture" and to control where audio output ends up, and what to do with it. The concept of an output channel could be the key ingredient to this.

[mfep]

However, implementing the whole effects range seems like something to be used primarily in scenarios where you'd also require or use a more complex channel / mixer setup - so I'd defer this part for now.

So my point was not to actually implement the effects, but to expose the OpenTk effects. These use a similar pattern to the currently exposed Filter, thus from this far it looks like a typing exercise (i.e. simple, not requiring structural changes). But, obviously, for that, we need to touch the Core and the AudioBackend. Like the currently present lowpass filter, these effects could be assigned per-SoundInstance.

This is a pre-requisite for the future implementation of the audio plugin, which would use the Core's SoundInstances as the backend for its channel constructs.

it would be nice to have the same functionality for audio as well, the ability to "render audio into a texture" and to control where audio output ends up

It reads like some sort of routing option for the channels. That means, that a channel's output can be another channel's input or the master channel (practically the application's audio output). We can go far with this... 🙂

[ilexp]

So my point was not to actually implement the effects, but to expose the OpenTk effects. These use a similar pattern to the currently exposed Filter, thus from this far it looks like a typing exercise (i.e. simple, not requiring structural changes). But, obviously, for that, we need to touch the Core and the AudioBackend. Like the currently present lowpass filter, these effects could be assigned per-SoundInstance.

Alright, makes sense.

If we really support a bigger range of effects, we need to be careful figuring out the API we expose for that, as I'd like to follow an "open set" approach here, where we might, at some point, allow users to define and implement their own effects in software and use them just the same as the builtin ones.

It's a bit different from the settings struct approach, but I'm not sure we should really stuff every possible effect layer into one big item anyway - currently I'm thinking something along the lines of a list of effect instances that are applied in sequence. For OpenAL native effects, they'd just be carrying parameters along to pass over.

This is a pre-requisite for the future implementation of the audio plugin, which would use the Core's SoundInstances as the backend for its channel constructs.

Not necessarily - I think it might make sense to actually depend the experimental audio plugin directly on the audio backend, with no detour through nested SoundInstances, especially since that plugin would define its own instances and emitters anyway. Working directly on the audio backend also allows, for example, directly streaming software-generated samples to OpenAL, which would be in line with what I think would be required for custom channel / mixer audio processing.

It reads like some sort of routing option for the channels. That means, that a channel's output can be another channel's input or the master channel (practically the application's audio output). We can go far with this... 🙂

Yes! Exactly. It would also allow to contain a scene's audio output within a scene's own global channel, like it can already be done with rendering a scene's graphics output to a texture - almost a natural extension of the self-contained scenes idea from #504.

We'd need to figure out what exactly a channel and a mixer is, and what other classes an structures take part in this, and how it's all interconnected. Ideally, we could lay out the basic design first and gradually add functionality.

On the downside, this pretty much means implementing our own audio system and using OpenAL only to output the final mix as a platform independent layer. If we see channels as fully contained audio streams, we'd even need to do the whole 3D audio deal manually.

Thinking of it that way, we probably should find some middle ground. What would be the smallest possible feature set of channels and mixers? How would that look like? Would it make sense to see channels not as audio data streams, but as hierarchical audio source parameter groupings of some kind?

[Schroedingers-Cat]

2. Adjust the volume via Gizmo
   Set the min. / max. distance of the audio emitter via scene editor like scaling an object

5. Audio Areas
   Define audio areas (like shape of rigidbody ?) so the audio emitter gets the same volume everywhere in this area.

Points 2 and 5 are usually combined into one solution by effectively making the area of the min distance the area where the source's volume isn't affected by the listener position. Additionally, blending between the 3D positioning and simple 2D playback via distance or a custom curve allows for an area where the listener's position doesn't affect the positioning of the audio source.
However, being able to define custom shapes like polygon colliders to define min and max distances of an audio source will allow for a lot of uncommon and creative use of these features.

3. Cone Angles
   the cone angles (inner and outer) specify how "sharp" the direction is. if inner and outer are 180, the sound is the same from all directions. if inner is 45, and outer is like 60, you'd have the full audio when the direction points at the listener within 45°, no (or minimum) audio at above 60°, and fading between both inbetween those angles

This sounds like emulating directivity patterns of a sound source. In the "real" world, a sound source's directivity pattern not only describes its direction-dependent volume but also its direction-dependent frequency spectrum. Furthermore, most sound sources don't have a "static" directivy pattern. E.g., a human making a vocal like an "a" has a different directivity pattern than a human making a sharp "s".

Usually, audio middlewares restrict themselves to a static directivity pattern, where the sound designer can define a direction-dependent volume attenuation and a direction-dependent filter, with the possibility to alter any parameter of any available effect based on the listener's position around a sound source.

If #482 was implemented somehow, these per-source effects wouldn't be available anymore.

In which way are these features mutually exclusive?

Many game engines have the concept of audio channels and mixers.

Having an audio mixer where each channel can be routed into another channel or different mixer (without looped dependencies, of course) would be extremely helpful for sound designers. Some more features for the mixer could be:

• Snapshots and transitioning between snapshots
• Instancing audio effects on different mixer channels
• API allowing to modify the mixer programatically
• Changing a sound source's mixer input during runtime

However, as it has already been pointed out, 3D positioning needs to happen before the mixer. That leaves you with two choices:

• Switch to OpenAL Soft and add the necessary mixing functionality there (not good for portability)
• Use OpenAL as a simple audio API abstraction layer and implement the 3D positioning in Duality

EDIT: OpenTK already seems to be using OpenAL Soft on Windows: andykorth/opentk#18 (comment)
I forgot that with the second option you'd have to write the mixing engine in C#, too.

it's a question whether managed code is performant enough for this sort of number-crunching.

I think implementing the 3D positioning and distance-based parameter modulation effects can be done efficiently in C#.
EDIT: However, doing all the audio mixer work in C#, where users can stack multiple audio effects, could turn out to be not the best choice, performance-wise.

Some more feature ideas:

• Implementing PureData would allow procedural audio assets in Duality
• Expand the sound asset with more common behaviour like pitch&volume randomization, attack-loop-release-behaviour and retrigger options

[ilexp]

EDIT: OpenTK already seems to be using OpenAL Soft on Windows: andykorth/opentk#18 (comment)
I forgot that with the second option you'd have to write the mixing engine in C#, too.

We're using a fork of OpenTK and handle this part manually in the platform backend of Duality: The default is the systems native OpenAL, but there's a special case for Windows to check whether a system implementation is available and fall back to OpenAL Soft, if the answer is no. It's there, but mainly as a fallback.

This sounds like emulating directivity patterns of a sound source. In the "real" world, a sound source's directivity pattern not only describes its direction-dependent volume but also its direction-dependent frequency spectrum. Furthermore, most sound sources don't have a "static" directivy pattern. E.g., a human making a vocal like an "a" has a different directivity pattern than a human making a sharp "s".

Usually, audio middlewares restrict themselves to a static directivity pattern, where the sound designer can define a direction-dependent volume attenuation and a direction-dependent filter, with the possibility to alter any parameter of any available effect based on the listener's position around a sound source.

Ah, makes sense - if we're doing both effects and direction dependence, we could combine the two to specify not only volume, but also effect parameters in a direction dependent way.

Random thought: From an audio designer's perspective, would there be any value in skipping the entire effects parameterization and instead having two audio sources that are crossfaded depending on the cone angle? It would allow to apply arbitrarily complex filtering and effects externally (in the audio software of your choice) and maximum artistic freedom, although still limited to an "inside the main direction" / "outside the main direction" distinction.

[Schroedingers-Cat]

We're using a fork of OpenTK and handle this part manually in the platform backend of Duality: The default is the systems native OpenAL, but there's a special case for Windows to check whether a system implementation is available and fall back to OpenAL Soft, if the answer is no. It's there, but mainly as a fallback.

I see.
I had another thought about the performance of the audio mixer plus user stackable audio effects. That part could be implemented in C++ without any OS-dependency, just plain objects and buffer calculations. So C# would hand the raw audio buffers into the C++ part where the 3D positioning + source effects + audio mixer + mixer effects are happening. Once the "master" channel buffer is ready, it gets fed into OpenAL. Would still require rewriting a lot of existing OpenAL functionality, but it would be portable and very fast.

I also checked the OpenAL specification to see if it supports feeding the output of a source into another source (which could be used to simulate a mixing engine) but found nothing in that regard. Maybe you can find something like that in there?

It has come to my attention that Unity is using an AOT compiler (they call it "Burst" compiler). Is there something similar available that we could use? It may help reducing the latency when doing the audio mixer + audio effects in C# and thus improving performance considerably.

Random thought: From an audio designer's perspective, would there be any value in [...] having two audio sources that are crossfaded depending on the cone angle?

Yes, but I makes more sense to not restrict that to the cone angle. Layering is a general and broad concept and should not be restricted to a specific parameter. Here is an example of how it works in audio middlewares:
This can be done without a fancy GUI. Have something like a layer list and define a curve and a parameter for each layer, that's all the data you need.

A crucial part with this technique is timing. Starting multiple sources at once needs to guarantee that they start exactly the same time/buffer/sample position. If not, phase effects will happen, most likely not to the enjoyment of the audio designer.

[mfep]

If #482 was implemented somehow, these per-source effects wouldn't be available anymore.

In which way are these features mutually exclusive?

The OpenAL effects can be applied per-audio-source. If we use the native audio sources for the bus channels, it's only possible to set the effect on a bus, not on a single audio source. (Of course as a workaround every source can be assigned to exactly one channel with the desired effects.)

5. Audio Areas
   Define audio areas (like shape of rigidbody ?) so the audio emitter gets the same volume everywhere in this area.

A quite intuitive way of connecting this to the mixer system would be, that an audio area triggered a mixer channel change on the entering and exiting audio sources. However, without using the physics system, the triggering is again far from trivial.

So as I can see, the conversation went quite far from exposing some parameters to implementing an audio workstation to Duality. I think at this point we should also consider switching to a different audio backend, which perhaps provides more functionality that we need. Unfortunately there's not a huge load of open-source options, among them SoLoud seems to be the best. It's a middleware itself, supporing various backends (including OpenAL). Also it has

• C# bindings
• channels/mixers
• 3d audio
• a couple of synthesizers
• a couple of filters and effects
• audio file streaming
• grouped sound instances (guaranteed to start together)

In my opinion, this project could be a valuable resource, used directly as well as a source code reference.

[Schroedingers-Cat]

The OpenAL effects can be applied per-audio-source. If we use the native audio sources for the bus channels, it's only possible to set the effect on a bus, not on a single audio source. (Of course as a workaround every source can be assigned to exactly one channel with the desired effects.)

I'm unaware of #482 having any involvement with bus mixing structures. The best implementation to #482 won't change a thing in how the current systems interact with each other. It boils down to some averaging between virtual audio listeners affecting the audio source's parameters in Duality.

5. Audio Areas
   Define audio areas (like shape of rigidbody ?) so the audio emitter gets the same volume everywhere in this area.

A quite intuitive way of connecting this to the mixer system would be, that an audio area triggered a mixer channel change on the entering and exiting audio sources. However, without using the physics system, the triggering is again far from trivial.

Why should the audio emitter's volume have any effect/change on the way it is mixed in the mixer system? Wouldn't it be simpler to treat this level-information at the sound source's 3D positioning/attenuationing stage than in the mixer stage?
Maybe I understood point 5 wrong. What functionality should it provide? To me, it sounds like creating an area with a constant volume.

Yes, SoLouds features are impressive. However, their OpenAL backend is probably not what one would expect. From the description:

Very experimental. Very high latency; if this is your only option, you're probably better off using OpenAL directly. No x64.