mac-capture: Add channel selection for CoreAudio input devices

[PatTheMav]

Description

This is an update of an older pull request (#2889) to fix the issues that were reported on the original PR.

The original PR added the ability to select and map input channels to the input channels supported by OBS Studio.

Motivation and Context

When OBS Studio is set to stereo output, it only accepts input on channels 0 and 1 of audio devices and "ignores" any output on other channels. Existing plugins cannot work around this issue, because they always access the audio data provided by libobs but not by the hardware.

This PR uses the capability of CoreAudio output units to be configured with a channel mapping that maps arbitrary input channels to the configured output channels.

This unlocks use cases like:

• Capturing input on channels 2+3 as channels 0+1
• Capturing input on channel 0 as channels 0+1 (duplicating mono microphone input as a stereo source)
• Capturing input on channel 5 as channel 1 (mono input)

All this happens before the audio is provided to libobs, so application can stay in "stereo mode".

Important

This PR also changes the output format used by CoreAudio to provide audio to libobs: CoreAudio flattens audio for AUHAL units to its "canonical" format by default (32-bit float, deinterlaced/planar, PCM) which fortunately aligns with OBS' internal format. The only adjustment is made to sampling rate (the output element of the AUHAL unit is set to the sampling rate of the input element).

This means that OBS will only add a resampling step if the sampling rates don't match.

The obvious benefit is that the entire CoreAudio code will assume the planar 32-bit float format to create buffers and render the input data (per Apple's documentation that this is what AUHAL units do by default). Using the "canonical" format is also suggested by Apple for any sound processing between Audio Units.

How Has This Been Tested?

Tested on macOS 13 with default macOS audio devices and external audio devices. Tested with multiple sample rate settings and channel configurations.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[WizardCM]

Please include screenshots.

[PatTheMav]

Please include screenshots.

Effectively the same as in the original PR - the number of selectable outputs depends on OBS' channel configuration:

[logsol]

I'd definitely be excited to see this 👍

[PatTheMav]

@pkviet could you try and confirm that it fixes the issue you had with the original PR?

[pkviet]

@PatTheMav good and bad news
Good: the bug is fixed on my side, namely there's no issue when trying to capture (0,0) or (1,1) for instance.
Bad: if I change once the channel selections, the audio streams. But if I try to change once more, the audio stops streaming. This needs confirmation though.
2023-11-24 08:24:47.328739+0100 OBS[1706:21972] [aqme] AQMEIO.cpp:199 timed out after 2.500s (2312 2312); suspension count=0 (IOSuspensions: ) 2023-11-24 08:24:47.328977+0100 OBS[1706:21972] [aqme] MEDeviceStreamClient.cpp:431 AQME Device-AppleUSBAudioEngine:ESI Audiotechnik GmbH:U22 XT USB:60300000:2,3: client stopping after failed start: <AudioQueueObject@0x7fe1ea1a5800; [1706]; play>; running count now 1

We might have to reset some stuff when swapping channels, maybe ?

[PatTheMav]

@PatTheMav good and bad news Good: the bug is fixed on my side, namely there's no issue when trying to capture (0,0) or (1,1) for instance. Bad: if I change once the channel selections, the audio streams. But if I try to change once more, the audio stops streaming. This needs confirmation though. 2023-11-24 08:24:47.328739+0100 OBS[1706:21972] [aqme] AQMEIO.cpp:199 timed out after 2.500s (2312 2312); suspension count=0 (IOSuspensions: ) 2023-11-24 08:24:47.328977+0100 OBS[1706:21972] [aqme] MEDeviceStreamClient.cpp:431 AQME Device-AppleUSBAudioEngine:ESI Audiotechnik GmbH:U22 XT USB:60300000:2,3: client stopping after failed start: <AudioQueueObject@0x7fe1ea1a5800; [1706]; play>; running count now 1

We might have to reset some stuff when swapping channels, maybe ?

Can't reproduce this - just did a recording, switching channel maps around during recording and all went fine - recording has everything coming in at the expected channels.

info: ==== Recording Start ===============================================
info: [ffmpeg muxer: 'adv_file_output'] Writing file '/Users/***/Movies/2023-11-24 17-28-27.mov'...
info: Channel map [0,0]. Inputs available: 6
info: coreaudio: Device 'Komplete Audio 6 MK2' [44100 Hz] initialized
info: Channel map [-1,0]. Inputs available: 6
info: coreaudio: Device 'Komplete Audio 6 MK2' [44100 Hz] initialized
info: Channel map [-1,1]. Inputs available: 6
info: coreaudio: Device 'Komplete Audio 6 MK2' [44100 Hz] initialized
info: Channel map [1,1]. Inputs available: 6
info: coreaudio: Device 'Komplete Audio 6 MK2' [44100 Hz] initialized
info: Channel map [1,-1]. Inputs available: 6
info: coreaudio: Device 'Komplete Audio 6 MK2' [44100 Hz] initialized
info: Channel map [0,-1]. Inputs available: 6
info: coreaudio: Device 'Komplete Audio 6 MK2' [44100 Hz] initialized
info: [ffmpeg muxer: 'adv_file_output'] Output of file '/Users/***/Movies/2023-11-24 17-28-27.mov' stopped
info: Output 'adv_file_output': stopping
info: Output 'adv_file_output': Total frames output: 4794
info: Output 'adv_file_output': Total drawn frames: 4821
info: Video stopped, number of skipped frames due to encoding lag: 1/4820 (0.0%)
info: ==== Recording Stop ================================================

[RockyYVR]

I am very interested in this feature!

[pkviet]

Approved for merging.

[shalako]

Really looking forward to a release of obs with this feature! It will enable me to significantly simplify my streaming solution.

[PatTheMav]

This PR just received an unexpected bigger update - as @Lain-B correctly pointed out, OBS will automatically downmix source audio to the number of channels set up for output.

The changes effectively removed this convenience feature by limiting the output to the specific amount of channels, which might be desirable in some scenarios, but not all.

With the recent push this change has been made optional with a new checkbox in the property window to "Enable Downmixing":

• When enabled, the source will report the full amount of channels supported by the device to OBS. All audio from all channels is provided to the audio subsystem which will apply downmixing automatically
• When disabled, the source will allow the selection of specific input channels for specific output channels

Downmixing is enabled by default to retain OBS' current default functionality, with the new ability available to users who require it.

[PatTheMav]

Audio Capture with Automatic Downmixing

Audio Capture with Channel Selection

[PatTheMav]

@RytoEX if you don't have any objections, I'd merge this soon - have tested this with several devices without issue but I guess we need to have this merged and actual "volume" of users on this to get any kinks out.

[RytoEX]

@RytoEX if you don't have any objections, I'd merge this soon - have tested this with several devices without issue but I guess we need to have this merged and actual "volume" of users on this to get any kinks out.

I do not have any substantial objections. Would have liked to maybe see another macOS dev (@gxalpha ?) look over this, but it's probably fine.

[RytoEX]

Tentatively approving this.

[RytoEX]

Tentatively approving this, for real.

[ghost]

Could you please share screenshots of this menu of as it looks now? Especially one that shows the string CoreAudio.None. I would like to add them as context to Crowdin.

[PatTheMav]

Could you please share screenshots of this menu of as it looks now? Especially one that shows the string CoreAudio.None. I would like to add them as context to Crowdin.

See #9766 (comment) for normal operations.

CoreAudio.None is the entry to mute a channel, so the OBS output is associated with no input channel from the hardware device:

[eVcreates]

hi there! how can i test this feature?

[morvius21227]

Send it to me! I want to test this :D