Custom-Audio-Device

Custom Audio Device

The Custom Audio Device sample application shows how to use a custom audio device in your application. A custom audio device lets you implement your own way of either capturing or rendering audio or both. It is important to note that only a custom audio device can be set at a time in an application.

This sample builds upon the Publisher Only sample, using a custom audio device that implements a custom way of capturing generated audio samples.

You will need a valid Vonage Video API account to build this app. (Note that OpenTok is now the Vonage Video API.)

Setting up your environment

OpenTok SDK

Building this sample application requires having a local installation of the OpenTok Linux SDK.

On Debian-based Linuxes

The OpenTok Linux SDK for x86_64 is available as a Debian package. For Debian we support Debian 12 (Bookworm). We maintain our own Debian repository on packagecloud. Follow these steps to install the packages from our repository.

• Add packagecloud repository:

curl -s https://packagecloud.io/install/repositories/tokbox/debian/script.deb.sh | sudo bash

• Install the OpenTok Linux SDK packages.

sudo apt install libopentok-dev

On non-Debian-based Linuxes

Download the OpenTok SDK from https://tokbox.com/developer/sdks/linux/ and extract it and set the LIBOPENTOK_PATH environment variable to point to the path where you extracted the SDK. For example:

wget https://tokbox.com/downloads/libopentok_linux_llvm_x86_64-2.28.2
tar xvf libopentok_linux_llvm_x86_64-2.28.2
export LIBOPENTOK_PATH=<path_to_SDK>

Other dependencies

Before building the sample application you will need to install the following dependencies

On Debian-based Linuxes

sudo apt install build-essential cmake clang libc++-dev libc++abi-dev \
    pkg-config libasound2 libpulse-dev libsdl2-dev

On Fedora

sudo dnf groupinstall "Development Tools" "Development Libraries"
sudo dnf install SDL2-devel clang pkg-config libcxx-devel libcxxabi-devel cmake

Building and running the sample app

Once you have installed the dependencies, you can build the sample application. Since it's good practice to create a build folder, let's go ahead and create it in the project directory:

$ mkdir Custom-Audio-Device/build

Copy the config-sample.h file as config.h at Custom-Audio-Device/:

$ cp common/src/config-sample.h  Custom-Audio-Device/config.h

Edit the config.h file and add your OpenTok API key, an OpenTok session ID, and token for that session. For test purposes, you can obtain a session ID and token from the project page in your Vonage Video API account. However, in a production application, you will need to dynamically obtain the session ID and token from a web service that uses one of the Vonage Video API server SDKs.

Next, create the building bits using cmake:

$ cd Custom-Audio-Device/build
$ CC=clang CXX=clang++ cmake ..

Note we are using clang/clang++ compilers.

Use make to build the code:

$ make

When the custom_audio_device binary is built, run it:

$ ./custom_audio_device

You can use the OpenTok Playground to connect to the OpenTok session in a web browser, view the stream published by the Custom Audio Device app, and publish a stream that the app can subscribe to.

You can end the sample application by typing Control + C in the console.

Understanding the code

The main.cpp file includes the OpenTok Linux SDK header:

#include "opentok.h"

We will focus here in the custom audio device implementation. This sample builds upon the Publisher Only sample.

Implementing the custom audio device

A custom audio device is represented by an otc_audio_device struct. And an otc_audio_device_callbacks struct includes function pointers to functions that act as the audio-related callbacks that the OpenTok Linux SDK invokes.

The implementation of the custom audio device in this sample defines a custom audio capturer. It does not set up a custom audio renderer, but its implementation would be similar.

This sample uses a struct audio_device user-defined struct that defines everything needed by the custom audio capturer:

struct audio_device {
  otc_audio_device_callbacks audio_device_callbacks;
  otk_thread_t capturer_thread;
  std::atomic<bool> capturer_thread_exit;
};

The application initializes it and sets the pointers to callback functions:

struct audio_device *device = (struct audio_device *)malloc(sizeof(struct audio_device));
device->audio_device_callbacks = {0};
device->audio_device_callbacks.user_data = static_cast<void *>(device);
device->audio_device_callbacks.destroy_capturer = audio_device_destroy_capturer;
device->audio_device_callbacks.start_capturer = audio_device_start_capturer;
device->audio_device_callbacks.get_capture_settings = audio_device_get_capture_settings;

The audio_device_start_capturer function starts a thread in which audio is captured:

static otc_bool audio_device_start_capturer(const otc_audio_device *audio_device,
                                            void *user_data) {
  struct audio_device *device = static_cast<struct audio_device *>(user_data);
  if (device == nullptr) {
    return OTC_FALSE;
  }

  device->capturer_thread_exit = false;
  if (otk_thread_create(&(device->capturer_thread), &capturer_thread_start_function, (void *)device) != 0) {
    return OTC_FALSE;
  }

  return OTC_TRUE;
}

The capturer_thread_start() function creates an array of 480 16-bit audio samples (from a generated waveform) and passes the array to the otc_audio_device_write_capture_data() function, defined in the OpenTok Linux SDK:

static otk_thread_func_return_type capturer_thread_start_function(void *arg) {
  struct audio_device *device = static_cast<struct audio_device *>(arg);
  if (device == nullptr) {
    otk_thread_func_return_value;
  }

  int16_t samples[480];
  static double time = 0;

  while (device->capturer_thread_exit.load() == false) {
    for (int i = 0; i < 480; i++) {
      double val = (INT16_MAX  * 0.75) * cos(2.0 * M_PI * 4.0 * time / 10.0 );
      samples[i] = (int16_t)val;
      time += 10.0 / 480.0;
    }
    otc_audio_device_write_capture_data(samples, 480);
    usleep(10 * 1000);
  }

  otk_thread_func_return_value;
}

The otc_audio_device_write_capture_data() function adds the audio byte array to the audio buffer that will be used for audio in the published stream.

Setting the audio device to be used by the application

Once the application initializes the struct audio_device struct and sets the pointers to the callback functions, the custom audio device can be set by using the otc_set_audio_device() function defined in the OpenTok Linux SDK.

  otc_set_audio_device(&(device->audio_device_callbacks));

The function takes a pointer to the audio_device_callbacks struct (of type otc_audio_device_callbacks) defined in the previous section.

Configuring the audio device

The OpenToken Linux SDK calls the get_capture_settings callback function that we added to the audio_device_callbacks struct. In this callback, we adjust configuration settings for the audio capturer:

static otc_bool audio_device_get_capture_settings(const otc_audio_device *audio_device,
                                                  void *user_data,
                                                  struct otc_audio_device_settings *settings) {
  if (settings == nullptr) {
    return OTC_FALSE;
  }

  settings->number_of_channels = 1;
  settings->sampling_rate = 48000;
  return OTC_TRUE;
}

The app sets the sampling rate to 48000 (samples per second). Note that the custom audio device implementation (described above) sets 480 audio samples each 10 microseconds (48,000 per second).

Next steps

See the Vonage Video API developer center for more information on the OpenTok Linux SDK.

See the Adjusting audio and video -- Linux developer for more information on setting custom audio captures and renderers, as well as other audio options, using the OpenTok Linux SDK.