Adventures in scheduling, buffers and parameters


Porting a dynamic audio engine to Web Audio



Chinmay Pendharkar, Peter B├Ąck and Lonce Wyse / Sonoport

Overview

  • 1. About us
  • 2. Sonoport Studio
  • 3. Dynamic Sound Models
  • 4. Porting to Web Audio
  • 5. Problems & Solutions
  • 6. Audio Worker
  • 7. Summary

About Me

Chinmay Pendharkar

@ntt / notthetup

  • we work with Interactive Dynamic Audio for Multimedia
  • we're building a Smart Sound Exchange Platform
  • we're located in Singapore & UK

2. Sonoport Studio

  • First application for the Smart Sound Exchange
  • Interactive Audio Authoring Platform
  • Core : Dynamic Sound Engine
    • - Collection of Sonoport Dynamic Sound Models
    • - Runtime + Utilities

2. Sonoport Studio

Demo Booth

  • Check out Sonoport Studio at Demo/Poster Session #2
  • Tuesday afternoon at MOZILLA
  • 15.00 - 17.00

3. Sonoport Dynamic Sound Models

  • Algorithms that make sound
  • Method to start/stop sounds
  • Parameters to change the sounds in real-time
  • Texture based or purely Synthesized

Sonoport Dynamic Sound Models

Now Open Source!!

http://wac.sonoport.com

3. Sonoport Dynamic Sound Models

  • Based on Lonce Wyse's work in Java
  • Moved to Flash (starting with Flash 10)
  • January 2014 - Starting migration to Web Audio

4. Porting to Web Audio - Aims

  • Port all existing functionality of Flash version
  • Keep API Consistency with Flash version
  • Leverage on Web Audio for 'plumbing'
  • Leverage the composibilty of Web Audio

3 Problems : 3 Solutions Hacks!

Problem #1 : Can't create custom Parameters in Web Audio

Problem #1 - Higher Level Nodes

  • All Dynamic Sound Models are Higher Level Nodes
  • Higher Level Nodes to leverage composibility of Web Audio
  • Internally composed of multiple AudioNodes

Problem #1 - Parameters

  • Higher Level Nodes need Parameters
  • Custom params should have similar interface as AudioParam
  • Custom params may be mapped to internal AudioNodes
  • Bonus : Custom Params should support Automation

Solution #1: SPAudioParam


Wrap around AudioParams, expose similar interface to AudioParams

SPAudioParam - Two Approaches

Wrapped SPAudioParam

  • Map SPAudioParam to underlying AudioParam
  • Use getters/setters to pass along values
  • Mapping/normalization on the fly
  • Pass along Automation method calls

Mocked SPAudioParam

  • Plain old JS Object
  • Fakes Automation using setInterval (URGH!)
  • For edge case without underlying AudioParam

Solution #1: SPAudioParam

  • Mostly used Wrapped SPAudioParam approach
  • Slight change in API for Parameters but free automation!!
  • Mocked SPAudioParam approach handy in edge case scenarios

SPAudioParam

						
/*Create Model*/
var looper = new Looper (context, "/audio.wav");
/*Play*/
looper.play();
/*Set SPAudioParam*/
looper.easeIn.value = 0.3;
looper.easeOut.value = 0.7;
/* Automate SPAudioParam*/
looper.playSpeed.exponentialRampToValueAtTime(2,10);
					
					

Problem #2 : Can't schedule some operations in Web Audio

Problem #2 - Queues

  • Scheduling is key, but can't un-schedule in Web Audio
  • Keep a Queue, schedule/unschedule on the Queue
  • Queue executes 'Events' close to the scheduled time
  • Common Queue Events : Start, Stop, Change Param

Problem #2 - Polyphony

  • Polyphonic Models with multiple voices
  • Typical archtecture : fixed number / pool of voices
  • Queue assigns buffers to a voice at the scheduled time

Problem #2 - Polyphony

  • New Queue Event : SetSource <- not schedulable!
  • Can't use setInterval for scheduling SetSource either
  • Executing SetSource at earlier could override another voice

Solution #2: Fire and Forget Voices


Create new voices as and when needed

Solution #2 - Fire and Forget Voices

  • Architecture : new BufferNode everytime a voice is needed
  • Ensure GC captures all the used up BufferNodes
  • Test no leaks using memory profiling and heap dumps
  • Tested to upto ~60x3 Nodes created per second

MultiTrigger Demo

 
/*Construct Sound Model*/
var aFiles = ["audio/Hit5.mp3", "audio/Hit6.mp3", "audio/Hit7.mp3", "audio/Hit8.mp3"];
var mt = new MultiTrigger(window.audioContext,aFiles);
/*Hook Up Button*/
multiTriggerButton.onclick = function (){
  if (multiTriggerButton.innerText === "Play"){
    multiTrigger.play();
    multiTriggerButton.innerText="Pause";
  }else{
    multiTrigger.pause();
    multiTriggerButton.innerText="Play";
  }
}
/*Hook Up Slider*/
multiTriggerSlider.oninput = function(){
  multiTrigger.eventRate.value = parseFloat(multiTriggerSlider.value);
}
						

eventRate

Problem #3 : Can't pause an AudioBufferSourceNode

Problem #3 - Buffers

  • AudioBufferSourceNode can only be played once
  • Pause/Play needs the position at which it was paused
  • No way to know current playbackPosition
  • Time based tracking only works for static playbackRate
  • Tracking through automated playbackRate is tedious and inaccurate

Problem #3 - Buffers

Solution #3: SPAudioBufferSourceNode


Wrap around AudioBufferSourceNode, and use a 'Scope' track playback

Solution #3 - SPAudioBufferSourceNode

  • Create a mirror AudioBuffer with index as value
  • Create a CounterBufferNode from this AudioBuffer
  • Connect this CounterBufferNode to a ScriptProcessorNode
  • onprocessaudio stores the last value of the inputBuffer
  • Perform every action (start, stop, automation..) on both AudioBufferSourceNodes

SPAudioBufferSourceNode Implementation

 
/*Create Nodes*/
var counterNode_ = audioContext.createBufferSource();
var scopeNode_ = audioContext.createScriptProcessor( 256, 1, 1 );

/*Connect Nodes*/
counterNode_.connect(scopeNode_);

/*Create Counter Buffer*/
counterNode_.buffer = createCounterBuffer( buffer.buffer );

function createCounterBuffer( audioBuf ) {
	var array = new Float32Array( buffer.length );
	var audioBuf = audioContext.createBuffer( 1, buffer.length, 44100 );

	for ( var index = 0; index < buffer.length; index++ ) {
	    array[ index ] = index;
	}

	audioBuf.getChannelData( 0 ).set( array );
}

/*Remember last element in inputBuffer*/
scopeNode_.onaudioprocess = function savePosition( processEvent ) {
  var inputBuffer = processEvent.inputBuffer.getChannelData( 0 );
  lastPos = inputBuffer[ inputBuffer.length - 1 ] || 0;
}
					

Solution #3 - SPAudioBufferSourceNode

  • Stored value exposed as playbackPosition property
  • playbackPosition is read only
  • Accurate to nearest ScriptNode buffer size
  • Not perfect buy good enogh for pausing/playing

Play and Pause

						
/*Create Model*/
var loop = new Looper(window.audioContext, "audio/loopy.mp3");
/*Hook Up Button*/
playPauseButton.addEventListener('click', function (){
  if (playPauseButton.innerText === "Play"){
    loop.play();
    playPauseButton.innerText = "Pause";
  }else{
    loop.pause();
    playPauseButton.innerText = "Play";
  }
});
/*Hook Up Slider*/
playPauseSlider.addEventListener('input', function(){
  loop.playSpeed.value = parseFloat(playPauseSlider.value);
});
						
						

playbackRate

Porting to WebAudio - Summary

  • Overcame most issues we faced while porting
  • Most solutions worked for our specific use case
  • Your mileage may vary based on what you're doing
  • There are creative ways to implement most use cases

Enter the AudioWorker!

Problem #1 - Parameters

  • AudioWorkers have an addParameter method!
  • Create a-rate parameters accessible inside AudioWorker!
  • Should solve all issues with creating custom Parameters

Problem #2 - Scheduling

  • AudioWorkers enables sample accurate scheduling
  • Queues can runs inside an AudioWorker
  • Load operations can be scheduled accurately

Problem #3 - Buffers

  • No playbackPosition property on AudioBufferSourceNode
  • .. but AudioBufferSourceNode could be reimplmented inside AudioWorker!
  • No idea of performance untill we get AudioWorker

7. Summary

  • Hacking around Web Audio was fun
  • Performance vs Functionality tradeoff
  • Might not work perfectly in all scenarios
  • AudioWorker should make life easier
  • Would love to know other ways of solving such problems!

Questions?

Slides, Papers, Links at http://wac.sonoport.com