What are you on about?

I'm glad you ask. Basically manipulating sound at runtime is what we're talking about here. Sounds simple right? It's not. See Flash has some built-in support for Volume and Panning but that is pretty much as far as it goes. That was until now.

Joe Berkovitz (respect again) is currently working on a library for full sound manipulation. Not sure where he has got to but it should be good. He has however not released anything yet which left me empty handed navigating away from his blog.

The only other major player's source I could get my hands on was the library called PopForge created by andre.michelle & joaebert. An open source library - at first glance PopForge looked very daunting. But soon after playing around I got that pleasant programmer surprise when you use an API for the first time. Things actually worked. It's like creating a primitive in Papervision and seeing it for the first time. It is one of the reasons why we do what we do. And why we use Actionscript to do it. There is no feedback like visual feedback.

ByteArray basics

Loading a sound file and manipulating it at runtime is not that easy. To understand how these guys did it requires some knowledge of the afore mentioned ByteArray and how files are read in the Flash Player.

With AS3 you can now access the actual bytes of a loaded file in the Flash Player memory. What does this mean? Well very simply, when you load a file using the URLLoader class you can access the actual bytes it takes up in memory. Still confused? Consider this example. I have an image file uploaded by a user. To optimise disk space I want to compress this image file using some JPEG compression algorythm before saving it on my server. Normally we would have the user upload the file, a server-side script would take care of encoding/compressing the uploaded file and then move it to a specified folder. A much faster way would be to have the server-side script save the already compressed file. But when does the compression take place? In the Flash Player of course! Being able to access the bytes we can perform a JPEG compression on the ByteArray and send the newly compressed ByteArray to the server, which means, we send less bytes to the server-side script. How cool is that?

You need to know a bit about the bytes you are going to change when working with ByteArray. So when it comes to sound we will need to know more about the make-up of a PCM WAV file. Sounds heavy but lets take it one step at a time. Popforge use PCM WAV because it is uncompressed. Remember MP3 is an encoded/compressed file and would require decoding before we can do anything with it. The Flash Player can play MP3 files because it has a decoder built-in.

So lets look at the raw WAV format a little closer. Because understanding how the file is made up will get us a little closer to understanding what is under the PopForge hood.

Brief history of WAVE and RIFF storage method

The WAV file format is a Microsoft and IBM standard format for storing audio in a digital format. It is based on the RIFF (Resource Interchane File Format) bitstream method which coincidently is also used in AIFF, a format commonly used on Apple computers. RIFF actually identifies the sotrage method used when storing audio. RIFF files consist entirely of chunks. Each chunk is made of some bytes describing that particular chunk. To see what each chunk breaks down to, view the RIFF resource at the end of this article. Now with this byte-level information, the guys from PopForge got crackin and actually wrote their own WAV file encoder/decoder. Now we're talking.

The Experiment

 
private function loadWave(): void
{
	filter = new ParametricEQ();
	var loader:URLLoader = new URLLoader();
	loader.dataFormat = URLLoaderDataFormat.BINARY;
	loader.addEventListener(Event.COMPLETE, onLoaderComplete );
	loader.addEventListener(ProgressEvent.PROGRESS, onLoaderProgress );
	loader.addEventListener(IOErrorEvent.IO_ERROR, onLoaderError );
	loader.load( new URLRequest( "sound/loop.wav" )); 
}

private function onLoaderComplete( event: Event ): void
{
	wav = WavFormat.decode( ByteArray( event.target.data ));
	initAudioEngine();
}

private function initAudioEngine(): void
{
	buffer = new AudioBuffer( 4, Audio.STEREO, Audio.BIT16, Audio.RATE44100 );
	buffer.onInit = onAudioBufferInit;
	buffer.onComplete = onAudioBufferComplete;
	phase = 0;
}  

private function onAudioBufferInit( buffer: AudioBuffer ): void
{
	buffer.start();
}

private function onAudioBufferComplete( buffer: AudioBuffer ): void 
{
	var samples: Array = buffer.getSamples();
	sampler( samples );
	filter.processAudio( samples );
	buffer.update();
}

private function sampler( samples: Array ): void
{
	var n: int = samples.length;  //-- some locals
	var output: Sample;
	var input: Sample;
	var speed: Number = wav.rate / buffer.getRate() * model.speed;
	
	for( var i: int = 0 ; i < n ; ++i )
	{
		output = samples[i];
		input = wav.samples[ int( phase ) ];
		output.left = input.left;
		output.right = input.right;
		phase += speed;
		if( phase < 0 ) phase += wav.samples.length;
		else if( phase >= wav.samples.length )
		phase -= wav.samples.length;
	}
}  

Start off by loading the WAV file. The URLLoader's dataFormat property is set to binary for us to access the byteArray through event.target.data. WaveFormat.decode is used to decode the WAV file's ByteArray and create samples. "A sample refers to a value or set of values at a point in time and/or space." (Wikipedia). Essentially sampling means breaking down the sound into chunks, i.e. samples. "A sampler is a subsystem or operator that extracts samples from continuous signal" (Wikipedia) like a sound wave. But wait, isn't a WAV file essentially made of chunks? Well yes it is. Well spotted. The custom WAV decoder class generates these samples for us from the raw ByteArray that was loaded. How do they do this? With the knowledge of what bytes in a WAV file refers to and based on the RIFF storage method, the PopForge guys went about their business. Now these samples are stored in the WAV instance for us to access at any given time. Excellent.

Next we intialise the PopForge AudioBuffer. The AudioBuffer generates a native Flash Sound object in ByteArray from the blueprint of the RIFF stream storage method. Once this is set up the buffer will start to play this empty Sound object. The AudioBuffer will then be filled with samples from the loaded WAV file. Remember we decode the loaded WAV file in the beginning and broke it down into samples. After each cycle a new AudioBuffer is created and the next WAV sample is pushed into the AudioBuffer. By having access to the samples being streamed into the buffer we can do loads of stuff to it. Like speeding up playback would require us to increase the pointer that indicates the next sample that is pushed into the buffer. The same would suffice for slowing down.

PopForge has some basic filters included and expose the structure for writing your own. And as most signal manipulation happens with Math, anyone should be able to write their own filter for this library. In the example above the onAudioBufferComplete handler applies some filter to the buffer's samples.

The event handler onAudioBufferComplete is called upon each time the new buffer is created. What this also means is that we can apply a filter to the next cycle by applying some calculations on the samples and then updating the buffer. All in all very impressive stuff. If none of the above made any sense view the example below. Use the LEFT and RIGHT arrow keys to increase or slow down the tempo. And press SPACE to apply an EQ filter to the sound at anytime. You will notice that the EQ will be applied regardless of the speed of the track.

View my experiment here »

Resources

• Adobe LiveDocs: ByteArray
• Wikipedia: Sampling signal processing
• Wikipedia: WAV format
• Wikipedia: Resource Interchange File Format