Here is part 2 of the series discussing the audio tech we implemented for FRACT OSC, check out part 1 here.
So once we’d decided to go with Pure Data for audio synthesis, the question was, how could we integrate that with the rest of FRACT’s engine?
When used in a live performance or gallery setting, the most common way to use Pure Data in tandem with other tools is simply to run them side by side as separate programs. The programs then communicate with each other in some way, often over the network using MIDI or Open Sound Control (a different OSC from the OSC in FRACT’s title). This is a flexible way to use any combination of programs together, but it has a few downsides for a case like ours where we need to put together a robust package that “just works.” There would be work involved in starting up Pure Data as a separate process, monitoring it, and closing it properly when the game exits, and since this sort of functionaly is usually platform-dependend it would all have to be rewritten on each platform. In addition, since Windows and OSX still see the two programs as separate entities, they might decide to force close (for whatever reason) one without closing the other, so you might end up with a soundless FRACT OSC (yuck) or an abandoned audio engine blaring forever (even more yuck).
Fortunately, there was another option. A fork of Pure Data called libpd wraps most of the core functionality of Pure Data into a library that can be embedded directly into native programs. In our case, we compiled it into a native plugin that we can use from Unity. (Native plugins are only available in pro or mobile versions of Unity.) This way Pure Data runs as a built-in part of our game.
There were also problems with this approach, though. Most notably, since the Unity editor runs in the same process as your game, it’s also affected by errors your game runs into. Normally these errors are in managed C# code, and are caught by Unity and logged. Unfortunately, errors in native code can’t be caught, and so if I make a mistake coding the plugin not only does it crash but it brings down the entire Unity editor with it! This wasn’t uncommon near the beginning of development, and so we formed a consistent habit of saving all changes before hitting “Play,” to prevent from losing unsaved work.
Next time I’ll talk about how the plugin works, and how we communicate through it to Pure Data!