In an effort to push my work further into the audio-visual realm, I’ve started an experiment using the Unity game engine to generate music. While it’s common to see visuals that are audio-reactive, I became much more interested in the possibilities of visual-reactive audio. In this case, meaning that the audio heard is generated from the visual content.
The above project shows a 3D physics simulation in Unity, involving cars crashing and bouncing off of each other in zero gravity. Underneath the Unity window is a drum kit loaded into an Ableton Live project. Using a special fork of Keijiro Takahashi’s MIDI Jack plugin with some of my own code, I utilize the collision data to generate MIDI notes. The MIDI notes then get sent from Unity into Ableton, using a virtual loopback MIDI-port device provided by the LoopMIDI software created by Tobias Erichsen. Different aspects of the collision are used to modify different aspects of the notes, such as which note is chosen and its given velocity.
The simulation itself can be manipulated using various inputs including gamepad and MIDI controllers, for example to increase the force of the collisions and resulting bounces, or to modify the flow of time, or to add and remove vehicles. Changes to the simulation of course creates changes in the music that emerges from it.
A common criticism I get from this project is that it is very chaotic music. Personally I enjoy the chaotic aspect of it. However, I have many proposed solutions that I am currently exploring: for example in the case of rhythmic quantization, the physics framerate can be lowered to a value that matches the desired BPM and value of subdivisions. If the physics framerate is set to 2 FPS, this of course lowers the accuracy of the simulation- but with the advantage of collisions only occurring at 2 equally spaced moments each second, resulting in sounds that play on a quarter note grid at 120 BPM.
Jason Gluck 