Nik's Final Project
Wiki in progress.
My final project has two main components: it uses a DDR pad to control a beat with some effects and uses the webcam to control three other instruments.
The Dance Pad
The DDR dance pad is just like any other HID device (joystick, keyboard, etc). I set up different SndBufs for the different arrows and they are activated when you stomp on that button, and turned off if you push it again. The diagonal pads are used to control effects. The left side is for a LPF on all the SndBufs and you control the cutoff frequency. The right side has three different modes (Reverb, Echo, and Gain) which are ChucKed to all the SndBufs and have universal control. You switch through the different modes with the small "Start" button at the top of the pad. To switch to different beats or play modes you cycle through them with the other small button at the top: "Select". The dance pad is spatially designed so you understand it pretty quickly. Any beats could be linked in and any effects could easily substitute for the ones I chose.
I'd been wanting to do something with the webcam all semester and now I finally did. I used Processing to get capture video from the built-in webcam. In Processing I also did a little analysis of the incoming video data. My code tracks where the brightest pixel currently is (so it's best to perform in a dark room with LED-type lights). It also determines the total amount of movement between each frame by analyzing the changes in Red Green and Blue content. It also draws a little dot on the screen where the brightest pixel is, and it only updates and displays the changes in RGB content, which gives it a sort of dark, eerie effect. Then it sends this data to ChucK via OSC. It sends the x and y position of the brightest pixel, the value of brightness (0-255), and the total amount of movement.
ChucK takes in this data and uses it to control some instruments. The primary instrument is a flute. The y-position of the brightest pixel controls the flute's frequency, and the x-postion controls it's Jet Reflection (which controls the overall tone/quality of the flute sound, kinda). The flute only turns on if the brightest pixel is above a certain threshold (in this case 245). The xy-position also controls a drone. When the brightest pixel is at the bottom of the screen it turns on the drone, and it turns off when you move to the top of the screen. Moving the light around will change the sound of the drone by adjusting the gain of different harmonic overtones.
The other instruments controlled by the webcam are mandolins and shakers. The more movement you make the denser the mandolins and shakers will be. I set up certain thresholds for the movement, and used probability to adjust the density. All of the instruments play in D minor using note arrays.
In the zip file (here) you'll find 2 ChucK files, one Processing file, and the wav files I used for the beats. (The beats were made with Kinetic 2). The 2 ChucK files are independent of each other. However the webcam ChucK file won't do anything unless you launch the Processing file first. It's all pretty straightforward and works with the Virtual Machine or through Terminal.
I recorded the audio and video directly using a ChucK file and some Processing code. However these these were recorded in slightly different formats and frame rates so my video example doesn't line up perfectly. When playing live the webcam and the dance pad are both very responsive. Also, the compression of the video isn't great so it flickers and looks jerkier and sounds choppier (more clicks) than it actually is. You can't see the dance pad in the video, but I'm standing on it and controlling everything you don't see that way.
This could be easily adaptable for whatever someone would want to do with it. The webcam and dance pad are independent, but they could be synced pretty easily. I chose to sync just the dance pad and allow for rhythmic freedom with the webcam. You can easily put in your own beats and change effects on the dance pad and add your own instruments to the webcam code. One issue I noticed is that the dac gets a little overloaded sometimes. When you have all the SndBufs, the drone, the flute, and the mandolins and shakers going it's too much for the computer to handle. Perhaps my code could be streamlined a bit or changed so it runs faster, but for now it'll have to do.