From CSWiki
Jump to: navigation, search

iPhone => ChucK

A project to natively read in data sent over OSC from the iPhone app OSCemote in ChucK.


This project began when I realized there was a bunch of iPhone apps in the App Store that could send Open Sound Control (OSC) messages to a computer. Many of these apps had patches/programs that connected to music-producing software like Max/MSP, PureData, and Ableton, but nothing for ChucK, so I set out to make one of the many iPhone-OSC apps talk to ChucK natively. If we could get one of these apps to talk to ChucK we could potentially compose a PLOrk piece with iPhones as the main device for musical production. I ended up going with the OSCemote app ($4.99):


The OSCemote apps works on both the iPhone and iPod Touch. It comes with four dashboards that show what can be done with the app. You can also design your own dashboards with Dashcode and use them for sending OSC messages instead of using the provided dashboards. To get OSCemote to communicate with your computer you need to specify the host's IP address (find it in System Preference -> Network) and the port number to send messages over (find this in the ChucK code) in the Settings pane of the program. A word of warning - OSCemote sometimes just will not communicate right off the bat. I found changing the port number in the ChucK program and then changing the port number in the Settings pane usually fixed any connection problems.

All of the code for this project can be found here. The code includes a bunch of files. Here is an explanation of what each file does:

  • - The class file that is used to create OSCemote objects. OSCemote objects read in data from the four standard dashboards that come with OSCemote and store it in global variables that can be accessed directly.

The following files need to be run alongside OSCemote. You can run things in MiniAudicle by first adding the OSCemote to the VM and then adding/removing each file, but I recommend you just run everything in the Terminal by cd-ing to the directory and running it with the chuck command line interface (e.g. chuck

  • - This is a simple tester/debugger for OSCemote. It creates an OSCemote object and then prints out all the data that OSCemote can read. Try running this one first to make sure your iPhone is connecting to your computer.
  • - This program uses the Buttons dashboard of OSCemote. It maps a different pitch on a PercFlute STK object to each of the 15 buttons on the screen. When you press a button a note is played. You can play multiple notes by holding down multiple buttons simultaneously.
  • - This program uses the Sliders dashboard of OSCemote. I adapted my Drum Machine to make use of this dashboard - there are six STKs playing different beats. Each slider is mapped to the gain of an STK so you can turn off a particular instrument or bring it out above all others. The letters buttons lets you change which instrument is playing which beat. The switches on the bottom print out their status to the console (not musical, but it works).
  • - This program uses the Multi-Touch dashboard of OSCemote. You can create up to five sine waves by placing up to five fingers on the iPhone (OSCemote does not support more fingers, but five is more than enough on the screen you're placing it on). Gain and frequency are manipulated by moving across the x and y axes respectively. KNOWN ISSUE: Removing your fingers should cut the gain on all the sine waves, but sometimes one or two may still be sounding.
  • - This program uses the Accelerometer dashboard of OSCemote. There are three standard UGens: SinOsc, PulseOsc, and a Phasor; each of their gains is mapped to an axis on the iPhone's accelerometer. Tilt the iPhone around to change the relative gains of each unit generator.

Each of these files works with a particular dashboard present in OSCemote. It certainly is possible to use all four dashboards in the same program which would allow for much more interesting, complex pieces. Also, as long as the accelerometer is enabled in the Accelerometer dashboard, data will be sent to the OSCemote class even if you are not in the Accelerometer dashboard! Here is another piece that utilizes both the MultiTouch and accelerometer panes/features:

  • - A very crude implementation of a lightsaber with powering on, powering off, humming, and swing. You turn it on by sliding your finger up quickly on the multitouch pane and turn it off by sliding your finger down quickly on the same pane. It turns on and off based on the movement vectors so do it like you mean it! The accelerometer (make sure it's on) will be reading data all throughout and when you swing the phone, a swing sound will activate. Use the force!

Video Demos

I recorded demonstrations of some of the above programs. Download the movies (.m4v) and check it out!

  • Ocarina ( video demonstration here.
  • Lightsaber ( video demonstration here.
  • Touch sine waves ( video demonstration here.

The Internals

To get this to work I first ran a PureData example off the OSCemote webpage that showed all the OSC messages being sent when the features in OSCemote were used. For example, each button on the Buttons screen, and each slider on the Sliders screen sent its own unique OSC message. So to get all the data I created an OscRecv object, set the port to the port OSCemote was sending data over and set that OscRecv object to listen. I then sporked shreds that would each create an event for each button/slider and then wait for that event. When a button/slider was manipulated, the particular shred would wake up, update the global variable for that action and then wait for the button/slider to be pressed again. The Accelerometer window required only a single shred as one OSC message contained the info on all three axes.

Getting the MultiTouch to communicate with ChucK was a bit more complicated because of the nature of messages that were sent from OSCemote. Each interaction with the MultiTouch window resulted in a group of messages being sent. Here is an example:

print: /tuio/2Dcur set 1 0.453125 0.347932 0 0 0

print: /tuio/2Dcur set 2 0.35625 0.734793 0 0 0

print: /tuio/2Dcur set 3 0.76875 0.111922 0 0.148027 9.08392

print: /tuio/2Dcur alive 1 2 3

print: /tuio/2Dcur fseq 40

The fseq message is a sequence ID sent with every group of messages and increases sequentially. The alive message is followed by a variable number of integers. The number of integers is how fingers are currently touching the screen and the actual numbers serve as identifiers for each finger that can be used to link finger to the data for that finger. When all fingers are removed, an alive message is sent with no integers attached. The set messages include the finger ID as well as the data for that finger. The first two numbers represent the position of the finger on the screen (x axis, y axis). The latter three numbers represent the x and y movement vectors and motion acceleration (big thanks to Cameron Britt for pointing that out to me).

To get ChucK to read in and parse all this data I created 5 OscEvent objects that each can read the alive messages followed by the variable number of integers. Since every group of messages includes the fseq message I halt the program (i.e. the oe => now business) and wait for that message to arrive. When it arrives, I then check on the 5 alive OscEvent objects and ask if anything of them have any new messages (using the if conditional and the nextMsg() method). Which ever alive message triggers tells me how many fingers are touching the screen and how many set messages to expect. I then loop over a for loop that reads in specified number of set messages and stores the data in a multi-dimensional array that the user can access. The only remaining issue is that not all set messages are sent with 5 floats. As you can see in the example messages above, the messages vary between sending integers and floats. To handle this, I just created a bunch of OscEvent objects that wait for different types of formats and then check which one is being used by just using the same process that is used to count the number of fingers.

Known Issues / Future Improvements

  • The current limitation of my OSCemote class is that it does not handle the finger IDs sent with the alive message. This presents problems for storing data when you are removing certain fingers from the screen and adding others while keeping some on. In the future, I hope to do some bookkeeping and be able to keep data associated with a finger in the same position in the array of stored values even as fingers come and go.
  • Also in the multitouch pane, removing fingers sometimes does not reset the array values are stored in.
  • After getting multitouch to work, I think it may be possible to make all other data readers into single functions so I do not have to spork tons of shreds to get all data to be read in.
  • Play around with creating my own dashboards and make something PLOrk-piece-worthy.