- 1 READ THIS FIRST
- 2 Getting started
- 2.1 1. Get a working version of ChucK, if you don't have one already.
- 2.2 2. Get the Wekinator: Choose 2a OR 2b
- 3 Running the Wekinator
- 4 Organization of the Wekinator
- 5 Implementing your own feature extractor in ChucK
- 6 Implementing your own feature extractor outside ChucK
- 7 Implementing your own synthesis class in ChucK
- 8 Implementing your own synthesis class outside ChucK
- 9 Implementing your own ScorePlayer in ChucK
- 10 Troubleshooting
READ THIS FIRST
A system for patching input features to output parameters
The Wekinator is, at heart, a system for patching input features to output parameters. Typically, your inputs will be gestural control signals (like an accelerometer indicating laptop tilt, or a joystick) or audio signals (like the audio captured from an acoustic flute player playing into a mic). The "features" are numbers describing the state of your inputs at a particular point in time. For example, for a joystick, we'd use a feature vector indicating whether each button is pressed and the current position of each axis. For audio, the raw audio samples are typically too complex a representation for use as a control input; instead we try to choose features that describe relevant aspects of the audio signal, such as the spectral centroid (which describes timbre).
At the other end of the system, you've got some parameters that you can plug into some piece of code that probably makes sound (or video, or something else, if you want), which we'll call your "synth." These parameters control your synth over time. For example, if you want your synth to play a melody, one parameter might control the pitch over time. Or if your synth is a drum machine, one parameter might control tempo, and another might control the number of loops currently playing.
You, the user, specify the relationship from input features to output parameters by supplying the Wekinator with a bunch of example input/output pairs. For example, if you want to use the Wiimote accelerometer to control the pitch of a sound, you'd provide training examples of Wiimote positions/gestures along with the corresponding synth parameters. The Wekinator's job is to learn a model of the relationship between features and parameters, which will produce parameter outputs for new inputs (even those that may be different from the training examples).
The picture below illustrates how a supervised learning algorithm can create a webcam-based hand gesture labeler, using a training set consisting of example gestures (inputs) and labels (outputs). The trained model is capable of applying appropriate labels to new hand gestures.
Option 1: Classification
The type of algorithm the Wekinator can use to learn relationships from input features to output parameters is dependent on the type of parameters. A classifier such as AdaBoost or k-nearest neighbor is used to classify features into discrete categories. For a gestural control system using the motion sensor, you might want to trigger a different, specific note when the laptop is tilting left versus when it is tilting right. In the Wekinator, discrete classes are represented as integers, and they are numbered starting from 0. So your training set would be constructed by giving the Wekinator some examples of you tilting left and specifying a corresponding parameter of 0, then you tilting right and specifying a parameter of 1. You would program your synth to play one pitch (say, middle C) whenever it received a parameter value of 0, and another pitch (say, A 440) whenever it received a 1.
Option 2: Neural networks
The other type of learning currently supported is a neural network, which can learn to output any real-valued parameter, not just integers. If you wanted to use a gestural control system in which the laptop motion sensor controlled your synth's pitch in a continuous way, you could use a neural network. For example, you could tilt all the way to the left and specify a parameter of 200, tilt in the middle and specify a parameter of 1000, and tilt to the right and specify a parameter of 500. Then in your synth, you could use the parameter value received and treat it as a frequency value; after doing some checking to make sure it's not below 0Hz and not above, say, 2000, you would set the frequency of an oscillator directly to this value.
1. Get a working version of ChucK, if you don't have one already.
- Install command line ChucK from http://chuck.cs.princeton.edu.
- I suggest installing chuck executable to /usr/bin/chuck if you're on OS X or linux, or copying bin\chuck.exe to c:\windows\system32\ if you're on Windows.
- You may also wish to install the miniAudicle, a ChucK IDE, if you want to experiment with writing ChucK code yourself: http://audicle.cs.princeton.edu/mini/.
2. Get the Wekinator: Choose 2a OR 2b
You have the choice of either downloading the very latest code (2a), with the possible downside that you have to build from source after each update (and get the svn and ant tools if you don't have them already), or of downloading the last packaged .zip (2b), with the possible downside that you'll have to be careful not to overwrite or lose track of any previously saved work.
2b is recommended for novices, 2a for people comfortable with the command line.
2a. Instructions for downloading the very latest code
Step 2.a.0 (setup: 1st time only):
- For Mac: Set up SVN and ant by installing OS X developer tools if you haven't already. (Not sure if you've got svn? Type "which svn" in the terminal. If you've got it, you'll see a line printed that looks something like "/usr/bin/svn"; if not, you won't see anything but another command prompt.)
- For Windows: Get SVN from http://subversion.apache.org/packages.html. Install it. Get ant and install it, for example from http://code.google.com/p/winant/downloads/list (this program may prompt you for the location of JDK on your computer-- assuming you have Java on your machine, it'll be something like C:\Program Files\Java\jdk_1.somethingsomething). Finally, restart your computer for good measure.
- For Linux: Get SVN and ant. You can do it.
Once you've got svn and ant, open up a terminal window and cd to a directory where you want the project to live from now on. For example you could cd to your desktop. Then type in the terminal:
svn checkout http://wekinator.googlecode.com/svn/trunk/ wekinator-read-only
This will grab you a copy of all the code.
Step 2.a.1 Do this every time you update:
Open terminal. Go to the wekinator-read-only directory. Type:
This will check out all the other changes into your directory. If you've made modifications to the code, it'll try to merge my changes with yours. Likewise, if you've created your own files, SVN will not move / delete / overwrite / change them.
Step 2.a.2 Do this after step 2.a.1, every time you update:
In terminal, go to directory wekinator-read-only/project/java/Wekinator/. At the prompt, type:
You should see "BUILD SUCCESSFUL" after this finishes running. If you don't see that, make sure you've done everything else correctly, otherwise email Rebecca because it's probably her fault!
(This creates and updates the java executable jar file, which we don't put in SVN.)
2b. Alternative instructions for those who don't want to use Terminal / SVN (But you'll have to do some manual directory management.)
Grab the latest .zip of the project (will require you to re-merge with your own files, if you've been working on things yourself) http://code.google.com/p/wekinator/downloads/list
Download it. Unzip it. Doesn't matter where, but avoid any place where a parent/ancestor directory has spaces or special characters in the directory name.
Feel free to unzip this to the same place every time, but be aware that any saved settings, learning systems, etc. that you've created in project/mySavedSettings/ will be overwritten unless you take care to copy them to another location first.
Running the Wekinator
0. Plug in any joysticks, controllers, etc. that you want to use.
1. Run the software.
- Find the directory where you downloaded the wekinator. For example, this directory might be /Users/rebecca/Dekstop/wekinator-read-only/ (if you downloaded from SVN) or /Users/rebecca/Desktop/wekinator/ . I'm going to call this directory WEKINATOR_DIR from now on.
- Go to WEKINATOR_DIR/project/java/Wekinator/dist/ and double-click on Wekinator.jar to run the Wekinator. (Alternatively, you can go to this directory in Terminal and type java -jar Wekinator.jar at the prompt.)
- You should now see a GUI pop up on your screen.
2. Optionally, start any external feature extractors or synthesis engines that haven't been started automatically in step 1. For example, if you want to use Max/MSP to extract features from a Wiimote, launch your max patch now (see below for details on how to configure your patch to actually send the Max data to the Wekinator). Or, say you want to use Wekinator to control a Max patch or Processing animation (i.e., a "synth" that's not a ChucK class): launch that code now, too.
3. The first time you run (and any time you update Wekinator by downloading the whole project .zip), you'll have to do some configuration for your system. Click on "Edit ChucK configuration...", then select the "System" tab.
- You'll need to specify the location of the ChucK executable on your system (i.e., where you downloaded it under "Getting started" above, probably /usr/bin/chuck if you're on a mac or linux and not a PLOrk student, [something]/PLOrk/bin/chuck if you're a PLOrk student, or C:\Windows\System32 if you're on a PC. Hit "Choose file," navigate to this file (not a directory), and select it.
- You'll then need to specify the location of the Wekinator project directory, which lives inside your wekinator project directory, which you've just installed/downloaded, and which we'll call WEKINATOR_DIR as I've said above. Hit "Choose directory," navigate to WEKINATOR_DIR/project, and select that directory.
- Hit "OK" to exit the configuration screen.
4. You'll probably need to do some additional configuration of ChucK (both the first time you run and in the future). Hit "Edit ChucK configuration" to configure the files that ChucK launches automatically for you:
- In the "Features (input)" tab, you can optionally enable a custom ChucK feature extractor. (Default is DISABLED.) This is what you'd use if you wrote your own piece of chuck code to extract features from something, say an arduino board, or a custom audio feature using UAnae. (Instructions for that are below.) If you did in fact do that, hit "Choose file" and select that .ck file, then enter in the number of features it extracts into the box. And check the checkbox. Otherwise leave it unchecked.
- Also in the "Features (input)" tab, you can optionally enable an OSC feature extractor. (Default value DISABLED.) This is what you'd use if you had a Max patch, custom Processing code, or some other piece of software outside of ChucK that's extracting features (inputs) and sending them out over OSC. (Instructions for setting up that patch/program are below.) If you want to do this, check the box and enter in the # of features you're receiving from that extractor. Otherwise leave it unchecked.
- In the "Synthesis (output)" tab, you must choose to use either a ChucK synth class (i.e., a piece of ChucK code that makes sound) or a synth running in some other environment (e.g., Max/MSP, Processing, etc.), which receives the Wekinator's output parameters via OSC.
- Selecting "Use a ChucK synth class" means that you're using the output of the Wekinator to control sound in a ChucK file. Hit "Choose file" and select the .ck that you want to run. Note that this file has to define a class called SynthClass that adheres to the Wekinator synth API (see below). There are some example synths ready to use in WEKINATOR_DIR/project/chuck/synths/, so you can choose one of these to start with.
- Selecting "Use a different Max/OSC synth module" means that you're using the output of the Wekinator to control sound/video/etc. in something other than ChucK, which will receive the Wekinator's output using OSC. You'll have to explicitly specify the number of parameters (i.e., output by Wekinator, and input by your synth) and whether they are real-valued or integer-valued (the Wekinator needs to know these things in order to know which types of learning algorithms to offer you). For integer-valued parameters, Wekinator needs to know how many values are legal; for example, if your synth expects a number from 0 to 3, you have a maximum of 4 values for that parameter. Finally, you must specify whether you want only the parameters from Wekinator, or if you want an array for each parameter specifying a probability distribution over the parameters. Just choose the parameters for now, the other option is buggy! Your synth should listen for these parameters on port 12000.
- In the "play-along learning" tab, you have the option of also loading a ChucK "playalong score." (Default DISABLED.) This is an advanced feature that allows you to create a ChucK "score" for your synth, which sets the parameters of your synth over time, bypassing the Wekinator system. This is used for play-along learning (see our ICMC 2009 demo). If you've created a score (see below), select Choose File and find that ChucK file; otherwise leave this unchecked. Make sure your score is sending the proper number and type of parameters for your synth. There are some sample scores included in WEKINATOR_DIR/project/chuck/score_players/.
- You can save this configuration for later use, if you like it. It'll also be loaded by default next time you launch wekinator, provided there are no errors.
- Hit OK!
5. Run the chuck component by hitting "Run" in the ChucK panel of the main GUI. You should see a status that tells you ChucK is running sucessfully, or be taken automatically to the "Features setup" tab.
- Troubleshooting: If ChucK does not run successfully, you can try to debug by going to View->Console in the top menu, then try running Chuck again. Most likely you did not specify your chuck executable or wekinator chuck directory correctly in the Chuck configuration, or there is a typo in one of the ChucK files you're running.
6. Once connected, go to the "Features Setup" tab. (You should be taken there automatically.)
- Check off any features that you want to use. This tells Wekinator what inputs you're going to use to control the sound. You can use multiple types of features. For starters, try built-in features like the motion sensor (Mac only).
- If you're running a custom OSC feature extractor or a custom ChucK feature extractor, check those boxes and enter the number of features being extracted.
- Optionally, you can add "meta-features" to these features. By selecting one or more features, then clicking "add meta-features from these features..." you can also extract 1st- and 2nd-order differences (estimating velocity and acceleration), a smoothed version (for noisy inputs), and/or a history buffer (i.e., a history length of 10 will store the last 10 values of each feature). Check the boxes of the features you want to use.
- When you're done choosing your features, hit GO!
7. Now you should be in the "Learning Setup" tab.
- In the "Advanced" subtab ("Simple" is disabled, sorry):
- The first time you run, check the radio button to "Create new dataset."
- For each parameter (i.e., each number output by Wekinator and input by your synth), you'll be able to choose the type of algorithm (a classifier if it's discrete, or a neural network if it's continuous) and choose which of the input features affect it. For example, if I have 3 synth parameters, and I'm using the trackpad as my input device, and I want the x position to affect parameters 0 and 1 and the y position to affect parameter 2 only, I would click "View & choose features" for each of the 3 params. For param 0 and param 1, I'd check only "Trackpad_0" feature; for param 2, I'd check only "Trackpad_1".
- When you're done configuring your models, hit GO!
8. Now you should be on the "Use it!" tab.
- COLLECT DATA:
- This view allows you to collect data used to train your model, telling the Wekinator about your intended relationship between input features and output parameters.
- For starters, put some numbers into the number boxes and hit "play" to hear what different parameter settings sound like with your synth. When you hear something you like, configure your inputs in a way that you want to correspond to that sound (e.g., by moving the laptop into a certain tilt position, if you're using the motion sensor features). Then hit "Begin recording examples into dataset" to collect some data, then "Stop recording" when you're done. Do this a few times, for different parameters and features.
- You can see exactly what Wekinator has recorded by hitting the "view examples" button at the bottom of this view. It's basically a spreadsheet where each column is a parameter or feature, and each row is a "snapshot" of features and the corresponding parameters at a moment in time.
- Advanced interaction: TODO explain this
- Once you've collected some data, you can train your model(s) (one model will be trained for each parameter).
- The simplest way to use the TRAIN view is to just hit "Train now."
- Advanced interaction: TODO explain this
- Once you've trained your models, you can run them!
- In the RUN view, hit the "run" button on the top right. This will send the input features to the model, get outputs from the model, and send them as parameters to the synth, repeatedly in real-time. Hit "stop" when you're done. You should hear your synth changing and see the parameters being updated in the GUI.
- Advanced interaction: TODO explain this
- CONFIGURE & EVALUTE:
- You can optionally reconfigure your models and evaluate them using cross-validation accuracy. TODO explain this. (advanced)
- Feel free to go back and collect more data and re-train and re-run your models as many times as you want. (You'll have to re-train after changing your dataset, for the changes to go into effect when you run.) You can also go back to the Features Setup and Model Setup tabs and change those settings.
Organization of the Wekinator
- TODO show picture of architecture
- The GUI is in Java. You won't need to recode any of this (but Rebecca will!). The GUI jar file lives in dist/wekinator.jar.
- The core ChucK system is in the core_chuck/ directory.
- You shouldn't have to change any of this.
- The main ChucK code responsible for communicating with Java and the feature extractors is main_chuck.ck.
- The ChucK synths are in the chuck/synths/ directory. Each synth implements a class called SynthClass, the structure of which you can read about below. If you want to create your own ChucK synth, you can start by editing synth_skeleton.ck.
- In play-along learning, a ScorePlayer object sends parameters to a synth object. (A score is simply a set of instructions at specific points in time.) The ScorePlayers live in the chuck/score_players/ directory. You don't have to implement a score player for your synth unless you want to do playalong learning from a ChucK score.
- A feature extractor is a module that extracts features (also called attributes) from your inputs. For example, if you're using an audio input, a feature might be an FFT, or the peak amplitude of the audio signal. If you're using the laptop motion sensor, a feature might just be the values of each of the 3 sensor outputs. Your input is fully described to the wekinator by a vector of real-valued features, which typically describe the state of your input at a single point (snapshot) in time (though you could imagine a feature vector that describes its current state and its previous state). Custom feature extractors in ChucK live in the chuck/feature_extractors/ directory. They implement the CustomFeatureExtractor class. You can add your own custom feature extractor in chuck by editing CustomFeatureExtractor.ck (see example_centroid.ck for a simple example). You can also choose from existing custom feature extractors for more specialized audio or gestural features.
- You can integrate synths in environments outside ChucK. For example, see the chuck/processing_synths directory. More info on how to do this below.
- You can also integrate feature extractors in environments outside ChucK. See below.
Implementing your own feature extractor in ChucK
Your custom feature extractor will live in a class named CustomFeatureExtractor. We recommend that you start with this file provided called CustomFeatureExtractor.ck (in the WEKINATOR_DIR/chuck/feature_extractors/ directory), which you can edit according to the "TODO" instructions in the code. Basically, you'll have to implement function bodies for setting up the feature extractor, and most importantly computing the features.
If you edit the code provided and put the computed features in the features array of your class, you're good to go! For an example of how to implement the setup and feature computation, check out WEKINATOR_DIR/chuck/feature_extractors/example_centroid.ck.
To run, choose your custom chuck feature extractor from the ChucK configuration pane in the GUI. Make sure you set the proper number of features.
Also, when you run the GUI, make sure to check the box that says "Custom ChucK features" on the Features panel, and set the number of features to the number of features your feature extractor actually extracts (numFeats in the skeleton code).
Implementing your own feature extractor outside ChucK
All your feature extractor needs to do is send a feature vector to the Wekinator via OSC at the (possibly variable) rate of its choosing. The feature vector is a list of floats (MUST be floats), of arbitrary length (though this length must not change over time, and the order of the features must not change over time). Your extractor sends the features in one OSC message, "/oscCustomFeatures", to port 6448.
The feature extractor can also optionally send an OSC message containing the feature names, so that these names are displayed in the Wekinator GUI. This message uses the string "/oscCustomFeaturesNames" and contains a list of string names (one per feature, in the same order as the feature values are passed in the oscCustomFeatures message).
To run, start your feature extractor before starting the Java GUI. In the ChucK configuration, in the Features tab, check the box to indicate that you are using a custom OSC feature extractor, and indicate the number of features. Then, after running ChucK, in the main Wekinator "Features" panel, enter the number of custom OSC features to be equal to the length of the feature vector specified in your oscCustomFeatures message above. (Yes, this is redundant for now -- a fix will come soon.)
Implementing your own synthesis class in ChucK
Your synthesis code will live in a ChucK class called SynthClass and implement a pre-defined set of functions.
We recommend starting with synths/synth_skeleton.ck, which has instructions on what you should edit to make your synth work.
You'll need to implement the following functions:
- fun int isDiscreteArray() : return an array whose length is equal to the number of parameters; each element is 1 if you want discrete classifier outputs, 0 if you want continuous outputs, for the corresponding parameter
- fun int getNumClassesArray() : return an array whose length is equal to the number of parameters; each element is the number of classes expected by your synth for the corresponding parameter (where a class here means a member of the set of discrete outputs of a classifier like kNN; for example, to classify gestures into "moving" and "not moving", you would have 2 classes). Only matters if doing discrete classification.
- fun int useDistributionArray() : return an array whose length is equal to the number of parameters; each element corresponds to a parameter. For each parameter, if you're using a discrete classifier, the array element is 1 if you want a vector of class membership probabilities for each output, or 0 if you only want a single class label for each output. If you're not discrete, it doesn't matter.
- fun int getNumParams() : the number of outputs expected from the model(s). For discrete or continuous learning. For example, to drive the frequency and volume of a SinOsc independently, you would use 2 parameters.
- fun void setup() : This function is called just once, when the system is setting up. Treat this like a constructor. In other words, do any work here that should be done before sound is made, like sporking shreds that should be running the whole time your synth is running (e.g., for smoothing parameters).
- fun void setParams(float params) : sets your synthesis parameters according to the contents of params, which will be of length equal to the number of parameters if you are doing continuous or discrete without distribution, or length equal to (# parameters) x (# classes) if you are doing discrete with the probability distribution.
- In other words, this is the most important part of your synth! It describes how you're using the outputs of the model.
- Note: You probably want to do some error checking here. Ideally, params will be of the appropriate size, but you should check. And ideally the values in this array will be what you'd expect, but you should never assume this is the case, particularly when using a neural network for learning a continuous value. Your "frequency" parameter value might come in as -1, 0, or 500,000 for example: it's your call how to use (or ignore) these types of values.
Your synth has a few other functions that need to be present, but which you shouldn't need to edit by default.
- sound() and silent() turn the sound on and off. In the examples, I use a main envelope called "e," and I patch all my objects into e instead of into the dac. sound() ramps up e, and silent() ramps down to 0. If you want to do something more complex, for example turning off expensive processing when your synth is silent, go ahead and edit these functions.
- Other functions: you really shouldn't edit. Just start from the skeleton, keep them at the bottom of your code, and ignore them.
When you've implemented your class, make sure you load it in the ChucK configuration panel in the Wekinator GUI.
Implementing your own synthesis class outside ChucK
Editing your synth
Your synth should listen on OSC port 12000 (fixed for now) for the message /OSCSynth/params containing the parameters as floats.
In order to set up Wekinator to send parameter values to your synth, edit the ChucK configuration upon opening the Wekinator. In the Synthesis (Output) tab, select the option to control an OSC synth module, and configure this module using the "Configure" button, adding the appropriate number and types of parameters to control your synthesis module.
If your synth contains its own GUI or other mechanism for changing parameter values independently of the Wekinator GUI (e.g., such as hooking your Max/MSP synth up to some max sliders), you'll want to communicate the parameter changes back to the Wekinator by sending the OSC message "/realValue" followed by the parameters as floats, to port 6448.
Play-along learning with an external synth
Currently, you will still use a ScorePlayer written in ChucK to control your OSC synth. Your ScorePlayer looks the same regardless of the type of synth.
Implementing your own ScorePlayer in ChucK
This is a lot like implementing your own synth class. Start by copying one of the standard examples, like score_players/icmc_melody.ck. Look at the "TODO" statements in icmc_melody.ck to see what you need to edit.
Your score must know the number of parameters required by your synth, but otherwise it is not tied to your particular synth. A score player can be compatible with multiple synths; a synth can be compatible with multiple score players.
Your new file will have to implement the ScorePlayer class, along with some required functions.
- fun void setup(SynthClass s, OscSend x) : Do any additional setup here
- fun void playScore() : in the simplest case, this will include a while(isPlaying) loop that sends new parameters to the SynthClass object over time, as long as isPlaying is true
- fun void sendMessage() : you may want to show the user a text message informing them what is happening in the score, and what the current parameter values are. This is optional.
There are a few other functions in the skeleton code that you can leave as-is (no need to edit).
- I can't establish an OSC connection from the GUI! Help!
- Wekinator requires that you only have one instance of the GUI and one instance of the ChucK code running at once. Make sure you don't have miniAudicle running. Try restarting both ChucK and the GUI and reconnecting. Finally, double check that the ports in the GUI match the ports ChucK is expecting (specified in main_chuck_playalong.ck).
- My controller isn't recognized! Help!
- Make sure you plug in the controller before running your chuck code (or starting miniAudicle).
- Every now and then a controller will fail to play nicely with ChucK. First, note that the controller configuration works only for controllers that act as HID joysticks or mice. If this describes your controller, check that ChucK recognizes it (outside the Wekinator) by running this code in miniAudicle or command line. If that code works, you've discovered a Wekinator bug and you should let us know. Otherwise, you may want to investigate alternative HID input environments (e.g., maybe Max/MSP will recognize your HID?), and you should follow the instructions above on implementing your own feature extractor outside ChucK.
- Neural net training is taking a long time! Help!
- Training time will increase if you are using a lot of features and/or asking the network to learn a more complex function. Try decreasing the learning rate (e.g., to 0.03 or lower). (This requires that you check the checkbox with the option to view the NN training GUI.) Also keep an eye on the epoch error rate. If it's relatively low, but the training goes on, you can stop the training early by hitting "stop" and then "accept."
- I get errors when I try to run the ChucK code.
- Make sure you are running a file containing all the "Machine.add" statements (i.e., one in main_machine_add_files/) and not the individual chuck classes. This file will add them in the correct order for you.
- Make sure you are running from the WEKINATOR_PATH directory (your top-level directory), and using the --caution-to-the-wind flag.
- If you've written your own synth or feature extraction code, you can do basic debugging yourself by running ONLY that code in miniAudicle (it'll catch syntax errors). You can also write some test scripts for your classes, like I've included in the tests/ directory.
- ChucK is core dumping / seg faulting / quitting! Help!
- ChucK does this now and then. First, we STRONGLY recommend running ChucK in the command line instead of in the miniAudicle. If you still experience problems, and you can reproduce this behavior reliably, write a description on the ChucK/Wekinator/Bugs wiki and we'll look into it.