From CSWiki
Revision as of 13:20, 4 July 2010 by RebeccaFiebrink (talk | contribs) (READ THIS FIRST)

Jump to: navigation, search


The Wekinator is, at it heart, a system for patching input features to output parameters. Typically, your inputs will be gestural control signals (like an accelerometer indicating laptop tilt, or a joystick) or audio signals (like the audio captured from an acoustic flute player playing into a mic). The "features" are numbers describing the state of your inputs at a particular point in time. For example, for a joystick, we'd use a feature vector indicating whether each button is pressed and the current position of each axis. For audio, the raw audio samples are typically too complex a representation for use as a control input; instead we try to choose features that describe relevant aspects of the audio signal, such as the spectral centroid (which describes timbre).

At the other end of the system, you've got some parameters that you can plug into some piece of code that probably makes sound (or video, or something else, if you want), which we'll call your "synth." These parameters control your synth over time. For example, if you want your synth to play a melody, one parameter might control the pitch over time. Or if your synth is a drum machine, one parameter might control tempo, and another might control the number of loops currently playing.


Example-based learning

You, the user, specify the relationship from input features to output parameters by supplying the Wekinator with a bunch of example input/output pairs. For example, if you want to use the Wiimote accelerometer to control the pitch of a sound, you'd provide training examples of Wiimote positions/gestures along with the corresponding synth parameters. The Wekinator's job is to learn a model of the relationship between features and parameters, which will produce parameter outputs for new inputs (even those that may be different from the training examples).

Option 1: Classification

The type of algorithm the Wekinator can use to learn relationships from input features to output parameters is dependent on the type of parameters. A classifier such as AdaBoost or k-nearest neighbor is used to classify features into discrete categories. For a gestural control system using the motion sensor, you might want to trigger a different, specific note when the laptop is tilting left versus when it is tilting right. In the Wekinator, discrete classes are represented as integers, and they are numbered starting from 0. So your training set would be constructed by giving the Wekinator some examples of you tilting left and specifying a corresponding parameter of 0, then you tilting right and specifying a parameter of 1. You would program your synth to play one pitch (say, middle C) whenever it received a parameter value of 0, and another pitch (say, A 440) whenever it received a 1.

Option 2: Neural networks

The other type of learning currently supported is a neural network, which can learn to output any real-valued parameter, not just integers. If you wanted to use a gestural control system in which the laptop motion sensor controlled your synth's pitch in a continuous way, you could use a neural network. For example, you could tilt all the way to the left and specify a parameter of 200, tilt in the middle and specify a parameter of 1000, and tilt to the right and specify a parameter of 500. Then in your synth, you could use the parameter value received and treat it as a frequency value; after doing some checking to make sure it's not below 0Hz and not above, say, 2000, you would set the frequency of an oscillator directly to this value.

Getting started

1. Get a working version of ChucK, if you don't have one already.

2. Get the Wekinator: Choose 2a OR 2b

2a. Instructions for downloading the very latest code

Step 2.a.0 (setup: 1st time only):

Set up SVN by installing OS X developer tools if you haven't already. (Not sure if you've got svn? Type "which svn" in the terminal. If you've got it, you'll see a line printed that looks something like "/usr/bin/svn"; if not, you won't see anything but another command prompt.)

Once you've got svn, open up terminal and cd to a directory where you want the project to live from now on. For example you could cd to your desktop. Then type in the terminal:

svn checkout wekinator-read-only

This will grab you a copy of all the code.

Step 2.a.1 Do this every time you update:

Open terminal. Go to the wekinator-read-only directory. Type:

svn update

This will check out all the other changes into your directory. If you've made modifications to the code, it'll try to merge my changes with yours. Likewise, if you've created your own files, SVN will not move / delete / overwrite / change them.

Step 2.a.2 Do this after step 2.a.1, every time you update:

In terminal, go to directory java/Wekinator/. At the prompt, type:


You should see "Build successful" after this finishes running. If you don't see that, make sure you've done Step 1, otherwise email Rebecca because it's probably her fault!

(This updates the java executable file, which we don't put in SVN.)

2b. Alternative instructions for those who don't want to use Terminal / SVN (But you'll have to do some manual directory management.)

Grab the latest .zip of the project (will require you to re-merge with your own files, if you've been working on things yourself)

Download it. Unzip it. That's it.

Running the Wekinator

0. Plug in any joysticks, controllers, etc. that you want to use.

1. Run the software.

  • Find the directory where you downloaded the wekinator. For example, this directory might be /Users/rebecca/Dekstop/wekinator-read-only/ (if you downloaded from SVN) or /Users/rebecca/Desktop/wekinator/ . I'm going to call this directory WEKINATOR_DIR from now on.
  • Go to WEKINATOR_DIR/project/java/Wekinator/dist/ and double-click on Wekinator.jar to run the Wekinator. (Alternatively, you can go to this directory in Terminal and type java -jar Wekinator.jar at the prompt.)
  • You should now see a GUI pop up on your screen.

2. Optionally, start any external feature extractors or synthesis engines that haven't been started automatically in step 1. For example, if you want to use Max/MSP to extract features from a Wiimote, launch your max patch now (see below for details on how to configure your patch to actually send the Max data to the Wekinator). Or, say you want to use Wekinator to control a Max patch or Processing animation: launch that code now, too. 3. The first time you run, you'll have to do some configuration for your system. In the "Chuck configuration," click on "Edit...". This is what you'll see:

  • Enable a custom ChucK feature extractor: Default value DISABLED. This is what you'd use if you wrote your own piece of chuck code to extract features from something, say an arduino board, or a custom audio feature using UAnae. (Instructions for that are below.) If you did in fact do that, hit "Choose file" and select that .ck file, then enter in the number of features it extracts into the box. And check the checkbox. Otherwise leave it unchecked.
  • Enable an OSC feature extractor: Default value DISABLED. This is what you'd use if you had a Max patch or custom Processing code that you wanted to use to extract features (e.g., from a Wiimote, 3D navigator button that works with Max, etc.). (Instructions for setting up that patch are below.) If you want to do this, check the box and enter in the # of features you're receiving from that extractor. Otherwise leave it unchecked.
  • Use a ChucK synth class: Default ENABLED. Selecting this option means that you're using the output of the Wekinator to control sound in a ChucK file. Hit "Choose file" and select the .ck that you want to run. Note that this file has to define a class called SynthClass that adheres to the Wekinator synth API (see below).
  • Use an OSC synth/composition module: Default DIASBLED. Selecting this option means that you're using the output of the Wekinator to control sound/video/etc. in something other than ChucK, which will receive the Wekinator's output using OSC. You'll have to explicitly specify the number of parameters (i.e., output by Wekinator, and input by your synth) and whether they are real-valued or integer-valued (the Wekinator needs to know these things in order to know which types of learning algorithms to offer you). For integer-valued parameters, Wekinator needs to know how many values are legal; for example, if your synth expects a number from 0 to 3, you have a maximum of 4 values for that parameter. Finally, you must specify whether you want only the parameters from Wekinator, or if you want an array for each parameter specifying a probability distribution over the parameters. Just choose the parameters for now, the other option is buggy! Finally, you'll have to specify which OSC port your synth is listening to; Wekinator will send the parameters there.
  • Enable play-along learning: Default DISABLED. This is an advanced feature that allows you to create a ChucK "score" for your synth, which sets the parameters of your synth over time, bypassing the Wekinator system. This is used for play-along learning. If you've created a score (see below), select Choose File and find that ChucK file; otherwise leave this unchecked.
  • VERY IMPORTANT: location of chuck executable: You must locate the ChucK binary on your system. If you are on OS X and installed ChucK to the default location, it'll be at /usr/bin/chuck.
  • VERY IMPORTANT: location of Wekinator's ChucK directory on your system: You must manually locate the chuck directory within wekinator, i.e. WEKINATOR_DIR/project/chuck/ If you don't do this right, nothing will work.
  • You can save this configuration for later use, if you like it. It'll also be loaded by default next time you launch wekinator, provided there are no errors.
  • Hit OK!

4. Run the chuck component by hitting "Run" in the ChucK panel of the main GUI. You should see a status that tells you ChucK is running sucessfully. If you don't see this, you can try to debug by going to View->Console in the top menu, then try running Chuck again. Most likely you did not specify your chuck executable or wekinator chuck directory correctly in the Chuck configuration, or there is a typo in one of the ChucK files you're running. 5. Once ChucK is running click "Connect" in the OSC pane of the GUI. This will get Java talking to ChucK. If you can't connect, try stopping and restarting ChucK. Again, using View->Console may help you to debug. 6. Once connected, go to the "Features Setup" tab.

  • Check off any features that you want to use. This tells Wekinator what inputs you're going to use to control the sound. You can use multiple types of features. For starters, try built-in features like the motion sensor (Mac only).
  • Optionally, you can add "meta-features" to these features. By selecting one or more features, then clicking "add meta-features from these features..." you can also extract 1st- and 2nd-order differences (estimating velocity and acceleration), and a smoothed version (for noisy inputs). Check the boxes of the features you want to use.
  • When you're done choosing your features, hit GO!

5. Go to the "Model Setup" tab.

  • In the "Advanced" subtab:
    • The first time you run, check the radio button to "Create new dataset."
    • For each parameter (i.e., number output by Wekinator and input by your synth), you'll be able to choose the type of algorithm (if it's discrete) and choose which of the input features affect it. For example, if I have 3 synth parameters, and I'm using the trackpad as my input device, and I want the x position to affect parameters 0 and 1 and the y position to affect parameter 2 only, I would click "View & choose features" for each of the 3 params. For param 0 and param 1, I'd check only "Trackpad_0" feature; for param 2, I'd check only "Trackpad_1".
    • When you're done configuring your models, hit GO!

6. Go to the "Use it!" tab to use your model.

    • This view allows you to collect data used to train your model, telling the Wekinator about your intended relationship between input features and output parameters.
    • For starters, put some numbers into the number boxes and hit "play" to hear what different parameter settings sound like with your synth. When you hear something you like, configure your inputs in a way that you want to correspond to that sound (e.g., by moving the laptop into a certain tilt position, if you're using the motion sensor features). Then hit "Begin recording examples into dataset" to collect some data, then "Stop recording" when you're done. Do this a few times, for different parameters and features.
    • You can see exactly what Wekinator has recorded by hitting the "view examples" button at the bottom of this view. It's basically a spreadsheet where each column is a parameter or feature, and each row is a "snapshot" of features and the corresponding parameters at a moment in time.
    • Advanced interaction: TODO explain this
  • TRAIN:
    • Once you've collected some data, you can train your model(s) (one model will be trained for each parameter).
    • The simplest way to use the TRAIN view is to just hit "Train now."
    • Advanced interaction: TODO explain this
  • RUN:
    • Once you've trained your models, you can run them!
    • In the RUN view, hit the "run" button on the top right. This will send the input features to the model, get outputs from the model, and send them as parameters to the synth, repeatedly in real-time. Hit "stop" when you're done. You should hear your synth changing and see the parameters being updated in the GUI.
    • Advanced interaction: TODO explain this
    • You can optionally reconfigure your models and evaluate them using cross-validation accuracy. TODO explain this. (advanced)

7. Interaction!

  • Feel free to go back and collect more data and re-train and re-run your models as many times as you want. (You'll have to re-train after changing your dataset, for the changes to go into effect when you run.) You can also go back to the Features Setup and Model Setup tabs and change those settings.

Organization of the Wekinator

  • TODO show picture of architecture
  • The GUI is in Java. You won't need to recode any of this (but Rebecca will!). The GUI jar file lives in dist/wekinator.jar.
  • The core ChucK system<b/> is in the core_chuck/ directory.
    • You shouldn't have to change any of this.
    • The main ChucK code responsible for communicating with Java and the feature extractors is
  • The ChucK <b>synths are in the chuck/synths/ directory. Each synth implements a class called SynthClass, the structure of which you can read about below. If you want to create your own ChucK synth, you can start by editing
  • In play-along learning, a ScorePlayer object sends parameters to a synth object. (A score is simply a set of instructions at specific points in time.) The ScorePlayers live in the chuck/score_players/ directory. You don't have to implement a score player for your synth unless you want to do playalong learning from a ChucK score.
  • A feature extractor is a module that extracts features (also called attributes) from your inputs. For example, if you're using an audio input, a feature might be an FFT, or the peak amplitude of the audio signal. If you're using the laptop motion sensor, a feature might just be the values of each of the 3 sensor outputs. Your input is fully described to the wekinator by a vector of real-valued features, which typically describe the state of your input at a single point (snapshot) in time (though you could imagine a feature vector that describes its current state and its previous state). Custom feature extractors in ChucK live in the chuck/feature_extractors/ directory. They implement the CustomFeatureExtractor class. You can add your own custom feature extractor in chuck by editing (see for a simple example). You can also choose from existing custom feature extractors for more specialized audio or gestural features.
  • You can integrate synths in environments outside ChucK. For example, see the chuck/processing_synths directory. More info on how to do this below.
  • You can also integrate feature extractors in environments outside ChucK. See below.

Implementing your own feature extractor in ChucK

Your custom feature extractor will live in a class named CustomFeatureExtractor. We recommend that you start with this file provided called (in the WEKINATOR_DIR/chuck/feature_extractors/ directory), which you can edit according to the "TODO" instructions in the code. Basically, you'll have to implement function bodies for setting up the feature extractor, and most importantly computing the features.

If you edit the code provided and put the computed features in the features[] array of your class, you're good to go! For an example of how to implement the setup and feature computation, check out WEKINATOR_DIR/chuck/feature_extractors/

To run, choose your custom chuck feature extractor from the ChucK configuration pane in the GUI. Make sure you set the proper number of features.

Also, when you run the GUI, make sure to check the box that says "Custom ChucK features" on the Features panel, and set the number of features to the number of features your feature extractor actually extracts (numFeats in the skeleton code).

Implementing your own feature extractor outside ChucK

All your feature extractor needs to do is send a feature vector to the Wekinator via OSC at the (possibly variable) rate of its choosing. The feature vector is a list of floats (MUST be floats), of arbitrary length (though this length must not change over time, and the order of the features must not change over time). Your extractor sends the features in one OSC message, "/oscCustomFeatures", to the port specified as recvPort in main_chuck (default is 6453).

To run, start your feature extractor before starting the Java GUI. In the "Features" panel, enter the number of custom OSC features to be equal to the length of the feature vector specified in your oscCustomFeatures message above.

That's all!

Implementing your own synthesis class in ChucK

Your synthesis code will live in a ChucK class called SynthClass and implement a pre-defined set of functions.

We recommend starting with synths/, which has instructions on what you should edit to make your synth work.

You'll need to implement the following functions:

  • fun int[] isDiscreteArray() : return an array whose length is equal to the number of parameters; each element is 1 if you want discrete classifier outputs, 0 if you want continuous outputs, for the corresponding parameter
  • fun int[] getNumClassesArray() : return an array whose length is equal to the number of parameters; each element is the number of classes expected by your synth for the corresponding parameter (where a class here means a member of the set of discrete outputs of a classifier like kNN; for example, to classify gestures into "moving" and "not moving", you would have 2 classes). Only matters if doing discrete classification.
  • fun int[] useDistributionArray() : return an array whose length is equal to the number of parameters; each element corresponds to a parameter. For each parameter, if you're using a discrete classifier, the array element is 1 if you want a vector of class membership probabilities for each output, or 0 if you only want a single class label for each output. If you're not discrete, it doesn't matter.
  • fun int getNumParams() : the number of outputs expected from the model(s). For discrete or continuous learning. For example, to drive the frequency and volume of a SinOsc independently, you would use 2 parameters.
  • fun void setup() : This function is called just once, when the system is setting up. Treat this like a constructor. In other words, do any work here that should be done before sound is made, like sporking shreds that should be running the whole time your synth is running (e.g., for smoothing parameters).
  • fun void setParams(float params[]) : sets your synthesis parameters according to the contents of params[], which will be of length equal to the number of parameters if you are doing continuous or discrete without distribution, or length equal to (# parameters) x (# classes) if you are doing discrete with the probability distribution.
    • In other words, this is the most important part of your synth! It describes how you're using the outputs of the model.
    • Note: You probably want to do some error checking here. Ideally, params[] will be of the appropriate size, but you should check. And ideally the values in this array will be what you'd expect, but you should never assume this is the case, particularly when using a neural network for learning a continuous value. Your "frequency" parameter value might come in as -1, 0, or 500,000 for example: it's your call how to use (or ignore) these types of values.

Your synth has a few other functions that need to be present, but which you shouldn't need to edit by default.

  • sound() and silent() turn the sound on and off. In the examples, I use a main envelope called "e," and I patch all my objects into e instead of into the dac. sound() ramps up e, and silent() ramps down to 0. If you want to do something more complex, for example turning off expensive processing when your synth is silent, go ahead and edit these functions.
  • Other functions: you really shouldn't edit. Just start from the skeleton, keep them at the bottom of your code, and ignore them.

When you've implemented your class, make sure you load it in the ChucK configuration panel in the Wekinator GUI.

Implementing your own synthesis class outside ChucK

Editing your synth

Instructions coming soon.

Play-along learning with an external synth

Currently, you will still use a ScorePlayer written in ChucK to control your OSC synth. Your ScorePlayer looks the same regardless of the type of synth.

Implementing your own ScorePlayer in ChucK

This is a lot like implementing your own synth class. Start by copying one of the standard examples, like score_players/ Look at the "TODO" statements in to see what you need to edit.

Your score must know the number of parameters required by your synth, but otherwise it is not tied to your particular synth. A score player can be compatible with multiple synths; a synth can be compatible with multiple score players.

Your new file will have to implement the ScorePlayer class, along with some required functions.

  • fun void setup(SynthClass s, OscSend x) : Do any additional setup here
  • fun void playScore() : in the simplest case, this will include a while(isPlaying) loop that sends new parameters to the SynthClass object over time, as long as isPlaying is true
  • fun void sendMessage() : you may want to show the user a text message informing them what is happening in the score, and what the current parameter values are. This is optional.

There are a few other functions in the skeleton code that you can leave as-is (no need to edit).


  • I can't establish an OSC connection from the GUI! Help!
    • Wekinator requires that you only have one instance of the GUI and one instance of the ChucK code running at once. Make sure you don't have miniAudicle running. Try restarting both ChucK and the GUI and reconnecting. Finally, double check that the ports in the GUI match the ports ChucK is expecting (specified in
  • My controller isn't recognized! Help!
    • Make sure you plug in the controller before running your chuck code (or starting miniAudicle).
    • Every now and then a controller will fail to play nicely with ChucK. First, note that the controller configuration works only for controllers that act as HID joysticks or mice. If this describes your controller, check that ChucK recognizes it (outside the Wekinator) by running this code in miniAudicle or command line. If that code works, you've discovered a Wekinator bug and you should let us know. Otherwise, you may want to investigate alternative HID input environments (e.g., maybe Max/MSP will recognize your HID?), and you should follow the instructions above on implementing your own feature extractor outside ChucK.
  • Neural net training is taking a long time! Help!
    • Training time will increase if you are using a lot of features and/or asking the network to learn a more complex function. Try decreasing the learning rate (e.g., to 0.03 or lower). (This requires that you check the checkbox with the option to view the NN training GUI.) Also keep an eye on the epoch error rate. If it's relatively low, but the training goes on, you can stop the training early by hitting "stop" and then "accept."
  • I get errors when I try to run the ChucK code.
    • Make sure you are running a file containing all the "Machine.add" statements (i.e., one in main_machine_add_files/) and not the individual chuck classes. This file will add them in the correct order for you.
    • Make sure you are running from the WEKINATOR_PATH directory (your top-level directory), and using the --caution-to-the-wind flag.
    • If you've written your own synth or feature extraction code, you can do basic debugging yourself by running ONLY that code in miniAudicle (it'll catch syntax errors). You can also write some test scripts for your classes, like I've included in the tests/ directory.
  • ChucK is core dumping / seg faulting / quitting! Help!
    • ChucK does this now and then. First, we STRONGLY recommend running ChucK in the command line instead of in the miniAudicle. If you still experience problems, and you can reproduce this behavior reliably, write a description on the ChucK/Wekinator/Bugs wiki and we'll look into it.