ChucK/Wekinator/Instructions

From CSWiki

Here you find instructions for version 0.2b (current as of 10/19/09).

Contents

[edit]

Getting started

1. Get a working version of ChucK, if you don't have one already.

2. Download the Wekinator zip file at the Wekinator website. Unzip to the location of your choosing, which we will refer to below as WEKINATOR_DIR.

[edit]

Running the Wekinator

0. Plug in any joysticks, controllers, etc. that you want to use.

1. Run the software.

  • If you're running one of the included examples (recommended to start with), you'll launch the software by running one of the chuck files in the WEKINATOR_DIR/main_machine_add_files/ directory.
    • You'll need to open up Terminal and cd to WEKINATOR_DIR
      • For example, I open terminal and type at the prompt: 
   cd /Users/rebecca/work/2008-09/smirk/wekinator/java/Wekinator
    • Then invoke chuck with the "chuck" command, using the flag "--caution-to-the-wind" (his tells chuck that it's allowed to launch other executables, namely the Java component of the wekinator and and other synths or feature extractors that you wish to launch from the chuck script) and the name of the file you wish to run.
      • For example, my terminal command for running my first ICMC example, located in WEKINATOR_DIR/main_machine_add_files/wekinator_icmc_1_melody_discrete.ck, looks like: 
  chuck --caution-to-the-wind main_machine_add_files/wekinator_icmc_1_melody_discrete.ck
  • If everything goes well, you should see the Java GUI pane pop up on your screen. Otherwise, you might not have installed chuck to the right place (typing "which chuck" at the command line should print out "/usr/bin/chuck"), or you might not be executing the command from the right directory (check that the directory you're in has subdirectories called main_machine_add_files, synths, score_players, etc.)

2. Optionally, start any external feature extractors or synthesis engines that haven't been started automatically in step 1. For example, if you want to use Max/MSP to extract features from a Wiimote, launch your max patch now (see below for details on how to configure your patch to actually send the Wiimote data to the Wekinator). Open up main_machine_add_files/wekinator_icmc_6_processing.ck and take a look at the Std.system("open processing_synths...") line for an example of how you can launch a processing feature extractor or synth from within your main chuck file.

3. Once everything is running (but not before!), click "Connect" in the OSC pane of the main GUI that popped up in step 1. This will get Java talking to ChucK. If you can't connect, kill everything using Ctrl-C in the terminal, then try steps 1-2 again.

4. Once connected, you'll see the Features pane. check off which features you want to use to learn from. Set their parameters, where necessary.

  • For starters, try checking boxes for a built-in feature extractor, like any of the audio features or the motion sensor (if you're on a Mac).
  • Or, if you're using custom chuck or OSC features (see below), enter that number in the box marked "Custom ChucK features: Enter #" or "Custom OSC features: Enter #".
  • Or, if you're using a HID device like a joystick, check the "Other HID" box, hit "Setup new...", and then engage all sensors/buttons/axes/etc. on the device. That is, move all the axes around, push all the buttons once, etc. Once you've done this, press "Hit to End Setup." You should see a description of the number of axes, buttons, and hats that have been seen on your device. You can save this device configuration by hitting "Save as...", and saving to myjoystick.controller, for example (file extension doesn't matter). In the future, you can reload this controller by checking "Other HID" and then hitting "Load from file..." and selecting this file.
  • You can check as many boxes on this pane as you want! If you're using an FFT, though, and your FFT size is 1024 or higher, OSC may crash and burn. Sorry.
  • If you think you want to use this set of features again in the future, you can save the configuration file by hitting "Save settings to file...", then reload it later by hitting "Load feature settings from file...".

5. Hit "Go!" The GUI will wait to make sure that it agrees with ChucK and your feature extractors on how many features you're using going forward.

6. You'll see the settings panel.

  • the choices you see will depend on whether the synthesis code you're running is continuous or discrete (see below for more info on this).
    • If it's continuous, you'll use a neural network (for now).
    • If it's discrete, you get your choice of classifier in the dropdown box.
    • You can also see the settings of your synthesis code on this pane. It'll tell you, for example, that your synth is expecting discrete or continuous outputs, and the number of classes it's expecting (if it's a discrete problem).
  • Choose to load a saved model or create a new model
    • If this is your first time running, you'll load a new model, and if you're doing discrete classification, you'll want to choose a classifier type from the drop-down box.
    • Or, you can load a saved model, from those you've saved in the Train/Run pane during previous runs (or models you've created using Weka).

7. Hit "Go!" A new set of classifiers/neural nets will be created for you.

8. Arrive at the Train/Run panel. Here is where you'll do most of your interaction with the Wekinator.

  • Here, your experience also depends on whether you're doing continuous or discrete learning.
    • In either case, you'll see an input box for every parameter that your synth is expecting (this number is specified in the synthesis class code itself, as discussed below).
    • Discrete learning: In each box, you can enter a value between 0 and N-1, where N is the number of "classes" your model is built to use. (For example, if you want to do pitch classification into a chromatic scale, you would have 12 classes.)
    • Continuous learning: You can enter whatever real numbers you'd like! (The synth is responsible for handling illegal or weird values that it may see).
  • You can type in values into the boxes and press "Listen to this value" to send those parameters to your synth.
  • Hitting "hold to record examples" captures a sequence of snapshots of your input features (e.g., the position of your motion sensor, or features of the audio coming into the microphone) and associates them with the parameter values currently in the text boxes. This list of pairs of input features with parameters becomes your training dataset!
  • Once you have recorded some training examples, the "Train!" button will become active, and you can train the model. Once you have a trained model, you can hit "Run!" to start using the model to actually generate parameter settings from the input features you supply, in real-time. You can hit "Run!" again at any time to stop running the model.
  • You can continue to add examples to your training set at any time by entering new parameters and hitting "Hold to record examples" again. You'll have to re-train each time you want to run the model using the new examples.
  • You can clear your model and training set by hitting "Forget everything!" (in which case you'll have to record new training examples and train a new model).
  • You can and save a trained model to a file by hitting "Save trained model." In the future, you can reload this trained model from the Settings pane, and add even more training examples to it in Train/Run (or just run it as is).
  • If you're using a discrete classifier, you'll see a button for "Classifier settings..." which allows you to adjust the settings of the classifier. Hitting "Compute accuracy" will allow you to compute the training and k-fold cross-validation accuracy of a trained model, giving you objective estimates of its performance.
  • If you're using a neural network, you'll see a checkbox for "view GUIs when training." If you keep this box checked, you'll see a series of GUIs that pop up when you hit "Train!," that allow you to interactively train and re-train the network associated with each parameter. You can change the architecture of the network by adding new nodes and connections. You can also adjust the learning rate, momentum, and number of epochs, and monitor the error per epoch as back-propagation training progresses.
    • For starters, try leaving all these settings as they are. To train using the GUI, hit "Start," then watch the Error per Epoch decrease. If the network training converges quickly on its own to a low error, training will stop, and you can hit "Accept" and move on to the network for the next parameter. Or, if the training error seems to be getting low but it isn't stopping on its own, you can hit "Stop" to stop training early, then hit "Accept." If training is going very slowly and error is still high, hit "Stop" and try changing the learning parameters, especially changing the learning rate to something lower. Then try hitting "Start" again and see if things improve.
    • Note that you *must* click "Start" and then later "Accept" at some point for the GUI for each parameter in order to train the network for that parameter!
    • If you like your neural net settings, you can uncheck "view GUIs when training" and the training will use these settings until you change them again.

10. You are free to experiment with adding new training examples, changing classifier/network parameters, choosing a different type of classifier, or choosing different features and feature parameters. Be aware that changing the features or the classifier will force you to train again from scratch!

11. Play-along learning Maybe you don't want to type the parameter values into the boxes every time you want to provide a new training example. Maybe it's better to listen to what your synth actually sounds like, and provide input gestures/audio/etc. that you would like to correspond to that sound. This is what the play-along learning tab is for. You can use this tab instead of or in the same session as the Train/Run tab to produce your training set.

  • Playing a score: If (in step 1) you've started a ScorePlayer that's compatible with your synth, you'll be able to hit the "Play score" button and listen to a "score" being played by your synth. The score is essentially a series of commands for your synth to set its parameters to particular values at particular times (just as if you were in the Train/Run tab changing the parameter values and hitting "listen").
  • Playing along with a score: Pressing the "Play along" button allows you to generate a sequence of inputs (gestures, audio, etc., using whatever controllers you've decided on in the Features pane) that you believe should correspond with the score. Wekinator will record your gestures/audio/etc. along with the parameters being sent by the score and use these to build your training set, without you having to type anything.
  • Training
    • Option 1: You can train using the Train/Run panel (go to that panel and hit the Train! button), just like I described above.
    • Option 2: Or, you can specify that the Wekinator should train in the background while you're playing along. Check the "Automatic training" box, then use the slider to specify how frequently you want training to happen. If you want the training to stop automatically once the Wekinator is successful at modeling the relationship between your inputs and the score, check the "Use auto-stop threshold" box. Here, the wekinator will be estimating the accuracy of its model using cross-validation. Practical advice here: more CV folds = more time to compute, so be careful. The threshold is a number from 0 (totally inaccurate) to 1 (completely perfect).
  • Adjusting parameters
    • For some classifiers, the "Use fast/accurate parameterization" slider lets you roughly specify the tradeoff you'd like to make in terms of speed of training and accuracy of the model.
[edit]

Organization of the Wekinator

  • TODO show picture of architecture
  • The GUI is in Java. You won't need to recode any of this (but Rebecca will!). The GUI jar file lives in dist/.
  • The core ChucK system<b/> is in the core_chuck/ directory.
    • You shouldn't have to change any of this, ideally.
    • The main ChucK code responsible for communicating with Java and the feature extractors is main_chuck.ck.
  • The ChucK <b>synths are in the synths/ directory. Each synth implements a class called SynthClass, the structure of which you can read about below. If you want to create your own ChucK synth, you can start by editing synth_skeleton.ck.
  • In play-along learning, a ScorePlayer object sends parameters to a synth object. (A score is simply a set of instructions at specific points in time.) The ScorePlayers live in the score_players/ directory. You don't have to implement a score player for your synth unless you want to do playalong learning.
  • A feature extractor is a module that extracts features (also called attributes) from your inputs. For example, if you're using an audio input, a feature might be an FFT, or the peak amplitude of the audio signal. If you're using the laptop motion sensor, a feature might just be the values of each of the 3 sensor outputs. Your input is fully described to the wekinator by a vector of real-valued features, which typically describe the state of your input at a single point (snapshot) in time (though you could imagine a feature vector that describes its current state and its previous state). Custom feature extractors in ChucK live in the feature_extractors/ directory. They implement the CustomFeatureExtractor class. You can add your own custom feature extractor in chuck by editing CustomFeatureExtractor.ck (see example_centroid.ck for a simple example). You can also choose from existing custom feature extractors for more specialized audio or gestural features.
  • You can integrate synths in environments outside ChucK. For example, see the processing_synths directory. More info on how to do this below.
  • You can also integrate feature extractors in environments outside ChucK. See below.
[edit]

Communication

  • TODO: diagram
[edit]

Running it all: The "machine add" file

ChucK does not support "include" or "import" statements, which means that the way we run large programs is using a file that merely instructs the ChucK virtual machine to run all of the component files of our program. In version 0.2, you can see a list of example such files in the main_machine_add_files/ directory. Inside these files, you'll see a list of Machine.add("....") statements, each of which adds a class definition and/or a set of instructions to ChucK.

In order to run the wekinator, your main "machine add" file must contain:

  • Machine.add instructions for the core chuck components: trackpad, motion, audio, custom OSC, and Processing feature extractors; plus the HID discoverer
  • One file containing the implementation of a custom feature extractor (use CustomFeatureExtractor.ck if you won't be using any custom features)
  • One file containing the implementation of a synth class
  • One file containing the implementation of a score player (doesn't matter which one you use if you won't be doing play-along learning)
  • The main_chuck_playalong.ck file

Finally, you can launch the GUI and any external synth / feature extractor using an Std.system command, for example:

Std.system("java -jar dist/Wekinator.jar &");

you MUST run chuck with the --caution-to-the-wind flag if you want these Std.system commands to actually do anything.

Caution: You MUST run this file from the WEKINATOR_PATH directory, otherwise ChucK will not be able to find the files you're trying to run in the Machine.add commands.

[edit]

Implementing your own feature extractor in ChucK

Your custom feature extractor will live in a class named CustomFeatureExtractor. We recommend that you start with this file provided called CustomFeatureExtractor.ck (in the feature_extractors/ directory), which you can edit according to the "TODO" instructions in the code. Basically, you'll have to implement function bodies for setting up the feature extractor, and most importantly computing the features.

If you edit the code provided and put the computed features in the features[] array of your class, you're good to go! For an example of how to implement the setup and feature computation, check out feature_extractors/example_centroid.ck.

To run, make sure your custom feature extractor class is the one added by your main Chuck "machine.add" file to implement CustomFeatureExtractor. That is, there should be a line that looks like:

Machine.add("MyCustomFeatureExtractorFileName.ck");

Also, when you run the GUI, make sure to check the box that says "Custom ChucK features" on the Features panel, and set the number of features to the number of features your feature extractor actually extracts (numFeats in the skeleton code).

[edit]

Implementing your own feature extractor outside ChucK

All your feature extractor needs to do is send a feature vector to the Wekinator via OSC at the (possibly variable) rate of its choosing. The feature vector is a list of floats (MUST be floats), of arbitrary length (though this length must not change over time, and the order of the features must not change over time). Your extractor sends the features in one OSC message, "/oscCustomFeatures", to the port specified as recvPort in main_chuck (default is 6453).

To run, start your feature extractor before starting the Java GUI. In the "Features" panel, enter the number of custom OSC features to be equal to the length of the feature vector specified in your oscCustomFeatures message above.

That's all!

[edit]

Implementing your own synthesis class in ChucK

Your synthesis code will live in a ChucK class called SynthClass and implement a pre-defined set of functions.

We recommend starting with synths/synth_skeleton.ck, which has instructions on what you should edit to make your synth work.

You'll need to implement the following functions:

  • fun int isDiscrete() : return 1 if you want discrete classifier outputs, 0 if you want continuous outputs
  • fun int getNumClasses() : the number of classes expected by your synth (where a class here means a member of the set of discrete outputs of a classifier like kNN; for example, to classify gestures into "moving" and "not moving", you would have 2 classes). Only matters if doing discrete classification.
  • fun int useDistribution() : if you're using a discrete classifier, return 1 if you want a vector of class membership probabilities for each output, or 0 if you only want a single class label for each output. If you're not discrete, it doesn't matter.
  • fun int getNumParams() : the number of outputs expected from the model(s). For discrete or continuous learning. For example, to drive the frequency and volume of a SinOsc independently, you would use 2 parameters.
  • fun void setup() : This function is called just once, when the system is setting up. Treat this like a constructor. In other words, do any work here that should be done before sound is made, like sporking shreds that should be running the whole time your synth is running (e.g., for smoothing parameters).
  • fun void setParams(float params[]) : sets your synthesis parameters according to the contents of params[], which will be of length equal to the number of parameters if you are doing continuous or discrete without distribution, or length equal to (# parameters) x (# classes) if you are doing discrete with the probability distribution.
    • In other words, this is the most important part of your synth! It describes how you're using the outputs of the model.
    • Note: You probably want to do some error checking here. Ideally, params[] will be of the appropriate size, but you should check. And ideally the values in this array will be what you'd expect, but you should never assume this is the case, particularly when using a neural network for learning a continuous value. Your "frequency" parameter value might come in as -1, 0, or 500,000 for example: it's your call how to use (or ignore) these types of values.

Your synth has a few other functions that need to be present, but which you shouldn't need to edit by default.

  • sound() and silent() turn the sound on and off. In the examples, I use a main envelope called "e," and I patch all my objects into e instead of into the dac. sound() ramps up e, and silent() ramps down to 0. If you want to do something more complex, for example turning off expensive processing when your synth is silent, go ahead and edit these functions.
  • Other functions: you really shouldn't edit. Just start from the skeleton, keep them at the bottom of your code, and ignore them.


When you've implemented your class, make sure you add it in your main "machine add" file, using a line like: Machine.add("MySynthClassName.ck");

(You will not be able to add any other chuck files that also include a SynthClass definition.)

[edit]

Implementing your own synthesis class outside ChucK

[edit]

Editing ChucK

Create a copy of synths/OSC_synth_skeleton.ck for your synth.

Edit this file as indicated in the comments (see the TODO statements): set the number of classes, whether it expects discrete or continuous outputs, the number of synthesis parameters, and whether it expects a probability distribution over classes or just a class label (in the case of a discrete problem). Set synthPort to the port on which your synth is listening, and set synthHostname to synth's hostname/IP if your synth is running on a different machine (otherwise keep it "localhost.").

Edit your "machine add" main file so that this file, and no other, is added as your synth.

[edit]

Editing your synth

Your synth MUST listen for the message "/OSCSynth/params", with a set of floating point values. The number of floating point values it receives will be equal to the number of parameters (set above), unless you are doing discrete classification and using probability outputs, in which case the number of floats it receives will be equal to (# params)x(# classes).

Your synth MUST be listening on the port specified in the ChucK file you just edited!!

This port CANNOT be the same as either of the ports you specified in the Wekinator GUI! Chuck & Java are already listening on those ports. You can send to them, but you've got to listen somewhere else.

Optionally (recommended but not necessary) listen for these messages, too:

  • /OSCSynth/silent
  • /OSCSynth/sound
  • /OSCSynth/startSendingParams SEE BELOW
  • /OSCSynth/stopParams
  • /OSCSynth/sendParams SEE BELOW


Listening for these optional messages will allow your synth to integrate most fully with the Wekinator GUI, e.g., being silent, using play-along learning. You have the option of manually configuring your synth to send its parameters directly to the "Receive port" specified in the GUI, or you may use the hostname & port sent along with the /OSCSynth/startSendingParams and /OSCSynth/sendParams commands, so you don't need to edit your code.

For an example of how all these OSC messages might be used by your synth, see processing_synths/icmc_visual_synth/ .

[edit]

Play-along learning with an external synth

Currently, you will still use a ScorePlayer written in ChucK to control your OSC synth. Your ScorePlayer looks the same regardless of the type of synth.

In the future, I'll be adding OSC messages to the list above where your synth itself might choose to implement its own score. Stay tuned.

[edit]

Implementing your own ScorePlayer in ChucK

This is a lot like implementing your own synth class. Start by copying one of the standard examples, like score_players/icmc_melody.ck. Look at the "TODO" statements in icmc_melody.ck to see what you need to edit.

Your score must know the number of parameters required by your synth, but otherwise it is not tied to your particular synth. A score player can be compatible with multiple synths; a synth can be compatible with multiple score players.

Your new file will have to implement the ScorePlayer class, along with some required functions.

  • fun void setup(SynthClass s, OscSend x) : Do any additional setup here
  • fun void playScore() : in the simplest case, this will include a while(isPlaying) loop that sends new parameters to the SynthClass object over time, as long as isPlaying is true
  • fun void sendMessage() : you may want to show the user a text message informing them what is happening in the score, and what the current parameter values are. This is optional.

There are a few other functions in the skeleton code that you can leave as-is (no need to edit).

[edit]

Troubleshooting

  • I can't establish an OSC connection from the GUI! Help!
    • Wekinator requires that you only have one instance of the GUI and one instance of the ChucK code running at once. Make sure you don't have miniAudicle running. Try restarting both ChucK and the GUI and reconnecting. Finally, double check that the ports in the GUI match the ports ChucK is expecting (specified in main_chuck_playalong.ck).
  • My controller isn't recognized! Help!
    • Make sure you plug in the controller before running your chuck code (or starting miniAudicle).
    • Every now and then a controller will fail to play nicely with ChucK. First, note that the controller configuration works only for controllers that act as HID joysticks or mice. If this describes your controller, check that ChucK recognizes it (outside the Wekinator) by running this code in miniAudicle or command line. If that code works, you've discovered a Wekinator bug and you should let us know. Otherwise, you may want to investigate alternative HID input environments (e.g., maybe Max/MSP will recognize your HID?), and you should follow the instructions above on implementing your own feature extractor outside ChucK.
  • Neural net training is taking a long time! Help!
    • Training time will increase if you are using a lot of features and/or asking the network to learn a more complex function. Try decreasing the learning rate (e.g., to 0.03 or lower). (This requires that you check the checkbox with the option to view the NN training GUI.) Also keep an eye on the epoch error rate. If it's relatively low, but the training goes on, you can stop the training early by hitting "stop" and then "accept."
  • I get errors when I try to run the ChucK code.
    • Make sure you are running a file containing all the "Machine.add" statements (i.e., one in main_machine_add_files/) and not the individual chuck classes. This file will add them in the correct order for you.
    • Make sure you are running from the WEKINATOR_PATH directory (your top-level directory), and using the --caution-to-the-wind flag.
    • If you've written your own synth or feature extraction code, you can do basic debugging yourself by running ONLY that code in miniAudicle (it'll catch syntax errors). You can also write some test scripts for your classes, like I've included in the tests/ directory.
  • ChucK is core dumping / seg faulting / quitting! Help!
    • ChucK does this now and then. First, we STRONGLY recommend running ChucK in the command line instead of in the miniAudicle. If you still experience problems, and you can reproduce this behavior reliably, write a description on the ChucK/Wekinator/Bugs wiki and we'll look into it.
Retrieved from "http://wiki.cs.princeton.edu/index.php/ChucK/Wekinator/Instructions"