ChucK/Wekinator/Walkthrough

From CSWiki
Jump to: navigation, search

WALKTHROUGH

This section will walk you through 2 runs of Wekinator, one with a discrete learning problem and one with a continuous learning problem.

First, let’s map from the motion sensor (our input features will be the 3 axes of the motion sensor) to a simple synth class whose only controllable parameter is its pitch. Run Wekinator by double-clicking on Wekinator.jar in the Finder, located in your downloaded Wekinator directory, in project/java/Wekinator/dist/Wekinator.jar. In “Chuck & OSC setup,” hit “Edit” to edit your chuck configuration, which specifies to Wekinator the location of crucial files and some other options it needs to run. First, under “Synthesis / Composition (Output),” you’ll specify which piece of code will be running to make sound. In the future, if you want to use a ChucK synth class; these files are located in wekinator/project/chuck/synths. For now, choose icmc_melody_discrete.ck. Next, you’ll have to specify the location of your chuck executable (probably /usr/bin/chuck if you're on a mac or linux, or C:\Windows\System32 if you're on a PC). Hit “Choose file” and navigate to that file. You also have to specify the absolute location of your wekinator project directory. Once you’ve done this, hit “OK.” (These settings will be reloaded the next time you run Wekinator.)

Back on the “Chuck & OSC Setup” pane, you need to launch the ChucK component of the Wekinator (ChucK does much of the feature extraction and sound synthesis, while Java does the graphical interface and machine learning). Hit “Run.” Once ChucK starts running you should be taken to the next pane, “Features setup.” If this doesn’t happen, ensure that you don’t already have another copy of Wekinator running, and that you’ve chosen a valid executable file and synth file. You can also go to View-> Console and try running again to view any error output.

On the “Features Setup” pane, you will choose which features (inputs, gestures) you want to use to control the sound. (If you want to use a HID device (such as a tether or joystick), you must plug this device in before running ChucK on the previous pane. Then hit “Configure,” “Start interactive setup now,” move all buttons/axes/hats of the HID device that you want to use, then “End interactive setup.” The window should show you the number of buttons/axes/hats that were detected.) For now, choose the motion sensor only, and make the extraction rate 30 ms (so that changes of the sensor value are detected more quickly). You also have the option here of adding “metafeatures” which estimate a feature’s velocity (1stDiff) and acceleration (2ndDiff), smooth its values, or keep a running history of the last N value. Don’t do this for now; just hit “Go” after choosing the motion sensor, and you will be brought to the next pane.

In the “Learning Setup” pane, you most importantly have the ability to choose which learning algorithms to use, and which features affect which parameters. For a discrete learning problem, in which the synth class takes parameters that are integer-valued, you have a choice among 4 classification algorithms (AdaBoost, Decision Tree, k-nearest neighbor, and support vector machine); for a real-valued learning problem, your learner will be a neural network. Each parameter of your synth gets its own model (which computes the parameter value as a function of the input features) and each parameter can have a different learning algorithm producing its model. Additionally, you can restrict which input features may affect which parameters. For example, if I am using laptop tilt and color tracking to control a synth whose parameters are pitch and volume, I could use the “View and choose features” button to enable only tilt to control pitch and only color tracking to control volume. For now, let all features control all parameters (the default). Hit “Go!” and you’ll advance to the next pane.

In the “Use it!” pane, you’ll be building and using your models. In “Collect data,” you’ll create a dataset consisting of input feature values matched with the corresponding synth parameter values. The easiest way to do this is to set the parameter values you want in the “set parameter values” pane, and perhaps listen to these values by selecting “my synth should play these parameters on demand” and then hitting “play.” Then, you will demonstrate the input gesture that should correspond to that sound, hit “Begin recording examples into dataset,” and then hit “stop recording.” This will take a series of snapshots of your current gesture or position. You can view this dataset of features and parameters by hitting “View examples,” and delete all this data by hitting “Clear examples.” For now, start by setting Param_0 to “0” in the dropdown box, tilting your laptop forward, and hitting “Begin recording examples into dataset” and (shortly after) hitting “Stop recording.” Then, set Param_0 to “7” in the drop down box, tilt your laptop backward, and record some examples like that. Click “View examples” to see this data, then “Done” to close that window. Next, you will want to create a model from this dataset, so that you can provide new input gestures and the model will output the corresponding synth parameters. Hit “Train” on the left menu, then hit the “Train now” button. Once training has finished, you’ll see a message in the “Training progress” box under the button, and you can use the model.

In order to run the model, go to the “Run” button on the left menu, then hit the “Run” button on the right. You should see the Param_0 value updating when you move the laptop back and forth. If not, go back to your dataset viewer and verify that you’ve actually recorded some examples with a param_0 value of 0 and others with a value of 1.

You can objectively evaluate your models’ performance by going to the “Configure & evaluate” option on the left menu. You can compute cross validation accuracy, a measure of how well the model generalizes to new data; the number of folds determines how this is computed and more folds may mean longer time to compute. You can also compute the training accuracy, a measure of how well the model has “memorized” the parameters of the training examples (your dataset). If this measure is quite low, it may mean that there is a problem with your training dataset or the algorithm’s ability to learn from it. For both these measures, 100% is the best possible accuracy (for measuring discrete models) and 0 is the best possible RMS error (for measuring neural networks). You can compute these measures for all parameters at once, or for individual parameters, by selecting from the drop-down box at the top of this pane.

You can also edit the settings for each of your learning algorithms by choosing them from the top drop-down box on the “Configure & Evaluate” pane. Each algorithm has different configuration options which may impact the training time of the algorithm, its accuracy, and other characteristics.

With the Wekinator, you are allowed (and encouraged) to change your training examples to modify the behavior of the learned model. To do this, go back to “Collect data…” at any time and record more examples, view & edit the dataset in the spreadsheet viewer, clear examples, or add more examples using the parameter clipboard “score.” After you’ve edited the data or changed any of the learning algorithm parameters in the “Configure & evaluate” pane, you’ll need to retrain the algorithms before the changes will take effect for running or computing training accuracy.

Now, go back to “Chuck & OSC setup” and stop ChucK. Edit the ChucK configuration and choose icmc_melody_continuous.ck (again, in project/chuck/synths/) to be your synth. This is the same synthesis method as icmc_melody_discrete, but this class requires a float parameter for pitch instead of an int (allowing you to change pitch continousouly). Run with this new synth, use the motion sensor again, and notice that when you get to “Learning Setup,” you will be creating a neural network for the parameter. Record examples similar to before, setting 0 to tilting your laptop forward, and 7 to tilting it back. Train your model and run it, and now notice that the pitch will now move gradually between the two extreme positions, rather than jumping from one to the other. This is what the neural network allows you to do.

Get comfortable with this. Play around with the following synths and feature extractors, and make sure you’re familiar with how they work. Synths: (all in project/chuck/synths/, except for blotar, which is in the top level of your unzipped wekinator directory):

  • Jeff_FM_Synth_wekinated2.ck (All parameters are continuous between 0 and 1)
  • icmc_bowed_physmod.ck (3 parameters continuous between 0 and 1)
  • Blotar (many continuous params, launch the max app separately!)
  • icmc_beats_mod.ck (simple drum machine, 1 discrete param that determines which drums play)
  • icmc_melody_discrete.ck (1 parameter that controls pitch, discrete from 0 to 24)

Input features:

  • Webcam color tracker (launch from Finder, located at project/chuck/colortracker2/application.macosx/colortracker2; if it doesn’t run, you may have to set to 32-bit mode using cmd+i on the filename. Once running, click 1st color, right-click/ctrl-click 2nd color, hit “=” to clear choices.)
  • Motion sensor
  • Joystick or tether (if you have one checked out)