MML fall2009

From CSWiki
Jump to: navigation, search

General Information

Welcome to the wiki home of the unofficial 2009 music & machine learning workshop. Who knows what will become of this page?

Participants

  • Rebecca Fiebrink, organizer
  • Dan Trueman
  • Anne Hege
  • Cameron Britt
  • Michelle Nagai
  • Konrad Kaczmarek
  • MR Daniel
  • Seth Cluett
  • Michael Early

Stuff we want to try out

  • pitch tracking
  • percussion gesture identification
  • general new instrument building (mapping)
  • add your request to this list!

Stuff we want to find out more about

  • Add your request to this list. For example, history of learning in NIME/ICMC/ISIMR; details of how one or more algorithms actually work; ??

Bugs / Feature requests

Code

  • File IO based communication (as alternative to OSC for large feature vectors)
  • Allow synthesis module to communicate current parameters to GUI (just once, not in play-along mode), to allow parameter text boxes to be auto-filled (Konrad's idea) (done)
  • Ability to designate N-M mappings (not All-All), e.g. ability to train one classifier on vocal features and one on gesture features (Anne's request) (done)
  • Integrate MFCC feature extractor, possibly ramp time feature extractor (Cameron's request, ref. Adam Tindale's paper)
  • Debug mode flag so that users don't get scared when they see code messages being printed out
  • Add a way to examine the current training dataset (useful for debugging); also good to be able to edit (for fine-grained fixing) (DONE)
  • Incorporate way to adjust annotations of audio & sensor inputs in Wekinator
    • Integrate w/ audacity labeling?
    • Integrate bonk-like onset-based detection? (Somewhat done: See TJ feature extractor)
  • Add customizability of foot/other controllers to control GUI (as opposed to sound inputs) (Somewhat DONE: FOOtcontroller works; need general OSC solution)
  • Allow training of 1 parameter (classifier) at a time (Konrad's request) (DONE)
  • Ability to work with multiple channels of audio input, for use both with audio and with sensors (Cameron)
    • This could start as another # in the feature extractor GUI
  • Possibly allow non-spreadsheet-like visualization & editing of training data
    • Possibly includes, for gestural inputs, indicator of features for a given training point, relative to current features, so you can match it
  • How to integrate envelope- or other-based selective application of Wekinator, e.g. to classify just after onset detected?
  • More thought on time-dependent features / modeling
  • Fix so that user gets indicator that training has finished after a long training round, not just 1st time
  • Better control of ChucK (get rid of machine add files) (done, but currently buggy)
    • Incorporate console to view chuck's error messages
    • Add --blocking, perhaps as an option to chuck
  • Ability to capture palette of sound you like during a 1st rough training, then re-play these during playalong mode.
  • Ability to quickly change mappings mid-performance
    • And ability to specify this via an OSC command
  • Fix OSC feature receiver so that ints are accepted as well as floats (Done, pending Konrad confirmation)
  • Fix confusion resulting from switching between settings & training tab when feature-parameter mapping settings have been changed (thanks Michelle!)
  • Major bug : switching classifier type after 1 classifier has already been trained doesn't in fact change the classifier (thanks Michael!)
  • Loading saved model from a file crashes everything if the model doesn't match your settings (Thanks MR!)
  • Visualization of complexity of learning problem - eg using MDS like IJCAI paper

Documentation

  • Document fact that GUI ports are for Chuck-GUI only; need to use different port for max-chuck synth communication, for example
  • Nice picture of architecture, what communicates with what and how
  • Tip: Require ./ before running .sh file from terminal
  • Tip: restart after changing OSC ports, and don't run miniAudicle concurrently if having OSC problems
  • Clear API for playalong learning synth class (update this from what's on webpage currently)
  • Clear OSC API for updated system (feature extractors and synthesizers)
  • ML bibliography page (Seth's request)
  • More on how to implement your own feature extractor in max/msp (for Dan 3D knob)
  • Tip: Drag from Finder to Terminal to get path
  • Explicit info on message used to send params to Wekinator from OSC synth (/realValue f+)
  • Better documentation on how to edit SynthClass for incorporating your own synthesizer from previous code: specifically, telling user to make all methods & variables class methods & variables & to spork everything in setup().
  • Updated build instructions: ant w/ svn

Examples

  • A clear stub / skeleton for play-along learning class, chuck feature extractor, and OSC feature extractor (DONE)
  • Add Anne's vowel feature extractor to example set (DONE)

General topics for conversation

  • Wekinator system architecture
    • What is required to add your own synth class? feature extractor? (Covered 10/12)
  • Metaphors and language for interaction, learning

Schedule

  • October 5, 2:30pm
    • Discuss goals for the semester
    • Possible outcomes:
      • Conference or journal publication
      • Compositions to be performed by ourselves, with PLOrk in Spring, other?
      • Community software repository: general tools for Wekinator, and/or specific to our compositions/interfaces
    • Scheduling - let's do every week for now, as long as people can make it
    • Possible papers to read together?
    • Things we want Rebecca or someone else to lecture on?
  • October 12, 2:30pm
    • Walked through system architecture, what a SynthClass looks like
    • Konrad showed his Max/MSP synthesis code, which integrates with a larger Max/MSP patch using patter
    • Dan brought in the magic 3D knob and hooked it up for feature extraction in Max, then we sent the features out to Wekinator via OSC. Turned it into obnoxious physical model controller.
    • Cameron brought in a drum w/ 2 pickups, verified that we could detect "hit" versus "silent" using basic audio features.
  • October 19, 2:30pm
    • Updates: Included in Wekinator 0.2 download
    • Mentioned OSC handout for those new to OSC
    • Tried out the foot pedal & a new feature extractor for Cameron's drum
    • Tried to try out vowel panning & LiSa code, not compatible with new version. (Sent out compatible code via email Monday evening.)
    • Discussed some interface options for data set creation / annotation
  • October 26, 2:30pm
    • Updates: SVN, snapshot of current synth params, spreadsheet-like view of current dataset w/ edit ability, fix bug that requires re-creation of training set when classifier changed, some prototypes of real-time annotation (perhaps in miniAudicle), try out Cameron code for envelope & feature-driven synthesis.
      • Motivators: What if we want to replace training data? Erase it? Edit it? Forget from a certain time window in the past?
      • Want to be able to view, audition, edit data points
      • Interesting simple and complex visualization methods to understand & edit training data spac
    • Discussed interface issues for creating drum detection dataset
      • Create traditional dataset & use CV accuracy to select features & classifier?
      • Integrate with methods like onset detection for selective application of Wekinator output?
  • November 9
    • New version includes check boxes for parameters, so you can decline to train certain parameters on the input you're currently providing
    • Konrad & Dan brought in their work to hack on
    • Briefly discussed upcoming Chuck-within-GUI support
  • November 16
    • Chuck within GUI is working (but buggy): We definitely need better debugging support for running in this mode, so that users have feedback on what Chuck is doing!
    • Dan, Cameron, and Konrad all brought their stuff to work on
      • Cameron has an Arduino controller speaking to ChucK using Chuck file I/O!
      • Dan using tether blotar with the help of several people; also wekinating sillicarbon code
    • Anne and others discussed still needing ability to specify which features control which parameters (this is coming for next week!)
    • Dan and I talked about allowing the user to create a play-along score from within the GUI, for example using parameter snapshots from a previous training round
  • November 24
    • Played around with new ability to set which features affect which parameters
    • Did some more debugging/setup for the Chuck-in-GUI version
    • Talked about Michelle's bark instrument
    • Made some _beautiful_ blotar sounds with the tether
    • Talked about ICMC & NIME: Consensus is that we want to do at least one paper together, possibly on our collective experiences with blotar
  • Dec. 1st (Tuesday with PRC)
  • Dec 7
  • Dec 14th

Bibliography

This is in the early stages. I'll put things here that participants might find useful, that we mention in the meeting, or that people are using to help build their own systems.

General machine learning resources

  • Data Mining: Practical Machine Learning Tools and Techniques (book). Witten & Frank.
    • This is a nice and very practical introduction to machine learning for people absolutely new to the field. It also serves as a tutorial to the Weka software. The Princeton library has a copy (currently checked out, but reservable). Rebecca highly recommends.
  • Pattern recognition and machine learning (book). Bishop.
    • This is a more technical introduction to machine learning. A great textbook. Rebecca has a copy and can bring it in if people want to see it.
  • Andrew Ng's online course (videos & materials): http://see.stanford.edu/see/courseinfo.aspx?coll=348ca38a-3a6d-4052-937d-cb017338d7b1

Machine learning in music

  • Music/audio analysis (ISMIR)
  • Gestural control systems
  • Compositional systems

Stuff happening at Princeton

  • Matt Hoffman's work on audio analysis

Related software

  • Marsyas
  • CLAM

Instructions for downloading latest code

Option 1 Fast. But you'll have to do some manual directory management.

Grab weekly .zip of all the latest stuff (will require you to re-merge with your own files, if you've been working on things yourself) http://www.cs.princeton.edu/~fiebrink/drop/wekinator/all.zip

Option 2. Advanced. Latest code, all the time!

Step 0 (setup: 1st time only): Set up SVN by installing OS X developer tools if you haven't already. (Not sure if you've got svn? Type "which svn" in the terminal. If you've got it, you'll see a line printed that looks something like "/usr/bin/svn"; if not, you won't see anything but another command prompt.)

Once you've got svn, open up terminal and cd to a directory where you want the project to live from now on. For example you could cd to your desktop. Then type in the terminal:

svn checkout http://wekinator.googlecode.com/svn/trunk/ wekinator-read-only

This will grab you a copy of all the code.

Step 1: Do this every time you update.

Open terminal. Go to the wekinator-read-only directory. Type:

svn update

This will check out all the other changes into your directory. If you've made modifications to the code, it'll try to merge my changes with yours. Likewise, if you've created your own files, SVN will not move / delete / overwrite / change them.

Step 2: After step 1, every time you update: NEW instructions as of 11/19/09 In terminal, go to directory java/Wekinator/. At the prompt, type:

ant

You should see "Build successful" after this finishes running. If you don't see that, make sure you've done Step 1, otherwise email Rebecca because it's probably her fault!

(This updates the java executable file, which we don't put in SVN.)