- 1 General Information
- 2 Participants
- 3 Stuff we want to try out
- 4 Stuff we want to find out more about
- 5 Bugs / Feature requests
- 6 General topics for conversation
- 7 Schedule
- 8 Bibliography
Welcome to the wiki home of the unofficial 2009 music & machine learning workshop. Who knows what will become of this page?
- Rebecca Fiebrink, organizer
- Dan Trueman
- Anne Hege
- Cameron Britt
- Michelle Nagai
- Konrad Kaczmarek
- MR Daniel
- Seth Cluett
- Michael Early
Stuff we want to try out
- pitch tracking
- percussion gesture identification
- general new instrument building (mapping)
- add your request to this list!
Stuff we want to find out more about
- Add your request to this list. For example, history of learning in NIME/ICMC/ISIMR; details of how one or more algorithms actually work; ??
Bugs / Feature requests
- File IO based communication (as alternative to OSC for large feature vectors)
- Allow synthesis module to communicate current parameters to GUI (just once, not in play-along mode), to allow parameter text boxes to be auto-filled (Konrad's idea)
- Ability to designate N-M mappings (not All-All), e.g. ability to train one classifier on vocal features and one on gesture features (Anne's request)
- Integrate MFCC feature extractor, possibly ramp time feature extractor (Cameron's request, ref. Adam Tindale's paper)
- Debug mode flag so that users don't get scared when they see code messages being printed out
- Add a way to examine the current training dataset (useful for debugging); also good to be able to edit (for fine-grained fixing)
- Incorporate way to adjust annotations of audio & sensor inputs in Wekinator
- Integrate w/ audacity labeling?
- Integrate bonk-like onset-based detection?
- Add customizability of foot/other controllers to control GUI (as opposed to sound inputs)
- Allow training of 1 parameter (classifier) at a time (Konrad's request)
- Ability to work with multiple channels of audio input, for use both with audio and with sensors (Cameron)
- This could start as another # in the feature extractor GUI
- Possibly allow non-spreadsheet-like visualization & editing of training data
- Possibly includes, for gestural inputs, indicator of features for a given training point, relative to current features, so you can match it
- How to integrate envelope- or other-based selective application of Wekinator, e.g. to classify just after onset detected?
- More thought on time-dependent features / modeling
- Document fact that GUI ports are for Chuck-GUI only; need to use different port for max-chuck synth communication, for example
- Nice picture of architecture, what communicates with what and how
- Tip: Require ./ before running .sh file from terminal
- Tip: restart after changing OSC ports, and don't run miniAudicle concurrently if having OSC problems
- Clear API for playalong learning synth class (update this from what's on webpage currently)
- Clear OSC API for updated system (feature extractors and synthesizers)
- ML bibliography page (Seth's request)
- More on how to implement your own feature extractor in max/msp (for Dan 3D knob)
- Tip: Drag from Finder to Terminal to get path
- Explicit info on message used to send params to Wekinator from OSC synth (/realValue f+)
- A clear stub / skeleton for play-along learning class, chuck feature extractor, and OSC feature extractor (DONE)
- Add Anne's vowel feature extractor to example set (DONE)
General topics for conversation
- Wekinator system architecture
- What is required to add your own synth class? feature extractor? (Covered 10/12)
- Metaphors and language for interaction, learning
- October 5, 2:30pm
- Discuss goals for the semester
- Possible outcomes:
- Conference or journal publication
- Compositions to be performed by ourselves, with PLOrk in Spring, other?
- Community software repository: general tools for Wekinator, and/or specific to our compositions/interfaces
- Scheduling - let's do every week for now, as long as people can make it
- Possible papers to read together?
- Things we want Rebecca or someone else to lecture on?
- October 12, 2:30pm
- Walked through system architecture, what a SynthClass looks like
- Konrad showed his Max/MSP synthesis code, which integrates with a larger Max/MSP patch using patter
- Dan brought in the magic 3D knob and hooked it up for feature extraction in Max, then we sent the features out to Wekinator via OSC. Turned it into obnoxious physical model controller.
- Cameron brought in a drum w/ 2 pickups, verified that we could detect "hit" versus "silent" using basic audio features.
- October 19, 2:30pm
- Updates: Included in Wekinator 0.2 download
- Mentioned OSC handout for those new to OSC
- Tried out the foot pedal & a new feature extractor for Cameron's drum
- Tried to try out vowel panning & LiSa code, not compatible with new version. (Sent out compatible code via email Monday evening.)
- Discussed some interface options for data set creation / annotation
- Motivators: What if we want to replace training data? Erase it? Edit it? Forget from a certain time window in the past?
- Want to be able to view, audition, edit data points
- Interesting simple and complex visualization methods to understand & edit training data spac
- Discussed interface issues for creating drum detection dataset
- Create traditional dataset & use CV accuracy to select features & classifier?
- Integrate with methods like onset detection for selective application of Wekinator output?
This is in the early stages. I'll put things here that participants might find useful, that we mention in the meeting, or that people are using to help build their own systems.
General machine learning resources
- Data Mining: Practical Machine Learning Tools and Techniques (book). Witten & Frank.
- This is a nice and very practical introduction to machine learning for people absolutely new to the field. It also serves as a tutorial to the Weka software. The Princeton library has a copy (currently checked out, but reservable). Rebecca highly recommends.
- Pattern recognition and machine learning (book). Bishop.
- This is a more technical introduction to machine learning. A great textbook. Rebecca has a copy and can bring it in if people want to see it.
- Andrew Ng's online course (videos & materials): http://see.stanford.edu/see/courseinfo.aspx?coll=348ca38a-3a6d-4052-937d-cb017338d7b1
Machine learning in music
- Music/audio analysis (ISMIR)
- Gestural control systems
- Compositional systems
Stuff happening at Princeton
- Matt Hoffman's work on audio analysis
<embed src="http://www.box.net//static/flash/box_explorer.swf?widget_hash=899765qtae&v=0&cl=0" width="460" height="345" wmode="transparent" type="application/x-shockwave-flash"></embed>