Office hours: Tuesday 1-2 PM, or Tuesday 4-5PM, at the EPD office #401 at 161 Mass Ave. Please email or call ahead.
Assignments and class notes will be posted to this blog: https://reactivemusic.net before or after the class. Search for: ep-426 to find the notes
Examples, software, links, and references demonstrated in class are available for you to use. If there is something missing from the notes, please ask about it. This is your textbook.
Syllabus:
Everybody calls this course “The Jitter class” – referring to Max/MSP jitter from Cycling 74. You will learn to use Jitter. But the object is to create interactive visual art. Jitter is one tool of many available.
The field of interactive visual art is constantly evolving.
After you take the course, you will have designed projects. You might design a new tool for other artists. You will have opportunities to solve problems. You will become familiar with how others make interactive art. You will explore the connection between sound, video, graphics, sensors, and data. You will be exposed to to a world of possibilities – which you may embrace or reject.
We will explore a range of methods and have opportunities to use them in projects. We’ll look at examples by artists – asking the question: How does that work?
Topics: (subject to change)
Jitter
Matrixes
Reverse engineering
Visualization of audio
Visualization of live data, API’s
Video analysis (realtime)
Video hardware and controllers
Prototyping
Video signal processing
OpenGL
Other tools: Processing, WebGL, Canvas, 2d graphics
Portfolios
Live performance
Grading and projects:
Grades are based on two projects that you will design – and class participation. Please see Neil Leonard’s EP-426 syllabus for details. I encourage and will give credit for: collaboration with other students, outside projects, performances, independent projects, and anything else that will foster your growth and success.
I am open to alternative projects. For example, if you want to use this course as an opportunity to develop a larger project or continue a work in progress.
Office hours: Tuesday 1-2 PM, or Tuesday 4-5PM, at the EPD office #401 at 161 Mass Ave. Please email or call ahead.
Assignments and class notes will be posted to this blog: https://reactivemusic.net before or after the class. Search for: ep-341 to find the notes
Examples, software, links, and references demonstrated in class are available for you to use. If there is something missing from the notes, please ask about it. This is your textbook.
Syllabus:
Prototyping is the focus. Max is a seed that has grown into music, art, discoveries, products, and entire businesses.
After you take the course, you will have developed several projects. You might design a musical instrument or a plugin. You will have opportunities to solve problems. But mostly you will have a sense of how to explore possibilities by building prototypes in Max. You will have the basic skills to quickly make software to connect things, and answer questions like, “Is it possible to make something that does x?”.
You will become familiar with how other artists use Max to make things. You will be exposed to to a world of possibilities – which you may embrace or reject.
We will explore a range of methods and have opportunities to use them in projects. We’ll look at examples by artists – asking the question: How does this work?
Success depends on execution as well as good ideas.
Topics: (subject to change)
Max
Reverse engineering
Transforming and scaling data
Designing user interfaces
Messages and communication, MIDI/OSC
randomness and probability
Connecting hardware and other devices
Working with sensors, data, and API’s
Audio signal processing and synthesis.
Problem solving, prototyping, portfolios.
plugins, Max for Live.
Basic video processing and visualization
Alternative tools: Pd
Max externals
How to get ideas
Computers and Live performance
Transcoding
Grading and projects:
Grades will be assigned projects, several small assignments/quizzes, and class participation. Please see Neil Leonard’s EP-341 syllabus for details. I encourage and will give credit for: collaboration with other students, outside projects, performances, independent projects, and anything else that will encourage your growth and success.
I am open to alternative projects. For example, if you want to use this course as an opportunity to develop a larger project or continue a work in progress.
Around the year 1700, several startup ventures developed prototypes of machines with thousands of moving parts. After 30 years of engineering, competition, and refinement, the result was a device remarkably similar to the modern piano.
What are the musical instruments of the future being designed right now?
new composition tools,
reactive music,
connecting things,
sensors,
voices,
brains
Notes:
predictions?
Ray Kurzweil’s future predictions on a timeline: http://imgur.com/quKXllo (The Singularity will happen in 2045)
Are there patterns in the ways that artists adapt technology?
For example, the Hammond organ borrowed ideas developed for radios. Recorded music is produced with computers that were originally as business machines.
Instead of looking forward to predict future music, lets look backwards to ask,”What technology needs to happen to make musical instruments possible?” The piano relies upon a single-escapement (1710) and later a double-escapement (1821). Real time pitch shifting depends on Fourier transforms (1822) and fast computers (~1980).
Artists often find new (unintended) uses for tools. Like the printing press.
New pianos
The piano is still in development. In December 2014, Eren Başbuğ composed and performed music on the Roli Seaboard – a piano keyboard made of 3 dimensional sensing foam:
Here is Keith McMillen’s QuNexus keyboard (with Polyphonic aftertouch):
Here are tools that might lead to new ways of making music. They won’t replace old ways. Singing has outlasted every other kind of music.
These ideas represent a combination of engineering and art. Engineers need artists. Artists need engineers. Interesting things happen at the confluence of streams.
Analysis, re-synthesis, transformation
Computers can analyze the audio spectrum in real time. Sounds can be transformed and re-synthesized with near zero latency.
Infinite Jukebox
Finding alternate routes through a song.
by Paul Lamere at the Echonest
Echonest has compiled data on over 14 million songs. This is an example of machine learning and pattern matching applied to music.
Harmonic Model Plus Residual (HPR) – Build a spectrogram using STFT, then identify where there is strong correlation to a tonal harmonic structure (music). This is the harmonic model of the sound. Subtract it from the original spectrogram to get the residual (noise).
“Detetcting Drops in EDM” – by Karthik Yadati, Martha Larson, Cynthia C. S. Liem, Alan Hanjalic at Delft University of Technology (2014) https://reactivemusic.net/?p=17711
Polyphonic audio editing
Blurring the distinction between recorded and written music.
What is the speed of electricity? 70-80 ms is the best round trip latency (via fiber) from the U.S. east to west coast. If you were jamming over the internet with someone on the opposite coast it might be like being 100 ft away from them in a field. (sound travels 1100 feet/second in air).
Global communal experiences – Bill McKibben – 1990 “The Age of Missing Information”
There is a quickening of discovery: internet collaboration, open source, linux, github, r-pi, Pd, SDR.
“Robots and AI will help us create more jobs for humans — if we want them. And one of those jobs for us will be to keep inventing new jobs for the AIs and robots to take from us. We think of a new job we want, we do it for a while, then we teach robots how to do it. Then we make up something else.”
“…We invented machines to take x-rays, then we invented x-ray diagnostic technicians which farmers 200 years ago would have not believed could be a job, and now we are giving those jobs to robot AIs.”
We instantly recognize people and animals by their voices. As an artist we work to develop our own voice. Voices contain information beyond words. Think of R2D2 or Chewbacca.
There is also information between words: “Palin Biden Silences” David Tinapple, 2008: http://vimeo.com/38876967
Ableton Live example: Local file: Max/MSP: examples/effects/classic-vocoder-folder/classic_vocoder.maxpat
Max vocoder tutorial (In the frequency domain), by dude837 – Sam Tarakajian https://reactivemusic.net/?p=17362 (local file: dude837/4-vocoder/robot-master.maxpat
Harmonic Model Plus Residual (HPR) – Build a spectrogram using STFT, then identify where there is strong correlation to a tonal harmonic structure (music). This is the harmonic model of the sound. Subtract it from the original spectrogram to get the residual (noise).
This method was used to send secret messages during world war 2. Its now used in cell phones to get rid of echo. Its also used in noise canceling headphones.
This is a Max patch that generates engine sounds (car, airplane, and spaceship) by reading RPM data from a bluetooth OBD-II sensor in an automobile. It uses Max adaptations of Pd patches by Andy Farnell from “Designing Sound”. And a Fourier filter patch (spaceship) by Katja Vetter.
In this audio clip, an airplane engine sound is mixed with a car engine sound.
The Max patch has been updated to detect available bluetooth devices. The audio example above was done with this device (Bluetooth Supper Mini OBD 2/OBD II ELM 327 Power 2)
max-pd-abstractions folder (needs to be in Max file path or a subdirectory)
Instructions
Follow the sequence of events as directed in the patch. Starting by selecting your device from the menu in the upper left corner. If there is a problem with the serial connection you will get “read 0” messages – or an error in the Max window.
Set the polling rate as slow as possible (700 ms.) and then work backwards.
The Korg NanoKontroller works with this patch too.
Notes, from: “Making Musical Apps with Csound using libpd and csoundapi~” at the 2nd International Csound Conference October 25th-27th, 2013, in Boston.
Overview
For about five months in 2013-2104 I worked as a programmer with Boulanger Labs to develop an app called Muse, using the Leap Motion sensor. Leap Motion is a controller that detects hand movement. Boulanger Labs is a a startup founded by Dr. Richard Boulanger “Dr. B” – to design music apps working with students in the Electronic Production and Design (EPD) department at Berklee College of Music.
Dr. B. was asked by a former student, Brian Transeau (BT), to help develop a music app in conjunction with Leap Motion. The goal was to have something in stores for Christmas – about 2 months from the time we started. BT would design the app and we would code it.
What would the app do? It would let you to improvise music in real time by moving your hands in the air. You would select notes from parallel horizontal grids of cubes – a melody note from the top, a chord from the middle, and a bass note from the bottom. It would be be beautiful and evolving like “Bloom” by Eno and Chilvers.
Getting started
We bought Leap Motion sensors. We downloaded apps from the Airspace store to learn about the capabilities of the sensor.
One of our favorite apps is “Flocking”. It displays glowing flames to represent fingers. When you move your fingers it causes a school of fish to disperse.
Making prototypes
We started to make prototypes in Max, using the aka.leapmotion external.
This was the first prototype and one of my favorites. It randomly plays Midi notes in proportion to how you move your fingers. It feels responsive.
“So this is an idea of the UI paralaxing. In the background it would be black with say stars. You could see your fingertips in this space and your hand movements would effect perspective changes in the UI. When you touch a cube it would light in 3D space radiating out (represented by the lens flares). This flare or light (like bloom) should continue in the direction you touched the cube. Instead of blocks, these would be grids *like 3D graph paper* subdivided into probably 12-24 cubes.”
Best,
_BT
Research:
Stephen Lamb joined the team as a C++ Open GL programmer, and began exploring the Leap Motion API in Cinder C++.
What kind of gestures can we get to work?
Darwin Grosse, of Cycling 74, sent us a new version of aka.leapmotion that handles predefined gestures, like swipes.
The next prototype was written, in CInder C++. An audio proof of concept. The FM oscillators and feedback delay are written at the sample level, using callbacks. The delay line code was borrowed from Julius O. Smith at CCRMA: https://reactivemusic.net/?p=7513
Christopher Konopka, sound designer and programmer, joins the team, but won’t be able to work on the project until October.
At this point we are having doubts about the utility of the Leap Motion sensor for musical apps. Because it is camera-based, the positioning of hands is critical. There is no haptic feedback. We are experiencing high rates of false positives as well as untracked gestures.
Dr. B asks us to consider RJDJ style environmental effects.
This is when we find out that audio input doesn’t work in Cinder. After staying up until about 6 AM, I decide to run a test of libPd in openFrameworks C++. It works within minutes. libPd allows Pd to run inside of C++. By the way, libPd is the platform used by RJDJ.
We can now write music using Pd, and graphics using OpenGL C++. This changes everything.
What about Csound? It also runs in Pd. Will it run in libPd? Dr. B introduces me to Victor Lazarrini – author of csoundapi~ and we figure out how to compile Csound into the project that evening.
Paul Batchelor joins the team. He is writing generative music in Csound for a senior project at Berklee. Paul and Christopher write a Csound/Pd prototype, in a couple of days – that will form the musical foundation of the app.
We build a prototype using Paul’s generative Csound music and connect it in to Leap Motion in openFrameworks.
Local file: (leapPdTest5ExampleDebug.app in applications)
In this next video, it feels like we are actually making music.
Note: local source code is ofx8 addons leapmotion : leapPdTest5 – but it probably won’t compile because we moved the libs into the proper folders later on
Combining three prototypes:
This was a week of madness. We had essentially three separate apps that needed to be joined: Steve’s Open GL prototype, my libPd prototype, and Paul’s Csound code. So every time Steve changed the graphics – or Paul modified the Csound code – I needed to re-construct the project.
Finally we were able to upload a single branch of the code to Github.
Tweaking the architecture
Steven Yi, programmer and Csound author, helped repair the xCode linking process. We wanted to be able to install the App without asking users to install Csound or Pd. Steven Yi figures out how to do this in a few hours…
Later that day, for various reasons Steve Lamb leaves the project.
I take over the graphics coding – even through I don’t know OpenGL. BT is justifiably getting impatient. I am exhausted.
Redesigning the graphics
Jonathan Heppner, author of AudioGL, joins the team. Jonathan will redo the graphics and essentially take over the design and development of the app in collaboration with Dr. B.
There is an amazing set of conference calls between Leap Motion, BT, Dr.B, and the development team. Leap Motion gives us several design prototypes – to simplify the UI. Dr. B basically rules them out, and we end up going with a Rubik’s cube design suggested by Jonathan. At one point BT gives a classic explanation of isorhythmic overlapping drum loops.
While Jonathan is getting started with the new UI, We forked a version, to allow us to refine the Osc messaging in Pd.
Christopher develops an extensive control structure in Pd that integrates the OpenGL UI with the backend Csound engine.
Christopher and Paul design a series of sample sets, drawing from nature sounds, samples from BT, Csound effects, and organically generated Csound motif’s. The samples for each set need to be pitched and mastered so they will be compatible with each other.
At this point we move steadily forward – there were no more prototypes, except for experiments, like this one: http://zerokidz.com/ideas/?p=9135 (that did not go over well with the rest of the team :-))
Tom Shani, graphic designer, and Chelsea Southard, interactive media artist, join the team. Tom designs a Web page, screen layouts, logos and icons. Chelsea provides valuable user experience and user interface testing as well as producing video tutorials.
Also, due to NDA’s, development details from this point on are confidential.
We miss the Christmas deadline.
The NAMM show
That brings us up to the NAMM show where BT and Dr. B produce a promotional video and use the App for TV and movie soundtrack cues.
There are more than a few loose ends. The documentation and how-to videos have yet to be completed. There are design and usability issues remaining with the UI.
This has been one of the most exhausting and difficult development projects I have worked on. The pace was accelerated by a series of deadlines. None of the deadlines have been met – but we’re all still hanging in there, somehow. The development process has been chaotic – with flurries of last minute design changes and experiments preceding each of the deadlines. We are all wondering how Dr. B gets by without sleep?
I only hope we can work through the remaining details. The app now makes beautiful sounds and is amazingly robust for its complexity. I think that with simplification of the UI, it will evolve into a cool musical instrument.
In the app store
We scaled back features and added a few new ones including a control panel, a Midi controller interface, a new percussion engine, and sample transposition tweaks. With amazing effort from Christopher, Jonathan, Paul, Chelsea, Tom S., and Dr. B – the app is completed and released!
