Sonification of Mass Ave buses, from Nubian to Harvard.
Updated for Max8 and Catalina
This patch requests data from MBTA API to get the current location of buses – using the Max js object. Latitude and Longitude data is mapped to oscillator pitch. Data is polled every 10 seconds, but it seems like the results might be more interesting to poll at a slower rate, because the updates don’t seem that frequent. And buses tend to stop a lot.
This project uses version 3 of the API. There are quality issues with the realtime data. For example, there are bus stops not associated with the route. The direction_id and stop_sequence data from the buses is often wrong. Also, buses that are not in service are not removed from the vehicle list or indicated as such.
The patch uses a [multislider] object to graph the position of the buses along the route – but due to the data problems described above, the positions don’t always reflect the current latitude/longitude coordinates or the bus stop name.
You will need to replace the API key in the message object at the top of the patch with your own key. Or you can probably just remove it. The key distributed with the patch is fake. You can request your own developer API key from MBTA. It’s free.
instructions
Open mbta.maxpat
Open the Max console window so you can see what’s happening with the data
click on the yellow [getstops] message to get the current bus stop data
Toggle the metro (at the top of the patch) to start polling
Turn on the audio (click speaker icon) and turn up the gain
Note: there will be more buses running during rush hours in Boston. Try experimenting with the polling rate and ramp length in the poly-oscillator patch. Also, you can experiment with the pitch range.
This implementation uses node.js for Max instead of Ruby to access the API. You will need set up a developer account with Spotify and request API credentials. See below.
Other than that, the synthesis code in Max has not changed. Some of the following background information and video is from the original version. ..
What if you used that data to reconstruct music by driving a sequencer in Max? The analysis is a series of time based quanta called segments. Each segment provides information about timing, timbre, and pitch – roughly corresponding to rhythm, harmony, and melody.
Edit spot1.js replacing the cliendID and clientSecret with your spotify credentials
node for max install instructions (first time only)
Open the Max patch: spotify-synth1.maxpat
Scroll the patch over to the far right side until you see this green panel:
Click the [script npm init] message – this initializes the node infrastructure in the current folder
Then click each of the 2 script npm install messages – this installs the necessary libraries
Instructions
Open the Max patch: spotify-synth1.maxpat
Click the green [script start] message
Click the Speaker icon to start audio
Click the first dot in the preset object to set the mixer settings to something reasonable
open the Max Console window so you can see the Spotify API data
From the 2 menus at the top of the screen select an Artist and Title that match, for example: Albert Ayler and “Witches and Devils”
Click the [analyze] button – the console window should fill with interest data about your selection.
Click [play]
Note: if you hear a lot of clicks and pops, reduce the audio sample rate to 44.1 KHz.
Alternative search method:
Enter an Artist and Song title for analysis, in the text boxes. Then press the buttons for title and artist. Then press the /analyze button. If it works you will get prompts from the terminal window, the Max window, and you should see the time in seconds in upper right corner of the patch.
troubleshooting
If there are problems with the analysis, its most likely due to one of the following:
artist or title spelled incorrectly
song is not available
song is too long
API is busy
Mixer controls
The Mixer channels from Left to right are:
bass
synth (left)
synth (right)
random octave synth
timbre synth
master volume
gain trim
HPF cutoff frequency
You can also adjust the reverb decay time and the playback rate. Normal playback rate is 1.
programming notes
Best results happen with slow abstract material, like the Miles (Wayne Shorter) piece above. The bass is not really happening. Lines all sound pretty much the same. I’m thinking it might be possible to derive a bass line from the pitch data by doing a chordal analysis of the analysis.
Here are screenshots of the Max sub-patches (the main screen is in the video above)
Timbre (percussion synth) – plays filtered noise:
Random octave synth:
Here’s a Coltrane piece, using roughly the same configuration but with sine oscillators for everything:
There are issues with clicks on the envelopes and the patch is kind of a mess but it plays!
Several modules respond to the API data:
tone synthesiszer (pitch data)
harmonic (random octave) synthesizer (pitch data)
filtered noise (timbre data)
bass synthesizer (key and mode data)
envelope generator (loudness data)
Since the key/mode data is global for the track, bass notes are probable guesses. This method doesn’t work for material with strong root motion or a variety of harmonic content. It’s essentially the same approach I use when asked to play bass at an open mic night.
additional notes
Now that this project is running again. I plan to write additional synthesizers that follow more of the spirit of the data. For example, distinguishing strong pitches from noise.
Also would like to make use of the [section] data as well as the rhythmic analysis. There is an amazing amount of potential here.
Overlapping loops of varying duration to represent natural cycles.
In October I collaborated with Wade Kavanaugh and Stephen P. Nguyen to compose and perform the sounds of a glacier for their installation at the Gem theatre in Bethel, Maine. The glacier was made from paper.
Wade and Stephen:
A time-lapse video of the project:
A time-lapse video of a similar project they did in Minnesota 2005:
The approach was to take a series of ambient loops and organize them by duration. The longer loops would represent the slow movement of time. Shorter loops would represent events like avalanches. One-shot samples would represent quick events, like the cracking of ice.
It took several iterations to produce something slow and boring enough to be convincing. I used samples from the Ron MacLeod’s Cyclic Waves library from Cycling 74 https://www.ableton.com/en/packs/cyclic-waves/. Samples were pitched down to imply largeness.
Each vertical column in an Ableton Live set represents a time-frame of waves. That is, the far left column contains quick events and the far right column contains long cycle events. Left to right, the columns have gradually increasing cycle durations. I used a Push controller to trigger samples in real time as people walked through the theatre to see the glacier.
The theatre speakers were arranged in stereo but from front to back. Since the glacier was also arranged along the same axis, a slow auto-panning effect sent sounds drifting off into the distance, or vice versa. Visually and sonically there was a sense that the space extended beyond the walls of the theatre.
In the “control room” above the theatre… using Push to trigger samples and a Korg NanoKontrol to set panning positions of each track:
The performance lasted about 45 minutes. Occasionally the cracking of ice would startle people in the room. There were kids crawling around underneath the paper glacier. Afterwards we just let the sounds play on their own. A short excerpt:
The Max patch is not available. From the video it appears that many channels of sound are playing concurrently. Color values are assigned to faders for each channel.
Around the year 1700, several startup ventures developed prototypes of machines with thousands of moving parts. After 30 years of engineering, competition, and refinement, the result was a device remarkably similar to the modern piano.
What are the musical instruments of the future being designed right now?
new composition tools,
reactive music,
connecting things,
sensors,
voices,
brains
Notes:
predictions?
Ray Kurzweil’s future predictions on a timeline: http://imgur.com/quKXllo (The Singularity will happen in 2045)
Are there patterns in the ways that artists adapt technology?
For example, the Hammond organ borrowed ideas developed for radios. Recorded music is produced with computers that were originally as business machines.
Instead of looking forward to predict future music, lets look backwards to ask,”What technology needs to happen to make musical instruments possible?” The piano relies upon a single-escapement (1710) and later a double-escapement (1821). Real time pitch shifting depends on Fourier transforms (1822) and fast computers (~1980).
Artists often find new (unintended) uses for tools. Like the printing press.
New pianos
The piano is still in development. In December 2014, Eren Başbuğ composed and performed music on the Roli Seaboard – a piano keyboard made of 3 dimensional sensing foam:
Here is Keith McMillen’s QuNexus keyboard (with Polyphonic aftertouch):
Here are tools that might lead to new ways of making music. They won’t replace old ways. Singing has outlasted every other kind of music.
These ideas represent a combination of engineering and art. Engineers need artists. Artists need engineers. Interesting things happen at the confluence of streams.
Analysis, re-synthesis, transformation
Computers can analyze the audio spectrum in real time. Sounds can be transformed and re-synthesized with near zero latency.
Infinite Jukebox
Finding alternate routes through a song.
by Paul Lamere at the Echonest
Echonest has compiled data on over 14 million songs. This is an example of machine learning and pattern matching applied to music.
Harmonic Model Plus Residual (HPR) – Build a spectrogram using STFT, then identify where there is strong correlation to a tonal harmonic structure (music). This is the harmonic model of the sound. Subtract it from the original spectrogram to get the residual (noise).
“Detetcting Drops in EDM” – by Karthik Yadati, Martha Larson, Cynthia C. S. Liem, Alan Hanjalic at Delft University of Technology (2014) https://reactivemusic.net/?p=17711
Polyphonic audio editing
Blurring the distinction between recorded and written music.
What is the speed of electricity? 70-80 ms is the best round trip latency (via fiber) from the U.S. east to west coast. If you were jamming over the internet with someone on the opposite coast it might be like being 100 ft away from them in a field. (sound travels 1100 feet/second in air).
Global communal experiences – Bill McKibben – 1990 “The Age of Missing Information”
There is a quickening of discovery: internet collaboration, open source, linux, github, r-pi, Pd, SDR.
“Robots and AI will help us create more jobs for humans — if we want them. And one of those jobs for us will be to keep inventing new jobs for the AIs and robots to take from us. We think of a new job we want, we do it for a while, then we teach robots how to do it. Then we make up something else.”
“…We invented machines to take x-rays, then we invented x-ray diagnostic technicians which farmers 200 years ago would have not believed could be a job, and now we are giving those jobs to robot AIs.”
Sonification of Mass Ave buses, from Harvard to Dudley.
This patch sends requests to the MBTA developer portal to get the current location of buses – using the Max js object. Latitude and Longitude data is mapped to oscillator pitch. Data is polled every 10 seconds, but it seems like the results might be more interesting to poll at a slower rate, because the updates don’t seem that frequent. And buses tend to stop a lot.
You will not need authentication to run run this patch. It uses the default developer API-key for testing. Please read the terms of service at the MBTA developer portal. Data should not be polled more often than 10 seconds. You can also request your own developer API key from MBTA.
instructions
Open mbta.maxpat
Toggle the metro (at the top of the patch) to start polling
Turn on the audio (at the bottom of the patch) and turn up the gain
Note: there will be more buses running during rush hours in Boston. Try experimenting with the polling rate and ramp length in the poly-oscillator patch. Also, you can experiment with the pitch range.
The Max patch is based on a tutorial by dude837 called “Automatic Silly Video Generator”
download
The patch at the download link in the video is broken – but the javascript code for the Max js object is intact. You can download the entire patch from the Max-projects archive: https://github.com/tkzic/max-projects folder: maxvine
Internet API’s
API’s (application programming interfaces) provide methods for programs (other than web browsers) to access Internet data. Any app that access data from the web uses an API.
For example, if you copy this URL into a web browser address bar, it will return a block of data in JSON format about the most popular videos on Vine: https://api.vineapp.com/timelines/popular
HTTP requests
An HTTP request transfers data to or from a server. A web browser handles HTTP requests in the background. You can also write programs that make HTTP requests. A program called “curl” runs http requests from the terminal command line. Here are examples: https://reactivemusic.net/?p=5916
Response data
Data is usually returned in one of 3 formats:
JSON
XML
HTML
JSON is the preferred method because its easy to access the data structure.
Max HTTP requests
There are several ways to make HTTP requests in Max, but the best method is the js object: Here is the code that runs the GET request for the Vine API:
function get(url)
{
var ajaxreq = new XMLHttpRequest();
ajaxreq.open("GET", url);
ajaxreq.onreadystatechange = readystatechange;
ajaxreq.send();
}
function readystatechange()
{
var rawtext = this._getResponseKey("body");
var body = JSON.parse(rawtext);
outlet(0, body.data.records[0].videoUrl);
}
The function: get() formats and sends an HTTP request using the URL passed in with the get message from Max. When the data is returned to Max, the readystatechange() function parses it and sends the URL of the most popular Vine video out the left outlet of the js object.
Playing Internet audio/video files in Max
The qt.movie object will play videos, with the URL passed in by the read message.
Unfortunately, qt.movie sends its audio to the system, not to Max. You can use Soundflower, or a virtual audio routing app, to get the audio back into Max.
