Category: interactive media art

Posted on

EchoNest segment analysis player in Max

The Echonest API provides sample level audio analysis.

http://developer.echonest.com/docs/v4/_static/AnalyzeDocumentation.pdf

What if you used that data to reconstruct music by driving a sequencer in Max? The analysis is a series of time based quanta called segments. Each segment provides information about timing, timbre, and pitch – roughly corresponding to rhythm, harmony, and melody.

download

https://github.com/tkzic/internet-sensors

folder: echo-nest

files

main Max patch
  • echonest-synth4.maxpat
abstractions and other files
  • polyvoice-sine.maxpat
  • polyvoice2.maxpat
ruby server
  • echonest-synth2.rb

authentication

You will need to sign up for a developer account at The Echo Nest, and get an API key. https://developer.echonest.com

Edit the ruby server file: echonest-synth2.rb replacing the API with your new API from echonest

 

installing ruby gems

Install the following ruby gems (from the terminal):

gem install patron

gem install osc-ruby

gem install json

gem install uri

instructions

1. In Terminal run the ruby server:

./echonest-synth2.rb

2. Open the Max patch: echonest-synth4.maxpat and turn on the audio.

3. Enter an Artist and Song title for analysis, in the text boxes. Then press the greet buttons for title and artist. Then press the /analyze button. If it works you will get prompts from the terminal window, the Max window, and you should see the time in seconds in upper right corner of the patch.

If there are problems with the analysis, its most likely due to one of the following:

  • artist or title spelled incorrectly
  • song is not available
  • song is too long
  • API is busy
If the ruby server hangs or crashes, just restart it and try again.

3. Press one of the preset buttons to turn on the tracks.

4. Now you can play the track by pressing the /play button.

The Mixer channels from Left to right are:

  • bass
  • synth (left)
  • synth (right)
  • random octave synth
  • timbre synth
  • master volume
  • gain trim
  • HPF cutoff frequency
You can also adjust the reverb decay time and the playback rate. Normal playback rate is 1.

programming notes

Best results happen with slow abstract material, like the Miles (Wayne Shorter) piece above. The bass is not really happening. Lines all sound pretty much the same. I’m thinking it might be possible to derive a bass line from the pitch data by doing a chordal analysis of the analysis.

Here are screenshots of the Max sub-patches (the main screen is in the video above)

Timbre (percussion synth) – plays filtered noise:

Random octave synth:

Here’s a Coltrane piece, using roughly the same configuration but with sine oscillators for everything:

There are issues with clicks on the envelopes and the patch is kind of a mess but it plays!

Several modules respond to the API data:

  • tone synthesiszer (pitch data)
  • harmonic (random octave) synthesizer (pitch data)
  • filtered noise (timbre data)
  • bass synthesizer (key and mode data)
  • envelope generator (loudness data)

Since the key/mode data is global for the track, bass notes are probable guesses. This method doesn’t work for material with strong root motion or a variety of harmonic content. Its essentially the same approach I use when asked to play bass at an open mic night.

The envelopes click at times – it may be due to the relaxed method of timing, i.e.., none at all. If they don’t go away when timing is corrected, this might get cleaned up by adding a few milliseconds to the release time – or looking ahead to make sure the edges of  segments are lining up.

[update] Using the Max [poly~] object cleared up the clicking and distortion issues.

Timbre data drives a random noise filter machine. I just patched something together and it sounded responsive – but its kind of hissy – an LPF might make it less interesting.

Haven’t used any of the beat, tatum, or section data yet. The section data should be useful for quashing monotony.

another update – 4/2013

tried to write this into a Max4Live device – so that the pitch data would be played my a Midi (software) instrument. No go. The velocity data gets interpreted in mysterious ways – plus each instrument has its own envelope which interferes with the segment envelopes. Need to think this through. One idea would be to write a device which uses EN analysis data for beats to set warp markers in Live. It would be an amazing auto-warp function for any song. Analysis wars: Berlin vs. Somerville.

Posted on

echonest API wind experiment

Since I had been thinking a lot about how to use data streams to compose music, it seemed like it would be cool to reverse the process. To use music as a data stream to control something.

The Echonest API http://developer.echonest.com analyzes audio content at the sample level. I requested an analysis of ‘Free Bird’ by Lynyrd Skynyrd, thinking this song has dynamic variation so it might sound like a wind event. Segment loudness data was connected to wind speed in the simulation and you could hear, as the song progressed, how the wind matched the song’s dynamic levels. But there wasn’t much variation – mostly a steady build from very low to very high. Tried connecting other analysis data to wind speed and found that the segment confidence values sounded like a ‘gusty’ kind of wind. Not wanting to lose the excitement of the song, the loudness peaks now trigger thunder – using the pd threshold object – with random timing to thin it out a bit. Also there’s no thunder until the loudness values stay above the threshold for several seconds.

API data is being served via OSC from a ruby script. For this first test I ran analysis using curl and saved it to a file. Was thinking of using this ruby GEM for the API calls:

https://github.com/youpy/ruby-echonest

But will first try hooking them up using Patron (curl) because it offers more flexibility to get at all aspects of the API.

This project is not included in internet-sensors repo yet.

Local Files:

  • tkzic/internetsensors/
  • wind-echonest.rb
  • echonest.txt (curl examples)
  • enfb-analysis.json (analysis data)
  • wind-echonest1.rb

 

 

Posted on

Open weather map API queries

edit 4/27/2015

This curl examples for this API are broken. – the API now requires a key: http://openweathermap.org/api

notes

I’m using this API now instead of the CORDC wind forecast data for the internet-sensors wind example. You can get 7 day forecast data for practically anywhere. Also historical data is available. And its free.

Get forecast by city id

Weather forecast in the city for the next 7 days.

http://api.openweathermap.org/data/2.1/forecast/city/{CITY_ID}

http://api.openweathermap.org/data/2.1/forecast/city/524901

{"message":"","cod":"200","calctime":0.0189,"list":[
{"dt":1345251600,
    "main":{"temp":286.6,"humidity":98,"pressure":1002,"temp_min":286,"temp_max":287},
    "wind":{"speed":0,"deg":-2},
    "rain":{"3h":2},
    "clouds":{"all":56},
    "weather":{"id":803,"main":"Clouds","description":"broken clouds", "img":"..." }
....

This gets city code for santa cruz – 5393052

curl http://api.openweathermap.org/data/2.1/find/name?q=santa%20cruz,US

this gets 7 days of santa cruz forecasts with time stamps

curl http://api.openweathermap.org/data/2.1/forecast/city/5393052

typical JSON response (for one datapoint):

{
		"dt": 1364176800,
		"main": {
			"temp": 285.54,
			"temp_min": 282.6,
			"temp_max": 287.5,
			"pressure": 1015.99,
			"humidity": 87.6,
			"temp_kf": 2.94
		},
		"weather": [{
			"id": 801,
			"main": "Clouds",
			"description": "few clouds",
			"icon": "02n"
		}],
		"clouds": {
			"all": 17,
			"low": 0,
			"middle": 0,
			"high": 17
		},
		"wind": {
			"speed": 4.29,
			"deg": 311,
			"gust": 5.1
		},
		"dt_txt": "2013-03-25 02:00:00"
	}

So… to get precipitation, we need to just look for “rain”, or “snow”  indicator

 

Posted on

domain ping machine in Web Audio

A ‘mini’ version of the Google domain ping synthesizer from the internet-sensors collection (Using the Mashape API). This one runs in Web Audio, using the Web Audio Playground with OSC.

Looks like a card game. Anyway it sounds cool. Doesn’t have the panning of the original, but it has an organic sound due to portamento in frequency changes, and more ‘beating’. Here’s a short excerpt.

Another example of Max controlling WAP https://reactivemusic.net/?p=6193

download

https://github.com/tkzic/WebAudio

folder is: WebAudio/osctest/

files

  • wapOSCserver-ping.rb
  • wapPingTest.maxpat
  • WAP patch: – ping2 (5 osc’s -> 5 gains, -> 1 master gain) – ping2.json
  • Web Page: WebAudio/index.html

instructions

update: you can run an online version of WAP Web client at http://zerokidz.com/wap/index.html – If you load this page, skip to step 3.

1. run the node webserver in WebAudio

node nodeserver.js

(it will run on localhost port 8081 – for example http://127.0.0.1:8081)

2. In Chrome web browser, run: 127.0.0.1:8081/index.html

3. From a terminal window, go to the osctest/ folder and start the server by typing:

./wapOSCserver-ping.rb

4. Load the Max patch:

wapPingTest.maxpat

5. In Chrome, click the OSC button – the ruby server should open a socket connection

6. Also in Chrome, load the patch: ping2 (note that there is a json copy of this patch ping2.json that can be pasted in, if it doesn’t show up in the menu)

6.5 In WAP, Click the square buttons on the 5 Oscillators to start them playing. You should hear sounds at this point.

7. Now back in Max patch – click green toggle to start polling and you probably want to increase the polling rate to about 50 ms instead of 1000 ms

suggestions
  • If it doesn’t seem like there is much action in the patch, try adjusting the FREQ_MULT and GAIN_MULT inside the ruby script.
  • You will probably also want to open the developer javascript console in Chrome to see what is going on.