Tag: sensors

Convert photocell data to MIDI

With Arduino and Max.

Screen Shot 2015-04-26 at 7.27.20 PM

An update to the basic Arduino/Max patches: http://reactivemusic.net/?p=11809

Replace the potentiometer in the “Arduino Serial Read” project with a photocell (LDR: Light dependent resistor)  and a 10K pulldown resistor, wired as shown in the image above and explained here: https://learn.adafruit.com/photocells/using-a-photocell (from Adadfruit)

The “Analog” lead represents the center terminal of a potentiometer and connects to A0.

The Arduino sketch is the example sketch: Analog | analogInOutSerial


Screen Shot 2015-04-26 at 7.50.42 PM


folder: arduino-basics

patch: arduino-serial-read-midi.maxpat

For instructions and circuit, refer to “Arduino Serial Read” project: http://reactivemusic.net/?p=11809

Automobile airplane engine in Max

An update of the automax project

This is a Max patch that generates engine sounds (car, airplane, and spaceship) by reading RPM data from a bluetooth OBD-II sensor in an automobile. It uses Max adaptations of Pd patches by Andy Farnell from “Designing Sound”. And a Fourier filter patch (spaceship) by Katja Vetter.

In this audio clip, an airplane engine sound is mixed with a car engine sound.


The Max patch has been updated to detect available bluetooth devices. The audio example above was done with this device (Bluetooth Supper Mini OBD 2/OBD II ELM 327 Power 2)


But any Elm 327 device should work, as long as it will connect with your  computer.

The device pictured above needs to be deleted and re-paired each time you use it (code: 1234). I would recommend looking for something else.




Main patch


Abstractions and other files
  • engine-overtone.maxpat
  • fourierfilter.maxpat
  • hextoint.maxpat
  • vz.nanoctrlr-tz.maxpat
  • max-pd-abstractions folder (needs to be in Max file path or a subdirectory)


Follow the sequence of events as directed in the patch. Starting by selecting your device from the menu in the upper left corner. If there is a problem with the serial connection you will get “read 0” messages – or an error in the Max window.

Set the polling rate as slow as possible (700 ms.) and then work backwards.

The Korg NanoKontroller works with this patch too.

Muse: development case study

Notes, from: “Making Musical Apps with Csound using libpd and csoundapi~” at the 2nd International Csound Conference ­ October 25th-27th, 2013, in Boston.


For about five months in 2013-2104 I worked as a programmer with Boulanger Labs to develop an app called Muse, using the Leap Motion sensor. Leap Motion is a controller that detects hand movement.  Boulanger Labs is a a startup founded by Dr. Richard Boulanger “Dr. B” – to design music apps working with students in the Electronic Production and Design (EPD) department at Berklee College of Music.

Dr. B. was asked by a former student, Brian Transeau (BT), to help develop a music app in conjunction with Leap Motion. The goal was to have something in stores for Christmas – about 2 months from the time we started. BT would design the app and we would code it.

What would the app do? It would let you to improvise music in real time by moving your hands in the air. You would select notes from parallel horizontal grids of cubes – a melody note from the top, a chord from the middle, and a bass note from the bottom.  It would be be beautiful and evolving like “Bloom” by Eno and Chilvers.

Getting started

We bought Leap Motion sensors. We downloaded apps from the Airspace store to learn about the capabilities of the sensor.

One of our favorite apps is “Flocking”. It displays glowing flames to represent fingers. When you move your fingers it causes a school of fish to disperse.

Making prototypes

We started to make prototypes in Max, using the aka.leapmotion external.

This was the first prototype and one of my favorites. It randomly plays Midi notes in proportion to how you move your fingers. It feels responsive.

Mac Os app: http://reactivemusic.net/?p=7434

Max code: http://reactivemusic.net/?p=11727

Local file: leapfinger3.app (in Applications)

Does it remind you of any of this?

Design sketches from BT

“So this is an idea of the UI paralaxing. In the background it would be black with say stars. You could see your fingertips in this space and your hand movements would effect perspective changes in the UI. When you touch a cube it would light in 3D space radiating out (represented by the lens flares). This flare or light (like bloom) should continue in the direction you touched the cube. Instead of blocks, these would be grids *like 3D graph paper* subdivided into probably 12-24 cubes.”


Stephen Lamb joined the team as a C++ Open GL programmer, and began exploring the Leap Motion API in Cinder C++.

What kind of gestures can we get to work?

Darwin Grosse, of Cycling 74, sent us a new version of aka.leapmotion that handles predefined gestures, like swipes.

The next prototype was written, in CInder C++. An audio proof of concept. The FM oscillators and feedback delay are written at the sample level, using callbacks. The delay line code was borrowed from Julius O. Smith  at CCRMA: http://reactivemusic.net/?p=7513


Delay line code: http://reactivemusic.net/?p=7513

Christopher Konopka, sound designer and programmer, joins the team, but won’t be able to work on the project until October.

At this point we are having doubts about the utility of the Leap Motion sensor for musical apps. Because it is camera-based, the positioning of hands is critical. There is no haptic feedback. We are experiencing high rates of false positives as well as untracked gestures.

More prototypes in Max

  • Finger painting
  • Left right hand detection
  • Detecting state changes
  • Defining gestures (air piano)



Reactive Music

Dr. B asks us to consider RJDJ style environmental effects.

This is when we find out that audio input doesn’t work in Cinder. After staying up until about 6 AM, I decide to run a test of libPd in openFrameworks C++. It works within minutes. libPd allows Pd to run inside of C++. By the way, libPd is the platform used by RJDJ.

Programming notes:


We can now write music using Pd, and graphics using OpenGL C++. This changes everything.

What about Csound? It also runs in Pd. Will it run in libPd? Dr. B introduces me to Victor Lazarrini – author of csoundapi~  and we figure out how to compile Csound into the project that evening.

Paul Batchelor joins the team. He is writing generative music in Csound for a senior project at Berklee. Paul and Christopher write a Csound/Pd prototype, in a couple of days – that will form the musical foundation of the app.

We build a prototype using Paul’s generative Csound music and connect it in to Leap Motion in openFrameworks.

Local file: (leapPdTest5ExampleDebug.app in applications)

In this next video, it feels like we are actually making music.

Note: local source code is ofx8 addons leapmotion : leapPdTest5 – but it probably won’t compile because we moved the libs into the proper folders later on

Combining three prototypes:

This was a week of madness. We had essentially three separate apps that needed to be joined: Steve’s Open GL prototype, my libPd prototype, and Paul’s Csound code. So every time Steve changed the graphics – or Paul modified the Csound code – I needed to re-construct the project.

Finally we were able to upload a single branch of the code to Github.

Tweaking the architecture

Steven Yi, programmer and Csound author, helped repair the xCode linking process. We wanted to be able to install the App without asking users to install Csound or Pd. Steven Yi figures out how to do this in a few hours…

Later that day, for various reasons Steve Lamb leaves the project.

I take over the graphics coding – even through I don’t know OpenGL. BT is justifiably getting impatient. I am exhausted.

Redesigning the graphics

Jonathan Heppner, author of AudioGL, joins the team. Jonathan will redo the graphics and essentially take over the design and development of the app in collaboration with Dr. B.

There is an amazing set of conference calls between Leap Motion, BT, Dr.B, and the development team. Leap Motion gives us several design prototypes – to simplify the UI. Dr. B basically rules them out, and we end up going with a Rubik’s cube design suggested by Jonathan. At one point BT gives a classic explanation of isorhythmic overlapping drum loops.

While Jonathan is getting started with the new UI, We forked a version, to allow us to refine the Osc messaging in Pd.

Christopher develops an extensive control structure in Pd that integrates the OpenGL UI with the backend Csound engine.

Christopher and Paul design a series of sample sets, drawing from nature sounds, samples from BT, Csound effects, and organically generated Csound motif’s. The samples for each set need to be pitched and mastered so they will be compatible with each other.

At this point we move steadily forward – there were no more prototypes, except for experiments, like this one: http://zerokidz.com/ideas/?p=9135 (that did not go over well with the rest of the team :-))

Tom Shani, graphic designer, and Chelsea Southard, interactive media artist, join the team. Tom designs a Web page, screen layouts, logos and icons. Chelsea provides valuable user experience and user interface testing as well as producing video tutorials.

Also, due to NDA’s, development details from this point on are confidential.

We miss the Christmas deadline.

The NAMM show

That brings us up to the NAMM show where BT and Dr. B produce a promotional video and use the App for TV and movie soundtrack cues.



There are more than a few loose ends. The documentation and how-to videos have yet to be completed. There are design and usability issues remaining with the UI.

This has been one of the most exhausting and difficult development projects I have worked on. The pace was accelerated by a series of deadlines. None of the deadlines have been met – but we’re all still hanging in there, somehow. The development process has been chaotic – with flurries of last minute design changes and experiments preceding each of the deadlines. We are all wondering how Dr. B gets by without sleep?

I only hope we can work through the remaining details. The app now makes beautiful sounds and is amazingly robust for its complexity. I think that with simplification of the UI, it will evolve into a cool musical instrument.

In the app store

We scaled back features and added a few new ones including a control panel, a Midi controller interface, a new percussion engine, and sample transposition tweaks. With amazing effort from Christopher, Jonathan, Paul, Chelsea, Tom S., and Dr. B – the app is completed and released!


But why did it get into the Daily Mail? http://www.dailymail.co.uk/sciencetech/article-2596906/Turn-Mac-MUSICAL-INSTRUMENT-App-USB-device-lets-use-hand-gestures-make-sounds.html

Arduino electric eye and musical stairs

Summary of experiments with IR beam detection.

Initial testing with Radio Shack sensors and IRremote library: http://reactivemusic.net/?p=4027

musical stairs project

Built by students at Gould Academy, Bethel, Maine 2013.

Detects movement on stairs using individual IR sensor pairs on each step. When an IR beam is broken, a note is triggered and status LED lights up. Using an Ethernet shield, the data is tracked in a feed at xively.com


Adafruit IR emitters and receivers



construction of sensor units

IR transmitters and receivers wired into terminal strips (no soldering):

transmitting unit:


receiving unit:


layout of stairs

Arduino connections and code


(local file: musicalStairsVersion3tz2)


xively.com feed with Arduino

Bi-directional communication from Arduino to a xively.com feed using an ethernet shield.

  • Initializes an internet connection (DHCP)
  • Connects to xively.com servers every minute
  • Stores random value in the feed using HTTP PUT
  • Retrieves current feed value using HTTP GET
  • Lights up LED when transmitting
By the way, xively used to be cosm used to be pachube… 
Arduino circuit
  • Use an ethernet shield.
  • Connect ethernet cable. (I am using a Netgear WNCE2001 ethernet to wiFi adapter)
  • LED is connected to pin 5 and ground. The shorter lead connects to ground.


xively arduino test

  • xively_test1 (Arduino sketch)
Arduino files and libraries

Copy the xively_test1/ folder to Documents/Arduino. This puts it in the Arduino sketchbook.

Notes on installing xively/cosm/pachube libraries for arduino: http://reactivemusic.net/?p=4900


  1. Connect Arduino to Macbook via USB.
  2. Open the Arduino serial monitor to initialize the ethernet connection and display the IP address.
  3. Every minute data gets send to the feed
  4. Monitor feed data here: https://xively.com/feeds/98281/workbench
Arduino sketch

5/20/2014 - Arduino/xively feed interaction
Uses Ethernet Shield and and LED connected between pin D5 and ground
Sends a random value to a xively.com feed every minute
The LED lights up during data transmissions
HTTP PUT - send data to xiveyly feed and store
HTTP GET - read xively feed value
#include <SPI.h>
#include <Ethernet.h>
#include <HttpClient.h>
#include <Cosm.h>
int ledPin = 5;
int upCount = 0; // counters for number of times going up and down
#define API_KEY "96PqSh4rj7HzNif3WtTpN7GjX96SAKxrWms3SUhwaDFGUT0g" // your Cosm API key
#define FEED_ID 98281 // your Cosm feed ID
// MAC address for your Ethernet shield
byte mac[] = { 0x90, 0xA2, 0xDA, 0x0D, 0x0B, 0xCE };
// note that pins 0 and 1 are used by the Ethernet shield
unsigned long lastConnectionTime = 0; // last time we connected to Cosm
const unsigned long connectionInterval = 60000; // delay between connecting to Cosm in milliseconds
// Initialize the Cosm library
// Define the string for our datastream ID
char sensorId[] = "count";
CosmDatastream datastreams[] = {
 CosmDatastream(sensorId, strlen(sensorId), DATASTREAM_FLOAT),
// Wrap the datastream into a feed
CosmFeed feed(FEED_ID, datastreams, 1 /* number of datastreams */);
EthernetClient client;
CosmClient cosmclient(client);
void setup() {

 // initialize the detector pins 

 pinMode(ledPin, OUTPUT ); // internet transmitting indicator
 // start the Monitor (console) serial port

// display happy messages 

 Serial.println("Xively test");
// Keep trying to initialize the Internet connection
 // Note - we should eventually timeout of this and just run the stairs independently

 Serial.println("Initializing network");
 while (Ethernet.begin(mac) != 1) {
 Serial.println("Error getting IP address via DHCP, trying again...");
Serial.println("Network initialized");
 // print your local IP address:
 Serial.print("Arduino IP address: ");
 for (byte thisByte = 0; thisByte < 4; thisByte++) {
 // print the value of each byte of the IP address:
 Serial.print(Ethernet.localIP()[thisByte], DEC);

} // end of setup function
//////////////////////////// control loop ///////////////////////////
void loop() {
 // main program loop

 ////////////////////////////// Internet sending/receiving code ////////////////////////////////

 if (millis() - lastConnectionTime > connectionInterval) {

 // uncomment this to just send a random value...
 upCount = random(256);

 digitalWrite(ledPin, HIGH ); // turn on transmitter light
 // read the datastream back from Cosm - comment out to save time
 digitalWrite(ledPin, LOW );
 // update connection time so we wait before connecting again
 lastConnectionTime = millis();


 ///////////////////// end of internet send/receive code /////////////////

} // end of main loop code
/////////////////// additional functions //////////////////////////
// send the supplied value to Cosm, printing some debug information as we go
void sendData(int sensorValue) {
Serial.print("Read sensor value ");
Serial.println("Uploading to Cosm");
 int ret = cosmclient.put(feed, API_KEY);
 Serial.print("PUT return code: ");
// get the value of the datastream from Cosm, printing out the value we received
void getData() {
 Serial.println("Reading data from Cosm");
int ret = cosmclient.get(feed, API_KEY);
 Serial.print("GET return code: ");
if (ret > 0) {
 Serial.print("Datastream is: ");
Serial.print("Sensor value is: ");