Tag: DSP

New musical instruments

A presentation for Berklee BTOT 2015 http://www.berklee.edu/faculty 


Around the year 1700, several startup ventures developed prototypes of machines with thousands of moving parts. After 30 years of engineering, competition, and refinement, the result was a device remarkably similar to the modern piano.

What are the musical instruments of the future being designed right now?

  • new composition tools,
  • reactive music,
  • connecting things,
  • sensors,
  • voices, 
  • brains



Ray Kurzweil’s future predictions on a timeline: http://imgur.com/quKXllo (The Singularity will happen in 2045)

In 1965 researcher Herbert Simon said: “Machines will be capable, within twenty years, of doing any work a man can do”. Marvin Minsky added his own prediction: “Within a generation … the problem of creating ‘artificial intelligence’ will substantially be solved.” https://forums.opensuse.org/showthread.php/390217-Will-computers-or-machines-ever-become-self-aware-or-evolve/page2


Are there patterns in the ways that artists adapt technology?

For example, the Hammond organ borrowed ideas developed for radios. Recorded music is produced with computers that were originally as business machines.

Instead of looking forward to predict future music, lets look backwards to ask,”What technology needs to happen to make musical instruments possible?” The piano relies upon a single-escapement (1710) and later a double-escapement (1821). Real time pitch shifting depends on Fourier transforms (1822) and fast computers (~1980).

Artists often find new (unintended) uses for tools. Like the printing press.

New pianos

The piano is still in development. In December 2014, Eren Başbuğ composed and performed music on the Roli Seaboard – a piano keyboard made of 3 dimensional sensing foam:

Here is Keith McMillen’s QuNexus keyboard (with Polyphonic aftertouch):



Here are tools that might lead to new ways of making music. They won’t replace old ways. Singing has outlasted every other kind of music.

These ideas represent a combination of engineering and art. Engineers need artists. Artists need engineers. Interesting things happen at the confluence of streams.

Analysis, re-synthesis, transformation

Computers can analyze the audio spectrum in real time. Sounds can be transformed and re-synthesized with near zero latency.

Infinite Jukebox

Finding alternate routes through a song.

by Paul Lamere at the Echonest

Echonest has compiled data on over 14 million songs. This is an example of machine learning and pattern matching applied to music.


Try examples: “Karma Police”, Or search for: “Albert Ayler”)

Remixing a remix

“Mindblowing Six Song Country Mashup”: https://www.youtube.com/watch?v=FY8SwIvxj8o (start at 0:40)

Screen Shot 2015-01-09 at 11.25.13 PM

Local file: Max teaching examples/new-country-mashup.mp3

More about Echonest

Feature detection

Looking at music under a microscope.

removing music from speech

First you have to separate them.


by Xavier Serra and UPF

Harmonic Model Plus Residual (HPR) – Build a spectrogram using STFT, then identify where there is strong correlation to a tonal harmonic structure (music). This is the harmonic model of the sound. Subtract it from the original spectrogram to get the residual (noise).

Screen Shot 2015-01-06 at 1.40.37 AM

Screen Shot 2015-01-06 at 1.40.12 AM

Settings for above example:

  • Window size: 1800 (SR / f0 * lobeWidth) 44100 / 200 * 8 = 1764
  • FFT size: 2048
  • Mag threshold: -90
  • Max harmonics: 30
  • f0 min: 150
  • f0 max: 200
Many kinds of features
  • Low level features: harmonicity, amplitude, fundamental frequency
  • high level features: mood, genre, danceability
Examples of feature detection
Music information retrieval

Finding the drop

“Detetcting Drops in EDM” – by Karthik Yadati, Martha Larson, Cynthia C. S. Liem, Alan Hanjalic at Delft University of Technology (2014) http://reactivemusic.net/?p=17711

Polyphonic audio editing

Blurring the distinction between recorded and written music.


by Celemony


A minor version of “Bohemian Rhapsody”: http://www.youtube.com/watch?v=voca1OyQdKk

Music recognition

“How Shazam Works” by Farhoud Manjoo at Slate: http://reactivemusic.net/?p=12712, “About 3 datapoints per second, per song.”

  • Music fingerprinting: https://musicbrainz.org/doc/Fingerprinting
  • Humans being computers. Mystery sounds. (Local file: Desktop/mystery sounds)
  • Is it more difficult to build a robot that plays or one that listens?

Sonographic sound processing

Transforming music through pictures.

by Tadej Droljc


(Example of 3d speech processing at 4:12)

local file: SSP-dissertation/4 – Max/MSP/Jitter Patch of PV With Spectrogram as a Spectral Data Storage and User Interface/basic_patch.maxpat

Try recording a short passage, then set bound mode to 4, and click autorotate

Spectral scanning in Ableton Live:

Web Audio

Web browser is the new black


by Joe Berkowitz 



by Dinahmoe


Can you jam over the internet?

What is the speed of electricity? 70-80 ms is the best round trip latency (via fiber) from the U.S. east to west coast. If you were jamming over the internet with someone on the opposite coast it might be like being 100 ft away from them in a field. (sound travels 1100 feet/second in air).

Global communal experiences – Bill McKibben – 1990 “The Age of Missing Information”

More about Web Audio

Conversation with robots

Computers finding meaning

The Google speech API


The Google speech API uses neural networks, statistics, and large quantities of data.

Microsoft: real-time translation

Reverse entropy


Making music from from sounds that are not music.

by Katja Vetter

. (InstantDecomposer is an update of SliceJockey2):   http://www.katjaas.nl/slicejockey/slicejockey.html

  • local: InstantDecomposer version: tkzic/pdweekend2014/IDecTouch/IDecTouch.pd
  • local: slicejockey2test2/slicejockey2test2.pd
More about reactive music

Sensors and sonification

Transforming motion into music

Three approaches
  • earcons (email notification sound)
  • models (video game sounds)
  • parameter mapping (Geiger counter)
Leap Motion

camera based hand sensor

“Muse” (Boulanger Labs) with Paul Bachelor, Christopher Konopka, Tom Shani, and Chelsea Southard: http://reactivemusic.net/?p=16187

Max/MSP piano example: Leapfinger: http://reactivemusic.net/?p=11727

local file: max-projects/leap-motion/leapfinger2.maxpat

Internet sensors project

Detecting motion from the Internet


Twitter streaming example


MBTA bus data

 Sonification of Mass Ave buses, from Harvard to Dudley


Screen Shot 2014-11-11 at 3.26.16 PM

Stock market music


More sonification projects
Vine API mashup

By Steve Hensley

Using Max/MSP/jitter

local file: tkzic/stevehensely/shensley_maxvine.maxpat

Audio sensing gloves for spacesuits

By Christopher Konopka at future, music, technology


Computer Vision

Sensing motion with video using frame subtraction

by Adam Rokhsar


local file: max-projects/frame-subtraction

The brain

Music is stored all across the brain.

Mouse brain wiring diagram

The Allen institute


“Hacking the soul” by Christof Koch at the Allen institute

(An Explanation of the wiring diagram of the mouse brain – at 13:33) http://www.technologyreview.com/emtech/14/video/watch/christof-koch-hacking-the-soul/

OpenWorm project

A complete simulation of the nematode worm, in software, with a Lego body (320 neurons)



Harold Cohen’s algorithmic painting machine


Brain plasticity

A perfect pitch pill? http://www.theverge.com/2014/1/6/5279182/valproate-may-give-humans-perfect-pitch-by-resetting-critical-periods-in-brain


Could we grow music producing organisms? http://reactivemusic.net/?p=18018


Two possibilities

Rejecting technology?
An optimistic future?

There is a quickening of discovery: internet collaboration, open source, linux,  github, r-pi, Pd, SDR.

“Robots and AI will help us create more jobs for humans — if we want them. And one of those jobs for us will be to keep inventing new jobs for the AIs and robots to take from us. We think of a new job we want, we do it for a while, then we teach robots how to do it. Then we make up something else.”

“…We invented machines to take x-rays, then we invented x-ray diagnostic technicians which farmers 200 years ago would have not believed could be a job, and now we are giving those jobs to robot AIs.”

Kevin Kelly – January 7, 2015, reddit AMA http://www.reddit.com/r/Futurology/comments/2rohmk/i_am_kevin_kelly_radical_technooptimist_digital/

Will people be marrying robots in 2050? http://www.livescience.com/1951-forecast-sex-marriage-robots-2050.html

“What can you predict about the future of music” by Michael Gonchar at The New York Times http://reactivemusic.net/?p=17023

Jim Morrison predicts the future of music:


More areas to explore

Processing shortwave radio sounds

Using the python sms-tools library.

sms-tools: https://github.com/MTG/sms-tools

Here is a song made from the processed sounds:

mp3 version:

This project was an assignment for the Coursera “Audio Signal Processing for Music Applications” course. https://www.coursera.org/course/audio

Source material

Sounds were recorded from a shortwave radio between 5-10MHz.

freesound.org links to the sounds:




The sound is an AM shortwave broadcast station from between 7-8 MHz. It is speech with atmospheric noise and a digitally modulated carrier at 440Hz in the background.

I tried various approaches to removing the speech and isolating the carrier. But ended up using the following parameters to remove noise and speech, but for most part leaving a 440hz digital mode signal with large gaps in it.

  • M=701
  • N=1024
  • minf0=400
  • maxf0=500
  • thresh=-90
  • max harmonics=50

After more experimentation, the following changes resulted in a cool continuous tone with speechlike quality (but not intelligible) and the background noise is gone.

Here is the full list of parameters:


Here is a plot:

Screen Shot 2014-12-16 at 8.03.55 PM

Here is the resulting sound of the sinusoidal part of the harmonic model:


The sound is continuous digital modulation (buzzing) from a shortwave radio between 7-8 MHz. The buzz is around 100Hz with atmospheric background noise.

Transformation using HPS (harmonic plus stochastic) model.

Not very impressive analysis, but the resynthesis had a very cool looking spectrogram due to some frequency shifting.


Screen Shot 2014-12-16 at 8.12.43 PM

I realized that I had set f0min too high. Went back to using the HPR model without transformation to see if I could separate the tone. Here is the plot:


Screen Shot 2014-12-16 at 8.33.50 PM

Here are the resulting sounds transformation (unused) and the sinusoidal/residual results that were used in the track.

source: digital_pulse_7hz.wav

A repeating pulse around from a shortwave radio between 7-8 MHz. The frequency of the pulse is around 1000Hz with a noise component.

Another noise filter – this was way more difficult due to high freq material.


Screen Shot 2014-12-16 at 8.40.12 PM

Instead, I went with a downward pitch transform, using the HPS model transform. Here are the resulting sounds from  the HPR filter (unused) and the HPS transform.


The sound contains typical amateur radio CW signals from the 40 Meter band, with several interfering signals (QRM) and atmospheric noise (QRN). Using the HPR model, I was able to completely isolate and re-synthesize the CW signal, removing all the noise and interfering signals.


Note that you can actually see the morse code letters “T, U, and W” on the spectrogram of model!

Screen Shot 2014-12-16 at 8.55.20 PM

Here is the re-synthesized CW sound:


The WWV National Bureau of Standards “clock” station at 5MHz. A combination of pulses, tones, speech, and background noise.

I was trying to separate the voice from the rest of the tones and noise. After several hours and various approaches, I gave up. The signal may be too complex to separate using these models. There were some interesting plots with the HPR model

Screen Shot 2014-12-16 at 9.11.54 PM

Finally decided to just isolate the 440 Hz. clock pulse from the rest of the signal:


Screen Shot 2014-12-16 at 9.06.09 PM

Here is the resulting sound (note that the tone starts several seconds into the sample)

ep-413 DSP week 15


  1. Syllabus: http://reactivemusic.net/?p=17122
  2. Ways to approach a project http://reactivemusic.net/?p=17132
    • Make machines that make art
    • Reverse engineering
    • Use the wrong tools
    • Abstraction and destruction
    • Backwards, extreme, opposite – connect two things
    • Ask questions
  3. Composition tools and dramatic shape http://reactivemusic.net/?p=17157
  4. Problem solving (pitch detection) and prototyping (Muse) http://reactivemusic.net/?p=17159
  5. Sound byte composition http://reactivemusic.net/?p=17190
  6. Convolution and voices http://reactivemusic.net/?p=17211
  7. (No class this week)
  8. Granular synthesis, the frequency domain, and phasors http://reactivemusic.net/?p=17360
  9. Data, Internet API’s, Vine API in Max http://reactivemusic.net/?p=17466
  10. Communication, Osc, Sonification, MBTA API in Max http://reactivemusic.net/?p=17518
  11. Filters: analog, digital, other, reversability http://reactivemusic.net/?p=17542
  12. Web Audio API http://reactivemusic.net/?p=17600
  13. Feature detection, and Music Information Retrieval http://reactivemusic.net/?p=17689
  14. Waves: light, radio, water http://reactivemusic.net/?p=17787
  15. This

John Coltrane: You can learn something from everybody, no matter how good or bad they play, everybody has something to say.

Sal Khan: In the future people will take agency for their own education.

For artists, everything is a tool.