Stock market music in Max

Make music from the motion of stock prices.

This program gathers stock prices into a database. It generates Midi data – mapping price to pitch, and mapping trading volume to velocity and rhythmic density. It uses ancient Web technology: HTML/javascript front-end with a php back-end accessing a mysql database.

Case study: http://zproject.wikispaces.com/stock+market+music

To run this project, you will need a server (preferably linux) with the following capabilities:

  • mysql + phpmyadmin
  • php (and ability to run php over the web)
  • netcat (nc)
  • network access

All of this is pretty standard – so I won’t talk about it here. I am running it on Ubuntu Linux. There are many other ways to get the project working, by using the layout described here.

download

https://github.com/tkzic/internet-sensors

folder: stock-market

files

Max

stock_market_music.maxpat

HTML/javascript web client
  • newstock3.html: (web page interface)
  • selectstock3.js:  (front-end)
php server
  • getstock3.php (back-end server to get quotes and save them to a database)
  • play3.php: (back-end server to retrieve quotes, analyze, and map to Midi sequence to send to Max)
  • udp.php: Osc library
Set execute privileges on php files so they can be run from your web server. (chmod +x)

 

database

The selectstock3.php program harvests stock quote data and stores it in a mysql database.

The database name is:  stocks – table is: quotes

Table structure:

The table is basic flat representation of a stock quote, indexed by the ticker symbol. It contains price, volume, high/low/change, timestamp, etc., For our purposes, the price, volume and timestamp are essentially all we need.

SQL to create the table:

 CREATE  TABLE  `stocks`.`quotes` (  `ticker` varchar( 12  )  NOT  NULL ,
 `price` decimal( 10, 2  )  NOT  NULL ,
 `qtime` datetime NOT  NULL ,
 `pchange` decimal( 10, 2  )  NOT  NULL ,
 `popen` decimal( 10, 2  )  NOT  NULL ,
 `phigh` decimal( 10, 2  )  NOT  NULL ,
 `plow` decimal( 10, 2  )  NOT  NULL ,
 `volume` int( 11  )  NOT  NULL ,
 `ttime` timestamp NOT  NULL  DEFAULT CURRENT_TIMESTAMP ,
 `id` int( 11  )  NOT  NULL  AUTO_INCREMENT ,
 `spare` varchar( 30  )  DEFAULT NULL ,
 UNIQUE  KEY  `id` (  `id`  ) ,
 KEY  `ticker` (  `ticker`  )  ) ENGINE  =  MyISAM  DEFAULT CHARSET  = latin1 COMMENT  =  'stock quote transactions';

creating the database, user, and table

  • Log into  phpmyadmin as root
  • Create a new database called ‘stocks’
  • In privileges, add a user called: ‘webdb1′ with a password of ’34door’ (note you can change the password later)
  • In SQL, copy in the above query to create the ‘stocks’ table

Instructions

Web client

The webpage control program allows you to select stocks by ticker symbol, and get either one quote or get quotes at regular time interval. Each quote is inserted into the stock table for later retrieval and analysis.

 

The web front end is quirky so I will describe it in terms of how you might typically use it:

market is open – and you just want to play music based on current stock prices
  1. Enter the ticker symbols for your stocks
  2. Press ‘tracking’ button – so the quotes get saved
  3. Enter the IP address of the computer running Max
  4. Press the ‘auto’ button in the upper left corner – it will run and play forever
market is closed – or you want to play historical data you have saved
  1. Enter the ticker symbols for your stocks
  2. Enter the IP address of the computer running Max
  3. set start end end dates
  4. Press the ‘play’ button to play once or press ‘loop’ to play continuously (using time interval in seconds)
  5. market is open – you just want to collect stock quote data
  6. Enter the ticker symbols for your stocks
  7. Press the tracking button so quotes will get saved
  8. Press the ‘get quotes’ button to get current quote or press ‘loop’ button (on the same line) to retrieve quotes  continuously every 30 seconds.

Instructions

Max patch

  1. Make sure the IP address is set to the address of your server
  2. Select the Midi port for output
  3. Play a few test notes
  4. Select either ‘one instrument’ mode (piano) or multi instrument mode. Each time you click the multi instrument button it randomly selects a new combination 

notes on stock market data

To look at historical trends, you would need access to historical stock data. To use it as a tool for short term analysis, you would need access to real-time quote data in an API. At the time, both of these cost money.

However, it doesn’t cost money to get recent quotes from Yahoo throughout the day and store them in a database – so that’s the approach I took.

If I were to do this project today, I’d look for a free online source of historical data, in machine-readable form – because the historical data provides the most interesting and organic sounds when converted into music. The instant high speeding trading data would probably make interesting sounds as well, but you still need to pay for the data.

notes on local files

Installing Ubuntu Linux 14.04 on MacBook Pro – dual boot

updated 3/17/2017 – for Sierra 10.12.3

  1. download latest version of refind (0.10.5) https://sourceforge.net/projects/refind/?source=typ_redirect
  2. unzip and copy folder under home directory
  3. restart computer and hold down cmd-R
  4. go to utilities and start a terminal session
  5. cd to the refind directory created above. (look under Volumes/Macintosh HD/Users/…
  6. run ./refind-install
  7. reboot

updated 6/18/2015 – For Yosemite

For upgrade to Yosemite, just get the newest version of refined (0.8.7) and rerun the install script. It will find the existing linux partition.

One annoying thing is that on restart it displays 3 separate versions of the same boot loader.

update 8/13/2014

Installed on new Macbook Pro retina and needed to use an earlier version of refind (0.7.8) than the one currently available. When running the newest version – it couldn’t see the USB stick with ubuntu. Also did not need to install the wifi drivers.

Another note: here are instructions for NVIDIA drivers. Though I haven’t tried them yet:

http://cberner.com/2014/04/20/installing-ubuntu-14-04-on-macbook-pro-retina/

The whole user experience on the Retina Macbook is so far not as good as on the older 2011 version. Problems with trackpad, mouse, and display.

update 4/25/2014 – how to repair dual boot after MacOS automatic update to 10.9.2

After an automatic update to 10.9.2 – the EFI boot menu disappeared. Solution is to just re-install rEFInd using the ./install.sh script as shown here, as in the original installation:

The installation needs to be done in the Terminal, by running the install.sh script:

cd ~/refind-bin-0.7.8
./install.sh --alldrivers
Then reboot...

Original post

This took several attempts to get right. So here are notes.

First – Make sure you have a Time Machine backup. You will probably need it.

The thing that worked was a combination of these two posts.

For the most part, use these instructions:

http://randomtutor.blogspot.com/2014/01/installing-ubuntu-on-retina-macbook-pro_19.html

One thing that was weird is that you reboot with the USB installer plugged in you get several boot choices with no explanation. You can choose the first one, but make sure to select “Try Ubuntu” and install according to above instructions…

note: I did not need to do step 5 – editing the refind config file.

But you may also need to get the wiFi drivers as explained here – just don’t follow the install instructions or you will trash your bootloader:

http://www.alexvictorchan.com/2013/05/01/installing-ubuntu-13-04-on-13-macbook-pro-retina-102/

Then you will probably want to do the first few of these suggestions to get Media playback:

http://itsfoss.com/things-to-do-after-installing-ubuntu-14-04/

For Pd-extended install – just use instructions from the Pd site

http://puredata.info/docs/faq/debian

(I ran the terminal commands with the ssh keys to get it, because It wasn’t in the Ubuntu software center)

 

 

Ubuntu in virtualbox

Ubuntu server guest, on Mac OS host.

instructions

http://www.lecloud.net/post/52224625343/the-ultimate-setup-guide-ubuntu-13-04-in-virtualbox

Skipped the SSH for now, also the shared folders are in /media/sf[folder-name] on the guest – will be trying again shortly.

In the meantime I installed a desktop version of Linux and the install was much easier – copy and paste works fine – and we were able to get sounds. now installing pd.

linux instructions for Pd: http://puredata.info/docs/faq/debian

After running into huge problems with copy/paste – and no audio, I was able to install 32 bit version of Ubuntu Studio. Audio works fine, but Jack doesn’t run – after a few hours, I’m abandoning this approach and will probably try running linux natively on something.

converting IQ audio files using sox

To get information about a file:

# sox --i 10meter96.wav

Input File : '10meter96.wav'
Channels : 2
Sample Rate : 96000
Precision : 16-bit
Duration : 00:00:16.50 = 1584387 samples ~ 1237.8 CDDA sectors
File Size : 6.34M
Bit Rate : 3.07M
Sample Encoding: 16-bit Signed Integer PCM

#

To convert the sample rate:

# sox 10meter96.wav -r 44100 10meter44.wav

More useful hints about sox by Selvaganeshan at “The Geek Stuff”

http://www.thegeekstuff.com/2009/05/sound-exchange-sox-15-examples-to-manipulate-audio-files/

Here are the commands that worked to get the raw IQ data from rtl_sdr into Max

rtl_sdr -f 94900000 -s 1024000 -g 50 iq.raw

To convert the above to 96k 16bit wav format

sox -e unsigned-integer -r 1024k -t raw -b 8 -c 2 iq.raw -r 96k -b 16 iq.wav

Note: I could not get the above conversion to work with device sampling rates below 1024k. Didn’t try anything higher.

 

 

writing delays in C (linux)

An alternative way to write a delay.

You can also use select(2) if you want microsecond precision (this works on platform that don’t have usleep(3))

The following code will wait for 1.5 second:

#include <sys/select.h>
#include <sys/time.h>
#include <unistd.h>`

int main() {
    struct timeval t;
    t.tv_sec = 1;
    t.tv_usec = 500000;
    select(0, NULL, NULL, NULL, &t);
}

how to open, read, and write from a serial port in C (linux)

With applications for Raspberry Pi

notes

by hobbytronics.com

http://www.hobbytronics.co.uk/raspberry-pi-serial-port

C – programming example:

http://stackoverflow.com/questions/6947413/how-to-open-read-and-write-from-serial-port-in-c

An R-Pi thread about connecting Arduino to R-Pi. R-Pi does TTL (3.3 v level and Arduino is rs-232 5v) so you need a level converter for them to communicate.

http://www.raspberrypi.org/phpBB3/viewtopic.php?f=31&t=23873

Another similar example, connecting Arduino and R-pi via USB

http://www.raspberrypi.org/phpBB3/viewtopic.php?f=32&t=28801

The WiringPi library for serial io:

https://projects.drogon.net/raspberry-pi/wiringpi/serial-library/