Tag: php

Stock market music in Max

Make music from the motion of stock prices.

This program gathers stock prices into a database. It generates Midi data – mapping price to pitch, and mapping trading volume to velocity and rhythmic density. It uses ancient Web technology: HTML/javascript front-end with a php back-end accessing a mysql database.

Case study: http://zproject.wikispaces.com/stock+market+music

To run this project, you will need a server (preferably linux) with the following capabilities:

  • mysql + phpmyadmin
  • php (and ability to run php over the web)
  • netcat (nc)
  • network access

All of this is pretty standard – so I won’t talk about it here. I am running it on Ubuntu Linux. There are many other ways to get the project working, by using the layout described here.

download

https://github.com/tkzic/internet-sensors

folder: stock-market

files

Max

stock_market_music.maxpat

HTML/javascript web client
  • newstock3.html: (web page interface)
  • selectstock3.js:  (front-end)
php server
  • getstock3.php (back-end server to get quotes and save them to a database)
  • play3.php: (back-end server to retrieve quotes, analyze, and map to Midi sequence to send to Max)
  • udp.php: Osc library
Set execute privileges on php files so they can be run from your web server. (chmod +x)

 

database

The selectstock3.php program harvests stock quote data and stores it in a mysql database.

The database name is:  stocks – table is: quotes

Table structure:

The table is basic flat representation of a stock quote, indexed by the ticker symbol. It contains price, volume, high/low/change, timestamp, etc., For our purposes, the price, volume and timestamp are essentially all we need.

SQL to create the table:

 CREATE  TABLE  `stocks`.`quotes` (  `ticker` varchar( 12  )  NOT  NULL ,
 `price` decimal( 10, 2  )  NOT  NULL ,
 `qtime` datetime NOT  NULL ,
 `pchange` decimal( 10, 2  )  NOT  NULL ,
 `popen` decimal( 10, 2  )  NOT  NULL ,
 `phigh` decimal( 10, 2  )  NOT  NULL ,
 `plow` decimal( 10, 2  )  NOT  NULL ,
 `volume` int( 11  )  NOT  NULL ,
 `ttime` timestamp NOT  NULL  DEFAULT CURRENT_TIMESTAMP ,
 `id` int( 11  )  NOT  NULL  AUTO_INCREMENT ,
 `spare` varchar( 30  )  DEFAULT NULL ,
 UNIQUE  KEY  `id` (  `id`  ) ,
 KEY  `ticker` (  `ticker`  )  ) ENGINE  =  MyISAM  DEFAULT CHARSET  = latin1 COMMENT  =  'stock quote transactions';

creating the database, user, and table

  • Log into  phpmyadmin as root
  • Create a new database called ‘stocks’
  • In privileges, add a user called: ‘webdb1′ with a password of ’34door’ (note you can change the password later)
  • In SQL, copy in the above query to create the ‘stocks’ table

Instructions

Web client

The webpage control program allows you to select stocks by ticker symbol, and get either one quote or get quotes at regular time interval. Each quote is inserted into the stock table for later retrieval and analysis.

 

The web front end is quirky so I will describe it in terms of how you might typically use it:

market is open – and you just want to play music based on current stock prices
  1. Enter the ticker symbols for your stocks
  2. Press ‘tracking’ button – so the quotes get saved
  3. Enter the IP address of the computer running Max
  4. Press the ‘auto’ button in the upper left corner – it will run and play forever
market is closed – or you want to play historical data you have saved
  1. Enter the ticker symbols for your stocks
  2. Enter the IP address of the computer running Max
  3. set start end end dates
  4. Press the ‘play’ button to play once or press ‘loop’ to play continuously (using time interval in seconds)
  5. market is open – you just want to collect stock quote data
  6. Enter the ticker symbols for your stocks
  7. Press the tracking button so quotes will get saved
  8. Press the ‘get quotes’ button to get current quote or press ‘loop’ button (on the same line) to retrieve quotes  continuously every 30 seconds.

Instructions

Max patch

  1. Make sure the IP address is set to the address of your server
  2. Select the Midi port for output
  3. Play a few test notes
  4. Select either ‘one instrument’ mode (piano) or multi instrument mode. Each time you click the multi instrument button it randomly selects a new combination 

notes on stock market data

To look at historical trends, you would need access to historical stock data. To use it as a tool for short term analysis, you would need access to real-time quote data in an API. At the time, both of these cost money.

However, it doesn’t cost money to get recent quotes from Yahoo throughout the day and store them in a database – so that’s the approach I took.

If I were to do this project today, I’d look for a free online source of historical data, in machine-readable form – because the historical data provides the most interesting and organic sounds when converted into music. The instant high speeding trading data would probably make interesting sounds as well, but you still need to pay for the data.

notes on local files

Internet sensors projects

overview

A series of projects that use Internet API’s for interactive media projects.

Projects have been tested on Mac OS 10.9 (Mavericks). All but one of the projects use Max/MSP 6.1.7 from: http://cycling74.com. The other project uses Pure Data 0.42.5 (extended) from: http://puredata.info. Other required programs are listed in the documentation for individual projects.

download

internet-sensors is on Github at:  https://github.com/tkzic/internet-sensors

Each project is in a separate folder.

authorization

Some projects require passwords and API-keys from providers.

For example, for the ‘Twitter streaming API in Max’ project you’ll need to set up a Twitter application from your account to get authorization credentials.

For projects that need authorization usually you’ll just need to modify the patches/source code with your user information – as directed in the instructions.

help

The API’s used in the projects change fairly often. So there’s no guarantee they’ll work. If you find problems or have ideas – please post to them to the github repository. Or email me at tkzic@megalink.net.

projects

1. Twitter streaming API in Max (FM, php, curl, geocoding, [aka.speech], Soundflower (optional), Morse code, OSC, data recorder, Twitter v1.1 API, Twitter Apps, Oauth)

http://reactivemusic.net/?p=5786

2. Sending tweets from Max using curl ([sprintf], [aka.shell], xively.com API, zapier.com API, JSON, javascript Twitter v1.1 API, Oauth)

http://reactivemusic.net/?p=5447

 

3. Send and receive tweets in Max using ruby (ruby, API, JSON, javascript Twitter v1.1 API, OSC, Oauth)

http://reactivemusic.net/?p=7013

New! – use the project above to send tweets from using a Fisher Price “Little Tikes” piano: http://reactivemusic.net/?p=6993

4. Speech to text in Max (Google speech API, JSON, javascript, sox, Twitter v1.1 API, Oauth)

Note: Send Tweets using speech as well.

http://reactivemusic.net/?p=4690

5. A conversation with a robot in Max (Google speech API, sox, JSON,  pandorabots API, python, [aka.speech]

http://reactivemusic.net/?p=9834

7. Playing bird calls in Max (xeno-canto API, [jit.uldl], [jit.qt.movie])

http://reactivemusic.net/?p=4225

8. Soundcloud API in Max (JSON, javascript, curl, [aka.shell], [jit.qt.movie])

http://reactivemusic.net/?p=5413

9. Real time train map using Max and node.js (XML, JSON, OSC, data recorder, web sockets, Irish Rail API)

http://reactivemusic.net/?p=5477

10. stock market music in Max (OSC, netcat,  php, mysql, html, javascript, Yahoo API, linux)

http://reactivemusic.net/?p=12029

11. Using weather forecast data to drive weather sounds in Pure Data (ruby, OSC, JSON, openweathermap API, “Designing Sound” by Andy Farnell)

http://reactivemusic.net/?p=5846

12. Using ping times to control oscilators in Max (Mashape ping-uin API, ruby, OSC, JSON)

http://reactivemusic.net/?p=5945

13. Echonest Segment analysis player – sonification of audio analysis data from The Echo Nest (echonest API, ruby, Osc, Max/MSP)

http://reactivemusic.net/?p=6296

14. Quadcopter AR_drone – Fly a quadcopter using Max – with streaming Web video. ( node.js, AR_drone, Google Chrome, Osc, Max/MSP)

http://reactivemusic.net/?p=6635

15. Adding markers to Google Maps in Max – ( node.js, ruby, Google Chrome, Osc, Max/MSP, websockets, Google Maps API, Jquery, javascript)

http://reactivemusic.net/?p=11412

16. Max data recorder –  Record and play back streams of data simultaneously at various rates

http://reactivemusic.net/?p=8053

 

17. Web Audio version of Google domain ping machine –  The sound of an even newer machine using Web Audio API, web sockets. Max, Ruby, Osc, Google API, and Mashape

Note: This project is not in internet-sensors. You will find it on github in the tkzic/WebAudio project: https://github.com/tkzic/WebAudio

Instructions: http://reactivemusic.net/?p=6243

 

16. MBTA bus data in Max –  Sonification of Mass Ave buses, from Harvard to Dudley

http://reactivemusic.net/?p=17524

Screen Shot 2014-11-11 at 3.26.16 PM

 

Twitter streaming API in Max

World map and radio simulation

features

  • Twitter streaming v1.1 API and Twitter Apps (using http requests and Oauth in php)
  • lat/lon conversion and map plotting in Max
  • sending data to Max using OSC in php
  • ‘speaking’ tweets using several voices (text to speech)
  • Using geo-coordinates to control an FM synthesizer
  • Converting Tweet text to Morse code
  • Using a data recorder to replay/save data streams (Max lists)

Compare to satellite photo of earth – note the pattern of lights.

download

https://github.com/tkzic/internet-sensors

folder: twitter-stream

files

Max
  • world3.maxpat (main patch)
  • data_recorder_list-tz.maxpat (abstraction for recording data)
  • data-recorder-wrapper.maxpat (abstraction for recording data)
  • worldMap.jpg
  • twitter-morning.txt (sample data – not required)
php
  • ctwitter_max3.php (main program)
  • ctwitter_stream_max3.php (twitter engine)
  • udp.php (Osc client)

externals

[note]  The project displays Tweets without these externals, but you won’t hear any speech

authorization

In addition to having a Twitter account, you will need to set up a Twitter application from the developer site here:

https://dev.twitter.com/apps

Good instructions on how to do this can be found in this stackoverflow.com post under this heading: So you want to use the Twitter v1.1 API?

http://stackoverflow.com/questions/12916539/simplest-php-example-for-retrieving-user-timeline-with-twitter-api-version-1-1

When you get to step 5 – in the instructions – instead of writing your own code, just use a text editor to copy your access tokens into this php program which is provided:

  • ctwitter_max3.php

Replace the strings in this line of code by copying and pasting the appropriate ones from your Twitter application:

$t->login('consumer_key', 'consumer secret', 'access token', 'access secret');

 

So it will end up looking something like this:

$t->login('ZdzfNaeflihFydfOHeOA', 'eXzUOfhif4riifgRbCTnnSN0T7neYtg8dIWDC7j3bs', '205589709-5kRI1fllJvU94jjffeerSn9LrTajtxSrvO8', 'u5MuSxPseBemUIBWlMxEFaw899feedXA0eHlReCnQ');

Yeah – its cryptic…

instructions

1. open the Max Patch: world3.maxpat

2. in a terminal window run the php program: ctwitter_max3.php. [note] it runs forever. Press <ctrl-c> when you want to stop streaming Tweets.

php ./ctwitter_max3.php

3. Switch back to world3.maxpat to see dots populating the map

4. In Max, press the speaker icon (lower left) to turn on audio.

5. Activate  voice synth/morse code using the blue toggle (lower left)

6. Clear the map by pressing the blue message box: “clear, drawpict a 0 0”

7. Stop the Tweet stream by pressing <ctrl-c> in the terminal window

special voice fx

If you have Soundflower installed, the Mac OS speech synth output can be routed back to Max for audio processing. This is somewhat complicated, but shows how to process audio in Max from other sources.

  • In MacOS System Preferences, set audio output device to Soundflower 2ch
  • Turn up hardware volume control on your computer
  • In Max, Options | Audio Status, set input device to Soundflower 2ch
  • In world3.maxpat double click on [p audio engine] (lower left). Then in the audio-engine sub-patch activate the toggle, (lower right) for voice-fx

data recording

The built-in data recorder/playback is on the left side of world3.maxpat:

  • toggle ‘record’ (red toggle)  to start or stop data recording
  • Note that data will only be recorded when the php program is streaming Tweets in the terminal window (see above)
  • Press /play message or other transport controls to replay data
<span style="font-family: 'Helvetica Neue', Helvetica, Helvetica, Arial, sans-serif; font-size: 23px; font-weight: bold; line-height: 1.1;">
</span>
<span style="font-family: 'Helvetica Neue', Helvetica, Helvetica, Arial, sans-serif; font-size: 23px; font-weight: bold; line-height: 1.1;">revision history</span>
<span style="font-family: 'Helvetica Neue', Helvetica, Helvetica, Arial, sans-serif; font-size: 23px; font-weight: bold; line-height: 1.1;">
</span>
  • updated 3/26/2014 – fixed runtime error in php server
  • updated 2/2/2014 – simplified user interface and updated audio engine
  • updated 9/2/2013 for Twitter v1.1 API with Oauth – note that older versions of this project are broken due to discontinued Twitter v1.0 API as of June 2013

Parsing Tweets

A method using regexp and php. Actually what this does is parse Tweets using regexp to reformat the text as html with links. A tutorial here:

http://saturnboy.com/2010/02/parsing-twitter-with-regexp/

This is a php library that breaks out hashtags, usernames, etc., but doesn’t really provide a way to isolate the remaining stuff. I have put it in tkzic/API –  there is an example php program provided.

https://github.com/mzsanford/twitter-text-php

hashtags – using regular expressions

http://stackoverflow.com/questions/11551065/parsing-tweets-to-extract-hashtags-in-r

twitter-text-rb : ruby gem which parses out usernames and hashtags

https://github.com/twitter/twitter-text-rb

 

Twitter streaming from php to Max

update 6/2014 – This project is part of the Internet sensors projects: http://reactivemusic.net/?p=5859. Check the link for current versions.

original post

notes

Got a test patch running today which breaks out tweets (in php and curl) and sends them to Max via Osc.

(update) Have parsed data to remove  hyperlinks and Twitter symbols.

It took some tweaking of global variables in php – and probably would be better written using classes (as in this example: http://stackoverflow.com/questions/1397234/php-curl-read-incrementally – see post from GZipp.

Max patch: tkzic/max teaching examples/twitter-php-streamer1.maxpat

php code: twitterStreamMax.php

<?php

// max-osc-play.php
//
//	collection of php OSC code from Max stock-market thing
//

include 'udp.php';		// udp data sending stuff

$DESTINATION = 'localhost';
$SENDPORT = '7400';
$RECVPORT = '7401';

//////////////////////////////////////////////////////////////////////////////////////////

	$USERNAME = 'username';
	$PASSWORD = 'password';
	$QUERY    = 'cats';		// the hashtag # is optional

	// these variables are defined as global so they can be used inside the write callback function
	global $osc;
	global $kount;

	// initialize OSC
	$osc = new OSCClient();  // OSC object
	$osc->set_destination($DESTINATION, $SENDPORT);

	// This amazing program uses curl to access the Twitter streaming API and breaks the data
	// into individual tweets which can be saved in a database, sent out via OSC, or whatever
	//

	/**
	 * Called every time a chunk of data is read, this will be a json encoded message
	 * 
	 * @param resource $handle The curl handle
	 * @param string   $data   The data chunk (json message)
	 */
	function writeCallback($handle, $data)
	{
	    /*
	    echo "-----------------------------------------------------------\n";
	    echo $data;
	    echo "-----------------------------------------------------------\n";
	    */

		$maxdata = "/tweet" ;				// header - begin   
		global $kount;					// test counter
		global $osc;						// osc object

	    $json = json_decode($data);
	    if (isset($json->user) && isset($json->text)) {

			// here we have a single tweet
	        echo "@{$json->user->screen_name}: {$json->text}\n\n";

			// do some cleaning up...
			// remove URL's
			$s = $json->text;		// raw tweet text

			// ok now need to do the same thing below for URL,s RT's @'s etc., 
			// and then remove redundant spaces	
			/* example
			Depending on how greedy you'd like to be, you could do something like:

			$pg_url = preg_replace("/[^a-zA-Z 0-9]+/", " ", $pg_url);

			This will replace anything that isn't a letter, number or space

			*/		

			// display all hashtags and their indices
			foreach( $json->entities->hashtags as $obj )
			{
			  echo "#:{$obj->text}\n";		// display hashtag
			  // get rid of the hashtag
			 	// note: this gets rid of all hashtags, which could obscure the meaning of the tweet, if
				// the hashtag is used inside a sentence like: "my #cat is purple" - would be changed to: "my is purple"
				// so we could use some intelligent parsing here...

			//  $s = str_replace("#{$obj->text}", "", $s );

			// this is a more benign approach, which leaves the word but removes the #

			$s = str_replace("#{$obj->text}", "{$obj->text}", $s );

			}

			foreach( $json->entities->urls as $obj )
			{
			  echo "U:{$obj->url}\n";		// display url			
			  $s = str_replace("{$obj->url}", "", $s );   // get rid of the url		
			}

			foreach( $json->entities->user_mentions as $obj )
			{
				echo "@:{$obj->screen_name}\n";		// display 			
				$s = str_replace("RT @{$obj->screen_name}:", "", $s );   // get rid of re-tweets
				$s = str_replace("@{$obj->screen_name}:", "", $s );   // get rid of other user mentions
				$s = str_replace("@{$obj->screen_name}", "", $s );   // get rid of other user mentions		
			}

			// $s = str_replace("RT ", "", $s );   // get rid of RT's (re-tweet indicators)

			// $s = preg_replace( '/[^[:print:]]/', '',$s); // remove non printable characters

			$s = htmlspecialchars_decode($s);		// decode stuff like &gt;

			$s = preg_replace('/[\x00-\x08\x0B\x0C\x0E-\x1F\x80-\x9F]/u', '', $s); // get rid of unicode junk

			$s = preg_replace('/[^(\x20-\x7F)]*/','', $s);		// get rid of other non printable stuff

			$s = preg_replace('!\s+!', ' ', $s);	// remove redundant white space

			echo "revised tweet: {$s}\n";

			$maxdata = "/tweet " . "{$json->text}";
			// $maxdata = $maxdata . " " . $kount++;
		   	$osc->send(new OSCMessage($maxdata));

	    }

	    return strlen($data);
	}

// initialize OSC 

// initialize curl

	$ch = curl_init();

	curl_setopt($ch, CURLOPT_URL, 'https://stream.twitter.com/1/statuses/filter.json?track=' . urlencode($QUERY));
	curl_setopt($ch, CURLOPT_USERPWD, "$USERNAME:$PASSWORD");
	curl_setopt($ch, CURLOPT_WRITEFUNCTION, 'writeCallback');
	curl_setopt($ch, CURLOPT_TIMEOUT, 20); // disconnect after 20 seconds for testing
	curl_setopt($ch, CURLOPT_VERBOSE, 1);  // debugging
	curl_setopt($ch, CURLOPT_ENCODING,  'gzip, deflate'); // req'd to get gzip
	curl_setopt($ch, CURLOPT_USERAGENT, 'tstreamer/1.0'); // req'd to get gzip

	curl_exec($ch); // commence streaming

	$info = curl_getinfo($ch);

	var_dump($info);

?>

Osc in php

The Osc code from the stock market music project http://reactivemusic.net/?p=12029 is not really doing Osc.

But… it works well going from php->max. In the other direction its using a kluge of nc and an alarm clock shell program – to receive messages from Max in UDP, but its really kind of horrible – so I’m going to look again for an OSC library in php.

update 2/2013 This is hard to believe, but I haven’t yet found a real OSC libraries for php. Apparently php is so uncool, that nobody wants to write for it anymore. Anyway, the code above, works unidirectionally, so its of some use for existing php code.

Local files are max-php-osc-tester.maxpat and max-osc-play.php in tkzic/api

Analysis that might help with parsing:

from Captain Caveman

http://forum.cockos.com/showthread.php?t=99076

 

Twitter streaming php decoder breaks out individual tweets

This code was adapted (i.e. stolen verbatim) from a stackoverflow post by drew010

http://stackoverflow.com/questions/10337984/using-the-curl-output

Here’s the code. It solves a huge problem for the class of projects which need to grab a large amount of tweets in real time to either save in a database, or trigger some action.

My version of the code is in tkzic/api/twitterStream1.php

<?php

$USERNAME = 'youruser';
$PASSWORD = 'yourpass';
$QUERY    = 'nike';

/**
 * Called every time a chunk of data is read, this will be a json encoded message
 * 
 * @param resource $handle The curl handle
 * @param string   $data   The data chunk (json message)
 */
function writeCallback($handle, $data)
{
    /*
    echo "-----------------------------------------------------------\n";
    echo $data;
    echo "-----------------------------------------------------------\n";
    */

    $json = json_decode($data);
    if (isset($json->user) && isset($json->text)) {
        echo "@{$json->user->screen_name}: {$json->text}\n\n";
    }

    return strlen($data);
}

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, 'https://stream.twitter.com/1/statuses/filter.json?track=' . urlencode($QUERY));
curl_setopt($ch, CURLOPT_USERPWD, "$USERNAME:$PASSWORD");
curl_setopt($ch, CURLOPT_WRITEFUNCTION, 'writeCallback');
curl_setopt($ch, CURLOPT_TIMEOUT, 20); // disconnect after 20 seconds for testing
curl_setopt($ch, CURLOPT_VERBOSE, 1);  // debugging
curl_setopt($ch, CURLOPT_ENCODING,  'gzip, deflate'); // req'd to get gzip
curl_setopt($ch, CURLOPT_USERAGENT, 'tstreamer/1.0'); // req'd to get gzip

curl_exec($ch); // commence streaming

$info = curl_getinfo($ch);

var_dump($info);