Audio from Video

Using video to generate audio.

jit.peek~ outputs an audio signal from video matrix data.

(jit.peek~ help)

from cycling74.com

At any point in the the process, matrix data can be tapped to produce an audio signal. Analyzing video is very much like analyzing audio. For example, using envelope following and spectral analysis. Here are examples.

examples

1. jit.peek~-additivesynth.maxpat (in Max/MSP examples/jitter-examples/audio)

Draw a song

By Adam Florin and Joshua Kit Clayton

2. Frame subtraction

By Adam Rokshar

https://reactivemusic.net/?p=7005

3. Sound Emotion2 – using Macbook built-in camera

By Andreas Wittich

https://reactivemusic.net/?p=9225

4. jit.peek~-osctrack.maxpat (in Max/MSP examples/jitter-examples/audio)

Derives rhythmic audio data from video. (Using bball.mov)

By cycling74.com

5. Whispering Heights (in Max/MSP examples/jitter-examples/3rd-party/image-to-spectral-filter/whispering_heights.maxpat)

Similar to above but uses video to create a moving spectral filter.

6. Wolframatic.maxpat  (in Max/MSP examples/jitter-examples/other/Wolframatic/)

Fractal generator also generates audio.

By R. Luke Dubois

 

maxradio updates

Upgrade to Mac OS Max version.

Windows, runtime, standalone, and documentation upgrades still in progress…

changes

This is an interim update to add new drivers.

  • driver for rtl-sdr
  • minor bug fixes

download

https://github.com/tkzic/maxradio

instructions

  • Connect a radio
  • Open maxsdr7a.maxpat 

note: 1/23/2015 – the repository was really messed up – missing externals, etc., have reloaded. The local source is now at tkzic/maxradio

Low resolution LED interface

Lots of information in a few pixels.

Computer displays have evolved to high resolutions. What about the other direction? This experiment  is a display interface using a grid of LED’s. Essentially, very large pixels.

information types
  • on/off 
  • small numbers (0-10)
  • large numbers (0-10,000,000)
  • clocks
  • level indicators
  • connections
  • map keys (i.e., explanations of symbols) 
communicating with LED’s

LED’s communicate information using

  • brightness
  • color
  • movement
ideas

With just a few LED’s its easy to display a clear message. A large matrix of LEDS can get confusing. Here are a few suggestions:

  • Use separate regions for each block of information.
  • Draw guide markers to help locate positions by giving frame of reference.
  • Animation catches the eye but also distracts, and confuses.

examples

traffic lights

An effective but inefficient signaling method.

resistor codes:

Resistors use a numeric color code.

Find the value of any resistor by looking at the first four 4 color bands. Colors represent  base 10 exponential notation.

abacus

An abacus uses 5 or 10 beads for each digit. Faster than decoding a resistor and works with one color – but takes up more space.

level meters

Segmented level meters convey information using a line of pixels:

binary clock

Represents digits using binary coded decimal notation.

An LED grid in Max

This grid design was used for the visual interface of a shortwave radio:

And an etch-a-sketch:

download

https://github.com/tkzic/max-projects

folder: LED-display/rx-320/

files:

  • rxpanel2.maxpat (main patch)
  • panel2.js (javascript)

instructions

There are 2 large toggles – one for etch-a-sketch, and another for the radio simulator. Try one, then toggle it off before trying another one. If you forget – just restart the patch.

The number box near the top can be used to expand or contract the display size (while it is active) The default size is 17.

how it works

The same patch generated both video examples above. It uses javascript to make a two dimensional array of Max led objects. Each object is addressable by its position in the array. Here’s the code to make the objects:

//
// makecells - create matrix of led objects
//
function makecells( x, y, color )
{

var p;		// this patcher
var tmpstr;
var objname;

	post( "makecells: ", x, y, color );
	post( );

	p = this.patcher;

// make cells

	for( i = 0; i < x; i++ )
		{
		for(j = 0; j < y; j++ )
			{
			cell[i][j] = p.newdefault(xorigin + (i * cellspace), yorigin + (j * cellspace), "led" );    // create leds
			if(color != 9 )
				{
				cell[i][j].hidden = 0;
				cell[i][j].message( "pict", color );
				}
			else
				{
				cell[i][j].hidden = 1;
				}

			cell[i][j].varname = "led" + i + "x" +  j;							//  assigns name for future use	
			}
		}

}

 

blobs

Although each LED in a grid is addressable, its easier to group sections of the grid into blobs. Each blob is a unit that displays data, like a number for example. There are several types of blobs:

Here are the properties of a blob:

//  blob data structure
//
//   x, y upper left
//   lengthx, lengthy, 
//
//   orientation: 0 = horizontal, 1 = vertical
//   step  1 = downward or rightward,  -1 = upward or leftward  (this defines the corner of origin too)
//
//   data lorange, hirange
//   scale: 0 = no, 1 = yes
//
//   blobtype: 0 = generic decimal,  1 = spare,  2 = pushbutton flash,  3 = radiobutton, 
//   color code 0-9
//   contrast color 0-9
//
//   signed ( 0 = no, 1 = yes)		// booooooooooooooolean  
//   colorshift ( 0 = normal , 1 = use different colors every 3 digits, like comma separators (frequency display)
//
//   blink  (milliseconds duration for pushbutton flash type only (led blinktime )
//      
//   radio number
//   name
//   value
//
//

 

Blob data examples

Lets look at examples of various ways to display data – as used in the shortwave radio video above.

Here are 3 blobs that represent numbers in three different ways.

The far left column and the bottom row are key graphics. They give a frame of reference for the data.

columns

Moving from left to right…

The LED’s far left column, are a graphic key, starting with red on the bottom, represent the numbers 1-9

The next blue column is just a divider

The 3rd column of white dots is the signal strength data ranging from the 0-9. The current value is ‘4’, represented by a column of 4 dots.

The next nine columns (4-12) represent the frequency in Hz. ranging from 0-999,999,999. The data is in groups of three (as you can see by looking at the graphical key in the bottom row). The number currently displayed is: 4,999,991.

Negative frequencies are displayed by shifting the colors to values that don’t match the key graphic.

The last column is a radio button with 4 possible values and is currently set to ‘3’

rows

The bottom row is a key graphic, showing a different color, or group of colors for each data item. So for example, there is one white dot under the signal strength data in column 3. There are 3 groups of 3 dots (yellow, green, yellow) in columns 4-12 representing the frequency data in the format: 999,999,999.

The 2nd blue row from the bottom is a divider.

The next shows which data items are being controlled by modulators. The 3 white LED’s show modulation of frequency data in the million’s, 100’s, and ten’s  places.

Modulator units

There are 5 modulator units in the display. Data is represented using a color code..

  1.  red
  2. green
  3. blue
  4. yellow
  5. white

Gray LED’s represent ‘momentary’ controls in the off state. When a momentary button is pressed, it will blink white.

Here is an example of a modulator unit

The blue LED’s are just dividers (background space)

columns

The first column of data on the left is the on/off indicator and the modulator’s ID number.

The top LED of the column is the on/off toggle. It is blank, which means off.

The next two red LED’s together represent the ID number of the modulator: red = 1

For the remaining columns, the top row indicates whether the input gate is open allowing other modulators to control the parameter. Grey indicates the gate is closed, A white LED means the input gate is open.

The second and third columns of data are the clock speed and wave type. The 2 LEDS in each column are grouped together and are using the color code above. The clock speed is 5 (white). The wave type is 4 (yellow)

The fourth and fifth columns of data are the low and high range. Low range value is 5 (white) and high range value is 2 (green) – which doesn’t make sense, but this is simulator data.

The last column is the modulator destination activity indicator: grey if zero (not assigned) or white if any non-zero value.

modulator data structure:
// modulator data structure
// these are fixed structures 8x2, with specific color rules
//
// ulx, uly
// mod id number 1-n 
// on	: 0 = off, 1 = on
// modin :  modulation source index 0-4
// clockspeed  :  0-4
// wavetype : 0-4
// lorange : 0-4
// hirange : 0-4
// ingate : 0 - 4  (tells which control is being modulated)
// spare
// destination : 0-127 destination index  // this is displayed elsewhere
//

 

hardware interface

Adafruit 32×32 RGB LED panel

https://www.adafruit.com/products/1484?gclid=CMzrnYDB1b4CFTQQ7AodjXkAFw

local file notes:

files are also in tkzic/new max radio project/

There is a newer version adapted for the Max radio project – basically same code, but file names are

  • rxpanel3.maxpat
  • panel3.js

 

rx-320 control

Operate the TenTec rx-320 receiver in Max.

A simple control program. Features save/recall memory presets with random scanning.

download

https://github.com/tkzic/rx-320

folder: max/receiver

instructions

  • Click the print message near the serial object to get a list of the serial ports in the Max window
  • Unlock the patch and edit the serial object to use the correct port letter

notes

  • runs in Mac OS and Windows
  • It would be worth checking the code that sets frequency and BFO against the newer code in the LED-project. I can’t remember if I updated this older version
  • There is also a version of this program that runs with ICOM ic-746 receiver

RTTY with Arduino and NTX2

A circuit from the UK High Altitude Balloon sight, that sends RTTY, from Arduino using the Radiometrix NTX2 transmitter on 434.650 Mhz.

http://ukhas.org.uk/guides:linkingarduinotontx2#where_to_buy_the_ntx2

circuit notes

substituted  2 100k resistors in parallel for the 47k connected to TX output in the circuit

RTTY settings

We used rtl-sdr with Max as the receiver, in SSB mode, sending audio via Soundcloud to dl-fldigi to decode the RTTY. The params that worked:

  • 50 baud
  • carrier shift 400-480 hz (varies)
  • 7 bits per char
  • no parity
  • 2 stop bits
  • reverse mode on
audio input monitor in dl-fldigi

Unfortunately there was no way to hear the signal while decoding because dl-fldigi doesn’t feed through the input audio. A workaround is to run  Audacity in the background. In audacity:

  • set input to soundflower,
  • set output to built-in output
  •  press pause
  •  press record – now you should hear the radio.

pwm circuit

An updated version of the system described above.

This circuit features no additional components. Just the NTX2 and Arduino. The voltages are generated using PWM (pulse width modulation).

Described in a 3 part series.

by Anthony Stirk at ava.upuaut.net

http://ava.upuaut.net/?p=617

Example 1 – testing

The first example generates a carrier shift of 310 Hz. It took several minutes for the transmit frequency to stop drifting. The AFC setting in dl-fldigi helps to keep things locked in – as long as their is a constant carrier.

Example 2 – RTTY

Reducing the filter bandwidth to less than 150 Hz helps, due to the narrow carrier shift.

Example 3 – dominoEX MFSK

Requires a 175 ohm resistor for precise timing. Didn’t have one, so I just ran the sketch without decoding. Here’s what it sounds like:

what’s next?
  • decoding RTTY in Max

 

Stock market music in Max

Make music from the motion of stock prices.

This program gathers stock prices into a database. It generates Midi data – mapping price to pitch, and mapping trading volume to velocity and rhythmic density. It uses ancient Web technology: HTML/javascript front-end with a php back-end accessing a mysql database.

Case study: http://zproject.wikispaces.com/stock+market+music

To run this project, you will need a server (preferably linux) with the following capabilities:

  • mysql + phpmyadmin
  • php (and ability to run php over the web)
  • netcat (nc)
  • network access

All of this is pretty standard – so I won’t talk about it here. I am running it on Ubuntu Linux. There are many other ways to get the project working, by using the layout described here.

download

https://github.com/tkzic/internet-sensors

folder: stock-market

files

Max

stock_market_music.maxpat

HTML/javascript web client
  • newstock3.html: (web page interface)
  • selectstock3.js:  (front-end)
php server
  • getstock3.php (back-end server to get quotes and save them to a database)
  • play3.php: (back-end server to retrieve quotes, analyze, and map to Midi sequence to send to Max)
  • udp.php: Osc library
Set execute privileges on php files so they can be run from your web server. (chmod +x)

 

database

The selectstock3.php program harvests stock quote data and stores it in a mysql database.

The database name is:  stocks – table is: quotes

Table structure:

The table is basic flat representation of a stock quote, indexed by the ticker symbol. It contains price, volume, high/low/change, timestamp, etc., For our purposes, the price, volume and timestamp are essentially all we need.

SQL to create the table:

 CREATE  TABLE  `stocks`.`quotes` (  `ticker` varchar( 12  )  NOT  NULL ,
 `price` decimal( 10, 2  )  NOT  NULL ,
 `qtime` datetime NOT  NULL ,
 `pchange` decimal( 10, 2  )  NOT  NULL ,
 `popen` decimal( 10, 2  )  NOT  NULL ,
 `phigh` decimal( 10, 2  )  NOT  NULL ,
 `plow` decimal( 10, 2  )  NOT  NULL ,
 `volume` int( 11  )  NOT  NULL ,
 `ttime` timestamp NOT  NULL  DEFAULT CURRENT_TIMESTAMP ,
 `id` int( 11  )  NOT  NULL  AUTO_INCREMENT ,
 `spare` varchar( 30  )  DEFAULT NULL ,
 UNIQUE  KEY  `id` (  `id`  ) ,
 KEY  `ticker` (  `ticker`  )  ) ENGINE  =  MyISAM  DEFAULT CHARSET  = latin1 COMMENT  =  'stock quote transactions';

creating the database, user, and table

  • Log into  phpmyadmin as root
  • Create a new database called ‘stocks’
  • In privileges, add a user called: ‘webdb1′ with a password of ’34door’ (note you can change the password later)
  • In SQL, copy in the above query to create the ‘stocks’ table

Instructions

Web client

The webpage control program allows you to select stocks by ticker symbol, and get either one quote or get quotes at regular time interval. Each quote is inserted into the stock table for later retrieval and analysis.

 

The web front end is quirky so I will describe it in terms of how you might typically use it:

market is open – and you just want to play music based on current stock prices
  1. Enter the ticker symbols for your stocks
  2. Press ‘tracking’ button – so the quotes get saved
  3. Enter the IP address of the computer running Max
  4. Press the ‘auto’ button in the upper left corner – it will run and play forever
market is closed – or you want to play historical data you have saved
  1. Enter the ticker symbols for your stocks
  2. Enter the IP address of the computer running Max
  3. set start end end dates
  4. Press the ‘play’ button to play once or press ‘loop’ to play continuously (using time interval in seconds)
  5. market is open – you just want to collect stock quote data
  6. Enter the ticker symbols for your stocks
  7. Press the tracking button so quotes will get saved
  8. Press the ‘get quotes’ button to get current quote or press ‘loop’ button (on the same line) to retrieve quotes  continuously every 30 seconds.

Instructions

Max patch

  1. Make sure the IP address is set to the address of your server
  2. Select the Midi port for output
  3. Play a few test notes
  4. Select either ‘one instrument’ mode (piano) or multi instrument mode. Each time you click the multi instrument button it randomly selects a new combination 

notes on stock market data

To look at historical trends, you would need access to historical stock data. To use it as a tool for short term analysis, you would need access to real-time quote data in an API. At the time, both of these cost money.

However, it doesn’t cost money to get recent quotes from Yahoo throughout the day and store them in a database – so that’s the approach I took.

If I were to do this project today, I’d look for a free online source of historical data, in machine-readable form – because the historical data provides the most interesting and organic sounds when converted into music. The instant high speeding trading data would probably make interesting sounds as well, but you still need to pay for the data.

notes on local files