Capturing App/Screen video with Syphon and NDI

How to get video from applications, like a web browser, into Syphon.

This example shows how to get a Youtube video from Firefox into Max/MSP Vizzie.

  • Use NDI Scan Converter (From NDI Tools https://ndi.video/tools/ ) and select Firefox from the Capture menu
  • Use NDISyphon to convert the NDI stream to Syphon. Use virtual audio routing (eg., black hole) for system audio, or route audio directly into Max.

  • In Max, use the Syphon client from the package manager to receive video. Here is an example using Vizzie abstractions described in previous posts:

notes:

NDI scan converter allows screen capture from apps and broadcasts in NDI format over the LAN. Then NDISyphon app converts NDI stream into Syphon. For youtube, use theatre mode to fill the window – full screen doesn’t really work. see this link: https://support.telestream.net/s/article/Wirecast-Remote-Computer-Screen-Capture-with-NDI-macOS

Recording Ableton Videosync output

Video recording options for Videosync in Ableton Live

Syphon Recorder.

https://syphon.github.io/recorder

  1. In Videosync preferences, set master Syphon out.
  2. In syphonrecorder set Videosync master as input.
  3. In syphonrecorder set virtual audio as input (eg., blackhole 2ch)
  4. In Ableton, set blackhole 2ch as audio output (or make a multioutput device in audoMidiSetup, combining headphones and blackhole.)

Vizzie Recorder.

(see this post about Vizzie Syphon abstractions https://reactivemusic.net/wp-admin/post.php?post=20896&action=edit)

  1. setup a Syphon client in Max
  2. Use abstractions to bring video into Vizzie. and use Vizzie recorder object.
  3. Route audio as described above.

Screenflick (or other screen recording software)

  1. Set screenflick to overlay videosync output window in Ableton Live
  2. Route audio as described above.

11 Km feedback delay

Using 40 meter SSB, Max/MSP, and Websdr to build a feedback delay from Maine to the Netherlands.

A Max patch plays an audio file into the transmitter in Maine. Then, using a websdr receiver in the Netherlands, the received signal is amplified and mixed back into the audio to be retransmitted. Effectively creating a feedback delay line.

Here is an example of what it sounds like on 40 meters.

http://www.pr0jex.com/longdelayline.mp3

This is the Max patch

The audio from Websdr is routed to the input of Max by creating a multi-output device (In Audio Midi Setup) combining “Blackhole 2ch” and external headphone output (for monitoring). The audio output of Max is assigned to the transmitter SSB input.

Patch not yet available. Local version is in max teaching examples.

Max/MSP Vizzie abstractions for Syphon

Vizzie objects can connect directly to Syphon clients and servers.

(work in progress – not available yet)

Syphon video as input to Vizzie

vizziesyphonclient.maxpat that gets syphon into vizzie. look at vizzieexamplewithsyphon.maxpat to see how it works.

Syphon as output from Vizzie

Use vizziesyphonserver.maxpat. Example: vizzieexamplewithsyphonserver.maxpat

(local files in maxteachingexamples)

X API update

Attempts to use the X API (formerly Twitter) for projects with Max/MSP have been disappointing at best. Most of the API is behind a paywall now.  The cost is $5000 per month. to implement streaming API used in projects like this:  https://reactivemusic.net/?p=5786

The free tier only allows basic tweeting and user lookup. Search is not available.

I was able to find only one node example that actually worked in the free tier. By “Coding with Ado”. The code requests a token from X, and then you enter a timestamped pin number to continue. Making it worthless for programs and bots. https://youtu.be/G5ZW5j5cwHk?si=vbAtGa0bQ3T_tga9

A local copy of the source code for this is in tkzic/nodetweet3/index.js

Other options

Another option with X, is to use a service like Socialdata. https://socialdata.tools/

There service sits in the middle to handle X API calls. You are charged by the number of calls. It doesn’t offer streaming either, but you can simulate it by calling a search every few seconds.

Other social media options

There are API’s for other social sites like facebook, instagram, tiktok, etc.,

Virtual serial port pairs in Mac OSx

I was unable to find any apps to create virtual serial ports for later versions of Mac OSx eg., Catalina.

There is a partial solution using the linux socat utility. It will create functioning vsp’s but the ports will not be detected by most apps, like Max/MSP.  I tried setting a symlink to files in /dev using tty. and cu. prefixes. eg., /dev/tty.x1. But apps that look for serial ports still couldn’t detect the vsp’s.

Instructions

  1. download and install socat using macports or brew. eg., “sudo port install socat”
  2. Disable SIP (system integrity protection). In Catalina, reboot holding cmd+r. Then type: “csrutil disable” into a terminal window. Reboot – Note: you probably don’t need to do this if you aren’t setting up a link in /dev – which so far doesn’t work anyway
  3. Run one of the following commands:

socat -d -d pty,raw,echo=0 pty,raw,echo=0

sudo socat -d -d pty,link=/dev/tty.x0,raw,echo=0,user=tkzic,group=staff pty,link=/dev/tty.x1,raw,echo=0,user=tkzic,group=staff

What works and what doesn’t

After setting up the ports, you can send data in one port and out the other, for example, type the following into 2 different terminal windows.

cat /dev/ttys004

echo “Hello World” > /dev/cu.x1

This works great. But, so far, using sym links – the ports don’t show up in any app that accesses serial ports. Still working on this…

Using socat you can also setup a serial/tcp pair – this may be a solution for some projects.

MBTA API in Max

Sonification of Mass Ave buses, from Nubian to Harvard.

Updated for Max8 and Catalina

This patch requests data from MBTA API to get the current location of buses – using the Max js object. Latitude and Longitude data is mapped to oscillator pitch. Data is polled every 10 seconds, but it seems like the results might be more interesting to poll at a slower rate, because the updates don’t seem that frequent. And buses tend to stop a lot.

Original project link from 2014: https://reactivemusic.net/?p=17524

MBTA developer website: https://www.mbta.com/developers

This project uses version 3 of the API. There are quality issues with the realtime data. For example, there are bus stops not associated with the route. The direction_id and stop_sequence data from the buses is often wrong. Also, buses that are not in service are not removed from the vehicle list or indicated as such.

The patch uses a [multislider] object to graph the position of the buses along the route – but due to the data problems described above, the positions don’t always reflect the current latitude/longitude coordinates or the bus stop name.

download

https://github.com/tkzic/internet-sensors

folder: mbta

patches:

  • mbta.maxpat
  • mbta.js
  • poly-oscillator.maxpat
authentication

You will need  to replace the API key in the message object at the top of the patch with your own key. Or you can probably just remove it. The key distributed with the patch is fake. You can request your own developer API key from MBTA. It’s free.

instructions
  • Open mbta.maxpat
  • Open the Max console window so you can see what’s happening with the data
  • click on the yellow [getstops] message to get the current bus stop data
  • Toggle the metro (at the top of the patch) to start polling
  • Turn on the audio (click speaker icon) and turn up the gain

Note: there will be more buses running during rush hours in Boston.  Try experimenting with the polling rate and ramp length in the poly-oscillator patch. Also, you can experiment with the pitch range.

Soundcloud API in Max8

This post describes an updated project.

The earlier version can be found here  https://reactivemusic.net/?p=5413

This project is part of the internet-sensors project: https://reactivemusic.net/?p=5859

Note

2024/01/21: not working:

Soundcloud deleted my API credentials and is not accepting new requests. So there is really no point to using this project unless you have active credentials.

console.error() function crashes node. -fixed locally

Overview

In this patch, Max uses the Soundcloud API to search tracks and then select a result to download and play. It uses the node.js soundcloud-api-client https://github.com/iammordaty/soundcloud-api-client

For information on the soundcloud API http://developers.soundcloud.com/docs/api/reference

download

https://github.com/tkzic/internet-sensors

folder: soundcloud

files

main Max patch
  • sc.maxpat
node.js files
  • scnode.js

authorization

  • The Soundcloud client-id is embedded in scnode.js – you will need to edit this file to replace the worthless client-id with your own. To get a client ID you will first need a Soundcloud account. Then register an app at: http://soundcloud.com/you/apps

first time instructions

  • Open the Max patch: sc.maxpat
  • In the green panel, click on [script npm init]
  • In the green panel , click on [script install soundcloud-api-client]

instructions

  • Open the Max patch sc.maxpat
  • open the max.console window so you can see the API data
  • click [script start]
  • click the speaker icon to start audio
  • type something into the search box and press <enter> or click the button to the left to search for what is already in the box.
  • select a track from the result menu, wait for it to start playing

Spotify segment analysis player in Max

Echo Nest API audio analysis data is now provided by Spotify. This project is part of the internet-sensors project: https://reactivemusic.net/?p=5859

There is an older version here using the discontinued Echo Nest API: https://reactivemusic.net/?p=6296

Note:  Last tested 2024/01/21

The original analyzer document by Tristan Jehan can be found here (for the time being):  https://web.archive.org/web/20160528174915/http://developer.echonest.com/docs/v4/_static/AnalyzeDocumentation.pdf

This implementation uses node.js for Max instead of Ruby to access the API. You will need set up a developer account with Spotify and request API credentials. See below.

Other than that, the synthesis code in Max has not changed.  Some of the following background information and video is from the original version. ..

What if you used that data to reconstruct music by driving a sequencer in Max? The analysis is a series of time based quanta called segments. Each segment provides information about timing, timbre, and pitch – roughly corresponding to rhythm, harmony, and melody.

spotify-synth1.maxpat

download

https://github.com/tkzic/internet-sensors

folder: spotify2

files

main Max patch
  • spotify-synth1.maxpat
abstractions and other files
  • polyvoice-sine.maxpat
  • polyvoice2.maxpat
node.js code
  • spot1.js
node folders and infrastructure
  • /node_modules
  • package-lock.json
  • package.json
dependencies:
  • You will need to install node.js
  • the node package manager will do the rest – see below.

Note: Your best bet is to just download the repository, leave everything in place, and run it from the existing folder

authentication

You will need to sign up for a developer account at Spotify and get an API key. https://developer.spotify.com/documentation/general/guides/authorization-guide/

Edit spot1.js replacing the cliendID and clientSecret with your spotify credentials

node for max install instructions (first time only)

  •  Open the Max patch: spotify-synth1.maxpat
  •  Scroll the patch over to the far right side until you see this green panel:

  • Click the [script npm init] message – this initializes the node infrastructure in the current folder
  • Then click each of the 2 script npm install messages –  this installs the necessary libraries

Instructions

  •  Open the Max patch: spotify-synth1.maxpat
  •  Click the green [script start] message
  • Click the Speaker icon to start audio
  • Click the first dot in the preset object to set the mixer settings to something reasonable
  • open the Max Console window so you can see the Spotify API data
  • From the 2 menus at the top of the screen select an Artist and Title that match, for example: Albert Ayler and “Witches and Devils”
  • Click the [analyze] button – the console window should fill with interest data about your selection.
  • Click [play]
  • Note: if you hear a lot of clicks and pops, reduce the audio sample rate to 44.1 KHz.
Alternative search method:

Enter an Artist and Song title for analysis, in the text boxes. Then press the buttons for title and artist. Then press the /analyze button. If it works you will get prompts from the terminal window, the Max window, and you should see the time in seconds in upper right corner of the patch.

troubleshooting

If there are problems with the analysis, its most likely due to one of the following:

  • artist or title spelled incorrectly
  • song is not available
  • song is too long
  • API is busy
Mixer controls

The Mixer channels from Left to right are:

  • bass
  • synth (left)
  • synth (right)
  • random octave synth
  • timbre synth
  • master volume
  • gain trim
  • HPF cutoff frequency
You can also adjust the reverb decay time and the playback rate. Normal playback rate is 1.

programming notes

Best results happen with slow abstract material, like the Miles (Wayne Shorter) piece above. The bass is not really happening. Lines all sound pretty much the same. I’m thinking it might be possible to derive a bass line from the pitch data by doing a chordal analysis of the analysis.

Here are screenshots of the Max sub-patches (the main screen is in the video above)

Timbre (percussion synth) – plays filtered noise:

Random octave synth:

Here’s a Coltrane piece, using roughly the same configuration but with sine oscillators for everything:

There are issues with clicks on the envelopes and the patch is kind of a mess but it plays!

Several modules respond to the API data:

  • tone synthesiszer (pitch data)
  • harmonic (random octave) synthesizer (pitch data)
  • filtered noise (timbre data)
  • bass synthesizer (key and mode data)
  • envelope generator (loudness data)

Since the key/mode data is global for the track, bass notes are probable guesses. This method doesn’t work for material with strong root motion or a variety of harmonic content. It’s essentially the same approach I use when asked to play bass at an open mic night.

additional notes

Now that this project is running again. I plan to write additional synthesizers that follow more of the spirit of the data. For example, distinguishing strong pitches from noise.

Also would like to make use of  the [section] data as well as the rhythmic analysis. There is an amazing amount of potential here.

Max8radio CubicSDR I/Q prototype

Another working prototype with Max and CubicSDR

Now working some better… The Max SDR patch is receiving an IQ audio stream at 96 KHz from CubicSDR and sending frequency data to rigctld daemon via a python script that recodes OSC to tcp data.

repository: https://github.com/tkzic/max8radio

Files:

max8sdr1.maxpat

py3rigctl2.py (python script)

Instructions:

Basically the same as instructions in the previous prototype here: https://reactivemusic.net/?p=19995

make sure to start the rigctl daemon before CubicSDR

 rigctld -m 1 4532 & 

And make sure there is some audio gain on CubicSDR

But… There is only one Max patch now and – after you start the rigcltd daemon, you need to run the python script in the max8radio folder like this:

python3 py3rigctl2.py

The most important thing is to start CubicSDR first before you run the Max patch. Make sure to get everything working correctly. Then start the Max SDR.

In CubicSDR make sure you only have one “modem” running – otherwise the IQ data stream will be a complete mess. Als0 make sure that the audio sample rate in CubicSDR is set to 96 KHz. It will revert to 48 KHz. everytime you load the program. You can use the ‘bookmarks’ from a previous CubicSDR session (lower left part of the screen) to load a previous session with the same parameters.

These are the necessary settings:

  • I/Q modem
  • Audio out: Existential Audio Inc. Blackhole 2 ch.
  • Audio sample rate: 96 KHz.
  • Rig Control Menu: enable rig and follow rig should be ‘checked’
  • Frequency should equal Center frequency and the V delta lock toggle should be on
  • Demodulator Gain level should be very low to prevent excess AGC (upper right corner)

Actually if you have loaded everything ok in a previous session, try this:

  • get the rigctld daemon running from the command
  • load CubicSDR
  • First thing: click ‘enable rig’ under rig control (this will probably load some crazy frequency like 145 Mhz
  • Then in the bookmarks (lower left) double click on your previous session, under ‘recents’ for example: 7007MHzI/Q – this should restore almost all the settings.
  • Then change the audio sample rate to 96 KHz if needed.
  • If the input to Max seems wrong, try clicking the S  (over near the top right)  to solo the modems. There may be more than one going.

Max settings

  • Set audio input to Blackhole 2ch @ 96 KHz. (to match output from CubicSDR
  • Click the ‘flip IQ’ toggle – for some reason CubicSDR sends out the I/Q signal flipped
  • The arrow key tuning and all other tuning methods should work now

Notes

One of the problems with CubicSDR is sometimes you’ll accidentally change something and all the settings go crazy.

note: I tried a new version of CubicSDR (2.6) from the sdrplay website. It would not detect any connected devices or audio drivers.

Once you get it working, the audio quality inside Max is excellent – using the Airspy HF+