This is one of my favorite things to teach on Day 1 of my creative coding workshops and courses. We start by learning what audio and video is made of, and then spend time exploring data mapping options and techniques for fine tuning. It is also a jumping off point for coding from scratch. For more patches and course materials check my GitHub.

What is video data?

One of the first things I teach is how video data is made, what it looks like from the machine perspective, what technical parameters are worth understanding and how we can use this knowledge for creativity. This image is from a Max patch that is playing a video of a ceiling fan which you can see moving in the RGB data matrix. Watch the video and see for yourself.

What is audio data?

The next thing I teach is what audio data looks like. The goal on day 1 is to create audio responsive videos using a volume envelope and a dozen or so video filters that are controlled by audio. Seeing the audio data helps to understand the data mapping process. This Max patch plots the volume at 20ms intervals and allows for basic filtering in order to isolate frequency ranges. You can see and hear this in the video.

Interactive Video Machine

A teaching tool I developed in Max for controlling video filters with sound. This particular version was presented for Cycling ‘74 at the Ableton Loop Conference in Los Angeles in 2018. You can download the patches and documentation here and watch a demo video at the link below.