ITCC 007 | Using Libraries, Sound Visualizations

Inspiration

Today’s Artists

Using Libraries

Video

If you try to search for video in the Processing language reference, you will notice that there is nothing to be found. The reason for that is that video playback is not part of the core Processing functionality, you need to use an additional library.

Libraries are collections of code that extend the functionality of Processing. There are a couple of libraries that are built-in (such as Serial and Network), but most of them need to be installed from the Library Manager.
(Sketch–>Import Library—>Add Library)

screen-shot-2016-10-14-at-02-31-55

Search for the official Video library from The Processing Foundation and install it.

screen-shot-2016-10-14-at-02-34-37

The best way to figure out how the library works, is to go through the examples that come with it. Note that the functionality is split into two main classes:

  • Movie: used to play video files. The library is a bit fussy about the codec of the video. H264 .mov files seem to work well.
  • Capture: Used to capture the image from a webcam connected to the computer.
screen-shot-2016-10-14-at-02-41-29

Sound

Playing sound in Processing also requires an external library. Try these libraries:

  • Sound library – The official sound library from the Processing foundation (has some bugs and issues. Especially on Windows)
  • Minim library – Often works better, so let’s use this mainly
screen-shot-2016-10-14-at-02-42-47

There are some other sound libraries also, if you need to do something more specific.

Again the best way to learn how it works is to go through the examples.

Other Useful Libraries

You might want to install some additional libraries while you are at it. Some of the features might be beyond your skill level right now, but the libraries listed below are some interesting ones that you might want to come back to later. I added links to the libraries, but you should be able to install these directly with the Library Manager.

Sound Visualizations

Example

This example uses local variables inside the for loop to draw multiple shapes. The index i is used to give each shape a unique location. See the code. Run the code in Processing so you can use the Tweak mode to see how changing the values influences the outcome.

  • for – Check the reference for the for loop, if you don’t remember how it works.
  • map() – Also see the map function. We use it a lot today to scale different values

Sound Visualizer

This example uses the Sound library. So please install it using the library manager. We are going to use the following classes of the library:

  • AudioIn – Allows us to use the audio input (microphone) of the computer
  • Amplitude – Gives the amplitude of the audio input
  • FFT –  Fast Fourier Transform (FFT) analyzer for the sound input.

1st Step: Enable Audio Input and Do Simple Amplitude Analysis

The code below shows you how you can modify the example above to control the size of the ellipses using amplitude analysis from the Sound library. The sound library is not available in OpenProcessing so you need to run the code in Processing.

import processing.sound.*;

AudioIn input;
Amplitude loudness;
int num = 16;

void setup() {
  size(600, 600);
  input = new AudioIn(this, 0);
  input.start();
  loudness = new Amplitude(this);
  loudness.input(input);
}

void draw() {
  background(135, 80, 135);
  // analyze the amplitude of the audio input
  float volume = loudness.analyze();
  textAlign(LEFT);
  textSize(32);
  text("volume: " + volume, 20, 100);

  for (int i=0; i<num; i++) {
    float s = map(volume, 0.0, 1.0, 10, 50);
    float x = map(i, 0, num-1, 50, 550);
    float y = 300;

    // ellipses
    fill(0);
    ellipse(x, y, s, s);

    // text
    textAlign(CENTER);
    textSize(16);
    text(i, x, y-s);
  }
}

2nd Step: FFT

Then we can add the FFT analysis, and connect each band of the FFT to the corresponding ellipse size.

What is FFT? The actual definition of it is very complex, but with what we are doing here, we can describe it in a bit simpler way. We are essentially able to get multiple amplitude values of different frequency bands. What does that mean? FFT frequency analysis allows you to isolate the different frequencies of a sound, like the low frequencies of a bass drum or high frequencies of a hi-hat. You could then make parts of your sketch react differently to different frequencies, create a beat detection algorithm, or make a game where you have to whistle at a specific frequency to play it.

import processing.sound.*;

AudioIn input;
Amplitude loudness;
FFT fft;
int num = 16;
float[] spectrum = new float[num];

void setup() {
  size(600, 600);
  input = new AudioIn(this, 0);
  input.start();
  loudness = new Amplitude(this);
  loudness.input(input);
  fft = new FFT(this,num);
  fft.input(input);
}

void draw() {
  background(135, 80, 135);
  // analyze the amplitude of the audio input
  float volume = loudness.analyze();
  // FFT
  fft.analyze(spectrum);
  textAlign(LEFT);
  textSize(32);
  text("volume: " + volume, 20, 100);

  for (int i=0; i<num; i++) {
    float s = map(spectrum[i], 0.0, 0.4, 10, 100);
    float x = map(i, 0, num-1, 50, 550);
    float y = 300;

    // ellipses
    fill(0);
    ellipse(x, y, s, s);

    // text
    textAlign(CENTER);
    textSize(16);
    text(i, x, y-s);
  }
}

3rd Step: Add Smoothing

We didn’t do this in class, but it is something that you might want to do to make the visualization a bit more pleasing to the eye. The raw values from the FFT change really quickly, which is not always what you want for the graphics.

import processing.sound.*;

AudioIn input;
Amplitude loudness;
FFT fft;
int num = 16;
float[] spectrum = new float[num];

// the sum[] array is used to store the smoothed values 
float[] smooth = new float[num];
// how smooth you want it? smaller number-->smoother animation
float smoothingFactor = 0.2; 

void setup() {
  size(600, 600);
  input = new AudioIn(this, 0);
  input.start();
  loudness = new Amplitude(this);
  loudness.input(input);
  fft = new FFT(this,num);
  fft.input(input);
}

void draw() {
  background(135, 80, 135);
  // analyze the amplitude of the audio input
  float volume = loudness.analyze();
  // FFT
  fft.analyze(spectrum);
  textAlign(LEFT);
  textSize(32);
  text("volume: " + volume, 20, 100);

  for (int i = 0; i < num; i++) {
    // smooth the values
    smooth[i] += (spectrum[i] - smooth[i]) * smoothingFactor;
    
    float s = map(smooth[i], 0.0, 0.4, 10, 100);
    float x = map(i, 0, num-1, 50, 550);
    float y = 300;

    // ellipses
    fill(0);
    ellipse(x, y, s, s);

    // text
    textAlign(CENTER);
    textSize(16);
    text(i, x, y-s);
  }
}

Examples

Some examples using p5js