Creative Programming: Sound

Lecture 5 (24 Feb)

Table of contents

• (Re)visiting slides?
• Milestone 1:
• Bounding Values
  • Constraining via if
  • Constraining via constrain
  • Constraining via min/max
  • Constraining via modulo
  • Constraining via mapping
  • Constraining in action
• The World of Sound
  • Sound in the real world
  • Paint with Music
  • Coded Instruments
  • The synthesizer
  • The p5.js synthesizer
  • Libraries
  • Downloading the p5js sound library (1)
  • Downloading the p5js sound library (2)
  • Downloading the p5js sound library (3)
  • Putting the p5js sound library in your sketch (1)
  • Putting the p5js sound library in your sketch (2)
  • Setting up your sketch HTML (1)
  • Setting up your sketch HTML (2)
  • Setting up audio in your sketch
  • Playing a note
  • Reacting to mouse position
  • Playing multiple notes
  • Synthesizing!
• Playing sounds
  • Sound files
  • The return of preload
  • Sound example
  • Finding sounds
  • Looping and pausing
• Recording
  • Microphone input
  • S(l)ide note
  • Volume levels
  • Mapping the microphone
  • Easing the microphone
  • Sound as vibrations
  • Microphone-controlled synthesizer
  • Sound reactive

The first slide of every lecture introduces what we can make with the things we learn this week! This week we are playing with sound and microphones, as well as go through some constraint techniques that are useful for all kinds of projects.

(Re)visiting slides?

This is where you can find the narration notes. Some slides don’t have any notes (especially this first week) as notes are typically reserved for slides that explain technical concepts.

You can jump to any slide with the menu to the (bottom) left.

Did you miss a slide or want to revisit? Open the narration tab while studying to get an explanation of difficult slides.

Milestone 1:

We go over some of Thomas’ slides here regarding the first milestone.

• We will go through the requirements for the first milestone, which is due next week

Milestone Guidance

• If you need some feedback/help for the first milestone, contact me during one of the class assignments or breaks
• As always, you can also ask and discuss with the TAs on Thursdays

Bounding Values

Constraining via if

This can be the correct choice, particularly if you need to do some more complex logic whenever the constrainment condition is met. However, for simple constrainment of values, it can be verbose and error-prone.

• if statements can be used to constrain values, but they can be verbose and error-prone.
• Example: constraining a variable constrainedX to the square.

Constraining via constrain

This is usually the best choice for simple constrainment.

• p5.js provides a built-in function constrain() that simplifies the process of constraining values.
• Example: constraining a variable constrainedX using constrain().

Constraining via min/max

Min and max allow you to constrain just a single side.

• Another way to constrain values is by using the min() and max() functions.
• min() returns the smaller of two values, while max() returns the larger.
• Example: constraining a variable constrainedX using min() and max().

Constraining via modulo

This is wrapping behaviour, which can be useful in some cases (e.g. looping animations, games, etc.). However, it is not suitable for all types of constrainment, especially if you want to stop values at the boundaries rather than wrapping them around. Also be careful of negative values.

The value of the modulo operator comes from the remainder of division, which can be used to wrap values around a certain range. For example, if you have a variable x that you want to wrap between 0 and 100, you can use x % 100 to ensure that x stays within that range. If x exceeds 100, it will wrap back around to the beginning of the range.

• The modulo operator % can be used to wrap values around a certain range.
• This is particularly useful for creating looping behavior, such as in animations or games.
• Example: wrapping a variable moduloX to be between 0 and 220.

Constraining via mapping

The term mapping comes from mapping two ranges to each other. For example, if you have a variable x that ranges from 0 to 400 and you want to map it to a range of 0 to 100, you can use the map() function to scale x accordingly. The map() function takes five arguments: the value to be mapped, the minimum and maximum of the input range, and the minimum and maximum of the output range. In this case, you would call map(x, 0, 400, 0, 100) to map x from the range of 0-400 to the range of 0-100.

• The map() function scales and constrains values.
• You map two ranges to each other, and the function will automatically scale the input value to fit within the output range.
• Example: mapping a variable mappedX from 0-400 to 100-220.

Constraining in action

Another nice challenge is to give the car a velocity and have it turn.

Class Assignment: Car

1. Create a shape that moves with the arrow keys. (Tip: check the reference!)
2. Use one of the constraining techniques to keep the shape within the canvas boundaries.
3. Can you make it so that the shape wraps around the canvas instead of stopping at the edges?

The World of Sound

Sound in the real world

Sound is crucial for many of the things we interact with in the world, such as music, movies, games, etc. It can enhance our experiences and create emotional connections.

Paint with Music

Paint with Music by Simon Doury and Caroline Buttet

Coded Instruments

The synthesizer

The nice thing about the synthesizer as an instrument is that it originally distilled tones to their basic components, which is great for an example here. Current-day synthesizers can be very complex and can produce a wide variety of sounds, but the basic principles of sound synthesis still apply.

The p5.js synthesizer

• We can play sound in p5.js using a synthesizer

let synthesizer;

function setup() {
  createCanvas(400, 400);
  synthesizer = new p5.MonoSynth()
}

function draw() {
  background(220);
  text("Click me to play 'tuning A'", 100, height / 2);
 
}

function mousePressed() {
  userStartAudio();
  synthesizer.play("A4", 1, 0, 1);
}

Libraries

• This code does not work by default! We need to include a library for extra functionality.
• The p5.js sound library provides various tools for working with sound.

Downloading the p5js sound library (1)

Go to the GitHub repository of the course and click on the libraries folder

Downloading the p5js sound library (2)

Open the p5.sound.min.js file

Downloading the p5js sound library (3)

Download the raw file

Putting the p5js sound library in your sketch (1)

Make your (new) sound sketch and go to its folder

Putting the p5js sound library in your sketch (2)

Go to the libraries folder in your sound sketch and put the p5.sound.min.js file there

Setting up your sketch HTML (1)

In Processing, open up the index.html file of your sketch

Setting up your sketch HTML (2)

Add the following to your index.html:

<script language="javascript" type="text/javascript" src="libraries/p5.sound.min.js"></script>

Setting up audio in your sketch

Extremely annoying yet necessary.

• Browser makers (Google, Mozilla, Apple, etc.) do not like it when websites play sound without the user’s consent.
• To prevent this, they require that audio can only be played in response to a user interaction (e.g. a mouse click or key press).
• We have to call the userStartAudio() function in a user interaction event (e.g. mousePressed()) to enable audio in our sketch.

Playing a note

• We can play single notes by specifying their note name (e.g. “A4”) or their frequency in hertz (e.g. 440).
• play(note, volume, delay, time) takes four arguments to specify the pitch, volume (0-1), delay before the note starts (in seconds), and duration of the note (in seconds).

Reacting to mouse position

• We can use frequencies to play notes that react to the mouse position.

Playing multiple notes

Websites such as these really help to understand how to make something sound nice instead of just creating random noise.

• Multiple notes can be played with p5.PolySynth()
• Check out a website for this (e.g. chords) https://www.michael-thomas.com/music/class/chords_notesinchords.htm

Synthesizing!

Class Assignment: Melody

1. Set up a synthesizer in p5js and play a note when a key is pressed.
2. Store a melody of notes in an array and play it back when a button is pressed instead.
3. By using a polySynth, can you play a melody of chords instead of notes?

Playing sounds

Sound files

Be careful with sound files, they tend to be extremely large.

• Of course we can also play sound files instead of synthesizing sounds ourselves.
• p5.js provides a loadSound() function to load sound files and a play() method to play them.
• You can stop, pause, and loop sounds as well using the stop(), pause(), and loop() methods.
• setVolume() sets the volume of the sound, similar to how we can set the volume of a synthesizer note.

The return of preload

• Just like with images, we need to use the preload() function to load sound files before the sketch starts.
• This ensures that the sound files are available when we want to play them.
• Remember! Loading files can be done with:
  • A website link (such as in the slides)
  • A local file in the same directory as your sketch (e.g. cymbal.wav if you have the file in the same folder as your sketch)

Sound example

• Let us look at an example of loading and playing sound files in p5.js.

Finding sounds

Some of these might require accounts.

• There are many websites where you can find free sound files to use in your projects, such as:
  • Freesound
  • Free Music Archive
  • SoundBible

Looping and pausing

• If you want to have a sound repeat, use loop() instead of play().
• You can pause and resume the sound using pause() and play().
• Finally, you can stop the sound completely with stop() .

Recording

Microphone input

It is (of course) possible to do way more with microphones than just read their level but that is outside of the scope of this course. You can definitely investigate this for your project, though.

• Sound goes both ways! We can also capture sound using the microphone.
• In p5js, we can use the p5.AudioIn() class to access the microphone and capture sound input.
• We have to make a new p5.AudioIn() object and call the start() method to begin capturing sound.

S(l)ide note

• Microphone input and interactive slides are not the best of friends.
• Some of the examples may not work properly in the slideshow, but you can copy the code into your own sketch to test it out.

Volume levels

• The getLevel() method of the audio input object returns the current volume level of the sound being captured by the microphone.
• This can be used to create visualizations or interactive elements that respond to sound input.

Mapping the microphone

• The values returned by getLevel() are typically between 0 and 1.
• We can use the map() function to scale the volume level from the microphone to a range that is suitable for our sketch.

Easing the microphone

Try to change the easing values (0.8 and 0.2) to create a smoother/less smooth effect.

• The microphone values are very jittery and can change rapidly, which can make it difficult to create smooth visualizations or interactions.
• Easing can get rid of the jitter and create smoother transitions between values.

Sound as vibrations

Keep in mind that you technically do not know whether the user is shouting or blowing.

• Microphone input does not just have to be sound
• Some media (such as games) use super high volume levels to check whether users blow into the microphone

Microphone-controlled synthesizer

Be careful of feedback (the microphone picking up the sound).

• We can combine the microphone input with a synthesizer to create a sound-reactive instrument.
• For example, we can use the volume level from the microphone to control the frequency or amplitude of a synthesizer note.

Sound reactive

We did not end up doing this assignment during the lecture due to time constraints, but it is a nice assignment to try out on your own!

Class Assignment: Sound Reactive

1. Set up a microphone input in p5.js and use the volume level to control a visual element (e.g. the size of a circle). You can also modify an old sketch!
2. Go on one of the free sound websites and find a sound file that you like. Load it into your sketch and play it when a button is pressed.
3. Can you make it so that the sound file loops as long as there is sound over the microphone?