Objectives

Questions that prompted this prototype:

  • How might we interpret the EQ concept and parameters without recreating DAW EQ interfaces?
  • How might we design the visualization to show the frequency “clashing” concept?
  • How might we introduce some common adjectives that are used to describe the image of the sound?

 

The design goal for this prototype is to create:

  • EQ visualization in VR
    • Adjectives such as bright and muddy.
    • Redesign 3-Band EQ: Hard to come up with different visualizations other than 3-Band EQ in DAWs because it is one of the best ways to explain EQ.

Inspiration

(Audio Cube inspiration)

We decided to go with the Audio Cube visualization as inspiration for our prototype. We felt that showing all the tracks and their frequencies occupying one space can help guests understand how the frequencies interact with each other. This allows them to both hear and see when frequencies are clashing, which is a major skill to learn when mixing. 

 

We also decided to try two different forms of EQ, 3-band and what we call “Simple” EQ. 3-band was selected because it is relatively simple to interact with, while still giving guests ample power to influence the tracks’ frequencies. Our Simple EQ was inspired by a conversation with Andrew Chu, one of our industry contacts. He recommended that we try to visualize common adjectives used in EQ, such as Muddy or Bright. The guest will be interacting with objects that represent these two adjectives, and placing them on the track to influence its frequency. This simple implementation is more of an abstraction of equalization, compared to the 3-band implementation.

Design Specifications

Here’s a demo of the prototype on the track field layout, tracks, basic control functions, and volume/panning:

Audio Cube EQ
3-Band EQ

Implement equalization and audio cube representation in the prototype to allow guests to: 

  •  See frequency distribution of all the tracks.
  • Change the frequency composition of individual tracks (high/mid/low).

 

  • The timeline and song control panel remain the same.
    • The timeline became a straight line to match the edge of the audio cube.
  • The Audio Cube shows the panning of tracks with the x-axis, the volume of tracks with z-axis, and the frequency composition with y-axis.
    • Guests can change the volume and panning of individual track orbs by dragging it while holding down the grip button (same as in track field). This only works in the x-z plane defined by the current y position of the orb.
    • Guests cannot directly move the orb along the y-axis. The y position of individual track orbs is defined by the approximate center frequency of the track (from 20 to 20k Hz along the y-axis).
    • For each individual track in the Audio Cube, there are “a bunch of audio-reactive rectangles” placed along the y-axis to show the loudness of each frequency range.
  • Guests can enter and exit the EQ edit mode by clicking on a track orb.
    • In the EQ edit mode, three translucent filter cubes will appear on top of the frequency spectrum of that track, representing the High, Mid, and Low filter.
    • Guests can hover over one of the filter cubes and push the joystick to change the filter value.
      • Push forward to increase and backward to decrease the value.
    • Guests can hover over one of the two filter thresholds (low-mid and mid-high) and push the joystick to change the threshold frequency.
      • Push forward to higher and backward to lower the threshold.

During the EQ edit mode, guests may still move the track on the x-z plane to change the volume and panning.

Guests switch between the Simple and 3-Band EQ by selecting the “Switch to ____” button on the console.

In the Simple EQ mode, guests change the frequency of each track by adding a star (bright) or a nebula (muddy) to the orb once in EQ Edit mode. 

    • When the guest points to either object and clicks the trigger, a small duplicate of the object will begin rotating around the orb.
    • Guests can add up to 3 stars or nebulas to rotate around the orb.
    • Adding a nebula after adding a star will get rid of a star since they cancel each other out.
      • The opposite case is also true, with stars cancelling nebulas out.

Tech Setup

FMOD Setup:

 From the FMOD side, since using EQ was like adding any other plugin like Reverb to the FX chain of the track, the setup for the FX chain is the same. 

The main part of this prototype needed FFT (Fast Fourier Transform) data from each of the audio tracks to drive visuals crucial to the implementation. 

Major source for FFT related setup was this FMOD forum post : FMOD Unity FFT.

The choice was to use the 3 Band Parametric EQ plugin supported by FMOD since that gave all the necessary control through the API and was easy to use.

Unity Setup:

The crucial part of the Unity audio setup was to take care of the chain of DSPs added to the track FX chain. Since FFT is also technically considered a DSP, the setup required to put FFT into the tail of the chain after EQ and Reverb (if used) since the effect of changing the EQ had to be shown in the visuals.

A TrackFFTContainer script is attached to every track that takes care of all the FFT related data to be used for visuals.

The FMOD engine was able to give us an impressive 32 window FFT for more than 12 tracks in real-time for us to provide rich visuals. 

The “Y-position” of the orb in real time is determined by the weighted average of the power of a frequency bin. This was to show around where the track in

Another important setup was to get the frequency scale to a logarithmic scale instead of a linear one to have good resolution in the “y-axis”. 

The formula used to convert the linear audible frequency scale to logarithmic scale: 

         F(frequencylin) = ( Log(frequency) - Log(20h) * axis-length / (Log(24000h) - Log(20h))

Each track then has two rectangles that define the left and right channel frequency band amplitude at a particular time and they continuously lerp within the values fed in by the FFT. 

The last part of the EQ implementation : Simple EQ does a simple +3 – -3 scale of the 3 frequency bands of the same plugin. A 1 unit increase means increasing the higher and middle frequencies by a certain amount and decreasing the lower band by some small amount and vice versa. The current implementation has an exaggerated increase just to help understand the concept better.

Playtest Results

Playtest statistics:

Results:

Overall (Audio Cube):

  • Visualization: Environment was quite busy – hard to get used to the space and understand what’s going on in the audio cube.
  • Interaction: Discover an advantage of audio cube – push all orbs to the back and bring one orb to the front, so that all tracks volume can be turned down to focus on EQ one orb, but can still hear other tracks in the background.
  • Interaction: Question the necessity of the audio cube – if needed to see which frequency is clashing, can bring all orbs to the front.

3-Band EQ:

  • Interface: No direct visual feedback when using a joystick – hard to notice that the opacity changes are associated with joystick movements.
  • Interaction: Hard to hover to select on a certain filter band, sometimes yellow focus state doesn’t appear when hover.
  • Visualization: Visual feedback (opacity change) while using the joystick to adjust a filter band was not noticeable.

Simple EQ: 

    • Interaction: Easy to perform and can clearly see feedback.
    • Interaction: Unsure about what parameters changed when made brighter or muddier.
    • Visualization: The concept was easy to understand for nonmusicians.
    • Visualization: Bright & muddy were straight forward, but too simple:
      • Adding nebula/muddy sound like volume ↓ to 2 non-musicians
      • Adding sun/bright sound “crispy” or “louder”

Insights From Professionals

Some thoughts from industry professionals after they watched/experienced our prototype:

  • Showing the frequency compositions and the band-filter seems contradictory to the purpose of introducing EQ to newcomers. The simple EQ seems to be fulfilling that purpose more.
  • When newcomers start learning EQ, they probably don’t need to care about volume or panning.

Lessons learned

There are many parameters that we chose to incorporate in this prototype. Maybe too many for newcomers. The space is full of colors, moving orbs, rectangular frequency graphs, filter bands, sun & nebula. Interface wise, keeping the volume and panning function on the X-Z plane caused some distraction on understanding EQ. In fact, we questioned ourselves: is it necessary to keep volume and panning functions to interpret EQ? If not, why not “audio square”?

The 2 modes of EQ both have it’s advantages: 

3-Band EQ gives guests the power to interact with low, mid, high frequencies, which is rich in interaction. Guests can understand the EQ concept in more creative and different ways. However, the visual feedback of opacity change while interacting with the filter bands was not noticeable enough, resulting in lots of confusion before explaining it to the guests. 

Simple EQ simplifies the EQ concept by applying “adjectives” bright (sun) and muddy (nebula) onto tracks. The interaction and visualizations are straightforward and easy to understand. Yet, some may wonder what are the changing parameters while applying sun or nebula? Applying it sounds different, is it the volume change? How does the frequency change?

Future Considerations

  • Audio cube → Audio square, without volume and panning functions, but clear with frequency clashes (Shown below)
  • Adding more adjectives into Simple EQ.