Objectives

Questions that prompted this prototype:

  • How could we represent volume and panning tools in the VR space without specific knobs and wording cues?
  • How could we make the song track representation obvious for guests to  understand the status of each track
  • How could we make the interactions easy to perform and understand? 
  • How could we design the basic control functions without blocking the song tracks, but also make it always accessible to guests?


The design goal for this prototype is to create:

  • Representation of individual music tracks
  • Basic control functionalities, including play/pause, stop, master mute (deactivate all), master solo (activate all). 
  • Volume tool
  • Panning tool

Inspiration

Paul Mc Cartney's "PRESS" Album cover

One of the main takeaways from interviews with industry professionals is that we should use the advantage of VR to create methods for people to understand sound mixing without the limitations of 2D interface.

Volume, panning, and layout inspiration: illustration made by Paul McCartney during mixing to visualize stereo sound. 

  • Layout (Trackfield): Seeing how the illustration positioned each instrument in a song gave us the thought of setting up a sector shaped track field to place all tracks. 
  • Panning: On the track field, tracks can be panned left or right to indicate the change of source sound.
  • Volume: On the track field, tracks can be dragged closer or further away to indicate the change of decibel volume.

Design Specifications

Here’s a demo of the prototype on the track field layout, tracks, basic control functions, and volume/panning:

Track Field:
    • No floor, highlight the edge – prevent guest from walking over
    • 90°-100° arc shape – designed for panning (left/right of the sector) and volume (forward and backward of the sector)
    • As orbs are further from guest, they are inclined (like an amphitheater) – prevent orb clusters
Tracks (orbs):
    • Half transparent orbs with 2D instrument floating in each orb – label track instrument
    • Status of track:
      • ON – colorful
      • PLAYING – colorful and glowing
      • OFF – grey
      • FOCUSED – enlarged/highlighted
Basic control functions:
    • In front of the guest – indirect control to prevent guests walking toward the track field and make sure orbs are far enough that they use point & drag to move the orbs instead of reaching out to it by hands.
    • Control panel with 5 pillars, holding 5 basic functions: play/pause, stop, loop, on(solo), off(mute) – these functions should always be accessible for guests to control each track, the height should be below eye level but still within guests’ vision.

VR Space:

  • Celestial aesthetic to indicate an ambiguous space prompting guests to think outside the box of normal DAW interfaces.
On boarding for stereo audio:
    • Based on feedback from some early playtesters, we created a “first step” for guests to follow–putting on a virtual headphone, which is the only thing guests would see initially. This first step prepares the guests for the upcoming stereo audio they’ll hear in the VR space and makes it clear that there won’t be spatial audio in this experience.

Tech Setup

The research that the team did before the start of the semester suggested to use an Audio middleware for what had to be achieved. Even though Unity’s audio system has improved significantly, there are still some features of an audio middleware that make the audio pipeline easy to handle. FMOD was the first choice since there are mentors in ETC who have FMOD knowledge. Hence, the pipeline is as follows :
  1. Each track (stem) from a DAW or any online license sources of stems are the source of the setup
  2. They are setup in FMOD as separate events (reasoning for this will follow)
  3. In Unity, a MasterBusController script serves as the main interface between FMOD and Unity. It helps setup song level controls like Play/Pause/Stop/Un-mute all/Mute all etc.
  4. The fmod banks that contain all the information about the FMOD setup are then built into Unity under the streaming assets.
  5. There was no specific customization necessary for Quest and hence this single setup was used to build both for Oculus Quest and Oculus Rift PC build.
 
Audio Setup:

Some advantages of using FMOD Audio Middleware: 

  • The DAW like layout makes it easy to do edits within FMOD itself instead of having to jump back to a DAW (the obvious!) 
  • The extent of DSP plugins that come with FMOD was the deal breaker and the interface that FMOD provides through it’s API is easy to use (even though it’s not documented very well)
  • More online content regarding FMOD and how to use it is available compared to extensively using Unity’s audio system. 

 

On that note, some of the most useful resources that was used: 

 
FMOD specific setup:

Since the sources are stems, and as the project demands individual control for each track through the FMOD API, the FMOD side setup is slightly different from a regular game audio like use case for FMOD. 

Each song is placed in a folder and each of its stems are setup as a separate 2D event and assigned to a bank as shown in the screenshot. They are set up as 2D events since it’s going to be strictly rendered as stereo and need not use more than 2 channels at any point. 

This set up allows for addition of any DSP module to any of these tracks independently through the API. The channel group that each event is routed to is the main source for addition of any sort of DSP chain, panning controls and volume controls.

Note: FMOD still allows to add separate plugins through the desktop interface for each audio track within an event but that isn’t straightforward to do from the API

Unity specific setup:
Apart from the MasterBusController script mentioned above, there is an additional TrackFMODManager that controls each track. It has functionalities to Mute/Unmute, send in the RMS metering value in real-time etc and is responsible for doing track specific operations.

Playtest Results

Playtest statistics:

Results:
  • Interface: The experiment of phrasing master Solo as ‘Activate’ and master Mute as ‘Deactivate’ was not easy to understand for half of the playtesters.
  • Interface: Colored buttons on the control panel led playtesters to wonder what the meaning of those colors was.
  • Interface: Function of the controller buttons (B, Y can mute/unmute each track) was not labeled in the VR space – playtesters overlooked the function of muting/unmuting each track
  • Interface: 2D instrument icons in the orbs only appear when pointing, which is not obvious for playtesters to know track instruments before hovering over.
  • Interaction: Volume function was easy to understand and operate for all playtesters. However, 3/13 playtesters expected moving the orb further and closer would present differently. 
  • Interaction: Panning function was easy to understand and operate for all playtesters.

Insights From Professionals

Some thoughts from industry professionals after they watched/experienced our prototype:
  • The orb and track field representation are intuitive to understand.
  • Think about sitting in an airplane cockpit when thinking about the track field view. Everything the pilot and the musician need to look at should be contained in a very narrow space so that they don’t have to look around to find stuff. This is helpful to think about the scale of the field and also what to have and what not to have.
  • The visual differences that show different states of the track orb should also make it possible for guests to anticipate when this track is going to have sound (similar to what the sound wave visual in DAW does).

Lessons learned

The purpose of the playtest was to find out:

  1. Whether the layout and basic control functions are understandable.
  2. If the volume and panning functions were clear and intuitive without instructions.

 

As indicated by our playtesting results, some 2D interfaces and buttons were misunderstood due to unclear labeling:

  • We “rebranded” the mute function from DAWs and called them “deactivate/activate” on the control panel to simplify what “mute” and “solo” do in a DAW. This decision led to a huge amount of confusion given most people are familiar with the term “mute”, even for those who have never used a DAW.
  • The color differences of the buttons on the control panel are misleading. The colors didn’t have specific meanings but the “deactivate” and “activate” uses red and green, which some playtesters relate more to play and pause.
  • Track orbs have no labeling shown unless they are hovered over, which makes it impossible for guests to know what instrument each orb is before selecting them.

 

Despite the confusions on the interface, the functionalities (including volume and panning) in this prototype are easy for most guests to understand and perform. Based on the data we collected from our playtesters, all playtesters were able to figure out how to adjust volume and panning in the track field presented to them, though a couple of them mentioned how they were expecting some different approaches to how these functionalities work. For example, some were expecting the further away the orb is the louder the volume is, which is opposite to what we implemented. Some also mentioned how they were expecting the size of the orb should show volume, and the distance should represent reverb.

Insights From Professionals

  • Have a more sophisticated 2D interface with color accessibility and more thought-through labelings. And consider the reasons behind coloring each button and track orb differently.
  • Having onboarding tutorials within the VR space can prevent unnecessary miscommunication and increase the efficiency of playtesting the prototype.