It’s 2017 and there still isn’t much online about designing 3D User Interface (UI) for Virtual Reality (VR).
The following is meant to be a synthesis and summary of existing UI’s in virtual creation programs– I want to share what I’ve learned about tool palette UI across the HTC Vive and Occulus Rift platforms.
I’ve played with a few programs and took some notes about UI layout that I thought might be helpful to consider when designing in the space.
I’ve played with a few programs and took some notes about UI layout that I thought might be helpful to consider when designing in the space.
Generally, the non-dominant is typically assigned toolbox and environmental options, while the right is assigned to active tool, and tool option functions. Many of the programs made switching the dominant hand easy (10% of the population is left-handed!), but for the sake of this post, right hand = dominant hand.
Medium by Oculus
Medium is a sculpting and modeling tool for VR. Oculus does a good job at utilizing all the functionality of the controllers. The non dominant controller serves as the toolbox, with tool selection, undo, redo, and layer adjustment through the thumbstick. The home menu and save/load functionality is assigned to the top most buttons.
Scaling is achieved by gripping triggers on both controllers and pushing/pulling them away from each other with a gesture. Additionally, positioning can be changed by depressing both controllers triggers and moving them relative to each other and the virtual canvas.
Unique to Medium, the right controller has a virtual representation of the active tool, tool-specific menu options, and the user may adjust the size of the brush by pulling the thumbstick up or down. Color selection is assigned to one of the top facing buttons, and is chosen by the dominant hand, so as not to make the user look away from their sculpture when choosing color options.
Undo and Redo functions are located on the non-dominant controller, and are accessible with the flick of the thumbstick left or right.
Scaling is achieved by gripping triggers on both controllers and pushing/pulling them away from each other with a gesture. Additionally, positioning can be changed by depressing both controllers triggers and moving them relative to each other and the virtual canvas.
Unique to Medium, the right controller has a virtual representation of the active tool, tool-specific menu options, and the user may adjust the size of the brush by pulling the thumbstick up or down. Color selection is assigned to one of the top facing buttons, and is chosen by the dominant hand, so as not to make the user look away from their sculpture when choosing color options.
Undo and Redo functions are located on the non-dominant controller, and are accessible with the flick of the thumbstick left or right.
 |
| Controller avatars. The active tool is displayed on the right controller, and the tool size can be adjusted by holding the thumbstick up or down. |
 |
| Game menu is accessed through the non-dominant controller buttons. Items are selected by a ray-cast from the dominant controller. |
 |
| The color pallet can be opened by pressing button B on the dominant controller. The tool pallet pops up in front of the dominant controller and color is chosen by pulling the dominant index trigger over the ray-cast selected color. |
 |
| Layers can be accessed through thumbstick down on the non-dominant controller. Selections are made by pointing the dominant controller at the UI. |
 |
| Tools are selected through the radial menu by holding the non-dominant thumbstick upwards and pointing the dominant controller over the tool type desired. |
Quill by Story Studio
Quill is definitely the least intuitive system, with the controllers being represented by a purple square and a sphere the current brush size and color.
The program has a somewhat unique UI selection mechanism– users must “poke” or “touch” elements on the left handed UI pallet in order to select them. There is no ray-cast/pointing system. In my experience, this was a novel and more intuitive system, especially with the haptic feedback provided with virtual button collisions and presses.
Overall, the Quill UI requires a lot of practice and trial and error from the user. Again, it is not the most intuitive system, with much of it’s functionality not even touched upon in the tutorial.
The program has a somewhat unique UI selection mechanism– users must “poke” or “touch” elements on the left handed UI pallet in order to select them. There is no ray-cast/pointing system. In my experience, this was a novel and more intuitive system, especially with the haptic feedback provided with virtual button collisions and presses.
Overall, the Quill UI requires a lot of practice and trial and error from the user. Again, it is not the most intuitive system, with much of it’s functionality not even touched upon in the tutorial.
 |
| The intro tutorial and controller avatars of Quill (purple square and black sphere). |
 |
| UI Load/Save functionality of the left controller, after pressing button X, as illustrated in the tutorial. |
 |
| Paint tool customization functionality on the left hand controller. Buttons are selected by touch gesture, as |
 |
| The color picker in Quill. It seems ill-designed, as the clunky interface is hard to navigate and change on the fly between brush strokes. |
 |
| A closeup of the controller avatars. |
Modbox by Alientrap
Modbox is a VR physics sandbox game for the HTC Vive. There’s a lot of functionality that needs to be available to the user throughout the controllers, and they do an alright job with cramming it all in.
The build pallet is ambidextrous and can be brought up by pressing either controller’s trackpads. Build and Play functionality are separated into two play modes, accessible through labeled controller UI. Teleportation is assigned to the grip buttons, and can be cast in an arc by either controller.
The build pallet is ambidextrous and can be brought up by pressing either controller’s trackpads. Build and Play functionality are separated into two play modes, accessible through labeled controller UI. Teleportation is assigned to the grip buttons, and can be cast in an arc by either controller.
 |
| Modbox’s four-faced UI pallet. |
 |
| Controller avatars in Play mode are hands, with a virtual controller overlaid onto them with button maps. |
 |
| UI menu selections are ambidextrous, they can be brought up on either controller, selections are made by pointing the dominant controller onto the UI of the pallet. |
 |
| Object selection from the UI pallet in build mode. Objects are one out of the four 3D GUI faces of the pallet. |
 |
| Advanced tool functionality (i.e. Paint, Delete), another face of the 3D GUI pallet. |
 |
| Color Pallet functionality. Color is pointed to by dominant controller, and selected with trigger squeeze. Objects are colored by pointed to them with the dominant controller and pulling the trigger. |
 |
| Scale can be adjusted in Modbox through a slider in the Settings UI face of the pallet. |
 |
| Each object has object specific functionality accessible through the radial menu while the object is being held in Build mode. The specific radial function is selected by manipulating the cursor over the designated option via the thumb trackpad. |
Tilt Brush by Google
Tilt Brush is a 3D painting program for VR. As such, there is a great amount of functionality that had to be designed into the controllers.
The active tool is easy to preview with the swish of a controller (a short fading trail of the active brush is always rendered), and the active color represented by the controlled color. Like Medium, Undo and Redo functions are controlled by the left thumb clicks, keeping the right controller functionality purely creative-oriented.
Movement is based on a Teleport function selected in the tools menu, but it should be noted that Tilt Brush makes it very easy to scale and move a 3D creation by depressing both grip buttons of the controller, and moving them relative to the the creation position.
 |
| Waving the dominant controller results in a temporary preview of the active brush that quickly fades away. |
 |
| Brush selection on the left handed pallet. The previous and following pages of brushes can be accessed either by pointing to the left or right arrows on the UI and pulling the trigger, or by swiping left or right on the right hand trackpad while pointing at the pallet UI, an intuitive and quick shortcut exclusive to Tilt Brush. |
 |
| The Color Picker is separated onto it’s own face in Tilt Brush. Color is selected by pointing the right hand controller to the desired color and pulling the trigger to select. This can be a bottleneck in creative workflow design, as the user would have to look at their left hand to choose the next color for their creative design, a two handed input. |
 |
| Tool functionality is separated on it’s own face in Tilt Brush. Note teleport functionality. |
 |
| Scaling is done by simultaneously depressing both grip buttons on the controller, and gesturing them towards or away from each other. Additionally, positioning can be changed by adjusting the controllers relative placement to each other to the virtual space (i.e. moving forward by depressing both grips and dragging both controllers towards the player) |
IKEA VR Experience
IKEA VR is a virtual kitchen experience released in 2016. The app is designed to showcase the potential for VR to be an effective interior design mockup medium, and the UI, although basic, has some cool functionality.
 |
| The UI on the left hand controller is always displayed in a single face. Selections can be made by pointing the right controller at the UI of the left hand pallet, and expanding the UI category out. |
 |
| Scaling functionality of the IKEA VR Experience. UI elements on the pallet can be expanded by pointing at them with the right hand and pulling the trigger. The pallet UI can be fully expanded all at once. |
 |
| Teleportation in the experience is completed by using the controller’s grip input and throwing an arced raycast from either controller. |