Week 2: Ako Goes Exploring!

This week our team dipped our toes into the tech and designs that we hope to use in the following months. We did some demos and prototyping to get ourselves familiar with them. This post will cover the following:

• Let’s play some games!
• Brainstorming agent roles
• The Design Workshop
• Character design research
• The Oculus-Zed-LeapMotion setup
• Speech-Text-Speech pipeline

Let’s play some games!

The first task was to explore table-top board and card games that could be good candidates for our project. We explored games that were suggested by both our advisors and by others. Here are some of the games we tried (more documentation here):

Set
A pattern identification based card-game that can be played by both one or multiple players.

Blueprint
A unique turn-based game where players build buildings based on the blueprint on their cards.

Few more Games

• 

• 

After some discussion and research we concluded that the ideal candidate game for our project would have the following characteristics:

Have simple rules – to facilitate feasible scope for game logic programming.

Have easily identifiable game pieces – Marker like and feature point rich with well defined shapes and solid colors for using computer vision algorithm libraries accurately.

Brainstorming agent roles

Ako is going to be an Augmented Reality agent that helps the players to learn a board game. However, initially, our client did not limit to just an agent that helps you learn but also had other roles (like an opposition) open to be chosen. Once we decided that our agent will be a learning companion, we went on to brainstorm how this agent might converse with a player.

We chose the card game Set as our base use case due to a variety of reasons that matched the criteria listed above. While it initially occurred to us that the game rules are simple and concise, drawing out a flowchart of the possible conversations the AR agent could have with the player helped us get a clear idea of the actual game itself.

The Design Workshop

This week, we also had a Design Workshop conducted by Melanie Lam, one of our instructors here at the ETC-SV campus. We went through different fun rounds of learning to brainstorm design ideas that would help us structure and focus on what we aim our project to be. We also were able to get inputs from other teams that made us think of unexplored and interesting ideas.

The main takeaways of the workshop was that it helped us form a clear definition of our project based on what our team members want it to be. We came up with the following goal.

A fun AR companion that helps you learn a board game.

We believe these values would help us create solid design pillars in the coming weeks.

Character design research

As seen in the goal above, we aim for our AR agent to be fun to hang out with even if the player is well-versed in playing the game. Therefore, we decided that our character will have a unique personality to it which would make it an interesting companion. We researched on a couple of possible ideas (more on the documentation here):

Humanoid Animal Character:
An animal-based character that is capable of human-like movement due to the following reasons:

• Easier to mimic movements of animals.
• More interesting interactions with players.
• Non-facial animations to express emotions.

Clockwise from Bottom: Cube-based characters, Humanoid-Animal characters, Human characters, Robots.

• 

• 

• 

• 

Robots:
Generic friendly robot-like characters that might be easier to animate but more difficult to express emotions. They have also been widely used and might not be able to build a connection with the player.

Human Character:
Human models can definitely make cute and toon-like styles but they are significantly harder to model and animate. It might also be disappointing to a player when its behavior doesn’t match their expectations.

Cube Shaped Characters:
Character design super-imposed on a cube-like mesh could be the easiest way to build a character. However, like robots, it might not be able to have a range of emotions and body movement.

The Oculus-Zed-Leap Motion Setup

We started off experimenting with a Passthrough VR setup. This setup was initially used by a project team (here) last semester.

Pass-through VR is a concept used with Virtual Reality headsets which allow the players to see the world around them within the VR headset. Thus, blurring the boundaries between Augmented Reality and Virtual Reality.

We use the following components in this setup:

• Oculus Rift VR headset
• Zed Camera
• Leap Motion Controller

Zed Camera
The Zed is a stereo camera that is capable of identifying the depth information of the environment it is capturing as well as do spatial mapping. It also consists of 6-Degrees of Freedom Positional Tracking which makes it a good candidate for a camera solution for a VR headset which also consists of 6-DoF. This enables us to setup a pass-through environment in the Oculus VR headset.

Leap Motion Controller
While traditional VR applications make use of a hand controller to facilitate interaction, passthrough VR would make more sense if the interactions were through actual hands. Leap Motion is a device which includes an IR camera that captures hand gestures and allows these gestures to be programmed into the application for various interactions. It also has pre-existing gestures as part of its API.

Other possible platforms

While the above setup sounds ideal for an interactive AR application, we realized that our application’s main form of interaction will be through vocal conversation between the player and the AR agent. We will also be relying on image recognition plugins and capabilities. Therefore, we also plan to research and prototype on one or more of the following devices:

• Microsoft HoloLens
• MagicLeap
• ARKit

Speech-Text-Response pipeline

We looked at couple of speech recognition and response models that are currently available. While every model had some pros and cons to it, we decided to test out the pipeline used by a past ETC project team Charlie. However, we didn’t go with using Cortana for Speech to text as that might not be part of our actual platform.

This involves the following steps:

• Speech to Text from the microphone input using Google Cloud Speech API. We used a wrapper built around Unity for this API(https://github.com/steelejay/LowkeySpeech).
• The above returns JSON file which consists of strings that can be identified using DialogFlow. DialogFlow then returns string responses for user queries.