Background
For the second half of the semester, we decided to dive deep into the direction of audio-based experiences. We have two experiences in total, both of which are audio-based experiences. Since we want to explore the possibility of not using the screen at all, we made one of the experiences have UI and the other one doesn’t.
Game Design Details
By making this experience, we want to explore how Tap works with audio-based games. To be accurate, this is audio-only games. Although this game can be played on Android devices, there is no Graphics at all. The ending credit doesn’t count, which would not affect the gameplay at all when removed.
This is a single player game and there is only audio. Basically, we want to create a feeling similar to the old radio shows. The player listens to the story and takes actions at certain key events. There are different kinds of actions types that the player can take, including tap matching, fast tapping, code input, and etc. Sometimes the player also needs to make choices and the choices made by the player will influence the story. By doing these, we hope to make the audio story more interactive and immersive.
Sound Design Details
Sound plays a dominant role in this audio-based experience. Some major topics to keep in mind during the process include having the theme consistent between dialogues and sound effects and painting the picture for the players sound wise.
Since we are diving in the space field, the sound elements in the experience are futuristic. But we wanted to have some familiar aspects to call the players’ attention to reality as well. So we had different whooshing sounds to create a distant feeling, the engine sounds to learn about the environment, and the AI dialogue that somehow connects these two.
In order to paint the perfect picture for our guests, we needed multiple layers of sound and make sure they work well with one another. For instance, we have cruising sounds running in the background to suit both as the ambience and the emotion builder. Now that the players know where they are and what to feel, we have dialogue and sound effects playing at certain times to provide them with more details about the story. The environment sounds at different phases are like the color palettes in an artwork while the dialogue and sound effects serve as linings. Altogether, they build a satisfying experience.
With Tap as the interaction device, sound needs to provide feedback for the players’ moves. In all, there are positive feedback sounds that are really satisfying and “negative” feedback sounds that tell the player that though they did not make the ideal move, we are only reminding them in a kind way rather than punishing them for their choices. The interaction sounds in this experience are responsive and positive.
UI Design Details
An animated logo is designed for the experience. Although the whole experience is based on audio and has non-visual, the brand identity can still become a working part of the storytelling process.
For the motion graphics part, the rotating planet evolves into a galaxy, and then it reveals to be an eyeball of the dinosaur. By extracting metaphor from the story and making them connected, some humorous, unpredictable yet serious feelings are conveyed. In terms of visuals, Hijrnotes is adopted as the main font to mimic handwriting. Rounded corners and curved lines are used for the credit scene to make it look younger and friendly
Implementation Details
Because this is a single player storytelling game, hard coding is inevitable. The remaining problem is how to make it easier to make changes and how to make the project clear so that the maintaining won’t be painful in the future.
Since there are many different states within the game, we decided to use a state machine to implements the game. We first created an abstract class GameState, then we created several actual classes based on the parent GameState class.
This is the implementation of GameState:
public abstract class GameState : MonoBehaviour
{
public enum Category
{
NOTHING = 0,
INPUT_CODE = 1,
CHECK_CODE = 2,
FAST_TAP = 3,
CHOICE_MAKING = 4,
MATCH_CHECK = 5,
TAP_COUNT_CHECK = 6
};
protected bool interactable = false;
[SerializeField]
protected GameState nextState = null;
[SerializeField]
protected AudioSource mainAudio = null;
[SerializeField]
protected AudioSource bgm = null;
[SerializeField]
protected AudioSource endBgm = null;
[SerializeField]
protected float overrideTime = -1;
[SerializeField]
protected bool fadeOutBGM = false;
public abstract void execute();
private IEnumerator waitForSoundFinish(float time)
{
if (overrideTime != -1)
{
time = overrideTime;
}
GameStateManager.Instance.nextGameState = nextState;
yield return new WaitForSeconds(time);
interactable = true;
yield return null;
if (endBgm != null)
{
GameStateManager.Instance.bgm = endBgm;
}
yield return null;
}
public abstract void onTapped(bool[] arr);
public static bool checkTapMatch(bool[] arr1, bool[] arr2)
{
if (arr1.Length != arr2.Length)
{
return false;
}
for (int i = 0; i < arr1.Length; i++)
{
if (arr1[i] != arr2[i])
{
return false;
}
}
return true;
}
public static int checkTapLength(bool[] arr)
{
int count = 0;
for (int i = 0; i < arr.Length; i++)
{
if (arr[i])
count++;
}
return count;
}
}
Each state is independent of each other so that each state only needs to manage its own logic. When a state’s goal is reached, it simply notices the GameStateManager and then the manager decides to go on with the next state. Sometimes there is information that needed to be shared between different states and this is done by using the GameStateManager.
There are 8 concrete classes in our game and they are enough for our implementation.
The GameStateManager manages the common logic of the different states and also the shared information. At the same time, it manages the sound effect of the game especially the background music and the input from Tap.
This is the code for GameStateManager:
public class GameStateManager : MonoBehaviour
{
private static GameStateManager _instance;
public static GameStateManager Instance
{
get
{
return _instance;
}
}
[SerializeField]
private GameState currentGameState;
[HideInInspector]
public GameState nextGameState;
[SerializeField]
private int[] playerCode;
private AudioSource currentBGM;
public AudioSource bgm
{
get
{
return currentBGM;
}
set
{
FadeTo(0, 1);
currentBGM = value;
if (currentBGM != null)
{
currentBGM.Play();
}
}
}
private AudioSource fadingBGM;
private void FadeTo(float volume, float time = 1f)
{
if (currentBGM == null)
return;
fadingBGM = currentBGM;
StartCoroutine(FadeToAsync(volume, time));
}
private IEnumerator FadeToAsync(float volume, float time)
{
float oldVolume = fadingBGM.volume;
float lastTime = 0f;
yield return null;
while (lastTime < time)
{
lastTime = Mathf.Min(time, lastTime + Time.deltaTime);
fadingBGM.volume = Mathf.Lerp(oldVolume, volume, lastTime / time);
yield return null;
}
}
public void notifyFinish()
{
currentGameState = nextGameState;
if (currentGameState != null)
{
currentGameState.execute();
}
else
{
OnScreenLogger.Log("GAME END & CREDIT");
}
}
public void setPlayerCode(int[] code)
{
playerCode = (int [])code.Clone();
}
public bool checkPlayerCode(int[] code)
{
for (int i = 0; i < code.Length; i++)
{
if (code[i] != playerCode[i])
{
return false;
}
}
return true;
}
private float timeFlag = 0;
private void Refresh()
{
if (timeFlag > 5)
{
setControllerMode();
timeFlag -= 5;
}
timeFlag += Time.deltaTime;
}
private void Awake()
{
_instance = this;
}
void Start()
{
TapInputManager.OnTapped += onTapped;
TapInputManager.OnBluetoothTurnedOn += onBluetoothOn;
TapInputManager.OnBluetoothTurnedOff += onBluetoothOff;
TapInputManager.OnTapConnected += onTapConnected;
TapInputManager.OnTapDisconnected += onTapDisconnected;
TapInputManager.setDebugLogging(true);
TapInputManager.startTAPInputManager();
currentGameState.execute();
}
void Update()
{
Refresh();
}
private string _identifier = "";
void onTapped(string identifier, int combination)
{
bool[] arr = TapCombination.toFingers(combination);
currentGameState.onTapped(arr);
}
void onTapConnected(string identifier, string name)
{
_identifier = identifier;
}
void onTapDisconnected(string identifier)
{
Debug.Log("UNITY TAP CALLBACK --- onTapDisconnected : " + identifier);
}
void onBluetoothOn()
{
Debug.Log("UNITY TAP CALLBACK --- onBluetoothOn");
}
void onBluetoothOff()
{
Debug.Log("UNITY TAP CALLBACK --- onBluetoothOff");
}
void setControllerMode()
{
TapInputManager.setControllerMode(_identifier);
}
void setTextMode()
{
TapInputManager.setTextMode(_identifier);
}
}
By doing all of these, we made it very modular and very designer-friendly. It becomes very easy for us to fo the iterations and also very easy for sound designers to directly test and modify.
Lessons Learned
From this particular experience, we have learned a lot. First of all, for an audio-based experience, especially this audio-only experience, audio is absolutely the most important part. To create an immersive world, we need not only an intriguing story but also the rich sound effects. Only by doing this can we make the players immerse in the game.
Another finding is about Tap. Although Tap has very high latency, it doesn’t matter too much for the audio-based games since there are not too many interactions that need very low response time.
Future Possibilities
People listen to the radio all the time, what if there could be interactions for the radio? This would be an interesting area to explore. Within the legal limitations, imagine that you can play some audio-based games when you are driving, which could be some game changer.
Google Home and Alexa Echoes are getting more and more popular, it is expected that we can see more and more games on those platforms. Tap as an input device is actually very suitable for smart speakers since most of the smart speakers don’t have a screen at all.
Appendix
Scripts
AI 1 Dear passenger, welcome onboard! Thank you for choosing Space Y! We are committed to providing the safest near-Earth orbit space experience. This aircraft-type AirBoat 7Q7 is equipped with the latest escape system, even if the spacecraft body has been completely destroyed, it can still ensure that you return to Earth safely. This experience includes near-Earth orbit sightseeing and spacewalking. Before taking off, we will conduct a series of safety inspections. Please be sure to cooperate. AI 2 Please tap all five fingers to activate the simulated gravity system. (The player taps five fingers.) AI 3 Thank you for your cooperation! The spacecraft is carrying out the final safety inspection before take-off. (The sound of pulling out the pipe and turning off the valves.) Effect_pull_pipe_valve_off AI 4 Ready to take off, please tap middle three fingers to release the takeoff command. (The player taps the middle three fingers.) AI 5 Countdown to takeoff, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, fire! (The sound of the spacecraft ignition and launching, the player feels the violent vibration. After 10 seconds.) Effect_spacecraft_launch AI 6 Dear passenger, we are now crossing the blackouts in the atmosphere, and your personal communication equipment may be temporarily unavailable. This is normal and there is no need to worry. (The high-temperature friction sound of the spacecraft crossing the black obstacle zone.) Effect_ out_of_atmosphere AI 7 Dear passenger, we have entered the low earth orbit! If you look to the left now, you can observe the complete blue earth. The first time humans have seen such a scene has been traced back to 117 years ago. If you would like to prepare to walk in space, please tap the index finger and... (A burst of noisy electromagnetic interference interrupted the AI's audio, and the player couldn't hear the specific instructions later.) (The player is likely to pick up some fingers, or not, there would be no effect.) AI 8 Dear passenger, we have encountered some technical problems and temporarily lost contact with the ground tower. We are trying to... (A burst of alerts, the player feels a violent shock.) AI 9 Dear passenger, we have encountered some unexpected situations. The onboard radar detected that an asteroid is flying to the Earth. According to current orbit estimates, it will hit Pittsburgh, Pennsylvania area after 10 minutes. The impact equivalent is... DATA OVERFLOW... Sorry for not detecting it earlier. I am not equipped with an ethics module. Do you need to manually override it? You can tap the thumb to override or tap the little finger to let me deal with it with the optimal calculation. (The player chooses to override or not. If no choice is made within 5 seconds, the default is not to override.) If the player chooses to override: AI 10 According to current estimates, you must abandon the spacecraft and take the escape cabin, I will control the main spacecraft to hit the asteroid, otherwise, the earth will be destroyed. This is the only option. Tap five fingers to activate the escape program. If the player chooses not to override: AI 11 According to current estimates, you must abandon the spacecraft to take the escape cabin, I will control the main spacecraft to hit the asteroid, otherwise, the earth will be destroyed. This is the optimal choice. The escape program will be deployed after 30 seconds, and you can also deploy it with five fingers. (Escape program deployment, the sound of the spacecraft escape cabin and the main body separating. Followed by a huge explosion of the main spacecraft hitting the asteroid.) Effect_escape_program_deployment AI 12 Dear passenger, thank you for your sacrifice! Your main spacecraft successfully hit the asteroid and detonated the fusion engine. The asteroid has been broken into pieces. You have just saved all human beings. On behalf of Space Y, I would like to pay the highest tribute to you! The return route of the escape cabin is being planned, and the cabin service will not be provided in the escape cabin. (A noisy radio contact sound) Effect_radio_contact AI 13 Dear passenger, we may have encountered some problems. All surface control towers cannot be connected. The radar cannot detect any other flying objects. The surface scanning system shows that the displacement has just occurred to the airport. This is really unbelievable, but there is no need to panic. This may be a radar fault. We can force landing on the nearby waters and contact the rescue team. Tap five fingers to activate the landing system. (The player taps the five fingers) (A noisy sound of traversing the atmosphere, followed by a sound of the spacecraft falling into the sea.) Effect_atmosphere_and_sea AI 14 Dear passenger, we have already landed. I have detected a civilian vessel in front and I am trying to contact. (Radio contacting noise...) Effect_radio_contact AI 15 Dear passenger, there are human beings on board, but they claim that they are in a zoo. They cannot help and suggest that we evacuate immediately. I do not understand the meaning of this suggestion, and the proposal cannot be executed. The current location is not recognized, and the GPS does not have any signal. To continue to contact the towers, please tap all your five fingers. (The player taps the five fingers) AI 16 The tower signal is received and the identification code is unknown. The rescue has been requested. (Loud noise of the ship approaching...) Effect_ship_approaching AI 17 Warning! A huge ship approaching is detected, and the image perception program shows that the pilot is a dinosaur... (AI was silent for a while) Effect_awe AI 18 I am sorry to inform you that our spacecraft has gone through a weak wormhole and traveled back in time for a while. We hit the asteroid 65 million years ago. You may have guessed that the asteroid was the one that hit the earth and destroyed the dinosaur species. The ship is approaching. The escape cabin is equipped with a self-destruct system for humanitarian reasons. I will try to communicate on your behalf. If necessary, you can choose to activate the self-destruct system by tapping all five fingers. Once again, on behalf of Space Y, thank you for choosing to take our flight... (Deep sound here?) Effect_deep_sound (Credits)