This article documents some technical challenges we faced during the semester, and how did we solve them. It can also serve as a beginner’s guide for coding with the Tap device.
Tap + Unity
Tap provides open-source SDKs for multiple platforms, including iOS, Android, Windows, and Unity. You can find them here. We chose to develop our prototypes in Unity because we saw more possibilities with Unity, especially for its power of creating the AR/VR experiences.
The Unity plugin for Tap can be downloaded here.
Once you have downloaded the Tap Unity plugin, you can unzip it, double click “tap-unity.unitypackage” to import it into Unity. After that, you will see a folder named “TAP” under Assets. Go to Tap -> Plugins -> TAP, you’ll see a script “TapInputTest.cs”, this script has all the stuff you need for coding with Tap.
Look inside, it has five methods that are very important for us:
- onTapped(string identifier, int combination)
- onBluetoothOn()
- onBluetoothOff()
- onTapConnected(string identifier, string name)
- onTapDisconnected(string identifier)
No need to worry about the parameters right now. All we need to do is simply putting the actions we want the app to take into the body of these methods. And add “TapInputManager.[MethodName] += [MethodName]” to the Start() method, just like the sample script did.
Now create an empty gameobject in your scene, drag “TapInputTest.cs” into that game object, then you’re all set. Easy, right? The methods in your script will be called when you turn on/off your Bluetooth, connect/disconnect with Tap, and tap with the device.
The current version of Tap Unity plugin only works on Android and iOS platform. So if you want to test your app, you need to build it and run on your mobile device. The best way to see debug message is to create a text UI in the corner of the screen and put the debug message on the UI.
onTapped(string identifier, int combination)
When you tap with the device, the onTapped method will be called. Meanwhile, a string and an integer will be passed as inputs.
Each Tap device has a unique string identifier. When you connect, disconnect or tap with the device, the identifier will be passed as a string parameter, so we can identify which device is calling this function. More information about the device’s identifier could be found in the API documentation here.
The integer “combination” tells us the combination of fingers one is using when tapping. It is an integer, but we can call the “static bool[] toFingers(int combination)” method the plugin provides to convert the integer into an array. This method will return an array of boolean value with a length of 5, representing the five fingers we can use.
After the conversion, if we want to know which fingers are tapped, we can just use the returned array and see if the value in the matching index is true. If it’s true, then that finger is tapped. Index 0 to index 4 represent fingers from thumb to the pinky.
- Arrar[0]: thumb
- Array[1]: index finger
- Array[2]: middle finger
- Array[3]: ring finger
- Array[4]: pinky
So for example, if we want to do something each time the thumb is tapped, we can write this code:
void onTapped(string identifier, int combination)
{
bool[] arr = TapCombination.toFingers (combination);
If (arr[0])
{
Debug.Log(“The thumb finger is tapped.”);
// Do something...
}
}
Tap + Unity + Windows?
As we mentioned before, the current tap plugin for Unity only works on Andoird and iOS. What if we really, really want to make a windows application with Tap using Unity? Can we do it?
Sure we can. In fact, think about what tap really is? A wearable keyboard! When you connect it to Windows via Bluetooth, you can type any character on your computer with Tap.
That makes things much easier. We can just treat it as a normal keyboard, and use our old friend Input.getKeyDown() in Unity. You will get the exact same behavior as you are using a real keyboard.
You can see the default Tap Map here. So when you want to detect a certain combination of finger tapping, just call Input.getKeyDown() with its matching character. Or you could set up your own map for tap and create a more proper map version for your WIndows application. The way to build your own tap mapper can be found here.
Tap + CAVE
CAVE stands for “Cave Automatic Virtual Environment”. It’s one kind of virtual reality environment, but instead of using a headset, it uses three or more giant screens that surround the guest to create an immersive experience. During the first half of the semester, we built a Tap + CAVE experience called “Utapia”. You can read more about it here.
CAVE runs on the Windows platform, so you can just follow the instructions from the previous section. However, during our development process, we found some issues worth addressing.
The best way to connect Tap to Windows is by using a Bluetooth adapter. After plugging the adapter into the computer, go to windows settings and find “Bluetooth & other devices”, click “Add Bluetooth or other devices”. Turn on Tap by pressing the button on your thumb finger, you’ll find a device called “Tap_XXXXXX” in the device list, click it and it will connect Tap to your computer.
In the CAVE, however, when the guest goes inside the CAVE wearing Tap, it’s very easy for the Bluetooth to lose connection with Tap. The distance between Tap and the Bluetooth adapter is a factor, also there are many metal frames in the CAVE, which further weakens the connection.
Our suggestion is to use a USB extension cable for the Bluetooth adapter. Put it in a place where it’s close enough to build a robust connection between the computer and Tap, but not too close to interrupt the guest experience.
Multiplayer with Tap
In our brainstorming process, we thought it would be cool if we can make a game where two people can play against each other with Tap, so we decided to make some multiplayer games.
As we mentioned in the previous sections, when you connect, disconnect, or tap with the device, a string named “identifier” will be passed to the app. Each device has a unique string identifier, and we can use that to implement our multiplayer games.
One of our prototypes is a two-player biking game, each person wears a Tap device and controls a bicycle. The faster you tap, the faster your bicycle will go. Whoever reaches the destination first wins. You can see the video of this game here.
What we did in the scripts was, we used two private strings to store the identifiers of the Tap device. We set their values in the onTapConnected(string identifier, string name) method, store the identifier of the first connected Tap into _identifier1, and the second into _identifier2.
In the onTapped(string identifier, int combination) method, we used the if statement to determine which player is tapping. If the identifier passed equals _identifier1, we move the player1’s bicycle forward, if the identifier passed to the method equals _identifier2, we move player2’s bicycle forward.
Here is our code:
public class RunningManager : MonoBehaviour
{
// use two strings to store the identifiers of the Tap devices
private string _identifier1 = "";
private string _identifier2 = "";
// two GameObject that players control with Tap, in this case, two bicycles
public GameObject player1;
public GameObject player2;
void Start()
{
…...
}
void onTapped(string identifier, int combination)
{
// if player1 is tapping, move the bicycle1 forward
if (identifier.Equals(_identifier1))
{
Vector3 currentPosition = player1.transform.position;
player1.transform.position = new Vector3(currentPosition.x + 0.2f, currentPosition.y, currentPosition.z);
}
// if player2 is tapping, move the bicycle2 forward
else if (identifier.Equals(_identifier2))
{
Vector3 currentPosition = player2.transform.position;
player2.transform.position = new Vector3(currentPosition.x + 0.2f, currentPosition.y, currentPosition.z);
}
}
// store the identifiers of the connect Tap devices to our strings
void onTapConnected(string identifier, string name)
{
if (_identifier1.Equals(""))
{
_identifier1 = identifier;
}
else if(_identifier2.Equals(""))
{
_identifier2 = identifier;
}
}
// Other methods
}
Finger Distinction with Tap
We made several prototypes using Tap and touch screen. We found that with Tap, we can achieve something that touch screen alone cannot do: finger distinction on the screen. We can now tell exactly which finger is used to touch the screen.
Here is one of our prototypes. As you can see in the video, when the user presses different fingers on the screen, different colors will show up. The thumb is green, the index finger is purple, etc.
Since Tap provides the information of which finger is down, adding it with the robust touch function of a touch screen shouldn’t be difficult. However, given that Tap has .18s latency problem, the challenge for combining them together is to synchronize the Tap input and the touch input.
To achieve this, we organized a stack to keep the record. As expected, the touch input will be detected first, with the tag of the touch’s state of “Began”, “Moved”, “Stationery”, “Ended”, we keep all the touch input recorded. When the Tap input is received, we can map it to the touch input by go through the latest touch input with possible time (Tap is not accurate so the touch input’s count may not equal to Tap input’s count). In this way, the two input event get combined together.
What can be done via this function is that it also extended Tap’s application. Tap can only detect touch down event which means no “Move”, “Stay” can be detected. With touch screen, now whenever a “Moved” touch input is received, it can work as a Moved Tap input as well, which brought more possibilities.
We packed it up with Tap plugin together, the link for download the unitypackage is here. The usage for the package can be view from the example in the package. You just need to override the 4 functions in “TouchTapTest.cs” to achieve the function you want.
- OnTouchTapBegan(TouchTap touchtap)
- OnTouchTapStay(TouchTap touchtap)
- OnTouchTapMove(TouchTap touchtap)
- OnTouchTapEnd(TouchTap touchtap)
The TouchTap is a class that encapsulates the Touch & Tap input.
//Data Structure for a touch tap input, which contains the identifier of the tap device, position of the current touch on screen, state of the touch input, index of the finger, and the gameobject assigned to the touch tap input if required. The gameobject can be modified with function UpdateGameObject(), the position and state will automatically updated with function UpdatePosition() and UpdateState().
public class TouchTap
{
// identifier of the tap device
public string _identifier;
// position of the current touch on screen
public Vector2 _position;
// state of the touch input, detail can be view from below
public FingerState _state;
// index of the finger
public int _index;
// the gameobject assigned to the touch tap input if required
public GameObject _object;
// construction function
public TouchTap(string identifier, Vector2 position, FingerState state, int index)
{
_identifier = identifier;
_position = position;
_state = state;
_index = index;
_object = null;
}
// call to change the binding gameobject
public void UpdateGameObject(GameObject gameObject)
{
_object = gameObject;
}
// automatically called to update the position
public void UpdatePosition(Vector2 position)
{
_position = position;
}
// automatically called to update the state
public void UpdateState(FingerState state)
{
_state = state;
}
}
public enum FingerState
{
Begin,
Stay,
Move,
End
}