It’s the last week of the semester!! We presented on Wednesday and we hope we delivered it well.
Sharing the post-mortem here:
Overview of the project
HotSpot is an ETC project in partnership with robotics company Boston Dynamics. Our team aims to explore and create an entertaining experience for Spot, a nimble robot designed for various functional purposes. We’re building a game to help BD explore the possibilities of using Spot is a unique gaming platform. The feedback could be used to validate that experience was indeed fun and worth people’s time. This would be a part of a growing portfolio of entertainment experiences centered on Spot.
The player is a captain who travels to Planet X, which is the room (the same size as our project room) and the player has a mission to complete by controlling Spot to do a bunch of activities on “Planet X”. In the game, we expect Spot to elicit positive emotions, the player will feel that Spot is helpful as Spot is the captain’s partner and the only Spot can do those activities on Planet X. The game is longer than 15 minutes and takes up to 30 minutes. Those are the main requirements of our client.
Deliverables
- Documentation
- Code
Our deliverable for the project will be, first, documentation that includes a story, game flow, level design, map, props description, art assets, operator console, control scheme, supporting videos, and anything that helps the client to create the game. Second, the code repository with tech documentation for the direct use of our client.
The Team
We are a team of six:
- Producing – Shelley Xiao
- Design/Producing – Amelia Li
- Design/Artist – Arnav Banerji
- Design/Artist – Yoli Shen
- Design/Programming – Yang Lei
- Programming – Tyler Thompson
Our faculty instructors are Carl Rosendahl and Scott Stevens. Our client is Boston Dynamics.
What went well
From the producing perspective
We checked carefully about the client’s needs when the semester started, which really helped us stay on track when exploring different directions. We worked under lots of challenges that we didn’t expect at the beginning – lack of experience in (physical/ location-based game), design challenges, covid restrictions, client requirements and hardware problems. However, we responded to those challenges by learning constantly and adjusting our strategies quickly enough and finally created a game that satisfied most of our client’s needs.
When we had trouble designing a game centered around Spot, an industrial robot not for human interactions, we looked for help from Jesse. Jesse gave us valuable advice, which was to explore fun activities Spot can do. We followed that advice and it turned out to be a great direction. When our first prototype didn’t work out well, we learned from the playtesting results and immediately started experimenting with new ideas. When we heard that we were not able to go to the ETC after Thanksgiving, we planned accordingly by leaving UI, sound feedback, and documentation touch-ups as tasks to finish after Thanksgiving. In order to iterate our work, we held 3 different types of playtests with more than 20 playtesters. We even held a paper prototype playtest virtually, which went well and we gained valuable feedback.
There were lots of examples like this. We kept tackling those problems and learned from feedback without giving up. Finally, we found a great direction that solved the design problems, incorporated our previous experimental outcomes in a fun and meaningful way, and satisfied our client’s needs. We really appreciated all the help from the ETC faculty and the clients.
From the design perspective
From a design perspective, to address our success, We would like to analyze “Spot” a little bit.
We need to work on both aspects of hardware and software to make it possible to generate fun interactive experiences.
Spot is originally designed for industrial purposes. Specifically speaking, it can independently complete tasks in places where it is not human-friendly. Boston Dynamics has done an amazing job achieving this goal. Their promotional videos have clearly demonstrated this point.
But from a game design standpoint, it isn’t necessarily a good news for us.
From a hardware perspective, it is not versatile for developing games.
First, it has no audio. This means we can not directly code in sound feedback.
Second, its camera is black and white. We can’t directly use color as a design element.
Third, even though we want to attach add-on elements to Spot, Spot needs to readjust itself frequently. This means we can only use its back as an attachment point.
From a software perspective, Boston Dynamic offered a very well-developed control system but this is only to control Spot’s movement. They also offered us a new system called “Choreographer” which has not been published yet. If we want to add more flexibility in Spot’s movement, we need to re-write the control system again and combine those two systems together.
Second, Spot has a very strong obstacle avoidance system. This will prevent Spot from hitting medium to big size physical objects and keep Spot balanced. This also is a great challenge for us.
Even though we face those challenges I mentioned above, we still successfully fulfilled the expectation from our client which is to generate more than 5 minutes, ideally 30 minutes interactive experience which gives the guests a positive experience towards Spot.
Every small step we move forward, we have left behind at least 3 prototypes, from game design, to props design, from story design to UI/UX design.
We bypass those limitations and find new ways to achieve our goal.
We use small objects for Spot to interact with so that Spot will not stop or bypass the objects.
We got lots of inspiration from real life, think of what a pet will do that seems adorable, and use those small moments to enrich the overall experience
We connect small objects to control big installations so that the player can get a more interesting experience.
We add extra hardware ( pixycam) on top of Spot so that we can play with color as our design element.
We reorganize our control system and merge the two systems together to make Spot more capable to do complex actions.
We create a background story so that all those experiences can be connected together. This will make the whole experience more smooth.
The design approach was correct. We started by brainstorming all possible ideas about Spot and narrowing it down into the most potential ones. Though we crossed most of them out because of the constraints Spot brought, the process gave us a vision of what we and our client wanted from Spot.
After we’ve done some prototyping and playtesting, we came to the conclusion that physical interactions, fixing players’ attention onto Spot and Spot’s easy but emotional expressions were vital to the success of the experience. We followed those conclusions and that led us to a cool interactive experience that all of us liked.
From the programming perspective
The use of additional hardware besides Spot allowed us to extend what Spot could accomplish by quite a bit. Using Arduinos for communicating with Unity and with each other was a smart move that allowed us to use more devices at once, using an Xbox controller to control Spot allowed for a control device that more players would be familiar with and that could communicate with Unity more easily, and PixyCam allowed us to use colors in a way that we would not have been able to do with just Spot alone. While this additional hardware also came with additional problems (which will be mentioned below), they also came with too many new opportunities to pass upon.
What could have been better
From the producing perspective
We needed to communicate more with the client and the faculty. We reported our “good” progress to them weekly but we missed some opportunities to show them what went bad. It was still valuable to show what didn’t work out and share the reasons for it with the client. It was also important to let the faculty know what “difficulties” exactly are so they would know better about our (individual) work or provide help accordingly.
In addition, when we were under pressure, we tended to solve the problem by ourselves rather than solving the problem together, especially under this “WFH” environment. Hence the producer should’ve taken the initiative to drive more meetings (not just stand-ups or team meetings but also about design & tech) to help the team to cooperate with each other.
Lastly, the producer could’ve conveyed the information more clearly to the team members, especially when not working physically together
From the design perspective
First, if we were not in a pandemic situation, this project would be more fun and our team would have more chances to test errors and make the whole production process faster.
Also, we hold more ideal expectations for our design object “SPOT”. Because we see the promotion video showing how powerful Spot is, we take it for granted that Spot can do complex tasks. Our project goal was too high at the very beginning.
When we try to add more hardware, we’d better put error testing time for hardwares to work seamlessly.
If we’d known from the start we needed to build a larger environment space for our experience, it would have been a good idea to think about the interior design and layout of the room so that it would be an immersive experience rather than just a showcase of what is possible. However since this wasn’t really the project goal, it was given lower priority as compared to actually trying out different ideas in the experience.
Spending some time thinking about or experimenting with the constraints of Spot in the early stage of the project may save us some time in brainstorming and iterating.
From the programming perspective
While the additional hardware we used allowed us to expand the design space we could work with by quite a bit, they also introduced new problems. For example, PixyCam was not as good at detecting color during the game as it was in a controlled setting. As lighting changed and as our environment became more colorful (blue walls instead of white walls, for example), it became much harder to consistently rely on PixyCam. If I were to do this project again, I would either look for a different way to detect color or insist on a play space where conditions could be kept more consistent. Another piece of hardware that presented a possible benefit to the project but ended up creating additional problems was the Xbox controller. While it allowed us to use a controller that more players would be familiar with as well as a controller that could communicate with Unity more easily, it also appeared to make Spot walk more shakily. Compared to the code being used by the Boston Dynamics provided controller, our code paired with the Xbox controller made Spot seem more unstable. However, we would have had great difficulty pairing Boston Dynamics’ controller with Unity and we could not have made Spot to the additional actions from Choreographer with the Boston Dynamics controller, so we had to make do with the Xbox controller and Spot’s shakiness. If I could do this semester again, I would focus more on the controller code to try to smooth out Spot’s walking to be more comparable to the Boston Dynamics controller’s walking.
Lessons learned and conclusion
From the producing perspective
Communication is the key. As we worked remotely, we understand that it might feel a little awkward communicating through Slack or Zoom but it was necessary. The producer should have taken more initiative in helping the team to better communicate and seeking help from the ETC faculty.
Trust each other and don’t give up. Stay calm when facing the challenges. We are a team and we keep trying and believe that we can make it work 🙂
From the design perspective
From the design perspective, in the beginning, we might need to make the turn over quickly so that we can test more basic ideas.
We might need to pay more attention to the use of extra hardware. They can be more affected by the environment than we expect.
Figuring out where players’ attentions are fixed and what got them motivated about the game or the experience can be inspiring in iterating the design.
From the programming perspective
This semester gave us the opportunity to learn more about networking code as it is used by Arduinos, how to code in Python, how to code on Arduinos, and how to use PixyCam. All of these lessons were learned through the non-PC hardware we were working with (Arduino for networking and Arduino code, Spot for Python, and PixyCam for how to use PixyCam). By learning these additional skills and additional hardware, we have expanded what we can effectively code and what we can effectively code on, which will be useful for us as developers going forward. For example, many projects with a physical component to them start out prototyping with an Arduino or a similar piece of hardware, and Python is an increasingly common coding language, so both of these lessons, in particular, could prove useful in our future careers and our future projects.
Final word:
Under covid restrictions, we created a game that meets most of the client’s needs (Spot is helpful, eliciting emotions, longer than 10 minutes, and environmental interactions despite that we didn’t incorporate remote control for remote testing due to technical challenges. We’ll share a design document and code repository with the client at the end of the semester. The client will test out on their end looking at the document and using our code. The project would provide insights into using Spot in-game for Boston Dynamics.
We’ve been experimenting with different directions throughout the semester. We initially came up with an idea of an escape room but at 1/4s we received the feedback from faculty that it’s out of scope as the escape room is hard to build. We switched our focus to the mini-games where we prototyped Gibbering Revelation and tested out Spot dance moves. From the client, faculty, and playtesters, we learned that direct interaction with Spot is preferred than with a computer. Then, we explored environmental interactions and emotional reactions. Finally, we made the whole “Mission on Planet X” game experience.
Alright. We’re about to say goodbye to Spots 🙂