ETC’s Physical Presence Pet Redefines VR Companionship

Physical Presence Pet (PPP), a project developed by a team of ETC students, began with a simple idea: making a VR pet that could substitute for the real ones they’d left behind to come to Carnegie Mellon University. What they ended up creating was much bigger: an unprecedented experiment in virtual companionship that cut across disciplines and anticipates new ways of integrating technology into our world.

For students at the ETC, technology is integrated into their work from the start. Students are introduced to cutting-edge immersive technology their very first week: virtual reality (VR) headsets are standard, and immersive extended reality (XR) experiences that combine the digital world with our physical one are a constant topic of discussion.

The members of PPP wondered what they could do with them. “We started thinking about the idea of a virtual pet ... one that would be a companion for you while you do work or go about everyday life,” Zheng said. “Because when we came to the ETC, many of us moved here from another country or into an apartment that didn’t allow pets — it was hard to take them with us.”

They pitched it to a committee of ETC faculty members, including ETC Director Derek Ham. When he heard their initial pitch, he was surprised. He’d proposed something similar himself, and had even begun prototyping it. “He took the head off a teddy bear, and put a Meta Quest controller in its neck,” said first-year student and PPP technical producer William Zhang. “Then, he made a VR teddy bear head that moved when you touched the controller.”

“I wanted to explore what happens when your eyes see one thing, and your hands feel another. It was about hacking your senses,” Ham said.

With the team’s agreement, they merged their ideas with Ham’s. “The tactile plush element became a big part, and that really made it different from any other VR project,” Zhang said.

The Physical Presence Pet team looks on while a guest playtests their project

Soft Technologies, Hard Problems

What followed was months of experimentation, troubleshooting and collaboration.

Early testing revealed a major issue: when embedded in a plush, the Meta Quest controller wouldn’t track properly. That challenge sparked an unexpected pivot to a new technology: the Apple Vision Pro.

The Vision Pro’s gesture and eye tracking eliminated the need for traditional controllers, and its ability to overlay digital onto physical spaces opened new possibilities. “No previous project has used the Vision Pro,” Zheng said. “We wanted to find out what it could do.”

They named the creature they planned to bring to life Luceal and crafted a plush version of her out of soft, white fur. Inside, she was packed with handmade sensors the team learned to build with a little help from IDeATe Teaching Professor Olivia Robinson.

“I showed them how to create custom soft sensors and textile circuits using conductive fabrics, yarns and threads. E-textiles were new to the team, and they were excited to create custom soft sensors that could be subtly and smoothly integrated into their stuffed animal form,” Robinson said.

By embedding them in Luceal, users could pet and squeeze the plush as much as they wanted without damaging it. “We created the exact same function as commercial variable resistors, but with the flexibility of cutting them in any shape we want,” said first-year student and PPP programmer Brian He.

But the most complex challenge the team faced was aligning the digital and physical pets in space — their self-described “holy grail.” “If people see a virtual thing floating in space — and the physical plush is somewhere else — they’ll want to touch the virtual one,” said Zhang.

For much of the semester, the digital and physical Luceals were separated due to the Vision Pro’s restrictions on spatial data and the imprecise nature of tracking hands in motion. A last-minute idea from Distinguished Professor of the Practice Jesse Schell turned out to be the key.

“The students’ ideas were way ahead of what the technology could handle,” Schell said. “They couldn’t find a way to reliably track the position and orientation of a stuffed animal. I’d suggested to the team that they should look at tracking what they could track, and in this case that ended up being the player’s hands. They embraced that idea, and they found a clever way to overlay the virtual pet onto the real one.”

With only a single week left, the team made their holy grail a reality.

Brian He troubleshoots the Apple Vision Pro in between playtesting sessions
 with help from Jerry Zheng, Paige Li, and Sophie Huang

A Team Effort, In All Senses

Physical Presence Pet required every team member to work across multiple disciplines — from user experience (UX) research to game development to spatial rendering. “It combined art, code, physical manufacturing, embedded systems, electronics,” said He. “All of them had to be present for the final product to work.”

Each team member brought their unique specializations and faced unique hurdles. First-year student and PPP technical artist Paige Li had to work around the inherent limitations of new technology. “What I saw when developing on my computer wasn’t the same on the Vision Pro — colors and scale looked completely different. I had to find a lot of workarounds to get the visual effects to work.”

For Joy Lim, sourcing the fabric posed a huge challenge. “After we decided on one, we found out it was out of stock almost everywhere,” Lim said. “I ran to three different fabric stores and luckily I found the last eight and just grabbed all of them. And then I was terrified every time I cut the fabric, because we had a limited amount.”

There were also challenges no one could have expected. Halfway through the semester, Zheng was rushed to the hospital by fellow team member Brian He. After emergency surgery, it was unclear if he could continue.

“We scheduled many, many meetings with Derek and other staff about how we should handle it,” said second-year student and PPP assistant producer and animator Sophie Huang. “But ultimately, we still figured things out in time for our midterm presentation that week.”

Zheng recovered, and he came back to work ready to contribute —- still managing to design a special Luceal mobile app they showed off in their final presentation.

The Physical Presence Pet team tests their game with the poster in the foreground

Luceal Lives On

Though the semester is over, Physical Presence Pet continues on.

The mobile app Zheng designed allows users to customize Luceal’s appearance and take selfies with her. “With XR, it’s so hard to share what you see with other people,” he said. “What if you just want to show off your Luceal?”

And while no formal post-semester plans are set, the team is optimistic and discussions about next steps have already begun. “I think there’s some interesting potential to extend it during the summer,” said He. “There might be companies out there who’d be interested in this hybrid of physical and virtual pets.”

“This is a perfect entrepreneurial product in my mind,” Ham said. “And now they’re at the point where they’re asking ‘Is this a product? Is this a company? Is this something you develop and try to sell to a toy company?’ They can talk about it — quite literally — as a project with multiple touch points.”

“The project was a wonderful exploration, and I feel sure that five or 10 years from now, stuffed animals with augmented reality overlays will be a popular toy,” Schell said. “My favorite thing about the ETC is how the students are always showing us concrete visions of the future.”