Marioneta

“Making History Come Alive”

Project Title: Marioneta
Team Members: Hyunghwan Byun, Emily Chang, Maria Alejandra Montenegro, Alexander Moser, Christina Tarn
Faculty Advisor: Shirley Saldamarco, Ruth Comley

The Idea

The Children’s Museum of Pittsburgh houses a collection of beautifully crafted antique puppets, many of them donated by local puppeteer Margo Lovelace. However, the puppets had to be displayed behind glass — too fragile for children to touch or play with. The Marioneta team from the ETC was invited to find a solution to this unique challenge: how can children engage with these puppets in a hands-on way without ever touching them?

The answer they came up with was a playful and intuitive digital installation, it allows young visitors to step into the role of a puppeteer using just their body. Through Microsoft Kinect v2 sensors and a Unity-powered experience, users can control a virtual puppet on screen — bringing it to life through movement, gesture, and imagination.

The Process

The team began by closely studying the museum’s puppet collection, building accurate 3D replicas using custom shaders and high-resolution photography. The puppets’ distinct designs were preserved, but reimagined for digital interaction in a whimsical, seasonal world. Marioneta’s interface was entirely hands-free. Users stood in front of a screen and were scanned by the Kinect, which then translated their motions to a virtual puppet avatar. The team paid special attention to accessibility, knowing that many children and their families would be new to gestural interfaces. Weekly playtesting sessions at the Children’s Museum — nine in total — shaped each design decision. They also wanted to ensure that the experience would continue to be interesting, even after multiple visits to the museum. As a result, they came up with four rotating seasonal scenes, each one unique. In spring, users could ring cowbells in a blooming meadow; in summer, they were able to smash lanterns to release glowing fireflies. In fall, they could hurl pumpkins in a harvest field, and in winter they’d have a virtual snowball fight.

What was key was that these seasonal moments came without explicit instruction. Instead, they invited exploration, and rewarded curiosity with the surprise interaction. And as users interacted with the experience, many of them began to connect the virtual puppets to the real ones in the nearby display case.

This intuitive link between the physical object and its digital representative was the team’s goal all along, and a key metric of success for the team.

The Impact

• Featured at 2015’s SXSW Film and Interactive Conference
• Presented at IEEE Virtual Reality Conference in 2015, where it was praised for its user-centered design and cultural impact

Installed inside the Children’s Museum of Pittsburgh, Marioneta became a hit with its intended audience: children under 13 and their families. Its intuitive interface, imaginative feedback loops, and freedom of movement offered something rare in a museum setting — an invitation to play openly with history.

Marioneta also carried forward a legacy. It was the spiritual successor to Virpets, an earlier ETC project from 2001 that had run at the museum for over a decade. Marioneta modernized the concept with improved tech, refined UX, and renewed storytelling grounded in the museum’s collection.

Marioneta’s work is still on display in the museum today, where visitors interact with it on a daily basis.