Development Blog

 

The show must go on! It’s a hoary chestnut of a showbiz phrase (as well as a mediocre Queen song), but it’s something we’ve been saying all throughout this process as a reminder to ourselves—despite whatever technical challenges we were facing or obstacles that seemed insurmountable, all of work would have been in vain if we weren’t able to present the performance to an audience.

And the show did indeed go on! We had 15 public performances of “Project Neverland,” and we received very encouraging reviews. One of our professors even brought his 5-year-old daughter who had her first experience with the HoloLens and loved it! It’s gratifying moments like those that warm our hearts and remind us that we are laying the foundation for the future of this kind of tech-enhanced performance.

Was everything perfect? Far from it. The clicker reliability issue was never solved; we simply learned to work around it; kudos to Will and Amara who figured out how to improvise during the spots where Tink wouldn’t respond to her next cue. The HoloLenses were not 100% reliable themselves; infrequently they would disconnect from the network and would have to be reset for the next performance. And depending on where each audience member was sitting, the holograms appeared to be in slightly different positions, thus occasionally breaking immersion depending on the effect we were trying to achieve.

But those are all quibbles. There were enough magical moments—whether it was Tink disappearing into and lighting up a dollhouse, flying into the window, or looking Peter right in the eye as they had a conversation—that justified the work we did and proved that AR has a role in the future of live theater. Eventually the hardware will mature, shrink, get more comfortable, and have a more capable battery. We also know a lot more about how holograms are rendered at a distance. Those that come after us (and perhaps ourselves one day, if we’re lucky enough to get the chance) will be better off for the work we did here, having pioneered and experimented as we did.

From all of us at Team TheatAR, thank you for reading this entries and joining us on our crazy journey to bring a dream to life. We’ll see you next time at the theater!

This was our final week of rehearsal. Our last chance to get things in order before we show our project to the ETC faculty (and to the world)!

Truth be told, it was kind of a slog. While everything ultimately came together relatively close to how we wanted it, there were moments of Sisyphean frustration, where it felt like we were pushing the technology well beyond what it was capable of. This manifested in some of the following ways:

1. The HoloLenses were draining power more quickly than they could be charged. Our small team of 6 people had a lot to manage this week, but it turned into an act of spinning plates to keep these devices powered. Even when plugged directly into an electrical outlet, our networked experiences was so resource intensive that by the end of the evening, some of the HoloLenses would be nearly or entirely dead. A bit scary heading into performance week!
2. We’d have long periods during rehearsals where the tech wasn’t cooperating and we had to hold the actors. Will and Amara (Peter and Wendy, respectively) developed a special bond as they waited…and waited…and waited some more while we ironed out problems and could start running the scene again.
3. Our livestreaming/recording solution was not looking great. With our dream of Spectator View (the solution proposed by Microsoft itself for high-quality capture) evaporating because of compatibility issues with out project, we were planning to record and stream directly from the HoloLens’ 720p camera. The result was a pixelated mess that (while low in latency) wouldn’t do our animations justice. Will, at one point trying to send his friends a photo of what he was working on, held his phone up to the HoloLens’ lens and the quality of the capture was good enough to get us thinking…we attempted to use Google Chromecast and an Android phone, and while the hologram quality was much, MUCH better, the lag between what was happening in the HoloLens and what would appear on the monitors was too long for acceptable livestreaming.It turned out, however, that an Apple iPhone captured similarly high-quality versions of the holograms and, when mirrored to an Apple TV connected to the monitors, also produced a low-latency livestream. And thus our streaming and recording solution was born: an iPhone duct-taped inside a HoloLens on top of a(n unused) DSLR camera on top of a tripod. We’ll be explaining all of this more in our final presentation.
4. The HoloLens clicker, our means of advancing Tink’s cues, was proving to be inconsistently unreliable. At random times during our run-throughs, a click would do absolutely nothing and Tink would remain in her previous position/idle, leaving our actors hanging and unsure of what to do next.

Will we fix the dreaded clicker catastrophe? Will all of our HoloLenses die right when we place them on our actual audience members’ heads? Stay tuned for next week, as the curtain finally rises (and falls) on “Project Neverland!”

Short week and short post. Most of us are taking off tomorrow to various destinations for Thanksgiving, so we only have one actual day—today—to work on the project. Most of the equipment has been loaded in, lighting levels have been set, and we’re in the process of figuring out audio. When we get back, we’ll be integrating final set pieces and dressings, props, and bringing over all the HoloLenses for our big tech week.

Happy Thanksgiving, from all of us at Team TheatAR!

There was perhaps no moment more exciting in this project than at last donning the HoloLens and directing not just two actors on stage, but three! At this performance, the role of Tinker Bell will in fact be played by Tinker Bell herself—it’s now possible to guide her around the stage (to predetermined locations, of course) and cue her acting beats in time with her human costars’ performances.

The HoloLens itself is already a few years old which with the way technology progresses these days already feels like ages ago, but directing a holographic actor on stage and seeing her convincingly “converse” with real people and move around and inside objects on stage feels like a milestone in theater/tech history. There’s still a (very) long two weeks between now and when we show this to the public, but there is a strong sense of magic happening that we are using to propel us through the final weeks of development.

Here’s what else happened this week:

Set design/construction: As evidenced from the photo above, we have a set! It’s much more painted (blue) than in that photo now, and there’s a lovely rug that covers the linoleum flooring. Additionally, the final virtual model of the set that gets run inside of the HoloLens has been created as well.

Programming: We’ve now successfully configured the Spectator View Pro setup, which will allow both audience members not wearing the HoloLens as well as our tech team cueing the performance to view the show as intended. We’re mounting two monitors on each side of the stage specifically for these purposes. We’ve also successfully networked multiple HoloLenses together, to maximize the size of the audience that can experience the performance in the optimal fashion—the way we envision audiences of the future viewing performances of this nature.

Art/Animation: Tinker Bell’s animations are really coming along now. She’s zipping around stage, emoting, and generally feels like an actual character. This will only get refined over the next couple of weeks.

Misc. Production: Rehearsals with the actors have been going really well. It’s important to remember that even in the midst of all the technological craziness, there’s a real human story to tell here between Peter and Wendy. It’s quite a touching little scene with a lot of warmth, and we’re spending time to make sure that isn’t lost with all of the tech layered on top of it.

 

Time. The inexorable forward march of progress. Despite our frequent yearnings for scientific devices or even magic that could turn back the clock, it only moves in one direction. Time is our enemy at Project Neverland and we are fighting the good fight against it every day.

The good news is that there are still 3 weeks left before we start performances. The bad news is that there are only 3 weeks left until we start performances. This realization has produced a considerable amount of stress, but we are harnessing it as a catalyst to sprint towards the finish line and deliver an amazing show; a kind of performance the likes of which (as we must occasionally remind ourselves) has never been seen before.

Here’s what else has been happening this week:

Set Design/Construction: We spent considerable time hauling assembled wooden flats, a dresser, a rocking chair, and a bookshelf up a dark flight of stairs to reach our wacky little performance space. All of the major pieces are up there now, in the process of being painted. The walls are all up and they’re getting painted now as well. As you can see from the picture, the window (the one Peter and Tink will enter from into the Darling nursery) has been constructed into one of the wooden walls.

Programming: We made considerable process with the HoloLens networking this week. Unity timeline and animation data can now be synced across multiple devices. This is big! As far as the other major hurdle is concerned, it’s proven to be more than a little complicated getting the livestreaming solution configured. We need this in order for those without HoloLenses on their heads (including the crew) to see where Tinker Bell is on stage at any given moment. Hopefully we’ll get this sorted out next week.

Art/Animation: We’re in full swing with the animating now that the rig is complete. Tink’s basic moveset has been created, which includes: fly forward, fly backward, fly up, fly left, and fly right. She also now has a handful of standby/idle animations for when she’s between cues. These include: angry, happy, and grumpy. We also refined her flight path a bit this week.

Misc. Production: We should have Tink’s sounds completed early next week. The speakers and sound system will be loaded into the space next week as well. We had our first rehearsals in the space with the set and actors this week, and it was a delight to see this scene on its feet. Our actors are essentially already off book (memorized) and this will only help as we begin to play around with the scene and refine their performances when we add Tink to the mix.

In some ways, animating Tinker Bell for this production is like being an air traffic controller. Unlike your average computer animation, you have to worry about Tink colliding with real-world obstacles, including people. While Tink crashing into Peter or Wendy won’t result in a fatal injury, it will decisively crush the audience’s faith that she’s a real character in this world.  To that end, as seen in the GIF above, Tink’s flight path is something that’s being extensively planned out in detail well before the actual animations are layered on to her character model.

Here’s what else has been happening this week:

Set Design: More furniture has been built and more set pieces have arrived for the nursery. In addition the dresser we had already constructed, we now have the rocking chair, the bookcase, and the frosted glass jug Tink will hide in during the scene. All our effects lighting has arrived and they’re ready to be assembled into the completed set, which is being (partially) constructed this weekend and into early next week.

Programming: Work continues on both setting up the Spectator View (which allows anyone outside of the headset to see Tinker Bell) and the networking (which allows more than one HoloLens to share the same experience). We also overhauled our surface detection solution and imported the virtual set into Unity, which is how the entire experience will be able to run.

Art/Animation: As mentioned earlier, Tink’s flight path has been established within Unity. After the final rig is received this weekend, all of the detailed animation can begin in earnest. A sprint to the finish line!

Production: We had our first table read of the script with Will and Amara, who will be playing Peter and Wendy, respectively. It was delightful to hear all of the dialogue spoken by our talented cast and we had a productive discussion about how the unique rehearsal process will work in the coming weeks. Next week we begin rehearsals in the space with the set. It’s like a real show!

 

This past Wednesday our team presented to the ETC faculty and student body, showing off the work we’d  accomplished to date.

The highlight for most was a live demo that showcased the crux of what we’ve been working on—it featured a real actor (in this case, a stand in for a real actor) looking at and talking to Tinker Bell, and her in turn responding, followed by her interacting with a physical prop that will be used in our set.

Proving that we could make all of these elements come together was crucial to the project’s viability and gives us confidence as we push forward throughout the remainder of the project. With the essential problems of the project addressed, we now can focus on creating the rest of the content knowing that it will, at its core, actually work.

Here’s what else happened this week:

Set Design: Lights for both practical effects and for actor cueing have been ordered. Labor has been contracted to build the walls of the set. A layout for where all cueing lights will be placed on set has been created.

Programming: Aside from the borderline-heroic work of getting the live demo to happen, a decision has been made to move away from Vuforia (and its associated image markers) in favor of spatial mapping and manual marker placement.

Animation: With the rig finally ready (thanks, Sahar Kausar!), we’ve been able to block out Tink’s general flight path in Unity, which is the first step to ensure that she’s ready to work with our human actors. Next week the rig will be updated to include cleaned up weight painting, finger controls, facial expression blenders, and adjusted wing control.

Misc. Production: We are working with a sound designer (Anthony Stultz) for Tink’s sound effects and “dialogue,” so we met with him and had a conversation about what we hope everything to sound like and how we’ll integrate them into the show. We also met with our actor playing Peter (Will Harrison), to get his measurements for his costume as well as onboard him into the whole process of how this is all going to work.’

Finally, we updated our production schedule. Wish us luck!

 

We’re many things over at team TheatAR, but probably least among them are furniture builders. Despite all of us having assembled IKEA furniture in our pasts, none of us were quite ready for the hell that was assembling the dresser we ordered off Amazon. We grabbed our tiny little Allen wrenches and got to work, however, and many hours later, we totally built the dresser for the Darling family nursery! In retrospect, we’re not sure why we didn’t take a photo of the completed thing; that would have been pretty cool to show right about now.

Some updates for the week:

Set design: We’ve acquired DMX lights to use for the practical effects where Tinker Bell lands on various objects around the nursery. We’re in the process of programming them now and should have one to show off next week as part of our halves presentation. We built the aforementioned dresser. We determined what materials we’ll need to construct the set after halves and have a full design ready to go.

Programming:

We’ve made some massive strides towards successfully networking the HoloLenses together. We’ve verified we can see the same character art in the same location in at least two devices. Now we have to make sure they can handle the large amounts of animation data we’ll be throwing at them.

Digital Art/Animation:

The rigging of our Tink model is nearly complete. Next week, we’ll be highlighting the process that went into creating this rig and featuring our guest artist, Sahar Kausar, who is handling the rigging process for us. In the meantime, we’ve done some work on establishing Tink’s flight path and tweaking her model textures.

Rehearsals with our actors begin in just a few short weeks. Will we be ready for them? Stay tuned to find out!

 

 

This week, visiting scholar Anthony Daniels dropped in to have a look at what we’re working on. As a performer who has quite a bit of experience acting alongside imaginary/invisible co-stars, his input was invaluable as we attempt to translate these well-established film-acting principles to the stage. With our Peter and Wendy now officially cast, we’ll be meeting with our actors next week to get their measurements for costumes (naturally) and (less naturally) for the purpose of creating digital representations of them in our AR world.

Here’s what else has been happening this week:

Set Design – A formal ground plan of our set was drafted that places all of the set pieces/furniture in their intended locations. This will allow us to digitize the set as well, enabling the programming magic that powers the AR effects to happen. Additionally, we taped out the dimensions of our set/playing area in the actual theater space, which does wonders in making everything feel more real.

Programming – We’ve spent most of the week working on networking multiple HoloLenses together to ensure that more than one audience member can experience the performance at the same time. We looked at a sample project where two people wearing HoloLenses could inhabit the same AR world an interact with a few augmented objects. We’re now in the process of inserting our Tinker Bell into such a shared world and seeing how it looks.

Additionally, our project would be greatly aided by the ability of non-HoloLens wearers (both audience members and crew) to see what Tinker Bell is doing on stage. To that end, we’re looking into a solution called “Spectator View,” which essentially bolts a HoloLens to a DSLR camera and streams out to external monitors. Setting this up requires us to use a 3D printer to make some kind of mount piece, so that’s always a fun experiment. We’ll be getting this set up hopefully by next week. Take a look at this weird little rig in the meantime:

Script/Story – We completed rough storyboarding/actor blocking, and made some edits to the script that came out of that process. This will help both our cast members and the animation/programming process as they begin to place Tinker Bell in the world!

Animation – By Monday, we’ll have a completed rig of our Tinker Bell model that will be ready for some animating! Our crack animation team has been poring over reference videos and creating plans based off the storyboarding/blocking we created.

Next week we’ll have updates on our networking process, reports on meeting with our (human) actors (!), and preparations for our mid-semester halves presentation the following week.

 

And we’re off! With our choices of story and hardware decided, we’ve moved into the production phase of the project. Even for such a short scene, we have our work cut out for us over the next 9 or so weeks. Our first step was to develop a schedule for the remainder of the project that includes deadlines for set design, prop purchasing, load ins, and costumes. With those established, each department is able to move forward on their individual tasks.

Script/Story: We’ve been delving into the nursery scene from Peter Pan and deciding exactly which elements we want to include.

We intend to stay as close to the original Barrie text as possible, but we are making some cuts where we can to cut down on time as well as to keep the focus of the audience experience on the AR moments. With a play as iconic as Peter Pan, this has proved to be an interesting challenge—practically every line is iconic from this scene and its beats are well known by many potential audience members. Ideally, a balance will be achieved wherein the scene is the right length for our needs, puts the spotlight on Tinker Bell, and feels completely coherent and seamless to the audience.

Technology/Animation: Our programmers and animator established a pipeline for the workflow on this project, involving creating the animations in one piece of software, exporting it to the game engine that will actually run the show in the AR hardware, and then having it run on demand as cue-able mini-sequences. The “timeline” feature in Unity has been tested as the facilitator that will make all of this possible.

Set design: Our designers are in the final stages of drafting a full set schematic that displays the precise location of walls, furniture, and props. Pending approval of our formal budget, we will begin to acquire the actual items we’ll be using on stage and start to modify them for our effects.

Next week we move on to creating the storyboards that will place Tinker Bell in the scene, and allow the actual animating and positional coding to begin!