You've probably already seen Andy Serkis and his performance capture demo for Magic Leap. And sure, the combination of Motion Capture Acting and Augmented Reality sounds compelling but how does really work as an experience?
A few days ago I had a chance to figure it out, thanks to the setting up of Imaginarium Studios in Los Angeles during the LEAP conference. That's right, I met Grishneck and a few of his friends.
After setting up the Magic Leap One, I was instructed to look at a flat screen monitor on which Grishneck was trapped and asked for release. Seconds later, this screen dynamic moved into Augmented Reality, and then Grishneck jumped from that virtual screen to the real table in front of me, while continuing to ask me to bring him home and save him from the obsolescence of Hollywood.
The experience is smart, but since it's on rails, it's something I did not really see before. But Grishneck was just the teaser. Only a few feet away, I was introduced to a motion capture actor wearing a series of sensors that covered a black body. Nearby, a monitor showed the actor as a Magic Leaps astronomer, reflecting each of her movements.
But this television monitor was for spectators only, the real fun began when I put on another Magic Leap One and suddenly saw a new character ̵
With the thumb on the controller of Magic Leap One, I was able to raise the rock creature up to 10 feet scale. I told the actor what I saw and asked her to circle me threateningly. It was great. And it really felt like the creature was towering over me in real space, while the actor followed my every movement in real time.
I can think of a number of ways in which this combination of motion capture and Magic Leap One could be used, but instead of guessing, I talked to The Brown's CEO, Matthew Brown, to find out where he believes that will reach us in the near future.
Next Reality: What is it? everything for? It's amazing, but what are you going to use this device for?
Matthew Brown: It's a content authoring tool. We can offer authentic performances through performance capturing. Unlike the keyframe animation that you see in many traditional VFX movies, our actors are essentially able to articulate digital characters. And as part of the capture process, you can get as close as possible to your cast members of the final version of the performance. We look forward to bringing that workflow and knowledge of performance capture to a new platform.
Next Reality: Interacting with One of Your Performances When I capture actors, I've asked them to pretend we're getting into a fight So if I could see what it would look like to the virtual rock monster in front of me – via the Magic Leap One – it would look like it was attacking me. Is it part of the goal of this line-up to provide actors with more realistic possibilities in addition to digital characters rather than imagining monsters in front of a green screen?
Brown: Yes, that's absolutely possible. And with the technique you just demonstrated, we could have an actor in our studio in London and the viewer can be anywhere on the planet, streaming that character in real time.
Next Reality: I noticed that there was a bit of a delay during my demo.
Brown: That was basically the [event space’s] Wi-Fi. All of this motion data goes into our system and goes into software called MotionBuilder. And then Motion Builder streamed it over Wi-Fi, competing with all the other Wi-Fi that is here to be rendered on the Magic Leap device in real-time.
Next Reality: Can you have, let's say, two or three motion-capture actors who work together and are visible through the Magic Leap One with this lineup
Brown : Sure, that's just a very small [space] set up for LEAP con. Our [space] back in the UK is about 30 feet by about 25 feet and we did shoots where we had up to a dozen people captured at the same time. We work for film, TV, video games, all sorts of projects. For example, we did all the movie sequences for Battlefield 1.
Next Reality: What about Magic Leap-specific experiments?
Brown: I think we're just getting started, using a technology like Magic Leap. We've been testing with other motion capture data we've already shot, and we had up to nine people in one scene [shown through the Magic Leap One].
We would like to experience the Grishneck experience and find out if we can Create a whole universe of characters. The idea with Grishneck is that he is a piece of rejected conceptual art [basedon Lord of the Rings Orcs, but not really on film production] and he did not make it into a movie. Each artist has tons and tons of rejected pieces of conceptual art. We think that there is a very interesting narrative idea to bring these characters into your own world through Magic Leap.