This paper addresses movement quality issues encountered during the creation of the theatrical performance The Shadow.
Working from a Computer Theater perspective as proposed by Pinhanez, we introduce the hyper-actor – an actor playing musical and visual digital instruments to enhance his performance.
The visual instrument used by the hyper-actor to enhance his performance is a troupe of five shadow avatars performing pre-recorded animations in a virtual shadow theater. We wrote and recorded a staging for the five avatars that consists in a score of about 150 cues that blend animations and are triggered step by step by the hyper-actor according to his progression in the story.
One of the main issues was to synchronize the acting of the 5 avatars with the rythm of the hyper-actor.
We use a linguistic model to organize movements as either idle or salient, and we describe two methods for staging the movements which are played step by step by the hyper-actor.
The second method applies idle looping inspired by video game techniques, detailing a way of programming the loops in real time with short pieces of non-looping idle movement.
The following video explains how it works. We first show a short non looping idle animation lasting about 6'' without any post-production effect. In consequence, we see a jump that breaks the avatar presence effect. In the second part of the video, we launch cue 10 that blends the idle animation with itself in reverse mode when it reaches its end point, and blends it again on itself in forward mode when it reaches its start point, as long as needed. When the hyper-actor triggers cue 20, the program blends the idle with the next salient animation. There is no blend with an idle animation when the salient animation finishes, to make clear why we need idle animation to keep avatars in action. Without idle animation, the avatar freezes, that breaks the presence effect.
Here is a view on the mixing system in Unreal Engine 4 :
From left to right, we show the 150 cues triggered by the hyper-actor during the performance, then a focus around cue 321, and finally what is programmed inside cue 321.
The concept of the Virtual Shadow Theater in which The Shadow has been realized is introduced in this paper :
Gagneré G. , Ternova A., « A CAstelet in Virtual reality for shadOw AVatar (CAVOAV) », in Proceedings of Virtual Reality International Conference (VRIC), S. Richir Ed, 22-24 April, Laval, France, Publisher: Laval Virtual, ISBN 978-2-9566-2517-9, 2020, pp. 129-136
In addition here are some slides about CAVOAV concept and a video introducing the paper
The Shadow has been performed in Ukraine and France over 12 times, with a warm welcome from audiences. The fact that audiences suspended their disbelief towards the staging of the avatars was the result we hoped. It validated our hypothesis about the salient and idle animations paradigm and the success of its implementation. Here is a 4 mns video excerpt of the performance in French and a short introduction to it with images.