How a Hyper-actor Directs Avatars in Virtual Shadow Theater (paper introduction)
Short video introduction (1:26)
Computer theater perspective and Hyper-actor
This paper addresses movement quality issues encountered during the creation of the theatrical performance The Shadow.
Working from a Computer Theater perspective as proposed by Pinhanez, we introduce the hyper-actor – an actor playing musical and visual digital instruments to enhance his performance.
The visual instrument used by the hyper-actor to enhance his performance is a troupe of five shadow avatars performing pre-recorded animations in a virtual shadow theater. We wrote and recorded a staging for the five avatars that consists in a score of about 150 cues that blend animations and are triggered step by step by the hyper-actor according to his progression in the story.
Synchronization issue - Idle and Salient animations paradigm
One of the main issues was to synchronize the acting of the 5 avatars with the rythm of the hyper-actor.
We use a linguistic model to organize movements as either idle or salient, and we describe two methods for staging the movements which are played step by step by the hyper-actor.
| Layer 1 shows a fixed timeline with a linking of three combinations of Salient + Idle animations. Layer 2 shows the moving timeline of the hyper-acting. We have to synchronize Cue 2 and 3 with the starting of Anim 2 and 3, that is not possible if we use a fixed timeline as in layer 1 above.. Layer 3 shows the first method that consists in recording long idle action after each salient animation. The idle animation must last the sufficient duration allowing to wait and adjust to the next cue that launches a blending to the next salient action. Layer 4 shows the second method that uses short looping idle animations, instead of long ones. We explain why using short idle animation improves movement quality during the recording session. |
Programmation of idle loops on the shelf
The second method applies idle looping inspired by video game techniques, detailing a way of programming the loops in real time with short pieces of non-looping idle movement.
The following video explains how it works. We first show a short non looping idle animation lasting about 6'' without any post-production effect. In consequence, we see a jump that breaks the avatar presence effect. In the second part of the video, we launch cue 10 that blends the idle animation with itself in reverse mode when it reaches its end point, and blends it again on itself in forward mode when it reaches its start point, as long as needed. When the hyper-actor triggers cue 20, the program blends the idle with the next salient animation. There is no blend with an idle animation when the salient animation finishes, to make clear why we need idle animation to keep avatars in action. Without idle animation, the avatar freezes, that breaks the presence effect.
Video:
Here is a view on the mixing system in Unreal Engine 4 :
Panorama on the recording and programming environment
The recording of the numerous animations has been done with AvatarStaging framework, introduced in this paper during MOCO 2018.
From left to right, we show the 150 cues triggered by the hyper-actor during the performance, then a focus around cue 321, and finally what is programmed inside cue 321.
Origin of the Virtual Shadow Theater and the Shadow Avatars
The concept of the Virtual Shadow Theater in which The Shadow has been realized is introduced in this paper :
Gagneré G. , Ternova A., « A CAstelet in Virtual reality for shadOw AVatar (CAVOAV) », in Proceedings of Virtual Reality International Conference (VRIC), S. Richir Ed, 22-24 April, Laval, France, Publisher: Laval Virtual, ISBN 978-2-9566-2517-9, 2020, pp. 129-136
In addition here are some slides about CAVOAV concept and a video introducing the paper
Artistic results
The Shadow has been performed in Ukraine and France over 12 times, with a warm welcome from audiences. The fact that audiences suspended their disbelief towards the staging of the avatars was the result we hoped. It validated our hypothesis about the salient and idle animations paradigm and the success of its implementation. Here is a 4 mns video excerpt of the performance in French and a short introduction to it with images.
Video :
