Working Together : development of synthetic actors , alongside human actors , in rehearsals
T. Meyer,C. Messom
Abstract:While film has long utilised computers to create otherwi se mpossible characters, similar use within live stage performances is relatively uncommon. Even within film, these characters have, until recently, bee n mere puppets controlled by some form of puppeteer. This paper outlines research carried out at Massey University into autonomous synthetic (computer) actors that are capable o f learning through the rehearsal process, alongside human counterparts. In particular, the paper describes a production of Orson Scott Card’s Porcelain Salamander , which utilised synthetic actors for one main role, and various minor roles, as well as four human actors. Throughout the rehearsal proc ess, the synthetic characters refined their performance, through the typical loop of imp rovisation followed by feedback from a director, until their final performanc e was optimal. The synthetic actors utilised visual data (from cameras), audio data (from a central microphone), and were actuated via projected graphical displays and output into the theatre sound system. This paper lightly covers the technical aspects of the lea rning and improvisational techniques utilised by the synthetic actors. It concent rates more on phenomenological analysis of qualitative data obtained through the rehearsa l p ocess, and quantitative analysis of audience reaction to the final performance. These results show that synthetic actors are capable of rapid learning with the very limit ed esting and training time available in rehearsals. The results also demonstrat e the ability for the synthetic actors to contribute, in a limited fashion, to the development of the performance, as human actors do. 1. A History of the Use of Computers in Theatrical Artistic Processes The film and music industries have been extensively using c omputers in artistic processes for decades [22], while theatrical productions ha ve r rely done so. Pinhanez provides a lengthy definition of the term ‘computer th atre’: “[it] is about providing means to enhance the artistic possibilities and experiences of professional and amateur actors, or of audiences clearly engaged in a re presentational role in a performance” [19]. This definition “restrict[s] the usage of the term computer theatre to performance situations (ruling out, for instance, user browsing and story-telling)” [19]. In brief, we are concerned with live, real-time, theatrical performances which use computers in the artistic process. 1.1. Scenery One use for computers in a live stage performance is gene ration of the scenery [24]; this is a logical progression from the use of lighting t o set a scene, and also allows more rapid and dynamic scenery, well suited to generatio ns that have grown up in front on the television, or watching movies, rather than at the theatre [31]. A group at the University of Kansas began testing virtual reality as a tool for traditional scenography [30], and then moved on to using virtual reality i n productions with the production of The Adding Machine in 1996 [27], Samuel Beckett’s Play [25], an adaptation of Arthur Kopit’s Wings [26], David Gustav Fraser’s Tesla Electric in 1998 [28], Machinal in 1999 [29], A Midsummer Night’s Dream in 2000 [32] and Dinosaurus in 2001 [33]. Just as computer generated images are integrated into live action sequences in film, Monsters of Grace , from the Brooklyn Academy of Music in 1998 [11], integrated computer generated scenes generated prior to the performance into an opera. The music for the opera was conducted and performed live for ea ch performance, but the imagery (projected onto a screen) was developed prior to th e performances. 1.2. Remote collaboration The simplest form of remote collaboration is utilisi ng computer technology to allow a number of authors to jointly develop a script for a perfo rmance, as in Stations of Desire [14] and the work of the Plaintext Players [23]. More complex collaboration occurs during real-time, for example the ‘Desktop Theatr e’ [35] performance of Beckett’s Waiting for Godot , Santaman’s Harvest , Spectacled Society , World of Park, and Water Wars[2] . These performances (other than Water Wars[2] ) did not have a live component – audiences watched it on any desktop computer with an Internet connection via chat software. Avatar Body Collision use similar technology, but incorporate a live performer into the performance, as in Screen Save Her , the[abc]experiment [13], s w i m, and BodyBeat . 1.3. Synthetic actors In 1995, Barbara Hayes-Roth suggested that artificial intell igence researchers expand their purview to include characters and actors whose perfor mances are designed to create particular kinds of impressions on their human aud iences [12]. In Play, one of their experiments in virtual scenography, the University o f Kansas group utilised a “piercing spotlight” as a “live, visible agent”, although t e spotlight was still under direct human control. Synthetic actors are similar to digital characters in film, but are (to varying extents) autonomous. These characters rang e from those that only interact with other synthetic characters, to those that intera c with humans as well; from characters that interact with a single user via a com puter to characters that entertain a large audience in a traditional theatre. From 1996 to 2001 Claudio Pinhanez at the Media Lab at M.I.T. pr oduced various experiments in computer theatre, concentrating on compute r, or synthetic, actors. In 1996’s SingSong , four disembodied heads interacted with a human conductor to perform a short musical item in front of a live audienc e [20]. All the characters were completely autonomous, working off a known script, and abl e to adapt to (limited) changes in the performance by the human actor. 1998’s It/I told the story of a computer actor ( It) who tries to keep the attention of a human actor ( I) by presenting various images, generating sounds, and changing lighting. It is an example of a disembodied computer actor, similar to the spotlight in Play [22]. It/I was followed by It, an interactive exhibit, based on It/I [20]. Pinhanez’s actors were able to adapt to temporal changes in the performance, and, to a certain e xte t, vary their performance each time it was presented [21]. The Virtual Theater project at Stanford University aimed t o provide a multimedia environment in which users could fill all the roles assoc iated with putting on a play in an improvisational theatre company, including producer, playw right, casting directory, set designer, music director, real-time dire ctory, and actor. Those roles not filled by the user were filled by intelligent agents. Pro ductions that formed a part of the Virtual Theater project included the Woggles , CyberCafe, The Forest Sauvage , Tigrito, and Master/Servant Scenarios . The focus of the Virtual Theater project was on improvisation in the classical mode of improvisation first practiced by the Commedia dell’Arte during the Italian Renaissance, rat her than performances that were scripted [36]. Ideally, synthetic actors are capable of following a sc ript, but also reacting according to its own role. Reilly and Bates state, “We are not necessarily interested in lifelike or rea listic agents. We don't really care if our characters only have four fingers or are talking animals. The arts have always abstracted away from reality an d created characters that were ‘larger than life’ and that were much more i nt resting than any mere ‘real’ people or animals we might know. We [want ] i eractive versions of these abstracted characters that have been so successful in non-interactive media” [34] Some synthetic characters are limited to interacting with users via the traditional human-computer interface, rather than in a theatrical setting. Mateas’s group aims to develop interactive drama that builds “dramatically int eresting virtual worlds inhabited by computer-controlled characters, within which the user ... experiences a story from a first person perspective” [15]. A key point is hat “it should not just be an interesting demo in a closed door lab, but be experienc d by people in the real world. Ultimately, this is the only way to validate the id as.” [15]. Synthetic actors are also, of course, used outside the t heatrical domain – for example in commentating soccer games [4], agent research [6], and education [18]. Computers have also been used in other ‘creative’ areas, such a s generating music [2; 3; 5] and line drawings [5]. 2. The Porcelain Salamander 2.1. Story Orson Scott Card created the short story The Porcelain Salamander [8] as a bedtime story for his wife, later publishing it in Unaccompanied Sonata and Other Stories [9] and then in Cardography [10]. Although it is not among his most well known storie s, it, one of the briefest he has written, is often judge d to be among his best [9]. The fable describes a young girl, Kiren, whose mother di es in childbirth; her father is so distraught at this that he curses Kiren to never move a muscle in her life, until she loses someone she loves as much as he loved her mothe r. Although the curse isn’t that strong, Kiren grows up barely capable to move, while her father, who regretted the curse immediately afterwards, spends his time searchi ng for a cure. At the start of the story, Kiren’s father meets a str nger on his journey who claims to have a cure for Kiren, and gives her father a porcelain s lamander to give to Kiren. The salamander moves, as if by magic, and can speak (a lthough this is only known to Kiren), but must never stop moving, or will transform to o rdinary lifeless porcelain. Kiren grows to love the salamander – to the point where she loves it as much as her father loved her mother. The stranger then arrives at Kiren’s house, and causes Kir n and the salamander to be trapped behind a wall, which gradually creeps in on them. In K iren’s weak state, she cannot jump out, although she could climb out if she had som ething to stan