The Pandora Project (long)

The Pandora Project was started in 2008 as a continuation of previous work we have done with interactive dance. The project involves experimenting with current motion-tracking technology for interactive dance. Several software composition environments were created, and experiments were made in collaboration with choreographer Brolin-Tani and several dancers using two different hardware systems: (1) an I-Cube Wi-micro System with four accelerometers attached to a dancer’s arms and feet and (2) the cv.jit (Pelletier n.d.) software library to extract motion data from the input from a digital camera connected to a computer. We have focused on using the same software composition environments with these two very different types of hardware interfaces to reveal their inherent differences, advantages, and drawbacks.

The goal of The Pandora Project was to examine and develop technologies for working with interactive sound in contemporary dance and to test these technologies in an artistic collaboration between composer, choreographer and dancers. Two intensive workshops were held in Aarhus, Denmark during the summer of 2008, where the composer, the choreographer and three dancers experimented with interactive dance.

The first workshop was held June 16-20, 2008. Three preliminary experiments were conducted. Each experiment consisted of two parts, one using computer vision (CV) and the other using accelerometers (AC). The two parts of each experiment were otherwise identical. For each experiment a solo dancer was asked to improvise in dialogue with the choreographer and the composer.
Both systems were configured to detect motion. For the CV system, a camera was placed in the center of the room facing the stage. The camera image received by the computer was divided into six image zones. The object cv.jit.framesub (Pelletier n.d.) was used to report changes between consecutive frames in each image zone. Data from this object was used as a simple motion detector: the greater the change between consecutive frames, the greater the activity level of the dancer in that particular image zone.

For the AC system, hardware accelerometers worn by a dancer were connected to an I-Cube system configured to transmit values representing the average difference in acceleration values over a period of two milliseconds. This data was similarly used for simple motion detection: the greater the difference between acceleration values, the greater the activity level of the dancer in relation to that particular sensor or limb.

The first experiment involved dynamic control of sounds similar to the techniques employed in “Sisters”. The dancer controlled the amplitude and timbre of six sounds. Mapping was simple: increased movement produced greater volume and brighter timbre dynamically. The second experiment involved control of a composition algorithm designed to generate rhythm and contrapuntal melodies similar to the techniques employed in the third section of “Movement Study.” Increased movement produced increased rhythmic activity and melodic activity as well as increased tempo. In the third experiment a motion threshold was defined to trigger drum-like sounds. In the CV controlled experiment, any motion in an image zone that exceeded the threshold would trigger a sound. In the AC controlled experiment, sensor data from any of the accelerometers that exceeded the threshold would trigger a sound. In both cases these triggered events were performed both with and a computer-generated rhythmic accompaniment.

The latency of the two systems was similar. The AC system appeared to respond slightly faster than the CV system, but no precise benchmarking was carried out. It was apparent that adjusting and fine-tuning hardware and software is extremely important regardless of which type of technology is being employed. In fact the similarity of the audio results for the same mapping using these two very different hardware systems was striking when the hardware and software was properly adjusted. Problems with occlusion generally found in camera-based systems were not immediately apparent because image processing was used for simple motion detection. Only the first two of Camurri’s four levels of motion tracking were employed: (1) sensor data; (2) low-level interpretation of sensor data (Camurri et al. 2004). The drawback of having to wear a sensor system was minimal with the AC system because of the small size of the system.

An informal discussion between the dancer, the choreographer, the composer and other workshop participants was held following the three experiments. The focus of the discussion was on how the dancer experienced each experiment and how the observers experienced them. Generally speaking, both dancer and observers found the first experiment to be the most successful. The relation between movement and sound was intuitive and expressive. The dancer had more precise control over the sounds when using the AC system, but the CV system provided a more obvious relationship between visual and audio elements. Despite these differences, both parts of the experiment were found to be technically and artistically viable.

It was generally agreed that the second experiment was less successful, mainly because mapping between motion and compositional parameters was less obvious. The third experiment was a bit more problematic. With the AC system the triggering of drum-like sounds quickly became tedious, especially from the dancer’s perspective. The mapping tended to inhibit the dancer’s freedom of movement, since movement always triggered a sound. With the CV system the interaction was less sensitive allowing for greater freedom of movement but also allowing less direct control.

After the discussion some new experiments were conducted, including experiments using both systems simultaneously with one dancer and with two dancers. In the weeks that followed, the composer and the choreographer discussed the three experiments and worked out a set a four choreographic sketches with interactive sound using some of the techniques that had been explored. A new workshop was held in Aarhus July 30 – 31. During this workshop the choreographer and composer worked with three dancers on four choreographic sketches with interactive sound. These four sketches were inspired by the myth of Pandora. According to this Greek myth, the Gods created the first woman, Pandora, as a “beautiful evil” sent to torment the race of men. She opened “Pandora’s box”, releasing all the evils that plague the world and leaving only hope locked inside on closing it again (Wikipedia 2009). The four sketches were finally performed for video documentation purposes in a theater with a lighting rig. Focus was on interaction between movement and sound but interactive computer graphics projected on the dancers were also incorporated with the assistance of Tobias Ebsen.

In the first sketch entitled Liquid, CV was used with a single camera front stage and the image divided into twelve zones. Each zone had a particular sound and tonality associated with it. The dancers controlled the amplitude and timbre of these twelve sounds, which are all obvious references to water. The sounds and tonalities associated with the twelve image zones were changed during performance by the composer.

In the second section, Who, the audio composition is closely related to that of the first sketch. In this sketch the AC system was active. Only one dancer was wearing the system controlling the sound and computer graphics. The choreography was fixed. All three dancers moved in synch, so that they all appeared to be affecting the sound. The transition between the first and second sketches was imperceptible. Who is a sort of guessing-game. The observer is fooled into thinking that all three dancers are controlling the sound, as in the first sketch. At one point the solo dancer begins moving independently of the others and the “secret” is revealed.

The third sketch was called The Box. The CV system was used with a single camera mounted above the stage and the image divided into twelve image zones. A specially focused red light illuminated one of the twelve image zones near center stage so that it clearly stood out as a bright red square on the stage representing Pandora’s box. The mapping between movement and sound was similar to the first sketch. However, the mapping between movement and audio was much more sensitive for the illuminated zone than for other zones. A loud, violent reaction occurred when a dancer moved into Pandora’s Box. This sketch was an improvisation with three dancers. The basic rule was that they all start in the same image zone stage left and gradually begin to move towards and explore Pandora’s box.

The final section, Fire, used the same techniques as the first section but with the camera mounted above the stage. Sounds evoking images of fire were used. The choreography was fixed, depicting a wild dance of fire.