"danceSense" Integrates Wearable Sensor Technology Into Zodiaque's Spring Program

Zodiaque Dance Company (ZDC) is UB Theatre and Dance’s (THD) in-house pre-professional dance company, celebrating its 49^th season with a spring program which includes diversified choreography in jazz, tap, ballet, modern, commercial, and contemporary dance works performed by current UB dance majors. The show is under the co-direction of faculty members Kerry Ring and Michael Deeb Weaver.

One of ZDC’s goals is to provide opportunities for students to engage with guest professional choreographers and designers, as well as new theater technology. The spring production is unique in that it features a transdisciplinary faculty research project with the UB Department of Media Study (DMS) and UB Arts Collaboratory called danceSense, exploring the dynamic between movement, music, data, and theatrical design in a new work of concert dance featuring dance students Lily Colligan, Abby Hankinson, Juliana Hassouna, and Fallon Tuholski.

In a four-section dance piece choreographed by THD Clinical Associate Professor Kerry Ring, lighting, projections and sound are triggered by signals generated from the movements of four student dancers. The technology was developed by DMS Assistant Professor Jason Geistweidt, who has long been intrigued by electroacoustic composition and sonic arts. Geistweidt enjoys engaging in generative and experimental approaches to formulate performance experiences.

Geistweidt constructed contact sensors which have been sown into the gloves and shoulders of the dancers’ open-backed unitard costumes to collect ambient activity from their motion, which is then sent to an array of computers to direct associated banks of sound and visualizations in the theater. Chance operations come into play as the computer processors re-contextualize the motion into images and sounds, creating unique, ephemeral events.

“I teach courses in physical computing, working with sensors and microcontrollers,” Geistweidt said. “I also teach courses in programming media systems, to think about how to take different types of media-whether it be video, audio or movement, or whatever happens to be the data (set)-and get it to coordinate into live performances. I’m also classically trained as a composer and mostly work in electronic music.

“I’m really interested in the idea of ‘intermedia,’ that everything is sort of interconnected. Zodiaque is a huge, multimedia type of event. It’s like an army, including stage managers, crew, performers, and is very hierarchical in that sense. The data that’s coming from the motion of the dancers and their placement in space on the stage directly affects the sounds that come out.

“Working with Kerry (Ring), we talked about choreographing in such a way that things sometimes freeze and stop, which you see in the opening section. And when the dancers stop, the sound stops. It kind of reverses the hierarchy of how you might think things get built. The choreographer sets up the choreography but it's the dancers who set up the sound and pacing of the video, augmenting what the lighting is capable of doing.”

Originally Geistweidt and Ring hoped to place sensors all over the body, but they found it was simply too much information and his computers couldn’t process it without crashing. “They were totally suited up with on-heel accelerometers and temperature sensors, but the machines kept overloading,” Ring explained. “So it’s really only the wrists we’re working with-rotation and bend and flex. That’s still a lot of information. We didn’t restrict the movement at all. The dancers had full range so we had to minimize the amount of sensor data (instead).”

“For this program the dancers wear gloves which have three-axis accelerometers on the back of each hand,” Geistweidt added. “You can see them in performance because they glow green; they have a green light on them which we kept so that you can see the technology. Wires go up their arms to a pack on their back which contains a microcontroller, which is a small computer, if you will. Then there’s a wireless radio on top of that and receiving antennas on stage.

“From there the data goes up over Ethernet to the sound booth and then to my computer. We have our own sound, lighting and video system separate from the system that’s in the Center for the Arts Drama Theatre. We’re using a protocol called OSC, Open Sound Control, developed at Berkeley some time back, to push everything. It just so happens that a program called QLab, which is used to run and affect video and lighting, speaks OSC, so the dancers are streaming OSC off their bodies, which is just this data from zero to 50 in two dimensions. It represents a closed system which the dancers are activating. The choreography provides the impulse which makes it go forward.”

“We began the project in early 2023,” Ring said. “Jason put out an initial email to our dance faculty that he was interested in doing a project. We did a Zoom meeting while at the same time an interdepartmental grant proposal was in play from the College of Arts and Sciences. We realized this would be an opportunity for us to work collaboratively and we presented the idea of incorporating it into Zodiaque because we have a pre-existing production machine at Theatre and Dance that the project could fit into.”

In spring 2023 Ring choreographed new test movement on some of her dancers in the wearable technologies by Geistweidt. “In the beginning the sensors were the priority and the movement was secondary,” Ring said. “We looked at the way the sensors were giving us data to inform how the dancers would move. We presented a small version of that work last spring for the Department of Media Studies Spring Show, but we were thinking ahead to the next version where the choreography would lead and the sensors would in some sense follow.”

Over the summer of 2023 Geistweidt continued to develop the sensors and more fittings were done with the dancers. By the fall of 2023 Ring’s choreography had grown to include four sections. Information from the sensors successfully controlled a wireless lighting rig designed by THD Design Technology Professor Lynne Koscielniak.  THD Clinical Assistant Professor Zechariah Saenz, who also manages the Costume Shop, then introduced costume props to the choreography.

“There’s one section of the work that’s based on percussion, while another is based on human breath and voice,” Ring said. “We defined these different categories so that the dancers wouldn’t necessarily be moving the same way in each section. When it’s breath-based with silence there would be more of a hold, and organic flow. When there was more percussion it would be more full-bodied, with leaping, etc. We had to create a palette in sound, lighting and projection, and that’s where the collaboration was.”

“For the sound design, I use a piece of software called Max/MSP, which is on my computer,” Geistweidt said. “It’s used in theater and computer music. As the choreography progressed, Kerry was using stand-in music, which she sent to me. I listened to it for a while and (created music that was similar). There are four dancers in the piece and each one has their own kind of sonic palette. I have about 17 pre-sets. For this piece the sound design is based upon the granular manipulation of pre-existing samples. I built the sensors and the sound(s) to go together, and then the next thing was how to get it out to the theater system.”

Regular work sessions in Center for the Arts Drama Theatre were supported by CFA Head Electrician Anthony Rajewski, who facilitated computer networking. Max/MSP triggering QLab software via OSC commands was a success, and THD Assistant Technical Director Tom Burke facilitated a projector rig along with theatre design and technology major Ethan Borrok, who is the Associate Projector Designer on the project.

“Since this is the first investigation, we have a kind of John Cage open work approach,” Geistweidt said. “The sensors on the dancers trigger things, which are random in a sense, but not completely, as it’s within a range of what can happen. When the dancers move past a certain degree with a hand, a few sensors will pop off and a random selection of video effects will go off. There’s a coordination of the entire systems; the lighting grid, audio, video, and the dancers are intricately linked by this network which pulls it all together. I think it creates something pretty novel. It’s very emotional. You can see when a dancer moves that there’s a direct reflection to what’s happening across the entire theatrical network.”

“The artwork that’s included in the projections was created by Lynne Koscielniak,” Ring added. “We’re using five LED lights, so additional work came from THD Design Tech student Lowden Flower. The sensor data draws the intensity and color of the light based on how quickly the dancers are moving. For the projection, Lynne created a palette which we agreed upon and then the dancers trigger a wide array of video effects including saturation point and movement.”

“I’ve totally enjoyed this project. It’s been scary and exciting, successful and fulfilling,” she said. “It’s great to think about creating a dance in a totally new way. It’s been wonderful to have the four of us-Zech, Jason, Lynne and I–all working together for an extended amount of time, and really staying on the same page.

“(All of the) students have been watching us. We’ve had the dancers in the studio throughout the whole process, so I think it’s been not only a good learning experience for me as an artist, but also for the students to be in the room to see how collaborations can go.

“There’s no way I would have made this dance without these influences. There are costume components that I would not have dreamed of. The same with the prop components which Lynne suggested. For example, in the third section there’s a vest that has fifteen foot strands of silk connected to it. That idea would not have come about without Zech and I talking about costume as a projection surface, to catch the images that Lynne was projecting. We also ended up with hoops the dancers use which catch and reflect the projected surfaces.

“It’s quite a big collaborative team with four professors, four dancers and an understudy working on the movement. Lynne had design students working on projection and lighting, so when we brought the whole group together it’s about 13 to 15 people who’ve really had their hands in the process.”

Plans for danceSense continue beyond the Zodiaque performance run this weekend.  There will be a free performance of the work at Spring into Art, presented by UB Center for the Arts on April 30, 2024. It will also be included in the 2024 DMS Spring Show. The group also hopes to schedule a regional or national gallery performance in later 2024-2025, as well as publish articles in dance and theatrical journal publications. Koscielniak will present the work at the United States Institute for Theatre Technology (USITT) National Conference, held this upcoming March in Seattle, Washington.