What makes live experiences special? Fusing art and science with Neurolive

In November 2022, the Siobhan Davies Studios in London hosted the live performance "How Shall We Begin Again". This contemporary dance event featured 50 performers who crafted a non-repetitive experience on stage. To delve into the intricacies of liveness – that is, the true essence of what makes a live performance special – the audience was invited to wear Pupil Invisible eye tracking glasses and ANT Neuro EEG headsets. These devices allowed researchers to capture the ocular and brain activity associated with the immersive nature of the performance.

The Neurolive project

"How Shall We Begin Again" marks the second live performance created as part of the Neurolive project. Neurolive, an interdisciplinary research collaboration, aims to unravel the essence of live experiences and the unique connection they forge between artists, scientists, and audiences. This five-year research program, supported by the European Research Council, merges artistic research, performance making, and cognitive neuroscience to explore the multifaceted nature of liveness. Guided by the artistic director, Matthias Sperling, and the scientific director, Guido Orgs, this collaboration fosters a fruitful exchange where artists and scientists co-design studies and blend their distinct practices. Together, they lead Neurolive in a visionary exploration at the intersection of art, science, and the intricacies of live experiences.

Unique challenges in a novel environment

The experimental setup, a new benchmark in the field, had an impressive ensemble of up to 20 Pupil Invisible glasses and ANT Neuro mobile EEG headsets operating at any given time. This presented unique challenges, such as synchronization of all equipment. To achieve this, the team utilized Pupil Labs' Real-time API, ensuring precise coordination of Pupil Invisible with the other data streams.

Figure 2. The control station where synchronisation of multiple sensors was coordinated

The audience, meanwhile, enjoyed the freedom to come and go as they pleased at any time during the performance, making it essential to have a dedicated team of research assistants efficiently overseeing the testing. And they managed as such, with a sleek and streamlined operation that was impressive to watch.

Figure 3. Setting up Pupil Invisible eye tracking glasses and ANT EEG system with a participant

The hardware and data logistics aspects were also crucial, given the vast amounts equipment and eye tracking footage generated over hours of the performance. Pupil Cloud proved to be a valuable tool, facilitating automatic and secure backups of the data at the end of the long days of recording.

Figure 4. Behind the scenes hardware logistics. Check out all those Pupil Invisible eye tracking glasses!

What's next?

The combination of EEG data and gaze patterns opens up significant possibilities for exploring the fundamental aspects of live experiences in new and inventive ways. With terabytes of eye tracking and EEG data at their disposal, the research team is now undertaking the task of processing this massive dataset, aiming to uncover intricate patterns that might shed light on the dynamics between performers and spectators.

The researchers aim to answer fundamental questions about the audience's engagement. Where did the audience direct their gaze? What elements of the performance captured their attention? How did brain activity fluctuate during pivotal moments in the performance, reflecting emotional responses and immersion? The aim is for these insights to provide a better understanding of the audience experience, revealing the interplay between individuals and the artistic expression unfolding before them.

"The end product for us is hopefully a number that tells us how live the experience of a person was, or how live a person felt, or how much they were immersed in the performance. An indicator that then contains both data about the gaze patterns, but also the brain activity." - Guido Orgs

We at Pupil Labs look forward to seeing insights gained about the different aspects of audience immersion, emotional connection, and the interplay between performers and spectators as they arise from this exciting and novel dataset.