Integrating Eye-Tracking into an Augmented Reality System for Surgical Training

Can augmented reality enhance surgical training?

Every day, around a million surgical procedures are carried out worldwide, providing vital and often lifesaving operations to those who need it most. To perform these procedures safely, trainee surgeons must learn a range of complex, multi-stage procedures, which involve memorizing detailed step-by-step instructions. For example, chest tube insertion, an emergency procedure carried out to drain the space surrounding the lungs, requires the surgeon to perform more than 20 steps in a specific sequence. However, having the necessary information at hand while practicing surgical skills is a challenge, with trainees often having to interrupt their learning to re-read the instructions.

Researchers from the Surgical Simulation Research Lab at the University of Alberta set out to improve surgical training methods using augmented reality. To achieve this, they combined Microsoft HoloLens augmented reality goggles with the Pupil Labs HoloLens Add-on, which adds Pupil Core eye-tracking capabilities to the HoloLens (1st gen) (Figure 1).

Although augmented reality systems have been developed previously to assist surgical training, these have not been designed to be seamlessly responsive to trainees’ behaviors. Instead, these systems display surgical instructions either continuously or at moments when the trainee may not even want instruction, providing unnecessary distractions from performing the procedure itself.

Figure 1. Combined augmented reality/eye-tracking headset for surgical training, integrating Microsoft HoloLens (1st gen) goggles with the Pupil Labs HoloLens Add-on (which adds Pupil Core-based eye tracking capabilities to the HoloLens (1st gen)).

Testing an augmented reality system that displays instructions on-demand

Using eye tracking, the research team aimed to build an augmented reality system that in response to trainees’ eye movements would provide them with surgical instructions at precisely the moments they were needed. To achieve this, the researchers capitalized on the customizability of the Pupil Labs open source library by modifying the “HoloLens Relay” plug-in to transfer eye-movement data between the Pupil Labs HoloLens Add-on and the HoloLens.

To test the system, the research team set up a chest tube insertion training simulation, consisting of a plastic human torso model with a soft synthetic skin pad positioned on the chest at the insertion site (Figure 2). They also positioned a blank sheet of cardboard near the torso to provide a surface onto which instructions could be projected in augmented reality.

Surface markers were attached in a square around the skin pad and on the cardboard instruction sheet, allowing the trainees’ eye movements within these areas to be automatically detected. If a trainee looked within the instruction sheet area, step-by-step instructions of the surgical procedure were immediately displayed on the sheet to assist them. In contrast, when their attention remained within the skin pad area, the augmented reality instructions were removed in order to avoid distractions.

Figure 2. Trainee surgeons wore the augmented reality/eye-tracking headset while practicing a chest tube insertion (left), a complex lifesaving surgical procedure, on a plastic dummy. Eye tracking (right) was used to determine whether the trainees were focusing on the area around the insertion site. To minimize distractions, surgical instructions were only projected onto the augmented reality display when trainees were looking outside of the insertion site.

Toward a seamless surgical training experience

The researchers found that the system provided a good level of assistance to the surgical trainees and now hope to develop it further. In particular, they are interested in how pupil diameter might be used to automatically identify moments at which the trainee is experiencing difficulty while learning a surgical procedure. Detecting these moments could allow the system to be even more responsive, matching the timing of instructions even closer to when they are needed and facilitating a smoother learning experience.

Pupil Labs helps to improve surgical training methods

Building on the customizability of the Pupil Labs system, the researchers successfully developed an augmented reality tool that offers trainee surgeons a more seamless learning experience. The use of eye tracking to control the display of surgical instructions is an innovative approach allowing trainees to simply move their eyes to intuitively access instructions at just the right moments.

With hundreds of millions of surgical procedures carried out worldwide every year, many of them lifesaving, it’s essential that surgeons receive the very best training they can. We at Pupil Labs are honored that our eye-tracking glasses can be a part of this, and wish the authors of the study the best in continuing to develop this important technology.

You can read the full paper here: Lu, S., Perdomo, Y. P. S., Jiang, X., & Zheng, B. (2020). Integrating Eye-Tracking to Augmented Reality System for Surgical Training. Journal of Medical Systems, 44(11), 1-7.

If you wish to include your published works or research projects in future digests, please reach out!

Figure 2 image credit: Figure provided by the article's corresponding author.