Guides
What makes a good eye tracker? Part 6: Gaze mapping
A wearable eye tracker measures gaze in the coordinates of its scene camera, as a point within the video it records. Some questions can be answered from gaze and eye-movement data alone, without identifying an object or area in the scene. Often, though, the question is what the wearer looked at. To answer it, gaze must be mapped to what appears at that position in the scene video. This article explains why this mapping is needed and how it can be achieved.
From scene camera coordinates to the world
The scene camera in a wearable eye tracker usually faces forward and records the wearer's field of view. Gaze is reported as a position within each frame of the scene camera video. Because the camera moves with the wearer's head, a stationary object in the world can appear in different parts of the video. For example, a sign fixed to a wall may shift from the centre to the edge of the frame as the wearer turns their head. In other words, a gaze point in scene camera coordinates shows where someone looked within a frame, but does not identify the object or the part of it they viewed. Gaze mapping is the process that connects each gaze point to what's been looked at.
Gaze mapping methods
How gaze is mapped depends largely on what needs to be identified, and in what conditions. The main approaches are manual review, marker-based mapping, and marker-less mapping.
Manual coding and mapping
Pupil Cloud interface showing manual mapping. Fixations in the scene video are manually mapped onto a reference image of the environment on a point-and-click basis. After one fixation is mapped, the software automatically skips to the next fixation.
The most direct approach is to manually review the video with gaze shown on it and label each fixation according to what the wearer looked at. In it's most basic form, you might think of this as semantic mapping. This type of mapping can also record a position. For example, a fixation on an info-screen can be marked at the corresponding point on a photograph of that screen. Both approaches are flexible, but reviewing every fixation takes time, especially for long recordings or many wearers. Clear coding rules also help different people make consistent decisions.
Marker-based mapping
Pupil Cloud interface showing marker mapping. Fixations in the scene video are mapped automatically onto a reference image of the environment, saving significant time when compared to manual mapping.
Marker-based methods add visible reference points to the environment, such as QR codes or AprilTags. Software tracks their positions in the scene video as the wearer moves. When placed around a surface, such as a magazine cover, the markers allow gaze and fixations to be mapped from the video onto that surface. Because this happens automatically, it can save considerable time compared with manual mapping. The markers do, however, take up space in the scene and need to remain visible to the camera.
Marker-less mapping
Pupil Cloud interface showing markerless mapping. Fixations in the scene video are mapped automatically onto a reference image of the environment, but without the requirement for markers affixed to surfaces in the scene.
Marker-less methods use features already visible in the scene video, without adding markers to the environment. This can save preparation time in environments where markers may be disruptive or impractical.
Depending on the method, software matches visual features in the video to a reference image or object model, or identifies a type of target.
Products displayed on a shelf, for example, can be photographed before a recording begins. As the wearer moves, software can recognise the display as it appears in the scene video, from its shape, colours, text, and edges, then map gaze onto the reference photograph. The same approach can be used for a poster, book page, or road sign.
Face and object detection algorithms can locate targets such as faces, vehicles, or phones, in the scene video. Their positions can then be compared with gaze to determine whether someone looked at them. This can show what type of target was viewed without positioning gaze on a shared reference image.¹,²
Some approaches also account for the three-dimensional shape of an object or environment. Images captured from different viewpoints can be used to build a 3D model through photogrammetry. Once the camera's position has been determined within that model, the gaze direction can be projected into it to identify what's being viewed. For a machine, this could distinguish gaze directed at its front, side, or top.³
Marker-less methods can reduce setup and manual review, and a prepared reference may be reused across recordings.
Why a shared reference matters
When several wearers encounter the same part of an environment, such as a sign or menu, a photograph of it can serve as a shared reference image for all their recordings. Mapping places their gaze points and fixations at corresponding positions on that image, allowing the data to be compared or combined despite different paths and viewpoints. Areas of interest can then be drawn around parts of the sign or menu, with fixation counts and durations calculated for each area. Heatmaps show where gaze was concentrated, while scanpaths show the order in which areas were viewed. The article on analysis will cover these tools in more detail
Gaze mapping tools
Pupil Cloud offers several powerful tools for both manual and automatic mapping. Manual Mapper lets a person review each fixation in the scene video and click its corresponding position on a reference image.⁴ Reference Image Mapper and Auto Image Mapper find the target in the scene video and map gaze onto a reference image automatically.⁵,⁶ Face Mapper detects when gaze falls on a face.¹ Marker Mapper uses AprilTags to track a surface.⁷,⁸
Beyond these tools, Neon's synchronized scene video and gaze data are available in open data formats. They can be used with custom mapping algorithms when a target does not fit one of the built-in methods. The articles on data streams and open APIs cover access to these data.
Together, the tools and open data support different ways of working. Pupil Cloud's mappers offer a turnkey experience, while access to the recordings allows a custom method to be built when needed.
Where to go next
This article is part of a series exploring what makes a good eye tracker. Related topics include:
¹ Pupil Labs, Face Mapper. https://docs.pupil-labs.com/neon/pupil-cloud/enrichments/face-mapper/
² Pupil Labs, Map Gaze Onto Anything (Alpha Lab). https://docs.pupil-labs.com/alpha-lab/map-onto-anything/
³ Pupil Labs, Map Gaze Into a User-Supplied 3D Model (Tag Aligner). https://docs.pupil-labs.com/alpha-lab/tag-aligner/
⁴ Pupil Labs, Manual Mapper. https://docs.pupil-labs.com/neon/pupil-cloud/enrichments/manual-mapper/
⁵ Pupil Labs, Reference Image Mapper. https://docs.pupil-labs.com/neon/pupil-cloud/enrichments/reference-image-mapper/
⁶ Pupil Labs, Auto Image Mapper. https://docs.pupil-labs.com/neon/pupil-cloud/enrichments/auto-image-mapper/
⁷ Edwin Olson, “AprilTag: A robust and flexible visual fiducial system,” Proceedings of the IEEE International Conference on Robotics and Automation, 2011. https://doi.org/10.1109/ICRA.2011.5979561
⁸ Pupil Labs, Marker Mapper. https://docs.pupil-labs.com/neon/pupil-cloud/enrichments/marker-mapper/
Explore gaze mapping in Pupil Cloud.
Mapping gaze onto something unusual? See Map Gaze Onto Anything in Alpha Lab.