Guides
What makes a good eye tracker? Part 7: Open APIs, SDKs, and data formats
Years of working with our users have shaped both the products we build and the access we provide to them. We develop ready-to-use recording and analysis tools so that researchers can get on with their work. We also know that the range of things people want to do with eye tracking extends well beyond the functions we can build into those tools.
Some projects need a small adjustment to an existing workflow. Others involve a specialised application, a different analysis method, or a question that has little to do with the metrics we provide. Those needs have consistently informed how we develop our software.
We believe researchers should be able to work with the device and its recordings in the ways their projects require. AI coding tools are making it easier for people with little or no programming experience to create their own plugins, extensions, and applications. That is why we provide documented data formats, accessible APIs, and open-source libraries. These let our users connect their own tools, develop their own methods, and build on what we provide.
Using data with familiar analysis tools
Any Neon eye tracking recording should be straightforward to use in the tools a research team already works with. That might be a spreadsheet for an initial review, statistical software for an analysis, or a custom program developed for a particular study.
Familiar formats enable this. Pupil Cloud and Neon Player timeseries exports provide numerical data in CSV files, with scene video available as MP4. JSON files contain recording metadata and scene-camera parameters.¹ These formats can be read by a wide range of software applications.
The data formats also need to be documented. Gaze measurements needs to be reported in a defined coordinate system. Pupillometry metrics need a unit. Timestamps need an explanation of which clock they used and which moments they represent. Our recording-format documentation describes all of these details and more.¹
This context helps users interpret the measurements, combine them with other data, and reuse the data in future analyses.
Building processing workflows in code
For users who want to process recordings using code, we provide the open-source pl-neon-recording Python library. It offers an intuitive application programming interface (API) to Neon's native recordings, with access to numerical measurements, video, and timestamps. It also provides functions for matching data streams in time.²
The library is a foundation for building a complete processing workflow. A program can read a native recording, select periods around experimental events, retrieve the relevant measurements and video frames, and apply an analysis method. Users can then calculate the outcome metrics their project needs and save the results. The same workflow can be applied across a collection of recordings.
For example, a team studying task demands could select each trial from its event markers, process the pupil measurements, calculate a response relative to a baseline, and produce a results table for statistical analysis. The researchers define the processing steps and outcome measures around their question.
The same flexibility matters for industry teams building products around eye tracking. A training platform, for example, could use pl-neon-recording to read recordings after each session, apply its own analysis, and calculate performance measures relevant to the task. This processing could run automatically and feed results into the product's own reports or interface. Integrators can therefore build a workflow from recording to feedback, with processing methods and outcome measures defined around their application.
Community projects extend these possibilities further. PyNeon is an independently developed Python package for processing and analysing Neon's multimodal data. Its authors acknowledge pl-neon-recording as an inspiration for its design. It supports native recordings and Pupil Cloud exports, with tools for preprocessing, dividing recordings into trials, working with video, and exporting to BIDS formats used in research data sharing.³
Contributions like this matter to us. They bring approaches and priorities from the people doing the research and make those methods available to others.
Automating work in Pupil Cloud
Processing workflows can also include recordings stored in Pupil Cloud. The Pupil Cloud API lets software work directly with recordings, projects, and annotations, allowing Cloud to become part of a larger workflow.⁴
A program can list recordings in a workspace, read their metadata, organise them into projects, and download recording data. It can also retrieve, create, and update event annotations.⁴ For example, a team could build a script that downloads the recordings belonging to a project, runs its own analysis, and adds detected events back to Cloud for review.
The API also provides access to area-of-interest statistics and exports.⁴ These results can be brought into a team's own analysis or reporting tools and combined with measurements collected elsewhere.
This gives users a way to automate repeated work across many recordings. A workflow can connect the organisation and analysis available in Cloud with custom processing, while keeping the results accessible through the tools the team already uses.
Streaming live data and controlling Neon
During data collection, an eye tracker may need to communicate with experiment software or another application. A program might start a recording with the first trial, mark when a stimulus appears, or use live gaze to change what happens next.
Live data and gaze mapping
Neon's Real-Time API gives offers a live connection to the device. Software can receive gaze and eye measurements, including pupil diameter and 3D eye poses, alongside scene video, eye video, audio, and head-motion data. These streams can be processed as they arrive, allowing users to build applications that respond to what the wearer is doing.⁵
For applications combining live video and gaze, the Real-Time API provides scene-camera frames matched with their corresponding gaze measurements. An application can pass these images to another algorithm and compare its output with gaze as the data arrives.⁵
When gaze needs to be expressed on a screen or another flat surface, pl-marker-mapper provides mapping using AprilTags. It can work with the live data received through the API or with recordings processed afterwards.⁶
Device control and application development
The same connection provides control over data collection. An application can discover devices, check their status, start and stop recordings, and save timestamped events. A team can therefore build a workflow that coordinates the experiment, records its key moments, and uses incoming measurements to determine what happens next.⁵
This supports anything from a short script that automates recording to an application that processes several live streams and provides feedback during a task. We provide open-source clients for Python, MATLAB, C++, and Unity C#.
The Python client and its worked examples make the API approachable for people with little or no programming experience. Users can start with a working script, see how it streams data or controls a recording, and adapt it step by step for their own application. This provides a practical way to learn the API while building something useful. Documentation of the underlying interface also supports developers building their own clients.⁵
Synchronising devices
The API also supports precise time synchronisation between multiple Neon devices and between Neon and other equipment. Neon provides UTC timestamps for its data, and the API can estimate the difference between each Companion device's clock and a computer's clock with millisecond accuracy. These offsets allow measurements and experimental events to be placed on a common timeline.⁷
Lower-level access
We also provide libraries for developers working more directly with cameras, video files, and camera geometry. pl-neon-usb connects to Neon's eye and scene cameras over USB, allowing applications to stream images, change exposure settings, and retrieve camera parameters.⁶
pl-video handles reading and writing video files, including selecting individual frames by index or timestamp. pl-camera provides functions for working with camera geometry, such as correcting lens distortion and converting between image positions and viewing directions.⁶ These components support developers building their own acquisition systems or specialised processing tools.
Complete workflows in Alpha Lab
In Alpha Lab, we publish experimental applications that show how these interfaces can be combined into complete workflows. The examples connect Neon with other algorithms and software, providing working code that users can adapt to their own questions.
Map Gaze Onto Anything demonstrates this with object detection and hand tracking. Its live workflow uses the Real-Time API to receive scene-camera frames and gaze, then passes the images to third-party computer vision models. Its post-hoc workflow uses pl-neon-recording to process recorded video and gaze. Both show how the same analysis idea can be built around live or recorded data.⁸
Real-time Visualizations uses the Real-Time API to build live eye models, scrolling measurement plots, and a view of the Neon module's orientation.⁸ These examples show how applications can turn incoming measurements into displays that help someone monitor or respond to what is happening.
Build Gaze-Contingent Assistive Applications demonstrates how these tools can turn gaze into an input for an interface. It uses the Real-Time API to receive data and the real-time-screen-gaze package to locate a screen using AprilTags and map gaze into screen coordinates. The example then uses those coordinates to move a cursor and trigger clicks when gaze remains on an element for a specified time.⁸ This provides a starting point for building interfaces that respond to where someone looks.
Audio-Based Event Detection demonstrates a workflow built around the Pupil Cloud API. It combines recorded audio with Whisper speech recognition to identify spoken events and add timestamped annotations to recordings in Cloud.⁸ This connects an external analysis method with the recording and review tools already available there.
A Guide to Multiperson Eye Tracking combines the Real-Time API with pl-neon-recording to synchronise and analyse recordings from several wearers. The API discovers and controls the devices, calculates their clock offsets, and saves synchronised event markers. The guide then shows how to use those markers to align the recordings and access their measurements together, supporting analyses of gaze behaviour during interactions between participants.⁸
Together, these examples show how the interfaces support different parts of an application, from accessing measurements and running another algorithm to saving results for further analysis.
Extending the Pupil Lab's analysis tools
The same foundations of open data formats and APIs also support our own applications. Neon Player uses pl-neon-recording at its core to load and access recordings.⁹ The library available to researchers is therefore also the basis for the recording access inside our desktop software.
Player adds a graphical interface for playback, review, analysis, and export. Its plugin architecture lets developers introduce new processing methods, visualisations, and exports within that environment.⁹ A specialised method can sit alongside the existing timeline and video tools, making its results easier to inspect against the recording.
The plugin collection includes tools for synchronising external camera video, displaying GPS data, transcribing audio, and processing scene depth. We publish these through a repository designed for sharing plugins from Pupil Labs and the community.¹⁰ These extensions demonstrate how additional data and methods can become part of the same recording workflow.
Keeping options open
For us, a good eye tracker should make it straightforward to begin and give users room to develop their work. Ready-to-use tools help turn recordings into useful results. Access to the data and interfaces allows users to adopt different methods, connect other equipment, or build eye tracking into their own products.
Our users and the wider community are an essential part of that development. They bring questions, methods, and experience that lead to new applications and tools that others can build on.
A team can choose the form that suits its work. It may build an automated processing program with pl-neon-recording, use a community package such as PyNeon, or develop a Player plugin with an interface for reviewing results.
We see community contributions as an essential part of this approach. Our users bring questions, methods, and experience that continually expand what can be done with eye tracking. Keeping the data formats and interfaces open gives those ideas room to develop and makes it possible for others to build on them. We see openness as part of the usefulness of an eye tracker. The measurements should remain accessible as the questions, methods, and software around them change.
Where to go next
This article is part of a series exploring what makes a good eye tracker. Related topics include:
¹ Pupil Labs, Recording Format.
² Pupil Labs, Neon Recording API and library documentation.
³ PyNeon developers, PyNeon documentation.
⁴ Pupil Labs, Pupil Cloud API reference.
⁵ Pupil Labs, Real-Time API and Python client documentation.
⁶ Pupil Labs, Open-Source Libraries.
⁷ Pupil Labs, Achieve Super-Precise Time Sync.
⁸ Pupil Labs, Alpha Lab.
⁹ Pupil Labs, Neon Player development and plugins.
¹⁰ Pupil Labs, Neon Player Plugins repository.