We are pleased to announce the release of Pupil Core software v3.5!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Overview

In Pupil v3.5, we cleaned up log messages, improved software stability, fixations are now cached between Pupil Player sessions, and we have fine tuned 3d pupil confidence.

Pupil v3.5 is the first release that supports the new macOS Monterey (with a small caveat). Read more below.

New

• Recording-specific annotation hotkeys

• Add option to filter out low-confidence values in raw data export

• Add option to disable world video recording

• macOS Monterey support

Improved

• Reduced number of user-facing log messages

• Export-in-progress file-name suffixes

• Windows: Handle exceptions during driver installation gracefully

• Windows: Option to skip automatic driver installation

• Windows: Create start menu entries and desktop shortcuts for all users on install

• Add Euler angles to headpose tracker result

• Normalize IMU pitch/roll values to [-180, 180] degrees

• Confidence adjustment for pye3d results

• Cache post-hoc fixation detector results

• Documentation

New

Recording-specific annotation hotkeys - #2170

Until now, the keyboard-shortcut setup for annotations was stored in Pupil Player's session settings. These were reset if you updated Pupil Core software to a new version.

Starting with this version, annotation keyboard shortcuts will be stored persistently within the recording.

The definitions are stored in <recording dir>/offline_data/annotation_definitions.json with the format:

{
    "version": 1,
    "definitions": {
        "<label>": "<hotkey>"
    }
}

This file can be shared between recordings.

When Pupil Player is opened, it will attempt to load the annotation definitions from the recording-specific file. If the file is not found or is invalid, Pupil Player will fallback to the session settings which contain the last known annotation definitions.

Add option to filter out low-confidence values in raw data export - #2210

Pupil Core software assigns "confidence" values to its pupil detections and gaze estimations. They indicate the quality of the measurement. For visualization and some analysis Plugins, Pupil Player hides gaze data below the "Minimum data confidence" threshold (adjustable in the general settings menu).

Prior to this release, the raw data exporter plugin exported all data, independently of its confidence. New users, therefore, often wonder why their exported data looks noisier than what they have seen in Pupil Player.

Starting with this release, users will have the option to filter out low-confidence data (i.e. below the "Minimum data confidence" threshold) in the export. The option can be enabled in the raw data exporter's menu.

Add option to disable world video recording - #2169

If you are not interested in recording the scene video, you can now disable the scene video recording in the Recorder menu of Pupil Capture's world window.

macOS Monterey support

Due to new technical limitations, Pupil Capture and Pupil Service need to be started with administrator privileges to get access to the video camera feeds. To do that, copy the applications into your /Applications folder and run the corresponding command from the terminal:

• Pupil Capture: sudo /Applications/Pupil\ Capture.app/Contents/MacOS/pupil_capture

• Pupil Service: sudo /Applications/Pupil\ Service.app/Contents/MacOS/pupil_service

Note: The terminal will prompt you for your administrator password. It will not preview any typed keys. Simply hit enter once the password has been typed.

Note: When recording with administrator privileges, the resulting folder inherits admin file permissions. Pupil Player will detect these and ask you for the administrator password to reset the file permissions. This will be only necessary once per recording.

Improved

Export-in-progress file-name suffixes - #2167, #2186

Exporting video can take a long time. Pupil Player displays the progress as a growing circle around the menu icon while writing the result incrementally to disk. Specifically, it writes the result directly to the final export location. While the export is ongoing, the video file is invalid but still available to the user. This can lead to users opening the video before it has finished exporting and then encountering playback issues.

Starting with this release, Pupil Core software changes the video exports to include a .writing file name suffix during the export. This prevents other applications from recognizing the partially exported video file as complete. Once the export is done, the suffix is removed and other applications are able to recognize the file type as video.

Reduced number of user-facing log messages - #2190

Over the years, Pupil Capture's functionality has grown and with it the number of user-facing log messages. They can be overwhelming to the point where important log messages are overlooked or ignored. For this release, we have reviewed the log messages that are being displayed in the UI during typical actions, e.g. calibration and recordings, and have reduced them significantly.

Windows: Handle exceptions during driver installation gracefully - #2173

Previously, errors during the driver installation could cause the software to crash. Now, errors will be logged to the log file and the user will be asked to install the drivers manually.

Windows: Option to skip automatic driver installation - #2207

Using the -skip-driv or --skip-driver-installation flags, one can now launch Pupil Capture and Pupil Service without running the automatic driver installation on Windows. This can be helpful to speed up the start up time or to avoid unwanted driver installations.

Windows: Create start menu entries and desktop shortcuts for all users on install - #2166

Pupil Core software is installed to C:\Program Files (x86)\Pupil-Labs by default. This directory should be accessible by all users but the start menu entry and desktop shortcut are only installed for the current user. Starting with this release, the installer will create start menu entries and desktop shortcuts for all users on install.

Add Euler angles to headpose tracker result - #2175

Next to camera poses as Rodrigues' rotation vectors, the headpose tracker now also exports the camera orientation in Euler angles.

We also fixed a typo in the exported file name.

- head_pose_tacker_model.csv

+ head_pose_tracker_model.csv

Normalize IMU pitch/roll values to [-180, 180] - #2171

Orientation values exported by the IMU Timeline plugin will now be normalized to a range of [-180, 180] degrees.

Confidence adjustment for pye3d results - #2195

In some cases, pye3d was overestimating the confidence of its pupil detections. These were visible as noisy gaze estimations in the Pupil Core applications. This release includes an updated version of pye3d with adjusted confidence estimations. As a result, the gaze signal should look more stable during lower confidence situations, e.g. while squinting or looking down. For more information, see the Developer notes below.

Cache post-hoc fixation detector results - #2194

Post-hoc fixation detector results no longer need to be recalculated if you close and reopen a recording. Pupil Player stores the results together with the used configuration (version of the gaze data and fixation detection parameters) within the recording directory. The cached data will be restored if the Player configuration matches the configuration stored on disk. If it does not match, Player will recalculate the fixations to avoid inconsistencies.

Documentation

We have improved our documentation in several places:

• Pupillometry best practices - pupil-docs#453

• Blink detection - pupil-docs#463

• Pupil Core API - pupil-docs#455

Fixed

• Fixed typo in exported headpose tracker files - #2185

• Fix confidence graphs in Pupil Player for pre-2.0 recordings - #2191

• Use correct socket API to retrieve local IP address for Service - #2201

• Fix loading 200 Hz gaze data in Pupil Invisible recordings - #2204

• Fix loading short audio streams - #2206

Known issues

• Windows: Connecting remotely to Pupil Remote on Pupil Service does not work in some cases

• Pupil Groups: Users have reported issues with Pupil Groups connectivity when running Pupil Capture on different devices.

We are investigating these issues and will try to resolve them with an update as soon as possible.

Developer notes

Dependencies updates

Python dependencies can be updated using pip and the requirements.txt file:

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

pye3d v0.2.0 and v0.3.0

Version 0.2.0 of pye3d has lowered 3d-search-result confidence by 40%. When the confidence of the input 2d pupil datum is lower than a specific threshold (0.7), pye3d attempts to reconstruct the 2d pupil ellipse based on its existing 3d eye model. We noticed that the confidence for these results was systematically overestimated given the actual detection result quality and have therefore adjusted the overall confidence assignment for them. See pye3d-detector#36 for reference.

Version 0.3.0 uses a new serialization format for the refractionizer parameters but is otherwise functionally equivalent to version 0.2.0.

pyglui v1.31.0

This new version of pyglui introduces support for multi-line thumb and icon labels.

Example:

from pyglui import ui
context = {"state": False}
ui.Icon(
    "state",
    context,
    label="Multi\nLine",
    label_offset_size=-10,  # smaller font
    label_offset_x=0,  # no horizontal adjustments
    label_offset_y=-10,  # move label upwards
    label_line_height=0.8,  # reduce line height
)

zeromq-pyre >= v0.3.4

Starting with version v0.3.4, zeromq-pyre is able to handle cases gracefully where it is not able to access any network interfaces. See #2187 for reference.

Plugin API

Easy plugin icon customization - #2196

Plugins can now overwrite the following class attributes to customize the menu icon:

• icon_pos_delta: relative positioning

• icon_size_delta: relative font resizing

• icon_line_height (multi-line icons only): line distance

Requires pyglui v1.31.0 or higher

Real-time Network API

Add surf_to_dist_img_trans and dist_img_to_surf_trans to surface events #2168

These two homographies are necessary to transform points from the surface coordinate system to the 2d image space of the scene camera, and vice versa. They have been available as part of the surface location exports in Pupil Player. With this release, they also become available to other plugins in Pupil Capture or via its Network API.

Example plugin that access the new homographies in real-time:

import numpy as np
from plugin import Plugin
class Example(Plugin):
    def recent_events(self, events):
        # access surfaces or fallback to empty list if not available
        surfaces = events.get("surfaces", [])
        for surface in surfaces:
            # access raw homography data, a list of lists
            homography = surface["surf_to_dist_img_trans"]
            # convert to a matrix
            homography = np.array(homography)
            # use homography, e.g. printing it
            print(f"{surface['name']}: {homography}")

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

Supported operating systems

• macOS Mojave 10.14, macOS Catalina 10.15, macOS Big Sur 11

• macOS Monterey 12 requires administrator privileges to access the video cameras

• Ubuntu 16.04 or newer

• Windows 10

Checksums

$ shasum -a 256 pupil_v3.5*
41428173c33d07171bb7bf8f0c83b7debfabb66a1dfc7530d1ae302be70ccc9c  pupil_v3.5-1-g1cdbe38_windows_x64.msi.rar
608aceaf4bf450254812b3fcaf8ea553450fcf2d626ae9c2ed0404348918d6ad  pupil_v3.5-7-g651931ba_macos_x64.dmg
786feb5e937a68a21821ab6b6d86be915bcbb7c5e24c11664dbf8a95e5fcb8cc  pupil_v3.5-8-g0c019f6_linux_x64.zip

We are pleased to announce the release of Pupil Core software v3.4!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Overview

Pupil v3.4 introduces a new color scheme for pupil detection algorithms, improved gaze-estimation stability for frozen eye models, and a new IMU visualization and export plugin for Pupil Invisible recordings within Pupil Player.

New

• Dynamic pye3d model confidence

• Pupil detection color scheme

• Added IMU Timeline plugin

Improved

• Gaze-estimation stability with frozen eye models

• Annotation log message

• Prevent log flood in Surface Tracker related to finding homographies

• Support upgrading legacy recordings

New

Dynamic pye3d model confidence

Pupil Core software version v3.4 ships the new pye3d version 0.1.1.

In the previous versions of pye3d model_confidence was fixed to a value of 1.0. Starting with version 0.1.0, pye3d assigns different model_confidence values based on the long-term model's parameters. Specifically, model_confidence is set to 1.0 by default and set to 0.1 if at least one model output exceeds its physiologically reasonable range. If the ranges are exceeded, it is likely that the model is either not fit well or the 2d input ellipse was a false detection.

The assumed physiological bounds are:

• phi: [-80, 80] degrees

• theta: [-80, 80] degrees

• pupil diameter: [1.0, 9.0] mm

• eyeball center x: [-10, 10] mm

• eyeball center y: [-10, 10] mm

• eyeball center z: [20, 75] mm

Phi and theta ranges are relative to the eye camera's optical axis. The eye ball center ranges are defined relative to the origin of the eye camera's 3d coordinate system.

The model_confidence will be set to 0.0 if the gaze direction cannot be calculated.

Pupil detection color scheme

The new pupil detection color scheme makes use of the dynamic model confidence values by visualizing within-bounds and out-of-bounds models in different colors.

Color legend:

• #FAC800: Pupil ellipse (2d)

• #F92500: Pupil ellipse (3d)

• #0777FA: Long-term model outline (within-bounds)

• #80E0FF: Long-term model outline (out-of-bounds)

Note: Even though the detected pupil ellipses fit the pupil well on the right-side picture, the model is outside of the physiological ranges defined above. This can lead to errors in gaze-direction and pupil-size estimates. You can improve such a model by looking into different directions.

While the 3d debug window is opened, pye3d also visualizes the short- and ultra-long-term model outlines:

• #FFB876: Short-term model outline

• #FF5C16: Ultra-long-term model outline

Added IMU Timeline plugin - #2151

This is a new plugin designed to visualize and export IMU data available in Pupil Invisible recordings.

In addition to the raw angular velocity and linear acceleration, the plugin also uses Madgwick's algorithm to estimate pitch and roll.

To export the data, the plugin must be enabled and a raw data export must be triggered. Once the export is complete, there should be a imu_timeline.csv file with the following columns

Improved

Improve gaze-estimation stability with frozen eye models

Freezing the eye models is the best way to reduce noisy data when in a controlled environment. Prior to this release, freezing the eye model only affected eye ball center and pupil diameter measurements. Starting with this version, freezing the eye model also stabilizes gaze direction estimates.

In order to adapt to slippage, pye3d estimates eyeball positions on different time scales. These serve specific downstream purposes. By default, the short-term model, which integrates only the most recent pupil observations, is used for calculating raw gaze-direction estimates.

In highly controlled environments with few eye movements (e.g. subject uses head rest and is asked to fixate a static target), the most recent pupil observations might not be sufficient to build a stable short-term model. If this is the case, the gaze direction varies more than one would expect given the controlled environment.Starting in pye3d version 0.1.0, the gaze direction will be inferred from the long-term model if it is frozen.

Freezing the eye model is only recommended in controlled environments as this prevents the model from adapting to slippage. In turn, it is able to provide very stable data.

Improve annotation log message - #2148

When creating a new annotation, the log message will no longer show the Pupil Timestamp. Instead, Pupil Capture will show how old the annotation is, while Pupil Player will show the frame index of the world video.

This change should help avoid the confusion associated with displaying Pupil Timestamp.

Prevent log flood in Surface Tracker related to finding homographies - #2150

Previously, failing to locate a surface because of missing homographies would print an error message in the main window. In some cases, this would result in a flood of error logs.

This change ensures that failures of such type are only issued once, and are written to the log file directly.

Support upgrading legacy recordings - #2161

Pupil Core software version v1.16 is no longer necessary to upgrade deprecated recording formats. Instead, Pupil v3.4 performs all necessary upgrades by itself. This release also includes a bug fix for an issue with pre v0.7.4 recordings.

In v1.16, recordings made with Pupil Capture v1.2 or earlier, andPupil Mobile r0.22.x or earlier have been deprecated due to the fact that these recordings are missing meta information that is required for the upgrade to the Pupil Player Recording Format 2.0. For details see "Missing Meta Information" in the v1.16 release notes.

Fixed

• Use UTF-8 decoding for known UTF-8 encoded text files - #2146

• Update circle drawing to display correctly on Apple Silicon devices - #2147

• Fix crash on surface tracker export if filling the marker cache is not complete - #2157

• Fix surface tracker marker export to use export range - #2158

Developer notes

Dependencies updates

Python dependencies can be updated using pip and the requirements.txt file:

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

pyglui v1.30.0

This new version of pyglui includes the menu icon for the new IMU timeline plugin as well as the new Color_Legend UI element.

from pyglui import ui
color_rgb_red = (1.0, 0.0, 0.0)
label = "red color label"
ui.Color_Legend(color_rgb_red, label)

pye3d v0.1.1

This version improves gaze-estimation stability for frozen eye models. For details see above and the pye3d changelog.

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

Supported operating systems

• Windows 10

• Ubuntu 16.04+

• macOS Mojave 10.14+

• macOS Monterey 12 is not supported by this release

We are pleased to announce the release of Pupil Core software v3.3!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Overview

Pupil v3.3 focuses on improving software usability, stability and performance. We have made improvements to how product-specific plugins are loaded in Pupil Player, updated keyboard shortcuts, and fixed several stability issues. Additionally, the surface tracker now exports marker detection results.

Note: Prior to this release,diameter_3d data for all Pupil Core v3 recordings did not have corneal refraction applied. See pye3d v0.0.7 notes below for updates and post-hoc fixes.

Improved

pye3d v0.0.7

This release comes with our latest pye3d version including the following improvements

• Performance improvements - #27

• Simplification of Conic parameter calculation - #26

• Correctly apply corneal-refraction correction to diameter_3d result - #28

All pye3d versions prior to 0.0.7 do not correctly apply corneal refraction to the diameter_3d field. To check if you are affected by this bug, take a look at the method column of your pupil_positions.csv export.

You have two options to get correctly calculated pupil size data:

1. Use the circle_3d_radius values and multiply them by 2, or

2. Re-run the post-hoc pupil detection in Pupil Player v3.3 or later

Marker detection export - #2140

The Surface Tracker plugin requires fiducial markers for defining and tracking areas of interest. Until now, the detection result of these was only stored as an intermediate result. With this release, the data is also stored as a CSV file next to other surface-tracking related exports.

The data is exported to surfaces/marker_detections.csv.

Improve surface detection consistency - #2139

Surfaces can now be defined as long as one or more markers are detected in the scene video. Previously, this was only sufficient in some cases -- usually, at least two markers were needed.

Note: Surface tracking works best if you use multiple markers spread-out over the area of interest to define and track your surface. The surface position export files contains information about how many markers were detected for each surface position. You can use this information as an estimation of the tracking stability.

Product-specific plugin loading in Pupil Player, part 2 - #2125

We continue work on finetuning the list of enabled plugins for Pupil Invisible recordings.

Starting with this release we have removed the “Pupil from recording” option, as there is no pupillometry data to load from Pupil Invisible recordings. Pupil Player will no longer generate an empty pupil_positions.csv file after export.

Keyboard shortcuts update - #2129

The default annotation keyboard shortcut was changed to x in both Pupil Capture and Pupil Player to avoid collision with other plugin shortcuts.

We've posted tables of keyboard shortcuts for Pupil Capture and Pupil Player below. You can find these tables in our Capture and Player documentation.

Fixed

• Fixed HMD calibration crashing Pupil Service - #2120

• Fixed Pupil Groups serialization issue - #2124

• Fixed toggling pupil detection on and off - #2126

• Fixed calibration area calculation errors - #2128

• Fixed video source replacement in eye process - #2131

• Fixed gaze visualization on Apple Silicon computers - #2132

• Fixed Pupil Invisible timestamp update procedure precision loss - #2134

• Made pye3d plugin's debug visualization colors consistent - #2136

• Fixed validation crash when not enough reference data is collected - #2137

• Fixed a crash when using Time Sync on macOS - #2138

Developer notes

Dependencies updates

Python dependencies can be updated using pip and the requirements.txt file:

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

We have specifically updated the version requirement for pye3d.

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

We are pleased to announce the release of Pupil Core software v3.2!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Overview

Pupil v3.2 focuses on quality-of-life and stability.

New

• We added accuracy calculation and visualisation for VR/AR calibrations.

• Enable plugins in Pupil Player only if they are able to be used with the product that created the recordings.

Improved

• Pupil Player has improved playback for Pupil Invisible recordings.

New

Accuracy visualisation and calculation for HMD calibrations - #2112

Prior to this release, when performing HMD calibration, accuracy was not calculated nor visualised in Pupil Capture. This release makes accuracy calculation and visualisation consistent across setups.

Product specific Pupil Player plugin loading - #2100

Load plugins in Pupil Player only if they are able to be used with the product that created the recordings.

Pupil Player is a very flexible tool. It can load, playback, visualize, and export data from recordings made with Pupil Core, Pupil Mobile, and Pupil Invisible! Each of these products has different capabilities and data formats for their recordings.

We have evaluated all Pupil Player plugins and their validity for each product. Starting with this release, Pupil Player will only enable plugins that are suitable for the active recording type.

Here is a list of what plugins are available based on the product that generated the recording:

Improved

pye3d detection speed improvement

We are aware that pye3d requires more computational resources than its predecessor, and are actively working to improve the speed. For some users, it has become more difficult to reach a consistent detection rate of 200Hz.

The first step we have taken to speed up pye3d is to disable some optional computations that are not currently used by Pupil Capture and Pupil Player (jump to developer notes for details).

Fixed

• Support recordings with decoding issues - #2107

• Support recordings when an audio device fails - #2103

• Fixed laggy playback after seeking on Pupil Invisible recordings - #2108

Developer notes

Dependencies updates

Python dependencies can be updated using pip and the requirements.txt file:

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

pye3d v0.0.6

pye3d v0.0.6 allows disabling optional computations. Pupil Core is not making use of these optional computations, they have been disabled, effectively reducing the computational load of pye3d pupil detections. Read about the changes here.

Release Note Updates

15.03.2021 - v3.2-16

• Fixes an issue causing Pupil Player to crash on Windows when enabling the eye video overlay or post-hoc pupil detection - #2117

21.03.2021 - v3.2-20

• Improves HMD Calibration stability in Pupil Service, fixes #2119 - eb8c232

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

We are pleased to announce the release of Pupil Core software v3.1!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Motivation

Pupil Core software is now compatible with macOS 11 Big Sur! In addition, we have added support in Pupil Player for 200Hz gaze data downloaded from Pupil Cloud.

Improvements

macOS 11 Big Sur compatibility - #2086, #2087, #2090, #2095

macOS 11 Big Sur introduced backwards-incompatible changes that caused earlier versions of Pupil Core and other software to not run properly.

There is currently only a single Python version, Python 3.9.1, that is fully compatible with macOS Big Sur. As a result, our new macOS Pupil Core bundle ships with this new Python version. This may have an effect on user plugin dependencies. For more information, check out the developer notes below.

Support for Pupil Cloud's 200Hz gaze recordings - #2082

Due to the computational limitations on the Companion Device, gaze data is recorded at 66Hz. Recent updates to Pupil Cloud automatically "upgrades" the gaze data recordings when they are uploaded to Pupil Cloud providing you with 200Hz gaze data. In version 3.1, Pupil Player can read and export the new data.

Note: Recordings that were downloaded prior to this Pupil Cloud update need to be re-downloaded in order to include the 200Hz gaze data.

Changes

Visual calibration marker appearance - #2095

The visual appearance of the calibration marker changed slightly, due to how we render the calibration marker on screen. The change was necessary to support screen-based calibrations on Apple Silicon Macs. As a result, the marker edges do not appear as smooth as the previous version. The overall design and detection process remains the same.

Bug Fixes

• Gracefully handle Pupil Invisible video files starting with a null frame - #2077, #2089

• Handle zero-byte video files when looking up Pupil Mobile intrinsics #2083

Developer notes

Dependencies updates

Python dependencies can be updated using pip and the requirements.txt file:

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

Support latest GLEW 2.2 - pyglui #117

This release also adds support for the latest version of GLEW. This means that the pyglui dependency needs to be updated to version 1.29.

pye3d v0.0.5

We have included pye3d==0.0.5 which fixes compatibility issues with numpy==1.20 and scikit-learn==0.24.1.

macOS bundle shipping with Python 3.9.1

To simplify the usage of the application bundles, each bundle is shipped with their own Python environment. All plugins and their dependencies will run in the context of this environment.

Especially compiled dependencies, e.g. pyrealsense2, are always compiled against a specific Python version. If you are running custom user plugins, you will have to reinstall compiled dependencies in order to make them work with the new Python version.

Bundles ship with the following Python versions:

• macOS: Python 3.9.1

• Windows bundle: Python 3.6

• Linux bundle: Python 3.6

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

macOS compatibility - #2086, #2087

Note: This is the first release to support macOS 11 Big Sur. We no longer support macOS 10.13 High Sierra since the macOS package manager homebrew no longer provides pre-compiled binaries for Pupil Core software dependencies.

We are pleased to announce the release of Pupil Core software v3.0!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! 😄

Motivation

We are excited to introduce pye3d - a complete overhaul of the 3D pupil detector employed by Pupil Core software. In building pye3d, we have left no stone unturned, reimagining many of the algorithmic building blocks constituting the final gaze-estimation pipeline. pye3d is the result of three years of hard work and we are excited to get it into your hands!

Improvements

Improved built-in 3D pupil detector - #2011, #2065

pye3d implements a novel eye-model fitting strategy that incorporates a corneal refraction-correction step, accounting for nonlinear effects in ocular optics. Gaze estimates and pupil diameter estimates are now more accurate than ever before [2,3]. Thanks to a new sphere center estimation methodology [3], it is also considerably faster than our previous 3D detector [1] in regards to fitting robust and accurate eye models. Read more about how it works in our documentation.

Academic references

[1] L. Świrski and N. A. Dodgson. A Fully-Automatic, Temporal Approach to Single Camera, Glint-Free 3D Eye Model Fitting. In Proceedings of ECEM 2013. "L. Świrski and N. A. Dodgson. A Fully-Automatic, Temporal Approach to Single Camera, Glint-Free 3D Eye Model Fitting. In Proceedings of ECEM 2013."[2] K. Dierkes, M. Kassner, A. Bullling, A novel approach to single camera, glint-free 3D eye model fitting including corneal refraction. In ETRA ’18: Symposium on Eye Tracking Research and Applications, 2018.[3] K. Dierkes, M. Kassner, A. Bulling. A fast approach to refraction-aware eye-model fitting and gaze prediction. In ETRA ’19: Symposium on Eye Tracking Research and Applications, 2019.

Changes

Removal of old Pupil 3D detector

With this release, we also stop the maintainance of the previous 3D detector. Starting with the new 2.0 release, our pupil-detectors package no longer includes the previous 3D detector. This allows us to remove Ceres as a dependency, making the project much easier to maintain and build from source.

Should you rely on this detector, you can still download previous Pupil Core software releases or install previous pupil-detectors via PyPI.

Bug Fixes

• Fixed issue that would leave an unresponsive eye window due to plugin crash - #2059

• Fixed crash on canceling single marker calibration window mode - #2062

• Fixed crash on post-hoc pupil detection when the frame count is zero - #2063

Developer notes

Network API Changes

Reworked pupil detector plugin and network API

In this release we reworked the pupil detector plugin base class. It is now easier than ever to implement a custom pupil detector and accompanying pupil detector plugin.

We have also introduced breaking changes to our network API for pupil detector plugins. Here is a list of possible notification payload descriptions that are handled by detector plugins:

# Notification for enabling/disabling any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_enabled',
    'subject': 'pupil_detector.set_enabled',
    'value': <is_enabled: bool>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}
# Notification for setting the Region-Of-Interest (ROI) for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_roi',
    'subject': 'pupil_detector.set_roi',
    'value': <roi: (min_x: int, min_y: int, max_x: int, max_y: int)>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}
# Notification for updating a partial set of pupil detector properties for a specific pupil detector plugin
{
    'topic': 'notify.pupil_detector.set_properties',
    'subject': 'pupil_detector.set_properties',
    'values': <detector_properties: dict>,
    'eye_id': <eye_id: int>,  # required; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # required
}
# Notification for requesting pupil detector properties broadcast for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.broadcast_properties',
    'subject': 'pupil_detector.broadcast_properties',
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

For more information about custom pupil detector plugins, please read the documentation. You can also see examples of custom pupil detection plugins here.

For an example script that showcases the network API, please consult this helper script.

Changed 3D pupil datum format

pye3d uses a slightly different pupil format than our previous 3D detector. Because pye3d continously updates a single model instead of fitting multiple parallel ones, the new 3D pupil datum no longer contains the following keys: model_id and model_birth_timestamp.

Our latest Hmd-eyes release handles this by falling back to "0" and 0.0, respectively.

Binocular 3D gaze data with string-only keys - #2068

In order to be compatible with the msgpack-python v1.0.0 (specifially the new strict_map_key=True default), this release uses strings only as dictionary keys for all msgpack-encoded data that is published via the Network API.

This change affects primarily binocular 3D gaze data which used to contain integer dictionary keys.

Raw Data Exporter - Optional model_id key - #2061

Some of the fields in the pupil datum related to the 3D model no longer exist in pye3d (see above). This change ensures the raw data export handles missing fields and outputs a compatible csv file.

Updated msgpack dependency to version 1.0 - #2068

We continue to streamline and simplify the installation of Pupil Core dependencies. In this release we updated the msgpack to the latest 1.0. To ensure you have the latest required versions of all dependencies, please install them via requirements.txt.

python -m pip install --upgrade pip wheel

pip install -r requirements.txt

Added support for custom receiver high-water mark for HMD Streaming source - #2058

This change adds the ability to set a custom ZMQ high watermark value for HMD Streaming source. This gives users more control over the streaming behaviour when using Pupil Core with an HMD headset.

Added launcher_process.should_stop notification handling #2070

This change adds the ability to request stopping Capture, Player or Service with the same notification.

import zmq
import msgpack
# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
pub_socket.connect(ipc_pub_url)
subject = "launcher_process.should_stop"
topic = "notify." + subject
payload = {
    'topic': topic,
    'subject': subject,
}
pub_socket.send_string(topic, flags=zmq.SNDMORE)
pub_socket.send(msgpack.dumps(payload, use_bin_type=True))

Simplified running from source on Windows - #2073

We have removed the necessity to run Pupil from source using the run_<app>.bat files on Windows. Instead of extending the PATH environment variable via the .bat file, we extend it via the application launcher. This also makes it easier to attach a debugger, e.g. via Visual Studio Code.

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

macOS (11) Big Sur compatibility - #2079

Note: Pupil Core is not currently supported on Mac OS Big Sur. We aim to release a compatability update within the next few weeks. :construction_worker:

We are pleased to announce the release of Pupil Core software v2.6!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Motivation

In v2.6, we are taking a big step towards our modularity goal. We now support custom user pupil detector plugins. This change makes it easier than ever for you to modify the existing pupil detection algorithms or to build your own! You will no longer have to run Pupil Core software from source to get started with your custom pupil detector! Read more about it in our documentation.

Improvements

Support custom pupil detection plugins - #2033

You can now run your own pupil detection plugins that run in the eye processes. These plugins can be used to implement custom algorithms for extracting pupillometry data from the eye images and feeding it to the rest of the gaze mapping pipeline.

Pupil detection plugins are supported in Pupil Capture, Pupil Player, and Pupil Service. Similar to regular user plugins, custom pupil detection plugins should be placed inside pupil_capture_settings/plugins for Pupil Capture, pupil_player_settings/plugins for Pupil Player, and pupil_service_settings/plugin for Pupil Service. The plugins are automatically loaded when the applications starts.

Read more about it in our documentation.

Support HMD Streaming Source in eye process - #2052

We have added support for using HMD Streaming Source in the eye process. This also adds support for BGR and Gray frame format for HMD streams.

Support Pupil Invisible recording format v1.4 - #2054

This new recording format has been introduced in Pupil Companion 1.0.0. This version provides support for the OnePus 8 phone and Android 11. On the OnePlus 8, eye videos will be h264-encoded.

Bug Fixes

• Removed import * statements - #2036

• Fixed typo in circle detector benchmark - #2038

• Fixed CPU graph and UI refresh performance - #2040, #2041

• Fixed gaze with negative z-component - #2043

• Fixed issue when minimizing 3D pupil detector debug window on Windows - #2047

• Fixed UVC camera auto-selection priority in case of multiple pattern matches - #2051

Developer Notes

Added ability to stop eye process plugins via notifications - #2039

Another small improvement for eye process plugin management in this release is the ability to stop plugins via notifications. This gives better control over the list of plugins running in the eye process.

import zmq
import msgpack
# class name of the plugin to stop
plugin_name = "..."
# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
pub_socket.connect(ipc_pub_url)
for eye_id in (0, 1):
    subject = "stop_eye_plugin"
    topic = "notify." + subject
    payload = {
        'topic': topic
        'subject': subject,
        'target': f"eye{eye_id}",
        'name': plugin_name,
    }
    pub_socket.send_string(topic, flags=zmq.SNDMORE)
    pub_socket.send(msgpack.dumps(payload, use_bin_type=True))

Added requirements.txt file - #2029

We continue to streamline and simplify the installation of Pupil Core dependencies. You can now use the requirements.txt file from the root of the pupil repository to install and update all Python dependencies.

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

We are pleased to announce the release of Pupil Core software v2.5!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Motivation

Pupil v2.5 focuses on further improvements to software stability. We have fixed multiple issues that prevented very old recordings from being opened in Pupil Player.

Improvements

Replacing internal GLFW bindings with pyGLFW - #2015

In this release we replaced the internal glfw.py bindings to GLFW with the external pyGLFW package. This change simplifies our dependency list, making easier to install GLFW on Windows.

New default camera intrinsics - #2023

We use camera intrinsics in multiple places to transform between the 2d distorted image space and the 3d undistorted camera space. With this release, we have updated the default scene camera intrinsics to perform this tansformation more accuractely at the image borders.

Note: Each camera is slightly different. Therefore, it is recommended to run your own camera intrinsics estimation to get the best possible transformations for your camera.

Bug Fixes

• Fixed a crash in Pupil Player when attempting to visualize invalid data - #2021

• Fixed an edge-case when handling recordings with negative time jumps - #2026, #2028

• Fixed a frame drop issue on macOS when windows were occluded - #2027

• Fixed an issue that caused instabilities when starting the Head Pose tracker - #2022

• Surface Tracker stability improvements - #2025

Developer Notes

Improved Observable class - #2022

Improvements to the Observable class enable you to observe monkey patched methods (e.g. replaced methods that were not in the original class definition). Additionally, trying to add private methods as observers now fails with a more explicit error.

Changed Requirements

Please install the new external glfw dependency:

pip install glfw

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

We are pleased to announce the release of Pupil Core software v2.4!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Motivation

In Pupil v2.4, we are adding support for eye camera intrinsics. These are used to improve 3d eyeball model positioning. Additionally, we have fixed a long-standing issue that caused application windows not to appear on startup when using Windows OS.

Improvements

Better 3D Eyeball Position - #1995

We now use the focal length of the eye cameras to improve the estimate of the 3D eyeball position. Previously, the 3d eye model assumed a fixed camera focal length. This will most likely not have any noticeable effect for most use-cases, as it does not influence the gaze estimation. We assume default focal lengths for the different types of eye cameras shipped with Pupil Labs's hardware.

Caveat: 200Hz Vive Add-on Recordings

Pupil Core software now saves the eye camera intrinsics to the recording folder. On opening recordings made with previous versions in Pupil Player, we patch up the recording by inserting the appropriate eye camera intrinsics.

It is important to note that the 200Hz Pupil Core and 200Hz Vive Add-on use eye cameras with different focal lengths. Unfortunately, it is not possible for Pupil Player to differentiate between existing eye videos made by these cameras. As a result, Pupil Player will use 200Hz Pupil Core eye camera intrinsics for existing 200hz Vive Add-on eye videos.

To avoid this, we recommend placing 200Hz Vive Add-on intrinsics into your exisiting recordings. You can download the default 200Hz Vive Add-on intrinsics from the Downloads section below. Remember to remove the vive_200hz_ prefix.

Recordings made with Pupil Capture v2.4 and higher will not require any action in this regard.

Bug Fixes

• Fixed issue when opening Pupil Invisible recordings with more than 10 parts - #1996

• Ensure timelines update when changing calibrations for post-hoc gaze-mapping - #2000

• Fixed a crash when opening a recording that contained recorded fallback intrinsics - #2002

• Fixed a very rare crash of the eye window caused by malformed data - #2008

• Fixed occasionally missing windows on startup on Windows in multi-screen setups - #2007

• Resolved an error message when minimizing the eye windows on Windows - #2009

Developer Notes

Changed Requirements

pupil-detectors - #1995

We have updated pupil-detectors to v1.1.1, please update with

pip install -U pupil-detectors

packaging - #1993

We now require packaging>=20.0, please update with

pip install -U packaging

Internal Version Parsing - #1993

We have switched to a more general version-parsing method by using packaging.version.parse to parse all internally used version strings (from Pupil and external dependencies).

Black Formatter 20.8 - #2003

We have upgraded our black formatter to version 20.8, which introduces a few changes to previous versions and reformats parts of the code which were fine previously. Please make sure to also upgrade black with

pip install -U black

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

We are pleased to announce the release of Pupil Core software v2.3!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Motivation

Pupil v2.3 is a compatibility release for Pupil Invisible, adding support for a simple gaze confidence measure.

Improvements

Pupil Invisible Gaze Confidence - #1984

With its latest 0.8.23 release, Pupil Invisible can detect if the glasses are being worn. If they are not worn, gaze output will be hidden.

We now use this information as a simple binary gaze confidence when loading Pupil Invisible recordings in Pupil Player. If the glasses are detected as being worn, the gaze confidence will be 1.0, otherwise 0.0.

Smaller Improvements

• Added better log message for pupil mobile network issues - #1994

• Renamed "camera calibration" to "camera intrinsics" to avoid confusion with gaze calibration - #1992

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

We are pleased to announce the release of Pupil Core software v2.2!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Motivation

With v2.2, we have focused on improving the user experience when using Pupil on high-pixel-density displays. Now, Pupil scales and layouts its content correctly based on the content-scaling factor set in your operating system's settings.

Improvements

Support For High-Pixel-Density Displays - #1976

On displays with a high pixel density, content looks much smaller than expected. Operating systems have the option to scale its content appropriately. Until now, Pupil only supported content-scaling on macOS. With this release, we have added support for content-scaling on Windows and Linux as well.

Smaller Improvements

• Better user feedback when Accuracy Visualizer reports nan values - #1974

• Disabled Duplicate and Delete buttons for read-only post-hoc calibrations - #1972

• Use a square default eye window aspect ratio, matching the latest default cameras - #1987

Changes

Remove Option To Manually Change The Interface Size - #1988

With native support for content scaling on the operating system level, there is no need to scale the UI manually anymore. Therefore, we have removed this feature to further simplify the user interface.

Bug Fixes

• Fixed non-monotonic progress visualization for post-hoc gaze mapping - #1965

• Fixed flickering on macOS when resetting the window size - #1973

• Fixed an empty side-menu when restarting Capture after using HMD-Eyes - #1979

• Fixed a crash when attempting to log any non-string object - #1980

• Fixed 2D monocular gaze calibration not working - #1983

Developer Notes

Changed Requirements

pyglui v1.28 - #1972

We updated the pyglui dependency to v1.28. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/pyglui

On Windows, please download and install the latest wheel from GitHub.

glfw 3.3 - #1976

In order to support content-scaling across all three platforms, we make use of the glfwGetWindowContentScale function which has been introduced in glfw 3.3.

We recommend installing the latest glfw 3.3 release via https://www.glfw.org/download.html.

Removed g_pool.gui_user_scale - #1976, #1988

With the removal of the Interface Size setting, the g_pool.gui_user_scale variable has been removed as well. If you draw custom UI content in your plugins without using pyglui, you should now instead use g_pool.gui.scale to scale the content appropriately for different native display scaling settings.

Removed Legacy Plugin Base Classes - #1971

The following legacy plugin base classes have been removed. All plugins inheriting from them now just inherit from Plugin:

• Visualizer_Plugin_Base

• Analysis_Plugin_Base

• Producer_Plugin_Base

• Experimental_Plugin_Base

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

We are pleased to announce the release of Pupil Core software v2.1!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Motivation

With v2.1, we have continued to focus on making Pupil Core more core. We were able to resolve a lot of stability and user experience issues, especially regarding post-hoc calibrations.

Bug Fixes

• Fixed that recorded calibration were displayed as being editable - #1932, #1938

• Fixed that recorded calibration were duplicatable - #1933

• Fixed that immutable calibration could be deleted - #1934

• Fixed inconsistencies in the naming of recorded calibrations - #1935, #1944

• Fixed minimum confidence slider for post-hoc calibration - #1942

• Ensured correct validation is run even when a different calibration is selected - #1921

• Removed UI for heatmaps across surfaces in Capture (only available in Player) - #1917

• Fixed incorrect warning about deprecated surface definitions - #1923

• Prevented crashes when rendering invalid eyeball outline - #1947, #1954

• Fixed that restarting background tasks would not clear already running ones - #1939

• Fixed frozen eye videos in HMD-Eyes - #1949

• Disabled automatic reference location detection again in post-hoc gaze - #1951

• Fixed crash when opening recording with unavailable gazer - #1922

• Fixed crashes of the timeline for very short recordings - #1958

• Fixed crash when deleting a running post-hoc calibration multiple times - #1955

• Fixed crashing post-hoc gaze mapping for an HMD-Eyes recording - #1961

• Fixed Service crashing when requesting a non-existing plugin - #1962

• Fixed Capture crashing when finishing a calibration without reference data - #1960

• Prevented starting gaze mappers on calibration fail - #1963

• Fixed an issue when running Pupil Capture bundle on macOS 10.14 Mojave - #1919

Developer Notes

Customizing Screen Marker Positions - #1953

Using custom marker positions for the screen marker calibration choreography might be a common use case. Previously this was not easy to achieve with a custom plugin, as the relevant function was a private method. We have now made it public so that you can more easily implement custom calibration choreographies.

Black Formatting Checks on Travis - #1956

We are internally using Black as automatic Python formatter for our codebase. This helps us ensure that PRs only contain relevant changes and no formatting/whitespace changes. To enforce this more strictly, we added a black formatting check to our Travis integration, which will run on all commits and PRs. Please make sure to format your code with black before submitting a PR to ensure that all Travis checks pass.

Downloads

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

We are pleased to announce the release of Pupil Core software v2.0!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Motivation

With v2.0 and beyond, we will be focusing on making Pupil Core more core.

Our vision for Pupil Core emerged early on: we wanted to make eye tracking hardware and software that was accessible. Accessibility is a useful keyword for us because it encourages us to think about the diverse group of researchers that use Pupil. Pupil Core strives for the following:

• Transparency: Open source is key here. Enable researchers to introspect into the algorithms. Be explicit about the methods and biases built into the software. Inspire others to make improvements and catch bugs.

• Modularity: Make it easy for users to develop and run their own custom code - plugin architecture.

• Connectivity: Make it easy to connect to other software - network API.

• Diversity: Pupil core is flexible. Pupil Core strives to accommodate a diverse range of research and applications. Pupil Core enables researchers who want to use it as a desktop app with GUI, researchers who want to integrate with other software and sensors via Network API and plugins, and developers who want to tinker with the source code directly.

Over the years, Pupil Core software grew organically to support the community, new hardware, and novel applications. With v2.0 took a few steps back to look at the big picture and to think about the future of Pupil Core. If we want Pupil Core to continue to be a useful platform for researchers and developers, then we need to boil it down to its essentials.

How are we making Pupil Core more core? By focusing on making essential (or core) parts of Pupil Core work really well. Here are the steps we have taken with v2.0:

1. Restructuring the core pipeline: We have made changes to the application architecture that allows it to run multiple pupil detectors in parallel as well as easily add custom gaze mapping algorithms via user plugins.

2. Removing hard to maintain features: We had to make some tough decisions. We decided to remove audio capture and native Intel RealSense support. These components were too much work to maintain.

3. Simplifying the user interface: Over the years we have added many features and the UI has grown organically in the process. We have decided to streamline and improve the UI. Be it merging of menus or more sane default settings, we felt that new users need a bit more help getting started with Pupil Capture and Player.

Improvements

Parallel 2D and 3D Pupil Detection

To allow for using different pupil detection algorithms, we have decoupled our existing pipeline of 2D and 3D detection. Instead of running either of those, both will now run as separate plugins with only minimal overhead. This will enable researchers (including our own R&D team) to test, optimize and replace parts of the pipeline much more conveniently in the future.

Additionally this change has allowed for the following improvements to other parts of the system:

Pupil Diameter Timelines - #1854

With 3D pupil data always available we can always display the meaningful 3D pupil diameter in the timelines. Additionally, we added outlier removal and included the range for a more expressive visualization.

Blink Detection - #1900

With 2D pupil data always available we can run a more reliable blink detection in all circumstances.

Calibration Choreography and Gazer Plugins

By far the biggest change in this release was the refactoring of calibration and gaze mapping part of the core pipeline into separate, extendable components that are easy to swap with custom implementations.

Calibration Plugin in Pupil Capture

When running Pupil Capture, in the plugin list you can find the unified Calibration plugin, which is responsible for configuring both the calibration method (with the Choreography selector), and the gaze mapping method (with the Method selector).

Extending Calibration and Gaze Mapping with User Plugins

Now users can implement their own calibration routines and gaze mapping methods. These will automatically show up in the Choreography and Method selectors.

To do this, users must subclass CalibrationChoreographyPlugin and GazerBase respectively. See the Developer Notes section below for a code example.

Improved Installation Workflow on Windows - #1853

We have wrapped Pupil in a Windows Installer package (MSI) in order to simplify the Windows workflow.

By default, all 3 apps (Capture, Player, and Service) will be installed in C:\Program Files (x86)\Pupil-labs\Pupil v<version>\. All apps will also get a start-menu entry, making it much easier to open Pupil. Installed versions of Pupil can be removed in the Windows Uninstall Settings.

New versions of Pupil will be installed alongside older versions and you can choose which to start. Switching versions will still overwrite your user settings as previously.

Changes

Fixation and Blink Detection by Default - #1870

In order to make Pupil Core even more useful with out-of-the-box settings, we have enabled the Fixation Detector and Blink Detector by default.

Simplified System Graphs - #1869

We have removed the configuration menu for the System Graphs plugin in order to simplify the user interface. The optional display of live graphs for the pupil diameter has been removed as well.

The confidence graphs will now always display the confidence of the 2D pupil detection, as this is a better indicator for correct positioning of the Pupil Core eye tracking headset.

Single Marker Calibration: Physical Markers - #1897

The single marker calibration is mostly intended for calibration with physical (printed) calibration markers, as opposed to the screen marker calibration. For sake of clarity, we changed the default mode for the single marker calibration to Physical Marker (previously this was called Manual).

Removed Manual Marker Calibration

With the restructuring of the calibration workflow, we have removed the deprecated Manual Marker Calibration. The Single Marker Calibration is a more general version and additionally allows for VOR-based choreographies. If you previously used the Manual Marker Calibration, try the Single Marker Calibration in Physical Marker mode (the new default)!

Parallel Pupil Detector Visualization - #1873

We have changed the color of the 2D ellipse visualization in all eye videos to blue in order to differentiate between 2D and 3D detection. You will now always see 3 ellipses on the eyes when the visualization is enabled:

Color Description

• blue: 2D pupil detection result

• red: 3D pupil detection result

• green: 3D eye model

Flip Eye 0 Display - #1844

The eye 0 (right eye) camera sensor is physically flipped, which results in an upside-down eye image for the right eye. This is by design and does not negatively affect pupil detection or gaze estimation. However, the upside-down eye 0 image repeatedly led users to believe that something was broken or incorrect with Pupil Core headsets. We flipped the eye 0 image now by default to better match user expectations with a right-side-up eye image.

Deprecated Audio Capture - #1868

Recording synchronized audio on 3 different operating systems (macOS, Linux, Windows) has always been a challenge for Pupil Capture. (See our motivation section above.)

Unified Network API Plugin - #1881

The previous plugins Pupil Remote and Frame Publisher were commonly used to interact with Pupil Core's Network API. For the sake of simplicity and clarity, we merged both plugins into the new Network API plugin.

Removed Blink Detector Visualization - #1901

The visualization of the blink detector (brief dark flashing on the screen) could interfere with other plugins and cause visual bugs. We have removed this feature.

Bug Fixes

• Fixed Player crashing when exporting while computing gaze history - #1843

• Fixed Player crashing when disabling the Surface Tracker while exporting - #1848

• Fixed wrong default camera selection in some cases - #1857

• Fixed a rare crash in the Surface Tracker - #1879

• Fixed deprecated RealSense cameras showing up in the camera selection - #1877

• Fixed that offline gaze mappers without data would show as completed - #1889

• Fixed corrupted gaze data when deactivating one of multiple gaze mappers - #1888

• Fixed new offline pupil data not being correctly announced sometimes - #1890

• Fixed a few small issues in offline pupil detection - #1899

• Fixed problems with opening the macOS bundles on newer versions of macOS

Developer Notes

Changed Requirements

Scikit-learn

We have added scikit-learn as a dependency for Pupil.

pip install scikit-learn

PyAV - 1908

We updated the PyAV dependency to v0.4.5, which includes some bug fixes. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/PyAV

On Windows, download and install the latest wheel from GitHub.

Changed data format

Both 2D and 3D pupil detector now publish their results independently of each other. To differentiate between them, we have added the origin to the topic. For a binocular setup, you will now get 4 different pupil topics:

• pupil.0.2d

• pupil.0.3d

• pupil.1.2d

• pupil.1.3d

This means that there can now be multiple pupil datums for one eye for the same timestamp (2D and 3D). When grouping by the entire topic however, the timestamps are still guaranteed to be unique.

Restarting Unique Plugins - #1871

We have added the previously missing feature to restart plugins with by_class uniqueness via notifications.

Recording Version 2.2 - #1885

The info.player.json meta-version has been bumped to 2.2 will be incompatible with Pupil Player version v1.

Allow specifying ZMQ HWM option - #1895

We added the option to start the eye processes with custom values for the ZMQ High Water Mark (HWM) via notification.

Custom Calibration Plugins

Below is an example with minimal implementation for a custom HMD 3D calibration choreography and gaze mapping method:

# To implement a custom calibration or gaze mapping method from scratch, subclass GazerBase and CalibrationChoreographyPlugin
from gaze_mapping.gazer_base import GazerBase
from calibration_choreography import CalibrationChoreographyPlugin
# Another option is to modify existing implementations by subclassing existing implementations
from gaze_mapping import Gazer3D
from calibration_choreography import ScreenMarkerChoreographyPlugin
class CustomGazer3D(Gazer3D):
    # Must be unique across all subclasses of GazerBase
    label = "Custom 3D Gazer"
class CustomChoreographyPlugin(ScreenMarkerChoreographyPlugin):
    # Must be unique accross all subclasses of CalibrationChoreographyPlugin
    label = "Custom Calibration"
    @classmethod
    def supported_gazer_classes(cls):
        return [CustomGazer3D]

Release Note Updates

17.06.2020 - v2.0-175

• Added compatibility for the HMD-Eyes VR integration

• Fixed an issue that prevented the validation from working correctly

24.06.2020 - v2.0-177

• Fixed an issue that prevented using the head pose tracker and creating new annotations in Player - #1927

25.06.2020 - v2.0-182

• Fixed an issue that prevented Pupil Service from starting - #1928

Downloads

Officially minimum supported operating system versions for the bundles:

Ubuntu: 16.04macOS: 10.13 (High Sierra)Windows: 10

To open the RAR-archive on Windows, you will need a decompression software, e.g. WinRAR.

We are pleased to announce the release of Pupil v1.23!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Improvements

Windows Driver Installation Stability - #1833

We have greatly improved the driver installation workflow on Windows. You should not experience problems with spaces or non-english characters in the path to your Pupil installation anymore. Additionally, you do no longer have to run the application as administrator. You still need an administrator to authorize the driver installation.

Changes

On-demand Loading of Multi-part Recordings - #1835

Multiple users have reported problems when opening Pupil Mobile recordings with a large number of video parts. We were able to fix this issue by loading the video parts on demand instead of opening them all at once. As a result, Pupil Player will open Pupil Mobile or Pupil Invisible recordings with multiple parts much quicker than before.

As a side-effect, during playback in Pupil Player, you might experience short frame jumps when a new video part needs to be loaded. This does not affect the underlying data or any exports.

Bug Fixes

• Fixed a crash when changing the ROI without a connected headset - #1827

• Fixed a crash when opening recordings with empty video parts - #1832

• Gracefully stop recordings when receiving non-monotonic timestamps - #1829

• Fixed an issue with calibration file storage on macOS - #1836

• Fixed an issue with gaze history settings restoration from session settings - #1836

Developer Notes

Changed Requirements - #1834

We updated the pyndsi dependency on v1.3. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/pyndsi

On Windows, please download and install the latest wheel from GitHub.

Removed All Direct C++ Dependencies - #1828

After the latest rework of the calibration codebase to remove C++ dependencies in v1.22, we have re-evaluated all dependencies for Pupil Core software -- especially on Windows. We are very pleased to announce that we have finally externalized or gotten rid of all C++ dependencies! :tada:

This greatly simplifies the developer setup on Windows. The biggest changes include:

• You no longer have to install Visual Studio to run from source

• You no longer need to download OpenCV

• You are no longer bound to a fixed path C:\work

We have updated the developer docs with the updated setup instructions. We hope that this simplifies your setup and enables you to develop awesome plugins for Pupil Capture and Pupil Player that can be shared with the community!

Release Note Updates

30.03.2020 14:00: Fix driver installation issue in Windows bundle - #1842

In Pupil v1.23-4, we fixed an issue that prevented users from installing the camera drivers on Windows.

06.04.2020 11:50: Include missing libuvc.dylib in macOS bundle

In Pupil v1.23-5, we have included missing library files for Pupil Player and Pupil Service on macOS.

08.04.2020 17:00: Include missing libusb.dylib in macOS bundle

In Pupil v1.23-5-fix0, we have included missing library files for Pupil Player and Pupil Service on macOS.

24.04.2020 14:00: Improve driver installation stability on Windows - #1856

In Pupil v1.23-10, we have made changes to the driver installation procedure that ensure a successful installation on a larger variety of Windows setups.

We are pleased to announce the release of Pupil v1.22!

Download the latest bundle (scroll down to the end of the release notes and see Assets).

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Features and Changes

Video Source Rework - #1792

We have reworked the video source input system to greatly improve stability, user experience, and maintainability of the related features:

Single Plugin UI

All previous plugins for managing the video source (managers and sources) have been unified under a single plugin menu: Video Source, which handles device/camera selection as well as adjusting camera settings.

Automatic vs Manual Camera Selection

The new source system offers two ways for selecting cameras: automatic and manual:

• In automatic mode (default) you can activate source devices, e.g. Local USB or Pupil Mobile. Activating a device selects the best matching available cameras as input for all windows in Pupil Capture.

• Enabling manual mode allows you to select cameras individually for every window.

UI Cleanup

The following non-essential user interface elements have been removed to provide a better overall user experience:

• The UI for selecting a file source has been removed. Starting a file source via network notification or via drag-and-drop is still possible.

• Manually selecting HMD-Eyes as input source has been removed, as the plugin is started from Unity via network notification.

• The Capture plugin UI in the eye windows has been removed entirely from Pupil Player as the displayed information was incomplete and the interactivity was limited.

• The Test Image source has been removed as it was only intended for development use.

Removed Built-in RealSense Video Backend

Intel RealSense sensors are designed as a prototyping/development platform. Intel deprecates RealSense hardware frequently. Driver support is appropriate for a prototyping tool, but it is not sufficient for a production ready tool.

We have tried to build our tools around Intel's changing APIs and patchwork driver support, but this became difficult and confusing for users.

You can still use RealSense Video Backend as a custom plugin. You can load it directly in the plugin folder of Pupil Capture. Check it out here.

Player: Add Gaze History to Polyline Visualization - #1785

We decided to use the Polyline Visualization as a replacement for the previously removed Scan Path Plugin. You can now visualize past gaze points up to a certain duration by adjusting the duration slider in the Vis Polyline Plugin. When first dragging the slider, the history for all frames will be computed, which will take some time. Afterwards you can see the gaze history and adjust the duration instantly.

Capture: Improved Region of Interest (ROI) selection - #1788

The ROI selector for the eye windows has been improved for better stability and better user experience. In doing so, we standardized the format of ROI boundaries across Pupil to the following order: (minx, miny, maxx, maxy). Please note that this changes the format for adjusting the ROI via network messages.

Player: Add 3D Eyeball Visualization - #1810

We have added the green circle overlay for the 3D eyeball to Player. The overlay will be added to the Eye Overlay and Eye Video Exporter plugins, when 3D pupil data is available. You can disable the visualization (including the pupil outline) via the corresponding plugin menus.

Bug Fixes

• Changed Windows disconnection log-level from warning to debug - #1786

• Fixed some of small bugs in the ROI selector - #1788

• Fixed mixed up export columns for monocular eye_center and gaze_normal - #1805

• Prevent a crash caused by incorrect UVC timestamps - #1812

Developer Notes

Getting Rid of C++ Dependencies - #1789

We have made the next step towards an easier installation process of Pupil. After the externalization of the Pupil Detectors in v1.20, we have been able to replace the last remaining C++ module with a Scipy re-implementation. This means that you do not have to compile any Python Extension Modules when starting Pupil.

As a result, you can now run Pupil from source on Windows without having to go through the trouble of setting up CERES. While the pupil-detectors still depend on CERES, we offer prebuilt wheels for Windows that you can install without having to set up the dependencies.

Changed Requirements

We updated the pyglui dependency to v1.27. Please update with:

pip install -U git+https://github.com/pupil-labs/pyglui

We Are Hiring Python & DevOps Engineers!

Hey - you're reading the developer notes, so this is for you! We're looking to hire developers to contribute to Pupil source code. If you love Python and enjoy writing code that is a joy to read, get in touch. Experience with the scientific Python stack is a plus, but not required. We have a lot of exciting projects in the pipeline.

We are also looking for Senior Full Stack/DevOps engineers that have experience with kubernetes, docker, and server-side Python.

Send an email to jobs@pupil-labs.com with a CV to start a discussion. We look forward to hearing from you.

Release Notes Update

25.02.2019 v1.22.3

• #1817 - Fixed a regression which caused all frames to be dropped after changing Pupil Capture's time base.

27.02.2019 v1.22.7

• Commit bcfbf7d - Fixed a typo that caused the v1.22.3 fix to not work correctly.

We are pleased to announce the release of Pupil v1.21!

Download the latest bundle (scroll down to the end of the release notes and see Assets). Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Features and Changes

Support for Pupil Invisible Audio Recordings - #1748

Pupil Player now supports audio playback and audio export for Pupil Invisible recordings with audio recordings.

Stream Head Pose over Network - #1774

You can now subscribe to the the topic head_pose to receive live data from the Online Head Pose Tracker plugin.

Bug Fixes

• Fixed eye processes always displayed the algorithm view - #1766

• Fixed potential crash when changing the eye video size - #1770

• Made pyrealsense warning a debug-level message in order to be less obtrusive - #1768

• Fixed broken interactivity of the ROI mask in the eye windows - #1778

• Fixed multiple stability issues with the fullscreen calibration window - #1767

Developer Notes

Changed Requirements

We updated the PyAV dependency to v0.4.4 for the PI audio support. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/PyAV

On Windows, download and install the latest wheel from GitHub.

We Are Hiring Python & DevOps Engineers!

Hey - you're reading the developer notes, so this is for you! We're looking to hire developers to contribute to Pupil source code. If you love Python and enjoy writing code that is a joy to read, get in touch. Experience with the scientific Python stack is a plus, but not required. We have a lot of exciting projects in the pipeline.

We are also looking for Senior Full Stack/DevOps engineers that have experience with kubernetes, docker, and server-side Python.

Send an email to jobs@pupil-labs.com with a CV to start a discussion. We look forward to hearing from you.

We are pleased to announce the release of Pupil v1.20!

Download the latest bundle (scroll down to the end of the release notes and see Assets). Please feel free to get in touch with feedback and questions via the #pupil channel on Discord 😄

Features and Changes

Deprecated Fingertip Calibration - #1753, #1757

We removed the fingertip calibration method.

The fingertip calibration method was rarely used and provided lower accuracy compared to other calibration methods. Additionally, dependencies needed for the fingertip detection required some work to set up correctly and accounted for almost 50% of the size of our application bundles.

Bug Fixes

• Fixed a crash when trying to open empty (but existing) recordings - #1751

• Fixed a crash on Windows when missing the latest Visual Studio redistributables - #1756

• Fixed visualization of circle markers in offline calibration for recordings with gaps - #1758

• Fixed broken combination of --hide-ui and --profile CLI flags - #1760

Developer Notes

Externalized Pupil Detectors - #1642

We have extracted the Pupil Detectors into their own repository: https://github.com/pupil-labs/pupil-detectors

This will make it very easy to use our pupil detectors in any standalone application or experimental setup without having to deal with all of Pupil's intricacies.

Additionally this will make it much easier to run Pupil from source (especially on Windows), since we are distributing pupil-detectors as package via PyPI and even provide precompiled binaries for Windows!

OpenGL Debugging - #1752

We occasionally receive crash reports involving error messages from PyOpenGL, which we cannot reproduce on our machines. We wrapped the PyOpenGL errors to prevent crashes and log additional messages in these cases. This can result in visual UI errors (instead of a crash) when there are OpenGL issues. If you notice any weird UI behavior, please save the log file and report back to us.

We Are Hiring Python & DevOps Engineers!

Hey - you're reading the developer notes, so this is for you! We're looking to hire developers to contribute to Pupil source code. If you love Python and enjoy writing code that is a joy to read, get in touch. Experience with the scientific Python stack is a plus, but not required. We have a lot of exciting projects in the pipeline.

We are also looking for Senior Full Stack/DevOps engineers that have experience with kubernetes, docker, and server-side Python.

Send an email to jobs@pupil-labs.com with a CV to start a discussion. We look forward to hearing from you.

We are pleased to announce the release of Pupil v1.19!

Download the latest bundle (at the end of the release notes under Assets) and let us know what you think via the #pupil channel on Discord 😄

Features and Changes

Improved Fixation Detector - #1743

We made improvements to the Fixation Detector to improve detection accuracy:

• We were able to get rid of large gaps in between detected fixations. Now there will be more sequential fixations detected.

• We decreased the false-positive detection rate by omitting low-confidence data.

• We fixed a bug that would sometimes result in wrong fixations on 3D gaze data.

API Changes

Due to the improvements mentioned above, the Fixation Detector now requires calibrated gaze in both 2D and 3D mode. To reflect these changes, the method field of a fixation datum now contains either 2d gaze or 3d gaze (previously pupil vs gaze). This effects both the real-time fixations published through the network API and the fixations.csv export file.

Removed Fixation-Boosted Calibration - #1744

Since the Fixation Detector now requires calibrated gaze, it will not be used anymore to speed up the calibration process.

Improved Binocular Gaze Mapping - #1731

We improved the mapping of monocular gaze data streams by using a dynamic windows size dependent on the frame rate. This will result in better temporal accuracy when matching binocular gaze data for eyes with frequent low-confidence phases.

We also investigated some other variants of binocular gaze mapping, but chose to stick with this simple improvement. For a full evaluation and discussion, see this jupyter notebook.

Bug Fixes

• Add/remove markers for legacy square markers works again - #1737

• Fixed some bad UX with switching surface detection modes - #1738

• Player is now able to load recordings with square brackets in the name - #1740

• Fixed a crash when setting 2D detector properties via network API - #1742

• Improved inconsistent background process behavior - #1741

Developer Notes

Automated Testing - #1721, #1724, #1730, #1735

We have done some work to consolidate old tests in the codebase and set it all up nicely with pytest. You can run the test suite from the repository root with:

pip install pytest
pytest

We set up a travis-ci integration to run pytest on all commits and pull requests. Now you will get feedback if you break something 😄 The test coverage is currently low. But from this point on, we - and you - are encouraged to write tests when adding new features and when fixing bugs.

We Are Hiring Python & DevOps Engineers!

Hey - you're reading the developer notes, so this is for you! We're looking to hire developers to contribute to Pupil source code. If you love Python and enjoy writing code that is a joy to read, get in touch. Experience with the scientific Python stack is a plus, but not required. We have a lot of exciting projects in the pipeline.

We are also looking for Senior Full Stack/DevOps engineers that have experience with kubernetes, docker, and server-side Python.

Send an email to jobs@pupil-labs.com with a CV to start a discussion. We look forward to hearing from you.

Release Notes Update

27.11.2019 17:00: v1.19.2

We have fixed an issue in v1.19.0 where multi-part Pupil Mobile and Pupil Invisible recordings were not displayed correctly in Pupil Player - #1747. If you have opened such a recording in Pupil Player v1.19.0 then delete all *_lookup.npy files and reopen the recording in Pupil Player v1.19.2.

We are pleased to announce the release of Pupil v1.18!

Download the latest bundle and let us know what you think via the #pupil channel on Discord 😄

Features and Changes

Player: Deprecated Recordings - #1711

We have removed support for opening recordings made with:

• Pupil Core < v1.3 or

• Pupil Mobile < r0.21.0

If you need to open a recording made with one of these versions, you will have to open it once in Pupil Player v1.16 or v1.17. This will upgrade the recording format to the new version. You can find more information about this in the v1.16 release notes.

Surface Tracker Export: Distorted Image <-> Surface Coordinates - #1678

We added the homographies to the surface tracker export, which can be used to convert pixel positions between surface coordinates and coordinates of the original camera image. You can use these to crop the surface from the distorted scene image, and get something similar to our surface debug view.

The homographies are exported as two additional columns in the surf_positions_<name>.csv export file:

• dist_img_to_surf_trans

• surf_to_dist_img_trans

Exporting Frame PTS - #1682

We added a column pts to the timestamps export in <name>_timestamps.csv. This information can be used to seek to and retrieve single frames from the exported video file without the need for sequentially iterating over the entire video.

Service: Start Binocular - #1714

For consistency with Pupil Capture, Pupil Service will now start in binocular mode.

Pupil Remote Plugin: Port No Longer Saved - #1715

The selected port for the Pupil Remote Plugin will no longer be stored in the Capture/Service settings. The default port is always 50020. If the port is already in use, Capture will choose a random other port, while Service will shut down again. You can choose a different port on startup with the -P command line flag.

Bug Fixes

• Eye movements overlayed in Player will now be correctly exported - #1675

• Shutting down eye processes while recording will correctly finish the recording - #1677

• Fixed some UX issues with the Head Pose Tracker visualization window - #1669

• Improved performance of apriltag detection for the Head Pose Tracker - #1669

• Fixed Pupil Service UI showing port None - #1715

• Pupil Service will now be shut down correctly if the selected port is already in use - #1716

• Windows run*.bat files will now correctly forward CLI arguments - #1717

• Fixed Pupil Invisible recordings displaying erroneous gaze positions at (0,0) - #1719

Developer Notes

Eye-Process Plugin List - #1674, #1681

The Eye processes are now using the Plugin_List class that we are already using in Capture, Player and Service. It takes care of managing the plugin lifecycle and makes it easier to add, remove or change plugins on the Eye process.

Command Line Interface Overhaul

We improved the handling of the Command Line Interface (CLI) of Pupil apps and added some additional start flags:

App Arguments Only - #1709

We changed the general structure of Pupil command line invocation to:

python main.py [-h] {Capture,Player,Service} [app-specific-arguments]

For the full list of arguments, see the README of the Pupil repository.

main.py takes no arguments except -h/--help; resulting in an info message. The <app> argument has been made required and will not default to Capture.

Version Flag - #1679

You can start any Pupil App with the --version flag. This will display the current version and exit the program afterwards.

Hide UI Flag - #1712

You can now start Pupil Capture and Pupil Service with the --hide-ui flag. This will suppress all UI creation while running the program. Use remote messages to control the behavior of the program. To stop the app gracefully, you can either send a SIGTERM signal or (when running via console) hit CTRL-C to send a SIGINT signal (see point below for additional info).

CTRL-C Shutdown - #1708, #1713

We added support for gracefully shutting down all apps (Capture, Player, Service) via hitting CTRL-C in the console from where it is running. All running operations and processes will be correctly finalized and terminated.

Recording Version 2.1

The info.player.json meta-version has been bumped to 2.1. This was necessary to reverse the effects of a bug that corrupted some recordings (#1719).

We Are Hiring Python & DevOps Engineers!

Hey - you're reading the developer notes, so this is for you! We're looking to hire developers to contribute to Pupil source code. If you love Python and enjoy writing code that is a joy to read, get in touch. Experience with the scientific Python stack is a plus, but not required. We have a lot of exciting projects in the pipeline.

We are also looking for Senior Full Stack/DevOps engineers that have experience with kubernetes, docker, and server-side Python.

Send an email to jobs@pupil-labs.com with a CV to start a discussion. We look forward to hearing from you.

Release Note Updates

06.11.2019 11:10: Fix opening Pupil Mobile recordings - #1722

In Pupil v1.18-4, we fixed an issue that prevented users from opening newly created Pupil Mobile recordings in Pupil Player.

11.11.2019 13:10: Fix Offline Pupil Detection on Windows - #1727

In Pupil v.1.18-35, we fixed an issue that caused Pupil Player to crash when running the Offline Pupil Detection on Windows.

We are pleased to announce the release of Pupil v1.17!

Download the latest bundle and let us know what you think via the #pupil channel on Discord 😄

Features & Changes

Surface Tracker Updates

We have fixed stability issues with surfaces defined with AprilTag markers. See issue #1696 for more information.

We now support more AprilTag marker types (#1700)! Checkout the docs for more info about tag types and settings.

Legacy Square Markers

If you have recordings with Legacy Markers, and would like to use the latest release to redefine surfaces, please get in touch with us via chat.

Heatmap Image Export Size - #1702

Heatmap images are now exported with a minimum size of 2000px. This will make it easier to use them out of the box without having to upscale them manually.

Streaming Pupil Invisible into Pupil Capture- #1693

You can now stream live video and gaze data from Pupil Invisible to Pupil Capture!

Streaming Scene Video

Select Pupil Mobile backend in Pupil Capture's Backend Manager. Select your Companion Device as Remote Host. Note that the Pupil Invisible Companion Device and Pupil Capture need to be connected to the same network and recording is not supported.

Streaming Gaze Data

Install the PI Preview Plugin.

Removed NSLR Eye-Movement Classifier - #1698

We found that the library did not perform well enough to be usefull and removed it from Pupil Player and Capture. Let us know if you need it back!

Bug Fixes

• Fixed Eye Movement Visualizations not being exported every time - #1675

• Fixed broken recordings when closing the eye windows while recording - #1677

• Fixed inaccurate UI when editing surface corners - #1692

• Fixed unresponsive UI for the gaze history length of online heatmaps - #1697

• Fixed an error with loading some Pupil Invisible recordings - #1704

• Fixed that surfaces would not get saved to recordings when they had never been modified - #1706

Developer Notes

We Are Hiring Python Developers!

Hey - you're reading the developer notes, so this is for you! We're looking to hire developers to contribute to Pupil source code. If you love Python and enjoy writing code that is a joy to read, get in touch. Experience with the scientific Python stack is a plus, but not required. We have a lot of exciting projects in the pipeline.

We are also looking for Senior Full Stack/DevOps engineers that have experience with kubernetes, docker, and server-side Python.

Send an email to jobs@pupil-labs.com with a CV to start a discussion. We look forward to hearing from you.

We are pleased to announce the release of Pupil v1.16!

Download the latest bundle and let us know what you think via the #pupil channel on Discord 😄

Features

Pupil Invisible Support - #1596, #1623

Preparing for the upcoming public release of Pupil Invisible (PI), we are adding support for recordings made with PI in Pupil Player.

You will be able to download the raw recording from Pupil Cloud or copy them directly from the Companion device. Drag and drop the recordings onto Pupil Player as usual.

Most features of Player will support recordings made with PI. Currently not supported are: The iMotions exporter and blink detector.

Surface Tracker

Removed Robust Detection from SurfaceTracker - #1600

The robust detection mode was a remnant of the legacy surface marker system. The new Apriltags markers are by default more robust than even the old robust detection mode. This option did not have an effect anymore, so it was removed.

Bugfixes

Disabled Hardware Timestamps on Windows Operating System - #1621

In the past we have received multiple reports of timestamps being out of sync on Windows. After some testing we discovered that the timestamps from the hardware were not always reliable on Windows. The exact cause of this is still unknown but we have disabled the use of hardware timestamps on Windows for now. We are now using software timestamps with an appropriate offset-mapping.

Improved Eye Overlay Visualizations - #1609

Previously, the Eye Overlay plugin would always display the closest eye frame to the current playback position. Therefore, in cases where the eye camera disconnected during a recording, the eye video was visualized incorrectly. Instead, now, we show a gray frame to indicate missing data.

Smaller Fixes

• Fixed minor performance issues in Apriltag surface marker detector

• Fixed broken SurfaceTracker heatmap rendering when no gaze data is available - #1584

• Fixed SurfaceTracker timeline drawing performance issues - #1595

• Fixed issues with EyeMovementDetector when using inconsistent 3D gaze data - #1587

• Fixed UI/UX issues when editing surfaces in SurfaceTracker - #1591

• Fixed flickering surface markers - #1600

• Updated the app icons for the Windows bundles to the latest design

• Fixed incorrect handling of missing projection matrices for HMD - #1603

• Fixed performance-related crashes of the SurfaceTracker on Windows - #1605, #1606

• Improved user feedback when opening invalid recordings - #1585

• Fixed crashing offline detection in recordings with timing irregularities - #1615

• Improved user feedback when trying to load an unknown plugin - #1613

• Fixed a crash when trying to use Pupil Invisible with Capture - #1620

• Fixed a crash when setting color values for CircleViz to 1.0 - #1624

• Improved user feedback when recording from Pupil Mobile without TimeSync - #1629

Deprecated Recordings

Previously released versions of Pupil Player v1.16 do not support recordings created with

• Pupil Capture v1.2 or earlier, and

• Pupil Mobile r0.22.x or earlier.

This was due to the fact that these recordings are missing meta information that is required for the upgrade to our new Pupil Player Recording Format 2.0. For details see "Missing Meta Information" below.

With this PR, instead of aborting with a "Recording too old" warning, Pupil Player will attempt to infer the missing values from other data in the recording. This inference is not perfect and results in imprecise values and might cause issues when converting recorded Pupil time to wall clock time. For details see "Missing Meta Information" below.

Upgrading will only work in Player v1.16

Since this inference is limited, we will remove the upgrade functionality for deprecated recordings in v1.17. This means, that if you want to open a deprecated recording in v1.17 or later, you will have to convert them to the new format using v1.16 first.

Missing Meta Information

Specifically the following keys in the info.csv file are missing:

• Start Time (System): Recording start time in Unix epoch

• Start Time (Synced): Recording start time in Pupil time

Pupil Player assumes that these timestamps were measured at the same time. This allows for the after-the-effect synchronization of Pupil data with externally recorded data that uses the Unix epoch.

Their value can be inferred from the existing recording, given a roughly known imprecision:

• Start Time (System): The system start time can be inferred using the existing Start Date and Start Time fields (precision: one second). Unfortunately, the Start Time is subject to the system's timezone while Start Time (System) is not. Therefore, its inference is off by the recording's amount of timezone offset.

• Start Time (Synced): This timestamp can be replaced by the earliest known recorded video timestamp in the given recording. In this case, the inference error depends on the startup delay of the cameras. This delay is typically less than a second for Pupil Capture recordings, and 2-5 seconds for Pupil Mobile recordings.

Incorrect Playback

In some cases, deprecated recordings might not be played back correctly in Pupil Player. If this is the case for you, please contact info@pupil-labs.com for support.

Developers Notes

Changed Requirements

• Updated pyndsi dependency to v1.0

pip install git+https://github.com/pupil-labs/pyndsi -U

• Added packaging dependency

pip install packaging

Pupil Player Recording Format 2.0

Variable Framerate Videos

Instead of writing videos with equally distant presentation timestamps, Capture now saves videos with their original timing. This means, that newly recorded videos can be played back correctly in VLC without having to export them first with Pupil Player.

Recording Info Files

Due to slight differences in how Capture, Mobile and Invisible store recording meta info, we introduced a unified recording meta info file for Pupil Player: info.player.json. When opening a recording made with Pupil Mobile, Pupil Invisible, or older versions of Pupil Capture, we are transforming the recording into the new format (as we did before already) and aggregate all necessary meta infos in info.player.json without changing the original meta info file. The original meta info files will be renamed in order to uniquely identify the origin of this meta info:

• Older versions of Pupil Capture*: info.csv -> info.old_style.csv

• Pupil Mobile: info.csv -> info.mobile.csv

• Pupil Invisible: info.json -> info.invisible.json

*Note that e.g. when a Pupil Mobile recording has been opened in an older version of Pupil Player before, we also rename to info.old_style.csv as the original info.csv will have been adjusted by Player before.

You can read about the exact specification of the new info.player.json format here.

Pretty Plugin Class Names - #1597

Custom plugin classes don't have to rely on underscores in their names anymore for displaying them nicely in the UI. You can now name your plugin classes in accordance with PEP 8 - Class Names:

class MyAwesomePlugin(Analysis_Plugin_Base):
    @classmethod
    def parse_pretty_class_name(cls) -> str:
        return "Awesome Plugin"

Note that currently all existing pupil classes (e.g. Analysis_Plugin_Base) keep their names with underscores for backward compatibility.

We Are Hiring Python Developers!

Hey - you're reading the developer notes, so this is for you! We're looking to hire developers to contribute to Pupil source code. If you love Python and enjoy writing code that is a joy to read, get in touch. Experience with the scientific Python stack is a plus, but not required. We have a lot of exciting projects in the pipeline.

We are also looking for Dev Ops engineers that have experience with kubernetes, docker, and server-side Python.

Send an email to jobs@pupil-labs.com with a CV to start a discussion. We look forward to hearing from you.

Release Note Updates

17.09.2019 22:40: Missing Microsoft Visual C++ Redistributable

We have received reports from two users that Pupil Capture failed to start on Windows with the following error message:

ImportError: DLL load failed: The specified module could not be found.

In both cases the "Microsoft Visual C++ Redistributable for Visual Studio" was missing on their system. If you should encounter the same error message, please download and run the vc_redist.x64.exe file from the official Microsoft support page. Afterwards, Pupil should start up as expected.

23.09.2019 17:22: Additional Bug Fixes

The following issues have been fixed in Pupil v1.16-43:

• Remove remaining dependencies on the info.csv file - #1652

• Fix frameless/worldless recordings in Pupil Capture - #1653

30.09.2019 15:30: Offline Calibration Bug Fix

The v1.16-71 release fixes #1656 Player will not create/ allow edits on new calibrations.

02.10.2019 15:35: Deprecated Recordings and Offline Calibration Bug Fix

The following issues have been addressed in Pupil v1.16-80:

• Deprecated Recordings - #1668: Recordings made with earlier versions than Capture v1.3 or Pupil Mobile r0.21.0 have been deprecated. Please see the dedicated section above for more information.

• Fixed issue with creating offline calibrations - #1667

We are pleased to announce the release of Pupil v1.15!

Download the latest bundle and let us know what you think via the #pupil channel on Discord 😄

Features

Apriltags support for Surface Tracking

Starting with v1.15, the Surface Tracker will detect apriltags instead of "square markers".

We highly recommend to replace "square markers" with apriltags in your setup since their improved detection performance translates to gaze being mapped to surfaces with higher accuracy and surfaces being detected more reliably, especially during movements.

You can still use the "square markers" by setting the Marker Detector Mode to Legacy square markers. This is especially useful, if you want to process recordings that were made previous to this release.

Apriltags version 3 and support for Windows

We started to maintain our own fork of @duckietown's apriltags Python bindings for @AprilRobotics' AprilTag 3 library. Forking the projects was mostly necessary in order to support them on Windows.

This change allows us to finally enable the Head Pose Tracker feature (initially released in v1.12) on Windows!

Freeze the current 3d eye model

You now have the option to freeze the current eye model. Doing so will prevent any changes to the current eye model and discard any alternative models that might have built up in the background.

Warning: Freezing the eye model will disable its ability to compensate for any kind of slippage.

Bugfixes

• Drop invalid frame after disconnect - #1573

• Revert change that caused eye process to crash when minimizing

Developers notes

Changed requirements

Please install apriltags via

pip install git+https://github.com/pupil-labs/apriltags

API changes

Improved fixations_on_surface export

A fixation is based on gaze data that full-fills the maximum-dispersion-minimum-duration-criterion. We use it to infer e.g. fixation's position. When mapping a fixation to a surface, we use the surface's homography to calculate the fixation's position in surface coordinates. The surface's homography is calculated based on the detected surface markers for a given world frame. In most cases, the fixation appears during multiple world frames, i.e. there are multiple surface homographies which could be used to map the fixation to the surface. Until now, we only exported fixations for the first frame during which the fixation appeared. This is not a problem, as long the surface does not move in relation to the world camera during the fixation. Since this is an assumption, that does not always hold true, we will export all fixation mappings starting with v1.15. This means, that if a fixation spans multiple world frames during which a surface was detected, we will export the fixation and its position for each of these world frames.

Therefore, we added two new columns, world_timestamp and world_index, to identify the world frame which was used to map the fixation.

Eye model freezing

You can un/freeze the 3d eye models via a notification:

{
    "subject": "pupil_detector.set_property.3d",
    "name": "model_is_frozen",
    "value": True,  # set to False to unfreeze
    # remove this line if you want
    # to freeze the model both eyes
    "target": "eye0",
}

See this example on how to use the Pupil Detector Network API.

Change pupil detection ROI by notification - #1576

You can change the pupil detector's region of interest via a notification:

{
    "subject": "pupil_detector.set_property",
    "target": "eye0",
    "name": "roi",
    # value: (minX, maxX, minY, maxY)
    "value": [0, 100, 0, 200],
}

HMD-Eyes Streaming - #1501

This new video backend allows to stream a virtual Unity camera to Pupil Capture.

HMD-Eyes Streaming is an experimental feature for the hmd-eyes project and can only be activated via a network notification.

More news on this topic will follow soon over at the hmd-eyes project!

We are hiring Python developers!

Hey - you're reading the developer notes, so this is for you! We're looking to hire developers to contribute to Pupil source code. If you love Python and enjoy writing code that is a joy to read, get in touch. Experience with the scientific Python stack is a plus, but not required. We have a lot of exciting projects in the pipeline.

We are also looking for Dev Ops engineers that have experience with kubernetes, docker, and server-side Python.

Send an email to jobs@pupil-labs.com with a CV to start a discussion. We look forward to hearing from you.

Release Updated

Update: Versions v1.15-65 and higher include multiple fixes for surface tracking. Please update Pupil if you have been using versions v1.15-4 or v1.15-5.

Update 2019-08-27 10:07: Please find the macOS release attached below.Update 2019-09-05 12:05: We updated the Windows bundle to include fixes for the HMD Streaming backend.

We are pleased to announce the release of Pupil v1.14! This update primarily contains bugfixes and optimizations (no new features).

Download the latest bundle and let us know what you think via the #pupil channel on Discord 😄

Bugfixes

• Surface Tracker stability improvements - #1547, #1555

  • Includes re-introduction of the fixations_on_surface_<name>.csv export

• Head Pose Tracker stability improvements - #1561, #1562, #1569

• Timeline visualization improvements - #1563, #1565

• Calculate eye movement segment frame indices - #1539

• Work around negative time-jump issue on Windows - #1551

• Correctly disable Stripe Detector for Pupil Cam3 - #1554

• GLFW: Raise with detailed error message - #1559

Developers notes

Changed Requirements

We removed our dependency on boost !

We are hiring Python developers!

Hey - you're reading the developer notes, so this is for you! We're looking to hire developers to contribute to Pupil source code. If you love Python and enjoy writing code that is a joy to read, get in touch. Experience with the scientific Python stack is a plus, but not required. We have a lot of exciting projects in the pipeline.

We are also looking for Dev Ops engineers that have experience with kubernetes, docker, and server-side Python.

Send an email to jobs@pupil-labs.com with a CV to start a discussion. We look forward to hearing from you.

We are pleased to announce the release of Pupil v1.13!

Download the latest bundle and let us know what you think via the #pupil channel on Discord 😄

Features

Eye Movement Detector - #1512, #1516, #1519

Our new eye movement detector is based on Naive Segmented Linear Regression and classifies fixation, saccade, post-saccadic oscillation, and smooth pursuit movements.

Event identification is based on segmentation that simultaneously de-noises the signal and determines event boundaries. The full gaze position time-series is segmented into an approximately optimal piecewise linear function in O(n) time. Gaze feature parameters for classification into fixations, saccades, smooth pursuits and post-saccadic oscillations are derived from human labeling in a data-driven manner.[1]

More details about this approach can be found here.

The open source implementation can be found here.

See this Pupil Helper script to filter eye movements in real time.

Surface Tracker Revamp - #1268, #1515

We have rewritten the surface tracker and included many improvements, e.g.:

• Compensate camera distortion correctly when mapping gaze onto the surface

• Do not block when recomputing surface location cache in post-hoc surface tracker in Pupil Player

• Improve heatmap generation

• General UI improvements

• Improved timeline visualizations

• Option to freeze the image in Capture for easier surface setup and editing

• Use more than a single video frame to define a surface in post-hoc surface tracker in Pupil Player

See the pull request description for a full list.

Please be aware that the names of the exported csv files has changed slightly. See the documentation for the new names.

Caveat: You might need to readjust your surface definitions after upgrading pre-v1.13 recordings. This is due to the internal surface definition having changed and not being fully compatible with previously saved surface definitions.

Pupil Invisible Companion Compatibility Improvements

We are gearing up for a wider release of our new eye tracking glasses Pupil Invisible, and are adding some features to Pupil desktop to support Pupil Invisible. Pupil Invisible will ship with its own Companion Android app. This app will enable one to stream video over WiFi for monitoring purposes.

Bugfixes

• Improved ui responsiveness - pyglui#105, #1527

• Fix Pupil Mobile intrinsics lookup - #1510

• Fix audio playback in cases where the audio is shorter than the world video - #1514

• Pupil data fields theta and phi changed to be consistently floats - #1526

Developers notes

Changed Requirements

• Eye movement detector: nslr and nslr-hmm

pip install git+https://github.com/pupil-labs/nslr
pip install git+https://github.com/pupil-labs/nslr-hmm

• ndsi v0.5

pip install git+https://github.com/pupil-labs/pyndsi -U

• pyglui v1.24