Pupil Core releases


Pupil core

January 7, 2021

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord! :smile:

Motivation

We are excited to introduce pye3d - a complete overhaul of the 3D pupil detector employed by Pupil Core software. In building pye3d, we have left no stone unturned, reimagining many of the algorithmic building blocks constituting the final gaze-estimation pipeline. pye3d is the result of three years of hard work and we are excited to get it into your hands!

Improvements

Improved built-in 3D pupil detector - #2011, #2065

pye3d implements a novel eye-model fitting strategy that incorporates a corneal refraction-correction step, accounting for nonlinear effects in ocular optics. Gaze estimates and pupil diameter estimates are now more accurate than ever before [2,3]. Thanks to a new sphere center estimation methodology [3], it is also considerably faster than our previous 3D detector [1] in regards to fitting robust and accurate eye models. Read more about how it works in our documentation.

Academic references

[1] L. Świrski and N. A. Dodgson. A Fully-Automatic, Temporal Approach to Single Camera, Glint-Free 3D Eye Model Fitting. In Proceedings of ECEM 2013. “L. Świrski and N. A. Dodgson. A Fully-Automatic, Temporal Approach to Single Camera, Glint-Free 3D Eye Model Fitting. In Proceedings of ECEM 2013.” [2] K. Dierkes, M. Kassner, A. Bullling, A novel approach to single camera, glint-free 3D eye model fitting including corneal refraction. In ETRA ’18: Symposium on Eye Tracking Research and Applications, 2018. [3] K. Dierkes, M. Kassner, A. Bulling. A fast approach to refraction-aware eye-model fitting and gaze prediction. In ETRA ’19: Symposium on Eye Tracking Research and Applications, 2019.

Changes

Removal of old Pupil 3D detector

With this release, we also stop the maintainance of the previous 3D detector. Starting with the new 2.0 release, our pupil-detectors package no longer includes the previous 3D detector. This allows us to remove Ceres as a dependency, making the project much easier to maintain and build from source.

Should you rely on this detector, you can still download previous Pupil Core software releases or install previous pupil-detectors via PyPI.

Bug Fixes

  • Fixed issue that would leave an unresponsive eye window due to plugin crash - #2059
  • Fixed crash on canceling single marker calibration window mode - #2062
  • Fixed crash on post-hoc pupil detection when the frame count is zero - #2063

Developer notes

Network API Changes

Reworked pupil detector plugin and network API

In this release we reworked the pupil detector plugin base class. It is now easier than ever to implement a custom pupil detector and accompanying pupil detector plugin.

We have also introduced breaking changes to our network API for pupil detector plugins. Here is a list of possible notification payload descriptions that are handled by detector plugins:

# Notification for enabling/disabling any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_enabled',
    'subject': 'pupil_detector.set_enabled',
    'value': <is_enabled: bool>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

# Notification for setting the Region-Of-Interest (ROI) for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.set_roi',
    'subject': 'pupil_detector.set_roi',
    'value': <roi: (min_x: int, min_y: int, max_x: int, max_y: int)>,
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

# Notification for updating a partial set of pupil detector properties for a specific pupil detector plugin
{
    'topic': 'notify.pupil_detector.set_properties',
    'subject': 'pupil_detector.set_properties',
    'values': <detector_properties: dict>,
    'eye_id': <eye_id: int>,  # required; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # required
}

# Notification for requesting pupil detector properties broadcast for any or all pupil detector plugins
{
    'topic': 'notify.pupil_detector.broadcast_properties',
    'subject': 'pupil_detector.broadcast_properties',
    'eye_id': <eye_id: int>,  # optional; possible values: 0 or 1
    'detector_plugin_class_name': <detector_plugin_class_name: str>,  # optional
}

For more information about custom pupil detector plugins, please read the documentation. You can also see examples of custom pupil detection plugins here.

For an example script that showcases the network API, please consult this helper script.

Changed 3D pupil datum format

pye3d uses a slightly different pupil format than our previous 3D detector. Because pye3d continously updates a single model instead of fitting multiple parallel ones, the new 3D pupil datum no longer contains the following keys: model_id and model_birth_timestamp.

Our latest Hmd-eyes release handles this by falling back to “0” and 0.0, respectively.

Binocular 3D gaze data with string-only keys - #2068

In order to be compatible with the msgpack-python v1.0.0 (specifially the new strict_map_key=True default), this release uses strings only as dictionary keys for all msgpack-encoded data that is published via the Network API.

This change affects primarily binocular 3D gaze data which used to contain integer dictionary keys.

Raw Data Exporter - Optional model_id key - #2061

Some of the fields in the pupil datum related to the 3D model no longer exist in pye3d (see above). This change ensures the raw data export handles missing fields and outputs a compatible csv file.

Updated msgpack dependency to version 1.0 - #2068

We continue to streamline and simplify the installation of Pupil Core dependencies. In this release we updated the msgpack to the latest 1.0. To ensure you have the latest required versions of all dependencies, please install them via requirements.txt.

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

Added support for custom receiver high-water mark for HMD Streaming source - #2058

This change adds the ability to set a custom ZMQ high watermark value for HMD Streaming source. This gives users more control over the streaming behaviour when using Pupil Core with an HMD headset.

Added launcher_process.should_stop notification handling #2070

This change adds the ability to request stopping Capture, Player or Service with the same notification.

import zmq
import msgpack


# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
pub_socket.connect(ipc_pub_url)


subject = "launcher_process.should_stop"
topic = "notify." + subject
payload = {
    'topic': topic,
    'subject': subject,
}

pub_socket.send_string(topic, flags=zmq.SNDMORE)
pub_socket.send(msgpack.dumps(payload, use_bin_type=True))

Simplified running from source on Windows - #2073

We have removed the necessity to run Pupil from source using the run_<app>.bat files on Windows. Instead of extending the PATH environment variable via the .bat file, we extend it via the application launcher. This also makes it easier to attach a debugger, e.g. via Visual Studio Code.

To open the RAR-archive on Windows, you will need to use decompression software, such as WinRAR or 7-Zip (both are available for free).

macOS (11) Big Sur compatibility - #2079

Note: Pupil Core is not currently supported on Mac OS Big Sur. We aim to release a compatability update within the next few weeks. :construction_worker:

Pupil core

November 16, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Motivation

In v2.6, we are taking a big step towards our modularity goal. We now support custom user pupil detector plugins. This change makes it easier than ever for you to modify the existing pupil detection algorithms or to build your own! You will no longer have to run Pupil Core software from source to get started with your custom pupil detector! Read more about it in our documentation.

Improvements

Support custom pupil detection plugins - #2033

You can now run your own pupil detection plugins that run in the eye processes. These plugins can be used to implement custom algorithms for extracting pupillometry data from the eye images and feeding it to the rest of the gaze mapping pipeline.

Pupil detection plugins are supported in Pupil Capture, Pupil Player, and Pupil Service. Similar to regular user plugins, custom pupil detection plugins should be placed inside pupil_capture_settings/plugins for Pupil Capture, pupil_player_settings/plugins for Pupil Player, and pupil_service_settings/plugin for Pupil Service. The plugins are automatically loaded when the applications starts.

Read more about it in our documentation.

Support HMD Streaming Source in eye process - #2052

We have added support for using HMD Streaming Source in the eye process. This also adds support for BGR and Gray frame format for HMD streams.

Support Pupil Invisible recording format v1.4 - #2054

This new recording format has been introduced in Pupil Companion 1.0.0. This version provides support for the OnePus 8 phone and Android 11. On the OnePlus 8, eye videos will be h264-encoded.

Bug Fixes

  • Removed import * statements - #2036
  • Fixed typo in circle detector benchmark - #2038
  • Fixed CPU graph and UI refresh performance - #2040, #2041
  • Fixed gaze with negative z-component - #2043
  • Fixed issue when minimizing 3D pupil detector debug window on Windows - #2047
  • Fixed UVC camera auto-selection priority in case of multiple pattern matches - #2051

Developer Notes

Added ability to stop eye process plugins via notifications - #2039

Another small improvement for eye process plugin management in this release is the ability to stop plugins via notifications. This gives better control over the list of plugins running in the eye process.

import zmq
import msgpack

# class name of the plugin to stop
plugin_name = "..."

# create and connect PUB socket to IPC
pub_socket = zmq.Socket(zmq.Context(), zmq.PUB)
pub_socket.connect(ipc_pub_url)


for eye_id in (0, 1):
    subject = "stop_eye_plugin"
    topic = "notify." + subject
    payload = {
        'topic': topic
        'subject': subject,
        'target': f"eye{eye_id}",
        'name': plugin_name,
    }

    pub_socket.send_string(topic, flags=zmq.SNDMORE)
    pub_socket.send(msgpack.dumps(payload, use_bin_type=True))

Added requirements.txt file - #2029

We continue to streamline and simplify the installation of Pupil Core dependencies. You can now use the requirements.txt file from the root of the pupil repository to install and update all Python dependencies.

python -m pip install --upgrade pip wheel
pip install -r requirements.txt

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

Pupil core

October 20, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Motivation

Pupil v2.5 focuses on further improvements to software stability. We have fixed multiple issues that prevented very old recordings from being opened in Pupil Player.

Improvements

Replacing internal GLFW bindings with pyGLFW - #2015

In this release we replaced the internal glfw.py bindings to GLFW with the external pyGLFW package. This change simplifies our dependency list, making easier to install GLFW on Windows.

New default camera intrinsics - #2023

We use camera intrinsics in multiple places to transform between the 2d distorted image space and the 3d undistorted camera space. With this release, we have updated the default scene camera intrinsics to perform this tansformation more accuractely at the image borders.

Note: Each camera is slightly different. Therefore, it is recommended to run your own camera intrinsics estimation to get the best possible transformations for your camera.

Bug Fixes

  • Fixed a crash in Pupil Player when attempting to visualize invalid data - #2021
  • Fixed an edge-case when handling recordings with negative time jumps - #2026, #2028
  • Fixed a frame drop issue on macOS when windows were occluded - #2027
  • Fixed an issue that caused instabilities when starting the Head Pose tracker - #2022
  • Surface Tracker stability improvements - #2025

Developer Notes

Improved Observable class - #2022

Improvements to the Observable class enable you to observe monkey patched methods (e.g. replaced methods that were not in the original class definition). Additionally, trying to add private methods as observers now fails with a more explicit error.

Changed Requirements

Please install the new external glfw dependency:

pip install glfw

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

Pupil core

September 17, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Motivation

In Pupil v2.4, we are adding support for eye camera intrinsics. These are used to improve 3d eyeball model positioning. Additionally, we have fixed a long-standing issue that caused application windows not to appear on startup when using Windows OS.

Improvements

Better 3D Eyeball Position - #1995

We now use the focal length of the eye cameras to improve the estimate of the 3D eyeball position. Previously, the 3d eye model assumed a fixed camera focal length. This will most likely not have any noticeable effect for most use-cases, as it does not influence the gaze estimation. We assume default focal lengths for the different types of eye cameras shipped with Pupil Labs’s hardware.

Caveat: 200Hz Vive Add-on Recordings

Pupil Core software now saves the eye camera intrinsics to the recording folder. On opening recordings made with previous versions in Pupil Player, we patch up the recording by inserting the appropriate eye camera intrinsics.

It is important to note that the 200Hz Pupil Core and 200Hz Vive Add-on use eye cameras with different focal lengths. Unfortunately, it is not possible for Pupil Player to differentiate between existing eye videos made by these cameras. As a result, Pupil Player will use 200Hz Pupil Core eye camera intrinsics for existing 200hz Vive Add-on eye videos.

Recordings made with Pupil Capture v2.4 and higher will not require any action in this regard.

Bug Fixes

  • Fixed issue when opening Pupil Invisible recordings with more than 10 parts - #1996
  • Ensure timelines update when changing calibrations for post-hoc gaze-mapping - #2000
  • Fixed a crash when opening a recording that contained recorded fallback intrinsics - #2002
  • Fixed a very rare crash of the eye window caused by malformed data - #2008
  • Fixed occasionally missing windows on startup on Windows in multi-screen setups - #2007
  • Resolved an error message when minimizing the eye windows on Windows - #2009

Developer Notes

Changed Requirements

pupil-detectors - #1995

We have updated pupil-detectors to v1.1.1, please update with

pip install -U pupil-detectors

packaging - #1993

We now require packaging>=20.0, please update with

pip install -U packaging

Internal Version Parsing - #1993

We have switched to a more general version-parsing method by using packaging.version.parse to parse all internally used version strings (from Pupil and external dependencies).

Black Formatter 20.8 - #2003

We have upgraded our black formatter to version 20.8, which introduces a few changes to previous versions and reformats parts of the code which were fine previously. Please make sure to also upgrade black with

pip install -U black

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

Pupil core

August 25, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Motivation

Pupil v2.3 is a compatibility release for Pupil Invisible, adding support for a simple gaze confidence measure.

Improvements

Pupil Invisible Gaze Confidence - #1984

With its latest 0.8.23 release, Pupil Invisible can detect if the glasses are being worn. If they are not worn, gaze output will be hidden.

We now use this information as a simple binary gaze confidence when loading Pupil Invisible recordings in Pupil Player. If the glasses are detected as being worn, the gaze confidence will be 1.0, otherwise 0.0.

Smaller Improvements

  • Added better log message for pupil mobile network issues - #1994
  • Renamed “camera calibration” to “camera intrinsics” to avoid confusion with gaze calibration - #1992

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

  • ...