All software releases

Here you will find a log of all features, changes, bug fixes, and developer notes for Pupil Labs software.


Pupil core

June 11, 2020

Pupil core

June 11, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Motivation

With v2.0 and beyond, we will be focusing on making Pupil Core more core.

Our vision for Pupil Core emerged early on: we wanted to make eye tracking hardware and software that was accessible. Accessibility is a useful keyword for us because it encourages us to think about the diverse group of researchers that use Pupil. Pupil Core strives for the following:

  • Transparency: Open source is key here. Enable researchers to introspect into the algorithms. Be explicit about the methods and biases built into the software. Inspire others to make improvements and catch bugs.
  • Modularity: Make it easy for users to develop and run their own custom code - plugin architecture.
  • Connectivity: Make it easy to connect to other software - network API.
  • Diversity: Pupil core is flexible. Pupil Core strives to accommodate a diverse range of research and applications. Pupil Core enables researchers who want to use it as a desktop app with GUI, researchers who want to integrate with other software and sensors via Network API and plugins, and developers who want to tinker with the source code directly.

Over the years, Pupil Core software grew organically to support the community, new hardware, and novel applications. With v2.0 took a few steps back to look at the big picture and to think about the future of Pupil Core. If we want Pupil Core to continue to be a useful platform for researchers and developers, then we need to boil it down to its essentials.

How are we making Pupil Core more core? By focusing on making essential (or core) parts of Pupil Core work really well. Here are the steps we have taken with v2.0:

  1. Restructuring the core pipeline: We have made changes to the application architecture that allows it to run multiple pupil detectors in parallel as well as easily add custom gaze mapping algorithms via user plugins.
  2. Removing hard to maintain features: We had to make some tough decisions. We decided to remove audio capture and native Intel RealSense support. These components were too much work to maintain.
  3. Simplifying the user interface: Over the years we have added many features and the UI has grown organically in the process. We have decided to streamline and improve the UI. Be it merging of menus or more sane default settings, we felt that new users need a bit more help getting started with Pupil Capture and Player.

Improvements

Parallel 2D and 3D Pupil Detection

To allow for using different pupil detection algorithms, we have decoupled our existing pipeline of 2D and 3D detection. Instead of running either of those, both will now run as separate plugins with only minimal overhead. This will enable researchers (including our own R&D team) to test, optimize and replace parts of the pipeline much more conveniently in the future.

Additionally this change has allowed for the following improvements to other parts of the system:

Pupil Diameter Timelines - #1854

With 3D pupil data always available we can always display the meaningful 3D pupil diameter in the timelines. Additionally, we added outlier removal and included the range for a more expressive visualization.

Blink Detection - #1900

With 2D pupil data always available we can run a more reliable blink detection in all circumstances.

Calibration Choreography and Gazer Plugins

By far the biggest change in this release was the refactoring of calibration and gaze mapping part of the core pipeline into separate, extendable components that are easy to swap with custom implementations.

Calibration Plugin in Pupil Capture

When running Pupil Capture, in the plugin list you can find the unified Calibration plugin, which is responsible for configuring both the calibration method (with the Choreography selector), and the gaze mapping method (with the Method selector).

Extending Calibration and Gaze Mapping with User Plugins

Now users can implement their own calibration routines and gaze mapping methods. These will automatically show up in the Choreography and Method selectors.

To do this, users must subclass CalibrationChoreographyPlugin and GazerBase respectively. See the Developer Notes section below for a code example.

Improved Installation Workflow on Windows - #1853

We have wrapped Pupil in a Windows Installer package (MSI) in order to simplify the Windows workflow.

By default, all 3 apps (Capture, Player, and Service) will be installed in C:\Program Files (x86)\Pupil-labs\Pupil v<version>\. All apps will also get a start-menu entry, making it much easier to open Pupil. Installed versions of Pupil can be removed in the Windows Uninstall Settings.

New versions of Pupil will be installed alongside older versions and you can choose which to start. Switching versions will still overwrite your user settings as previously.

Changes

Fixation and Blink Detection by Default - #1870

In order to make Pupil Core even more useful with out-of-the-box settings, we have enabled the Fixation Detector and Blink Detector by default.

Simplified System Graphs - #1869

We have removed the configuration menu for the System Graphs plugin in order to simplify the user interface. The optional display of live graphs for the pupil diameter has been removed as well.

The confidence graphs will now always display the confidence of the 2D pupil detection, as this is a better indicator for correct positioning of the Pupil Core eye tracking headset.

Single Marker Calibration: Physical Markers - #1897

The single marker calibration is mostly intended for calibration with physical (printed) calibration markers, as opposed to the screen marker calibration. For sake of clarity, we changed the default mode for the single marker calibration to Physical Marker (previously this was called Manual).

Removed Manual Marker Calibration

With the restructuring of the calibration workflow, we have removed the deprecated Manual Marker Calibration. The Single Marker Calibration is a more general version and additionally allows for VOR-based choreographies. If you previously used the Manual Marker Calibration, try the Single Marker Calibration in Physical Marker mode (the new default)!

Parallel Pupil Detector Visualization - #1873

We have changed the color of the 2D ellipse visualization in all eye videos to blue in order to differentiate between 2D and 3D detection. You will now always see 3 ellipses on the eyes when the visualization is enabled:

Color Description

  • blue: 2D pupil detection result
  • red: 3D pupil detection result
  • green: 3D eye model

Flip Eye 0 Display - #1844

The eye 0 (right eye) camera sensor is physically flipped, which results in an upside-down eye image for the right eye. This is by design and does not negatively affect pupil detection or gaze estimation. However, the upside-down eye 0 image repeatedly led users to believe that something was broken or incorrect with Pupil Core headsets. We flipped the eye 0 image now by default to better match user expectations with a right-side-up eye image.

Deprecated Audio Capture - #1868

Recording synchronized audio on 3 different operating systems (macOS, Linux, Windows) has always been a challenge for Pupil Capture. (See our motivation section above.)

Unified Network API Plugin - #1881

The previous plugins Pupil Remote and Frame Publisher were commonly used to interact with Pupil Core’s Network API. For the sake of simplicity and clarity, we merged both plugins into the new Network API plugin.

Removed Blink Detector Visualization - #1901

The visualization of the blink detector (brief dark flashing on the screen) could interfere with other plugins and cause visual bugs. We have removed this feature.

Bug Fixes

  • Fixed Player crashing when exporting while computing gaze history - #1843
  • Fixed Player crashing when disabling the Surface Tracker while exporting - #1848
  • Fixed wrong default camera selection in some cases - #1857
  • Fixed a rare crash in the Surface Tracker - #1879
  • Fixed deprecated RealSense cameras showing up in the camera selection - #1877
  • Fixed that offline gaze mappers without data would show as completed - #1889
  • Fixed corrupted gaze data when deactivating one of multiple gaze mappers - #1888
  • Fixed new offline pupil data not being correctly announced sometimes - #1890
  • Fixed a few small issues in offline pupil detection - #1899
  • Fixed problems with opening the macOS bundles on newer versions of macOS

Developer Notes

Changed Requirements

Scikit-learn

We have added scikit-learn as a dependency for Pupil.

pip install scikit-learn

PyAV - 1908

We updated the PyAV dependency to v0.4.5, which includes some bug fixes. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/PyAV

On Windows, download and install the latest wheel from GitHub.

Changed data format

Both 2D and 3D pupil detector now publish their results independently of each other. To differentiate between them, we have added the origin to the topic. For a binocular setup, you will now get 4 different pupil topics:

  • pupil.0.2d
  • pupil.0.3d
  • pupil.1.2d
  • pupil.1.3d

This means that there can now be multiple pupil datums for one eye for the same timestamp (2D and 3D). When grouping by the entire topic however, the timestamps are still guaranteed to be unique.

Restarting Unique Plugins - #1871

We have added the previously missing feature to restart plugins with by_class uniqueness via notifications.

Recording Version 2.2 - #1885

The info.player.json meta-version has been bumped to 2.2 will be incompatible with Pupil Player version v1.

Allow specifying ZMQ HWM option - #1895

We added the option to start the eye processes with custom values for the ZMQ High Water Mark (HWM) via notification.

Custom Calibration Plugins

Below is an example with minimal implementation for a custom HMD 3D calibration choreography and gaze mapping method:

# To implement a custom calibration or gaze mapping method from scratch, subclass GazerBase and CalibrationChoreographyPlugin
from gaze_mapping.gazer_base import GazerBase
from calibration_choreography import CalibrationChoreographyPlugin

# Another option is to modify existing implementations by subclassing existing implementations
from gaze_mapping import Gazer3D
from calibration_choreography import ScreenMarkerChoreographyPlugin


class CustomGazer3D(Gazer3D):
    # Must be unique across all subclasses of GazerBase
    label = "Custom 3D Gazer"


class CustomChoreographyPlugin(ScreenMarkerChoreographyPlugin):
    # Must be unique accross all subclasses of CalibrationChoreographyPlugin
    label = "Custom Calibration"

    @classmethod
    def supported_gazer_classes(cls):
        return [CustomGazer3D]

Release Note Updates

17.06.2020 - v2.0-175

  • Added compatibility for the HMD-Eyes VR integration
  • Fixed an issue that prevented the validation from working correctly

24.06.2020 - v2.0-177

  • Fixed an issue that prevented using the head pose tracker and creating new annotations in Player - #1927

25.06.2020 - v2.0-182

  • Fixed an issue that prevented Pupil Service from starting - #1928

Officially minimum supported operating system versions for the bundles:

Ubuntu: 16.04 macOS: 10.13 (High Sierra) Windows: 10

To open the RAR-archive on Windows, you will need a decompression software, e.g. WinRAR.

Pupil core

March 27, 2020

Pupil core

March 27, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Improvements

Windows Driver Installation Stability - #1833

We have greatly improved the driver installation workflow on Windows. You should not experience problems with spaces or non-english characters in the path to your Pupil installation anymore. Additionally, you do no longer have to run the application as administrator. You still need an administrator to authorize the driver installation.

Changes

On-demand Loading of Multi-part Recordings - #1835

Multiple users have reported problems when opening Pupil Mobile recordings with a large number of video parts. We were able to fix this issue by loading the video parts on demand instead of opening them all at once. As a result, Pupil Player will open Pupil Mobile or Pupil Invisible recordings with multiple parts much quicker than before.

As a side-effect, during playback in Pupil Player, you might experience short frame jumps when a new video part needs to be loaded. This does not affect the underlying data or any exports.

Bug Fixes

  • Fixed a crash when changing the ROI without a connected headset - #1827
  • Fixed a crash when opening recordings with empty video parts - #1832
  • Gracefully stop recordings when receiving non-monotonic timestamps - #1829
  • Fixed an issue with calibration file storage on macOS - #1836
  • Fixed an issue with gaze history settings restoration from session settings - #1836

Developer Notes

Changed Requirements - #1834

We updated the pyndsi dependency on v1.3. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/pyndsi

On Windows, please download and install the latest wheel from GitHub.

Removed All Direct C++ Dependencies - #1828

After the latest rework of the calibration codebase to remove C++ dependencies in v1.22, we have re-evaluated all dependencies for Pupil Core software – especially on Windows. We are very pleased to announce that we have finally externalized or gotten rid of all C++ dependencies! :tada:

This greatly simplifies the developer setup on Windows. The biggest changes include:

  • You no longer have to install Visual Studio to run from source
  • You no longer need to download OpenCV
  • You are no longer bound to a fixed path C:\work

We have updated the developer docs with the updated setup instructions. We hope that this simplifies your setup and enables you to develop awesome plugins for Pupil Capture and Pupil Player that can be shared with the community!

Release Note Updates

30.03.2020 14:00: Fix driver installation issue in Windows bundle - #1842

In Pupil v1.23-4, we fixed an issue that prevented users from installing the camera drivers on Windows.

06.04.2020 11:50: Include missing libuvc.dylib in macOS bundle

In Pupil v1.23-5, we have included missing library files for Pupil Player and Pupil Service on macOS.

08.04.2020 17:00: Include missing libusb.dylib in macOS bundle

In Pupil v1.23-5-fix0, we have included missing library files for Pupil Player and Pupil Service on macOS.

24.04.2020 14:00: Improve driver installation stability on Windows - #1856

In Pupil v1.23-10, we have made changes to the driver installation procedure that ensure a successful installation on a larger variety of Windows setups.

Pupil core

February 20, 2020

Pupil core

February 20, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Features and Changes

Video Source Rework - #1792

We have reworked the video source input system to greatly improve stability, user experience, and maintainability of the related features:

Single Plugin UI

All previous plugins for managing the video source (managers and sources) have been unified under a single plugin menu: Video Source, which handles device/camera selection as well as adjusting camera settings.

Automatic vs Manual Camera Selection

The new source system offers two ways for selecting cameras: automatic and manual:

  • In automatic mode (default) you can activate source devices, e.g. Local USB or Pupil Mobile. Activating a device selects the best matching available cameras as input for all windows in Pupil Capture.
  • Enabling manual mode allows you to select cameras individually for every window.

UI Cleanup

The following non-essential user interface elements have been removed to provide a better overall user experience:

  • The UI for selecting a file source has been removed. Starting a file source via network notification or via drag-and-drop is still possible.
  • Manually selecting HMD-Eyes as input source has been removed, as the plugin is started from Unity via network notification.
  • The Capture plugin UI in the eye windows has been removed entirely from Pupil Player as the displayed information was incomplete and the interactivity was limited.
  • The Test Image source has been removed as it was only intended for development use.

Removed Built-in RealSense Video Backend

Intel RealSense sensors are designed as a prototyping/development platform. Intel deprecates RealSense hardware frequently. Driver support is appropriate for a prototyping tool, but it is not sufficient for a production ready tool.

We have tried to build our tools around Intel’s changing APIs and patchwork driver support, but this became difficult and confusing for users.

You can still use RealSense Video Backend as a custom plugin. You can load it directly in the plugin folder of Pupil Capture. Check it out here.

Player: Add Gaze History to Polyline Visualization - #1785

We decided to use the Polyline Visualization as a replacement for the previously removed Scan Path Plugin. You can now visualize past gaze points up to a certain duration by adjusting the duration slider in the Vis Polyline Plugin. When first dragging the slider, the history for all frames will be computed, which will take some time. Afterwards you can see the gaze history and adjust the duration instantly.

Capture: Improved Region of Interest (ROI) selection - #1788

The ROI selector for the eye windows has been improved for better stability and better user experience. In doing so, we standardized the format of ROI boundaries across Pupil to the following order: (minx, miny, maxx, maxy). Please note that this changes the format for adjusting the ROI via network messages.

Player: Add 3D Eyeball Visualization - #1810

We have added the green circle overlay for the 3D eyeball to Player. The overlay will be added to the Eye Overlay and Eye Video Exporter plugins, when 3D pupil data is available. You can disable the visualization (including the pupil outline) via the corresponding plugin menus.

Bug Fixes

  • Changed Windows disconnection log-level from warning to debug - #1786
  • Fixed some of small bugs in the ROI selector - #1788
  • Fixed mixed up export columns for monocular eye_center and gaze_normal - #1805
  • Prevent a crash caused by incorrect UVC timestamps - #1812

Developer Notes

Getting Rid of C++ Dependencies - #1789

We have made the next step towards an easier installation process of Pupil. After the externalization of the Pupil Detectors in v1.20, we have been able to replace the last remaining C++ module with a Scipy re-implementation. This means that you do not have to compile any Python Extension Modules when starting Pupil.

As a result, you can now run Pupil from source on Windows without having to go through the trouble of setting up CERES. While the pupil-detectors still depend on CERES, we offer prebuilt wheels for Windows that you can install without having to set up the dependencies.

Changed Requirements

We updated the pyglui dependency to v1.27. Please update with:

pip install -U git+https://github.com/pupil-labs/pyglui

Release Notes Update

25.02.2019 v1.22.3

  • #1817 - Fixed a regression which caused all frames to be dropped after changing Pupil Capture’s time base.

27.02.2019 v1.22.7

  • Commit bcfbf7d - Fixed a typo that caused the v1.22.3 fix to not work correctly.

Pupil core

January 9, 2020

Pupil core

January 9, 2020

Features and Changes

Support for Pupil Invisible Audio Recordings - #1748

Pupil Player now supports audio playback and audio export for Pupil Invisible recordings with audio recordings.

Stream Head Pose over Network - #1774

You can now subscribe to the the topic head_pose to receive live data from the Online Head Pose Tracker plugin.

Bug Fixes

  • Fixed eye processes always displayed the algorithm view - #1766
  • Fixed potential crash when changing the eye video size - #1770
  • Made pyrealsense warning a debug-level message in order to be less obtrusive - #1768
  • Fixed broken interactivity of the ROI mask in the eye windows - #1778
  • Fixed multiple stability issues with the fullscreen calibration window - #1767

Developer Notes

Changed Requirements

We updated the PyAV dependency to v0.4.4 for the PI audio support. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/PyAV

On Windows, download and install the latest wheel from GitHub.

Pupil core

December 12, 2019

Pupil core

December 12, 2019

Features and Changes

Deprecated Fingertip Calibration - #1753, #1757

We removed the fingertip calibration method.

The fingertip calibration method was rarely used and provided lower accuracy compared to other calibration methods. Additionally, dependencies needed for the fingertip detection required some work to set up correctly and accounted for almost 50% of the size of our application bundles.

Bug Fixes

  • Fixed a crash when trying to open empty (but existing) recordings - #1751
  • Fixed a crash on Windows when missing the latest Visual Studio redistributables - #1756
  • Fixed visualization of circle markers in offline calibration for recordings with gaps - #1758
  • Fixed broken combination of --hide-ui and --profile CLI flags - #1760

Developer Notes

Externalized Pupil Detectors - #1642

We have extracted the Pupil Detectors into their own repository: https://github.com/pupil-labs/pupil-detectors

This will make it very easy to use our pupil detectors in any standalone application or experimental setup without having to deal with all of Pupil’s intricacies.

Additionally this will make it much easier to run Pupil from source (especially on Windows), since we are distributing pupil-detectors as package via PyPI and even provide precompiled binaries for Windows!

OpenGL Debugging - #1752

We occasionally receive crash reports involving error messages from PyOpenGL, which we cannot reproduce on our machines. We wrapped the PyOpenGL errors to prevent crashes and log additional messages in these cases. This can result in visual UI errors (instead of a crash) when there are OpenGL issues. If you notice any weird UI behavior, please save the log file and report back to us.

  • ...
  • ...