Pupil Core releases


Pupil core

August 25, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Motivation

Pupil v2.3 is a compatibility release for Pupil Invisible, adding support for a simple gaze confidence measure.

Improvements

Pupil Invisible Gaze Confidence - #1984

With its latest 0.8.23 release, Pupil Invisible can detect if the glasses are being worn. If they are not worn, gaze output will be hidden.

We now use this information as a simple binary gaze confidence when loading Pupil Invisible recordings in Pupil Player. If the glasses are detected as being worn, the gaze confidence will be 1.0, otherwise 0.0.

Smaller Improvements

  • Added better log message for pupil mobile network issues - #1994
  • Renamed “camera calibration” to “camera intrinsics” to avoid confusion with gaze calibration - #1992

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

Pupil core

August 4, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Motivation

With v2.2, we have focused on improving the user experience when using Pupil on high-pixel-density displays. Now, Pupil scales and layouts its content correctly based on the content-scaling factor set in your operating system’s settings.

Improvements

Support For High-Pixel-Density Displays - #1976

On displays with a high pixel density, content looks much smaller than expected. Operating systems have the option to scale its content appropriately. Until now, Pupil only supported content-scaling on macOS. With this release, we have added support for content-scaling on Windows and Linux as well.

Smaller Improvements

  • Better user feedback when Accuracy Visualizer reports nan values - #1974
  • Disabled Duplicate and Delete buttons for read-only post-hoc calibrations - #1972
  • Use a square default eye window aspect ratio, matching the latest default cameras - #1987

Changes

Remove Option To Manually Change The Interface Size - #1988

With native support for content scaling on the operating system level, there is no need to scale the UI manually anymore. Therefore, we have removed this feature to further simplify the user interface.

Bug Fixes

  • Fixed non-monotonic progress visualization for post-hoc gaze mapping - #1965
  • Fixed flickering on macOS when resetting the window size - #1973
  • Fixed an empty side-menu when restarting Capture after using HMD-Eyes - #1979
  • Fixed a crash when attempting to log any non-string object - #1980
  • Fixed 2D monocular gaze calibration not working - #1983

Developer Notes

Changed Requirements

pyglui v1.28 - #1972

We updated the pyglui dependency to v1.28. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/pyglui

On Windows, please download and install the latest wheel from GitHub.

glfw 3.3 - #1976

In order to support content-scaling across all three platforms, we make use of the glfwGetWindowContentScale function which has been introduced in glfw 3.3.

We recommend installing the latest glfw 3.3 release via https://www.glfw.org/download.html.

Removed g_pool.gui_user_scale - #1976, #1988

With the removal of the Interface Size setting, the g_pool.gui_user_scale variable has been removed as well. If you draw custom UI content in your plugins without using pyglui, you should now instead use g_pool.gui.scale to scale the content appropriately for different native display scaling settings.

Removed Legacy Plugin Base Classes - #1971

The following legacy plugin base classes have been removed. All plugins inheriting from them now just inherit from Plugin:

  • Visualizer_Plugin_Base
  • Analysis_Plugin_Base
  • Producer_Plugin_Base
  • Experimental_Plugin_Base

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

Pupil core

July 17, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Motivation

With v2.1, we have continued to focus on making Pupil Core more core. We were able to resolve a lot of stability and user experience issues, especially regarding post-hoc calibrations.

Bug Fixes

  • Fixed that recorded calibration were displayed as being editable - #1932, #1938
  • Fixed that recorded calibration were duplicatable - #1933
  • Fixed that immutable calibration could be deleted - #1934
  • Fixed inconsistencies in the naming of recorded calibrations - #1935, #1944
  • Fixed minimum confidence slider for post-hoc calibration - #1942
  • Ensured correct validation is run even when a different calibration is selected - #1921
  • Removed UI for heatmaps across surfaces in Capture (only available in Player) - #1917
  • Fixed incorrect warning about deprecated surface definitions - #1923
  • Prevented crashes when rendering invalid eyeball outline - #1947, #1954
  • Fixed that restarting background tasks would not clear already running ones - #1939
  • Fixed frozen eye videos in HMD-Eyes - #1949
  • Disabled automatic reference location detection again in post-hoc gaze - #1951
  • Fixed crash when opening recording with unavailable gazer - #1922
  • Fixed crashes of the timeline for very short recordings - #1958
  • Fixed crash when deleting a running post-hoc calibration multiple times - #1955
  • Fixed crashing post-hoc gaze mapping for an HMD-Eyes recording - #1961
  • Fixed Service crashing when requesting a non-existing plugin - #1962
  • Fixed Capture crashing when finishing a calibration without reference data - #1960
  • Prevented starting gaze mappers on calibration fail - #1963
  • Fixed an issue when running Pupil Capture bundle on macOS 10.14 Mojave - #1919

Developer Notes

Customizing Screen Marker Positions - #1953

Using custom marker positions for the screen marker calibration choreography might be a common use case. Previously this was not easy to achieve with a custom plugin, as the relevant function was a private method. We have now made it public so that you can more easily implement custom calibration choreographies.

Black Formatting Checks on Travis - #1956

We are internally using Black as automatic Python formatter for our codebase. This helps us ensure that PRs only contain relevant changes and no formatting/whitespace changes. To enforce this more strictly, we added a black formatting check to our Travis integration, which will run on all commits and PRs. Please make sure to format your code with black before submitting a PR to ensure that all Travis checks pass.

To open the RAR-archive on Windows, you will need to use decompression software, e.g. WinRAR.

Pupil core

June 11, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Motivation

With v2.0 and beyond, we will be focusing on making Pupil Core more core.

Our vision for Pupil Core emerged early on: we wanted to make eye tracking hardware and software that was accessible. Accessibility is a useful keyword for us because it encourages us to think about the diverse group of researchers that use Pupil. Pupil Core strives for the following:

  • Transparency: Open source is key here. Enable researchers to introspect into the algorithms. Be explicit about the methods and biases built into the software. Inspire others to make improvements and catch bugs.
  • Modularity: Make it easy for users to develop and run their own custom code - plugin architecture.
  • Connectivity: Make it easy to connect to other software - network API.
  • Diversity: Pupil core is flexible. Pupil Core strives to accommodate a diverse range of research and applications. Pupil Core enables researchers who want to use it as a desktop app with GUI, researchers who want to integrate with other software and sensors via Network API and plugins, and developers who want to tinker with the source code directly.

Over the years, Pupil Core software grew organically to support the community, new hardware, and novel applications. With v2.0 took a few steps back to look at the big picture and to think about the future of Pupil Core. If we want Pupil Core to continue to be a useful platform for researchers and developers, then we need to boil it down to its essentials.

How are we making Pupil Core more core? By focusing on making essential (or core) parts of Pupil Core work really well. Here are the steps we have taken with v2.0:

  1. Restructuring the core pipeline: We have made changes to the application architecture that allows it to run multiple pupil detectors in parallel as well as easily add custom gaze mapping algorithms via user plugins.
  2. Removing hard to maintain features: We had to make some tough decisions. We decided to remove audio capture and native Intel RealSense support. These components were too much work to maintain.
  3. Simplifying the user interface: Over the years we have added many features and the UI has grown organically in the process. We have decided to streamline and improve the UI. Be it merging of menus or more sane default settings, we felt that new users need a bit more help getting started with Pupil Capture and Player.

Improvements

Parallel 2D and 3D Pupil Detection

To allow for using different pupil detection algorithms, we have decoupled our existing pipeline of 2D and 3D detection. Instead of running either of those, both will now run as separate plugins with only minimal overhead. This will enable researchers (including our own R&D team) to test, optimize and replace parts of the pipeline much more conveniently in the future.

Additionally this change has allowed for the following improvements to other parts of the system:

Pupil Diameter Timelines - #1854

With 3D pupil data always available we can always display the meaningful 3D pupil diameter in the timelines. Additionally, we added outlier removal and included the range for a more expressive visualization.

Blink Detection - #1900

With 2D pupil data always available we can run a more reliable blink detection in all circumstances.

Calibration Choreography and Gazer Plugins

By far the biggest change in this release was the refactoring of calibration and gaze mapping part of the core pipeline into separate, extendable components that are easy to swap with custom implementations.

Calibration Plugin in Pupil Capture

When running Pupil Capture, in the plugin list you can find the unified Calibration plugin, which is responsible for configuring both the calibration method (with the Choreography selector), and the gaze mapping method (with the Method selector).

Extending Calibration and Gaze Mapping with User Plugins

Now users can implement their own calibration routines and gaze mapping methods. These will automatically show up in the Choreography and Method selectors.

To do this, users must subclass CalibrationChoreographyPlugin and GazerBase respectively. See the Developer Notes section below for a code example.

Improved Installation Workflow on Windows - #1853

We have wrapped Pupil in a Windows Installer package (MSI) in order to simplify the Windows workflow.

By default, all 3 apps (Capture, Player, and Service) will be installed in C:\Program Files (x86)\Pupil-labs\Pupil v<version>\. All apps will also get a start-menu entry, making it much easier to open Pupil. Installed versions of Pupil can be removed in the Windows Uninstall Settings.

New versions of Pupil will be installed alongside older versions and you can choose which to start. Switching versions will still overwrite your user settings as previously.

Changes

Fixation and Blink Detection by Default - #1870

In order to make Pupil Core even more useful with out-of-the-box settings, we have enabled the Fixation Detector and Blink Detector by default.

Simplified System Graphs - #1869

We have removed the configuration menu for the System Graphs plugin in order to simplify the user interface. The optional display of live graphs for the pupil diameter has been removed as well.

The confidence graphs will now always display the confidence of the 2D pupil detection, as this is a better indicator for correct positioning of the Pupil Core eye tracking headset.

Single Marker Calibration: Physical Markers - #1897

The single marker calibration is mostly intended for calibration with physical (printed) calibration markers, as opposed to the screen marker calibration. For sake of clarity, we changed the default mode for the single marker calibration to Physical Marker (previously this was called Manual).

Removed Manual Marker Calibration

With the restructuring of the calibration workflow, we have removed the deprecated Manual Marker Calibration. The Single Marker Calibration is a more general version and additionally allows for VOR-based choreographies. If you previously used the Manual Marker Calibration, try the Single Marker Calibration in Physical Marker mode (the new default)!

Parallel Pupil Detector Visualization - #1873

We have changed the color of the 2D ellipse visualization in all eye videos to blue in order to differentiate between 2D and 3D detection. You will now always see 3 ellipses on the eyes when the visualization is enabled:

Color Description

  • blue: 2D pupil detection result
  • red: 3D pupil detection result
  • green: 3D eye model

Flip Eye 0 Display - #1844

The eye 0 (right eye) camera sensor is physically flipped, which results in an upside-down eye image for the right eye. This is by design and does not negatively affect pupil detection or gaze estimation. However, the upside-down eye 0 image repeatedly led users to believe that something was broken or incorrect with Pupil Core headsets. We flipped the eye 0 image now by default to better match user expectations with a right-side-up eye image.

Deprecated Audio Capture - #1868

Recording synchronized audio on 3 different operating systems (macOS, Linux, Windows) has always been a challenge for Pupil Capture. (See our motivation section above.)

Unified Network API Plugin - #1881

The previous plugins Pupil Remote and Frame Publisher were commonly used to interact with Pupil Core’s Network API. For the sake of simplicity and clarity, we merged both plugins into the new Network API plugin.

Removed Blink Detector Visualization - #1901

The visualization of the blink detector (brief dark flashing on the screen) could interfere with other plugins and cause visual bugs. We have removed this feature.

Bug Fixes

  • Fixed Player crashing when exporting while computing gaze history - #1843
  • Fixed Player crashing when disabling the Surface Tracker while exporting - #1848
  • Fixed wrong default camera selection in some cases - #1857
  • Fixed a rare crash in the Surface Tracker - #1879
  • Fixed deprecated RealSense cameras showing up in the camera selection - #1877
  • Fixed that offline gaze mappers without data would show as completed - #1889
  • Fixed corrupted gaze data when deactivating one of multiple gaze mappers - #1888
  • Fixed new offline pupil data not being correctly announced sometimes - #1890
  • Fixed a few small issues in offline pupil detection - #1899
  • Fixed problems with opening the macOS bundles on newer versions of macOS

Developer Notes

Changed Requirements

Scikit-learn

We have added scikit-learn as a dependency for Pupil.

pip install scikit-learn

PyAV - 1908

We updated the PyAV dependency to v0.4.5, which includes some bug fixes. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/PyAV

On Windows, download and install the latest wheel from GitHub.

Changed data format

Both 2D and 3D pupil detector now publish their results independently of each other. To differentiate between them, we have added the origin to the topic. For a binocular setup, you will now get 4 different pupil topics:

  • pupil.0.2d
  • pupil.0.3d
  • pupil.1.2d
  • pupil.1.3d

This means that there can now be multiple pupil datums for one eye for the same timestamp (2D and 3D). When grouping by the entire topic however, the timestamps are still guaranteed to be unique.

Restarting Unique Plugins - #1871

We have added the previously missing feature to restart plugins with by_class uniqueness via notifications.

Recording Version 2.2 - #1885

The info.player.json meta-version has been bumped to 2.2 will be incompatible with Pupil Player version v1.

Allow specifying ZMQ HWM option - #1895

We added the option to start the eye processes with custom values for the ZMQ High Water Mark (HWM) via notification.

Custom Calibration Plugins

Below is an example with minimal implementation for a custom HMD 3D calibration choreography and gaze mapping method:

# To implement a custom calibration or gaze mapping method from scratch, subclass GazerBase and CalibrationChoreographyPlugin
from gaze_mapping.gazer_base import GazerBase
from calibration_choreography import CalibrationChoreographyPlugin

# Another option is to modify existing implementations by subclassing existing implementations
from gaze_mapping import Gazer3D
from calibration_choreography import ScreenMarkerChoreographyPlugin


class CustomGazer3D(Gazer3D):
    # Must be unique across all subclasses of GazerBase
    label = "Custom 3D Gazer"


class CustomChoreographyPlugin(ScreenMarkerChoreographyPlugin):
    # Must be unique accross all subclasses of CalibrationChoreographyPlugin
    label = "Custom Calibration"

    @classmethod
    def supported_gazer_classes(cls):
        return [CustomGazer3D]

Release Note Updates

17.06.2020 - v2.0-175

  • Added compatibility for the HMD-Eyes VR integration
  • Fixed an issue that prevented the validation from working correctly

24.06.2020 - v2.0-177

  • Fixed an issue that prevented using the head pose tracker and creating new annotations in Player - #1927

25.06.2020 - v2.0-182

  • Fixed an issue that prevented Pupil Service from starting - #1928

Officially minimum supported operating system versions for the bundles:

Ubuntu: 16.04 macOS: 10.13 (High Sierra) Windows: 10

To open the RAR-archive on Windows, you will need a decompression software, e.g. WinRAR.

Pupil core

March 27, 2020

Please feel free to get in touch with feedback and questions via the #pupil channel on Discord :smile:

Improvements

Windows Driver Installation Stability - #1833

We have greatly improved the driver installation workflow on Windows. You should not experience problems with spaces or non-english characters in the path to your Pupil installation anymore. Additionally, you do no longer have to run the application as administrator. You still need an administrator to authorize the driver installation.

Changes

On-demand Loading of Multi-part Recordings - #1835

Multiple users have reported problems when opening Pupil Mobile recordings with a large number of video parts. We were able to fix this issue by loading the video parts on demand instead of opening them all at once. As a result, Pupil Player will open Pupil Mobile or Pupil Invisible recordings with multiple parts much quicker than before.

As a side-effect, during playback in Pupil Player, you might experience short frame jumps when a new video part needs to be loaded. This does not affect the underlying data or any exports.

Bug Fixes

  • Fixed a crash when changing the ROI without a connected headset - #1827
  • Fixed a crash when opening recordings with empty video parts - #1832
  • Gracefully stop recordings when receiving non-monotonic timestamps - #1829
  • Fixed an issue with calibration file storage on macOS - #1836
  • Fixed an issue with gaze history settings restoration from session settings - #1836

Developer Notes

Changed Requirements - #1834

We updated the pyndsi dependency on v1.3. On Linux and macOS, you can update with:

pip install -U git+https://github.com/pupil-labs/pyndsi

On Windows, please download and install the latest wheel from GitHub.

Removed All Direct C++ Dependencies - #1828

After the latest rework of the calibration codebase to remove C++ dependencies in v1.22, we have re-evaluated all dependencies for Pupil Core software – especially on Windows. We are very pleased to announce that we have finally externalized or gotten rid of all C++ dependencies! :tada:

This greatly simplifies the developer setup on Windows. The biggest changes include:

  • You no longer have to install Visual Studio to run from source
  • You no longer need to download OpenCV
  • You are no longer bound to a fixed path C:\work

We have updated the developer docs with the updated setup instructions. We hope that this simplifies your setup and enables you to develop awesome plugins for Pupil Capture and Pupil Player that can be shared with the community!

Release Note Updates

30.03.2020 14:00: Fix driver installation issue in Windows bundle - #1842

In Pupil v1.23-4, we fixed an issue that prevented users from installing the camera drivers on Windows.

06.04.2020 11:50: Include missing libuvc.dylib in macOS bundle

In Pupil v1.23-5, we have included missing library files for Pupil Player and Pupil Service on macOS.

08.04.2020 17:00: Include missing libusb.dylib in macOS bundle

In Pupil v1.23-5-fix0, we have included missing library files for Pupil Player and Pupil Service on macOS.

24.04.2020 14:00: Improve driver installation stability on Windows - #1856

In Pupil v1.23-10, we have made changes to the driver installation procedure that ensure a successful installation on a larger variety of Windows setups.

  • ...