We are happy to release v.0.7 of the Pupil platform! These notes are for new features introduced in v0.7 and v0.7.3. and serve as the official v0.7 release notes.

Contributors

We want to thank everybody that has contributed! Be it in code, bug reports or feature request and feedback! Pupil would not be possible without you! Check out the list of contributors here.

New features

Pupil Detection

Ported the existing detector to C++ and improved its performance. The detector CPU load has been cut in half!v0.7 introduces our new 3d model based pupil detector. We call it 3d detector and renamed the old detector 2d detector.

3d Pupil Detector

The old pupil detector and gaze mapper hit the limits of regression based calibration: It assumed the headset was "screwed to the head". When the headset moves the gaze mapping shifts. This is usually referred to as drift.Additionally, a model-less search for the ellipse yields poor results when the pupil is partially obstructed by reflections, eyelashes, and/or eyelids.

With the use of a 3d model of the eye and a pinhole model of the camera based on Swirski’s work in “A fully-automatic, temporal approach to single camera, glint-free 3d eye model fitting - PETMEI 2013 we model the eyeball as a sphere and the pupil as a disk on the sphere. The sphere size used is based on an average human eyeball diameter of 24mm. The state of the model is the position of the sphere in eye camera space and two rotation vectors that describe the location of the pupil on the sphere.

Using temporal constraints and competing eye models we can detect and compensate for slippage events when 2d pupil evidence is strong. In case of weak 2d evidence we can use constraints from existing models to robustly fit pupils with considerably less evidence than before.

Using the 3d model we can now compute some exciting and very useful properties from the eye video footage:

• 3d pupil normal: The normal vector of the pupil in relation to the 3d location of the eye camera.

• 3d pupil diameter: Using the 3d model we correct the pupil diameter for perspective scale and transform it to millimeter (mm) space using the assumption that the eye ball size has a diameter of 24mm.

  3d sphere location: The location of the eyeball in relation to the eye camera.

  Augmented 2d detector confidence: Using a strong 3d model we have a better handle on false positives.

This new 3d pupil detector still has areas that we will continue to work on, but we feel that it already outperforms the 2d detector and recommend giving it a try!

3d Gaze Mapper & Calibration

With a 3d location of the eye and 3d vectors of gaze we don’t have to rely on a generic polynomial or similar for gaze mapping.

Instead we use a geometric gaze mapping approach: We model the world camera as a pinhole camera with distortion, and project pupil line of sight vectors onto the world image. For this we need to know the rotation translation of world and eye camera. This rigid transformation is obtained in a 5 point calibration routine. Under the hood we use a custom bundle adjustment routine.

Since we are now detecting and modeling the location of the eyeball we are taking movement of the headset into account. This means that the 3d model in combination with the geometric gaze mapper can compensate for slippage. You can take off the headset and put it back on without having to re-calibrate! We are still working on this feature, and you can expect this to become better over time.

For robust 3d pupil detection and fast slippage compensation we found that detecting the eye(s) at 90 Hz or greater is very helpful.

Restructured Pupil Server & Updated pupil-helper scripts

• Pupil server now sends JSON data, that conforms to a standard format structured topic.

• Every subject in the events like pupil and gaze data is broadcast as a separate topic.

• Motivated by @zenithlight’s pull request: https://github.com/pupil-labs/pupil/pull/326

• See updated pupil (and much easier to use) helper scripts: https://github.com/pupil-labs/pupil-helpers

Introducing Plugin Notifications

This scheme allows plugins to notify their in-app environment and when using pupil sync to notify other programs and machines.

• This simplifies developing more intertwined behaviour while keeping plugins isolated.

• Plugin notifications that carry the flag record are saved during recording and can be retrieved.

Pupil Sync

This is not a new feature but we added some cool new stuff!

• Added proper network time synchronization for multiple source and multi user synchronous recordings. Check out the updated pupil sync examples: https://github.com/pupil-labs/pupil-helpers/tree/master/pupil_sync -- check out a short video here: https://youtu.be/2a6awbn9Cao

• Plugin notifications that carry the flag network propagate are shared with all other Pupil Sync nodes.

Full Support for Binocular Operation

• Start and stop left and right eye during runtime.

• 2d and 3d gaze mappers support binocular mapping.

Better Visual Feedback in Pupil Capture

Pupil and Gaze confidence is represented as transparency in the gaze dot in the world window and the pupil dot in the eye window(s). More opaque dot = higher confidence.

Changes to the Recording Format

We have depreciated the pupil_positions.npy and gaze_postions.npy files and now record all events and notifications that happen during recording into the pupil_data file. pupil_data is a pickled python dictionary. All raw data can be easily exported as .csv from Pupil Player :point_down:

Exporting Data

We have restructured export logic in Pupil Player.

• Now you can press e or click the UI button e and all open plugins that have export capability will export.

• All exports are separated from your raw data and contained in the exports directory. The exports directory will live within your recording folder. Structural change prompted by @cpicanco's issue.

• We have added an exporter in Pupil Player that will give you access to all raw data that was acquired during the recording.

Bugfixes and Small improvements

• GLFW MacOS fixes for render issues and crashes

• Player GUI design is more consistent

• Various pyglui fixes and improvements

• Audio modes are settable

• Camera default values for 120fps capture

• Fixed bugs in libuvc and pyuvc that appeared when using Logitech C525/B525 cameras.

• Fixed bug in Pupil Player that prevented exports when world.wav file was present.

Licence Change

On 2016-12-15 we transitioned to a new license. Pupil is now licensed under LGPLv3. For more information see the License page on the wiki.

We hope you find these new features useful and look forward to feedback!

Best,

The Pupil Dev Team