gms | German Medical Science

Background: State-of-the-art deep learning [1] requires vast amounts of accurately labeled training data to enable high classification performance [2]. To obtain sufficient amounts of data, data donation is a feasible approach [3]. Yet, the data is only usable, if correct annotations are present. One way to create such annotations is crowd-sourcing via gamification. In this paper, we present an integrated training and annotation approach that allows large-scale annotation of ophthalmic diseases.

Methods: We created a game called “Odin's Eye” in order to make image classification an exciting and rewarding experience. The game has three modes that are used to slowly lead the player to the complex field of ophthalmic diseases. To do so, we used image data from the ODIR 2019 Challenge (https://odir2019.grand-challenge.org/dataset/). The dataset contains more than 7000 images showing healthy eye data and different pathologies.

Figure 1 [Fig. 1]

In “Normal Mode”, the player is shown one fundus image and is asked to select whether the shown image is “normal” or shows Cataract, Macular Degeneration, Glaucoma, Retinopathy, or Myopia. After a sufficient number of correct answers, the player can access the unlabeled mode, in which the user is able to give annotation to unlabeled image.

Figure 2 [Fig. 2]

The “Difficult Mode” is inspired by puzzle games such as Candy Crush or Zookeeper. Here, the player is displayed various fundus images and is asked to align them such that three images showing the same pathology form either a row or a column. Once such a triplet is found, it is eliminated and new images enter the game canvas from the top. To make the game more challenging, a curtain enters the field of view from the top that gradually increases the pressure on the player as well as increases game difficulty via decreasing the image brightness. After each successful elimination, the curtain is raised a little and the time limit is increased. Rewarding strategies like reward animation and image shuffling are applied to achieve better user experience. The idea of using triplets is appealing as images of unknown pathology can be mixed in the game area. The player will implicitly classify such images when he tries to align them with two more images of the same class. Only unique assignments will add to ground truth.

A leader board shows high scores in order to encourage players to a large number of annotations.

Results: A prototype was implemented in Unity [4]. To test the game idea, only five prototypical images were chosen for each pathology at present. The difficulty of the game can easily be increased by increasing types of diseases. First experiences with test players confirmed that in particular, the “Difficult Mode” has a very rewarding game experience. Prototypes are available for Android and WebGL (https://www.medicaldatadonors.org/index.php/odins-eye/). An in-game video was created to demonstrate the gameplay (https://youtu.be/UehnND9gkvY).

Conclusion: A believe that we created a challenging yet rewarding game experience. In future versions, we will make use of additional images from the ODIR 2019 dataset in order to create slowly increasing difficulty to convince even expert players to keep playing Odin's Eye.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.