gms | German Medical Science

Objective: The objective of this study was to explore the feasibility of a visual fixation-based retinal prosthetic simulation using a simulated saccade mechanism, and to assess the improvements achieved through corresponding end-to-end optimization approaches.

Materials and Methods: Fixations were predicted using images from the ImageNet dataset, leveraging self-attention from a pre-trained Vision Transformer. Out of the 256 patches (16x16) from each image (224x224 pixels), the top 10% most salient fixation patches were preserved to mimic the saccade mechanism. Each fixation patch (14x14 pixels) was encoded with a trainable U-Net optimizer and then simulated using the Axon-Map Model from the pulse2percept library to predict percepts. The resulting masked percepts were evaluated with a self-supervised foundation model (DINOv2), with an optional learnable linear layer for classification accuracy.

Results: Classification accuracy was measured on a subset of the ImageNet validation set (3,952 images, 10 classes). The visual fixation-based approach achieved 81.99% accuracy, compared to 38.70% using a downsampling approach. The accuracy was further improved to 87.72% with the inclusion of an end-to-end U-Net encoder. For comparison, the healthy upper bound achieved 92.76% accuracy.

Discussion: The visual fixation-based retinal prosthetic simulation shows promising potential, drawing inspiration from the saccade mechanism of the human eye while efficiently utilizing the limited number of electrodes in retinal implants. End-to-end optimization further enhances classification accuracy, making this approach a compelling advancement for retinal prosthetics.

Acknowledgment: This work was supported by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) with the grant GRK2610: InnoRetVision (project number 424556709).