Public Report Of The Project's Results: WP2 - System Architecture Definition and Modeling

WP2: System Architecture Definition and Modeling

1. Modeling and simulation of the experimentally-determined retinal architecture

PV retina experimental data was provided by Laboratory for the Structure and Function of Neural Circuits at Friedrich Meinscher Institute (FMI) for Biomedical Research in Basel The data consisted of electrophysiological recordings of the cell membrane voltage and excitatory and inhibitory currents for 286 isolated cells (in ~4400 files). We first developed data mining software for data preprocessing: loading experimental data and headers with experiment details, rescaling/converting the cat natural scenes into mouse natural scenes, spike detection and sorting. To reveal the roles of parallel RGC circuits in visual processing, we examined the firing patterns of eight different genetically-identified RGCs from the mouse retina during presentation of white/black
and short presentations of natural movies.

1.1. Spot stimulus

We first analysed each cell type's response to simple visual input in the form of black and white spot stimulation for a set of six spot sizes (125, 250, 375, 500, 650 and 1200 mm), This enabled the classification of each RGC based on several key physiological metrics. Our quantitative approach was based on strict biophysical parameters, including the receptive field size, surround inhibition ratio, transient/sustained firing (also dependent on the light intensity), bimodal sensitivity to static spatial contrast, and response latency.

1.2. Natural scene stimulus

Next we analysed the firing patterns during presentation of short duration (~10 sec) complex visual scenes (natural movies). Natural stimuli should contain the features that the cells have evolved to become most sensitive to. In order to link the visual input with the spiking output we use information theory. Using a modified form of mutual information ("Quadratic Mutual Information"), we probed the high dimensional space formed by the visual input for a much smaller dimensional subspace of Receptive Field Vectors (RFVs) that give the most information about the response of each cell. RFVs are special types of visual inputs that the RGC circuits will be most sensitive to Derivation of novel types of RFVs formed by the natural scene visual input was possible even with limited numbers of spikes per cell. This approach enabled us to estimate the 'visual memory' of each cell type and the corresponding receptive field area by calculating Mutual Information as function of the number of frames and radius. Finally, we made predictions of biologically relevant functions based on the RFVs of each cell type, including edge detection, spatial contrast sensitivity and motion sensitivity. Some of these biological roles could be confirmed based on previously published data on corresponding cell types, including local edge detectors, approach sensors, and direction selective ganglion cell types. Others (such as 'horizon detector', 'hole in the wall detector', etc) can now be tested in the future based on our mathematical predictions.

Thus RFVs lead to predictions of biological roles based on limited data and facilitate analysis of sensory-evoked spiking data from single defined cell types.

Figure 1: Example of a Receptive Field Vector (RFV): (a) single RFV for a PV2 cell that maximally separate spiking from non-spiking inputs. Redder responses represent higher (illumination) values. Outer circle (diameter 440um) and inner circles (D=200 and 100um_ assist in the estimation of the size of the structures. ),. (b) Average number of spikes generated vs. projections of the input vectors onto the RFV. c) MutuaI Information (MI) contained within an increasing radius across the entire RFV. The decrease in the MI indicates overfitting. The vertical red line represents the identified radius before the onset of the overfitting artifacts. (d) MI vs. number of frames that the RFV contains. The relevant receptive field history (or the Cell Memory) was estimated as in c) and marked with a red arrow.

1.3. Predicitive algorithms

A new algorithm for predicting the spiking behaviour of an RGC on the basis of the known visual input was developed which uses the new multi-class/multi-RFV results. Data was split into Training and Validation Sets and the methodology for a Linear-Nonlinear Model was implemented to compare the results.

Figure 2: Example of the predictive power of the model: a) generator signal (proportional to spiking response function) against projected value of stimuli on receptive field vector. b) Normalised simulated and experimentally measured spike counts for each stimulus frame. Cell class: PV7.

2. New Vision Sensor - System Behavioural Emulation

The following accomplishments were made during the duration of the project:

  • Designed and constructed a vision sensor behavioural emulator (VSBE) based on PS3Eye camera and Java Address Event Representation (jAER) framework. a platform for behavioural emulation of the proposed architecture and functionality of the silicon retina.
  • To interface PS3Eye with jAER a bespoke C++ Windows driver was developed.
  • The VSBE was tested and characterised using several metrics.
  • Testing environment was created (isolation box), with two systems: a 120Hz LCD monitor with an OpenGL Matlab mex interface wrapper used to create rapid, controllable visual stimuli and a electroluminescence panel system for flat field measurements.

Figure 3: Comparison of the ouput of VSBE and a DVS 128 running through jAER.

3. Feasibility study for machine vision and retinal prosthetic applications

3.1. Machine Vision Applications

We successfully combined a vision sensor, specifically a Dynamic Vision Sensor (DVS128) and conventional frame-based QVGA PS3-Eye webcam. For that purpose we used the jAER software. Conventional synchronous imaging sensors provide frame-based RGB video with a relatively high degree of temporal redundancy. On the other hand, activity-driven, event-based imaging sensor provides low resolution, monochromatic video feeds with low latency. By integrating the output from both camera systems we could leverage on the strengths of both types of sensors. We describe and demonstrate various video processing applications achieved using the combined camera system:

  1. An object tracking system that provides faster tracking capabilities than conventional object tracking systems.
  2. Implementation of 2-dimensional Velocity Estimation for Tracked Features.
  3. Foviation - focusing only the moving object(s) - Lock-on tracker.
  4. Real time video compression.

The following youtube videos show examples.

Figure 4: Tracking and estimation of the velocity of a ping-pong ball.

3.2. Retinal Prosthetic/Vision Augmentation Applications

In general retinal prostheses function by bridging the interrupted flow of visual information in damaged retina. To do this, they must emulate the processing characteristics of the retina’s neural circuitry. Insights and understandings of the RGC functions brought by our partner from the FMI Basel were an important input.

We developed a system that can be used as the front-end of a real-time retinal prosthesis system (RP) as well as for a complete vision augmentation system (VA) for severely visually impaired individuals. Such devices usually consist of three components: a sensory block to capture the visual scene, a processing block to manage the collected data and generate stimulus patterns, and an output block. For the sensory block we use a Dynamic Vision Sensor (eDVS) instead of a conventional camera. A microcontroller is used as the processing block, which receives asynchronous inputs from the DVS in the form of ON/OFF events and treats them like post-synaptic potentials. A simple algorithm based on an Integrate & Fire neuron model is used to emulate temporal contrast sensitive Retinal Ganglion Cells (RGCs). For a VA system the output block could consist of an LED matrix (for displaying simplified images), whereas for an RP system the output would be an implanted electrode array. In our case the results are shown in the form of ON or OFF spikes on the LED matrix (equivalent to the stimulation pattern on an electrode array in the case of an RP). Main advantages of the proposed system is that it is a low-power, battery operated body worn system, which avoids image processing on a PC/smart-phone-based system by utilizing an embedded solution that combines a DVS and a microcontroller unit.

In addition we have developed another system which also has a DVS as a front end for collecting the video input, and a microcontroller to process it, but the output is a sound of a particular pitch and intensity played by stereo headphones. We expect for this system to be of more immediate relevance since it does not need any kind of implants.

We have also developed a retinal ganglion cell algorithm that emulates a new type of retinal circuit that is sensitive to approach motion, which was discovered by our partners at FMI. It was implemented using: (a) a microcontroller (PIC18) and (b) an FPGA board.





Figure 5: a) Development prototype of Front End of a Retinal Prosthesis system consisting of a DVS camera, a PIC microcontroller development board and an LED matrix to stimulate RGCs. b) Complete setup in demonstration on display at Science Museum in London.

Figure 6: Sensory substitution system converts the visual input (eDVS) into sound output (headphones) using an Android smartphone.