Eye tracking and virtual reality in the rehabilitation of mobility of hemianopia patients: a user experience study

Materials

We used a head mounted eye tracker (Pupil Labs: Pupil Core; Kassner, 2014), which field of view horizontally and two eye cameras recording the eye movements at a sampling rate of 200 Hz. The eye tracker was connected to a laptop, which had the Pupil Labs software installed (Pupil Labs, Berlin; https://github.com/pupil-labs/pupil). The Pupil Labs software provides an interface through which the eye tracker can be calibrated, the data can be recorded, and the recordings can be replayed on top of the video recordings from the world camera.

Procedure

Study 1 was performed at Royal Dutch Visio in Amsterdam. The study was approved by the Ethical Committee of the Psychology department of the University of Groningen and all participants provided informed consent. Each participant completed one, two or three exercises from the IH-CST protocol while their eye movements were tracked with a head mounted eye tracker. Some exercises were repeated several times during a training session and eye movements were recorded each time. Participants were instructed to perform the scan pattern that they had already learned in previous steps of IH-CST training. As participants with hemianopia were clients of Royal Dutch Visio, who were at different stages of their rehabilitation, they only performed exercises that were suitable for their current training stage. They could be split into three groups: beginners (2 participants), intermediate (3 participants) and advanced (2 participants). Beginners only performed exercise 1, intermediates performed exercise 1 and 2, and advanced participants performed exercise 1, 2 and 3. Table 1 shows which participant performed which exercises. The eye movement datasheet and the movie of the eye movements projected onto the visual field of the patients were saved. Individuals with hemianopia and their occupational therapist watched the video recordings of the eye movements directly after each exercise, with one exception where the video recording was discussed about a week later. The occupational therapists assessed whether the required scanning pattern had been performed correctly based on the video recordings and gave feedback to the individuals accordingly. Individuals with hemianopia filled out part A and the occupational therapist, who accompanied them, filled out part B of the questionnaire after a test session. This means that beginners rated the eye tracker based on their experience in exercise 1, intermediates based on their experience in exercise 1 and 2, and advanced participants based on their experience in all exercises. This is the same for the therapists who accompanied them. We determined the average score over one questionnaire of each participant, as well as the average score over all questionnaires of each participant group (beginners, intermediates, advanced and their corresponding occupational therapists). A score of above 2 out of 4 points indicates an overall positive user experience. We also collected the comments that were made by all participants about a certain criterion to illustrate which specific aspects need to be changed to improve the usability of the mobile eye tracker.

Exercise 1: Applying the scanning pattern in a static condition

Participants had to call out 30 or 60 numbers presented on a large screen (2x2 m, viewing distance: 85 height of the numbers: 3cm). They initially fixate a cross in the centre. Then numbers were displayed at different locations on the screen. The patient needed to use the learned scan pattern (without head movements) to see all numbers. Each exercise took on about 1-3 minutes, the exact time depending on the reaction times of the participant. Participants wore the head mounted eye tracker while performing exercise 1, seated in a chair with the laptop, to which the eye tracker was connected, outside their field of view.

Exercise 2: Applying the scanning pattern in a simple indoor environment with static obstacles.

Individuals with hemianopia walked along a hallway of 17 m length and 1.5 to 3 m widths in the therapy centre, while performing the learned scanning pattern. This exercise can be varied, by introducing obstacles and targets on the walls, that the participant needs to see. In this study participants performed the exercise with coloured cards pinned to the walls, of which they have to report the colour to the therapist. Individuals with hemianopia performed this exercise wearing the head mounted eye tracker, while the occupational therapists accompanied them carrying the laptop, which was connected to the eye tracker. Individuals with hemianopia were instructed to perform the scanning pattern while walking, as they would usually do this exercise during the IH-CST. This exercise takes approximately 3 minutes, in which the participant walks along the hallway in both direction.

Exercise 3: Applying the scanning pattern in outdoor mobility situations.

Individuals with hemianopia and occupational therapists crossed a road and walked down a footpath, while the individuals with hemianopia were wearing the head mounted eye tracker and a backpack with the laptop was connected to. The individual with hemianopia was instructed to perform the learned scanning pattern while walking. All participants crossed the same road and walked along the same footpaths. As this exercise was performed in public space, the traffic conditions were not the same for all participants. Crossing a road once took about 1 min and walking along a footpath was recorded for 1:40 minutes per recording.

Study 2) Virtual reality

Materials

The exercises in virtual reality were presented with a HTC Vive (Manufacturer, City; horizontal visual field size: 110 degrees, refresh rate: 90 Hz, resolution: 1080 x 1200 pixels per eye). Two motion sensors (called Vive base stations) tracked the position of the headset in space by creating infrared pulses that are detected by the headset. The range of motion, which that can be tracked, is determined by the distance between the two sensors. They are therefore placed at opposite ends of the setup.

We used two different setups of the hardware. The first setup was used while developing the virtual environments. The second one was used in the final experimental session with our participants.

Setup 1: The VR headset was connected to a laptop that was placed on a table, with a cable of 4 m length. The maximum range of movement of the person wearing the headset walking in a straight line was about 4 m. due to the size of the room in which the experiment was performed. The motion sensors were placed at either end of the setup, meaning that they were approximately 4 m apart.

Setup 2: The VR headset was connected to a PC that was placed on a cart. The motion sensors were placed about 5.5 m apart. The therapist moved the cart with the PC so that the range of movement for the participant was 5.5 m. Virtual environments were developed by the ‘The Virtual Dutch Men’ using Unity (https://thevirtualdutchmen.com).

Procedure

Study 2 was performed at Royal Dutch Visio in Haren. The study was approved by the Ethical Committee of the Psychology department of the University of Groningen and all participants provided informed consent. We first met with our VR developers for two development sessions, where they presented a first version of the requested virtual environments for three different exercises. These first versions were tested by one individual with hemianopia, an age matched control and authors of the study. Two of these exercises were further developed into their final versions and tested in an experimental session by the participants mentioned in the ‘participants’ section above. For completeness we will describe all three exercises, including the first version of exercise 3, which was performed by one individual with hemianopia (70 years) and 3 controls (mean age 43). Individuals with hemianopia and normal-sighted controls filled out a questionnaire after each exercise. The occupational therapist filled out one questionnaire for each exercise at the end of the experimental session. We computed the average score that was given for exercises 1 and 2 of each participant group (individuals with hemianopia, normal-sighted controls and the occupational therapist) in their final versions. We also computed the score that was given by the individual with hemianopia and each of the controls given to exercise 3. In addition, we collected the comments that were made by each participant about each exercise to evaluate how we can improve the exercises in the future.

Exercise 1: Walking along a virtual hallway

Normal-sighted controls and individuals with hemianopia started walking at one end of the virtual hallway and had to walk to the other end. The view for the participant when starting this exercise is shown in Figure 1A. Figure 1B shows the entire hallway from above to get an overview over the full trajectory (not shown to the participant). The position of the headset was tracked in space so that participants perceived motion in VR in synchrony with their actual motion in the setup. As the range of motion, that our final setup allowed, was too short to complete a walk along the entire trajectory of the hallway, participants turned around after 5.5 m. While turning around, a black screen was shown and the virtual environment was turned 180 degrees, so that the participants continued to walk along a new part of the hallway.

Figure 1:

Three different versions and six scenarios (differing in the amount of obstacles in the hallway) of a virtual hallway scenario were developed. Details about these versions and scenarios can be found in Table 3. Participants tested several scenarios of this exercise, depending on how long completing a certain scenario took. The number of scenarios tested differed between participants.

Results

Study 1) Eye tracking

All exercises

The overall feedback for the head mounted eye tracker was very positive with a mean score of 3.7 ± 0.5 given by the beginners, a mean score of 3.8 ± 0.5 given by the intermediates and a mean score of 4 ± 0 given by the advanced individuals with hemianopia. The corresponding therapist questionnaires gave an average score of 3.6 ± 0.6, 3.4 ± 0.9 and 3.8 ± 0.6. Figure 2 shows the average score and standard deviations of each participant.

Figure 2:

We identified four main criteria that were most important for the user experience, based on the comments mentioning them:

• 1. Usability in mobile situations and outside

• 2. Usability with glasses

• 3. Improvement of feedback

• 4. Being easy to use (no disruption of regular procedure of therapy session)

Table 4 provides an overview of the comments made by individuals with hemianopia and occupational therapists.

Table 4.

Score and comments for the head mounted eye tracker for each criterion evaluated by patients and therapists.

Study 2) Virtual reality

Exercise 1: Walking along a virtual hallway

The feedback of individuals with hemianopia and normal-sighted controls was more positive than the feedback from the occupational therapist for this exercise.

Individuals with hemianopia rated this exercise with an average score of 3.9 ± 0.5, controls with an average score of 3.8 ± 0.5 and the occupational therapist with an average score of 2.5 ± 1.4. Figure 3 shows the score given by each participant.

Figure 3:

The most important challenge in this exercise was to simulate the movement of the participant over a distance of several meters in the virtual environment, which is much longer than the range of movement in exercise 2. The current setup of the hardware did not allow for a large enough range of movement to walk all the way to the other end of the hallway, although using setup 2 improved it. Our approach of letting the participants turn around before continuing along the next section of the hallway, was not a good solution as it led to a feeling of insecurity by participants, when they had to turn around without visual input. This comment was made in several questionnaires. An overview over the comments, which were specifically relevant for this exercise, is shown in Table 5.

Discussion

The main finding of this study is that the user experience of individuals with hemianopia and normal-sighted controls was generally positive for both tested devices. While the occupational therapists, who evaluated the head mounted eye tracker, also reported a positive user experience the occupational therapist, who evaluated the virtual reality setup, reported a more negative experience. Below, we describe our findings, conclusions and these experiences in more detail.

Providing feedback by showing replay of eye movements improves the user experience of individuals with hemianopia and therapists in mobility training.

A positive aspect, noted by individuals with hemianopia and therapists, of using the head mounted eye tracker during the therapy sessions was that it offers the possibility to replay the eye movements to give feedback to the participant. This was seen as a way to improve therapy progress, as individuals with hemianopia are able to reflect on their own viewing behaviour. Moreover, it helps the therapists to explain the (need for) compensatory strategies. All participants and therapists saw the recording of the eye movements during the four exercises as a beneficial addition to the training The device fulfilled most of the requirements of individuals and therapists.

A problematic aspect was the usability with glasses, as the eye tracker was less precise in tracking the pupil when participants were wearing their glasses, especially under varying luminance conditions. This is a common problem for eye trackers, as glasses and contact lenses and changes in luminance distort the image of the eye that is captured by the eye tracker (Dahlberg, 2010; Fuhl et al., 2016). The Pupil Labs eye tracker provides a confidence interval of the recorded data, which can be used to judge the reliability of the recorded eye tracking data.

VR Scenarios

All participants (individuals with hemianopia and normal-sighted controls) reported a preference for ‘lively’ and ‘realistic’ scenarios in virtual reality. Most participants thought that adding exercises in virtual reality would benefit vision rehabilitation therapy. They also think that if we can add the possibility to replay a recording of the exercise from the point of view of the participant, which also shows the eye movement behaviour, this will improve feedback.

These findings are in line with a study by Ishoel and Kanstad (2017), who tested different head mounted VR displays and concluded that VR is a promising method for vision rehabilitation therapy. While Ishoel and Kanstad developed a mini game, which is supposed to support therapy progress, we attempted to make the current virtual reality scenario resemble the real-world exercise as much as possible by letting our participants walk within the setup. This approach makes it easy for participants to understand the task and, especially for individuals with hemianopia, to understand the objective of the exercise and transfer learned skills. It has previously been shown that using VR to train patients with unilateral spatial neglect can improve their visual – spatial performance, as well as their ability to cross a real street (Katz et al., 2005). However, the setup described by Katz et al. was not immersive and did not require the participants to move. In a study by Iorizzo et al. (2011) individuals with hemianopia and normal-sighted controls were presented with an immersive virtual environment in which they had to detect basketballs that were presented in the periphery. This task was performed seated as well as while walking in an L-shaped path. They found that individuals with hemianopia had more difficulty detecting targets in the periphery when walking than the control group. This finding emphasizes that it is necessary for individuals with hemianopia to practice eye movements while walking. Under ideal circumstances, the immersive VR environment that we created can be used while a large space is available in which the participats can safely move the required distances. To walk through the complete virtual hallway at once, the available real walking distance should be long enough, for example the same length as the actual hallway, in which the exercise is performed. Preferably, it would be even larger to allow for directional errors. At present, this was technically not feasible as it would also require that the sensors of the VR headset had to be placed much further than 5.5 meters apart, which would have made tracking of the headset impossible. In the first version of this exercise, we tested two options to compensate for this. The first idea was that one step in reality corresponded to several steps in the virtual world. The second idea was to let the participant continue along a new part of the hall virtual hallway when they turn around at the end of the setup. The participant could continue to walk along the virtual hallway by walking back to the other side of the room. As the first caused dizziness and insecurity when walking, the second idea was implemented in the following sessions. Our solution to let participants turn around after a few meters however interrupted the flow of the task and this was experienced as irritating by the participants, both the normal-sighted controls and the individuals with hemianopia. Another problem of our current setup 2 was that the cart with the computer had to be dragged along next to the participant. In our setup, the power cable for the computer was often in the way causing problems to move the cart. Therefore the occupational therapist was assisted by two people to move the cart, which is an undesirable solution in an actual therapy sessions. When turning around at the end of the setup, the cable of the VR headset was also often in the way and participants had to be instructed such that they would only turn around in one direction.

Therefore, to be able to perform these exercises within a regular training session, the setup will need to change fundamentally. Despite these difficulties, the participants and the therapist were of the opinion that adding VR exercises to the regular training sessions will be beneficial. Instead of using a larger space, which can be costly or simply not available, the space issue may also be solved by adding an alldirectional treadmill to the VR setup that will enable walking longer distances in all directions (e.g. https://www.virtuix.com). A further way to improve future setups is to add eye tracking to the VR exercises, which is technically feasible by now (e.g. https://pupillabs.com/products/vr-ar/).

Limitations

In this study, not all of our participants completed and evaluated the same exercises. This has some practical reasons. In study 1 the eye tracker was tested during regular training sessions of the participants. Therefore they only performed exercises that were suitable for their current training stage. VR exercise 3 was not included in the final test session, due to time constraints, meaning that there was too little time for the developers to continue developing this exercise and there would also not have been enough time on the day of testing to let all participants perform another exercise. We still decided to report the results of the questionnaire for this exercise here, as the participants liked the virtual environment and we therefore plan to continue its development in the future. This study does not evaluate the benefit in terms of training progress of adding eye tracking and VR to the current protocol. This will be the topic of a follow up study. The results of this study represent the subjective experiences of our participants. In our opinion a positive user experience is an important aspect for the successful integration of these devices in the IH-CST training.

Conclusion

This study shows that integrating eye tracking and VR technologies into vision rehabilitation therapy may positively impact training outcomes. Individuals with hemianopia are able to use these technologies and they consider these helpful additions to the existing training exercises. Therapists welcome the additional objective information provided but indicate that VR setups first need to allow for a larger range of mobility. Moreover, integrating eye tracking into the setup could further increase the value of VR. relevant for this exercise, is shown in Table 5.