Multimodal Feedback Techniques to Increase Accessibility of Immersive Virtual Reality

Virtual Reality (VR) users often experience balance problems and gait (i.e., walking patterns) disturbances, which could be a major barrier to universal usability and accessibility for all, especially for persons with balance impairments (BI). Prior research has confirmed the imbalance and gait disturbance issue, but minimal research has been conducted to reduce this effect. To solve this issue, this dissertation proposes 16 novel assistive feedback techniques (four auditory, four vibrotactile, four visual, and four multimodal) to improve balance and gait while immersed in VR. In seven different studies, 431 participants (178 with BI due to multiple sclerosis, 75 with BI due to diabetes, and 178 without BI) were recruited to evaluate the developed techniques. In my first user study, I developed several auditory feedback techniques and recruited 42 participants (with BI: 21, without BI: 21) to investigate the impact of the auditory techniques on balance and gait in VR, specifically spatial audio, static rest frame audio, rhythmic audio, and audio mapped to the center of pressure (CoP). Participants performed three types of tasks - standing visual exploration, standing reach and grasp, and timed walking task using GAITRite. Within-subject results showed that each auditory technique in VR improved balance and gait significantly more (p < .001) compared to the no feedback in VR for both persons with and without BI. Spatial and CoP auditory feedback improved balance and gait significantly more (p < .001) than other auditory conditions. However, there was no significant difference between spatial and CoP auditory feedback or between rhythmic and static auditory feedback.In my second user study, I developed several vibrotactile feedback techniques and recruited 42 participants (with BI: 21, without BI: 21) to examine the effect of the vibrotactile feedback techniques on balance and gait in VR, specifically spatial vibrotactile, static vibrotactile, rhythmic vibrotactile, and vibrotactile feedback mapped to the center of pressure (CoP). Participants completed standing visual exploration, standing reach and grasp, and walking tasks. According to within-subject results, each vibrotactile feedback enhanced balance and gait in VR significantly (p < .001) for those with and without BI. Spatial and CoP vibrotactile feedback enhanced balance and gait significantly more (p < .001) than other vibrotactile feedback. However, there was no statistically significant difference between spatial and CoP conditions, nor between rhythmic and static conditions.In my third user study, I investigated how to increase VR balance and gait by utilizing additional visual cues. To examine how different developed visual techniques (static, rhythmic, spatial, and center of pressure (CoP) based feedback) affect balance and gait in VR, I recruited 50 people (25 with BI due to multiple sclerosis and 25 without BI). Participants completed standing visual exploration, standing reach and grasp, and walking tasks. Results demonstrated that static, rhythmic, and CoP visual feedback approaches enhanced balance and gait significantly (p < .05) in VR for people with BI. Static and rhythmic visual feedback performed significantly better (p < .05) than the CoP visual feedback. This study was inconclusive between static and rhythmic visual feedback, with no significant difference. Spatial visual feedback had no significant effect. Also, there was no significant effect of the visual feedback conditions for participants without BI.In my fourth, fifth, and sixth user studies, I replicated the first, second, and third user studies respectively. However, I recruited the participants from Bangladesh instead of the United States to investigate the effect of the developed techniques on populations from different geographic locations. I also included participants with BI due to type 2 diabetes for these three user studies to investigate how the developed techniques affect a different group of participants. For each project, I recruited 25 people with BI due to MS, 25 people with BI due to type 2 diabetes, and 25 people without BI. I found similar results in Bangladesh. Also, I found similar results for participants with BI due to MS and type 2 diabetes.In my final user study, I conducted a study with 72 participants (36 with BI due to MS and 36 without BI) to investigate the effect of multimodal feedback - combined auditory, vibrotactile, and visual feedback - on balance in immersive VR. Results showed that multimodal feedback improved balance and gait significantly more (p < .001) than any other condition. Combining two feedback modalities performed better than using just one. However, mental load (p < .05) and fatigue levels(p < .05) were significantly higher for multimodal condition compared to the conditions where auditory, vibrotactile, or visual feedback were applied separately.The results will help researchers better understand the different kinds of auditory, vibrotactile, visual, and multimodal feedback for maintaining balance and gait in HMD-based virtual environments. Moreover, this research can help developers create VR experiences that are more usable and accessible to people with and without balance impairments and push VR closer to universal usability.