Towards Automatic Cybersickness Detection, Early-Prediction, and Reductions for Virtual Reality Applications
Virtual reality (VR) has gained immense popularity in recent years with the rapid development of head-mounted displays (HMDs). The use of VR applications in entertainment, education, training, rehabilitation, and low-cost HMDs, created a widespread userbase for VR. However, VR experience often creates motion-sickness-like discomforts, commonly known as Cybersickness which can cause general discomfort, headache, nausea, fatigue, etc. Commonly, researchers use pre and post-immersive subjective questionnaires (e.g., simulator-sickness questionnaire) to measure the severity of Cybersickness, which does not provide a sufficient granular understanding of the severity during immersion. Recent research has reported that Cybersickness affects a user's physiological signals such as heart rate, breathing rate, galvanic skin response, eye blinking rate, delta, and beta power of electroencephalogram - brain signals. These feedback data are highly correlated with Cybersickness reported by prior research, which can be utilized to predict the onset of Cybersickness during VR immersion. In this dissertation, I have developed several novel approaches for automatic cybersickness detection, prediction, and reduction. In my first research, I collected the user's physiological feedback such as heart rate, breathing rate, galvanic skin responses during a roller-coaster VR immersion from 23 participants. I used a convolutional long-short-term memory-based neural network to detect cybersickness severity with an accuracy of 97.4%.Furthermore, I also presented how to predict the onset of Cybersickness two minutes earlier using physiological data with an accuracy of 87.38%. However, this work heavily relies on external physiological sensors such as heart rate and galvanic skin response sensors, which are often difficult to integrate as standalone systems. Therefore, it was not suitable for consumer VR use. To address this problem, I conducted further user studies with 30 individuals. I developed a multimodal deep-fusion network to predict the onset of Cybersickness using only eye-tracking and head-tracking data with an accuracy of 87.7%, which is more effective as it does not rely on any external sensors for cybersickness prediction. Later, I developed a closed-loop framework that utilized the prior cybersickness severity prediction approaches and based on the predicted cybersickness severity. It applies different cybersickness reduction strategies in real-time. However, prior research reported that once cybersickness onsets have started, it is likely to persist. Therefore, applying reduction techniques once the onset has begun might not be effective (i.e., real-time cybersickness detection). Forecasting (i.e., early prediction) of cybersickness severity could facilitate sufficient time to apply cybersickness reduction techniques, potentially reducing the severity before the discomforts occur. Therefore, I evaluated several state-of-the-art forecasting models such as Attention-based LSTM, Neural basis expansion analysis for interpretable time series(NBEATs), deep temporal convolutional neural network (DeepTCN) to forecast (i.e., 30, 60, 90, 120, and 150 seconds prior) the onset of Cybersickness. The results suggested that the multimodal DeepTCN model can forecast the onset of Cybersickness 90 seconds earlier than the actual Cybersickness occurs, with a root-mean-square error value of 0.54 (on a scale from 0-10). The results also suggested that fusing eye-tracking, heart-rate, and galvanic skin response data outperformed the other data fusion approaches. This investigation provides insight into utilizing deep neural networks to automatically forecast and detect cybersickness severity during VR immersion and adaptively apply cybersickness reduction techniques. Furthermore, this research can be used as a guideline to develop automated cybersickness reduction toolkits for VR applications.