The usability evaluation of the serious game was approved by the Ethics Committee for Human Research of the Federal University of Uberlandia, under the Certificate of Presentation of Ethical Appreciation: 43229921.8.0000.5152.
To evaluate the usability of the RehaBEElitation, we used user-centered methods. These analyses are performed with two different approaches: using survey-based methods, in which the user fills out evaluation questionnaires after interacting with the system; and using observational analysis, in which the user interacts with the system while the developers observe and evaluate that interaction.
A group of five PD patients was involved in the study (one female, four male, 68 ± 5.83 years). The participants received brief instructions about the context of the game and were asked to play the four phases for 20 minutes each, or until they complete the phases’ objectives. Users were instructed to speak aloud during the game, expressing their thoughts and impressions. Participants’ interactions with the game were recorded on video using a camera, which captured the user’s hand and reactions; and the game screen was simultaneously recorded.
Immediately following the end of the session, each participant answered the System Usability Scale – SUS, the Game Experience Questionnaire – GEQ and the Short Flow State Scale – SFSS.
After the first session of the participants’ interaction with the game, the recorded video showing the user interacting with the game was synchronized with the one showing the game screen. Next, the videos of the five participants were submitted to four evaluators, who composed a multidisciplinary team from the fields of game development and design, physical therapy, biomedical engineering and physical education. The purpose of analysing the videos was to evaluate the participants’ interaction with the game, identifying and recording all significant events, which occur when the user finds a problem or visibly reacts to the game. The evaluators’ observations were conducted using the Serious Game Usability Evaluator – SeGUE.
Then, as the evaluators watched the videos and took notes independently, a reconciliation of their results was performed to obtain a good agreement. Finally, a list of game changes was prepared. After making the changes in the game, the five PD patients interacted with the RehaBEElitation again and answered the questionnaires to validate the changes made. The t-test and Wilcoxon-Mann-Whitney test were used to confirm differences in questionnaire responses before and after the game modifications.
Statistically significant differences were found for all questionnaires except for the GEQ “negative affect”, “tension”, “negative experience” and “tiredness” items. The mean scores of all questionnaires increased after the game modifications, indicating that the changes improved the usability of the game (with the exception of the GEQ items that did not show significant differences and the “challenge” item). The item that showed the greatest difference after the game improvements was the users’ competence to play the game (p = 0.0002).
The main problems found in the game were: 1 – the glove did not correctly identify the movements of opening and closing the hand and finger tapping; 2 – the position of the targets in Phase 1 was not well adjusted; 3 – the 3D hand picture that appeared on the screen when the player reached the end of the scenario in Phase 1 still appeared open, even though the player was holding the pollen; and 4 – because the game scenario is three-dimensional, the targets in Phases 2 and 4 that were located closer to the edges of the screen appeared to be distorted.
The actions suggested by the evaluators to solve these problems were: 1 – improve the ergonomics and structure of the glove. Use a thinner fabric glove and sew more conductive lines on the palm of the hand and fingers; 2 – turn the entire target area into the target itself in Phase 1; 3 – leave the hand that appears in 3D on the game screen in the same state as the player’s hand is at the end of the scenario in Phase 1; 4 – remove targets from areas too close to the edges of the screen in Phases 2 and 4.
With these results we could observe that, as the game presents narrative and interface that are simple, intuitive and in accordance with reality, the game is consistent with the individuals’ mental models, which facilitates their interaction. In addition, the participants of the study reported that the experience of playing the RehaBEElitation was extremely pleasant, indicating a great acceptance of the game, and the modifications made in the game contributed significantly for increasing the usability of the overall system developed.
The individuals with PD who participated in this research were recruited from a philanthropic institution for patients with Parkinson’s disease and Parkinsonism (Associação de Parkinson do Triângulo), located in the city of Uberlândia – Brazil. All data collection was performed at the Center for Innovation and Assessment of Technologies in Health (NIATS).