In order for the user to interact with the game, two versions of a wearable HMI device were developed. Both versions of the HMI have a motion measurement unit, consisting of inertial sensors with a three-axis XYZ accelerometer, a three-axis XYZ gyroscope and a three-axis XYZ magnetometer, capable of estimating the orientation of the hand.
In the first developed version, touch/tactile sensors connected to the palm of the hand and to the fingers estimate the finger tapping and opening and closing movements of the hand. In the second version, light sensors (LDR – Light Dependent Resistor) present on the thumb and at the base of the middle finger, which convert the energy of a light into an electrical signal output, are able to estimate these same movements. The two versions are shown in the following figure.
The HMI measures the movements of the user’s hand as he moves it. The hand orientation in 3D space can be defined by the rotation angles in the X, Y and Z axes. The HMI uses the values measured in the inertial sensor and estimates, through suitable mathematical equations, three rotation angles, which represent the three dimensions of player’s hand movement: roll (extension and flexion of the hand), pitch (supination and pronation of the forearm) and yaw (adduction and abduction of the hand). A microcontroller (ESP32) processes the information from the sensors, and sends it to the game via WiFi so that the user can control the avatar and fulfill the objectives of the phases.
Data acquisition during serious game