A face-machine interface utilizing EEG artifacts from a neuroheadset for simulated wheelchair control

Signal acquisition and preprocessing

The Emotiv EPOC X is the latest version of the 14-channel EEG neuroheadset produced by the Emotiv company (Hernandez-Ossa et al., 2020). This device was designed for human brain research to provide access to professional-grade brain data with an improved and easy-to-use design. For software, EmotivPRO has been developed for neuroscience research and education. In addition, EmotivBCI software was developed and implemented for brain-computer interface research. For the 14-channel EEG acquisition, electrodes were positioned on both sides of the brain lobe at positions AF3, F3, F7, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4. The reference electrodes were positioned on the parietal lobe at positions M1, P3, M2, and P4 to enhance the generated EEG signals. The electrode positions followed the international 10-20 electrode placement system. The Cortex application programming interface (API), an application programming interface developed by Emotiv, was used for streaming the obtained data to develop the abovementioned third-party applications. The Cortex API was built on JavaScript object notation (JSON) and WebSocket, which enables it to easily access various programming languages and platforms. This API provides EEG data in JSON format with a sampling rate of 256 Hz, and the program was implemented in Python for McGill immersive wheelchair simulator (miWE) control (https://emotiv.gitbook.io/epoc-x-user-manual/introduction/introduction-to-epoc-x; Pirani et al., 2018; Routhier et al., 2018; https://atrehab.ca/research/wheelchair-training/). The components are shown in Figure 2.

Figure 2:

Feature extraction and algorithms

In this paper, we used the Emotiv EPOC X neuroheadset to acquire EEG signals from channels AF3, F7, F3, and FC5, which exhibit strong features when the left eye is winked. For right eye winks, EEG signals from channels AF4, F8, F4, and FC6 also demonstrated a strong feature. Winking with both eyes generated EEG signal patterns from channels AF3, F7, F3, FC5 AF4, F8, F4, and FC6, as shown in Figure 3. Moreover, in this study, we utilized another EEG artifact: the signals induced by jaw movements. During chewing, including chewing on both the left and right sides of the jaw, the EEG channels that exhibited patterns during eye winking again showed clear features but different patterns, as shown in Figures 3 and 4. The process of the proposed face-machine interface system is shown in Figure 5.

Figure 3:

Figure 4:

Figure 5:

Following the determination of EEG features, we selected four channels from among the eight total EEG channels for each participant during preprocessing by using the channel amplitudes. The EEG signals from F7 and F8 were employed for left and right eye wink detection. EEG signals from FC5 and FC6 were used to capture signals while jaw chewing. Using real-time processing, the EEG signals were used to detect actions, and commands were issued every second for the direction control of the virtual wheelchair.

Thresholding

Before executing the proposed face-machine interface system, threshold parameters must be acquired during left and right eye winks as follows:

(1)

$$T_{JL}=mean\left(J_{LF(1)},J_{LF(2)},J_{LF(3)},\dots ,J_{LF(10)}\right)$$

(2)

$$T_{JR}=mean(J_{RF(1)},J_{RF(2)},J_{RF(3)},\dots ,J_{RF(10)})$$

where J_LF represents the variance of the EEG signal obtained from channel FC5, while that of left jaw chewing is calculated similarly to J_L according to (5). J_RF represents the variance of the EEG signal obtained from channel FC6, while that of right jaw chewing is calculated similarly to J_R according to (6). To create the threshold parameters of jaw actions such as T_JL and T_JR, 10 variance values were used to calculate the mean variance:

(3)

$$T_{WL}=mean(W_{LF(1)},W_{LF(2)},\dots ,W_{LF(10)})$$

(4)

$$T_{WR}=mean(W_{RF(1)},W_{RF(2)},\dots ,W_{RF(10)})$$

where W_LF represents the amplitude range of the 10 EEG signals obtained from channel F7 during left eye winking. W_RF represents the range of 10 EEG signals obtained from channel F8 during left eye winking. Then, each set of 10 range numbers was used to calculate the threshold parameter of the corresponding winking action (T_WL and T_WR).

Parameter setting

The features acquired during facial movements, J_L and J_R, are defined as the feature parameters of left and right jaw chewing, respectively, and W_L and W_R are defined as the feature parameters of left and right eye winking, respectively; these four parameters are calculated as follows:

(5)

$$J_L=variance(X_{FC5}[1],X_{FC5}[2],\dots ,X_{FC5}[n])$$

(6)

$$J_R=variance(X_{FC6}[1],X_{FC6}[2],\dots ,X_{FC6}[n])$$

(7)

$$W_L=range(X_{F7}[1],X_{F7}[2],\dots ,X_{F7}[n])$$

(8)

$$W_R=range(X_{F8}[1],X_{F8}[2],\dots ,X_{F8}[n])$$

where X_FC5, X_FC6, X_F7, and X_F8 are the EEG signals acquired in real time at channels FC5, FC6, F7, and F8, respectively, with a sampling rate (n) of 256 Hz.

Decision making

As the feature parameters were acquired, simple decision rules were used to compare the real-time features and threshold parameters that were obtained. The classification decisions for the seven commands were produced as follows:

Command translations

For command translations, the first proposed command modality, we controlled the forward movement by jaw chewing (turning left by moving the left side of the jaw and turning right by moving the right side of the jaw) and the backward movement by winking with both eyes. Moreover, we created a second modality that uses both eyes and the jaw; this modality is similar to the first proposed command, but we changed the command activities from those utilized in the previous command. We used the jaw to control the forward movement with normal chewing and used eye winking for turning left and right. For backward commands, we used the winking of both eyes, as shown in Figure 6.

Experimental results and discussions

Experiment II

Normally, the user’s confidence level has a relationship with the result. Before starting Experiment II, we tried to control the participant’s confidence by achieving a greater than 85% accuracy in Experiment I and allotted 20 min for a training session. We also recorded the time of each participant for steering the simulated power wheelchair using a joystick for the user and system evaluations.

Each participant was tested with three modalities to freely control a virtual wheelchair, as shown in Figure 7a. Each route was performed three times for each modality. The times taken from start to stop were recorded to evaluate the proposed control modalities and the resulting user performances. An example of the experiment is illustrated in Figure 7b.

Figure 7:

Figures 8 and 9 present efficiency comparisons between the proposed control modalities and a joystick based on the time required to steer the simulated power wheelchair along route 1 and route 2, respectively. For route 1, the average time required by the joystick (conventional control modality) was 55 sec, the average time required by the first control modality was 156 sec, and time required by the second control modality was 122 sec. The least amount of time taken using the first control modality was 118 sec, that using the second control modality was 107 sec, and that using joystick control took only 47 sec. For route 2, the average time taken by the joystick control modality was 57 sec, the average time required by the first control modality was 160 sec, and the time required by the second control modality was 127 sec. The least amount of time taken using the first control modality was 102 sec, that using the first control modality was 63 sec, and that using joystick control took only 47 sec.

Figure 8:

Figure 9:

Comparing all modalities, we found that the second control modality could achieve a higher efficiency than that of the first control modality for all tested routes but lower efficiency than that of the joystick. The difference between the average times taken by the second control modality and the joystick on route 1 was 67 sec, and that on route 2 was 70 sec. Participants 1 and 2 with BCI experience demonstrated a high efficiency when using the second control modality; the efficiency was close to that achieved using joystick control. However, some participants may have difficulty performing and may need more time for training. Efficiency comparisons with previous works in real-time discontinuous control (Jang et al., 2016) showed that the proposed system can produce an elapsed time and command transfer rate similar to those of previous works. According to the results, the proposed face-machine interface can be further implemented in a real-power wheelchair.

Conclusion

In this work, we proposed utilizing EEG artifacts obtained from an Emotiv neuroheadset for human-machine interface system-based practical machine control. The advantages of the EEG neuroheadset are that it is flexible and easy to install for signal acquisition. For the proposed control modalities, we employed eye winking and jaw chewing to create seven command channels. The two control modalities were demonstrated via simulated wheelchair control. Incorporating eye winking and jaw chewing into the system can result in high efficiency, and this approach can be developed to increase the efficiency further until it is close to that of using joystick control. Nevertheless, some limitations of the use of the proposed real-time face-machine interface system to control a simulated wheelchair are as follows. (i) The system required the training of some users who had difficulty controlling only the left or right sides of the eye and jaw movements to generate clear features for achieving high user and system performances. (ii) Over a long period of time, the system required adaptive threshold calibration and detection of the fatigue period to avoid a high error rate. (iii) Following the initial verification of the proposed system with only the directional control of the simulated wheelchair, we aimed to further enable additional speed control. We expect that the face-machine interface system can achieve performance equivalent to using a joystick and hands-free control. For future applications, we will employ the proposed system to control real-power wheelchairs or electric devices for serving people with quadriplegia.