ISSN Print 2500–1094
ISSN Online 2542–1204
Bulletin of RSMU
BIOMEDICAL JOURNAL OF PIROGOV UNIVERSITY (MOSCOW, RUSSIA)
ORIGINAL RESEARCH
Development of a neurodevice with a biological feedback for compensating for lost motor functions
About authors
1 Institute of Chemistry and Biology,Immanuel Kant Baltic Federal University, Kaliningrad, Russia
2 Department of Hospital Surgery No. 1, Faculty of General Medicine,Pirogov Russian National Research Medical University, Moscow, Russia
3 Department of Human Pathology, Faculty of Postgraduate Professional Training of Physicians,The First Sechenov Moscow State Medical University, Moscow
Correspondence should be addressed: Evgeny Bogdanov
ul. A. Nevskogo, d. 14, Kaliningrad, Russia, 236041; moc.liamg@vonadgobue
About paper
Funding: the work was supported by the Ministry of Education and Science of the Russian Federation (Grant Agreement no. RFMEFI57815X0140 dated October 27, 2015).
Acknowledgements: the authors thank Alexandr Romanov of the Rehabilitation Center of the Administrative Department of the President of the Russian Federation, Moscow; Raphael Oganov of State Research Center for Preventive Medicine, Moscow; Daniil Borchevkin, Alexey Belousov, Vladimir Savinov, Sergey Sokolov, and Alexey Medvedev for their scientific contribution and productive collaboration.
Received: 2016-03-31
Accepted: 2016-04-07
Published online: 2017-01-05
|
Applied biorobotics improves the quality of life in patients with neurological disorders and traumas. Neuroprostheses, exoskeletons, robotic wheelchairs and telecontrol robots contribute to rehabilitation of patients, substitute for lost functions and enhance physical abilities of healthy people.
Choosing the right control scheme is very important for the development of such devices. It must ensure the accuracy, stability and safety of the device performance, given that the device will be used continuously. The majority of existing solutions are based on recording human body biopotentials using electromyography (EMG), electroencephalography (EEG), electrooculography (EOG) and electrocardiography (ECG) [1, 2, 3, 4, 5, 6].
Robotic wheelchairs, prostheses and exoskeletons are good examples of the effectiveness of EMG-based schemes [7, 8, 9]. However, EMG alone is not enough if an individual who had a stroke or a spinal cord injury cannot generate a muscle signal of the required intensity. In such cases we turn to brain-computer interfaces (BCIs) that transform signals from damaged brain areas into commands for external devices. One of the recent works [10] has demonstrated a high effectiveness of a BCI for neuroprosthesis control tested by a tetraplegic patient with intact sensory and cognitive functions.
Among various methods of brain signals recording, EEG is the most convenient due to its availability, safety, cost- effectiveness and portability. The brain cortex consists of multiple areas of functional specialization in which waves of different frequency are observed [11]. The EEG spectrum is unique for every individual and changes constantly depending on a person’s physiological condition and the activity performed, as long-term measurements have proved [12] By decoding EEG signals, we can discriminate between limb movements quite accurately. For example, the algorithm proposed for the reconstruction of the trajectory of finger joint angles during reach to grasp movements ensured 76 % accuracy of EEG signal [13]. Another work showed that it was possible to correctly identify one out of five actual or imaginary movements of the wrist and fingers with 65–71 % accuracy [14].
For better classification accuracy,a large number of EEG channels is thought to be necessary. However, Yang et al. [15] were able to remove irrelevant noise and improve the EEG signal classification technique that can be applied to a neural network or used for robotic device control. EEG was recorded with only 6 channels out of 32; still, the classification accuracy reached 86 % in some motor tasks. However, EEG-based BCIs have certain drawbacks resulting from incorrect electrode placement, shifting of electrodes, noises, artifacts, imperfect algorithms of filtration and signal processing.
Some researchers suggested that EMG and EEG methods should be fused [16, 17, 18]. For example, in case of paresis or limb loss, EEG signals can be used to compensate for weak EMG signals, ensuring that a prosthesis or exoskeleton is moved by mental effort. If EMG signals are of normal intensity, EEG signals can help reduce the impact of tremor, fatigue or artifacts.
Leeb et al. [19] proposed a hybrid EEG-EMG-based control system; it was tested on 6 healthy individuals. The subjects moved their left or right arm for 5 seconds (there were 60 trials in total). Brain activity was recorded by 16 sensors placed in accordance with the international 10–20 system. Muscular activity was recorded over left and right forearm flexors and extensors. The obtained EMG signals were rectified and averaged (0.3 s) to get the envelopes. The data from two classifiers were fused together to get one control signal. The hybrid system showed high classification accuracy in all subjects. Despite the fact that EMG signals were quite informative (classification accuracy was 83 % in average), the hybrid approach was more effective (classification accuracy was 91 %), especially in case of increasing muscle fatigue.
Xie et al. also developed a hybrid EEG-EMG-based BCI (visualization of movement intention) [20]. Their study enrolled 10 post stroke patients with non-severe hemiparesis, 10 patients with peripheral nerve injury and 10 healthy individuals. All patients were between 20 and 58 years of age. For calibration, subjects were asked to lie on the bed and perform knee flexion and extension tasks. The sensor measured the angle and the force of movements; the obtained data were later used as target levels. Then EEG/EMG sensors were attached, and the experiment was carried out. The aim of the experiment was to establish the correlation between EEG/EMG signals and leg movements and to measure the accuracy of potential control commands for the external device. First, EEG data were processed followed by EMG data processing; in the third experiment the hybrid approach was applied. The results showed that the hybrid approach led to increased classification accuracy in all groups of subjects, compared to single modality approach. In healthy individuals, classification accuracy was 98 %, in post stroke patients — 84 %, for patients with peripheral nerve injury — 85 %.
Kiguchi et al. [21] carried out a study of a hybrid EEG-EMG system for controlling arm movements using SUEFUL-7 robotic device, and assessed its effectiveness [22]. The robot was equipped with a video camera and could detect arm position by rotational angle and force sensors. A 16-channel EMG interface for recording signals coming from arms and shoulders was used as a control system. The experiment enrolled four 23-year-old healthy individuals. Some of them wore an exoskeleton and a device for EEG recording and response monitoring. In the first experiment, the subjects performed arm flexion/extension tasks, and the robot did the opposite impeding the movement. In the second experiment, one full and two empty cups were put on the table. When the subject grasped the empty cup, the robot used the assistance algorithm that estimated the position of empty cups using the video camera, and randomly selected one of them; after that, the robot assisted the subject in pouring the liquid. The accuracy of choice was assessed using EEG and EMG signals. If the subject did not resist, the robot inferred that the target had been chosen correctly. In that experiment, the flexibility of the robot and its ability to correctly interpret the intentions of the subject were tested. The results showed the increased accuracy of interpretation of human actions by the robot.
BCI performance can be improved by oculography data recorded parallel to EEG. A group of scientists designed a hybrid EEG-EOG-based BCI to enhance the reliability of hand exoskeleton for continuous grasping movements [23]. EEG signals were recorded at 5 EEG sites in accordance with the international 10–20 system. The experiment consisted of two parts. In the first part, the subjects controlled the exoskeleton through EEG signals only; they made grasping movements when the visual indicator appeared (green for the movement onset, red for rest). The robotic hand opened automatically if the commands issued by the operator’s brain were not intense enough to get over a preset threshold. In the second part of the experiment, EOG signals were used as a switch. When the subject looked to the left or to the right, the exoskeleton hand opened regardless of EEG signals. The hybrid model increased system safety. When only EEG signals were used, the motion of the robotic hand exceeded 25 % of a full hand closing in half of subjects at rest. The hybrid system showed the increase in the threshold value in 10.4 % of subjects, with maximal grasp being less than 28 % of a full hand closing (in a single modality system it was 60 %) .
Cardiovascular system performance is usually assessed by monitoring arterial blood pressure and heart rate (HR); it correlates to brain activity, including that, during motor tasks [24, 25, 26]. Studies of the effect of changing mental activity on heart activity assessed by EEG show that hybrid EEG- ECG systems are a promising practical tool [27, 28, 29]. In the experiment involving 6 healthy right-handed men (mean age was 28 years), who imagined movements of their left leg or left arm, researchers assessed classification accuracy of EEG signals and ECG signals separately; then a fused EEG-ECG recording was processed [29]. For every subject, 180 sessions were held (60 sessions for each assessment method). They consisted of three parts; the subject rested for the first 6 seconds (while, data from previous sessions were processed); then the subject was presented with an indicator that randomly indicated the action that the subject had to perform (arm or leg movement visualization or rest); that part of the experiment lasted for 6 s; finally, there was a pause of unfixed length (up to several seconds). Three EEG channels were recorded (C3, C4, Cz, according to the international 10–20 system) along with ECG, R–R intervals were calculated as a difference between QRS complexes, which show heart rate, filtered at 5–10 Hz frequencies. The obtained data allowed for a few interesting conclusions. First, active visualization of limb movements induced heart rate change. Second, ECG classification accuracy was very high in almost all subjects: in many subjects the use of ECG modality was more effective than EEG. Third, the hybrid approach increased classification accuracy in almost all subjects, especially in those, whose results in a single modality mode were low.
Thus, a hybrid approach to the implementation of systems for external device control is very promising. Considering how fast these technologies are developing, we believe that such high-accuracy neurodevices will appear in the market in the nearest future. The laboratory of Neurobiology and Medical Physics of the Institute of Chemistry and Biology of Immanuel Kant Baltic Federal University is working on a multifunctional neurodevice capable of detecting different electrophysiological signals simultaneously (EEG, EMG, EOG supported by the use of photoplethysmogram, SpO2 and temperature modules), ensuring a biological feedback and transmitting the processed data to exoskeletons and robotic devices in real time. This article presents the results of the first tests of the prototype model of such a neurodevice and assesses the possibility of fused EEG and EMG signal recording based on it.
METHODS
We have implemented a prototype model of electrophysiological and biometrical recorder capable of converting biosignals into commands for an electromechanical device; we have also tested our model in a two-stage experiment. At the first stage, the neurodevice was used to study the motor activity of the subjects by recording electrophysiological signals. For the unbiased assessment of the device performance, the resulting data were compared to the data obtained with analytical devices that had proved to be reliable and are now successfully applied in medical practice. At that stage, 2 healthy men participated in the experiment, (22 and 23 years of age, height of 175 and 177 cm, respectively, weight of 70 and 75 kg). At the second stage, a possibility of fused EEG and EMG recording using the neurodevice was assessed. The experiment enrolled 10 healthy right-handed men aged 22–29 years (mean age was 25 years).
Brain electrical activity was measured by encephalogram via scalp leads; bioelectrical potentials in skeletal muscles were measured by electromyography; bioelectrical potentials related to eye ball movements were measured by oculography; body temperature was measured by thermometry; pulse rate was measured using photoplethysmography. The obtained data were recorded digitally and graphically.
For EEG recording, silver cup electrodes (Ag/AgCl) were used; EEG caps were used to place the electrodes on subjects’ heads. For EMG and EOG recording, silver plate electrodes were used. In the experiments aimed at the assessment of physiological signal parameters, the most common artifacts were detected, such as artifacts resulting from bad electrode attachment or electrical noise caused by subject’s movements, artifacts caused by upper body muscle tension and forehead wrinkling, muscle potentials, skin potentials, eye blinking, pulse waves.
To study motor activity, the subjects were asked to do physical exercises, including bending and turning the head to the right and left, tilting it down and back, with the prototype model of the neurodevice attached to it. Before the experiment, we had written a program for real-time visual representation of Euler angles rotation.
Results of EEG, EMG and EOG signal recording and pulse rate data were compared to those obtained with KARDi3 device (Medical Computer Systems, Russia), intended for recording and analyzing ECG, EOG, EEG and some other parameters. Measurements were first done with KARDi3, then with the neurodevice of interest. The subject remained in the same position throughout the experiment. The electrodes attached to the body were not moved when switching from KARDi3 to the neurodevice prototype model. To reduce the amount of artifacts, electrode cables were bundled and twisted.
During EEG recording, we focused on alpha-rhythm, which is normally the most stable electrophysiological signal. To record the alpha rhythm, a bipolar lead system was used. Electrodes were attached to the back of the subject’s neck, reference electrodes were attached to ear lobes. To achieve the maximal relaxation of neck and head muscles and to reduce myographic artifacts, the subject was seated in the reclined position. In total, 100 trials were conducted. For both the neurodevice prototype model and KARDi3, the same recording mode was used, with a 30 Hz low-pass filter, a 0.5 Hz high- pass filter, a 50 Hz band-reject filter, speed of 30 mm/s (X-axis), sensitivity of 50 mcV/mm (Y-axis).
EOG signals were recorded during the eye movement task. The total number of trials was 100. The subjects were asked to do the following exercises: look at the yellow dot in the center of the board – then up (red circle) – center (yellow dot) – down (blue circle) – center (yellow dot) – left (red cross) – center (yellow dot) – right (blue cross) – center (yellow dot). The subject was seated in front of the board with graphic symbols. To register the signal, a bipolar montage scheme was used. Electrodes were attached to the temples, close to the right eye and on the forehead. For both the neurodevice prototype model and KARDi3, the same recording mode was used, with a 40 Hz low-pass filter, a 1 Hz high-pass filter, a 50 Hz band-reject filter, speed of 15 mm/s (X-axis), sensitivity of 50 mcV/mm (Y-axis).
To record EMG signals during thigh muscles contraction, the subject was asked to move the right leg forward for a step- like movement. There were 100 trials in total. The left leg did not move, the subject did not lean on the right leg on which electrodes were placed. The subject was standing, using his left leg and right arm as points of support; his right leg with electrodes on it was relaxed. Electrodes were placed 5 cm apart from each other. To record the signal, a referential montage scheme was used. Electrodes were placed over the femoral muscle using adhesive rings. For both the neurodevice prototype model and KARDi3, the same recording mode was used, with a 100 Hz low-pass filter, a 1 Hz high-pass filter, a 50 Hz band-reject filter, speed of 120 mm/s (X-axis), sensitivity of 10 mcV/mm (Y-axis).
To record EEG, EMG and EOG signals using our neurodevice prototype, neurodevice, we have developed original software. To record EEG, EMG and EOG by KARDi3, Neurocortex software by Neurobotics, Russia was used.
To measure the pulse rate the subject was seated. For ECG recording (100 trials in total), KARDi3 electrodes were attached to the wrists by adhesive rings. A referential montage scheme was used. Then, the subject put his finger on the photoplethysmogram module of the neurodevice, the output being the pulse-related signal. Pulse measurement was supported by pulse oximetry (SpO2). The following recording mode was chosen for both the neurodevice and KARDi3: a 0.1 Hz low-pass filter, a 50 Hz high-pass filter, a 50 Hz band-reject filter, speed of 60 mm/s (X-axis), sensitivity of 20 mcV/mm (Y-axis). To process data obtained with KARDi3, Neocortex software was used; to process data obtained with the neurodevice, Heart Rate Monitor Demo software was used (Silicon Labs, USA).
To compare temperature measurement accuracy, Fluke 17b multimeter with a plug-in thermistor was used. The temperature sensor was attached to the subject’s forehead by the adhesive ring. The total number of trials was 136. Temperature data were transmitted to the PC via Blootooth protocol every second.
At the second stage of the experiment, a possibility of EEG and EMG fused recording by the neurodevice of interest was studied to ensure its good performance in a complex with robotic devices, such as an exoskeleton.
The subjects were instructed to imagine their left leg movements and then to flex and extend the thigh (10 sessions for every participant). Classification accuracy was first assessed for EMG signals only and then for a hybrid EMG-EEG system.
Physiological parameters were continuously monitored during motor tasks and idle periods (5 s long). Fisher linear discriminant analysis was used for classification.
RESULTS
We studied motor activity involved in performing such tasks as turning and bending the head to the right or left, tilting it down and back. The results demonstrate high accuracy and precision of the data obtained with the motor sensor of the studied neurodevice. The diagram in fig. 1 shows Euler angles rotation (X-axis represents time, Y-axis represents angle): 1) pitch is rotation around the transverse axis (green line); 2) roll is rotation around the longitudinal axis (blue line); 3) yaw is rotation around the vertical axis (red line).
EEG data obtained with the studied neurodevice showed the same artifacts as EEG data obtained with KARDi3, and their amounts were comparable. It indicates that the studied neurodevice could compete with similar tools for EEG recording. Muscular activity artifacts were associated with small neck and head movements resulting from subject’s fatigue. Quite a few encephalograms showed traces of cardiogram artifacts, which is possibly related to the individual specifics of the subject’s cardiovascular system and the placement of electrodes over subcutaneous blood arteries.
EOG data obtained with the neurodevice were comparable to the data obtained with KARDi3 in their informative value; in case of our neurodevice, the amount of artifacts was lower. The most common artifact was eye blinking, which appeared on EOG as a sharp amplitude increase; artifacts of mimic muscles that accompanied the subject’s growing fatigue were also present.
During EMG signal quality assessment, it was found that the data obtained with our neurodevuce were as informative as the data obtained with KARDi3. No artifacts were detected.
The results of temperature measurements obtained with the neurodevice were comparable to the data from Fluke 17b reference device. In average, temperature variance was 0.3%.
Pulse signal was obtained from the electrocardiogram recorded with KARDi3 and the neurodevice photoplethysmogram module. ECG R–R interval data from KARDi3 showed the same pulse values as data from the photoplethysmogram module. The mean HR in the first subject was 78 and 77 beats per minute (measured with KARDi3 and the neurodevice, respectively). The mean HR in the second subject was 72 and 71 beats per minute (measured with KARDi3 and the neurodevice, respectively). No artifacts that could affect the result were observed (fig. 2).
It is worth mentioning that the module for the assessment of cardiovascular system performance estimates blood oxygen saturation, thus providing some valuable data that can be used for exoskeleton control.
Our study of EEG and MG signal hybridization yielded results that support the idea electrophysiological signal fusion approach. The experiment showed that mean classification accuracy of EMG signals was 74.3 %. EMG-EEG hybridization led to the increased classification accuracy by an average of 12.5 % with a mean of 86.8 % (75–97 %) in all subjects. The results are presented in the table below.
DISCUSSION
It is obvious that development of a high-accuracy multifunctional neurodevice that allows for continuous recording of physiological signals and transmits data to the external device (exoskeleton) can yield very inspiring results. We have carried out a truly multidisciplinary study, at the first stage of which a prototype model of such a neurodevice was created and tested. It was demonstrated that the signals obtained with our device were identical to those obtained with reliable analytical tools.
During some motor activity measurement procedures, gyroscope drift was observed associated with a changing magnetic field generated by the accumulator battery. It was the result of the relatively weak attachment of the accumulator to the model; rigid fixation of the accumulator helped to solve the issue. With weak attachment of the model to the subject’s body, a simultaneous change of two angles was observed occasionally when the subject was performing a task. It can be explained by the “multilayered” scheme used in the experiment: gyroscope components were placed on the motor module, and the motor module was placed on the subject’s head. Steady movements of the subject’s head also made their contribution. We will consider it in the fabrication of the experimental sample and will use software and hardware automatic calibration of the device position with respect to the subject’s position The main motor activity parameters measured by the experimental sample that we plan to fabricate will be linear acceleration of the accelerometer, angular acceleration of the gyroscope and a magnetic field vector of a magnetometer. EMG and EOG muscular artifacts will be removed using additional band-reject filters or special software. Cardiogram artifacts can be removed by changing electrode attachment mode from stationary to dynamic, with a possibility to shift electrodes by no less than 10 mm. Thus, the electrode can be moved if it has been placed over an artery. Besides, improving accessories for electrode attachment will also reduce the amount of artifacts.
The results of EEG-EMG fusion experiment showed the considerable advantage of hybrid BCIs over single-modality BCIs and confirmed the feasibility of simultaneous recording of various physiological signals [19, 20, 21, 23, 29]. Due to the increased classification accuracy and flexibility, a hybrid system is more reliable and exhibits higher performance. The obtained results lead us to conclude that fused EEG-EMG recording improves the interpretation of intended and actual physical activity. EEG signals unrelated to muscular activity are an additional identification tool that can be used in robotics. We speculate that improvements to the system and simultaneous use of various physiological signals will result in almost 100 % classification accuracy.
By now, very few works describing such experiments have been published. All of them are non-representative with respect to the number of participants. To increase signal classification accuracy and safety of robotic devices, further research is necessary. Still, certain difficulties remain. First, electrode shifting is a problem, because the correct placement of electrodes is what defines intensity, quality and reproducibility of signals. With respect to that, non-contact technologies can be a solution. Second, complex movements involving several muscles (hand, forearm, shoulder girdle and trunk muscles) are generated by a large number of motor cortex areas, and the size of each area is unique for every person, which impedes reconstruction of complex movements. To solve this problem, new technologies capable of isolating target movements from unrelated ones are necessary. Some solutions have been proposed so far, including invasive interfaces based on electrocorticography [30, 31].
CONCLUSIONS
The tests of the neurodevice prototype capable of simultaneous detection of different electrophysiological signals confirmed the feasibility of hybrid approach to the development of systems for external device control. Fusion of several modalities or switching from one to another to select the one that best interprets human intention increases signal classification accuracy and can possibly improve robotic device performance. Further research is necessary with a larger number of participants involved, including those with different pathologies.