Electronic apppliances are increasingly a part of our everyday lives. In particular, mobile devices, with their reduced dimensions with power rivaling desktop computers, have substantially augmented our communication abilities offering instant availability, anywhere, to everyone. These devices have become essential for human communication but also include a more comprehensive tool set to support productivity and leisure applications.
However, the many applications commonly available are not adapted to people with special needs. Rather, most popular devices are targeted at teenagers or young adults with excellent eyesight and coordination. What is worse, most of the commonly used assistive control interfaces are not available in a mobile environment where user's position, accommodation and capacities can vary even widely.
To try and address people with special needs new approaches and techniques are sorely needed. This paper presents a control interface to allow tetraplegic users to interact with electronic devices. Our method uses myographic information (Electromyography or EMG) collected from residually controlled body areas. User evaluations validate electromyography as a daily wearable interface. In particular our results show that EMG can be used even in mobility contexts.
Keywords: Electromyography, Interaction, Evaluation, Usability, Accessibility, Mobile, Recall, Wearable
Subjects: Electromyography, Human Computer Interaction, Usability
We find ourselves surrounded by technology, whether in public spaces, our homes or even within our body space. Indeed, it is difficult to imagine how we could go about our daily business without these devices and their functions, that we take for granted today but were not available only a few years ago. Thus, technology touches our lives in ways we no longer think about and creates new opportunities for many, offering new forms of social interaction, instant access to information, constant availability and higher control of the environment that surrounds us.
Moreover, although a few years ago computers were meant to be used only in static environments, the extraordinary development of RF technology dictated the success of mobile computing devices. Indeed, advances in communication technology and component miniaturization were the main reasons behind the mobile technology success and its enormous penetration on society. These small, portable and stylish devices extend our capacities through different scopes in our daily lives. Considering available functionalities, they are increasingly becoming similar to desktop computers. Therefore, we are now able to edit a document on a mobile device and send it to a colleague in another country. Indeed, and this is the most important and basic function, using them one can be always available and communicate with anyone else in the world just by dialing a number.
While many people are able to operate both static and mobile devices with relative ease, there are still large groups that experience difficulties when interacting with these small devices to varying degrees. While such inability can be due to different impairments or conditions, in this paper we focus our attention on severely motor disabled individuals and tetraplegics in particular. A tetraplegic is an individual with motor limitations on both upper and lower limbs usually caused by traumatic spinal cord injuries. Thus, most barriers to interaction are physical while cognitive capabilities remain intact.
Limitations imposed by tetraplegia deprive the injured individuals from operating electronic devices such as computers or mobile devices. Besides the drastic quality of life reduction directly imposed by these impairments, individuals also face a communication bottleneck as they are often incapable of operating IT devices to communicate with others, such as computers, cell phones or personal digital assistants (PDAs).
It has become a world wide concern to empower these disabled users with communication and control skills in order to improve their quality of life. Hence, if we are able to provide tetraplegics with control over basic input functions, such as pointing, selecting and entering text, tetraplegics can then operate many devices, easing communication and overall autonomy, allowing these people to regain control over many tasks in their daily lives.
Although many assistive technologies attempt to overcome motor impairments and offer tetraplegic people control and communication capabilities, they are still insufficient and fragmentary. Even with the latest assistive technologies, tetraplegic users experience many difficulties to control devices in everyday settings. This is because body position, shifting ambient conditions or miscalibration issues still require constant caretaker assistance to keep many devices in a stable operating condition. As an example, cursor emulation approaches normally require the user to be placed in front of the computer (i.e., gaze-tracking approaches). However it is often difficult for tetraplegic users to assume the required body posture and most important, often they may not be able to position themselves properly without third-party assistance. Therefore, although the interaction techniques are designed to be similar, the pre-requisites are not satisfied equally by full-capable individuals and tetraplegic users.
The considerations and requirements above lead us to design and develop an electromyographic mobile device control interface for tetraplegics, whereby users can control mobile and desktop devices through muscular contractions.
Electromyography (EMG) is defined as the study of the muscular function through the analysis of the electric signals generated during muscular contractions. The electrical potential obtained in the muscle fibers can be registered at the surface of the human body through surface electrodes due to biological tissues conducting properties [ Luc02, CMHVC92 ].
A large set of candidate muscles is available so we can interact flexibly with the computer. Indeed, being able to detect and to evaluate muscle activity gives us the possibility to associate such activity with predetermined interface commands, using the myographic signal as input. By using this technique (EMG) we can focus on a wide target group - any voluntarily contracted/moved muscle can be monitored - and be exhaustive in exploring the capacities the user has to offer while keeping the system both usable and simple. Once again, Electromyography provides the setup liberty and adaptation to all the scenarios providing the necessary mobile device interaction tool.
In this article, we present an electromyographic control interface and evaluate it in several scenarios. First, we present exploratory studies, aimed at validating electromyography as a general control interface, assessing its response accuracy and speed as well as its robustness. In a second phase, we evaluate the recognition of input commands with both full-capable and disabled users, determining to what extent our prototype is effective and easy to use.
The recurrent and increasing medical applications of electromyography led to considerable scientific investments to improve both myographic signal acquisition and analysis. These advances have made it possible to use portable electromyographic devices that communicate via wireless signals with a processing system. Indeed, portability makes it possible for any individual to carry and use an EMG device with great social acceptance [ CPIA04 ]. Furthermore, EMG device portability and reduced size easily lead to their use in Human-Computer Interaction where research work is carried out in Accessibility, Robotics, Affective Computing, among many other areas.
[ TCLK97 ] presented a myographic-controlled human-computer interface for quadriplegic users with injuries at the level of C4 vertebrae or below. In this system, five electrodes are bilaterally placed on and between the upper trapezius and sternocleidomastoid muscles: for each pair of electrodes, one is located over the sternocleidomastoid and the other over the upper trapezius; the ground electrode is located near the right earlobe. Through this system the user is able to control the a cursor pointer (four directions and double-click) although the feature space and several parameters need to be adjusted before achieving good classification ratios.
[ PKCP99 ] suggested a single-switch EMG-based communication device for disabled users with severe motor and speech impairments. Users operate this system by chewing with the masseter muscle achieving communication using morse-code through dots and dashes, according to the duration of each contraction (chew).
Myographic activity is also used as an input mechanism within other accessibility tasks. As an example [ Col01 ], studied EMG to improve and induce movements in elderly people. By including EMG biofeedback in the system patients retain muscle contraction capabilities to participate in practicing motor control activities, regardless of their ability to generate joint movements. The system detects muscle contractions and provides information to the users in a form of feedback that induces the patients to contract body muscles in a specific way. The author chose to use entertainment applications to provide feedback to get patients going.
Several researchers have leaned on gesture recognition using electromyography. The majority of the projects target either arm-operated joysticks or mice. An example of this is the Biofeedback pointer [ Ros98 ] which enables users to control the mouse pointer by wrist motion. The system receives user commands via four electrodes placed in the forearm.
In the same vein, [ WJ03 ] presented Neuroelectric Joysticks and Keyboards, recognizing up to nine wrist and hand motions (keypad) with a forearm band (Figure 1). Besides developing an EMG-joystick controlled flight simulator, the authors also presented a system that detects typing keypad numbers on the knee.
Figure 1. EMG Arm Joystick [ WJ03 ]
[ CPIA04,CIA05 ] presented electromyography as a subtle and intimate interface for mobile interaction. They argue that a technique requiring a simple pair of electrodes, that can be worn and operated unbeknownst to others, is perfect for interacting while on the move. Indeed, users are able to respond to system questions without disrupting the environment.
Despite their wide scope, all projects described above use EMG as an input interface. In this article we present a system supported by user studies to go further and use electromyography in daily wearable interfaces. We developed a prototype to control a wide variety of computer applications enabling us to conduct exploratory studies. Moreover, we were able to evaluate the system wearability with target users within real life scenarios. Additionally, we introduce applications to ease text-entry tasks both in static and mobile settings. The next section describes our approach. Then we present user evaluations, results and discussion. Last, we present our conclusions from user evaluations and outline future work.
Understanding muscle activity and interpreting accurately the myographic signals it generates make us able to design systems controlled by users using conscious muscle contractions. However, to build such systems, several phases must be considered, from signal acquisition to its processing and muscle onset detection. One difficulty is that electromyographic signals present several interferences that must be removed. To achieve a good accuracy, we conducted system design while paying careful attention to these three phases: signal acquisition, signal processing and onset detection.
Our electromyography device collects electric samples at a rate of 1000Hz in five independent channels. To collect the signals we use surface differential electrodes, which are disks with a 1.5 cm radius. The device includes a 110dB CMRR operational amplifier and a band pass filter between 25 and 500 Hz with gain 1000. It is a relatively small device (14cm * 8cm * 4cm) that can be carried in either a belt or pocket (Figure 2). The portable device communicates with the host at a range up to 100 meters via a Bluetooth interface.
The first stage in any myographic interface is signal acquisition. Besides the material to be used, we must consider the mounting sites as well as the hardware pre-processing steps. Both these aspects greatly influence the fidelity of the collected signal. The electromyographic device includes processing components that improve the signal quality, removing interferences and noise (i.e., 25-500 Hz band-pass filter).
In order to get useful information concerning the muscular activity it is necessary to carefully analyze some aspects, from technical details of the electrode placement in the surface of the human body to the points where this placement must be done. Several aspects influence signal quality: skin preparation, electrodes placement position, electrodes fixation, electrodes distance and outside interferences [ Luc02 ].
We have discarded all the skin preparation techniques as we do not think they are appropriate to an user interface. Besides, after several tests we observed good signal quality with small interference. However, as an example, to reinforce the surface electrodes adherence we created an elastic band for the neck (Figure 3) and two elastic bands for the forearm. We have not focused our attention in this special purpose devices as one of the contributions of our system is the setup freedom. The users can select the locations to be monitored. However, we also believe that the approach can be complemented with a set of special components that ease interaction and maximize success.
We used a 2cm distance between electrodes which guarantees good acquisition results, collecting the signal of a significant portion of the muscle and restricting, simultaneously, undesired signals to insignificant values [ Luc02 ].
Basically, these electrodes allow us to monitor any voluntarily contracted muscle. However, the signal frequencies and amplitudes vary from muscle to muscle. Obviously, we only use surface electrodes as we are studying a wearable daily interface and want save users as much pain or discomfort as possible.
The electrodes should not be placed on the muscle motor point at which the signal low frequency components are substantially dampened. Besides electrode placement it is also important to check the orientation of the electrodes in relation to muscular fibers. The imaginary line that joins the two surfaces (two surface electrodes as we are using a differential setup) should be parallel to the muscular fiber orientation. Since we do not expect that caretakers are electromyographic experts nor that they pay special attention when performing the system setup, one of our goals is to provide a solution that is able to cope with slight shifts in either position or orientation of the electrodes.
Considering the target population and aiming at a wider number of possible users, there are some face muscles that can be selected as as probable monitored locations. Moreover, some body gestures (i.e., blinking an eye) can be monitored in a particular muscle and isolated (from other muscle action potentials). If the user detains higher control of his body, the possible locations also increase.
In order to extract useful information from the signal we need to process it. This processing phase transforms the signal into a more meaningful indicator and it includes components to gather a digital signal representation, amplify it to enable analysis, improve the signal quality and represent it in a transformed yet more comprehensible form.
The outcome of this module is a signal representation that is aligned with the system's purpose. Thus, the goal of the signal processing module is to transform the signal so we can identify muscle contractions, independently from the monitored muscle and with no previous information.
Some of the projects mentioned in the area of electromyographic human-computer interaction have strong pattern classification algorithms that offer them great reliability. However, the drawback is that, besides the need for long training sessions for each user, the systems require that the mounting sites and ”body gestures” remain constant. As we require high versatility and adaptability, a feature-based algorithm is not usable. In our work we try to make a simpler approach adaptable to every user with no training required. Thus, we have developed a signal processing based approach.
Our signal processing module is composed by a hardware pre-processing and a smoothing phase (Figure 4).
The electromyographic device, wires and electrodes gather the first component set in order to acquire and process the myographic activity. While some of these components are required in a system that aims at a digital signal analysis, other components relate to the signal quality improvement.
Analog signals are voltage signals that are analogous to the physical signal they represent. The amplitude of these signals typically vary continuously throughout their range. The analog-to-digital conversion process generates a sequence of numbers, each number representing the amplitude of the analog signal at a specific point in time. [ Luc02, reviewed in].
One important technical item is the selection of the proper Sampling Frequency. To obtain a proper conversion of the complete frequency spectrum of a signal, the sampling rate must be at least twice as high as the maximum expected frequency of the signal. This relationship is described as the Nyquist sampling theorem [ Nyq02 ] and it shows that sampling a signal at a low frequency results in aliasing effects. Considering electromyographic signals, almost all of the signal power is located between 25 and 250 Hz and scientific recommendations require a minimum band maximum of 500 Hz [ HFM99, FH99 ], which translates in a sampling frequency of at least 1000 Hz.
The EMG amplifiers used are differential amplifiers that subtract the value between two electrodes. This step of the process is essential to remove exterior interferences from the signal and to limit the motor units to be considered. Also, as EMG amplitudes are reduced, the amplifier augments the signal voltage so it can be analyzed afterwards.
Surface EMG, as a sum of several frequency waves, has its useful information located in a determined frequency band. Most researchers agree that the relevant information is between 25 and 250 Hz. The band-pass filter used in our system rejects below 25 Hz and above 500 Hz. This band-pass filter cleans several interference patterns that can reduce signal fidelity. For example, motion artifacts have most of their energy in the frequency range from 0 to 20 Hz. On the other hand, ambient noise is around 50/60Hz, a rich EMG frequency band, that can hardly be removed without reducing the signal significance.
After the acquisition phase, we have a digitalized and amplified signal with a restricted frequency range. However this signal, called raw signal, although it has already been pre-processed, can hardly be interpreted by the computer if no further processing stages are executed. To allow an accurate signal interpretation we have to clean the signal and represent the muscle energy in a way that we can easily identify activation and deactivation times.
The signal received from the electromyography device has a gamma of values between 0 and 4096, having this to be adjusted, since, really, the signal oscillates between negative and positive values. The centralization is a very basic operation and consists of deducting the base value (2048) from the signal.
While most of the amplifiers work with an offset correction, it is possible that the signal baseline is shifted away from the true zero line. If this offset occurs and it is not corrected, all amplitude based calculations are invalid. This condition can be identified by averaging the raw EMG signal (the mean value should be zero if no offset is present).
Thus, to remove the baseline offset, we add the samples to the set of received values already acquired and, with the average calculated on these, we calculate and remove this DC (direct current) offset: y(t) = f(t) - m(t)(1)
The interference pattern of EMG is of random nature. Indeed, the set of motor units changes within the diameter of available motor units and the way motor unit action potentials superpose is arbitrary. Thus, even if all the procedures are repeated exactly the same way, the raw EMG signal cannot be reproduced a second time by its precise shape. To overcome this problem, the non-reproducible part of the signal is minimized by applying digital smoothing algorithms that outline signal evolution during time. The steep pikes are discarded and the signal receives a linear envelope.
One common approach to envelope the signal is to apply a Moving Average algorithm. In this technique, based on a pre-defined time window, the window samples are averaged. The samples in the window are rectified before performing the average. Commonly, Full-Wave Rectification (the negative samples are reflected by the baseline) is applied as all the signal energy remains available, in contrast to Half-Wave Rectification, where the negative values are discarded [ Luc02 ]. The signal averaging can be performed with a linear moving average but it can also be based on a Hanning, Hamming or Bartlett window. They are different in the way they weight the several samples in the window to be smoothed.
Another approach to signal smoothing is Root Mean Square. The root-mean-square (RMS) of a variate X, sometimes called the quadratic mean, is the square root of the mean squared value of X (Equation 2). The RMS reflects the mean power of the signal and is the preferred recommendation for smoothing [ Luc02 ]. The RMS EMG is also applied to a moving window and time duration values between 50-100 ms are likely to work well as the real time impression is kept and the signal is smoother as desired.
In our prototype we applied the RMS algorithm value as it is a measure of the power of the signal, thus it has a clear physical meaning. Also, the application of the RMS translates in a signal with higher amplitudes and therefore, with easier recognizable onsets and offsets.
Figure 5 presents a tetraplegic user lying down operating the system using two input commands. It also presents one of the monitored samples (both raw and processed representations).
Several clinical applications, like gait analysis and coordination studies, require the accurate detection of when and for how long the muscles are active. Therefore, several methods have been proposed for detecting the on and off timing of the muscle.
In clinical applications, the most common method for resolving motor-related events from EMG signal is still visual inspection by trained observers. However, as this method is unsuitable for the majority of the applications, other automatic solutions must be analyzed. Single-threshold methods, which compare the EMG signal with a fixed threshold, are the most common computer-based approaches of detecting the onset of muscle contraction. The accuracy of this detection depends on an appropriate threshold definition. The most popular approach to threshold definition calculates the baseline standard deviation (SD) (before muscle activity) and multiplies it with a pre-determined multiplication factor. Although a popular method, the standard deviation threshold definition can be difficult to set up for valid and repeatable results. The EMG signal varies between trials and subjects and the SD is likely to be largely different. This kind of method rely on criteria that are too heuristic.
Our system requires that muscle contractions are detected. Moreover, we are required to detect contractions from different muscles, at different times, with no manual identification. Also, as the user is able to select the monitored locations, the system has to deal with contractions that although aimed at firing a certain action, can affect more than one monitored muscle.
On the other hand, we are not required to detect the exact firing time as a delay of some milliseconds is tolerable. Also, most of the research on onset detection occurs in the clinical area where, as an example, the goal may be to identify problems in gait and coordination. The solutions surveyed analyze a signal independently from the other monitored locations.
We have developed two onset detection methods adapted to our requirements and goals, that take advantage of the characteristics and context of use. The first requires an initial Calibration Step while the second is similar to the standard deviation threshold-based solutions described above.
We determine thresholds for each signal collected by performing a preliminary calibration phase. This phase consists of asking users to perform each monitored contraction at a time. With this procedure we aim to achieve two goals. One is to define a user and situational threshold for each channel. The other is to use the information from the other channels to improve overall intention recognition. Moreover, this calibration phase should also allow identifying erroneous situations such as cable disconnections, wire related interferences, etc.
In a multi-electrode setup, one likely problem is the influence of a given contraction to more than one monitored location (even though that may not be the user intention). Therefore, besides defining a threshold that suits a single channel configuration, we also intend to deal with such unexpected interferences. To achieve this goal, we take advantage of the other monitored signals and time windows. Thus, we ask users to perform a contraction (a near-maximum voluntary contraction) within a determined time window. However, to define a threshold to the recalled location, in spite of analyzing its time window, we only analyze the remaining ones. With this approach, we define a threshold that deals with the worst case scenarios, when other muscles are being contracted. The threshold is defined as the maximum sum of the higher RMS value and a scale of the maximum standard deviation (calculated with inner 50-100ms samples), of those remaining time windows. Figure 6 presents the calibration step for the temporalis and frontalis setup. It is clear that the thresholds for right and left eye blink are easy to determine as during the remaining contractions the signal in that specific channel is almost stable. On the other hand, the frontalis presents contractions when the blinks are performed. Therefore, the threshold is higher and is near the High threshold value.
Figure 6. Calibration Step: The first chart presents the three signals. The other three charts represent the temporalis right, temporalis left and frontalis signals with the calculated low thresholds.
During the calibration process, in a time window referent to a certain contraction, we also collect the maximum voluntary contraction (MVC) as the RMS peak value of that window. This value summed with a scale of the maximum standard deviation value (calculated using 50-100ms inner time windows) is our high threshold.
This value enables us to detect abnormal situations and ignore erroneous commands. We only consider contractions that are quantified between the two thresholds. Moreover, this high threshold allows us to identify unfeasible setups. Indeed, if the low threshold is higher, equal or even with a value too close (a scale of the maximum standard deviation related to that contraction) to the high threshold, the monitored locations are discarded and the user is warned. This detects setups that are not feasible but also situations where the user does not use a certain location (and does not perform its particular calibration). With these conditions the user is able to define acceptable intensity ranges for each muscle contraction independently. It enables personalization and adaptability but also restricts the acceptable signals and therefore reduces the likelihood of unexpected behaviors.
One particular situation occurs when only one signal is acquired: in this scenario the low threshold is defined using a window where no other contractions are performed. In this situation, and regarding the lower threshold, the two methods presented here are very similar. However, concerning high threshold, both single-location and multiple setup are equivalent.
Although the signal is smoothed, single spontaneous spikes can easily exceed the threshold range and possibly be marked as a muscle contraction. To neglect this undesired commands, we define a minimum time (minimum sub-period duration) that the processed signal has to constantly stay between thresholds to be accepted as a muscle onset (i.e., 50 ms). Figure 7 present a set of recognized and unrecognized states. We can observe one false negative in the last chart. Also, we can observe two spikes that were correctly discarded. The correct choices are shaded in green while the incorrect (one) is shaded in red.
Besides the aforementioned procedures, we have also included optional verification mechanisms that are able to improve recognition. Particularly, and although we believe our system to be accurate, it is still possible that a user performing a contraction still creates an acceptable contraction following the desired contraction. This can happen in the same location or even in one of the other monitored locations. As an example, we can observe neck lateral movement (leaning the head towards the shoulder) with both sternocleidomastoid sides being monitored. While leaning the head to one of the sides, is likely to be detected as a contraction in that side, finishing the motion and returning to normal position can also be detected as a contraction in the other monitored location. Although, this is not an expected behavior by the user, the contraction is probably being performed and it will not be distinguished from the normal behavior.
Thus, if it is required and can be applied, we also define a mute window after a certain contraction (i.e., 100 ms). During that time window, after a detected action, the system will discard any possible actions.
One thing that is also possible is that two muscles are contracted in the same time window and the system detects both onsets. We believe that this can be a possible situation and useful for certain applications. Therefore, we have not included in the main recognition system, a component to analyze these situations. However, we have provided an optional component that can be used if the application requires. This component can eliminate this detections or decide between them. This decision is made according to the percentage of the contraction performed. This percentage is calculated in the range between the low and high thresholds. The signal with higher contraction percentage is selected. Thus, we select the contraction that is most likely to be the desired, as it reflects an higher intention considering the user initial calibration.
The calibration mode allows us to detect and adapt the system to the user with a determined setup. If a shift is performed or a serious error situation is detected the user is able to request to calibrate the system again.
On-the-Fly mode is very similar to the SD threshold algorithms described above. With this option, the user is able to use the system, with no previous calibration steps. The system detects muscle onset with an approach where threshold is estimated as a multiple h of standard deviations [ SFDW01 ]. Another ”problem” with this mode is that modern EMG amplifiers are so noise free that the multiplication factor has to be increased to 8 times or higher to give reliable results. This situation can also be improved if the user is able to define a sensitivity feature that influences the multiplication factor.
To improve recognition, this mode also includes Minimum Onset Window and Offset Window procedures. This mode is not as accurate as the calibration mode but it can be used if the monitored locations are not likely to interfere with each other.
To assess electromyography as a suitable control interface in a non-restrictive context, some preliminary questions need to be answered. While it is important that our prototype allows application control, it is also important that it allows it in a daily context, subject to the interferences that a wearable interface may suffer. Moreover, some aspects, like the response accuracy and speed need to be studied, as future work depends on the assurance of those characteristics.
These exploratory evaluations were performed with a small group of users as they intend to identify possible insufficiencies or problems with a myographic control interface and our particular prototype system. Also, the tasks performed are not the main goal of our project, yet they are ideal to evaluate the following features: accuracy, response speed, robustness, wearability.
To evaluate the system response speed and accuracy we have performed an user evaluation on desktop control. The mouse cursor requires a high response rate to be properly controlled therefore this task seems appropriate to evaluate the system accuracy and speed. However, cursor control or any kind of continuous control is not our goal. In this particular evaluation, the scenario itself evaluates the characteristics we aim to observe so we thought of it as a good candidate challenge. In this evaluation context we performed two evaluations: Target acquisition (point and click) and continuous control mode in a text-entry application.
This evaluation was performed with three full-capable users with 19, 24 and 51 years. All of them were used to deal with computers. Although none of the users is part of our target group, this evaluation is focused on the system response time and therefore we believe that the results are independent from the user physical condition. All the users performed both evaluations.
The first evaluation features target acquisition in a desktop environment and besides evaluating the time to acquire and click a certain target also evaluates the difficulties the users may have to hold the pointer steady over a target. With this approach we can easily collect the time to acquire targets, erroneous clicks and even compare our system with other approaches. As our goal is not cursor control we have not compared the performance with other solutions (i.e., trackball, mouse, eye-tracking,..). However, our test application is similar to the one used by [ BSA99 ] and consists in a point and click timed exercise. Thus, we are able to compare our approach with one that is also EMG-based but also features EEG signals that work as a clutch (recognize user' s attention) to improve recognition.
The setup is created with enough electrodes to emulate mouse moving directions and left-click. The users were equipped with two pairs of electrodes in each forearm (four directions) and another pair near one eye to detect blinking (click emulation).
The recognition system was set up to On-the-fly mode (RMS window = 75ms, SD Scale = 8), requiring no previous calibration. The users were able to adapt themselves to the system and observe the required strength to move the cursor in the desired direction.
Clicking a Start Button, starting a timer;
Moving the cursor towards the Stop button, with any trajectory;
Clicking the Stop Button, the time is presented to the user and saved.
The Start Button dimensions are always 8,5 x 8,5mm but there are four Stop Button dimensions (8.5 x 8.5mm; 12.5 x 12.5mm; 17 x 17mm; 22 x 22mm). We made 80 evaluations, 20 of each for every Stop Button size. The Start Button changed between the four corners.
We developed a simple OpenGL application with a Start Button (presented in a corner position) and a Stop button presented in the middle of the screen. The application collects all the user interactions and logs the timestamp for all the application events. This enabled us to post-process the data and analyze the results.
Figure 8 shows the average values taken for each subject to complete the 80 trials. The subjects required an average of 7.5 seconds to achieve the experience goal.
Although the acquisition times are slow compared to absolute pointing solutions (i.e., eye-trackers) the system is presented as accurate as no clicking errors were detected and the users easily achieved the targets. The acquisition times are conditioned by the relative navigation mode used, with no acceleration features. Our approach seems to be quite efficient compared to others. We have duplicated an experiment already made by [ BSA99 ] and the results are quite better. The subjects in our trials required around 7.5 seconds to move the cursor from the corner button to the center button and performing a Left-Click as [ BSA99 ] had an average result of 16 seconds.
In general, projects using EMG concentrate on a point and click approach, which is inappropriate to the writing activity (very slow). We propose a synergy between applications where a pointer is continuously controlled by myographic activity, which appears to be a faster and efficient approach. Although tested in a desktop environment, this application scenario can be transported to mobile device text-entry. The downside is that it is also a continuous control approach and it is difficult to use EMG as a continuous control interface as prolonged contractions become attenuated (and unrecognizable) and fatigue issues arise.
The hardware used was similar to the previous evaluation scenario (Section 4.1.2).
The users were asked to write the sentence ”Dasher is a fine text entry interface and I enjoy it”. These evaluations were performed with the forearm setup (similar to the one described in Section 4.1.2) and neck setup with only two electrodes (one electrode in each side, monitoring the sternocleidomastoid muscle) as the application gives the one-dimensional control possibility.
We also tested the forearm position setup and asked the users to write the sentences using Windows On-Screen Keyboard to compare our synergy navigation application with point and click approaches. We used the same goal sentence. The users alternatively started using whether onscreen keyboard, whether Dasher (described below).
Dasher [ WBM00 ] is a text-entry interface based on a zooming technique shown in Figure 9. This application was developed considering situations or users associated with an incapability to write with the keyboard. The user basically navigates in a ”sea of letters” which appear accordingly to word prediction techniques. It allows two-dimensional and one-dimensional control.
To control the desktop independently from the control interface we have created an application that emulates operating system events. Therefore, using this application and any control interface we can set up a set of relations between input commands and application events. In this particular application we have related the four forearm detected motions and the eye blink with cursor events (four directions and single-click).
The experiment results are presented in Table 1. In order to understand the evaluation we need to define precisely the meaning of each metric:
Error: an error is detected when the user misses a letter and has to go back. Some of these errors may be users fault, i.e. skipping a letter or a space by distraction.
Time: time until the user ends his sentence correctly.
All the users succeeded and were capable to write the entire message. The errors detected were related to user' s distraction, i.e. skipping letters and having to go back in the writing. There were no errors in the forearm control + Dasher task. One of the users made an error in the neck control (missed a letter) but was capable to go back and complete the trial. We had one error in the On-Screen Keyboard (hitting between letters).
We presented a synergy between applications that outperforms the On-Screen Keyboard scenario: our approach averaged 124 seconds against 480 seconds in the keyboard. Both the Speed as the Dasher results are quite interesting and present electromyography as an auxiliary interface for impaired individuals. These tests included the writing through neck movements which were successful. Electromyographic interaction is an opportunity for tetraplegic individuals and we improve this opportunity with a faster and accurate approach. These results show that even though an electromyographic interface is slower than absolute positioning approaches, it can be improved if the interaction is designed accordingly. Also, and considering accuracy and speed, it was clear in this application scenario that the users were able to navigate through the ”sea of letters” with a real time response and few, and always controlled, unexpected movements.
Over a day, users are likely to shift positions in bed as well as performing voluntary or involuntary movements. Therefore, besides providing a suitable dialogs to make erroneous commands controls unlikely, the system shouldbe robust enough to filter the unwanted interferences. Moreover, it is also important that these movements do not damage the system and users are able to regain control afterwards.
[ CIA05 ] give relevance to EMG technology in mobile contexts mentioning it as a subtle interface which translates to great social acceptance. They rely on users of the system being able to interact privately without disrupting the environment around them. Similarly we are motivated to using EMG with mobile devices. To this end the authors evaluate EMG usability while walking and making contractions of different durations. However, they use only one input channel for simple subtle intimate response events. In our evaluation, we try to assess if the electromyographic setup is robust, whether it keeps its characteristics when used for lengthy periods and whether the users feel comfortable when using the setup while on the move.
This experience intended to evaluate the system in standing and walking conditions while responding to voice impulses. The users were already familiarized with the system due to first evaluation trials (Section 4.1.1). To create an application scenario we have used the same framework used in the Text-Entry Evaluation, emulating cursor events (Section 4.1.3).
Our method tries to validate EMG wearability and mobility, but with a more complex prototype than previous works, where there are several monitored input channels/muscles and several corresponding actions previously selected. The aim of this experiment was to evaluate whether the system responds as it is expected even in standing and walking conditions.
To evaluate the system's correct response we designed a walking circuit (similar to [ CIA05 ]) which the user has to follow as he responds to orders. Several variants were tested from the Walking with no contractions setup to the Walking with 5 contractions involved. The variations are:
Walking with no contractions.
Standing with stimulus response.
Walking with stimulus response.
The users were equipped with two pairs of electrodes in each forearm (four directions) and another pair in the right temporalis (click by blinking). Another setup was created with one pair of electrodes in each side (two directions) of the neck (Figure 10).
One of the users had one false positive in the Walking with Stimulus Response task. The other two had no false positives. The false positive was due to wire misplacement. No false negatives were detected in any of the users' experiments.
The designed system is robust to movement but if the monitored muscle is contracted (voluntarily or not) an action will be performed. However, we believe that these results point out that small movements and other common possible interferences (i.e., wheelchair guidance) are not likely to create erroneous commands.
The main goal of our project is to bridge the gap between disabled users and electronic devices. Moreover, we want to allow device control in several scenarios, with different accommodations, different users and their residual and momentary capacities. Also, we must consider the user' s environment and particularly, the prototype ease of use.
To analyze and evaluate the extensibility and versatility of the system we have performed an user evaluation. In this phase, we have focused our efforts in validating the electromyographic signal as a suitable control interface, evaluating contraction recall in several sites as well as possible setup collisions.
The purpose of this study was to evaluate the electromyographic signal processing module. It is independent from any application and it does not feature any feedback component. From these studies, we aimed to validate the electromyographic signal as a suitable multi-command issuer while validating our signal processing module as robust, versatile and extensible. Furthermore, we aimed to identify possible preferred electrodes placement positions, setup collisions and compare the system recall when used by impaired and able-bodied users.
We performed an evaluation featuring several electrodes placement, varying the placement and number of monitored sites. To this purpose we have identified a set of voluntarily contracted muscles in the face. Each task featured the recall of a set of determined muscle contractions (the command order was issued by the evaluation monitor). Each task featured the recall of each site 5 times and each command is issued every 5 seconds.
The selected acquisition sites (Figure 11) were:
Temporalis [Left and Right], Frontalis (3 sites, 15 commands)
Frontalis and mentalis (2 sites, 10 commands)
Masseter [Left and Right], frontalis (3 sites, 15 commands)
Although several other scenarios could have been tested, the performed evaluations gathered a meaningful set of placement sites. For each of the aforementioned electrodes placement, the user was asked to perform a maximum (but still comfortable) contraction for all the monitored muscles (Calibration step). Then the user was asked randomly to recall a muscle contraction, with a 5 seconds interval between commands. The order of the several possible placements was randomly selected for each user.
The electromyographic signal was collected with Ag/Acl surface electrodes connected to a BioPlux6 physiological system. The collected systems were transmitted via Bluetooth to a laptop computer (Toshiba Satellite A30), a Pentium 4 3.0 Ghz with a memory of 512 Mb.
The evaluation was performed with eight users, two of which were tetraplegic (Figure 12) and six were able-bodied. Of the eight participants, two (25%) are female and six (75%) male. Their ages range from 21 to 53 years old and their education level ranged from 12th Grade to Post-graduate degrees. Both tetraplegic users have C5/C6 injuries and are dependent on caretakers. Both users sport full face, neck, shoulder and partial arm control.
The results analyzed in this section were recorded in log files during the evaluation sessions. We collected the raw signal, processed signal and threshold values for each of the monitored channels. Afterwards, we performed an offline analysis to evaluate the system precision and recall. This evaluation was performed using the Calibration Mode (Root Mean Square Window - 75 ms; Minimum Onset Window - 50 ms; Offset Window - 150 ms; Standard Deviation multiplication factor - 2).
As it might be expected, the setup with the closest set of monitored positions (temporalis and frontalis) was the one that revealed a lower True Positive Classification (91.67%). The distinction between eye blinks (temporalis) is clear and this pair is presented as a good choice (Table 2). Some of the users presented collisions between the temporalis and the frontalis as it was visible that frowning lead to eye blinks and vice-versa. However, we believe that this recognition could be improved if the user repeated the calibration as it was clear in the performed post-analysis that the detected collisions were due to unacceptable calibration procedures (i.e., low thresholds too close to high thresholds).
On the other hand, the masseter and frontalis setup presented frowning as a suitable action. Moreover, in opposite to the temporalis pair, the masseter and the associated action (clenching teeth) are likely to lead to errors if the user is not able to perform clearly distinctive actions. It was clear during the evaluation and in the post-analysis that the most successful users used their lips to create different gestures. However, in general the system presented a good accuracy (94.17%).
One aspect that is interesting to compare is the precision and recall from able and disabled users. The results are quite similar and it is difficult to assess any particular difference. It is interesting to notice that, in the temporalis and frontalis setup, the tetraplegic users performed even better than the fully-capable population.
In general, the results achieved showed that the recognition system is accurate. The recognition rates are high and it is clear that a body gesture can be identified. However, if the set of monitored locations is close, the recognition accuracy can decrease. Also, if the user is not able to clearly make different gestures/contractions, the accuracy could also be damaged. However, the results were still highly accurate and the errors were minimal. The best placement positions are those with a clear associated body gesture (i.e., temporalis, frontalis, mentalis). If a multi-electrodes setup is applied, locations that have different motor actions associated are the best choices.
One of the most important factors to achieve an effective interface is that the system is easy to use. This is important while the system is in use but also in the mounting and configuration stages. Thus, although we have already validated the recognition system, those evaluations were performed with an experienced helper which performed the electrodes placement and configuration. However, in a daily scenario that experienced helper is not present. Moreover, the electromyographic signal characteristics imply variations from day to day and signal repeatability is impossible to achieve.
While designing our approach, the main focus and concerns go to the target user, a tetraplegic person. However, our approach influences and depends not only on the user, but also on the caretaker, at specific times. The mounting and initial system configuration is the most important step that both stakeholders (user and caretaker) are present. Thus, to state our approach as effective and usable, one question must be answered: can the user use the system and achieve good recognition results without the help of an experienced helper?
This evaluation was performed with one 26 years old, male tetraplegic user. It is also important to refer that the caretaker was a 24 years old female with no previous medical or electromyography related expertise. Her background is on housekeeping and she has completed the 6th grade.
This evaluation was composed by three sessions performed in three different days (spaced by two days) with one tetraplegic user at his home. Each individual session was similar to the one described in the Input Recognition Evaluation (Section 5.1.2 ) with the difference that all the mounting procedure was performed by the caretaker. For the caretaker to be able to perform the task for the three recalled setups (both temporalis and frontalis, both masseter and frontalis, frontalis and mentalis) we presented pictures of the monitored areas. Also, a set of recommendations on electrodes placement and system configuration were handed to the caretaker. Between tasks both users were asked for any problem thatmight have occurred. The test monitor tried to keep the users comfortable.
Figure 13 presents a summary results side-by-side with the results from the same user in the Input Recognition Evaluation. The first bar represents the overall results for the System Recognition Evaluation already presented. The other three bars present the results for the sessions where all the process was conducted by the caretaker and the user. Overall, the results are quite similar. In the first session, the number of errors is a little higher with the Caretaker placement than the Experienced Helper. However, by analyzing the electrodes placement and the signal afterwards, we concluded that there are not significant shifts in the placement but rather an evolution of the knowledge of the system by both stakeholders that adapted themselves to each other and to the application.
Another important analysis is on the error situations, which occurred in these evaluations, mostly in the first Caretaker experience. Most of the erroneous situations were created between right and left teeth clenching. This scenario is error prone as some users are unable to clearly clench teeth independently. The user showed difficulties with this scenario in all sessions. However, the results show that there was an improvement and we believe that it was related with both subjects' performance. The resumed results for this evaluation are available in Table 3.
Overall, the results are similar to the ones gathered in the Input Recognition Evaluation and present the system as usable even when no experienced user is involved. Moreover, we believe that both stakeholders can learn and get used to the system and improve performance. One important factor to mention is that the collection sites were established without the user intervention. These influences the results as some users are likely to dislike or be incapable of creating a distinct contraction. Once again, if the system is experienced for some sessions we believe that this would evolve and the users would achieve greater performances and accuracy rates.
Finally, the caretaker found the system easy to set up although she has showed a little discomfort in the beginning of the evaluation. The main problem that she mentioned was the ”wire mess” that was all around the user. This is a problem as the wires used were too small to offer the required flexibility for a more comfortable setup. However, this problem is easy to solve as they can be replaced with longer and thicker wires.
The primary goal of our work is to provide tetraplegic users with an effective mobile device control interface. Not only it was required an interface that enabled device control, but also one that was able to cope with a large spectrum of situations. This variability goes from physical differences among individuals to accommodating shifts even for the same individual.
We presented an Electromyographic control interface that allows users to input commands to a device thereby controlling applications through muscle contractions. The capability to monitor any voluntarily contracted muscle allows the system to adapt to different impairments individuals and the special needs they entail. Furthermore, evaluation studies showed that a such daily wearable interface is both feasible and suitable to control mobile devices for tetraplegics. Moreover, considering the actual state of assistive computer access technologies for tetraplegics, we believe that the work presented in this dissertation can easily be extended beyond mobile device interaction. As future work, we plan to develop a prototype to enable mobile interaction using the electromyographic control interface and evaluate it with the target users in varied scenarios.
[BSA99] SIGCAPH Comput. Phys. Handicap., (1999), 6—16, ACM Press, New York, NY, USA, issn 0163-5727.
[CIA05] Toward subtle intimate interfaces for mobile devices using an EMG controller, CHI '05: Proceedings of the SIGCHI conference on Human factors in computing systems, 2005, Portland, Oregon, USA, ACM Press, New York, NY, USA, 481—489, isbn 1-58113-998-5.
[CMHVC92] Estudo da funcao neuromuscular com recurso a electromiografia: desenvolvimento e fundamentacao de um sistema de recolha e processamento, Faculdade de Motricidade Humana, 1992.
[Col01] Electromyography based human-computer-interface to induce movement in elderly persons with movement impairments, WUAUC'01: Proceedings of the 2001 EC/NSF workshop on Universal accessibility of ubiquitous computing, 2001, Alcácer do Sal, Portugal, pp. 75—79, ACM Press, New York, NY, USA, isbn 1-58113-424-X.
[CPIA04] EMG as a Subtle Input Interface for Mobile Computing, Mobile HCI, 2004, Lecture Notes in Computer Science, 2004, , pp. 426—430, isbn 3-540-23086-6.
[FH99] SENIAM 9: European Recommendations for Surface Electromyography, Enschede, Netherlands, 1999, Roessingh Research and Development, isbn 90-75452-14-4.
[HFM99] SENIAM 8: European recommendations for surface electromyography. Results of the SENIAM project, Enschede, Netherlands, 1999, Roessingh Research and Development, isbn 90-75452-15-2.
[Luc02] Surface Electromyography: Detection and Recording, Tutorial, DelSys Inc, 2002.
[Nyq02] Certain topics in telegraph transmission theory, Proceedings of the IEEE, (2002), no. 2, 280—305, issn 0018-9219.
[PKCP99] Adaptive Emg-Driven Communication for the disability, Proceedings of the First Joint Engineering in Medicine and Biology, 1999. 21st Annual Conf. and the 1999 Annual Fall Meeting of the Biomedical Engineering Soc. BMES/EMBS Conference, 1999, p. 656, IEEE Press, isbn 0-7803-5674-8. ,
[Ros98] The Biofeedback Pointer: EMG Control of a Two Dimensional Pointer, ISWC '98: Proceedings of the 2nd IEEE International Symposium on Wearable Computers, 1998, p. 162, IEEE Computer Society, Washington, DC, USA, isbn 0-8186-9074-7.
[SFDW01] Onset Detection in Surface Electromyographic Signals: A Systematic Comparison of Methods, Journal on Applied Signal Processing, (2001), 67—81, issn 1110-8657.
[TCLK97] Design of the human/computer interface for human with disability using myoelectric signal control, Engineering in Medicine and Biology Society, 1997. Proceedings of the 19th Annual International Conference of the IEEE, 1997, , pp. 1909—1910, issn 1094-687X.
[WBM00] Dasher - a data entry interface using continuous gestures and language models, UIST '00: Proceedings of the 13th annual ACM symposium on User interface software and technology, 2000, San Diego, California, United States, pp. 129—137, ACM, New York, NY, USA, isbn 1-58113-212-3.
[WJ03] Gestures as Input: Neuroelectric Joysticks and Keyboards, IEEE Pervasive Computing, (2003), no. 2, 56—61, IEEE Educational Activities Department, Piscataway, NJ, USA, issn 1536-1268.
Fulltext as PDF. ( Size 4.4 MB )
Any party may pass on this Work by electronic means and make it available for download under the terms and conditions of the Digital Peer Publishing License. The text of the license may be accessed and retrieved at http://www.dipp.nrw.de/lizenzen/dppl/dppl/DPPL_v2_en_06-2004.html.
Tiago João Vieira Guerreiro, and Joaquim Armando Pires Jorge, Assessing Electromyographic Interfaces. JVRB - Journal of Virtual Reality and Broadcasting, 5(2008), no. 12. (urn:nbn:de:0009-6-16061)
Please provide the exact URL and date of your last visit when citing this article.