For individuals communicating using a switch, a new interface learns how they choose, and then adjusts itself accordingly.
Image credits: Pixel, CC0 Public Domain
In 1995, Jean-Dominique Bouby, editor of the French fashion magazine, suffered a seizure while driving a car, leading to a condition known as locked-in syndrome, a neurological disease in which the patient becomes completely paralyzed and only muscles are affected. can control. Eyes.
Bobby, who had signed a book contract shortly before his accident, wrote the memoir “The Diving Bell and the Butterfly” using a dictation system in which his speech therapist recited the alphabet and when he was correct. He used to blink when he said the letter. He wrote a 130-page book in a blink of an eye.
Technology has come a long way since the accident of Bobby. Many individuals with severe motor impairment due to locked-in syndrome, cerebral palsy, amyotrophic lateral sclerosis, or other conditions can communicate using computer interfaces where they select letters or words in an onscreen grid by activating a switch, Often by pressing a button, releasing a gust of wind, or blinking.
But these row-column scanning systems are very rigid, and, similar to the technology used by Bowby’s speech therapists, they highlight each option one at a time, making them frustratingly slow for some users. . And they’re not suitable for tasks where the options can’t be arranged in a grid, such as drawing, browsing the web, or gaming.
A more flexible system being developed by researchers at MIT places individual selection indicators next to each option on a computer screen. Indicators can be placed anywhere – next to anything a person can click with the mouse – so the user doesn’t have to cycle through a grid of options to make a selection. The system, called Nomon, includes probabilistic logic to learn how users select, and then adjusts the interface to improve their speed and accuracy.
Participants in a user study were able to type faster using Nomon than a row-column scanning system. Users also did better on the picture selection task, demonstrating how Nomon can be used for more than just typing.
“It’s great and exciting to be able to develop software that has the potential to really help people. Being able to find those signals and turn them into communication is what we’re used to,” says senior author Tamara Broderick , an associate professor and a member in the MIT Department of Electrical Engineering and Computer Science (EECS). Laboratory for Information and Decision Systems and Institute for Data, Systems and Society.
Joining Broderick on the paper is lead author Nicolas Bonaker, an EECS graduate student; Emilie-Marie Nel, Head of Innovation and Machine Learning at Everly and Visiting Lecturer at the University of the Witwatersrand in South Africa; and Keith Vartanen, associate professor at Michigan Tech. The research is being presented at the ACM Conference on Human Factors in Computing Systems.
clock in
In the Nomon interface, a small analog clock is placed next to each option the user can select. (A gnome is the part of a solarium that casts a shadow.) The user looks at an option and then clicks their switch when that clock hand passes the red “noon” line. After each click, the system changes the clocks’ phases to isolate the most likely next targets. The user clicks repeatedly until their target is selected.
When used as in a keyboard, Nomon’s machine-learning algorithms try to guess the next word based on the previous words and each new letter as the user selects.
Broderick developed a simplified version of Nomon several years ago, but decided to revisit it to make the system easier for motor-handicapped individuals to use. He enlisted the help of then-graduate Bonker to redesign the interface.
He previously consulted with non-profit organizations that work with motor-impaired individuals, as well as motor-impaired switch users, to gather feedback on the Nomon design.
They then designed a user study that would better represent the abilities of motor-disabled individuals. Broderick explains that they wanted to make sure to thoroughly test the system before motor-impaired users were using up too much of their valuable time, so they tested it on non-Switch users first.
switch switch
To collect more representative data, Bonaker designed a webcam-based switch that was harder to use than simply clicking a key. Non-Switch users had to tilt their body to one side of the screen and then back to the other side to register a click.
“And they have to time it exactly, so it really slows them down. We did some empirical studies that showed they were very close to the reaction times of motor-impaired individuals,” Broderick says.
They ran a 10-session user study with 13 non-switch participants and a single-switch user with an advanced form of spinal muscular dystrophy. In the first nine sessions, participants used Nomon and a row-column scanning interface for 20 min to perform text entry, and in the 10th session they used two systems for the picture selection task.
Non-Switch users typed 15 percent faster using Nomon, while motor-impaired users typed even faster than non-Switch users. When typing unfamiliar words, users were 20 percent faster overall and made half the mistakes. In their final session, they were able to complete the picture selection task 36 percent faster using Nomon.
“Nomon is a lot more forgiving than row-column scanning. With row-column scanning, even if you’re a little off, you now chose B instead of A and that’s an error,” Broderick says.
adapt to click noise
With its probabilistic logic, Nomon knows everything a user is likely to click to make the process faster, easier and less error-prone. For example, if the user selects “Q”, Nomon will make it as easy as possible for the user to select “U”.
Noman also learns how A user clicks. So, if the user always clicks a bit after the clock strikes noon, the system adapts to that in real-time. It is also adaptable to noise. If a user’s click is often off the mark, the system requires additional clicks to ensure accuracy.
This probabilistic logic makes Nomon powerful, but it also requires a higher click-load than a row-column scanning system. Clicking multiple times can be a daunting task for severely motorized users.
Broderick hopes to reduce click-load by incorporating gaze tracking into Nomon, which will give the system more robust information about what a user can choose next based on which part of the screen. The researchers want to find a better way to automatically adjust clock speeds to help users be more precise and efficient.
They are working on a new series of studies in which they plan to partner with more motor-disabled users.
“So far, feedback from motor-disabled users has been invaluable to us; We are very grateful to the motor-impaired user who commented on our initial interface and the isolated motor-impaired user who participated in our study. We are currently expanding our study to work with a larger and more diverse group of our target population. With their help, we are already further improving our interface and working to better understand Nomon’s performance,” she says.
“Non-speaking individuals with motor disabilities are not currently provided with efficient communication solutions to interact with speaking partners or computer systems. This ‘communication gap’ is a known unsolved problem in human-computer interaction, and So far there is no good solution. This paper demonstrates that a highly creative approach based on a statistical model can provide tangible performance benefits to the users who need it most: a way for non-speaking individuals to communicate. are dependent on switches,” says Per Ola Christensen, professor of interactive systems engineering at Cambridge. University, which was not involved in this research. “The paper also demonstrates the value of complementing insights from computational experiments with the participation of end users and other stakeholders in the design process. I find this a highly creative and important paper in an area where it is extremely difficult to make significant progress.” “
Written by Adam Zewe
Source: Massachusetts Institute of Technology
