Helping People With Limited Fine-Motor Skills Play Video Games

Carnegie Mellon University researchers are using sound to help people with hand tremors, cerebral palsy, nervous system damage and other fine-motor limitations enjoy video games.

But the research won’t stop there. Acoustic resonance sensors could aid physicians in monitoring vital signs, improve farming or environmental monitoring, or even help people read Braille.

“The game controller idea is really interesting,” said Justin Chan, an assistant professor in both the Software and Societal Systems Department (S3D) and the Electrical and Computer Engineering Department (ECE), who advised the project. “But acoustic resonance sensors have a wide array of potential uses.”

The researchers paired this sensor technology — which uses an embedded speaker and microphone in a touch pad to analyze acoustic pattern changes — with machine learning to build a computer game controller that is easier to manipulate.

The game controller’s touch pad consists of a round, soft surface a little less than 3.5 inches in diameter that sits atop a 3D-printed case. Inside the controller rests a microphone and tiny speaker that transmits high-frequency sound, undetectable to the human ear. The pair create a distinctive acoustic pattern inside the case, which a user can change by pressing anywhere on the soft surface. The researchers used a machine learning model to map the various frequency profiles to particular points of contact.

“The idea is that we create unique acoustic patterns inside the controller,” said Kevin Xu, an ECE doctoral student who led the project. “The fundamental principle is that there is a unique frequency profile at any contact position when you press down on the controller.” 

Users can also tailor the sensor to their own movements by clicking a button when pressing on the sensor, said Xu, a National Science Foundation Graduate Research Fellow.

Developing the game controller originated as a final project in a course on building user-focused sensing systems taught by School of Computer Science faculty members Yuvraj Agarwal and Mayank Goel. The controller showed promise, so Xu and a small team of faculty and students continued to work on it.

“The thought process was that it would be kind of cool to apply it to something that was a problem for gamers with disabilities,” said S3D Ph.D. student Anupama Sitaraman, noting that adaptive controllers are often expensive and can require some experimentation to become fully functional.

“One advantage of the technology is that speakers and microphones are affordable and small, and they are on almost all computing devices,” Chan said. “A camera-based sensor might require a custom, embedded board to interface with a device, but an inexpensive acoustic sensor can simply be plugged into the audio jack of a phone, tablet or computer.”

Acoustic resonance sensing is still in early development but has the potential for many applications. In his own lab, Chan's group is incorporating the technology into a pen for reading Braille. As a user swipes the pen across a line of Braille letters, the sensor can detect the raised dimples of the Braille letters. The researchers are now determining how to make sense of the acoustic changes in real time as the sensor moves across the page.

“The motivation is that 90 percent of blind people don’t know how to read Braille,” Chan said, adding that many people lose sight in adulthood, when it’s difficult to learn Braille.

Other applications include fields such as robotics. Most grippers are rigid tools, but incorporating soft acoustic resonance sensors might enable a robot to know what it’s picking up, whether the object is brittle, and where the robot has grasped it. 

“You’re giving it a sense of physical intelligence of its environment,” Chan said. 

Acoustic resonance sensors could also be used in medicine to monitor vital signs, or in environmental applications, such as sensing changes in soil moisture.

One of the CMU researchers' unique technical contributions was overcoming “mirror image” signals that initially caused sensor performance problems. Because the cylindrical sensor pad is symmetrical, its internal geometry was also symmetrical in its initial design. Pressing on the pad at locations equidistant from the center, such as at the 3 o’clock and 9 o’clock positions, generated signals hard to distinguish from each other.

The solution, Xu said, was to add randomly spaced shapes inside the pad, which created an asymmetric geometry that produced unique acoustic signals wherever the sensor pad was pressed. In smaller sensors, where changing the internal geometry of the sensor is difficult, researchers achieved the same effect by using multiple microphones.

Xu presented a paper describing the controller at the IEEE Sensors Conference. Read the paper on the IEEE Xplore database.