“EMG sensors aren’t very accurate,” said Weinberg, director of Georgia Tech’s Center for Music Technology. “They can detect a muscle movement, but the signal is too noisy to infer which finger the person wants to move. We tried to improve the pattern detection from EMG for Jason but couldn’t get finger-by-finger control.”
But then the team looked around the lab and saw an ultrasound machine. They partnered with two other Georgia Tech professors – Minoru Shinohara, Chris Fink (College of Sciences) and Levent Degertekin (Woodruff School of Mechanical Engineering) — and attached an ultrasound probe to the arm. The same kind of probe doctors use to see babies in the womb could watch how Barnes’ muscles moved.
“That’s when we had a eureka moment,” said Weinberg. When Barnes tries to move his amputated ring finger, the muscle movements differ from those seen when he tries to move any other digit. Weinberg and the team fed each unique movement into an algorithm that can quickly determine which finger Barnes wants to move. The ultrasound signals and machine learning can detect continuous and simultaneous movements of each finger, as well as how much force he intends to use.
“It’s completely mind-blowing,” said Barnes. “This new arm allows me to do whatever grip I want, on the fly, without changing modes or pressing a button. I never thought we’d be able to do this.”
This is the second device Weinberg’s lab has built for Barnes. His first love is the drums, so the team fitted him with a prosthetic arm with two drumsticks in 2014. He controlled one of the sticks. The other moved on its own by listening to the music in the room and improvising.
The device gave him the chance to drum again. The robotic stick could play faster than any drummer in the world. Worldwide attention has sent Barnes and Weinberg’s robots around the globe for concerts across four continents. They’ve also played at the Kennedy Center in Washington, D.C. and Moogfest.
“If this type of arm can work on music, something as subtle and expressive as playing the piano, this technology can also be used for many other types of fine motor activities such as bathing, grooming and feeding,” said Weinberg. “I also envision able-bodied persons being able to remotely control robotic arms and hands by simply moving their fingers.”