Facebook
Twitter
Pocket
Reddit
Email
Evelyn Glennie began percussion lessons around age 12, after losing much of her hearing to nerve deterioration. Her teacher struck a timpani drum and let the sound resonate, wondering aloud how they could make use of the drum’s vibrations. “He asked me to put my hands on the wall of the music room,” Glennie said recently, in a conversation from her home in England. She could feel the first impact of a drumbeat, but she could also feel the vibrations reverberating afterward. “It really slowed the body down, because I was paying attention to the whole journey of that sound,” she explained. “The whole body was participating in paying attention to the sound. And it really then just changed everything for me. It changed my sensitivity toward touch.” Glennie learned to discern the different pitches of musical notes this way, and eventually built a career as a celebrated solo percussionist.
Glennie’s exquisite sensitivity to vibration and remarkable talent for composing music is emblematic of the powerful relationship that millions of deaf and hearing-disabled people have to music. Many go to concerts and enjoy music in their homes through touch, vision, and movement.1 A select few, like Glennie, play musical instruments or sing professionally. But as scientists learn more about how our bodies and brains process vibration, they are producing a wealth of new research that will help deaf people better appreciate the complexities and emotional range of music, both as listeners and performers. In the process, they are also discovering how music moves through us to create symphonies of feeling.
Many deaf concertgoers clasp balloons to better feel acoustic vibrations through the thin rubber.
Humans with hearing perceive music and other sounds when certain frequencies of vibration—those within the audible range of 20 hertz to 20,000 hertz—travel through the air and are captured by tiny sensory cells in the cochlea, a snail-shaped organ in the inner ear. But music can also be perceived through the sense of touch. Sound waves can be felt as air pressure on the skin or travel through solid materials such as the soundboard of a guitar or the floor of a stage, where they can be picked up by the hands, body, and feet. Vibrations can also be sensed by the membranes between bones2 and the walls of internal cavities, such as the lungs and chest.3, 4
Deaf musical traditions have long taken advantage of the power of touch to perceive vibration, which is called the vibrotactile sense. Many deaf concertgoers clasp balloons to better feel acoustic vibrations through the thin rubber, and Glennie is known for performing barefoot so she can better sense vibrations pulsing through the stage. After he became deaf, Beethoven is said to have fashioned himself a kind of hearing aid from a stick: He clenched one end with his teeth and let the other end rest on his piano so as to pick up vibrations from the music.
Thanks to some combination of brain wiring and practice, many people who are born deaf seem to develop enhanced sensitivity to the sense of touch,5 and this can be expressed in various ways. Neuroscientist and musician Frank Russo is investigating how electrical signals in the brain track beats detected through the vibrotactile sense, and his initial data shows different results in deaf and hearing people. “What we’ve seen so far is that the deaf brain is doing a better job of representing the beat than your typical hearing person’s brain,” said Russo, who studies musical cognition, acoustics, and auditory perception at Toronto Metropolitan University.
In the brains of deaf people, the sound-processing part of the brain is often repurposed. Without an influx of sound signals to interpret, the auditory cortex can rewire itself to receive signals from other senses, such as touch and sight.3 Every part of the brain usually ends up being used in one way or another, said Karolina Finc, a cognitive scientist at Nicolaus Copernicus University. Research Finc published in 2020 revealed that for those who became deaf early in life, circuits throughout the brain reorganized to accommodate hearing loss.6 The different networks had to learn to talk to each other in new ways, she said. For example, a network involved in attention and problem-solving was found to be more strongly linked to networks that coordinate motor responses, visual systems, and memory. “The whole brain has to adjust,” Finc said.
Because every hard-of-hearing person has a different history with music, and because every brain is different, how and to what extent an individual’s vibrotactile sense may fill in for the loss of hearing will vary. Either way, vibration communicated through touch offers a wealth of musical subtlety that researchers are now quantifying.
Neuroscientists who research sensory perception have historically focused on the auditory and visual systems. The vibrotactile sense is less well understood, said Mario Prsa, a neuroscientist at the University of Fribourg in Switzerland who studies how the brain coordinates movement, among other topics. “We somehow neglect this vibration sense,” he said. In 2016 and 2017, when Prsa was a postdoc in Daniel Huber’s neuroscience lab at the University of Geneva, he was researching how to enhance mind control of prosthetics. To train mice to activate certain neurons in the motor cortex, a key area for regulating movement, he tried giving them feedback in the form of tactile vibrations.7
While documenting the results, he chanced upon surprising behavior from neurons in the somatosensory cortex: Different sets of neurons responded most intensely to different frequencies of vibration. Prsa dug into the scientific literature to see what others before him had learned about the behavior of the neurons he was observing and came up empty. He could find no other accounts of the neurons’ frequency-specific responses to vibration. So he and his colleagues at the University of Geneva decided to run some experiments. When they sent vibrations at different frequencies and intensities through a metal rod to human fingertips8 and mouse forelimbs,9 they confirmed that different frequencies elicited responses from different neurons.
This was a significant finding because the human auditory system10 behaves in a similar manner.11 In fact, the sensory systems for hearing and touch overlap in many intriguing ways. Even in hearing people, signals from vibrotactile input can be detected by the auditory cortex,12 and tactile and auditory stimuli can be confused when perceived together or in alternating fashion.4 In the hearing, auditory input can also alter the perception of whether an object such as parchment paper feels rough or smooth.13 In both hearing and touch, the mechanics of sensing vibration is essentially the same: sensory nerves called mechanoreceptors, located in the cochlea and throughout the body, bend in response to pressure changes, triggering a neural impulse that is sent to the brain.4 Many creatures, like some insects and rodents, still primarily communicate through vibrations and some neuroscientists have even proposed that hearing evolved in more complex animals from the vibrotactile sense.
In the past decade or two, engineers have built an array of vibrating wearables and devices that aim to help deaf people experience more of the nuances of music: jackets, gloves, and wristbands for listeners, and drum stools and platforms for musicians.14 One wearable, recently beta-tested at a concert in Las Vegas by Not Impossible Labs, a company that engineers innovative open-source healthcare tech, looks a little bit like a harness or a backpack. It consists of a series of wireless sensors—designed to amplify and transmit vibrations of different frequencies—strapped to a person’s back, shoulders, and waist, as well as at the ankles and wrists.
But many of the existing instruments are still relatively crude and primarily convey only bass and beats. Glennie, who has tried out vibrating platforms, chairs, and wristbands, says she’s concerned that devices like these can flatten the tactile experience of music. Scientists are only just beginning to understand how to capture and convey the true complexity, depth, and emotional range of music through touch, to translate sweeping vocals, deep bass notes, and jangly guitar riffs into tactile vibrations in intricate combinations of frequencies and intensities that can be exquisitely felt.
A deaf person’s brain can do a better job of representing the beat than the typical hearing person’s brain.
“If we can optimize the use of vibration through technology to bring better music experiences to people who are living with hearing loss or deafness, then why wouldn’t we?” asks Russo. “We’re just seeing that blossoming now.”
When the Musical Vibrations project at the University of Liverpool, a group that provides equipment to translate music into vibration, offered their technology to deaf rapper SignKid, he found it helped him keep up with the beat. They fed one of his songs into the equipment, which transformed tracks with bass guitar, guitar, kick drum, and snare drum into vibrations. Each of SignKid’s bare feet rested on two shakers—red and white canisters topped with circular plates that produced the vibrations he felt through his heels and forefeet. “I like this,” he told the Musical Vibrations team. “I could use this.” The team loaned him the equipment, which he used in producing his 2019 EP, The Visual Experience.
Some musical features of tactile vibrations are obvious: They can convey rhythm effectively. Russo’s analyses with EEG show brain signals measured through the scalp pulsed in time with a musical beat, whether that beat was communicated via sound or touch.15 The pattern isn’t quite as clean with “funkier” sequences, which feature unequal time intervals, he said. But touch can convey much more detailed aspects of music, as well. The amplitude of a sound wave, which translates to volume in the auditory system, is felt as intensity of pressure by the vibrotactile sensory system. Frequency creates a sense of pitch in both modes.16
We’re not as sensitive to changes in vibration as we are to changes in sound, however, which makes pitch detection more difficult. Even with training, using vibration to identify the difference between nearby notes like C and C sharp was difficult for both deaf and hearing people, according to research by Carl Hopkins, a leader of the Musical Vibrations project.17 Still, our bodies can detect through touch the major rises and falls in pitch that make up melodies. We can also discern through the vibrotactile sense the pitches of many of the notes produced by musical instruments and human vocals, though this range is narrower than it is for sound.18
Without sounds to interpret, the auditory cortex can rewire itself to receive signals from other senses.
Whereas our ears are most sensitive to frequencies between 2,000 and 5,000 hertz19—the vibrotactile range generally falls between 5 and 1,000 hertz.16 (For reference, deep voice territory for men is generally between 75 to 100 hertz.) This means that many of the highest tonal frequencies that the ear commonly detects cannot be picked up by the vibrotactile sense, for example—or cannot be identified precisely. But it also means that we can feel certain low frequencies even when we can’t hear them.
Through vibrations, people can also identify differences in timbre, a quality that shapes the “color” of sound we hear. “Even if you take two pianos, they don’t have exactly the same timbre,” Russo said. “They could both be playing middle C, but there’s something brighter about a grand piano, as compared to an upright piano.” Just from touching a vibrating gadget, both hearing and deaf people have demonstrated the ability to detect the differences between a cello, a piano, and a trombone, as well as between dull and bright sounds.4
Prsa and his collaborators are studying whether certain combinations of frequencies are more pleasing to the vibrotactile sensory system than others. “Is there such a thing as a vibrotactile octave?” Prsa wondered. “Is there something similar to dissonant notes?” He and his collaborators have begun to explore these questions with a simple melodic experiment: testing how easily deaf and hearing participants could identify basic melodies, like “Happy Birthday,” through vibrations sensed by touch. Translating audio into vibrations that can be felt through touch is an imperfect science. Because the range of frequencies detectable by touch is narrower and lower than the range detectable by ear, and because it is harder to distinguish between neighboring tones, some notes are altered in the process. Prsa wanted to see how these alterations—in particular the spacing between one frequency and another—influenced subjects’ ability to identify the melody. The results of this work have not yet been published.
Researchers are also attempting to understand how much of the emotion in music comes across through the vibrotactile sense. “Through vibration, you can get critical aspects of vocal emotion,” Russo said. “You can get the tiredness in someone’s voice, the grit in someone’s anger.” For Glennie, lower-frequency vibrations carry the most emotion, while higher-frequency vibrations lack it. Byron Remache-Vinueza of the University of Málaga described the emotional range of tactile vibrations in different terms: Switching music from sound to vibration is like eating a favorite meal that your mother used to make, only puréed. It’s just not the same emotional experience. But he’s excited about the idea of making new music from scratch purely in vibrations—no sound—perhaps through unique instruments designed to produce only vibrations or through digital composition.
Another obstacle to translating existing music into vibration is separating out the different sounds. Many of the current technologies can so far only process music that is already divided up into discrete audio tracks for each instrument or layer of notes, said Mark Fletcher of the Institute of Sound and Vibration Research at the University of Southampton. But it’s hard to “de-mix” sound well, he said. Even the most advanced algorithms that feed music into vibrating equipment struggle to account for the way lower and more intense vibrations mask others, and the way different instruments may draw more attention at different points in a song. Without robust ways to separate the different instruments in a piece and compile balanced signals in the form of vibrations, the vibrations that come out of these systems will lack the nuance of the music that goes into them.
Through years of research, Hopkins has seen many hard-of-hearing people test out his team’s system, absorbing the vibrations through their hands and feet. Some preferred the ways they already interacted with sound. Glennie, for example, prizes the complexity of the vibrations she can sense with her body when these come directly from the different instruments that are being played and the built environments of concert halls and practice rooms. But she’s glad these experiments are continuing and that a larger demographic will get the opportunity to try the technologies out.
Some deaf people get carried away by the feeling of the music conveyed by the vibrotactile instruments. One subject told Hopkins that she had never realized the difference between the white and black keys on the piano. And when the team brought their vibrating system to a deaf school, the teachers reported that students were much more engaged in music lessons. “The children really took to it,” Hopkins said. One boy just kept coming back in at break time and knocking on the door and asking, “Can I play on the equipment again?” 
Lead image: Donatas1205 / Shutterstock
References
1. Holmes, J.A. Expert listening beyond the limits of hearing: Music and deafness. Journal of the American Musicological Society 70, 171–220 (2017).
2. Rowe, M., Tracey, D., Mahns, D., Sahai, V. & Ivanusic, J. Mechanosensory perception: Are there contributions from bone-associated receptors? Clinical and Experimental Pharmacology and Physiology 32, 100-108 (2005).
3. Tranchant P., et al. Feeling the beat: Bouncing synchronization to vibrotactile music in hearing and early deaf people. Frontiers in Neuroscience 11 (2017).
4 Russo, F.A., Ammirante, P., & Fels, D.I. Vibrotactile discrimination of musical timbre. Journal of Experimental Psychology: Human Perception and Performance 38, 822-826 (2012).
5. Levänen, S. & Hamdorf, D. Feeling vibrations: Enhanced tactile sensitivity in congenitally deaf humans. Neuroscience Letters 301, 75-77 (2001).
6. Bonna, K. Early deafness leads to re-shaping of functional connectivity beyond the auditory cortex. Brain Imaging and Behavior 15, 1469-1482 (2021).
7. Prsa, M., Galiñanes, G.L., & Huber, D. Rapid integration of artificial sensory feedback during operant conditioning of motor cortex neurons. Neuron 93, 929-939.e6 (2017).
8. Prsa, M., Kilicel, D., Nourizonoz, A., Lee, K.-S., & Huber, D. A common computational principle for vibrotactile pitch perception in mouse and human. Nature Communications 12, 5336 (2021).
9. Prsa, M., Morandell, K., Cuenu, G., & Huber, D. Feature-selective encoding of substrate vibrations in the forelimb somatosensory cortex. Nature 567, 384-388 (2019).
10. Humphries, C., Liebenthal, E., & Binder, J.R. Tonotopic organization of human auditory cortex. Neuroimage 50, 1202-1211 (2010).
11. Barbour, D.L. Intensity-invariant coding in the auditory system. Neuroscience & Biobehavioral Reviews 35, 2064-2072 (2011).
12. Caetano, G. & Jousmäki, V. Evidence of vibrotactile input to human auditory cortex. Neuroimage 29, 15-28 (2006).
13. Auer, E.T. Jr, Bernstein, L.E., Sungkarat, W., & Singh, M. Vibrotactile activation of the auditory cortices in deaf versus hearing adults. Neuroreport 18, 645-648 (2007).
14. Remache-Vinueza, B., Trujillo-Léon, A., Zapata, M., Sarmiento-Ortiz, F., & Vidal-Verdú, F. Audio-tactile rendering: A review on technology and methods to convey musical information through the sense of touch. Sensors 21, 6575 (2021).
15. Gilmore, S.A. & Russo, F.A. Neural and behavioral evidence for vibrotactile beat perception and bimodal enhancement. Journal of Cognitive Neuroscience 33, 635-650 (2021).
16. Mahns, D.A., Perkins, N.M., Sahai, V., Robinson, L., & Rowe, M.J. Vibrotactile frequency discrimination in human hairy skin. Journal of Neurophysiology 95, 1442-1450 (2006).
17. Hopkins, C., Maté-Cid, S., Fulford, R., Seiffert, G., & Ginsborg, J. Perception and learning of relative pitch by musicians using the vibrotactile mode. Musicae Scientae 1-24 (2021).
18. Hopkins, C., Maté-Cid, S., Fulford, R., Seiffert, G., & Ginsborg, J. Vibrotactile presentation of musical notes to the glabrous skin for adults with normal hearing or a hearing impairment: Thresholds, dynamic range and high-frequency perception. PLoS One 11, e0155807 (2016).
19. Kirsten, L., et al. Endoscopic optical coherence tomography with wide field-of-view for the morphological and functional assessment of the human tympanic membrane. Journal of Biomedical Optics 24, 031017 (2018).
Mario Prsa’s work translating audible sounds into vibrations for the hearing impaired is funded by the Bertarelli Foundation.
Get the Nautilus newsletter
Cutting-edge science, unraveled by the very brightest living thinkers.
