Rocks, Paper or Scissors? Electrodes Implanted in the Brain Can Read Which Hand Signs You Intend to Make

Author: Kate Baggaley


Scientists are using the parlor game Rocks-Paper-Scissors (and its extended version, Rocks-Paper-Scissors-Lizard-Spock) to develop better prosthetics to read and act on the intentions of people who can't move.

In a new study, researchers asked a paralyzed man with electrodes implanted in his brain to imagine making different hand shapes from the game. The scientists were able to read patterns of brain activity and know what hand sign he was imagining.

It's the first time scientists have been able to decode hand shapes from a part of the brain called the posterior parietal cortex. This is the area of the brain where intentions about moving a limb take place, before being sent to the motor cortex, which is in charge of actually executing the movement. Reading movement intentions could allow paralyzed people to direct robot hands with their minds and form more dexterous, complex hand shapes than those allowed by taking the signals from the brain areas traditionally tapped for neuroprosthetics.

Where to read from?

Erik G. Sorto has been paralyzed from the neck down for more than a decade. A few months ago, the same team of researchers who produced the new study announced that implanting electrodes in Sorto's brain allowed him to control a robot limb well enough to pick up and sip from a cup with a straw in it.

Traditionally, scientists have implanted electrodes in the motor cortex to read signals of voluntary muscle movements. But in fact, when reaching out to grab a cup, we don't think about each muscle movement that makes up that action. We just "intend" to reach out and grab, and the brain works out the rest. Therefore, neuroprosthetics that use direct movement signals from the motor cortex end up making delayed, jerky movements because people are forced to focus on each individual muscle move.

So for Sorto, scientists turned to another part of the brain, the posterior parietal cortex. "It takes sensory information and processes it in a way that it can be used to plan actions that are then sent to motor cortex where they are executed," said coauthor Richard Andersen, a neuroscientist at the California Institute of Technology in Pasadena.

Because this area controls the intent to move, connecting electrodes to this region instead allows Sorto to make smoother movements. "This intent can be interpreted by smart computers that can handle the fine details of the robotic movement rather than relying on the brain to provide all this detailed information," Andersen said.

In nonhuman primates, scientists had previously decoded neuron activity from the posterior parietal cortex that corresponds with the intention to grasp an object. Andersen and his colleagues wanted to know with more complicated hand shapes could also be read from the posterior parietal cortex in humans. This would allow for neuroprosthetics with better motor skills.

Rock, paper, scissors, neurons

For the new study, Andersen and his colleagues showed Sorto pictures of a rock, some paper, scissors, a lizard, or a photo of Leonard Nimoy in his role as Spock. Sorto then had to imagine making a corresponding hand shape.

For each hand shape, different groups of neurons became active. This meant that, based on which nerve cells were firing, the scientists could figure out what shape Sorto was imagining.

The hand shapes that Sorto imagined forming for each symbol were not directly related to the hand shape he would have had to make to actually grasp its corresponding object. This means that the neurons whose activity Andersen and his team read can be used for tasks like communication. "The area can go beyond just the simplistic mapping of an object to a grasp," Andersen said.

This means that neuroprosthetics reading from the posterior parietal cortex could make hand shapes that are complicated and more abstract than those used to grab objects.

The team also saw two distinct populations of neurons become active for each of the hand shapes. "Some cells are coding the intent of the subject and others are coding what the subject is seeing," Andersen said.

Some of these neurons became active when Sorto saw a picture of a rock or lizard but did not fire when he just heard those words spoken aloud. The other population of neurons still spiked in activity if Sorto heard the word "rock" and had to imagine a fist shape. These cells "are more generally coding the imagined movement triggered by vision or hearing," Andersen said.

"This finding of two populations is important for separating attentional-visual components of neuronal response from motor-related components," he and his colleagues concluded in the paper, which they published November 18 in the Journal of Neuroscience. This "can be used to drive neuroprostheses in an intuitive manner."