Devices that Read Human Thought now Possible

Brain Implants Could Help Severely Disabled

CARL T HALL / SF Chronicle 10nov03

[More below]

New Orleans, LA—Less than a month after a widely heralded experiment showed how thought-reading implants can work in monkeys, scientists presented new findings Sunday suggesting such machines could work in people, too.

Dr. Miguel A.L. Nicolelis of Duke University said previously unreported human experiments demonstrated success with one type of a so-called brain computer interface, or BCI.

He and others discussed their latest findings Sunday at the annual meeting in New Orleans of the Society for Neuroscience, the world's largest gathering of brain researchers. About 28,000 people are attending the weeklong event.

Much of the attention on Sunday was given to technology designed to overcome paralyzing injuries or illnesses afflicting the nervous system. About 11,000 new cases arise every year, adding to a total estimated at more than 200,000.

Nicolelis said the new study had been done in a few Parkinson's disease patients while they were undergoing open-skull neurosurgery for their disease.

Full results, he said, have been submitted for peer review to a scientific journal and were not a formal part of the program, in which he and colleagues reported new details from the monkey experiments already published.

Nicolelis said the important point was that the principle had been shown to work: People can control devices merely by thinking.

Ultimately, it may be possible to design high-tech implants that can read and direct the muscles using the patient's own intentions and natural sensory equipment.

For now, it's a much less grandiose business of just tuning the equipment to the human brain's frequency.

In the Duke experiments, patients were being fitted with standard electrical stimulator devices, which can help to control Parkinson's symptoms.

This procedure requires the patient to be awake while the surgeon identifies a safe route through brain tissue, taking care not to harm brain cells needed for essential functions. As part of that process, the surgeon periodically asks the patient to speak or move while recording localized brain activity.

Nicolelis and his colleagues took advantage of the opportunity and recorded the information the surgeon was obtaining. Then, for five-minute periods while the patient was being operated on, they conducted simple reaching-and-grasping experiments to determine whether the patient's intentions could accurately be read—the first essential step in controlling a limb by computer implant.

That's a far cry from proving that a workable long-term implant would be safe and effective. Nicolelis said it was much too soon to "even think about" moving any particular device into full-blown clinical trials.

A competing group, however, led by founders and collaborators of a company called Cyberkinetics Inc., has announced plans to begin a small safety study next year of an implant designed to allow a paralyzed patient to control a desktop computer.

That device, called "BrainGate," is based on research at Brown University, led by scientist John Donaghue. He and other company officials described the technology on Sunday as a "novel gateway" for people with no other options.

"These are the opening days of a new era in neurotechnology," Donaghue said.

The competition, however, has gotten somewhat testy of late amid an explosion of interest. Some scientists accuse Nicolelis of overreaching, noting that his latest monkey experiment actually wasn't the first to show a "thoughts-into-action" device could function in a primate; he was merely the first to show that a monkey's brain firings could be harnessed to direct complicated movement, involving both reaching and grasping.

Meanwhile, Nicolelis decried the entry of corporate interests into a field once thought to be purely science fiction, now being taken seriously as modern medicine at the cutting edge of technology.

"I am a university professor," Nicolelis said. "I have no interests in any business. I am Brazilian—I want to have fun, I don't want to make money. What I am very afraid of is that people who really want to make a buck out of this will be rushing into the clinical thing. I don't believe in that. A lot of important science needs to be done, and we need to go step by step in a very careful way."

All the labs claim to be pursuing the technology responsibly.

Donaghue and his colleagues pointed out they were also university scientists who realized the only way to fully exploit the technology was to form a company capable of raising the money needed to carry out very expensive clinical studies. Cyberkinetics is proceeding with the guidance of the U.S. Food and Drug Administration.

In the latest studies on people, Nicolelis' Duke group had to use a simplified version of the animal study protocol to stay within the bounds of a five-minute surgical window. But that was still enough, Nicolelis said, to show animal and human brains can be read much in the same way.

"We are showing the same computational algorithms work, the same technology in general works, suggesting the principle would work in a patient that is severely handicapped," Nicolelis said. "We are able to predict the hand position, and the hand force, while they are doing the task during the surgery."

Before you can lift even a finger, nerves fire in the brain, along the spinal cord and nerve pathways of the arm, then back again in a tightly controlled feedback loop.

Douglas J. Weber, of the University of Alberta in Edmonton, reported new research Sunday suggesting that the motion of a limb can be accurately predicted by reading the firings of just a handful of brain cells—only 10 or so in one case.

That means it may be simpler than once imagined to tap into the body's own sensory apparatus to keep some natural motion going with a brain implant merely as a detour around a damaged spinal cord or other problem in the brain's natural circuitry.

Dr. Jonathan Wolpaw of the New York State Department of Health's Wadsworth Center described new methods of reading signals that can be detected outside and just beneath the surface of the skull, suggesting the possibility that some devices may not even have to be implanted into the brain. Implants run some risk of infections and other problems.

But he and others emphasized it might be several years before the first such devices were ready for widespread use, and they noted that the technology worked only in individuals who might be utterly disabled and "locked in," with no ability to move even their eyes, and yet had enough healthy brain activity to drive the implants.

The revolution will start slowly, Wolpaw said, in a few people "who are the most disabled and who have no other options."

Monkeys Consciously Control a Robot Arm Using Only Brain Signals; Appear to "Assimilate" Arm As If it Were Their Own

PRESS RELEASE 13ocr03

DURHAM, N.C. -- Researchers at Duke University Medical Center have taught rhesus monkeys to consciously control the movement of a robot arm in real time, using only signals from their brains and visual feedback on a video screen. The scientists said that the animals appeared to operate the robot arm as if it were their own limb.

The scientists and engineers said their achievement represents an important step toward technology that could enable paralyzed people to control "neuroprosthetic" limbs, and even free-roaming "neurorobots" using brain signals.

Importantly, said the neurobiologists, the technology they developed for analyzing brain signals from behaving animals could also greatly improve rehabilitation of people with brain and spinal cord damage from stroke, disease or trauma. By understanding the biological factors that control the brain's adaptability, they said, clinicians could develop improved drugs and rehabilitation methods for people with such damage.

The advance was reported in an article published online Oct. 13, 2003, in the Public Library of Science (PLoS), by neurobiologists led by Miguel Nicolelis, M.D., who is professor of neurobiology and co-director of the Duke Center for Neuroengineering. Lead author of the paper was Jose Carmena, Ph.D., in the Nicolelis laboratory. Besides Nicolelis, the other senior co-author is Craig Henriquez, Ph.D., associate professor of biomedical engineering in the Pratt School of Engineering, who is also the other center co-director. The research was funded by the Defense Advanced Research Projects Agency and the James S. McDonnell Foundation.

Nicolelis cited numerous researchers at other institutions whose work has been central to the field of brain-machine interfaces and in understanding the brain -- and whose insights helped lead to the latest achievement. They include John Chapin, Ph.D., State University of New York Health Science Center, Brooklyn; Eberhard Fetz, Ph.D., University of Washington, Seattle; Jon Kaas, Ph.D., Vanderbilt University; Idan Segev, Ph.D., Hebrew University, Jerusalem, and Karen Moxon, Ph.D., Drexel University.

In previous research, Nicolelis and his colleagues demonstrated a brain-signal recording and analysis system that enabled them to decipher brain signals from owl monkeys in order to control the movement of a robot arm.

The latest work by the Duke researchers is the first to demonstrate that monkeys can learn to use only visual feedback and brain signals, without resort to any muscle movement, to control a mechanical robot arm -- including both reaching and grasping movements.

In their experiments, the researchers first implanted an array of microelectrodes -- each smaller than the diameter of a human hair -- into the frontal and parietal lobes of the brains of two female rhesus macaque monkeys. They implanted 96 electrodes in one animal and 320 in the other. The researchers reported their technology of implanting arrays of hundreds of electrodes and recording from them over long periods in a Sept. 16, 2003, article in the Proceedings of the National Academy of Sciences.

The researchers chose frontal and parietal areas of the brain because they are known to be involved in producing multiple output commands to control complex muscle movement.

The faint signals from the electrode arrays were detected and analyzed by the computer system the researchers had developed to recognize patterns of signals that represented particular movements by an animal's arm.

In the initial behavioral experiments, the researchers recorded and analyzed the output signals from the monkeys' brains as the animals were taught to use a joystick to both position a cursor over a target on a video screen and to grasp the joystick with a specified force.

After the animals' initial training, however, the researchers made the cursor more than a simple display -- now incorporating into its movement the dynamics, such as inertia and momentum, of a robot arm functioning in another room. While the animals' performance initially declined when the robot arm was included in the feedback loop, they quickly learned to allow for these dynamics and became proficient in manipulating the robot-reflecting cursor, found the scientists.

The scientists next removed the joystick, after which the monkeys continued to move their arms in mid-air to manipulate and "grab" the cursor, thus controlling the robot arm.

"The most amazing result, though, was that after only a few days of playing with the robot in this way, the monkey suddenly realized that she didn't need to move her arm at all," said Nicolelis. "Her arm muscles went completely quiet, she kept the arm at her side and she controlled the robot arm using only her brain and visual feedback. Our analyses of the brain signals showed that the animal learned to assimilate the robot arm into her brain as if it was her own arm." Importantly, said Nicolelis, the experiments included both reaching and grasping movements, but derived from the same sets of electrodes.

"We knew that the neurons from which we were recording could encode different kinds of information," said Nicolelis. "But what was a surprise is that the animal can learn to time the activity of the neurons to basically control different types of parameters sequentially. For example, after using a group of neurons to move the robot to a certain point, these same cells would then produce the force output that the animals need to hold an object. None of us had ever encountered an ability like that."

Also importantly, said Nicolelis, analysis of the signals from the animals' brains as they learned revealed that the brain circuitry was actively reorganizing itself to adapt.

"It was extraordinary to see that when we switched the animal from joystick control to brain control, the physiological properties of the brain cells changed immediately. And when we switched the animal back to joystick control the very next day, the properties changed again.

"Such findings tell us that the brain is so amazingly adaptable that it can incorporate an external device into its own 'neuronal space' as a natural extension of the body," said Nicolelis. "Actually, we see this every day, when we use any tool, from a pencil to a car. As we learn to use that tool, we incorporate the properties of that tool into our brain, which makes us proficient in using it." Said Nicolelis, such findings of brain plasticity in mature animals and humans are in sharp contrast to traditional views that only in childhood is the brain plastic enough to allow for such adaptation.

According to Nicolelis, the finding that their brain-machine interface system can work in animals will have direct application to clinical development of neuroprosthetic devices for paralyzed people.

"There is certainly a great deal of science and engineering to be done to develop this technology and to create systems that can be used safely in humans," he said. "However, the results so far lead us to believe that these brain-machine interfaces hold enormous promise for restoring function to paralyzed people."

The researchers are already conducting preliminary studies of human subjects, in which they are performing analysis of brain signals to determine whether those signals correlate with those seen in the animal models. They are also exploring techniques to increase the longevity of the electrodes beyond the two years they have currently achieved in animal studies.

Henriquez and the research team's other biomedical engineers from Duke's Pratt School of Engineering are also working to miniaturize the components, to create wireless interfaces and to develop different grippers, wrists and other mechanical components of a neuroprosthetic device.

And in their animal studies, the scientists are proceeding to add an additional source of feedback to the system -- in the form of a small vibrating device placed on the animal's side that will tell the animal about another property of the robot.

Beyond the promise of neuroprosthetic devices, said Nicolelis, the technology for recording and analyzing signals from large electrode arrays in the brain will offer an unprecedented insight into brain function and plasticity.

"We have learned in our studies that this approach will offer important insights into how the large-scale circuitry of the brain works," he said. "Since we have total control of the system, for example, we can change the properties of the robot arm and watch in real time how the brain adapts."

source: http://news.mc.duke.edu/news/article.php?id=7100 10nov03

Monkeys Control Robot Arm Via Brain Signals

PRESS RELEASE 15nov00

DURHAM, N.C. - Duke University Medical Center researchers and their colleagues have tested a neural system on monkeys that enabled the animals to use their brain signals, as detected by implanted electrodes, to control a robot arm to reach for a piece of food. The scientists even transmitted the brain signals over the Internet, remotely controlling a robot arm 600 miles away.

According to the scientists, their recording and analysis system, in which the electrodes remained implanted for two years in one animal, could form the basis for a brain-machine interface that would allow paralyzed patients to control the movement of prosthetic limbs. Their finding also supports new thinking about how the brain encodes information, by spreading it across large populations of neurons and by rapidly adapting to new circumstances.

In an article in the Nov. 16, 2000, Nature, Miguel Nicolelis, associate professor of neurobiology, and his colleagues described how they tested their system on two owl monkeys - implanting arrays of as many as 96 electrodes, each less than the diameter of a human hair, into the monkeys' brains.

The technique they used, called "multi-neuron population recordings" was developed by co-author John Chapin and Nicolelis. It allows large numbers of single neurons to be recorded separately, and then combines their information using a computer coding algorithm.

The scientists implanted the electrodes in multiple regions of the brain's cortex, including the motor cortex from which movement is controlled. The scientists then recorded the output of these electrodes as the animals learned reaching tasks, including reaching for small pieces of food.

The scientists fed the mass of neural signal data generated during many repetitions of these tasks into a computer, which analyzed the brain signals to determine whether it was possible to predict the trajectory of the monkey's hand from the signals. In this analysis, the scientists used simple mathematical methods to predict hand trajectories in real-time as the monkeys learned to make different types of hand movements.

Said Chapin, who is at the State University of New York Health Science Center, "In a previous paper [published in the July 1, 1999, Nature Neuroscience], we found that rats were able to use their neuronal population activity to control a robot arm, which they used to bring water to their mouths. At the beginning of the experiments, the animals had to press down a lever to generate the brain activity needed to move the robot arm. Over continued training, however, their lever movements diminished while their brain activity remained the same."

Said Nicolelis, "We found two amazing things, both in the earlier rat studies and in our new studies on these primates. One is that the brain signals denoting hand trajectory shows up simultaneously in all the cortical areas we measured. This finding has important implications for the theory of brain coding which holds that information about trajectory is distributed really over large territories in each of these areas - even though the information is slightly different in each area.

"The second remarkable finding is that the functional unit in such processing does not seem to be a single neuron," Nicolelis said. "Even the best single-neuron predictor in our samples still could not perform as well as an analysis of a population of neurons. So, this provides further support to the idea that the brain very likely relies on huge populations of neurons distributed across many areas in a dynamic way to encode behavior."

Once the scientists demonstrated that the computer analysis could reliably predict hand trajectory from brain signal patterns, they then used the brain signals from the monkeys - as processed by the computer - to allow the animals to control a robot arm moving in three dimensions. They even tested whether the signals could be transmitted over a standard Internet connection, controlling a similar arm in MIT's Laboratory for Human and Machine Haptics - informally known as the Touch Lab.

Said co-author Mandayam Srinivasan, director of the MIT laboratory, "When we initially conceived the idea of using monkey brain signals to control a distant robot across the Internet, we were not sure how variable delays in signal transmission would affect the outcome. Even with a standard TCP/IP connection, it worked out beautifully. It was an amazing sight to see the robot in my lab move, knowing that it was being driven by signals from a monkey brain at Duke. It was as if the monkey had a 600-mile-long virtual arm."

Besides Nicolelis, Srinivasan and Chapin, other co-authors of the paper were, from Duke, Johan Wessberg, Christopher Stambaugh, Jerald Kralik, Pamela Beck and Mark Laubach; and from MIT, Jung Kim and James Biggs. The scientists' work is supported by the National Institutes of Health, National Science Foundation, Defense Advanced Research Projects Agency and the Office of Naval Research.

"The reliability of this system and the long-term viability of the electrodes lead us to believe that this paradigm could eventually be used to help paralyzed people restore some motor function," Nicolelis said.

"This system also offers a new paradigm to study basic questions of how the brain encodes information. For example, now that we've used brain signals to control an artificial arm, we can progress to experiments in which we change the properties of the arm or provide visual or tactile feedback to the animal, and explore how the brain adapts to it. Understanding such adaptation will allow us to make inferences about how the brain normally encodes information."

Nicolelis and his colleagues will soon begin such "closed-loop" experiments, in which movement of the robot arm generates tactile feedback signals in the form of pressure on the animals' skin. Also, they are providing visual feedback by allowing the animal to watch the movement of the arm. The scientists' experiments with learning in rats that were reported in Nature last July have already indicated that the analysis system can detect adaptive brain changes associated with learning.

Such feedback studies could also potentially improve the ability of paralyzed people to use such a brain-machine interface to control prosthetic appendages, said Nicolelis. In fact, he said, the brain could prove extraordinarily adept at using feedback to adapt to such an artificial appendage.

"One most provocative, and controversial, question is whether the brain can actually incorporate a machine as part of its representation of the body," he said. "I truly believe that it is possible. The brain is continuously learning and adapting, and previous studies have shown that the body representation in the brain is dynamic. So, if you created a closed feedback loop in which the brain controls a device and the device provides feedback to the brain, I would predict that as people or animals learn to use the device, their brains will basically dedicate neuronal space to represent that device.

"If such incorporation of artificial devices works, it would quite likely be possible to augment our bodies in virtual space in ways that we never thought possible," Nicolelis said. "For example, in our modest experiment at using brain wave patterns to control the robot arm over the Internet, if we extended the capabilities of the arm by engineering in feedback - such as visual, force or texture - such closed-loop control might result in the remote arm being incorporated into the body's representation in the brain. Once you establish a closed loop, you're basically telling the brain that the external device is part of the body representation. The major question in my mind now is what is the limit of such incorporation."

Besides experimenting with such feedback systems, Nicolelis and his colleagues are planning to increase the number of implanted electrodes, with the aim of achieving 1,000-electrode arrays. They are also developing a "neurochip" that will greatly reduce the size of the circuitry required for sampling and analysis of brain signals.

"We envision that this neurochip can become an essential component of the type of hybrid-brain-machine interfaces that may one day be used to restore motor function in paralyzed patients," said Nicolelis. "These activities will serve as the backbone of a new Center for Neural Analysis and Engineering currently being created at Duke."

source: http://dukemednews.org/news/article.php?id=69 10nnov03