Abstract:BCI: brain–computer interface DCN: dorsal column nuclei ICMS: intracortical microstimulation LED: light-emitting diode PDMS: polydimethylsiloxane RF: radiofrequency The dexterous hand is a defining feature of human existence. Evolved over tens of millions of years, modern humans are able to perform remarkable tasks with their hands. From typing hundreds of words per minute to playing Rachmaninoff's Piano Concerto No. 2, the dexterous hand defines us. Unfortunately, a number of maladies rob us of this defining human characteristic. In the most extreme case, paralyzed individuals lose communication between the brain and the periphery. This condition affects an estimated 5.4 million people, or 2% of the US population.1 At present, no effective treatment restores function to these individuals. Regaining hand function is a principal concern for paralyzed patients. Toward this aim, significant advances in motor—or efferent—brain–computer interface (BCI) systems have occurred in recent years. Efferent BCI systems extract movement-relevant information from electrocorticography (ECoG) or electroencephalography (EEG). These analogue signals are transformed into control commands to drive robotic arms2 or evoke muscle contractions in paralyzed limbs.3-8 In the later example, compound wrist flexion may be evoked by brain-controlled functional electrical stimulation of forearm flexors. Planned clinical trials aim to capitalize upon these scientific advances to test efferent BCI across a range of conditions and control routines. While these proof-of-principal systems are encouraging, a number of substantial hurdles remain. Perhaps the most pressing barrier to restoring dexterous hand movements is the lack of systems to restore somatosensory feedback. Even in the presence of intact descending motor systems, precise hand movements are abolished when somatosensation is missing.9-16 Indeed, the majority of efferent BCI systems currently in testing rely solely upon visual guidance. This constraint is unnatural and unlikely to be useful if deployed clinically. Visual guidance requires constant vigilance and introduces substantial time-lags to error correct each movement. To restore naturalistic movements, bi-directional BCI systems that link movements and real-time sensory feedback must be developed. The feedback loop of bi-directional BCI is closed with sensory feedback. Unfortunately, the field of sensory—or afferent—brain–computer interface has not kept pace with the maturation of efferent systems. This is due, in part, to the challenges concerning sensory research in animals. Sensory perception is a uniquely subjective experience that does not lend itself readily to the quantitative metrics. For decades, experimentalists have attempted to characterize the perceptual experiences associated with stimulation of the sensory cortices, including primary somatosensory cortex (S1), secondary somatosensory cortex (S2), and parietal association areas in animal models. From this body of literature, we know that intracortical microstimulation (ICMS) of S1 yields sufficient percepts to permit limited binary decisions, such as differentiating between 2 stimulation frequencies or amplitudes.17-21 Despite exhaustive investigation, no study has convincingly reproduced the complex sensory phenomena that are fundamental to our routine encounters with the physical world. Compounding the problem, very limited human data are available to assess the efficacy of S1 stimulation. Animal studies do not answer the question of how stimulation feels. To answer these qualitative questions, we need human data. Most human data have been obtained during brief testing sessions in awake craniotomies or during stimulation in patients with implanted ECoG electrodes.22-24 Invariably, these patients reported that S1 stimulation yielded only vague ‘tingling’ sensations with modest regional localization. Flesher and colleagues recently reported the first human data using ICMS encoding in S1 with chronic penetrating arrays.25 In this experiment, a 28-yr-old male with a spinal cord injury underwent implantation of 2 32-channel multi-electrode arrays into primary somatosensory cortex (S1). Over the course of several months, the investigators mapped perceptual responses to ICMS up to 100 μA. The majority of responses (93%) were categorized as ‘possibly natural,’ ‘pressure’ sensations. The perceptual intensity was modulated by stimulation amplitude with increased pressure corresponding to increase stimulus amplitude. This finding mirrors that of ICMS in primary visual cortex where phosphine brightness is modulated by stimulus amplitude.26 These data constitute a substantial step toward clinical sensory BCI. However, there were a number of findings that tempered enthusiasm for immediately clinical implementation. For instance, none of the S1 electrodes activated sensory representations of the distal fingers where feedback is most needed. Instead, the majority of responses were localized to the palmar crease region of the hand proximal to the fingers. Also, the detection thresholds of a many electrode sites rose significantly over the short course of the study, raising the concern that the effect of S1 encoding will fade over time. Finally, few of the stimuli evoked properly ‘naturalistic’ percepts. These limitations and the disappointing results from similar work in visual cortex raise the question of whether cortical ICMS encoding is the optimal solution for sensory restoration. These unanswered questions motivate our research program. Our work aims to bridge the divide between current state-of-the-art and the clinical needs of our patients. Our overarching strategy is to develop closed-loop, autonomous bidirectional brain–machine interface systems. These systems, as conceived, provide real-time communication between the brain and body. Because the field of efferent BCI has vastly outpaced that of afferent BCI, our work primarily focuses on developing sensory-brain interfaces to couple with existing BCIs (see Bouton et al27 for example). Our strategy focuses on 3 critical intersections of engineering and neuroscience. The first is development of a suite of sensors that serve as mechanoreceptors for the paralyzed, insensate hand. The second is development of a chronic neural interface for artificial sensory encoding. The third is a body area network that links peripheral sensors with novel neural interfaces. The integration of these components is illustrated in Figure 1.FIGURE 1: Body area network. Fully integrated system with implantable force and flex sensors (1, 2), wearable analyzer (3), electrogoniometer (4), and neural interface (5).In this brief overview, we outline our approach, preliminary data, and future directions. This work collectively represents a fruitful collaboration between neurosurgery and electrical engineering. We are grateful to the National Science Foundation for funding our work. RESEARCH APPROACH Our research strategy follows 3 central aims: development of novel sensors, characterization of novel neural interfaces, and development of an autonomous body-area network. Novel Sensors Hand somatosensation can be characterized by a multidimensional space with axes defined by sensory modality (eg, light touch, proprioception), somatotopy, temporal dynamics, the influence of descending central inputs, and brain state. Restoring native somatosensation is perhaps too lofty a goal for a first-generation sensor–brain interface. Instead, we reduce the dimensionality of the problem to a single sensory modality at a single somatotopic location. We have developed a number of force sensors and a proprioceptive sensor as our first aim. The design of our force sensors is constrained by the form and function of the human hand. Relevant design features include: sensor sensitivity, range, power, form-factor, and complexity. Sensitivity is defined as a sensor's accuracy to convert mechanical force into voltage changes on the sensor. Dynamic range captures the extremes of mechanical force spanning interactions between the hand and the physical environment. The feature of power concerns both the requirements of the sensor (active or passive) as well as the sensor's efficiency to convert physical energy into electrical energy. For wireless sensors, the power feature also includes power harvesting and wireless transmission of data. Form-factor is defined as the mechanical properties of the sensor (size, shape) as well as the flexibility and elasticity of the substrate. Finally, the complexity of the sensor constrains fabrication and durability. These competing design constraints inevitably require engineering trade-offs. In the interest of brevity, we focus on 2 prototype force sensors and a proprioceptive sensor to illustrate these engineering trade-offs in the context of sensor–brain interface. First we consider scattering force sensors and optical force sensors before moving toward proprioceptive electrogoniometers. Scattering force sensors operate under the principle of radiofrequency (RF) back scatter. RF identification is a common technique used to track tags, like those attached to garments at a department store to prevent theft or those implanted subdermally in house pets to identify them when they are lost. The central concept is that RF energy polarizes conductive elements, such as the linear segments of an antenna, and scatter energy back in a measurable way. Deformations of the segment length or shape cause a shift in the back-scatter pattern as the polarization of each segment is related to its orientation in a pulsed electromagnetic field. By calibrating the back-scatter patterns induced by force-induced deformations of RF antenna segments, one may indirectly measure forces applied to a flexible antenna implanted under the skin. In the first series of experiments, our group characterized the back-scatter signatures of a number of antenna designs serving as passive sensor nodes. An advantage of passive sensors is that they do not require active power supplies. Therefore, flexible antennas can be implanted under the skin without the need of wires or batteries. Initial antennas were made with copper tape for rapid prototyping. Antenna shapes were constructed into space filling curves (eg, Hilbert, Peano curves) that varied in the number and length of conductive segments (Figure 2). Changes in size and shape of copper RF antennas were associated with reproducible batter scatter properties.FIGURE 2: Passive scattering force sensor design. A, Antenna shapes with different linear segments in second order Hilbert and Peano curves. B and C, Polarization of antenna segments within electromagnetic field. D, Radiofrequency response curves as a function of area of RF tag (left), and shifts in curves with ±2% change in area (right). E, Prototype indium–gallium tags in PDMS substrate. Central reservoir visible in series with antenna segments. F, RF tuning curve of indium–gallium tags in response to forces applied to central reservoir. Rapid shift noted in low end of force axes indicates appropriate sensitivity for precise finger grip.To build force sensitivity, our second series of experiments examined the flexibility of antennas across a range of forces routinely encountered by the human hand. Liquid metal indium–gallium antennas were designed within a flexible, skin-like polydimethylsiloxane (PDMS) substrate. Indium–gallium is a highly conductive eutectic alloy whose melting point is sufficiently low (∼ –2°F) to allow the alloy to remain in liquid phase at room temperature. Channels were laser-etched into the PDMS in the shape of space filling curves to house the alloy (Figure 2). Force sensitivity was amplified by creating a central compressible metal reservoir in series with the channels. When force was applied, the liquid metal filled the channel segments proportionally. As each successful segment of the antenna was filled with conductive metal, the RF back-scatter properties shifted (Figure 2). As can be seen in the RF response curve, the antenna was sufficiently sensitive to capture force changes within 5 N of fingertip pressure, appropriate for precision grip activities. These experiments verified the feasibility of force sensing RF tags. However, limitations to this technique include the need for sensitive detecting antennas to measure back scatter. For this reason, we examined force sensor designs that were independent of RF signal. Optical force sensing is a method to detect fingertip pressure without electromagnetic interference. An optical force sensor layers PDMS membrane on SiO2 within an implantable chip (Figure 3) that could be implanted subdermally. At one end of the floor of the sensor, an internal 80 μm2 light-emitting diode (LED) emits light. The light is reflected by the internal ceiling of the chip that is constructed of PDMS in an inverse lenticular structure. Reflected light is detected by a photodiode at the opposite end of the sensor. The intervening SiO2 acts as an optical waveguide. In the absence of force (or compressing pressure), the waveguide allows reflected light to excite the photodiode with an efficient electric-to-optical conversion, a high sensitivity (0.02 kPa−1) and a pressure sensing resolution (38 mPa). When force is applied, the PDMS ceiling bows downward, opening light channels in the membrane. This allows light to escape, which in turn decreases the voltage at the photodiode monotonically, and yields a scaled readout.FIGURE 3: Optical force sensor design. A, Side view of optical sensor in absence of load. Directional path of light shown in yellow reflected from internal surface of PDMS ceiling. LED emitter located in lower left of sensor; photodiode (PD) located in lower right. B, Applied forces reduce light received by photodiode end. C, Diagram of optical force sensor circuit. D, Idealized relationship between applied force and photodiode voltageBoth scatter sensors and optical sensors achieved their desired engineering goals of converting force into measurable data. Neither system represented optimal solutions. In the case of scatter sensors, environmental noise may obscure the back-scatter energy detected by a horn antenna. In the case of optical sensors, an active circuit is required. On-going experiments aim to address these limitations by increasing the signal-to-noise ratio (RF sensors) and integrating rechargeable power (optical sensors). Beyond touch sensation, proprioception is a fundamental sensory modality that informs us about limb position. To restore proprioception across large joints, we developed a wireless electrogoniometer.28 Unlike other electrogoniometers that require strain gauges or power-hungry potentiometers, our system was designed to have very low power requirements (∼20 μW) both in terms of sensing and wireless data transmission. This was achieved using a pair of impulse-radio ultrawide band wireless smart sensor nodes interfacing with low-power 3-axis accelerometers through event-driven analog-to-digital converters. Electrogoniometers are designed to operate across large joints, such as the elbow, which are too large for strain sensors or other position sensors. On-going experiments aim to combine multiple sensor modalities in the same organism. Novel Central Nervous System Targets Our second aim is to identify optimal sensory encoding nodes along the neuraxis. Cortical encoding has been attempted for decades in animals, and recently in humans, with mixed results. It remains to be seen how well S1 ICMS will faithfully reproduce naturalistic perception. ICMS in other sensory areas, like primary visual cortex, generates phosphenes but not complex visual images.26 This may be due to the fact that cortical representations are distributed. Complex experiential phenomena, like rich somatosensory percepts, are therefore unlikely to be reproduced with focal stimulation without activation of a larger network. Upstream sensory circuits have remained largely unexplored. To begin to fill this knowledge gap, we developed the first chronic neural interface of the dorsal column nuclei (DCN) to enable recording and stimulation in awake behaving animals.29,30 The DCN represent a logical target for sensory encoding. These compact nuclei on the dorsal surface of the brainstem receive somatotopically organized tactile and proprioceptive signals directly from primary afferents (Figure 4).31,32 Terminals from the DCN relay high fidelity ascending information to the thalamus33 for further sensory integration. And, the DCN receive descending input from sensorimotor cortex34 that may modulate sensory gating.FIGURE 4: Dorsal column nuclei interface. A, Relationship between gracile and cuneate nuclei and obex (white arrowhead) in human. Both are readily accessible with gentle elevation of the cerebellar tonsil. B, Chronic multielectrode array implanted in the DCN of a macaque at time of formalin fixation. Array remained safely in place for several months. C, Brainstem cross section of DCN showing termination of electrode tip in proximity to cuneate nucleus (black arrow).Prior studies of the mammalian DCN were limited to sedated or terminal acute experiments. In our first set of in Vivo studies, macaque DCN were implanted with floating multi-electrode arrays to explore the feasibility of a chronic brainstem interface. Over several months, we demonstrated that these arrays are safe and well tolerated in unrestrained animals without adverse effects. First-ever data from chronically implanted primate DCN yielded a number of interesting observations. Over 300 units were recorded. The most important observation was that tactile receptive fields remained stable over time, confirming that DCN are somatotopically organized. Spontaneous spike frequencies occurred with a dominant alpha rhythm (8-14 Hz) in the >300 units. Using spike-triggered fields, we confirmed that individual units could be held over multiple days (and even weeks in some cases) with chronic brainstem arrays. Encouraged by the results of brainstem recording, we designed a series of stimulation experiments. In subsequent experiments, we performed the first-ever attempts at sensory encoding through DCN ICMS. Encoding at DCN effectively activates downstream networks in a highly precise manner. Single pulse stimulation of the cuneate nucleus evoked layer-specific responses in primary sensory cortex. DCN stimulation evoked lasting rhythmic unit and field activity in the S1 granular layer (Figure 5), which is known to receive efferents from thalamic sensory nuclei. The induced rhythm persisted for up to 800 ms. This intriguing finding may explain the mechanism of why perceptual experiences outlast primary mechanoreceptor activations. If replicated, it suggests that circuits between DCN and S1 granular layer have a gain function for fleeting sensory volleys. To test perceptual thresholds of DCN stimulation, animals were trained on a vibrotactile detection and discrimination task. When vibrotactile stimuli were replaced with DCN microstimuli, animals learned to detect the electrical stimuli over approximately 10 sessions.35 Detection probabilities then rose to 75% above chance. Detection thresholds for DCN are comparable to cortical thresholds (∼30-40 μA). This verification that DCN encoding effectively activates downstream networks justifies behavioral experiments to characterize the efficacy of DCN evoked perception.FIGURE 5: Cortical responses to cuneate encoding. A, Layer specific evoked responses to single-pulse DCN stimulation. Layer 4/5 corresponds to the granular cell layer, which receives ascending thalamocortical terminals. B, Spectrogram of frequencies (vertical axis) following single-pulse DCN stimulation. Following the initial stimulus artifact at time 0, a lasting alpha rhythm persists well beyond the fleeting stimulus (white arrow).The feasibility of chronic, safe DCN recording and encoding enables hypothesis testing not previously possible. Data from these experiments have broad implications for somatosensory research. Experiments currently underway will further characterize DCN responses and their relation to downstream nodes including the thalamus and sensory cortices. Novel Bidirectional BCI Systems Novel hardware systems are required to link peripheral sensor nodes and sensory encoding electrodes. We developed a multipurpose bidirectional brain–machine interface, termed the PennBMBI,36-40 as our third aim. This system, when fully implemented, wirelessly links a suite of implantable and wearable peripheral sensor nodes with neural recording and stimulating electrodes (Figure 1). Collectively, the system and its linked nodes are referred to as a body area network. At the core of the PennBMBI are 4 battery-powered wireless devices, including a neural signal analyzer, a neural stimulator, a sensor node, and a graphic user interface.38 The system-on-chip consists of a 16-channel neural recording front-end, neural feature extraction units, 16-channel programmable neural stimulator back-ends, closed-loop controllers, and associated circuits. The neural feature extraction units are tunable for unit discrimination or field detection. Specifically, the system includes an ultralow-power neural energy extraction unit enabling 64-step natural logarithmic domain frequency tuning, and a current-mode action potential detection unit with time-amplitude window discriminator. A control policy is implemented on-chip in the form of a proportional-integral-derivative controller that maps sensor data from peripheral sensors to desired patterns related to somatosensory cortex activity (Figure 6).37FIGURE 6: Block diagram of closed-loop brain–computer interface applications. A, Block diagrams illustrating intersections between BCI systems and body, in the case of paralyzed limb, or prosthetic limb. Proportional integral derivative controller permits feedback control from nodes within block diagram (shown in red). B, Illustration schematically demonstrating control loop integrating neural recording and stimulation in separate regions.A multimode stimulator affords the flexibility to activate paralyzed muscles or depolarize neural elements. Stimulation parameters may be programmed to generate monopolar or bipolar, symmetrical or asymmetrical charge balanced stimulation with a compliance voltage of ± 12 V.36 Our second-generation stimulator addresses the ubiquitous safety issues endemic to high-duty-cycle current matching stimulator designs—namely the irreversible charge transfer that creates destructive chemical species that injure neural tissue and damage electrodes. Traditional current matching stimulators employ biphasic current stimulation with a reversal charge meant to balance a stimulating phase that depolarizes neural elements. However, even well-designed stimulator output stages accumulate residual changes due to unrecoverable charge diffusion during the stimulating phase. Over millions of duty cycles, oxidative free radicals develop and set in motion oxidative reactions that degrade the interface and damage tissue. To properly account for this charge diffusion, we implement a feedback controller that terminates the reversal phase when a net-zero charge point is detected (Figure 7).41 The essential architecture of this circuit is an error compensation comparator that minimizes charge error by measuring the residue charges during prior stimulus pulses. If the residue charges are outside the safe range, subsequent stimulation pulses are recalibrated as error correction. In-Vitro tests demonstrated that this method prevents charge accumulation over long durations, suggesting that the system will have improved safety in Vivo. Additional experiments are underway to test this hypothesis.FIGURE 7: Guiding principle behind net-zero charge stimulator. A, Current-time (top) and charge-time (bottom) plots of traditional charge-balanced stimulators. Traditional matching between stimulating phase and recovery phase shown (red arrows). Idealized theory supporting charge-balancing shown in green tracing (‘theory’), while actual charge dissipation shown in blue (‘real’). B, Net-zero charge stimulator modulates the recovery charge duration based on measurements of actual zero point crossing. No residual charge build-up (red circle).The body area network requires real-time communication between the components. Communication is rendered wirelessly with an impulse-radio ultra-wide band transceiver. The transmitter and receiver use commercial components to transmit and recover signal between components. For clinical implementation, communication between nodes must be robust and operate within an acceptable tolerance of data fidelity. The PennBMBI features an on-the-air data rate of 2 Mbps in 2.4 GHz industrial, scientific, medical band. The measured bit error rate was 10−3 over a distance of 3 m. Both of these measures are well within the desired specifications for human use. Before moving toward human testing, a number of assurances must be satisfied. Among them, safe operation of the system must be demonstrated in Vivo. Efficacy must also be demonstrated to compel further preclinical investigation. To establish whether closed-loop operation of the system was safe and effective at providing intelligible percepts, we designed in Vivo experiments based on the classical psychophysical apparatus, the Morris water maze. Rodents were trained to seek out a submerged hidden platform to escape the water bath. Unaided by visual cues, rodents utilized a random swim pattern until the platform is encountered. Rodents were then conditioned to associate ICMS driven by the PennBMBI as a homing beacon centered on the hidden platform (Figure 8). As the animal approached the platform, stimulation frequency increased. As expected, maze performance significantly improved in the presence of augmented perception. When the PennBMBI was turned off, during catch trials, performance was random. No adverse events occurred. Thus, animals are able to use closed-loop cyber-physical systems to augment perception in a safe and effective manner.FIGURE 8: In vivo feasibility testing of novel systems. A, Rodent wearing PennBMBI system. B, Experimental design wherein rodent is guided to submerged platform in water maze by ICMS acting as homing beacon. Idealized swim paths illustrated from above maze with PennBMBI ‘off’ and ‘on’ illustrating optimal swim trajectory in presence of augmented perception.CONCLUSION In summary, our strategy to develop a sensor–brain interface system focuses on 3 aims: development of novel sensors, characterization of novel neural interface, and development of autonomous body-area network. We have made progress in each of these areas, but substantial work remains. We continue to scale our systems up to maximize channel count, optimize peripheral sensors and refine our circuits. In parallel, we subject our systems to increasingly rigorous animal models. It is our long-term goal to integrate this sensor brain interface with existing efferent systems to yield a bidirectional BCI to reanimate paralyzed limbs. Disclosures This work was sponsored in part by the National Science Foundation, grant NSF CBET 1404041. The authors have no personal, financial, or institutional interest in any of the drugs, materials, or devices described in this article.

Strategies for Autonomous Sensor–Brain Interfaces for Closed-Loop Sensory Reanimation of Paralyzed Limbs

A Fully Integrated Sensor-Brain-Machine Interface System for Restoring Somatosensation

2327 Decoding/encoding somatosensation from the hand area of the human primary somatosensory (S1) cortex for a closed-loop motor/sensory brain-machine interface (BMI)

Restoring tactile and proprioceptive sensation through a brain interface

Reconnecting the Hand and Arm to the Brain: Efficacy of Neural Interfaces for Sensorimotor Restoration After Tetraplegia

Synergetic Innovation of Cranial Nerves and Biomedical Engineering: Promoting the Frontier of Brain Science

Real-time Decoding Algorithm in Brain Machine Interfaces

Biohybrid Robotic Hand to Investigate Tactile Encoding and Sensorimotor Integration

A Fully Integrated Wireless Sensor-Brain Interface System to Restore Finger Sensation.

Development and evaluation of a non-invasive brain-spine interface using transcutaneous spinal cord stimulation

A distributed, high-channel-count, implanted bidirectional system for restoration of somatosensation and myoelectric control

Towards machine to brain interfaces: sensory stimulation enhances sensorimotor dynamic functional connectivity in upper limb amputees

Broadband Prosthetic Interfaces: Combining Nerve Transfers and Implantable Multichannel EMG Technology to Decode Spinal Motor Neuron Activity

Biomimetic approaches to bionic touch through a peripheral nerve interface

Long-Term Home-Use of Sensory-Motor-Integrated Bidirectional Bionic Prosthetic Arms Promotes Functional, Perceptual, and Cognitive Changes

Sensing and decoding the neural drive to paralyzed muscles during attempted movements of a person with tetraplegia using a sleeve array

Editorial: Brain-computer interfaces in neurological disorders: expanding horizons for diagnosis, treatment, and rehabilitation

Restoring the sense of touch with a prosthetic hand through a brain interface

A somatotopic bidirectional hand prosthesis with transcutaneous electrical nerve stimulation based sensory feedback

Human Perception of Electrical Stimulation on the Surface of Somatosensory Cortex

A Brain to Spine Interface for Transferring Artificial Sensory Information