Brain-computer interface research : a state-of-the-art summary 6 978-3-319-64373-1, 3319643738, 978-3-319-64372-4

Table of contents :
Front Matter ....Pages i-vi
Introduction (Christoph Guger, Brendan Z. Allison, Mikhail A. Lebedev)....Pages 1-8
Advances in BCI: A Neural Bypass Technology to Reconnect the Brain to the Body (Gaurav Sharma, Nicholas Annetta, David A. Friedenberg, Marcia Bockbrader)....Pages 9-20
Precise and Reliable Activation of Cortex with Micro-coils (Seung Woo Lee, Shelley I. Fried)....Pages 21-33
Re(con)volution: Accurate Response Prediction for Broad-Band Evoked Potentials-Based Brain Computer Interfaces (J. Thielen, P. Marsman, J. Farquhar, P. Desain)....Pages 35-42
Intracortical Microstimulation as a Feedback Source for Brain-Computer Interface Users (Sharlene Flesher, John Downey, Jennifer Collinger, Stephen Foldes, Jeffrey Weiss, Elizabeth Tyler-Kabara et al.)....Pages 43-54
A Minimally Invasive Endovascular Stent-Electrode Array for Chronic Recordings of Cortical Neural Activity (Thomas J. Oxley, Nicholas L. Opie, Sam E. John, Gil S. Rind, Stephen M. Ronayne, Anthony N. Burkitt et al.)....Pages 55-63
Visual Cue-Guided Rat Cyborg (Yueming Wang, Minlong Lu, Zhaohui Wu, Xiaoxiang Zheng, Gang Pan)....Pages 65-78
Predicting Motor Intentions with Closed-Loop Brain-Computer Interfaces (Matthias Schultze-Kraft, Mario Neumann, Martin Lundfall, Patrick Wagner, Daniel Birman, John-Dylan Haynes et al.)....Pages 79-90
Towards Online Functional Brain Mapping and Monitoring During Awake Craniotomy Surgery Using ECoG-Based Brain-Surgeon Interface (BSI) (L. Yao, T. Xie, Z. Wu, X. Sheng, D. Zhang, N. Jiang et al.)....Pages 91-96
A Sixteen-Command and 40 Hz Carrier Frequency Code-Modulated Visual Evoked Potential BCI (Daiki Aminaka, Tomasz M. Rutkowski)....Pages 97-104
Trends in BCI Research I: Brain-Computer Interfaces for Assessment of Patients with Locked-in Syndrome or Disorders of Consciousness (Christoph Guger, Damien Coyle, Donatella Mattia, Marzia De Lucia, Leigh Hochberg, Brian L. Edlow et al.)....Pages 105-125
Recent Advances in Brain-Computer Interface Research—A Summary of the BCI Award 2016 and BCI Research Trends (Christoph Guger, Brendan Z. Allison, Mikhail A. Lebedev)....Pages 127-134

Citation preview

SPRINGER BRIEFS IN ELEC TRIC AL AND COMPUTER ENGINEERING

Christoph Guger Brendan Allison Mikhail Lebedev Editors

Brain-Computer Interface Research A State-of-the-Art Summary 6 123

SpringerBriefs in Electrical and Computer Engineering Series editors Woon-Seng Gan, Nanyang Technological University, Singapore, Singapore C.-C. Jay Kuo, University of Southern California, Los Angeles, CA, USA Thomas Fang Zheng, Tsinghua University, Beijing, China Mauro Barni, University of Siena, Siena, Italy

More information about this series at http://www.springer.com/series/10059

Christoph Guger Brendan Allison Mikhail Lebedev •

Editors

Brain-Computer Interface Research A State-of-the-Art Summary 6

123

Editors Christoph Guger g.tec Guger Technologies OG Schiedlberg Austria

Mikhail Lebedev Department of Neurobiology Duke University Durham, NC USA

Brendan Allison g.tec Guger Technologies OG Schiedlberg Austria

ISSN 2191-8112 ISSN 2191-8120 (electronic) SpringerBriefs in Electrical and Computer Engineering ISBN 978-3-319-64372-4 ISBN 978-3-319-64373-1 (eBook) DOI 10.1007/978-3-319-64373-1 Library of Congress Control Number: 2017938537 © The Author(s) 2017 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Printed on acid-free paper This Springer imprint is published by Springer Nature The registered company is Springer International Publishing AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christoph Guger, Brendan Z. Allison and Mikhail A. Lebedev Advances in BCI: A Neural Bypass Technology to Reconnect the Brain to the Body . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gaurav Sharma, Nicholas Annetta, David A. Friedenberg and Marcia Bockbrader Precise and Reliable Activation of Cortex with Micro-coils . . . . . . . . . . . Seung Woo Lee and Shelley I. Fried Re(con)volution: Accurate Response Prediction for Broad-Band Evoked Potentials-Based Brain Computer Interfaces . . . . . . . . . . . . . . . . J. Thielen, P. Marsman, J. Farquhar and P. Desain Intracortical Microstimulation as a Feedback Source for Brain-Computer Interface Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sharlene Flesher, John Downey, Jennifer Collinger, Stephen Foldes, Jeffrey Weiss, Elizabeth Tyler-Kabara, Sliman Bensmaia, Andrew Schwartz, Michael Boninger and Robert Gaunt A Minimally Invasive Endovascular Stent-Electrode Array for Chronic Recordings of Cortical Neural Activity . . . . . . . . . . . . . . . . . Thomas J. Oxley, Nicholas L. Opie, Sam E. John, Gil S. Rind, Stephen M. Ronayne, Anthony N. Burkitt, David B. Grayden, Clive N. May and Terence J. O’Brien Visual Cue-Guided Rat Cyborg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yueming Wang, Minlong Lu, Zhaohui Wu, Xiaoxiang Zheng and Gang Pan

1

9

21

35

43

55

65

v

vi

Contents

Predicting Motor Intentions with Closed-Loop Brain-Computer Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matthias Schultze-Kraft, Mario Neumann, Martin Lundfall, Patrick Wagner, Daniel Birman, John-Dylan Haynes and Benjamin Blankertz

79

Towards Online Functional Brain Mapping and Monitoring During Awake Craniotomy Surgery Using ECoG-Based Brain-Surgeon Interface (BSI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . L. Yao, T. Xie, Z. Wu, X. Sheng, D. Zhang, N. Jiang, C. Lin, F. Negro, L. Chen, N. Mrachacz-Kersting, X. Zhu and D. Farina

91

A Sixteen-Command and 40 Hz Carrier Frequency Code-Modulated Visual Evoked Potential BCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Daiki Aminaka and Tomasz M. Rutkowski

97

Trends in BCI Research I: Brain-Computer Interfaces for Assessment of Patients with Locked-in Syndrome or Disorders of Consciousness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Christoph Guger, Damien Coyle, Donatella Mattia, Marzia De Lucia, Leigh Hochberg, Brian L. Edlow, Betts Peters, Brandon Eddy, Chang S. Nam, Quentin Noirhomme, Brendan Z. Allison and Jitka Annen Recent Advances in Brain-Computer Interface Research—A Summary of the BCI Award 2016 and BCI Research Trends . . . . . . . . . 127 Christoph Guger, Brendan Z. Allison and Mikhail A. Lebedev

Introduction Christoph Guger, Brendan Z. Allison and Mikhail A. Lebedev

1 What Is a BCI? Brain-computer interfaces (BCIs) are devices that directly read brain activity and use it in a real-time, closed loop system with feedback to the user. Unlike all other interfaces, BCIs do not require movement. Instead, the information from the brain is translated into messages or commands without relying on the body’s natural output pathways. Thus, BCIs can be very helpful to people with severe motor disabilities that prevent them from speaking or using most (or even all) other devices for communication. BCI research has continued to provide new ways to help these types of patients. In the last several years, BCIs have also broadened far beyond communication and control devices for severely paralyzed users. Today, BCIs are rapidly gaining attention for people with a wide variety of other conditions. Major entities and individuals such as Facebook and Elon Musk have recently announced extremely ambitious BCI-related projects. These research activities and announcements could lead to new ways to help many more people, and new hope for future developments. On the other hand, announcements of overly ambitious and unrealistic goals could lead to false hope and sour public opinion. It is certainly a dynamic and eventful time for the BCI research community.

C. Guger (&) Schiedlberg, Austria e-mail: [email protected] B.Z. Allison San Diego, USA M.A. Lebedev Durham, USA © The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_1

1

2

C. Guger et al.

2 The Annual BCI-Research Award G.TEC is a leading provider of hardware, software, and complete systems for BCI research and related directions. G.TEC is headquartered in Austria and with branches in Spain and the USA. In 2010, G.TEC decided to create an Annual BCI-Research Award to recognize and study top new BCI projects. The competition is open to any BCI group worldwide. There is no limitation or special consideration for the type of hardware and software used in the submission. Since the first award in 2010, we have followed more or less the same process: • G.TEC selects a Chairperson of the Jury from a well-known BCI research institute. • This Chairperson forms a jury of top BCI researchers who can judge the Award submissions. • G.TEC publishes information about the BCI Award for that year, including submission instructions, scoring criteria, and a deadline. • The jury reviews the submissions and scores each one across several criteria. The jury then determines twelve nominees and one winner. • The nominees are announced online, asked to contribute a chapter to this annual book series, and invited to a Gala Award Ceremony that is attached to a major conference (such as an International BCI Meeting or Conference). • At this Gala Award Ceremony, the twelve nominees each receive a certificate, and the winner is announced. The winner earns $3000 USD and the prestigious trophy. The 2nd place winner gets $2000 USD and the 3rd place gets $1000 USD. We have made some changes over the years, such as increasing the number of nominees from ten to twelve and adding second and third place awards. Otherwise, the overall process has not changed. The 2016 jury was: Mikhail A. Lebedev (chair of the jury 2016), Alexander Kaplan, Klaus-Robert Müller, Ayse Gündüz, Kyousuke Kamada, Guy Hotson. Consistent with tradition, the jury included the winner from the preceding year (Guy Hotson). The chair of the jury, Dr. Mikhail A. Lebedev, is a top figure in BCI research and leads the prestigious BCI lab at Duke University, USA. Dr. Mikhail Lebedev said: “I was very fortunate to work with the 2016 jury. All of the jury members that I approached chose to join the jury, and we had an outstanding team”. How does the jury decide the nominees and winners? We have used the same scoring criteria across different years. These are the criteria that each jury uses to score the submissions. Earning a nomination (let alone an award) is very

Introduction

3

challenging, given the number of submissions and the very high quality of many of them. Submissions need to score well on several of these criteria: • • • • • •

Does the project include a novel application of the BCI? Is there any new methodological approach used compared to earlier projects? Is there any new benefit for potential users of a BCI? Is there any improvement in terms of speed of the system (e.g. bit/min)? Is there any improvement in terms of accuracy of the system? Does the project include any results obtained from real patients or other potential users? • Is the used approach working online/in real-time? • Is there any improvement in terms of usability? • Does the project include any novel hardware or software developments?

3 The BCI Book Series The annual BCI Book Series is another way that we recognize and study the top BCI projects over time. Each year, the nominees are invited to contribute a chapter. The authors have considerable flexibility in their chapters. Aside from their nominated work, authors might present even newer achievements, work from related groups, or future directions and challenges. In addition to the work that was nominated, authors may also present related material, such as new work since their submission. We have also had some flexibility across different years, such as including chapters from “honorable mention” submissions that were not nominated but had new improvements since their submission (Figs. 1 and 2). In addition to providing the authors with flexibility, we also asked them to present material in a relatively readable format. While chapters present advanced work, the authors and editors have worked to explain some underlying concepts and why the work is important. The chapters include numerous color figures to help illustrate the authors’ ideas and results. Thus, we hope that the chapters herein are of interest not only to experts in different fields, but also to non-experts. For example, chapters might be useful for students who are enrolled in a relevant course or are considering a new research or career direction. Each book also contains an introduction and conclusion. Across different years, we have used the submissions, nominees, and winners to study trends and issues within BCI research. These chapters have already led to some conclusions about what has and hasn’t changed. For example, the types of imaging approaches that are described in submitted and nominated projects has been fairly consistent over the years. EEG-based approaches are prevalent, while intracranial methods including ECoG and depth electrodes are fairly well represented, and other approaches such as fMRI and fNIRS are relatively less common. Most submissions, nominees, and winners have come from the USA and Europe, with some submissions from Japan and China, and many projects that span different groups.

4

C. Guger et al.

Fig. 1 This picture shows the nominees at the BCI Award 2016 ceremony. Tomek Rutkowski, Eberhard Fetz, Jaime Pereira, Benjamin Blankertz, Jordy Thielen, Shelley I. Fried, Lin Yao, Sharlene Flesher, Gaurav Sharma, Kyousuke Kamada (jury), and Christoph Guger (organizer)

Fig. 2 Christoph Guger (organizer), Sharlene Flesher (nomination), and Kyousuke Kamada (jury)

Introduction

5

What has changed? The types of applications and patient groups have broadened considerably over the years. In 2010, projects were relatively focused on communication and control for persons with severe motor disabilities. Recently, many more projects have presented achievements such as assessment of consciousness, rehabilitation, and functional brain mapping, which could benefit persons with disorders of consciousness (DOC), stroke, brain injury, cerebral palsy, epilepsy, tumors, and other conditions. These and other developments that we have noted in prior books are consistent with, and often precede, more general consensus across other BCI publications. This year might introduce other new directions that will soon become prominent. For example, projects focusing on sensory restoration and new directions with intracranial BCIs were nominated, which are directions that were also nominated in recent years. Two of the 2016 nominees explored autism, and another 2016 nominee included a new game that also addresses classic issues in free will. This year, we have decided to extend our focus on growing trends and issues with a new type of chapter: “Trends in BCI Research”. This is a new type of chapter that spans different authors and research groups, including some nominees and winners along with top outside experts. Each year, our book may include one or more of these special chapters that highlights a topical research field. Our first such chapter focuses on BCI technology for persons with disorders of consciousness (DOCs). This direction has advanced well beyond initial research. Several groups worldwide have published dozens of papers that include bedside assessment, communication, and/or outcome prediction with patients in real-world settings. Our new chapter includes recent achievements from different groups that were presented at the BCI Meeting 2016 in Pacific Grove, CA, the same conference where the 2016 BCI Awards Ceremony occurred.

4 Projects Nominated for the BCI Award 2016 This year’s jury reviewed all of the submissions based on the scoring criteria presented above. After tallying the scores across all reviewers, the twelve submissions that were nominated for a BCI Award 2016 were: A P300-based brain-computer interface for social attention rehabilitation in autism Carlos Amaral1, João Andrade1, Marco Simões1, Susana Mouga1,2, Bruno Direito1, Miguel Castelo-Branco1,3 1 IBILI-Institute for Biomedical Imaging and Life Sciences, Faculty of Medicine— University of Coimbra, Coimbra, Portugal 2 Unidade de Neurodesenvolvimento e Autismo do Serviço do Centro de Desenvolvimento da Criança, Pediatric Hospital, Centro Hospitalar e Universitário de Coimbra, Coimbra, Portugal 3 ICNAS—Brain Imaging Network of Portugal.

6

C. Guger et al.

Sixteen Commands and 40 Hz Carrier Frequency Code-modulated Visual Evoked Potential BCI Daiki Aminaka, Tomasz M. Rutkowski University of Tsukuba, Japan. Natural movement with concurrent brain-computer interface control induces persistent dissociation of neural activity Luke Bashford1,2, Jing Wu3, Devapratim Sarma3, Kelly Collins4, Jeff Ojemann4, Carsten Mehring2 1 Imperial College London, Bioengineering, UK 2 Bernstein Centre, Faculty of Biology, BrainLinks-BrainTools, Univ. of Freiburg, Germany 3 Bioengineering, Ctr. For Sensorimotor Neural Eng. 4 Dept. of Neurolog. Surgery, Ctr. For Sensorimotor Neural Eng., Univ. of Washington, USA. Intracortical Microstimulation as a Feedback Source for Brain-Computer Interface Users Sharlene Flesher2,3, John Downey2,3, Jennifer Collinger1,2,3,4, Stephen Foldes1,3,4, Jeffrey Weiss1,2, Elizabeth Tyler-Kabara1,2,5, Sliman Bensmaia6, Andrew Schwartz2,3,8, Michael Boninger1,2,4, Robert Gaunt1,2,3 1 1,2,5,8 Departments of Physical Medicine and Rehabilitation, Bioengineering, Neurological Surgery, Neurobiology, University of Pittsburgh, Pittsburgh, PA, USA 2 3 Center for the Neural Basis of Cognition, Pittsburgh, PA, USA 3 4 Department of Veterans Affairs Medical Center, Pittsburgh, PA, USA 4 6 Department of Organismal Biology and Anatomy, University of Chicago, Chicago, IL, USA. Minimally invasive endovascular stent-electrode array for high-fidelity, chronic recordings of cortical neural activity Thomas J. Oxley, Nicholas L. Opie, Sam E. John, Gil S. Rind, Stephen M. Ronayne, Clive N. May, Terence J. O’Brien Vascular Bionics Laboratory, Melbourne Brain Centre, Departments of Medicine and Neurology, The Royal Melbourne Hospital, The University of Melbourne, Parkville, Victoria, Australia. Brain-Computer Interfaces based on fMRI for Volitional Control of Amygdala and Fusiform Face Area: Applications in Autism Jaime A. Pereira1,2, Ranganatha Sitaram1,3, Pradyumna Sepulveda2,4,5, Mohit Rana2, Cristián Montalba5, Cristián Tejos3,4,5, Sergio Ruiz1,2,3 1 Department of Psychiatry and Interdisciplinary Center for Neuroscience, School of Medicine, Pontificia Universidad Católica de Chile 2 Laboratory of Brain-Machine Interfaces and Neuromodulation, Pontificia Universidad Catolica de Chile

Introduction

7

3 Institute for Medical and Biological Engineering, Schools of Engineering, Medicine and Biology, Pontificia Universidad Católica de Chile 4 Department of Electrical Engineering, Pontificia Universidad Católica de Chile. 5 Biomedical Imaging Center, Pontificia Universidad Católica de Chile. Reclaiming the Free Will: A Real-Time Duel between a Human and a Brain-Computer Interface Matthias Schultze-Kraft, Daniel Birman, Marco Rusconi, Carsten Allefeld, Kai Görgen, Sven Dähne, Benjamin Blankertz, John-Dylan Haynes Neurotechnology Group, Technische Universität Berlin, Berlin, Germany. An Implanted BCI for Real-Time Cortical Control of Functional Wrist and Finger Movements in a Human with Quadriplegia Gaurav Sharma1, Nick Annetta1, Dave Friedenberg1, Marcie Bockbrader2, Ammar Shaikhouni2, W. Mysiw2, Chad Bouton1, Ali Rezai2 1 Battelle Memorial Institute, 505 King Ave, Columbus, OH 43201 2 The Ohio State University, Columbus, OH, USA 43210. Broad-band BCI: finding structure in noisy data Jordy Thielen, Pieter Marsman, Colleen Monaghan, Jason Farquhar and Peter Desain Donders Center for Cognition, Radboud University Nijmegen Vision-Augmented Rat Cyborg Yueming Wang1, Minlong Lu2, Zhaohui Wu2, Liwen Tian2, Kedi Xu1, Xiaoxiang Zheng1, Gang Pan2 1 Qiushi Academy for Advanced Studies, Zhejiang University, China. 2 College of Computer Science, Zhejiang University, China Precise and reliable activation of cortex with micro-coils Seung Woo Lee and Shelley I. Fried Boston VA Healthcare System, Boston, Massachusetts, USA, Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA. Towards Online Functional Brain Mapping and Monitoring during Awake Craniotomy Surgery using ECoG-based Brain-Surgeon Interface (BSI) L. Yao1, T. Xie2, Z. Wu3, X. Sheng2, D. Zhang2, C. Lin1, F. Negro1, L. Chen3, N. Mrachacz-Kersting4, X. Zhu2, D. Farina1 1 Institute of Neurorehabilitation Systems, University Medical Center Goettingen, Goettingen, Germany. 2 State Key Laboratory of Mechanical System and Vibration, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, China. 3 Department of Neurosurgery, Huashan Hospital, Fudan University, China.44 Center for Sensory-Motor Interaction, Aalborg University, Aalborg, Denmark.

8

C. Guger et al.

5 Summary Since 2010, the annual BCI Awards and book series have recognized the top BCI projects worldwide. Our books have also identified and highlighted major trends and issues in BCI research. The procedures relating to jury selection, scoring criteria, and the awards have been updated somewhat over the years, and this book introduces the new Trends in BCI Research chapter. We plan to continue administering and editing the BCI Awards and book series, and look forward to next year’s submissions!

Advances in BCI: A Neural Bypass Technology to Reconnect the Brain to the Body Gaurav Sharma, Nicholas Annetta, David A. Friedenberg and Marcia Bockbrader

1 Introduction Millions of people worldwide suffer from diseases that lead to paralysis through disruption of signal pathways between the brain and the muscles. Neuroprosthetic devices aim to restore or substitute for a lost function such as motion, hearing, vision, cognition, or memory in patients suffering from neurological disorders. Current neuroprosthetics systems have successfully linked intracortical signals from electrodes in the brain to external devices including a computer cursor, wheelchair and robotic arm [1–11]. In non-human primates, these types of signals have also been used to drive activation of chemically paralyzed arm muscles [12, 13]. However, technologies to link intracortical signals in real time to a neuroprosthetic device to re-animate a paralyzed limb to perform complex, functional tasks had not yet been demonstrated. We recently showed, for the first time, that intracortically-recorded signals can be linked in real-time to muscle activation to restore functional and rhythmic movements in a paralyzed human [14, 15]. We utilized a chronically-implanted intra-cortical microelectrode array to record multiunit activity from the motor cortex in a study participant with quadriplegia from cervical spinal cord injury. Then, using an innovative system of our design, signals from the cortical implant were decoded and re-encoded continuously, in real-time, to drive a custom neuromuscular electrical stimulation (NMES) cuff that enabled the patient to regain lost function. In essence, we have demonstrated an electronic ‘neural bypass technology (NBT)’ that has the ability to circumvent disconnected neurological pathways. G. Sharma (&)
N. Annetta
D.A. Friedenberg Battelle Memorial Institute, 505 King Ave, Columbus, OH 43201, USA e-mail: [email protected] M. Bockbrader The Ohio State University, Columbus, OH 43210, USA © The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_2

9

10

G. Sharma et al.

Figure 1 shows the NBT system used by the participant. The system translates the patient’s intentions to move his wrist and fingers into evoked movements that smoothly combine stimulated wrist and finger movements with voluntary shoulder and elbow movements and enables him to complete functional tasks relevant to daily living. Clinical assessment showed that when using the system, the patient’s motor impairment level improved from C5-C6 to a C7-T1 level unilaterally, conferring on him the critical abilities to grasp, manipulate and release objects. This is the first demonstration of successful control of muscle activation utilizing intracortically-recorded signals in a paralyzed human. These results have significant implications in advancing neuroprosthetic technology for people worldwide living with the effects of paralysis.

Fig. 1 Experimental neural bypass technology (NBT) system in use with the participant (seated in a wheelchair) in front of a table with a computer monitor

Advances in BCI: A Neural Bypass …

11

2 Methods 2.1

Study Design and Surgery

The neural bypass technology has been successfully demonstrated during a Food and Drug Administration (FDA) and Institutional Review Board (IRB)-approved study [14, 15]. The study participant is a 25-year-old male who sustained a C5/C6 level spinal cord injury (SCI) from a diving accident. At baseline (without the BCI), he retains the ability to voluntarily control shoulder and some elbow movements, but has lost finger, hand and wrist function. A 96-channel microelectrode array (Utah array, Blackrock Microsystems, Salt Lake City, UT) was implanted in the left primary motor cortex of the participant. As shown in Fig. 2a, the hand area of the primary motor cortex was identified preoperatively by performing functional magnetic resonance imaging (fMRI) while the participant attempted to mirror videos of hand movements. The NeuroportTM system was used to acquire neural data.

2.2

Novel Hardware and Software Development

The study required the development of novel hardware and software components. A custom neuromuscular electrical stimulator system was developed, including a high-definition, flexible, circumferential NMES cuff that adheres to the user’s forearm. The cuff is comprised of up to 160 electrodes, allowing precise control of individual forearm muscles (Fig. 2b). The high number of electrodes not only allowed stimulation of isolated superficial forearm muscles but also enabled electric field steering to activate individual deep muscles. This combination proved essential for generating isolated finger movements as well as multiple forms of

Fig. 2 Implant locations and NMES cuffs a Red regions are brain areas active during attempts to mimic hand movements, where the t-values for the move-rest T1-weighted fMRI contrast are greater than 7; The implanted microelectrode array location from post-op CT is shown in green; The overlap of the red and green regions is shown in yellow. b Neuromuscular electrical stimulation cuffs

12

G. Sharma et al.

functional grips. Electrical stimulation was provided intermittently in the form of current-controlled, monophasic, rectangular pulses of a 50 Hz pulse rate and 500 µs pulse width. Pulse amplitudes ranged from 0 to 20 mA and were updated every 100 ms. Software development included novel machine learning-based decoding algorithms that are robust to context changes (such as arm position), which allowed the participant to perform complex tasks using a combination of both non-paralyzed and paralyzed muscles simultaneously. Algorithms also included methods to remove stimulation artifacts from the neural data due to electrical stimulation being applied to the arm, giving the participant the ability to start and stop stimulation at will. Our approach to intra-cortically recorded neural data was also innovative: Instead of using single unit activity, which is known to decline over months, we used a wavelet decomposition method to approximate multi-unit neural activity. Wavelet decomposition has shown to be an effective tool in neural decoding applications and provides information encompassing single unit, multiunit, and LFP, without requiring spike sorting [16]. In this study, four wavelet scales (3–6) were used, corresponding to the multiunit frequency band spanning approximately 235–3750 Hz to estimate a feature termed mean wavelet power (MWP). This allowed for recording and analysis of signal features that were detectable and robust over time, providing the potential for a system that could be used for long term applications.

2.3

Participant Sessions and Neural Decoder Training

The study sessions with the participant were typically conducted three times per week, lasting approximately 3–4 h. Stimulation patterns were first calibrated for the desired movements. Decoders were trained for a given movement by asking the participant to imagine mimicking hand movements cued to him by an animated virtual hand on a computer monitor. The neural decoders were trained in training blocks, each consisting of multiple repetitions of each desired motion. This full set of data was used as input for training a nonlinear Support Vector Machine (SVM) algorithm to generate a robust set of decoders. A decoder for each motion (against all other motions and rest) was built using a nonlinear Gaussian radial basis function kernel [17] to process this full set of data and a non-smooth SVM algorithm that uses sparsity optimization to improve performance [18]. During the test period, all decoders ran simultaneously and the decoder with the highest output score above zero was used to drive the NMES.

Advances in BCI: A Neural Bypass …

13

3 Results The applicability of NBT was demonstrated in three different contexts, highlighting different facets of the technology. In the first experiment, the participant was asked to mimic a virtual hand on a computer screen in front of him. The virtual hand cued him to perform six different movements with his right hand: thumb extension, wrist flexion, wrist extension, middle finger flexion, thumb flexion, and hand open. Each movement was cued five times and the presentation order was randomized so the participant could not anticipate the next movement. The participant was able to successfully achieve the movement on 29 of the 30 cues, although he was not always able to maintain the movement for the duration of the cue. Overall, he was able to match the cue 70.4% ± 1.0% (mean ± S.D., P < 0.01 by permutation test) of the time. Examples of neural modulation, decoder output, and physical movement for each of the six cues are shown in Fig. 3. This was the first demonstration of a tetraplegic human regaining volitional control of six distinct hand and wrist movements with an intracortical BCI system. The second demonstration used the Graded and Redefined Assessment of Strength, Sensibility, and Prehension (GRASSP) test [19] to quantify the participant’s sensorimotor impairment level both with and without the neural bypass system. Five domains were evaluated; strength, dorsal sensation, ventral sensation, gross grasping ability (qualitative prehension), and prehensile skills (quantitative prehension). Since the NBT was only expected to improve motor function and not sensory outcomes, we focused on the strength, quantitative prehension and qualitative prehension measures.

Fig. 3 Mean wavelet power and system performance for individual hand movements For each movement, (top) heat maps of MWP and (bottom) neural decoder (dashed line) with physical hand movements (solid line). The vertical dashed lines indicate the start and end of the movement cue, while the break in the heat map indicates when the stimulation turns on. When the stimulation is on, we introduce stimulation artifacts into the data, hence the modified color scale. These artifacts can be partially removed as detailed in Bouton et al. [14]

14

G. Sharma et al.

Fig. 4 GRASSP performance on the three motor function domains The brown triangle shows the participant’s baseline score without the use of the system, and the green triangle shows his scores while using the system. The grayscale triangles show the International Standards for Neurological Classification of Spinal Cord Injury and the American Spinal Injury Association Impairment Scale for comparison

Figure 4 shows that when the participant used the NBT, his Manual Muscle Test (MMT) strength improved from C6 to C7–C8 level, his gross grasping ability improved from C7–C8 to C8–T1 level, and his prehensile skills improved from C5 to C6 level. Taken together, these results quantify the improvement the participant gained while using the system, and suggest that a system that users could take home would significantly improve their ability to live independently. Finally, the participant demonstrated that he could use the system to complete complex functional tasks that are relevant to tasks of daily living. The functional task required him to pick up a bottle, pour the contents of the bottle into a jar, replace the bottle, then pick up a stir stick and stir the contents of the jar (Fig. 5). This task required the participant to combine his residual shoulder and elbow movement with three hand movements using the NBT (hand open, cylindrical grip, pinch grip). We observed differences in neural patterns when the participant was performing shoulder and elbow movements, which necessitated including those movements in the training process to assist in building robust decoders. In this study, for the first time, a human with quadriplegia regained volitional, functional movement using intracortically-recorded signals linked to neuromuscular stimulation in real-time. Using our investigational system, our C5/C6 participant gained wrist and hand function consistent with a C7/T1 level of injury.

Advances in BCI: A Neural Bypass …

15

Fig. 5 Grasp-pour-and-stir functional movement task Sequential snapshots (a–f) from the functional movement task showing the participant opening his hand (a), grasping the glass bottle (b), pouring its contents (dice) into a jar (c), grasping a stir stick from another jar (d), transferring the stir stick without dropping it (e), and using it to stir the dice in the jar (f)

This improvement in function is meaningful for reducing the burden of care in patients with SCI, as most C5/C6 patients require assistance for activities of daily living, while C7/T1 level patients can live independently. The technology also has potential applications in the field of BCI-controlled neuroprosthetics, which could improve patient independence through improved motor function.

4 Current Work and Outlook Towards Future Our current efforts are focused on adapting the NBT for home use. To make this technology ready for home use, the system must be made smaller and easier to use, with fewer adjustments needed from the user over long-term use. Making these improvements to this system will require several technological hurdles to be overcome as detailed below. The current NBT system was designed for the research setting where space and mobility is not a constraint. However, for home use, the technology will need to be miniaturized. On the recording side, Blackrock Microsystems has made progress in developing a wireless headstage that can handle the high bandwidth data, but it does not yet have FDA approval for human use. It also uses a large receiver to interface with the PC, which must be shrunken down. The PC used to control the system would ideally be replaced with a small device such as a tablet or a custom designed, small form factor device with an embedded processor. This can be challenging due to the complexity of the algorithm and the amount of data that needs to be processed, and it will require the algorithms to be streamlined. The NMES will also need to be simplified and made more user friendly. The high voltage and high

16

G. Sharma et al.

channel count as well as the size of the battery that is needed makes it challenging to shrink the NMES. Shrinking the entire system will increase portability, but even more improvements must be made to the electrode cuff before the system can be easily used in home settings. These stimulation electrodes will need to be embedded in a sleeve form that can be donned as a piece of garment that can keep the electrodes in good contact with the skin. The decoding algorithms need to be adapted to make them more robust to any variability in neural modulation. Currently, the user needs to go through a retraining process every few hours to build new decoders. The decoders need to be rebuilt because neural activity in the brain changes, even over the course of just a few hours. Many environmental conditions, mental states of the user (e.g. emotions, focus level, etc.), sensory feedback, and other movements the user is making, among other things, will all factor into how the neural activity changes. Decoders must be developed that can account for these changes so that the user does not constantly have to go through time-consuming retraining. Use of deep neural networks can be one possible way to improve decoder performance.

5 Neurorehabilitation Outcomes and Need for Standardized Tests for Evaluating SCI Neuroprosthetics As the options for BCI-neuroprosthetics expand to include a range of more or less invasive control options (from brain implants to surface EEG to myoelectric control) and more or less cybernetic effector mechanisms (from surface electrical stimulation, surgical implants and tendon transfers, to robotic arms), it is increasingly important to be able to counsel consumers on both costs and risks—whether financial, technological, surgical, or self-image related—and comparative device performance. However, there is no consensus for how to evaluate device performance. Multiple upper limb standardized tasks have been evaluated by expert reviews [20–22] and consensus panels (SCI EDGE task force [23, 24], and the SCI RE project [25, 26]), with general agreement that the ideal evaluation tasks have the following established psychometric properties: • Ecological and construct validity, such that arm and hand functional tasks be relevant to Activities of Daily Living (ADLs) but do not confound hand function with other impairments, like balance; • Sensitivity to detect small clinically significant changes important for evaluating treatment effects and comparing interventions; • Performance range sufficient to avoid ceiling and floor effects; • Reliability associated with repeatable, standardized, unambiguous scoring that (1) does not confound performance speed with degree of ability to complete the task or level of assistance needed, (2) provides some estimate of trial-to-trial

Advances in BCI: A Neural Bypass …

17

performance variability, (3) is based on observed measurements and not on subjective reports, and (4) is not subject to practice effects; • Clinical relevance, such that the measurement domain falls within the arm and hand activity domain of the International Classification of Functioning, Disability and Health [27]; and • Prognostic implications for functional independence, established by presence of normative performance data for patients with SCI by ASIA Impairment Scale level. A recent review from the tendon-transfer literature [21] identifies 8 measures that fall within the ICF Arm and Hand Activity domain: The Grasp and Release Test (GRT [28]), the Capabilities of Upper Extremity Questionnaire (CUE-Q [29, 30]), the Van Lieshout Test (VLT, [31, 32]), the Action Research Arm Test (ARAT, [33]), the Tetraplegia Hand Activity Questionnaire (THAQ [34]), the AuSpinal Test [35], the Sollerman Hand Function Test (SHFT [36]) and the Graded and Redefined Assessment of Strength, Sensibility, and Prehension (GRASSP [19, 22, 37, 38]). Of these, only 6 are based on rater observations of performance (GRT, VLT, ARAT, AuSpinal, SHFT, GRASSP), only 4 of which (GRT, VLT, SHFT, GRASSP) have been endorsed for research in spinal cord injury either by the SCI EDGE task force of the Neurology Section of the APTA [23, 24] or the Canadian SCI Research Evidence (SCIRE) Project [25, 26]. To date, BCI-neuroprosthetic studies have alternatively used the GRT (Freehand and/or tendon transfers [39–43] [44]), ARAT (Robotic Applied Physics Laboratory arm [45, 46]), Box and Block Test (BBT [47]—which is also endorsed by SCIRE; Robotic Applied Physics Laboratory arm [45]), or GRASSP (NBT system [14]). Individually, these 4 measures address different aspects of the performance profile of BCI-neuroprosthetics (see Fig. 6), so they may be best utilized as a battery. Of these 4 measures, only the ARAT assesses fine pincer grip, which is important for manipulating small objects. However, the ARAT incorporates only the power grip into dynamic object manipulation (pouring a cup), and not fine fingertip grasps. The GRASSP evaluates power and fine grips in dynamic tasks, but has potential floor and ceiling effects. The GRASSP can provide prognostic implications for functional independence, as its scoring is normed to AIS Impairment Scale levels, which are widely recognized by patients and clinicians alike. For example, the participant using our NBT system was scored on the GRASSP as improving from C5/6 to C7/T1 level function using the device, which would correlate to a significant improvement in functional independence if used in the home setting. The BBT does not specify grip type, and some patients can do the object transfer task without the neuroprosthetic, with only their baseline adaptive grip. This task may help identify speed and efficiency limitations of BCI systems, and when patients should not use the BCI-neuroprosthetic over their adaptive grip. Lastly, the GRT was developed to assess hand and wrist function in isolation from trunk and arm control for a range of light to heavy objects. It has also been widely used to assess recovery of function after tendon transfer surgery, which is

18

G. Sharma et al.

Power/ Palmar

Lateral Key

Tip-to-Tip Opposition

Fine Pincer

Grip Type

GRT BBT

ARAT

BBT

ARAT

BBT

ARAT

GRASSP

GRT

GRASSP

GRT

GRASSP

ARAT

Static

ARAT

Dynamic GRASSP

GRASSP

GRASSP

Fig. 6 Grip types featured in standardized tests of upper limb motor function for BCI-neuroprosthetic research and between-device comparisons No single outcome measure adequately assesses performance across static and dynamic performance across the four essential grip types (power/palmar, lateral key, tip-to-tip opposition, and fine pincer). Static tasks isolate hand and wrist function from other upper limb movements, while dynamic tasks require stable grip through forearm pronation/supination for successful completion

the only neuroprosthetic-like intervention that has been widely translated into clinical practice. In summary, research and development, clinical translation and future prescriptions for upper limb BCI-neuroprosthetics depend on the rational development of a battery of upper limb functional measures to compare devices in meaningful ways. Together, the ARAT, BBT, GRT, and GRASSP provide complementary measures to assess device strengths and limitations across 4 essential grip types (power/palmar, lateral key, tip-to-tip opposition, and fine pincer) in static and dynamic hand and wrist tasks.

References 1. Aflalo T et al. (2015) Neurophysiology. Decoding motor imagery from the posterior parietal cortex of a tetraplegic human. Science 348(6237):906–910 2. Bansal AK et al. (2012) Decoding 3D reach and grasp from hybrid signals in motor and premotor cortices: spikes, multiunit activity, and local field potentials. J Neurophysiol 107(5): 1337–1355

Advances in BCI: A Neural Bypass …

19

3. Chapin JK (1999) et al. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nat Neurosci 2(7):664–670 4. Hochberg LR et al. (2012) Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485(7398):372-U121 5. Hochberg LR et al. (2006) Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442(7099):164–171 6. Kennedy PR, Bakay RAE (1998) Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport 9(8):1707–1711 7. Santhanam G et al. (2006) A high-performance brain-computer interface. Nature 442(7099): 195–198 8. Serruya MD et al. (2002) Instant neural control of a movement signal. Nature 416(6877): 141–142 9. Taylor DM, Tillery SI, Schwartz AB, (2002) Direct cortical control of 3D neuroprosthetic devices. Science 296(5574):1829–1832 10. Velliste M et al. (2008) Cortical control of a prosthetic arm for self-feeding. Nature 453(7198): 1098–1101 11. Wessberg J et al. (2000) Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408(6810):361–365 12. Ethier C et al. (2012) Restoration of grasp following paralysis through brain-controlled stimulation of muscles. Nature 485(7398):368–371 13. Moritz CT, Perlmutter SI, Fetz EE (2008) Direct control of paralysed muscles by cortical neurons. Nature 456(7222):639-U63 14. Bouton CE et al. (2016) Restoring cortical control of functional movement in a human with quadriplegia. Nature 533(7602):247–250 15. Sharma G et al. (2016) Using an artificial neural bypass to restore cortical control of rhythmic movements in a human with quadriplegia. Sci Rep 6:33807 16. Sharma G et al. (2015) Time stability of multi-unit, single-unit and LFP neuronal signals in chronically implanted brain electrodes. Bioelectronic Medicine (in press) 17. Scholkopf B et al. (1997) Comparing support vector machines with Gaussian kernels to radial basis function classifiers. IEEE Trans. Signal Process 45:2758–2765 18. Humber C, Ito K, Bouton C (2010) Nonsmooth formulation of the support vector machine for a neural decoding problem. arXiv 19. Kalsi-Ryan S et al. (2012) Development of the Graded Redefined Assessment of Strength, Sensibility and Prehension (GRASSP): reviewing measurement specific to the upper limb in tetraplegia. J Neurosurg Spine 17(1 Suppl):65–76 20. Mulcahey MJ, Hutchinson D, Kozin S (2007) Assessment of upper limb in tetraplegia: Considerations in evaluation and outcomes research. J. Rehabil Res Dev 44(1):91–102 21. Sinnott KA et al. (2016) Measurement outcomes of upper limb reconstructive surgery for tetraplegia. Arch Phys Med Rehabil 97(6 Suppl 2):S169–81 22. Kalsi-Ryan S et al. (2016) Responsiveness, sensitivity, and minimally detectable difference of the graded and redefined assessment of strength, sensibility, and prehension, version 1.0. J Neurotrauma 33(3):307–314 23. Kahn J et al. (2013) SCI EDGE outcome measures for research, N.S. American Physical Therapy Association, Alexandria, VA 24. Kahn J et al. (2013) Spinal Cord Injury EDGE Task Force Outcome Measures Recommendations, American Physical Therapy Association, Neurology Section 25. Mille WC et al. (2013) Outcome measures. In: Eng JJ et al. (eds) Spinal cord injury rehabilitation evidence, version 4.0. Vancouver, CA, pp 28.1–28.366 26. Hsieh JTC et al. (2011) Outcome Measures Toolkit: Implementation Steps, S. Project. London, ON, CA, p 1–58 27. Organization WH, International Classification of Functioning, Disability and Health: ICF2001: World Health Organization 28. Wuolle KS et al. (1994) Development of a quantitative hand grasp and release test for patients with tetraplegia using a hand neuroprosthesis. J Hand Surg 19(2):209–218

20

G. Sharma et al.

29. Oleson CV, Marino RJ (2014) Responsiveness and concurrent validity of the revised capabilities of upper extremity-questionnaire (CUE-Q) in patients with acute tetraplegia. Spinal Cord 52(8):625–628 30. Marino RJ et al. (2015) Reliability and validity of the capabilities of upper extremity test (CUE-T) in subjects with chronic spinal cord injury. J Spinal Cord Med 38(4):498–504 31. Spooren A et al. (2006) Measuring change in arm hand skilled performance in persons with a cervical spinal cord injury: responsiveness of the Van Lieshout Test. Spinal Cord 44(12): 772–779 32. Franke A et al. (2013) Arm hand skilled performance in persons with a cervical spinal cord injury—long-term follow-up. Spinal cord 51(2):161–164 33. Yozbatiran N, Der-Yeghiaian L, Cramer SC (2008) A standardized approach to performing the action research arm test. Neurorehabilitation Neural Repair 22(1):78–90 34. Land N et al. (2004) Tetraplegia Hand Activity Questionnaire (THAQ): the development, assessment of arm–hand function-related activities in tetraplegic patients with a spinal cord injury. Spinal Cord 42(5):294–301 35. Coates S et al. (2011) The AuSpinal: a test of hand function for people with tetraplegia. Spinal Cord 49(2):219–229 36. Sollerman C, Ejeskär A (1995) Sollerman hand function test: a standardised method and its use in tetraplegic patients. Scand J Plast Reconstr Surg Hand Surg 29(2):167–176 37. Kalsi-Ryan S et al. (2012) The graded redefined assessment of strength sensibility and prehension: reliability and validity. J Neurotrauma 29(5):905–914 38. Kalsi-Ryan S et al. (2009) Assessment of the hand in tetraplegia using the Graded Redefined Assessment of Strength, Sensibility and Prehension (GRASSP) impairment versus function. Top Spinal Cord Inj Rehabil 14(4):34–46 39. Peckham PH et al. (2001) Efficacy of an implanted neuroprosthesis for restoring hand grasp in tetraplegia: a multicenter study. Arch Phys Med Rehabil 82(10):1380–1388 40. Smith B, Mulcahey M, Betz R (1996) Quantitative comparison of grasp and release abilities with and without functional neuromuscular stimulation in adolescents with tetraplegia. Spinal Cord 34(1):16–23 41. Kilgore KL et al. (1997) An implanted upper-extremity neuroprosthesis. follow-up of five patients. J Bone Jt Surg Am 79(4):533–541 42. Kilgore KL et al. (2008) An implanted upper-extremity neuroprosthesis using myoelectric control. J Hand Surg 33(4):539–550 43. Mulcahey M et al. (1999) A prospective evaluation of upper extremity tendon transfers in children with cervical spinal cord injury. J Pediatr Orthop 19(3):319–328 44. Mulcahey M, Smith B, Betz R (2003) Psychometric rigor of the Grasp and Release Test for measuring functional limitation of persons with tetraplegia: a preliminary analysis. J Spinal Cord Med 27(1):41–46 45. Wodlinger B et al. (2015) Ten-dimensional anthropomorphic arm control in a human brain-machine interface: difficulties, solutions, and limitations. J Neural Eng 12(1):016011 46. Collinger JL et al. (2013) High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381(9866):557–564 47. Mathiowetz V et al. (1985) Adult norms for the box and block Test of manual dexterity. Am J Occup Ther 39(6):386–391

Precise and Reliable Activation of Cortex with Micro-coils Seung Woo Lee and Shelley I. Fried

1 Introduction The optimization of brain-computer interfaces (BCIs) will require the delivery of feedback signals to the somatosensory and/or proprioceptive cortices of the device user. Ultimately, the precision and reliability with which such signals can be delivered will underlie the quality and consistency of the information that can be conveyed. Unfortunately, the use of implantable micro-electrodes to deliver electrical signals directly into cortex has several inherent drawbacks that limit their efficacy and can reduce their consistency over time. For example, implantation triggers a series of complex biological reactions that can alter the structural and functional properties of the surrounding neural tissue [9, 22]. In severe cases, these changes can lead to high-impedance glial encapsulation around individual electrodes, thereby disrupting the flow of current into the surrounding tissue. Even without the loss in effectiveness that can occur over time, conventional electrode implants are limited by their inability to selectively target (or avoid) specific types of neurons. This is of particular concern with passing axons from distal neurons, as their high sensitivity to stimulation can greatly expand the size of the region activated by a given electrode (Fig. 1a) and can also lead to a wide array of undesirable side effects [1, 10, 28]. In addition to limiting the potential effectiveness of BCI feedback, the challenges associated with implantable micro-electrodes are similarly problematic for other types of cortically-based prostheses that require focal S.W. Lee (&)
S.I. Fried Department of Neurosurgery, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA e-mail: [email protected] S.I. Fried e-mail: [email protected] S.I. Fried Boston VA Healthcare System, Boston, MA, USA © The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_3

21

22

S.W. Lee and S.I. Fried

Fig. 1 Enhanced control of cortical activation with micro-coil magnetic stimulation. a Schematic illustration of a micro-electrode implanted into cortex; conventional electrodes produce electric fields that are largely symmetric (red arrows) and therefore create uniform activating forces on all nearby neurons and processes (red shaded region). b Similar to (a) except for implantation of a micro-coil. The induced electric field is spatially asymmetric and creates a relatively strong activating force on vertically oriented neurons while creating only a relatively weak activating force on horizontally oriented processes

activation, e.g. visual prostheses that strive to restore sight to the blind by stimulating primary visual cortex (V1) [6, 19, 25]. Focal and predictable activation of cortex is also crucial for fundamental research studies in which specific regions of cortex are targeted by electric stimulation in order to resolve details of brain structure and function. It is well known that magnetic stimulation can overcome many of the limitations associated with microelectrode-based stimulation of cortex. Unlike electric fields, magnetic fields pass readily through biological tissue. Thus, even if coils become severely encapsulated, their ability to stimulate remains stable over time. While magnetic fields are not thought to directly activate neurons, time-varying magnetic fields induce the electric fields that are effective. Magnetic fields can thereby ‘carry’ the electric field beyond any region of encapsulation. The magnetic fields arising from coils are also spatially asymmetric and can therefore produce stronger activating forces in some directions than others (Fig. 1b). Thus, in the cortex for example, a suitably oriented coil can create a strong activating force for vertically-oriented pyramidal neurons without simultaneously creating a strong activating force for the horizontally-oriented passing axons that can arise from distant regions of the brain. As a result, activation with micro-coils can be confined to a focal region around the coil, a considerable advantage over the spatially broad activation that arises with conventional electrode implants [10]. While most

Precise and Reliable Activation of Cortex with Micro-coils

23

previous studies with magnetic stimulation have focused on the use of large coils for non-invasive activation of neurons, several recent efforts have shown that tiny coils, e.g. small enough to be safely implanted into cortex, can strongly activate surrounding neurons. Here, we describe some of this recent work and review some of the advantages of this approach. We also discuss some of the challenges that will need to be overcome before micro-coil based implants can be safely utilized in clinical applications.

2 Neuronal Activation with Submillimeter-Sized Inductors Despite the potential benefits of magnetic stimulation, the reduction in the size of the coil needed for safe implantation into the brain greatly limits the strength of the electric field that can be induced. Fortunately, the initial computational analyses with micro-coils [2] suggested that field strengths in excess of the known thresholds required for activation [17] could still be obtained, as long as the distance between the coil and targeted neurons was limited. Much previous work with electric stimulation has shown that in many cases the magnitude of the gradient of the electric field is actually the driving force for activation [24], and so the electric field gradients arising from small coils were also confirmed to exceed known thresholds [14, 15]. Initial electrophysiological testing utilized a millimeter-sized, commerciallyavailable inductor (Panasonic ELJ-RFR10JFB, 21 turns, 500 um diameter, 1 mm length, 5  10 µm copper wire, 100 nH) and confirmed its ability to activate neurons. Further, initial testing also showed that the asymmetric electric fields (and electric field gradients) could be harnessed to preferentially activate specific neuronal sub-populations within the region surrounding the coil. For example, if the central axis of the coil was held parallel to the surface of a retinal explant (Fig. 2a), the component of the electric field penetrating vertically into the retina, e.g. parallel to the long axes of bipolar cells, was strong and therefore optimized to activate these cells. The resulting spiking patterns that arose in ganglion cells closely matched the characteristic patterns known to arise when bipolar cells are artificially activated (Fig. 2c), e.g. bursts of spikes with a relatively long onset latency [7, 12, 13]. This same orientation did not simultaneously activate ganglion cells or their axons (Fig. 2, pink neurons), consistent with the relatively weak electric fields and gradients arising in the horizontal direction. Rotation of the coil so that its central axis was now perpendicular to the retinal surface (Fig. 2b) resulted in induced electric fields that were now strongest in the horizontal direction, e.g. along the length of ganglion cell axons, and resulted in single, short-latency (75% rated restoration of arm/hand function as very important to improving their quality of life, making this technology a priority for the population [4].

J. Collinger
S. Foldes
J. Weiss
E. Tyler-Kabara
M. Boninger
R. Gaunt (&) Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA e-mail: [email protected] S. Flesher
J. Downey
J. Collinger
J. Weiss
E. Tyler-Kabara
A. Schwartz
M. Boninger
R. Gaunt Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA e-mail: [email protected] S. Flesher
J. Downey
J. Collinger
S. Foldes
A. Schwartz
R. Gaunt Center for the Neural Basis of Cognition, Pittsburgh, PA, USA J. Collinger
S. Foldes
M. Boninger Department of Veterans Affairs Medical Center, Pittsburgh, PA, USA E. Tyler-Kabara Department of Neurological Surgery, University of Pittsburgh, Pittsburgh, PA, USA S. Bensmaia Department of Organismal Biology and Anatomy, University of Chicago, Chicago, IL, USA A. Schwartz Department of Neurobiology, University of Pittsburgh, Pittsburgh, PA, USA © The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_5

43

44

S. Flesher et al.

Our prior work with iBCIs has shown clinically significant restoration of arm control with both seven and ten controllable degrees-of-freedom using a robotic arm [1, 2]. In both cases, users had continuous and simultaneous control of three translational degrees of freedom (hand location) and three degrees of orientation (wrist rotation). In addition, whole-hand grasp, as a single dimension, was controlled in the seven degree-of-freedom paradigm, while 10 degree-of-freedom control expanded independent movements of the hand. These dimensions consisted of combined flexion of the thumb, index and middle fingers, opposition of the thumb, combined flexion of the ring and pinky fingers, as well as ab/adduction of the four fingers. This increased the capabilities of the user from performing simple grasping to include dexterous hand movements. These new capacities were intended to enable appropriate handling of a variety of objects, however, from a functional perspective, there was no significant improvement in performance on object transfer tasks [2]. Limited improvements in functionality, even with added control dimensions in the hand, may have been due to the fact that current iBCI systems are limited to visual feedback. Lacking any cutaneous somatosensory feedback, it is possible that additional control degrees-of-freedom cannot be used skillfully. It is also possible that restored sensation itself could prove more beneficial than additional controllable movements themselves. Somatosensory feedback is necessary for skilled movement [5–9]. In healthy subjects, the loss of cutaneous feedback alone can make even simple motor tasks nearly impossible [5, 8]. However, state of the art iBCI paradigms do not provide somatosensory feedback and instead rely solely on visual feedback. Therefore, since somatosensation is a critical component of natural movement, and the loss of sensation impairs movement, we believe that cutaneous sensations should be restored for iBCI systems (Fig. 1). One possible method of delivering this feedback is through intracortical microstimulation (ICMS) of primary somatosensory cortex (S1). Indeed, ICMS in S1 can be used to guide the behavior of implanted animals [11–14] and can be delivered safely over many months [10]. Recently, we extended this work to a human participant and showed that ICMS in S1 is spatially selective and evokes percepts that are naturalistic and are perceived to span a range of intensities [15]. Here, we summarize these findings and begin to investigate the impact of providing sensation on motor control tasks designed to benefit from ICMS feedback.

2 Methods Implant This study was conducted under an Investigational Device Exemption from the Food and Drug Administration, approved by the Institutional Review Boards at the University of Pittsburgh (Pittsburgh, PA) and the Space and Naval Warfare Systems Center Pacific (San Diego, CA), and registered at ClinicalTrials.gov (NCT01894802).

Intracortical Microstimulation as a Feedback Source …

45

Fig. 1 iBCI paradigm. Neural activity, in the form of threshold crossings, was recorded from two intracortical microelectrode arrays (Utah arrays, top left) placed in M1. These signals were transformed into velocity commands to control the endpoint of a prosthetic limb via an optimal linear estimator decoder. Up to 10 degrees of freedom were simultaneously controlled. The paradigm depicted above includes the addition of somatosensory feedback, provided via ICMS delivered to microelectrode arrays (Utah arrays, bottom left) implanted in S1 (Picture courtesy of Timothy Betler, UPMC Media Relations)

Informed consent was obtained before any study procedures were conducted. To test the feasibility of ICMS as a feedback source for BCI users, a twenty-eight-year-old participant with a chronic C5 motor and C6 sensory AIS B spinal cord injury was implanted with two, 32-channel, stimulating intracortical microelectrode arrays in S1 and two, 88-channel, recording arays in M1. All arrays were implanted in the left hemisphere and were placed based on pre-surgical imaging. The two recording arrays were placed in the upper limb representation in M1 targeting the shoulder and hand region. Neural activity from these arrays was decoded to control a robotic limb. The stimulating arrays were targeted to the hand region of area 1 of S1. Microelectrode arrays were placed so that ICMS would elicit cutaneous percepts that projected to the right hand and thus relay tactile information from the sensors in the robotic limb (Modular Prosthetic Limb, Johns Hopkins Applied Physics Lab). Microstimulation Tasks To assess the viability of ICMS as a feedback source for BCI users, we sought to characterize the sensory quality, location, detection threshold, and the ability to evoke a range of perceived intensity. Stimulus pulse trains consisted of cathodic-first, asymmetric, charge-balanced pulses delivered at 100 Hz (Fig. 2). Pulse amplitude was modulated by task parameters. To determine the location and perceptual qualities of percepts evoked via ICMS, individual electrodes were stimulated for 1 s at a supraliminal intensity (60 µA). Pulse trains were repeated as many times as necessary for the participant to fully describe the evoked sensation. The subject was shown either a segmented hand or an unlabeled schematic of a hand, which he used to describe the location of the percepts. In the first 10 months, the participant reported which of the predefined segments (see Fig. 3a) were closest to the location of the projected fields. In later

46

S. Flesher et al.

Fig. 2 Pulse trains and 2 alternative forced-choice ICMS tasks. a Pulse waveform. Pulses were delivered at 100 Hz and pulse amplitude was modulated by task parameters. b Detection thresholds and just noticeable differences (JND) were measured using a 2-alternative forced-choice paradigm, as shown above. For detection thresholds (middle row), the interval that contained the pulse train was identified. For JNDs (bottom row), the interval containing the more intense pulse train was identified

Fig. 3 Locations of projected fields. a Segmented hand that was shown to the participant during 60lA surveys to describe the locations of evoked sensations. b Arrays color-coded by location based on segmentation and colors in (a). Colors indicate participant’s reported projected field for each electrode. Gray squares indicate electrodes that were not in use. Pink squares indicate electrodes that had diffuse projected fields. c Adapted from Flesher et al.15. Pre-operative MEG imaging showing array locations in S1 and areas of activation when the participant watched different regions of a hand being stroked by a cotton swab. The somatotopy observed from the MEG imaging is largely reflected in the spatial arrangement of projected fields shown in (b)

experiments, the participant drew the locations of the projected fields using a tablet computer and a stylus that was placed in his hand. The quality of the evoked sensations was described using the set of descriptors listed in Table 1. Evoked percepts could be described using any of the suggested words, a combination thereof, or any words the participant chose. We also investigated the relationship between stimulation amplitude and the perceived intensity of evoked percepts in a free magnitude estimation task. Pulse trains ranging in intensity from 10 to 80 µA were presented, in random order, to the participant. Following each pulse train, the participant was asked to report how

Totally natural Almost natural Possibly natural Rather unnatural Totally unnatural

Naturalness

Pain

10

4,5,6

0 (no pain) 1,2,3

0.0

94.7

3.0

2.3

%

7,8,9

Skin Surface Below Skin Both

Depth

8.2

78.0

13.6

0.2

%

0

0

0

0

100

%

Sharp

Pressure

Touch

Mechanical

3.3

36.9

1.8

%

Movement

Vibration

Movement

0

10.4

%

Cool

Warm

Temperature

0

24.5

%

Itch

Electric current Tickle

Tingle

Misc

0.0

0.0

3.3

79.2

%

Table 1 Perceptual qualities of evoked sensations. The participant described the conscious perception of ICMS using a variety of words that are included in this table. The percentage of stimulus trains that elicited each type of sensations are shown. Only one option for naturalness, depth, and pain could be selected. However, any combination of mechanical, movement, temperature, or miscellaneous descriptions could be reported

Intracortical Microstimulation as a Feedback Source … 47

48

S. Flesher et al.

intense the pulse train felt, using a self-selected numerical scale. Each amplitude was presented in a random order, and the process was repeated a total of six times. Unbeknown to the participant, the first presentation of each amplitude was excluded from analysis. The participant was instructed to report a “0” if the stimulus was not felt and to report a number twice as large for a stimulus that felt twice as intense as a previous stimulus. A two-alternative forced choice (2AFC) paradigm was used to measure detection thresholds and just noticeable differences (JNDs). To measure detection thresholds, the participant was instructed to indicate which of two consecutive time windows contained an ICMS train. Pulse amplitude was selected dynamically based on task performance such that it decreased if the participant correctly identified the window that contained the stimulus pulse on three consecutive trials at the same pulse amplitude. Pulse amplitude was increased on any trial following the incorrect identification of the window that contained the stimulus. Detection thresholds were measured for each electrode at least twice in the 18-month study period, with some electrodes measured more frequently. The JNDs from a subset of electrodes were measured in a similar fashion. Pulse trains were presented in both time windows and the participant was instructed to identify which window contained the more intense stimulus. The amplitude of one pulse train was held constant in all trials, at either 20 or 70 µA. If this standard amplitude was 20 µA, all comparison amplitudes were greater than this value. For the high standard amplitude of 70 µA, all comparison amplitudes were smaller. Using this approach, we could compare the participant’s ability to distinguish between pulse trains that had the same amplitude difference, but were presented in a low or high amplitude regime. Neural Decoding and Control Neural signals were acquired at 30 kHz using the NeuroPort signal processor (Blackrock Microsystems). Raw signals were acquired with a 0.3–7.5 kHz band pass filter and then further filtered using a first order Chebyshev high-pass filter (250 Hz, 10 dB ripple) filter for spike thresholding. Thresholds were set at −4.5 times the root mean squared value of the raw signal. Threshold crossings were counted in 20 ms bins, then filtered using a 440 ms exponential filter. To train the decoder, the participant observed the robotic limb completing a two-dimensional hand shaping task. The task consisted of 9 targets made up of all combinations of the flexed, neutral, and extended positions for both “pinch” (thumb/index/middle flexion-extension) and “scoop” (ring/pinky flexion/extension) hand shapes. Each of the 9 targets had a unique name, which was presented as an audio cue at the beginning of a trial. After the audio cue, the hand automatically moved to achieve the appropriate target position and hold it for 1 s. The participant was instructed to act as though he were controlling the robotic hand, essentially attempting to perform the movements with the robotic hand. The firing rates from the M1 electrodes during 27 trials of this task were then fit to the movement velocities of the robotic limb to create an optimal linear estimator decoder using methods described in detail elsewhere [1, 2]. Once this decoder was

Intracortical Microstimulation as a Feedback Source …

49

trained, the participant completed the same task with orthogonal assistance, where the computer constrained the decoded movement velocities to the ideal path [16]. Once 27 trials had been collected with orthogonal assistance, a new decoder was trained on the most recent data. This velocity decoder was then used, without computer assistance, to complete a two-dimensional force matching task, described below. Real-Time ICMS Feedback Tasks To investigate the ability of the participant to use ICMS as a feedback source, we first performed a location discrimination task. An experimenter touched individual fingers on the prosthetic hand and the blindfolded participant was asked to respond with the identity of the finger. Torque sensor data derived from the D2-D5 finger motors of a prosthetic limb were linearly mapped to groups of electrodes that elicited percepts on the corresponding fingers. When the fingers were touched, motor torque increased, and these torque values were used to modulate pulse train amplitude in real-time. To investigate the utility of providing feedback about contact location and intensity in a motor control task, the participant performed a continuous two-dimensional force matching task. The participant was instructed to pinch (index and middle finger flexion), scoop (ring and little finger flexion), or grasp (all finger flexion) a foam object either gently or firmly. ‘Gentle’ targets were defined to be 12-36% of the maximum grasp torque, while ‘firm’ targets were specified to be 36–60% of the maximum grasp torque. The participant had to apply the instructed torque with the specified fingers for 750 ms within 7 s of the start of a trial to be successful. During all trials, the participant used the BCI to continuously control both the pinch and scoop dimensions while trying to achieve instructed torque targets. This task was performed with and without ICMS feedback. The task was conducted in blocks of 6 trials, such that each combination of the three grasp postures and two force targets were presented once, in random order. The success rate per block was used as the metric for task performance.

3 Results 3.1

Projected Fields and Perceptual Quality

The projected fields of the electrodes (see Fig. 3a and b) were located in digits 2–5, primarily at the base of each digit. Sensations were usually reported as originating from a single digit, and if projected fields were reported for multiple fingers, reports were from adjacent fingers. During supraliminal intensity surveys of all the implanted electrodes, sensations were reported from 55 of the 60 electrodes used in this task. Due to some electrodes exhibiting an abnormally high interphase voltage at 60 uA, four electrodes were not used in this task. No painful sensations or paresthesias were ever reported. Of the

50

S. Flesher et al.

reported sensations, 36.9% were described as “pressure” and 79.2% as “tingle” with most being described as “possibly natural” (Table 1). Percepts mostly felt as though they contained sensory elements that occurred both at and below the skin surface. Sensation qualities, as listed in Table 1, were not mutually exclusive, so the participant could describe sensations using combinations of the qualities, and/or using any other descriptors he deemed appropriate.

3.2

Psychometric Evaluation

Detection thresholds were measured for all 62 tested electrodes (Fig. 4a). The median detection threshold was 29.5 µA with lower and upper quartiles of 19.3 µA and 37.6 µA, respectively. Additionally, we measured JNDs on seven electrodes (Fig. 4a), as previously shown [15]. Data were fit with cumulative normal curves, and the JND was defined to be the difference in stimulus amplitudes where the more intense stimuli was correctly identified 75% of the time. The JNDs were found to be 15.4 ± 3.9 µA and were the same regardless of standard amplitude (Wilcoxon signed rank, p = 1). Using the free magnitude estimation task, we found the relationship between pulse amplitude and perceived intensity to be highly linear (R2 = 0.996, linear regression). The results from all ten tested electrodes are shown in Fig. 4c.

Fig. 4 Psychometric evaluation of evoked sensations. a. Distribution of mean detection threshold for each electrode. b. JND results fit with a cumulative normal curve. Low and high standard amplitudes yielded similar curves and JNDs. c. Perceived intensity increased linearly with stimulation amplitude. Responses from all ten electrodes were normalized and a line fit to the raw data points. The mean (dots) and S.E.M. (error bars) for each test amplitude are shown

Intracortical Microstimulation as a Feedback Source …

3.3

51

Real-time ICMS Feedback Tasks

As we previously reported [15], the participant could accurately identify which robotic fingers were being touched while blindfolded. Across 14 sessions containing a total of 69 repetitions on each finger, the correct finger was identified 84 ± 12.2% (mean accuracy across fingers +/− standard deviation) of the time. Repeated tests did not improve the accuracy, indicating that training was not a factor. The index (D2) and little (D5) fingers were correctly identified most consistently (96.9 ± 7.2% and 93.9 ± 12.1%, respectively), while the middle (D3) and ring (D4) fingers were less accurately identified (73.5 ± 18.1% and 73.1 ± 24.6%, respectively), with errors typically being reports of an adjacent finger (see Fig. 5a). In the continuous force matching task, the participant continuously controlled the flexion/extension of two grasp dimensions while the applied torque for each of these dimensions was evaluated. The success rate for achieving the six possible targets (pinch, scoop or grasp with gentle or firm forces) was significantly improved with the addition of ICMS feedback (Fig. 5, p < 0.001, Wilcoxon signed-rank test). This was even though many of the trials could be successfully completed without ICMS feedback, which we attribute to the simple nature of the task. On successful trials, time to target was not significantly different between feedback paradigms (Wilcoxon rank-sum test, p > 0.05). Interestingly, the participant could describe why failed trials were unsuccessful with ICMS feedback, but was unable to do so without it. That is, if the instructed target was “firm pinch”, the pinch fingers should exert a force from 36–60% of the maximum torque while the scoop fingers should not make contact with the object at all. Immediately following a failed trial, the participant could report that he did not exert enough force with the pinch fingers but also made contact with the scoop fingers. Reports of this nature were voluntarily

Fig. 5 Performance on real-time ICMS feedback tasks. a. Confusion matrix of participant’s ability to correctly identify which robotic finger was being touched, using the data reported by Flesher et al. b. Proportion of trials correct in the continuous force matching iBCI task under the two feedback paradigms. Proportion correct was calculated on a block-by-block basis, with blocks consisting of 6 trials each. These six trials included one repetition of each combination of hand posture and force level. ICMS feedback was either provided or not for each block

52

S. Flesher et al.

provided and demonstrated that ICMS feedback provided additional interpretable information, even if the information could not be acted upon. With no object present, the subject was very proficient at achieving position-only targets, with a success rate of 93.5 ± 7.4% (mean ± standard deviation).

4 Discussion ICMS delivered to area 1 of S1 has the potential to provide behaviorally relevant somatosensory feedback to people who use iBCI systems. We found that percepts were evoked at expected somatotopic locations and that the perceived intensity of stimuli scaled linearly over the tested amplitude range (up to 100 lA). These features enable us to relay both the location and intensity of object contact, two sources of information that would be helpful for BCI users to interact with objects. Further, we have shown that a BCI user can use ICMS feedback, generated by a prosthetic limb in real-time, to improve performance in a simple motor control task. There are many challenges in implementing experiments that involve both neural recording and microstimulation. In these experiments, developing a task that was within the motor control capabilities of the decoder, yet could also benefit from somatosensory feedback without vision being artificially removed, proved challenging. Further, robotics control issues and sensor data stability increase the technical complexity of even simple experiments. Perhaps the most significant challenge is that the optimal way to encode measured sensor data into stimulus trains across many electrodes is unknown. Here, only the most simple stimulus encoding function was tested. Measured reaction torque values from the prosthetic fingers were linearly scaled to the stimulus amplitude even though it has been shown that non-linear function more accurately represents the relationship between stimulus amplitude and skin indentation [17]. The actual neural activity in the cortex in response to skin indentation is a complex pattern that represents the activity of both slowly and rapidly adapting neurons [18]. Encoding prosthetic sensor data using these biomimetic principles may improve the integration of somatosensory feedback into motor control tasks. However, in these experiments, the stimulus pulse frequency was held constant at 100 pulses per second and pulses were delivered synchronously across all electrodes. This synchronous pulsing scheme was used to minimize stimulus artifact. In future, more complex encoding schemes will be tested and it remains to be seen what aspects of the natural code must be replicated to improve the naturalness of sensation or improve sensorimotor integration. Nevertheless, even with the simplistic encoding scheme used here, performance for the two dimensional force matching task was significantly improved with the addition of ICMS feedback. We expect that this is because in the absence of ICMS feedback, it was difficult, if not impossible, for the participant to correct errors in the applied grasp force. This interpretation is supported by verbal reports that the study participant began supplying during the task itself. After trials that included

Intracortical Microstimulation as a Feedback Source …

53

ICMS, the participant often reported why he was unsuccessful. For instance, he might report that he was pinching too hard. Such reports were only occurred when ICMS feedback was provided. In the absence of ICMS feedback, the study participant was unable to provide any information about what was wrong in an incorrect trial. This suggests the feedback was accurately relaying both intensity and location of object contact in such a way that was easily interpretable to the user with no training. The inability to correct these errors, despite accurate knowledge of what needed to occur, may reflect the short duration of the trials (7 s) or an issue with the decoding performance of the controller.

5 Conclusions We have shown that intracortical microstimulation in the hand area of primary somatosensory cortex provides intuitive feedback about the intensity of force applied by each finger individually. This feedback enabled a user to improve his ability to provide metered force to an object with different finger combinations. This proof-of-concept is an important first step in the development of bidirectional neuroprosthetic arms for people with paralysis to allow them to have more natural interactions with their environment and ultimately increase their independence by enabling them to complete a wide variety of tasks without assistance. Acknowledgements This study was funded by the Defense Advanced Research Projects Agency’s (Arlington, VA, USA) Revolutionizing Prosthetics program (contract number N66001-10-C-4056) and Office of Research and Development, Rehabilitation Research and Development Service, Department of Veterans Affairs (Washington DC, USA, grant numbers B6789C, B7143R, and RX720). S.N.F. was supported by the National Science Foundation Graduate Research Fellowship under Grant No DGE-1247842.

References 1. Collinger JL, Wodlinger B, Downey JE, Wang W, Tyler-Kabara EC, Weber DJ, McMorland AJ, Velliste M, Boninger ML, Schwartz AB (2012) High-performance neuroprosthetic control by an individual with tetraplegia. Lancet. doi:10.1016/S0140-6736 (12)61816-9 2. Wodlinger B, Downey JE, Tyler-Kabara EC, Schwartz AB, Boninger ML, Collinger JL (2015) Ten-dimensional anthropomorphic arm control in a human brain-machine interface: difficulties, solutions, and limitations. J Neural Eng 12:016011 3. Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, Branner A, Chen D, Penn RD, Donoghue JP (2006) Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442(7099):164–171 4. Collinger JL, Boninger ML, Bruns TM, Curley K, Wang W, Weber DJ (2013) Functional priorities, assistive technology, and brain-computer interfaces after spinal cord injury. J Rehabil Res Dev 50(2):145

54

S. Flesher et al.

5. Rothwell JC, Traub MM, Day BL, Obeso JA, Thomas PK, Marsden CD (1982) Manual motor performance in a deafferented man. Brain 105(Pt 3):515–542 6. Ghez C, Gordon J, Ghilardi MF (1995) Impairments of reaching movements in patients without proprioception. II. Effects of visual information on accuracy. J Neurophysiol 73: 361–372 7. Sainburg RL, Poizner H, Ghez C (1993) Loss of proprioception produces deficits in interjoint coordination. J Neurophysiol 70:2136–2147 8. Johansson RS, Hger C, Bäckström L (1992) Somatosensory control of precision grip during unpredictable pulling loads. III. Impairments during digital anesthesia. Exp Brain Res 89:204–213 9. Jenmalm P, Johansson RS (1997) Visual and somatosensory information about object shape control manipulative fingertip forces. J Neurosci 17:4486–4499 10. Monzée J, Lamarre Y, Smith AM (2003) The effects of digital anesthesia on force control using a precision grip. J Neurophysiol 89:672–683 11. Chen KH, Dammann JF, Boback JL, Tenore FV, Otto KJ, Gaunt RA, Bensmaia SJ (2014) The effect of chronic intracortical microstimulation on the electrode-tissue interface, J Neural Eng 11:026004. Kim S, Callier T, Tabot GA, Gaunt RA, Tenore FV, Bensmaia SJ (2015) Behavioral assessment of sensitivity to intracortical microstimulation of primate somatosensory cortex. Proc Natl Acad Sci USA, 201509265 12. Dadarlat MC, O’Doherty JE, Sabes PN (2014) A learning-based approach to artificial sensory feedback leads to optimal integration. Nat Neurosci. doi:10.1038/nn.3883 13. Romo R, Hernández A, Zainos A, Salinas E (1998) Somatosensory discrimination based on cortical microstimulation. Nature 392:387–390 14. O’Doherty JE, Lebedev MA, Ifft PJ, Zhuang KZ, Shokur S, Bleuler H, Nicolelis MAL (2011) Active tactile exploration using a brain-machine-brain interface. Nature 479:228–231 15. Flesher SN, Collinger JL, Foldes ST, Weiss JM, Downey JE, Tyler-Kabara EC, Bensmaia SJ, Schwartz AB, Boninger ML, Gaunt, RA (2016) Intracortical microstimulation of human somatosensory cortex. Sci Transl Med 8(361):361ra141–361ra141 16. Velliste M, Perel S, Spalding MC, Whitford AS, Schwartz AB (2008) Cortical control of a prosthetic arm for self-feeding. Nature 453(7198):1098–1101 17. Tabot GA, Dammann JF, Berg JA, Tenore FV, Boback JL, Vogelstein RJ, Bensmaia SJ (2013) Restoring the sense of touch with a prosthetic hand through a brain interface. Proc Natl Acad Sci USA 110(45):18279–18284 18. Saal HP, Harvey MA, Bensmaia SJ (2015) Rate and timing of cortical responses driven by separate sensory channels. Elife 4:e10450

A Minimally Invasive Endovascular Stent-Electrode Array for Chronic Recordings of Cortical Neural Activity Thomas J. Oxley, Nicholas L. Opie, Sam E. John, Gil S. Rind, Stephen M. Ronayne, Anthony N. Burkitt, David B. Grayden, Clive N. May and Terence J. O’Brien

1 Introduction Cross discipline collaboration is heralded as an evolutionary pathway when exploring alternate means of tackling complex biological disorders. The field of Neural Bionics epitomizes this statement, merging the worlds of both engineering and medicine. It has not only made significant advances where conventional methods have failed to address physical disablement but also captured the public imagination in the process. The development of chronic recording devices has facilitated advancement across the field, notably in the areas of movement disorders [1], seizure monitoring and prediction [2, 3], willful control of prosthetics [4, 5], and restorative devices for the auditory [6] and visual [7] senses. Behind every BCI control system lays the heart of its technology: the interface. Chronic interfaces can take the form of scalp electrodes, epidural, subdural and penetrating arrays. Whilst desirably non-invasive, signal quality and positional stability are pitfalls when deliberating on the use of scalp arrays. Penetrating arrays provide superior spatial resolution but must breach the blood brain barrier, as can surface arrays. Consequences of opening this membrane are chronic inflammation and glial scarring. In turn this has negative bearing on the device itself with a gradual reduction T.J. Oxley (&)
N.L. Opie
S.E. John
G.S. Rind
S.M. Ronayne Vascular Bionics Laboratory, Departments of Medicine and Neurology, Melbourne Brain Centre, The Royal Melbourne Hospital, The University of Melbourne, Parkville, VIC, Australia e-mail: [email protected] T.J. Oxley
N.L. Opie
S.E. John
G.S. Rind
S.M. Ronayne
C.N. May The Florey Institute of Neuroscience and Mental Health, The University of Melbourne, Parkville, VIC, Australia S.E. John
A.N. Burkitt
D.B. Grayden
T.J. O’Brien The Department of Electrical and Electronic Engineering, The University of Melbourne, Parkville, VIC, Australia © The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_6

55

56

T.J. Oxley et al.

in the number of viable electrodes for signal acquisition. Placement of intracranial arrays carries further complexities as they require intricate craniotomic surgery. The Vascular Bionics Laboratory proposed a device that could minimise the invasiveness of electrode array placement and circumnavigate the long-term issues associated with crossing the blood brain barrier. The results of the feasibility study were published in Nature Biotechnology in March 2016 [8]. In this chapter, the major outcomes of the paper are outlined.

2 Mapping Cerebral Vessels Yanagisawa et al. have demonstrated the potential for information rich recordings that may be acquired from the anterior sulcus of the motor area [9]. As such, the central sulcus became an appropriate target for the deployment of a BCI. The initial inquiry to using an endovascular approach was primarily the accessibility of suitable cerebral vessels. A study was conducted on vasculature in close proximity to the sensorimotor sulcus. Magnetic Resonance Imaging (MRI) was used to identify the venous anatomy surrounding the sensorimotor cortex in human subjects (n = 50, 34.5, 18–73 years). Veins were categorized as pre or post CSV by their position relative to the central sulcal vein. 4 interconnected structures were identified in the immediate surrounds, the superior sagittal sinus (SSS), precentral sulcal vein (preCSV), central sulcal vein (CSV) and post central sulcal vein (postCSV). Vessel diameters were characterised at set points. The study yielded the results shown in Table 1. An interrogation into the vessel sizes of venous structure in the sheep brain (n = 13, 4.3, 2.5–5 years) revealed the ovine superior sagittal sinus as an apt correlate to human vessels to embark on an animal feasibility study (Figs. 1 and 2).

Table 1 Comparison between the relevant vessel diameters of the ovine and human brain Human vasculature

Proximal diameter (5 mm)

Mid diameter (40 mm)

Distal diameter (80 mm)

Pre CSV CSV Post CSV Ovine vasculature SSS

4.8 mm (4.8–3.3 mm) 4.9 mm (2.2–4.6 mm) 4.8 mm (1.7–3.7 mm) Proximal diameter (0 mm) 2.4 mm (1.6–1.8 mm)

3.3 mm (3.6–8.5 mm) 3.1 mm (2.2–4.5 mm) 3.5 mm (1.6–4.5 mm) Mid diameter (30 mm) 1.7 mm (1–1.5 mm)

2.3 mm (3.4–6.1 mm) 2.3 mm (2.5–5.3 mm) 2.7 mm (1.8–5 mm) Distal diameter (60 mm) 1.1 mm (1–1.2 mm)

A Minimally Invasive Endovascular Stent-Electrode …

57

Fig. 1 MRI based reconstruction of the ovine brain with overlying superior sagittal sinus and branching vessels

Fig. 2 Device Design and Delivery. (Top) Images depicting the deployment action of the interface from within a 1.1 mm lumen catheter. (Bottom) An X-ray image showing the implanted device

58

T.J. Oxley et al.

3 Device Design and Delivery Following the determination of a suitable animal vessel to test the feasibility of an endovascular recording device, attention was turned towards creating a system that could access it. A co-axial catheter system was used to achieve SSS entry via a vascular puncture in the external jugular vein in the neck. X-ray angiography was employed to navigate to the system to the sinus. Early experiments showed that a 4F catheter was the largest sized catheter that could reliably access the ovine sinus without causing vessel damage. To navigate through a catheter to the desired location, the interface had to fit inside a 1.1 mm lumen catheter (044 DAC, Concentric Medical). It then needed to expand from within it with adequate radial force to create an appropriate vessel wall apposition to allow recording. To fit this design requirement, the concept of a stent like scaffold was developed. Precedence exists for this technology as intracranial stents are used to alleviate both arterial and venous complications [10, 11]. Initial interface prototypes were constructed using commercially available, self-expanding, Nitinol stent retrievers (Solitaire SAB, Covidien). Platinum disc electrodes were mounted to the Nitinol scaffold with a trailing lead to transmit acquired signal out of the vessel to a percutaneous connector (Micro plastic series, Omnetics), which exited the skin above the sternocleidomastoid. This design allowed device connection at the convenience of the researcher.

4 Sinus Endothelialisation With the device in situ, the next logical progression became the exploration of its interaction with the vessel wall. The process of incorporation has been demonstrated for arterial stenting with endothelialisation occurring in as little as a week [12]. Cerebral venous stenting has not been as widely characterised thus far. Endothelialisation of the recording head serves the function of removing the structure from direct interaction with the blood flow. Other potential benefits are an increased proximity to the neuronal population and positional stability of the device inhibiting migration. An exploratory study was undertaken to assess the level of endothelial growth and what effect it may have on the electrode interface properties. To quantify growth Synchrotron X-ray imaging was used to determine the distances between the scaffold struts and the vessel lumen. The results show that the struts move away from the lumen and deeper into the vessel in a relatively short amount of time, as can be seen in Table 2 and Fig. 3. Prior to cull, impedance measurements were taken daily for 2 weeks and weekly thereafter. Significant changes (p < 0.0001) were noted at 100 Hz in both impedance measurements and phase angle, within the first 6 days. This is indicative of biological activity at the tissue electrode interface. Further changes from day 8 to day 28 show no significant effects (P > 0.619). These results enforce the

A Minimally Invasive Endovascular Stent-Electrode … Table 2 Scaffold strut to lumen distance showing vessel incorporation over time

Time point

Strut to lumen distance (mean ± SEM)

Day 1 3 Weeks 4 Months

21 µm ± 8 µm (n = 97 struts, 2 subjects) 309 µm ± 22 µm (n = 89 struts, 4 subjects) 320 µm ± 22 µm (n = 72 struts, 4 subjects)

59

Fig. 3 Synchrotron imaging of the superior surface of the brain and sinus showing both vessel patency and device incorporation

Synchrotron data suggesting that incorporation of the device begins almost immediately following implantation and interface stability occurs as early as the first week.

5 Chronic Vessel Patency The assumption of vascular incorporation also brought forth the question of vascular occlusion. A study was undertaken to assess chronic sinus patency to ensure venous drainage was maintained following implantation. Implanted animals (n = 20) were administered with 100 mg Aspirin daily as antiplatelet therapy for the duration of the study. A 3-day loading dose period was provided prior to implantation. Repeated lumen diameter measurements were taken up to a 12-week time point after which the animals were culled and ex vivo brain samples were Synchrotron imaged Fig. 4. • Synchrotron imaging confirmed patency of the SSS in all animals at the loci of stentrode implantation • Imaging analysis (n = 78 slices) on 4 subjects with implantation periods longer than 20 weeks had an observed SSS median lumen diameter of 4.77 mm2 (2.19–6.03 mm2) • Cortical veins entering the sinus demonstrated mild obstruction following implantation with – 92% (11/12) patency after 2 weeks – 63% (5/8) patency after 3 months

60

T.J. Oxley et al.

Fig. 4 Plot of anaesthesia induced neural signal modulation transitioning between deep and light states (left to right). The colour key denotes Minimum Alveolar Concentration (MAC)

• No animals were seen to present with symptomatic behavioural indicators during the study

6 Vascular ECoG Recordings via the a cerebral blood vessel has been described previously; however, the studies conducted have not lasted more than a few hours in highly controlled environments [13–18]. We took several approaches to verify the devices capacity to record cortical signal via the vasculature. Somatosensory Evoked Potentials (SSEPs) were elicited using direct tibial nerve stimulation. Recording were taken over a 28-day period. • SSEP’s were detectable in 98% of all functional channels • Peak to Peak amplitudes experienced no significant change over this period, P = 0.42, n = 703 (linear regression model) indicating stability during recording. • Correlating with endothelialisation and impedance data, SSEP detection was seen to improve in the initial days following implantation i.e. the number of channels detecting SSEP signal increased, possibly due to interface changes. Day1 50% (25–100%), n = 62 channels, 5 subjects Day2 79% (62–96%), n = 44 channels, 5 subjects Day4 92% (77–100%), n = 34 channels, 5 subjects

6.1

Modulatory Effects of Anaesthesia

Comparisons were made between states of deep and light anaesthesia at day 0 and 1 month following implantation. Anaesthesia has been shown by Lukatch to induce

A Minimally Invasive Endovascular Stent-Electrode …

61

theta burst suppression within neural activity [19]. Duration of implantation had a significant effect (F1,8 = 12.2, P = 0.008, n = 5 (2-way ANOVA)) and larger burst-suppression ratio was detected at the 1 month time point. • Day 0 0.12 ± 0.05 (mean ± SEM) • Day 28 0.51 ± 0.07

6.2

Bandwidth and Power Spectra

Recordings were taken from freely moving animals implanted with an endovascular array and epidural and subdural surface recording arrays. Power spectra and bandwidth were assessed within neural signals of subjects in a physically relaxed state. All 3 devices presented with a characteristic 1/f decrease in power typically associated with neural signal. The subdural array performed better on power averages than the endovascular device in the mid–upper gamma bands with no significant differences in the lower bands. There was no significant difference between the endovascular device and the epidural array across the relevant spectra. This finding suggests that any signal attenuation caused by the dura was not furthered by the vessel itself, and that the performance of the endovascular array in its prototype form can match the performance of an epidural surface array.

7 Discussion and Long Term Perspectives BCI research has shown enormous potential in a wide range of applications. Public awareness about BCIs is growing, along with curiosity and enthusiasm. As a medical aid, BCIs can provide fundamental utility to those who have lost or were born without ability. Progressing past a medical aid, its implications are only limited by the mind considering them. The key to pushing this technology forward is to create the safest possible surgical delivery with long term biological and functional stability. Reducing the capacity for clinical complication, long term reliability and ease of use for patient/surgical/research users shall be vital for the growth of a reputable technology. The StentrodeTM system in its initial manifestation is being interrogated for use in patients suffering severe paralysis. It is nonetheless being considered as a platform technology with capabilities to benefit numerous clinical indications. Presently, the group is pushing towards a first in human trial with a highly developed fully implantable system. Successful outcomes will be used to drive future clinical translation.

62

T.J. Oxley et al.

Acknowledgements The Vascular Bionics Laboratory would like to acknowledge all participants and contributors to our work thus far. In particular, we would like to recognise the input of The University of Melbourne • Dept. of Medicine • Dept. of Electrical and Electronic Engineering The Florey Institute of Neuroscience and Mental Health The Royal Melbourne Hospital

References 1. Deuschl G, Schade-Brittinger C, Krack P et al (2006) A randomized trial of deep-brain stimulation for parkinson’s Disease. N Engl J Med 355:896–908. doi:10.1056/ NEJMoa060281 2. Cook MJ, O’Brien TJ, Berkovic SF et al (2013) Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: A first-in-man study. Lancet Neurol 12:563–571. doi:10.1016/S1474-4422(13)70075-9 3. Morrell MJ (2011) Responsive cortical stimulation for the treatment of medically intractable partial epilepsy. Neurology 77:1295–1304. doi:10.1212/WNL.0b013e3182302056 4. Hochberg LR, Serruya MD, Friehs GM et al (2006) Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442:164–171. doi:10.1038/nature04970 5. Yanagisawa T, Hirata M, Saitoh Y et al (2012) Electrocorticographic control of a prosthetic arm in paralyzed patients. Ann Neurol 71:353–361. doi:10.1002/ana.22613 6. Wilson BS, Finley CC, Lawson DT et al (1991) Better speech recognition with cochlear implants. Nature 352:236–238. doi:10.1038/352236a0 7. Weiland JD, Cho AK, Humayun MS (2011) Retinal Prostheses: Current Clinical Results and Future Needs. Ophthalmology 118:2227–2237. doi:10.1016/j.ophtha.2011.08.042 8. Oxley TJ, Opie NL, John SE et al., Minimally invasive endovascular stent-electrode array for high-fidelity, chronic recordings of cortical neural activity. Nat Biotechnol 34:320–327. doi:10.1038/nbt.3428 9. Yanagisawa T, Hirata M, Saitoh Y et al (2009) Neural decoding using gyral and intrasulcal electrocorticograms. Neuroimage 45:1099–1106. doi:10.1016/j.neuroimage.2008.12.069 10. Chimowitz MI, Lynn MJ, Derdeyn CP et al (2011) Stenting versus Aggressive Medical Therapy for Intracranial Arterial Stenosis. N Engl J Med 365:993–1003. doi:10.1056/ NEJMoa1105335 11. Puffer RC, Mustafa W, Lanzino G (2013) Venous sinus stenting for idiopathic intracranial hypertension: a review of the literature. J Neurointerv Surg 5:483–486. doi:10.1136/ neurintsurg-2012-010468 12. van der Giessen WJ, Serruys PW, van Beusekom HM et al., (1991) Coronary stenting with a new, radiopaque, balloon-expandable endoprosthesis in pigs. Circulation 83:1788LP–1798. http://circ.ahajournals.org/content/83/5/1788.abstract 13. Watanabe H, Takahashi H, Nakao M et al (2009) Intravascular neural interface with nanowire electrode. Electron Commun Japan 92:29–37. doi:10.1002/ecj.10058 14. Mikuni N, Ikeda A, Murao K et al (1997) ‘Cavernous Sinus EEG’: A new method for the preoperative evaluation of temporal lobe epilepsy. Epilepsia 38:472–482. doi:10.1111/j.15281157.1997.tb01738.x 15. Bower MR, Stead M, Van Gompel JJ et al (2013) Intravenous recording of intracranial, broadband EEG. J Neurosci Methods 214:21–26. doi:10.1016/j.jneumeth.2012.12.027 16. Boniface SJ, Antoun N (1997) Endovascular electroencephalography: the technique and its application during carotid amytal assessment. J Neurol Neurosurg Psychiatry 62:193–195

A Minimally Invasive Endovascular Stent-Electrode …

63

17. Penn RD, Hilal SK, Michelsen WJ et al (1973) Intravascular intracranial EEG recording technical note. J Neurosurg 38:239–243. doi:10.3171/jns.1973.38.2.0239 18. Driller J, Hilal SK, Michelsen WJ et al., (1969) Development and use of the POD catheter in the cerebral vascular system. Med Res Eng 8:11–6. http://europepmc.org/abstract/MED/ 5823257 19. Lukatch HS, Kiddoo CE, Maciver MB (2005) Anesthetic-induced burst suppression EEG activity requires glutamate-mediated excitatory synaptic transmission. Cereb Cortex 15:1322– 1331. doi:10.1093/cercor/bhi015

Visual Cue-Guided Rat Cyborg Yueming Wang, Minlong Lu, Zhaohui Wu, Xiaoxiang Zheng and Gang Pan

1 Introduction An animal robot is an animal that is connected to a machine system, usually via a brain-computer interface or BCI [1, 2]. This BCI is combined with a device to deliver electrical stimuli to specific brain areas, thereby driving the animal to take actions that are specified by humans [3]. The stimuli are delivered to specific brain areas via implanted electrodes. In particular, animal robots can be controlled by humans to navigate along a specified path. Because of the specific motion and perceptual abilities of animals [4], animal robots have great potential for use in search and rescue applications [5]. A rat robot is a typical animal robot [6], which can navigate along a human-specified route. A major disadvantage is that humans need to identify the arrangements of objects in the environment before giving appropriate instructions to

This is a brief version of the published article in IEEE Computational Intelligence Magazine, 2015 [14]. Y. Wang (&)
X. Zheng Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou, China e-mail: [email protected] X. Zheng e-mail: [email protected] M. Lu
Z. Wu
G. Pan College of Computer Science, Zhejiang University, Hangzhou, China e-mail: [email protected] Z. Wu e-mail: [email protected] G. Pan e-mail: [email protected] © The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_7

65

66

Y. Wang et al.

facilitate navigation in the environment. This limits the possible applications of rat robots in environments that humans cannot observe. In some applications, only a few objects are of interest to a rat robot, such as human faces or indication signs. If the rat robot system can find these objects and a motion action is specified for each object, this would allow the rat robot to perform human-specified navigation automatically. In this preliminary study, we attempt to address this problem. We construct a rat robot where the rat can accept stimuli and perform a few basic actions, such as turning left, turning right, and walking forward. Our novel system makes two major contributions, as follows. (1) To allow the rat robot to find “human-interesting” objects, i.e., the objects that easily attract humans’ attentions such as human faces and indication signs, a miniature camera is mounted on the back of the rat robot and the video captured by the camera is transferred to a computer. Interesting objects in the video, such as human faces and arrow signs, are then identified by object detection algorithms and the detection results are used to control the rat robot. (2) To allow the rat robot to navigate automatically while being guided by the identified objects/cues, we develop a stimulation model that drives the rat robot to perform a unique motion action in response to the detection of an object. A problem with automatic control is that a single stimulus, e.g., a stimulus for turning left, does not allow the rat to perform a successful turning left motion. Humans usually give a series of stimuli to the rat for this purpose, according to the state of the rat and the objects in front of it. Inspired by this manual control process, we develop a closed-loop stimulation model that mimics the human control procedure, which issues a stimulus sequence automatically according to the state of the rat and the objects detected until the rat completes the motion successfully. We refer to our system as a rat cyborg. We evaluate the key features of our rat cyborg through extensive experiments, which demonstrate that the rat cyborg can achieve successful visual cue-guided automatic navigation. The present study comprises neuroengineering [7] and a preliminary study in the new field of “cyborg intelligence,” i.e., incorporating biological intelligence and machine intelligence to obtain more powerful capacities in a system [8–13].

2 Overview Figure 1 shows the three main components of our rat cyborg, i.e., implanted electrodes, a rat-mounted pack, and a computational component. The electrodes are implanted in specific regions of the rat’s brain and electrical stimuli can be delivered to the rat’s brain via the implanted electrodes. The stimuli control the rat cyborg to perform turning left, turning right, or moving forward behaviors.

Visual Cue-Guided Rat Cyborg

67

Fig. 1 Three main components of our rat cyborg system. The electrode picture is taken under a microscope. The rat-mounted pack includes a miniature camera, a wireless module, and a stimulator

The rat-mounted pack consists of the following components (Fig. 1). • A stimulator generating electrical stimuli that are delivered to the rat’s brain via the electrodes implanted in the brain. • A miniature camera that captures real-time video of the scene in front of the rat. The miniature camera measures 20 mm  8 mm  1 mm and its optical axis is in the same direction as the rat’s head. • A wireless module that receives stimulus instructions from a PC and sends videos from the miniature camera to the computational component on the PC to identify interesting objects. Thus, the rat-mounted pack includes an instruction receiver and a video transmitter. The computational component comprises object detection algorithms and a closed-loop stimulation model. Face and colored object detection algorithms are developed to search for faces and colored objects in the video data transferred from the rat-mounted pack. Based on the results, the closed-loop stimulation model estimates the motion state of the rat and determines the stimulus sequence delivered to the rat-mounted pack to stimulate the rat to take the correct actions. All of the rats used in these experiments were cared well by the animal keepers. All of the experiments were performed in accordance with the guidelines issued by the Ethics Committee of Zhejiang University and they complied with the China Ministry of Health Guide for the Care and Use of Laboratory Animals.

3 Basic Rat Robot First, we construct a basic rat robot system, as described in [5]. The rat robot can perform navigation tasks under manual control using the basic instructions: “left,” “right,” and “forward.” The behavior of the rat robot is controlled by the implanted

68

Y. Wang et al.

electrodes and the electrical stimuli sent to the rat brain. In this section, we describe the basic principles of the rat robot system, including the underlying mechanism that connects electrical stimulation with the rat’s actions, the hardware, and the rat robot training procedure.

3.1

Stimulation-Action Principles

Electrical stimuli can be delivered to specific brain regions as rewards [15, 16] and as steering cues [17] to control rat behavior. The MFB in the rat’s brain is known as a pleasure center, thus the application of electrical stimuli to the MFB can be used as rewards [15]. Applying a stimulus to the MFB will increase the level of dopamine (a neurotransmitter with an important role in reward-motivated behavior) in the rat’s brain [15, 18], thereby motivating its motion and reinforcing its behavior [6]. The application of stimulation to the SI can be used as a steering cue [19]. Rats use their vibrissae (whiskers) to sense object surfaces while exploring the environment. The whisker barrel fields in the SI receive projections from the contralateral facial vibrissae. A stimulus on one side of the SI is represented as a virtual touch on the contralateral vibrissae, which makes the rat perform a turn [6]. Thus, three pairs of electrodes are implanted in the rat’s brain. One of the electrodes is placed in the MFB and the other two are implanted symmetrically in the whisker barrel field of the left and right SIs.

3.2

Hardware Modules

The rat robot system contains two hardware modules/circuits: a stimulator and a wireless module. The stimulator circuit generates stimulation pulses. The size of the circuit is minimized by using surface-mounted devices. The main processor in the stimulator is a Mixed-Signal ISP FLASH MCU (C8051F020), which is characterized by its high speed, small size, and low power consumption. These features make it suitable for use in the small rat-mounted pack. This processor has two 12-bit Digital to Analog Converters (DACs), which produce outputs for jitter-free waveform generation. The electrical stimulation pulses exported from the two DACs of the C8051F020 MCU are used to control a constant voltage driver circuit and a constant current driver circuit, thereby producing a monopolar pulse. The pulses of the constant voltage/current pass through three analog switches and are then delivered to the implanted electrodes. The wireless module contains an instruction receiver, which receives the manual control instructions sent from the computer, and a transmitter, which sends the video from the camera to the computer to search for interesting objects (see Sect. 4 for details).

Visual Cue-Guided Rat Cyborg

69

4 Rat Cyborg We construct our rat cyborg system based on the basic rat robot. We intend that the rat cyborg is able to find objects to guide its motion to perform automatic navigation. Thus, we mount a miniature camera on the back of the rat and the captured video is sent to the computer. Two object detection algorithms are developed to search for interesting objects in the video, i.e., faces and colored objects. Furthermore, we want the rat cyborg to perform a unique motion in response to a found object, thereby allowing it to achieve human-specified navigation in an automatic manner where it is guided by the objects of interest. However, it is not sufficient to simply link an object and a single instruction, i.e., “left,” “right,” or “forward,” because the actual motion performed by the rat depends on its current state and its response delay, as well as the instruction. In manual control, humans usually observe the state and responses of the rat, before giving a group of instructions to ensure that the rat achieves the motion successfully. Thus, inspired by the human control procedure, we develop a closed-loop stimulation model to estimate the rat’s motion state and to determine the stimulus sequence that allows the rat cyborg to achieve the corresponding action.

4.1

Object Detection

In this study, we require that the rat cyborg can find two common objects: colored objects and human faces. In colored object detection, a specific color is treated as a random variable c, which conforms to a single Gaussian distribution, c  Nðl; RÞ, where c ¼ ðR; G; BÞT is the color vector, and l and R are the mean vector and the covariance matrix of the distribution respectively. The parameters for a specific color are estimated from a group of natural training images by: l¼

n n 1X 1 X cj ; R ¼ ðcj  lÞðcj  lÞT ; n j¼1 n  1 j¼1

ð1Þ

where n is the total number of color training samples cj . The probability of a pixel with color vector x belonging to the specific color can be computed as: pðxÞ ¼

1 ð2pÞ

1=2

e2ðxlÞ 1

jRj

1=2

T

R1 ðxlÞ

:

ð2Þ

During detection, if pðxÞ is greater than a threshold Tc , the pixel is considered to be this color. When the area of the bounding box of connected pixels in a frame exceeds a threshold Ta , the object is considered to be detected. In our experiments,

70

Y. Wang et al.

Tc is set to 0.9 and Ta is set to 25  25 pixels, which obtains satisfactory performance. For face detection, we develop a modified version of the fast face detection method called soft cascade [20], which employs real AdaBoost as the learning algorithm. The classifier used by soft cascade is: HT ðxÞ ¼

T X

hi ðxÞ;

ð3Þ

i¼1

where x is a test sample and hi ðxÞ denotes a weak classifier. Given a set of rejection thresholds fc1 ; c2 ; . . .; cT g, x will be accepted as a face if and only if every partial sum Ht ðxÞ [ ct . This cascade structure makes the detector fast. The Haar feature is used in the detector and a stump method is used to train the weak classifiers [20]. The detector is trained on a face image set containing more than 20,000 face images and 100,000 non-face images, which measures 10  10 pixels, and the face images were collected from the Internet.

4.2

Closed-Loop Stimulation Model

We want the rat cyborg to perform a unique motion when an object is found. However, an explicit rule that simply issues a “left,” “right,” or “forward” instruction when finding an object does not work well. This is because the action of the rat depends on its current motion and its response delay, as well as on the objects detected. For example, let us suppose that the rat reaches a junction and we want it to take a left turn. One “left” instruction may be sufficient if its head is pointing in the same direction as its body, whereas two “left” instructions would be necessary if its head is currently pointing to the right of its body. In addition, to obtain a successful and continuous motion, the “left” instruction should be followed by a “forward” instruction if the rat’s head actually turns left. The triggering time for the “forward” instruction depends on the response and the action delays of the rat. Thus, during manual control, humans provide a series of instructions to the rat cyborg based on observations of the rat’s states and the objects. Therefore, to construct an automatic instruction control system, we develop a closed-loop stimulation model, which learns the manual control process and imitates the human instruction-issuing process to steer the rat cyborg automatically. As shown in Fig. 2, the loop comprises a rat state extraction module and a human-like instruction model. When an instruction is given, the program checks the rat’s motion state to determine whether it has made the correct motion indicated by the instruction. Based on the changes in the rat’s state and the object detection results, the program uses the human-like instruction model to issue the next instruction, which imitates the decisions made by humans when they observe similar states. The human-like instruction model is learned from a training dataset

Visual Cue-Guided Rat Cyborg

71

Fig. 2 Closed-loop stimulation model

that contains instruction sequences, state change data, and detected objects collected during the manual control procedure. Next, the loop passes to the next round and continues until the mission is complete. Rat State Extraction The rat’s motion state provides feedback for the closed-loop stimulation model to allow appropriate instructions to be given. We define the rat motion state as S ¼ ðh; VÞ, where V is the rat head motion direction and h is the rat head orientation. The motion states of the rat cyborg are extracted using the video captured by the rat-mounted camera. The rat head motion direction is estimated based on the average motion of the feature points in two consecutive video frames. It should be noted that when the rat head moves in one direction, the feature points in the video move in the opposite direction. The main steps used to estimate the motion direction of the rat’s head comprise feature detection, feature tracking, and direction computation. • Feature detection: In this step, we initialize a set of feature points for tracking in consecutive frames. We use the Harris corner detection method [21] to extract corner features from the frame Iðx; y; tÞ. The autocorrelation matrix M is computed from the image derivatives as follows: M¼

X x;y



I2 wðx; yÞ x Ix Iy

 Ix Iy ; Ix2

ð4Þ

where wðx; yÞ is a window function and Ix denotes the partial derivative of the pixel value with respect to the x direction. The corner response is defined as R ¼ detðMÞ  k  traceðMÞ2 , where k is a constant, and detð
Þ and traceð
Þ are the determinant and the trace of a matrix, respectively. The Harris detector finds the points where the corner response function R is greater than a threshold, and it takes the points with the local maxima of R as the corner feature points. • Feature tracking: The Lucas-Kanade method [22] is applied to compute the optical flow between consecutive frames, Iðx; y; tÞ and Iðx; y; t þ 1Þ. We assume that u and v are the x and y components of the velocity of the corner feature ðx; yÞ. Thus, we have Ix u þ Iy v þ It ¼ 0. This equation is computed over a 5-by-5 window around the pixel ðx; yÞ, thereby yielding the following overconstrained system:

72

Y. Wang et al.

Fig. 3 Rat head orientation estimation

2   Ix ðp1 Þ u A ¼ b; where A ¼ 4 n v Ix ðp25 Þ

2 3 3 Iy ðp1 Þ It ðp1 Þ n 5; b ¼ 4 n 5: Iy ðp25 Þ It ðp25 Þ

ð5Þ

The solution is ½u vT ¼ ðAT AÞ1 AT b. The corresponding pixel in Iðx; y; t þ 1Þ of the corner ðx; yÞ is then found using u and v. • Direction computation: The corner feature point ðx1 ; y1 Þ in image Iðx; y; tÞ and its corresponding pixel ðx01 ; y01 Þ in the next frame Iðx; y; t þ 1Þ form a vector v1 ¼ ða1 ; b1 Þ, which indicates the motion of the feature point between the two frames. The rat’s head motion direction V is calculated Pn as the opposite direction of the average feature point motion, i.e., V ¼  i vi =n, where n denotes the number of feature points. For the rat head orientation, the line between the rat cyborg and the current target is considered to be the reference direction. The rat’s head orientation is defined as the angle h between the camera’s optical axis and the reference direction, as shown in Fig. 3. Assume that the position of the target in the frame is d pixels from the center in the x direction. The rat’s head orientation h is computed as h ¼ arctanðd=f Þ, where f is the focal length. If the rat cyborg deviates from the reference direction by a large distance, the target will move outside the video frame. In this case, the distance d is estimated by the last target offset distance dold and the rat’s head motion, d ¼ dold  Vx , where Vx is the x component of V. Human-like Instruction Model In the closed-loop stimulation method, the human-like instruction model issues an instruction, given a rat state and an object. This method operates in a similar manner to humans when they encounter a similar situation during the manual control of a rat cyborg. In the manual control process, after the instruction Ci1 is issued to the rat cyborg, its posture changes from the state Si1 to the next state Si . Next, humans observe the change in the rat state and determine the current instruction Ci to adjust the incorrect action or to reward the correct action. Thus, the state change is an important factor when deciding the current instruction. In addition, the current

Visual Cue-Guided Rat Cyborg

73

object affects the selection of the instruction. Different objects are related to different motion expectations for the rat cyborg. Thus, different instructions will be sent even if the same state change is observed for different objects. Therefore, the current motion expectation is treated as another factor when deciding the current instruction. We assume that the state change extracted by our method is DSi ¼ fhi  hi1 ; Vi  Vi1 g, the current motion expectation Ei indicated by the object detection result is one of Forward (0), Left Turn (-1), and Right Turn (1), and the current instruction issued by humans is Ci . We use Xi ¼ ðDSi ; Ei Þ as input features and Ci as output labels to train a support vector machine (SVM) classifier to construct the human-like instruction model. Because there are three possible values for an instruction: C (Left (-1), Right (1), or Forward (0)), we finally build a three-class classifier using a one-against-all scheme. The training data were collected during the manually controlled navigation of the rat cyborg. Both the control instructions and the videos from the rat-mounted camera were recorded. The rat motion state Si that corresponded to each instruction Ci was extracted from the videos and used to compute the rat state change DSi . The action expectation Ei was obtained based on the object detection results. In the testing stage, we computed the real-time state change DS and motion expectation E of the rat cyborg. The instruction for the rat cyborg was then obtained based on the classification results produced using our model.

5 Experiments In this section, the accuracy of the rat state extraction procedure and the classification performance of the human-like instruction model were evaluated. Finally, we developed a video showing the rat cyborg performing full cue-guided navigation tasks.

5.1

Evaluation of the Rat State Extraction Method

The rat states provide important information that allows the closed-loop model to issue suitable stimulus instructions. In this section, we present an assessment of the accuracy of our state extraction method. This experiment was conducted using a four-armed maze (see Fig. 4a). We designed eight routes for rat state estimation. For each arm of the maze, the rat cyborg was initially placed at the end and a colored arrow was placed at a junction in the maze, where the rat cyborg was required to move from the starting point to the end of the adjacent arm indicated by the direction of the arrow, as shown in Fig. 4a. There were two possible directions for the arrow, so each arm had two possible routes. Thus, there were eight routes in total for the four arms. As the rat cyborg traversed the routes, we continuously

74

Y. Wang et al.

Fig. 4 a Estimation of the rat’s head orientation (viewed from the top by a bird’s eye camera). Left Head orientation estimated using the rat-mounted video (blue arrow) and the top-mounted video (yellow arrow) when the sign was visible. Right Estimated orientations when the sign was not visible. b Estimation of the rat’s head motion direction (viewed from the rat-mounted camera). Left Corner features (red dots) detected in the first frame. Right Original feature location (red dots), the registered features (green dots) in the next frame, and the estimated rat’s head motion direction (yellow arrow)

Table 1 Average difference and standard deviations between the rat’s head orientations estimated from the rat-mounted camera and from the top-mounted camera, and the accuracy of the estimated rat’s head motion direction Trial 1– 4

R1

AD/SD* ACC AD/SD ACC AD/SD ACC AD/SD ACC

8.9/6.6 88.4 8.7/6.6 91.0 10.6/7.3 94.2 8.3/7.2 86.1

R2

R3

R4

13.1/6.1 12.7/6.9 7.7/5.1 84.7 82.5 93.3 6.9/6.0 8.4/6.8 7.7/5.3 89.8 90.8 91.7 9.7/6.9 11.0/7.3 10.1/7.0 89.8 90.0 91.6 13.6/7.2 8.6/5.9 8.5/6.7 85.1 5 94.2 91.7 88.2 [*] AD denotes the average difference (degree), SD denotes Accuracy (%)

R5

R6

R7

R8

10.2/6.7 91.2 8.2/5.5 93.6 8.4/6.8 93.3 11.0/6.7 89.6

8.4/6.3 88.9 8.6/6.0 84.2 11.9/6.4 85.2 8.4/5.9 91.7

9.1/6.2 89.3 11.5/6.2 85.2 10.9/7.0 87.2 9.9/6.6 89.4

9.7/6.7 92.3 11.4/7.1 90.7 11.5/7.0 92.1 8.1/6.4

denotes the standard deviation, and ACC

estimated the rat’s head motion direction V and the rat’s head orientation h from the videos recorded by the mounted camera (see Fig. 4a, b). We tested each route four times, thus there were four trials and 32 routes. To obtain the ground truth for V and h, we used a bird’s eye camera, which was mounted above the scene, to record videos while the rat cyborg performed the tests. In these stable videos, we labeled the rat’s head in the first frame and used the Lucas-Kanade method [22] to track the head and to compute the rat’s head orientations h. These results were compared with the results estimated from the rat-mounted camera and we computed the average differences and standard deviations, as shown in Table 1. For the rat’s motion direction V, a similar method also obtained the rat’s head motion directions from the videos captured by the bird’s eye camera. However, it should be noted that the motion direction is in a vector space and the computational results obtained from the videos use different scales compared with those computed from the videos recorded by the rat-mounted camera

Visual Cue-Guided Rat Cyborg

75

because the two videos use different views and different cameras. Thus, we performed a qualitative comparison to determine whether the two motion estimates were in the same direction: “left” or “right.” Table 1 shows the average difference and standard deviations between the rat’s head orientations estimated from the rat-mounted camera and those from the bird’s eye camera, as well as the accuracy of the estimates of the rat’s head motion direction. In all trials, the average differences are usually about 8 degrees. These differences generally have trivial effects in determining whether the rat’s head is currently located left or right of its body. On average, approximately 90% of the rat’s motion directions are estimated correctly. Because the closed-loop model continuously estimates the rat state, one estimation error may be followed by several correct estimations. The error action caused by an error state can be rectified by the subsequent estimations in the model. Thus, the performance can satisfy the requirement of the closed-loop stimulation model.

5.2

Evaluation of the Closed-Loop Stimulation Model

During the evaluation of the closed-loop stimulation model, we manually controlled the rat cyborg to navigate the four-armed maze and a small urban planning model (Fig. 5) to collect data for training the human-like instruction model. After training, we employed the unused data to perform an off-line test of the human-like instruction model. Next, we tested whether the closed-loop stimulation model could automatically direct the rat cyborg to perform a single action or two successive actions given one or two object(s) and the rat’s states. Several simple scenes were designed for this experiment, each of which contained one or two object(s), and the rat cyborg was required to perform a single action or two actions. To train the human-like instruction model and to verify its performance in an off-line test, we performed 60 trials in the small urban planning model and the four-armed maze. In these trials, we placed the pictures of colored arrows and human faces in different positions. The rat cyborg was manually controlled to walk toward the arrow pictures, to turn in the directions indicated by the arrows, and it finally reached the human face target. During navigation, we collected the instructions issued by humans and the videos from the rat-mounted camera. The

Fig. 5 Automatic cue-guided navigation. The rat cyborg was expected to follow the signs to reach the target face

76

Y. Wang et al.

Table 2 Average confusion matrix for the off-line instruction classifications obtained using our human-like instruction model %

Forward

Left

Right

Forward Left Right

93.83 11.81 12.37

2.95 88.19 0

3.22 0 87.63

Table 3 Comparison of the success rates obtained for motions using automatic control with our method and manual control based on four simple tasks

R1

R2

Left Turn Right Turn Left ! Right Right ! Left Left Turn Right Turn Left ! Right Right ! Left

Ours Success/Total

Ours Speed (m/min)

Manual Success/Total

Manual Speed (m/min)

42/50 44/50 15/20 17/20 45/50 44/50 16/20 15/20

2.88 2.60 2.83 2.33 3.78 3.42 3.26 3.34

47/50 48/50 17/20 19/20 49/50 43/50 18/20 18/20

2.84 2.63 2.89 2.38 3.85 4.10 3.74 3.34

objects in the videos were detected using the object detection methods and the rat’s states were extracted. For each manually issued instruction, we determined the synchronous rat state changes and the objects detected to form a dataset. There were 1171 instructions in 60 trials, thus the dataset contained 1171 samples. We selected 574 random samples as the training data to learn the human-like instruction model and used the remaining 597 samples as testing data. The off-line test results are shown in Table 2. There are three types of instruction: “Forward,” “Left,” and “Right.” The average confusion matrix shows that the accuracies of classification for the three instructions are 93.83%, 88.19%, and 87.63%, respectively. To verify whether the closed-loop stimulation model could automatically direct the rat cyborg to perform a specified motion, we designed four simple routes: a single left/right turn in the four-armed maze and a left/right turn followed by a right/left turn in the small urban planning model. The pictures of arrows were placed at junctions to indicate the motion directions. We used two rat cyborgs and tested each rat cyborg on the first two routes 50 times and on the other two routes 20 times. If the rat cyborg completed the motion successfully, we treated it as a successful trial. We recorded the time costs and speed during each trial to evaluate the efficiency of the closed-loop model. Moreover, the same experiments were conducted using manual control for the purposes of comparison. Table 3 compares the success rates for achieving specified motions using our automatic control method and manual control based on four simple tasks. In all cases, the success rates with the closed-loop stimulation model are very similar to those with manual control. The speed of motion completion is also similar to the two methods. In some cases, our method is even faster than manual control. This

Visual Cue-Guided Rat Cyborg

77

indicates that the closed-loop stimulation model can automatically control the rat cyborg to perform specified motions. A video demo of the automatic navigation process can be found at the following link: http://www.cs.zju.edu.cn/gpan/demo/RatCyborg.mp4.

6 Conclusion and Discussion In this study, we have developed a rat robot system, called a rat cyborg, which is able to find colored objects and faces using a miniature camera. The detection results have been used to trigger stimuli to guide the behavior of the rat cyborg based on a closed-loop model. Our extensive experiments demonstrate that the rat cyborg is capable of performing visual cue-guided automatic navigation. This work could inspire totally new search and rescue applications, such as finding victims trapped by earthquake debris. Acknowledgements This work was supported by the grants from the National 973 Program (no. 2013CB329500), National Natural Science Foundation of China (No. 61673340) and Zhejiang Provincial Natural Science Foundation of China (LZ17F030001, LR15F020001)

References 1. Bin G, Gao X, Wang Y, Hong B, Gao S (2009) VEP-based brain-computer interfaces: time, frequency, and code modulations [research frontier]. IEEE Comput Intell Mag 4(4):22–26 2. Wolpaw J, Wolpaw EW (2012) Brain-computer interfaces: principles and practice. Oxford University Press 3. Holzer R, Shimoyama I (1997) Locomotion control of a bio-robotic system via electric stimulation. IEEE/RSJ Int Conf Intell Robots Syst 3:1514–1519 4. Paxinos G (2004) The rat nervous system. Academic Press 5. Feng Z, Chen W, Ye X, Zhang S, Zheng X, Wang P, Jiang J, Jin L, Xu Z, Liu C, Liu F, Luo J, Zhuang Y, Zheng X (2007) A remote control training system for rat navigation in complicated environment. J Zhejiang Univ Sci A 8(2):323–330 6. Talwar S, Xu S, Hawley E, Weiss S, Moxon K, Chapin J (2002) Behavioural neuroscience: rat navigation guided by remote control. Nature 417(6884):37–38 7. Li Z, Hayashibe M, Fattal C, Guiraud D (2014) Muscle fatigue tracking with evoked EMG via recurrent neural network: Toward personalized neuroprosthetics. IEEE Comput Intell Mag 9 (2):38–46 8. Wu Z, Pan G (2013) Smartshadow: models and methods for pervasive computing. Springer 9. Wu Z, Pan G, Zheng N (2013) Cyborg intelligence. IEEE Intell Syst 28(5):31–33 10. Wu Z, Pan G, Principe JC, Cichocki A (2014) Cyborg intelligence: Towards bio-machine intelligent systems. IEEE Intell Syst 29(6):2–4 11. Wu Z, Yang Y, Xia B, Zhang Z, Pan G (2014) Speech interaction with a rat. Chin Sci Bull 59 (28):3579–3584 12. Wu Z, Zhou Y, Shi Z, Zhang C, Li G, Zheng X, Zheng N, Pan G (2016) Cyborg intelligence: recent progresses and future directions. IEEE Intell Syst 31(6):44–50

78

Y. Wang et al.

13. Yu Y, Pan G, Gong Y, Xu K, Zheng N, Hua W, Zheng X, Wu Z (2016) Intelligence-augmented rat cyborgs in maze solving. PLoS ONE 11(2):e0147754 14. Wang Y, Lu M, Wu Z, Tian L, Xu K, Zheng X, Pan G (2015) Visual cue-guided rat cyborg for automatic navigation. IEEE Comput Intell Mag 10(2):42–52 15. Hermer-Vazquez L, Hermer-Vazquez R, Rybinnik I, Greebel G, Keller R, Xu S, Chapin J (2005) Rapid learning and flexible memory in “habit” tasks in rats trained with brain stimulation reward. Physiol Behav 84(5):753–759 16. Reynolds J, Hyland B, Wickens J (2001) A cellular mechanism of reward-related learning. Nature 413(6851):67–70 17. Romo R, Hernández A, Zainos A, Brody C, Lemus L (2000) Sensing without touching: psychophysical performance based on cortical microstimulation. Neuron 26(1):273–278 18. Schultz W (2002) Getting formal with dopamine and reward. Neuron 36(2):241–263 19. Wang Y, Su X, Huai R, Wang M (2006) A telemetry navigation system for animal-robots. Robot 28(2):183–186 20. Bourdev L, Brandt J (2005) Robust object detection via soft cascade. IEEE Comput Soc Conf Comput Vis Pattern Recognit 2:236–243 21. Harris C, Stephens M (1988) A combined corner and edge detector. In: Alvey vision conference, vol. 15. p 50 22. Lucas BD, Kanade T (1981) An iterative image registration technique with an application to stereo vision. IJCAI 81:674–679

Predicting Motor Intentions with Closed-Loop Brain-Computer Interfaces Matthias Schultze-Kraft, Mario Neumann, Martin Lundfall, Patrick Wagner, Daniel Birman, John-Dylan Haynes and Benjamin Blankertz

1 Introduction The ability of modern brain-computer interfaces (BCIs) to study the relationship between brain processes and mental states in real-time and provide immediate feedback to the person has put forth novel application possibilities. While BCI research has primarily been focused on its use as an assistive technology in the medical context with the aim to provide paralyzed patients with a direct communication and control channel (Birbaumer et al. [4], Wolpaw and Wolpaw [28]), since the turn of the century research has expanded towards BCI applications that go beyond control (Blankertz et al. [9], Allison et al. [1], Brunner et al. [12], Blankertz et al. [5]). Control-directed BCIs have been characterized by their “closed-loop” nature (Blankertz et al. [6]), because establishing communication channels relies on feeding the user’s decoded intentions back to the user in real-time. Non-control BCIs, on the other hand, have been predominantly open-loop systems, since their primary goal has been the monitoring and prediction of mental states (Kohlmorgen et al. [19], Müller et al. [23], Schultze-Kraft et al. [27], Naumann et al. [24]), without requiring a direct interaction with the user. A further distinction is that while for control-BCIs the goal is to achieve “explicit control”, non-control BCIs on the other hand aim to exploit “implicit information” obtained from neurophysiological markers in the EEG. M. Schultze-Kraft (&)
M. Neumann
M. Lundfall
P. Wagner
B. Blankertz Neurotechnology Group, Technische Universität Berlin, Berlin, Germany e-mail: [email protected] M. Schultze-Kraft
J.-D. Haynes
B. Blankertz Bernstein Focus: Neurotechnology, Berlin, Germany M. Schultze-Kraft
D. Birman
J.-D. Haynes
B. Blankertz Bernstein Center for Computational Neuroscience Berlin, Berlin, Germany J.-D. Haynes Berlin Center for Advanced Neuroimaging, Berlin, Germany © The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_8

79

80

M. Schultze-Kraft et al.

Here, we present findings from two recent studies, which While the results from one study are reported here for the first time, the findings from the other study have already been published (Schultze-Kraft et al. [26]). Both employ instances of BCIs outside the medical context that are at their very core closed-loop systems, relying on fast responsive feedback. In both experiments, we aimed at detecting movement intentions of subjects in real-time from the ongoing EEG and using this decoded information in order to interact with the subjects’ behavior. The unique possibility to intervene in an experimental paradigm based on the momentary intention or decision state of a person opens the potential for employing such BCIs for multiple purposes.

1.1

EEG Signals Predictive of Movement Intentions

What both studies have in common is that they both aim at detecting movement intentions using two related and well-known motor preparatory signals in the EEG. One is the so-called readiness potential (RP), a slow, negative cortical potential that starts more than one second before voluntary, self-initiated movements and is observed over motor areas in the EEG (Kornhuber and Deecke [20], Cui et al. [14]). This potential gained particular fame in the seminal experiment by Libet et al. [21], who found that the conscious decision to move occurs several hundred milliseconds after onset of the RP. The early onset of the RP suggests that it is a good neurophysiological marker for predicting whether and when a spontaneous movement will occur. About 400 ms before movement onset, the RP suddenly increases its slope and becomes asymmetrically distributed on the scalp, with a stronger negativity over the hemisphere contralateral to the moved body part (Coles et al. [13], Eimer [15]). This late component of the RP has been coined the lateralized readiness potential (LRP). This property of the LRP suggests that it is a good neurophysiological marker for predicting what kind of movement will occur, e.g. whether the left or the right hand will be moved.

1.2

Two Studies

LRP-study In one of the two studies we present here (termed LRP-study), we aimed at predicting a binary movement decision in real-time using the lateralized readiness potential as a predictive signal. The ability to predict a binary movement decision from neural signals in real time before the movement occurs was first demonstrated by Blankertz et al. [7] and later investigated in a systematic study by Maoz et al. [22], where epilepsy patients with implanted intracranial electrodes played a “matching pennies” game against an opponent. In this game, the player wins a fixed amount of money if they raise a different hand than the opponent at the

Predicting Motor Intentions with Closed-Loop …

81

end of a countdown and loses that amount otherwise. Using low-frequency signals from the implanted electrodes allowed them to predict which hand the patient would move 0.5 s before the movement with good accuracy. In the LRP-study presented here, we aimed at implementing the “matching pennies” paradigm, however using the lateralized readiness potential in the EEG as a non-invasive approach (as opposed to the invasive approach used by Maoz et al. [22]) for predicting the laterality of the movement in real-time. The study was conceived as a proof of concept to test whether an EEG potential like the LRP has enough predictive power to allow for a successful prediction of the player’s decision, thus enabling the BCI to win the game. RP-study In the other study (termed RP-study) the BCI was setup to predict whether/when a movement would occur by detecting the occurrence of a readiness potential in the ongoing EEG. Here, the experimental paradigm was designed to address a fundamental question in cognitive neuroscience. It has to date remained unclear whether the onset of the RP triggers a chain of events that unfolds in time and cannot be cancelled, or whether people can still cancel movements after onset of the RP. One intriguing way to test the underlying hypothesis is to interrupt a person with a stop signal once a RP has started, but before they have started the intended movement, thus potentially giving them the opportunity to stop the movement (Haynes [18]). In a study with 10 participants, we implemented this idea with a real-time BCI in order to test the underlying hypothesis (Schultze-Kraft et al. [26]).

2 Methods 2.1

Experimental Task

LRP-study Six subjects participated in the experiment in which they played the “matching pennies” game against the computer. The game consisted of single trials. A trial began with the subject pressing down the bottom leftmost and the bottom rightmost keys of a standard keyboard with the fingertips of the left and right hand, respectively, while resting the hands calmly on the table. This started a three-second countdown that was presented on a computer screen as a continuous shrinking of a horizontal bar. Immediately at the end of the countdown, subjects were asked to raise one of the two hands with a fast movement and high enough “as if to show the palm to the computer”. Subjects were asked to perform the movement precisely timed with the end of the countdown. To ensure this, there was a 100 ms window at the end of the countdown during which the movement was required to occur. If the movement occurred outside this time window or if both buttons were released, the trial was considered invalid. Otherwise, the computer’s choice (left or right) was shown on the screen. If the subject’s choice was the same as the computer’s, this was a win trial for the BCI. Otherwise, it was a win trial for the subject.

82

M. Schultze-Kraft et al.

The experiment was divided into three stages of 100 trials each. During stage I, the choice of the computer was random. After stage I, the recorded data were then used to train a classifier on the LRP in order to learn to discriminate between the movement of either the left or the right hand. During stages II and III, the classifier was applied to the ongoing EEG data, and the output of the classifier at the end of the countdown was taken as the computer’s choice. While subjects were not aware of the change after stage I, after stage II but before stage III they were informed about the origin of the computer’s choices. RP-study During this experiment, participants (N = 10) made spontaneous, self-initiated movements with their right foot, which consisted of pressing a button that was attached to the floor. Subjects were instructed to terminate their decision and withhold any movement whenever a stop signal was elicited on a screen. The task was designed as a “duel” against the BCI. If the subjects pressed the button while a light on a computer screen was green, this was a win trial for the subject. If they pressed the button after the computer had turned the light red (stop signal), this was a lose trial. The experiment had three consecutive stages. In stage I, stop signals were elicited at random onset times. The EEG data from stage I were then used to train a classifier to detect the occurrence of RPs. In stages II and III, movement predictions were made in real-time by the BCI with the aim of turning on the stop signal in time to interrupt the subject’s movement. For details, see Schultze-Kraft et al. [26].

2.2

Data Acquisition

In both studies, EEG was recorded at 1 kHz from 32 (LRP-study) or 64 (RP-study) Ag/AgCl electrodes (EasyCap; Brain Products), respectively, referenced to the nose (LRP-study) or channel FCz (RP-study) and re-referenced offline to a common reference. In the RP-study, EMG was additionally recorded from the calf muscle of the moved foot in order to determine the time of movement onset. The amplified signal was converted to digital (BrainAmp: Brain Products), saved for offline analysis, and simultaneously processed online by the Berlin Brain-Computer Interface toolbox (BBCI1).

2.3

Online Classifier

In both studies, before stage II, a linear classifier was trained using segments of EEG data from trials in stage I. In the RP-study, during stages II and III, the classifier was then applied to the ongoing EEG and its output used to determine

1

https://github.com/bbci/bbci_public.

Predicting Motor Intentions with Closed-Loop …

83

with above chance accuracy if a movement was being prepared, which then elicited a stop signal. For details, see Schultze-Kraft et al. [26]. In the LRP-study, the EEG data used to train the classifier consisted of 600 ms segments preceding the time point of the release of one of the two buttons, where the laterality defined the two classes. The data was then downsampled to 10 Hz and the difference between the channel pairs FC1 − FC2, FC5 − FC6, C3 − C4, CP 1 − CP2 CP5 − CP6 was concatenated to obtain a feature vector, which was used to train a regularized linear discriminant analysis (LDA) classifier with automatic shrinkage (Blankertz et al. [8]). The so-trained classifier was used during stages II and III to predict laterality. Every 10 ms, a classifier output was generated from the immediately preceding 600 ms of EEG data, and the classifier output at the end of the countdown was then used as the computer’s decision in the task.

3 Results 3.1

Mean Event-Related Potentials

Let us first have a look at the two EEG signals relevant in each of the studies. RP-study Figure 1a shows the mean readiness potential for spontaneous, voluntary foot movements. It displays the well-known shape of the RP, which starts to become negative around 1000 ms before movement onset and has its maximum around the time of movement onset. The RP has its highest amplitude at channel Cz and, as expected for foot movements, there is no lateralization of the potential (Brunia et al. [11]). Furthermore, a comparison with EEG segments where no movement occurred shows a very high class discriminability that is already apparent several hundred ms before movement onset. LRP-study Participants performed movements with either the left or the right hand, which resulted in the readiness potential becoming asymmetrical around 400 ms before button release. The difference between the two classes was strongest in channels C3 and C4, respectively (Fig. 1b). This is confirmed by signed r2 values, which furthermore are highest during the time interval −200 to −50 ms w.r. t. the time of button release.

3.2

Performance of Online Predictors

Let us next examine the performance of the real-time BCI in both studies to predict the correct laterality of hand movements and to detect RPs and predict self-initiated movements, respectively. LRP-study The experimental paradigm of the LRP-study required subjects to move either their left or right hand 100 times in each of the three stages. All

84

M. Schultze-Kraft et al. Cz

(a) 0

0.5 2

0

-10

sgn r

µV

-5

-0.5

-15 -1000

-500

0

Time to movement onset [ms]

foot

foot - no move

µV -1.7

(b)

-0.05

C3

2

2

sgn r

1.7

C4

2

1

0.05

1 0.05

0

-1 -2 left right

-3 -4 -600

-400

-200

-1

0

-2

-0.05

-3 -4 -600

0

Time to button release [ms]

left hand

4

-400

-200

0

Time to button release [ms]

right hand

µV -4

sgn r

µV

µV

2

0

left - right

µV -4

4

sgn r -0.04

2

0.04

Fig. 1 Mean readiness potential for foot movements and lateralized readiness potential for movements of left and right hand. a The top panel shows the grand average RP at channel Cz, computed by averaging the EEG signal time-locked to movement onset and baseline correcting in the time interval −1200 to −1100 ms. The color bar on top shows the class discriminability (signed r2 values) between movement and no-movement trials. The two bottom panels show as scalp topographies the spatial distribution of the voltage and signed r2 values, respectively, during the time interval −700 to −500 ms, as indicated in the top panel (dotted lines). b The top panels show the LRP recorded over channels C3 and C4, respectively, time-locked to the button release and baseline corrected w.r.t. time interval −600 to −550 ms. The color bar on the top shows the class discriminability (signed r2 value) between the two classes (left, right). The bottom panels show as scalp topographies the mean voltage (left, middle) and signed r2 value (right), averaged in the time interval −200 to −50 ms w.r.t. time of button release, as indicated in the top panels (dotted lines)

Predicting Motor Intentions with Closed-Loop …

85

subjects moved the right and left hand with approximately equal probability, and there was also no significant difference across stages. During stage I, the computer’s choice was random. As expected—and given the equal probability of subjects to raise either the left or the right hand—the accuracy of the online predictor was 48% (Fig. 2), which was not significantly better than chance (one-sided tð5Þ ¼ 1:87, p ¼ 0:94). During stages II and III, however, when the computer’s choice was controlled by the real-time classifier output, the mean accuracy increased substantially to 62 and 65%, respectively, which in both cases was significantly better than chance (one-sided tð5Þ ¼ 6:52, p\0:001 and one-sided tð5Þ ¼ 3:96, p\0:01). RP-study In the RP-study, the evaluation of the real-time BCI is more complex because the possible trial outcomes were manifold (Fig. 3a). If the button was pressed without a preceding stop signal, the current trial ended. We refer to this as a missed button press trial. If a stop signal was issued and the subject pressed the button during the subsequent second, we term the trial a predicted button press trial. If no button press but an EMG onset occurred despite there being a stop signal we term the trial an aborted button press trial. Otherwise, if no observable movement followed a stop signal we refer to this as an ambiguous trial that reflected either an early cancellation or a false alarm. Furthermore, during stages II and III 40% of trials were silent (not shown here). In these trials, the time of a planned stop signal was recorded but the red stop signal itself was not presented. These trials always ended when the participant eventually pressed the button. Figure 3b shows that roughly 2 out of 3 button presses were missed while during stage I, but only 1 out of 3 button presses were missed during stages II and III. Furthermore, predicted or aborted button presses were almost absent during stage I, while during stages II and III, when the BCI was actively predicting subjects’ movements, they occurred in roughly 20 and 15% of trials, respectively.

3.3

The Cancelling of Self-initiated Movements

The experimental paradigm of the RP-study furthermore allowed us to test whether people are able to cancel self-initiated movements after onset of the RP and, if so, if there is a point of no return. We therefore assessed how the timing of stop signals was related to movement onset (as assessed by EMG). Figure 4 shows that subjects mostly pressed the button if the stop signal occurred after EMG onset (failed cancellations) but that they were able to stop the movement in time if the stop signal occurred earlier around EMG onset (late cancellations). Interestingly, subjects rarely moved despite seeing stop signals earlier than 200 ms before EMG, even though RP onset occurred more than 1000 ms before EMG onset. Examining the distribution of “silent predictions” (cyan distribution) shows that, while a majority of them occurred around movement onset, many also occurred more than 200 ms before EMG onset. This suggests that the BCI was indeed able to predict movements at such early stages and that subjects were caught early enough to cancel their decision without

86

M. Schultze-Kraft et al.

Accuracy (%)

70 60 50 40 30

Stage I

Stage II

Stage III

Fig. 2 Mean accuracy of the online laterality predictor during the three stages. The average across subjects is shown (error bars = SEM). The chance level accuracy at 50% is indicated with a dashed line

(a)

Trial outcomes

(b) 70

Stage I Stage II Stage III

60

Trials (%)

50 40 30 20 10 0

Missed button press

Predicted button press

Aborted button press

Early cancellation / False alarm

Fig. 3 Possible trial types and outcomes. a The four possible trial categories, as detailed in the preceding text. b Percentage of trial outcomes across stages for the four trial categories described in a. All trial categories in one stage (bars of same color) add up to 100%. The average across subjects is shown (error bars = SEM). In missed button press trials, the participant wins. In predicted button press trials, the BCI wins. Aborted button press trials and the ambiguous early cancellation/false alarm trials constitute draws because the participant’s task was to press a button without being detected. Figure reproduced with permission from Schultze-Kraft et al. [26]

any overt sign of movement (Fig. 4, yellow). Evidence for such early cancellations was finally obtained by means of an offline analysis that detected the occurrence of event-related desynchronization (ERD), an EEG marker for movement preparation

Predicting Motor Intentions with Closed-Loop …

Number of predictions

125 100 75

87

Silent predictions Failed cancellations Late cancellations Early cancellations (partly)

50 25 -1000 -800 -600 -400 -200

0

200

400

Time relative to EMG onset (ms)

Fig. 4 Distribution of BCI predictions in four different cases, relative to EMG onset. The cyan line shows the predictions in silent trials which were only recorded but not shown to participants. The red and orange bars show the joint distribution of failed cancellations (stop signals in predicted button press trials) and late cancellations (stop signals in aborted button press trials), respectively. The yellow area indicates the timing of presumed early cancellations. Figure reproduced with permission from Schultze-Kraft et al. [26]

that is independent of the RP (Bai et al. [3]). In conclusion, our results suggest that humans can still cancel voluntary movements even after onset of the RP. However, this is only possible until a point of no return around 200 ms before movement onset.

4 Discussion The two EEG signals used in each of the studies, the readiness potential and the lateralized readiness potential, both share a critical feature: they can predict a specific motor intention several hundred milliseconds before the corresponding action begins. However, they also differ in many ways. While the RP is predictive of whether/when a self-initiated movement will occur, the LRP predicts movement content, i.e. a what decision. This distinction fits well into what has boon described as the what, when, whether model of intentional action (Brass and Haggard [10]). Furthermore, the predictive power of both signals occurs at different time scales. While the early onset of the RP (Fig. 1a) allows us to make predictions as early as several hundred milliseconds before movement onset (a key feature for finding the point of no return in the RP-study), in the LRP-study, the relatively late lateralization of the RP (Fig. 1b) shifts the time window for good prediction accuracies closer to movement onset. Examining the performance of the online predictors in both studies shows that the BCI was successful in making predictions about the subjects’ intentions. During stages II and III, when the BCI was actively predicting subjects’ movement intent, mean prediction accuracies of 62 and 65%, respectively, were achieved in the LRP study. In the RP-study, in those two stages the rate of predicted movements (both

88

M. Schultze-Kraft et al.

completed and cancelled button presses) increased from virtually absent (1.5%) during stage I to around 36%. Furthermore, the offline ERD analysis revealed that the roughly 30% of trials with predictions but no overt movements were in fact in part early cancellations and not merely false alarms. This remarkable performance of the online predictor in the RP-study eventually allowed us to probe the coupling between the RP as a preparatory signal and its corresponding action and identify a point of no return in cancelling self-initiated movements (Schultze-Kraft et al. [26]). With the two presented studies, we demonstrated both the technical feasibility and resulting application possibilities of BCIs capable of real-time prediction and immediate feedback of movement intentions. Early attempts on single-trial EEG aimed at predicting the laterality of finger movements from the lateralized readiness potential with the goal of improving the responsiveness of control-based BCIs (Blankertz et al. [7]). This work led to the development of a system capable of online predictions of externally evoked actions such as in an emergency braking situation (Haufe et al. [16, 17]). Other studies have used event-related desynchronization in the EEG to predict movement intentions from single trials both offline (Salvaris and Haggard [25]) and online (Bai et al. [2]). To the best of our knowledge, the two presented studies are the first studies that demonstrated the successful real-time prediction of when and what movement intentions using the RP and the LRP, respectively. Most importantly, however, the RP-study is the first realization of the idea of employing a real-time, closed-loop BCI as a research tool, thereby paving the way for future experiments that address previously unapproachable questions from cognitive neuroscience. Acknowledgements This work was supported by the Bernstein Focus: Neurotechnology from the German Federal Ministry of Education and Research (BMBF grant 01GQ0850), by the Bernstein Computational Neuroscience Program (BMBF grant 01GQ1001C), the Research Training Group “Sensory Computation in Neural Systems” (GRK 1589/1-2), the Collaborative Research Center “Volition and Cognitive Control: Mechanisms, Modulations, Dysfunctions” (SFB 940/1) and the German Research Foundation (DFG grants EXC 257 and KFO 247).

References 1. Allison BZ, Dunne S, Leeb R, Millán JDR, Nijholt A (2012) Towards practical brain-computer interfaces: bridging the gap from research to realworld applications. Springer Science & Business Media, Heidelberg 2. Bai O, Rathi V, Lin P, Huang D, Battapady H, Fei D-YY, Schneider L, Houdayer E, Chen X, Hallett M (2011) Prediction of human voluntary movement before it occurs. Clin Neurophysiol 122(2):364–372 3. Bai O, Vorbach S, Hallett M, Floeter MK (2006) Movement-related cortical potentials in primary lateral sclerosis. Ann Neurol 59(4):682–690 4. Birbaumer N, Ghanayim N, Hinterberger T, Iversen I, Kotchoubey B, Kübler A, Perelmouter J, Taub E, Flor H (1999) A spelling device for the paralysed. Nature 398 (6725):297–298

Predicting Motor Intentions with Closed-Loop …

89

5. Blankertz B, Acqualagna L, Dähne S, Haufe S, Schultze-Kraft M, Sturm I, U2¢umlic M, Wenzel MA, Curio G, Müller K-R (2016) The Berlin brain-computer interface: progress beyond communication and control. Front Neurosci 10:530 6. Blankertz B, Dornhege G, Krauledat M, Müller K-R, Curio G (2007) The non-invasive Berlin brain-computer interface: fast acquisition of effective performance in untrained subjects. NeuroImage 37(2):539–550 7. Blankertz B, Dornhege G, Lemm S, Krauledat M, Curio G, Müller K-R (2006) The Berlin brain-computer interface: machine learning based detection of user specific brain states. J UCS 12(6):581–607 8. Blankertz B, Lemm S, Treder M, Haufe S, Müller K-R (2011) Singletrial analysis and classiffcation of ERP components - a tutorial. NeuroImage 56(2):814–825 9. Blankertz B, Tangermann M, Vidaurre C, Fazli S, Sannelli C, Haufe S, Maeder C, Ramsey LE, Sturm I, Curio G, Müller KR (2010) The Berlin brain-computer interface: non-medical uses of BCI technology. Front Neurosci 4:198 10. Brass M, Haggard P (2008) The what, when, whether model of intentional action. Neuroscientist 14(4):319–325 11. Brunia CH, Voorn FJ, Berger MP (1985). Movement related slow potentials. II. A contrast between finger and foot movements in left-handed subjects. Electroencephalogr Clin Neurophysiol 60:135–145 12. Brunner C, Birbaumer N, Blankertz B, Guger C, Kübler A, Mattia D, del R. Millán J, Miralles F, Nijholt A, Opisso E, Ramsey N, Salomon P, Müller-Putz GR (2015). BNCI Horizon 2020: towards a roadmap for the BCI community. Brain Comput Interfaces 2 (1):1–10 13. Coles MG, Gratton G, Donchin E (1988) Detecting early communication: using measures of movement-related potentials to illuminate human information processing. Biol Psychol 26 (1):69–89 14. Cui RQ, Huter D, Lang W, Deecke L (1999) Neuroimage of voluntary movement: topography of the Bereitschaftspotential, a 64-channel DC current source density study. NeuroImage 9 (1):124–134 15. Eimer M (1998) The lateralized readiness potential as an on-line measure of central response activation processes. Behav Res Methods Instrum Comput 30(1):146–156 16. Haufe S, Kim J-W, Kim I-H, Sonnleitner A, Schrauf M, Curio G, Blankertz B (2014) Electrophysiology-based detection of emergency braking intention in real-world driving. J Neural Eng 11(5):056011 17. Haufe S, Treder MS, Gugler MF, Sagebaum M, Curio G, Blankertz B (2011) EEG potentials predict upcoming emergency brakings during simulated driving. J Neural Eng 8(5):056001 18. Haynes J-D (2011) Decoding and predicting intentions. Ann N Y Acad Sci 1224(1):9–21 19. Kohlmorgen J, Dornhege G, Braun M, Blankertz B, Müller K-R, Curio G, Hagemann K, Bruns A, Schrauf M, Kincses W (2007) Improving human performance in a real operating environment through real-time mental workload detection. In: Dornhege G, del R. Millán J, Hinterberger T, McFarland D, Müller K-R (eds) Toward brain-computer interfacing. MIT press, Cambridge, MA, pp 409–422 20. Kornhuber HH, Deecke L (1965) Hirnpotentialänderungen bei Willkürbewegungen und passiven Bewegungen des Menschen: Bereitschaftspotential und reafferente Potentiale. Pflügers Arch 284:1–17 21. Libet B, Gleason CA, Wright EW, Pearl DK (1983) Time of conscious intention to act in relation to onset of cerebral activity (readiness-potential). The unconscious initiation of a freely voluntary act. Brain J Neurol 106(3):623–642 22. Maoz U, Ye S, Ross IB, Mamelak AN, Koch C (2012) Predicting action content on-line and in real time before action onset - an intracranial human study. In: Bartlett PL, Pereira FCN, Burges CJC, Bottou L, Weinberger KQ (eds) NIPS, pp 881–889 23. Müller K-R, Tangermann M, Dornhege G, Krauledat M, Curio G, Blankertz B (2008) Machine learning for real-time single-trial EEG-analysis: from brain-computer interfacing to mental state monitoring. J Neurosci Methods 167(1):82–90

90

M. Schultze-Kraft et al.

24. Naumann L, Schultze-Kraft M, Dähne S, Blankertz B (2017) Prediction of difficulty levels in video games from ongoing EEG. In: Gamberini L, Spagnolli A, Jacucci G, Blankertz B, Freeman J (eds) Symbiotic interaction, vol 9961. Springer International Publishing, Cham, pp 125–136 25. Salvaris M, Haggard P (2014) Decoding intention at sensorimotor timescales. PLoS ONE 9 (2):e85100 26. Schultze-Kraft M, Birman D, Rusconi M, Allefeld C, Görgen K, Dähne S, Blankertz B, Haynes J-D (2016a) The point of no return in vetoing self-initiated movements. Proc Nat Acad Sci 113(4):1080–1085 27. Schultze-Kraft M, Dähne S, Gugler M, Curio G, Blankertz B (2016) Unsupervised classification of operator workload from brain signals. J Neural Eng 13(3):036008 28. Wolpaw J, Wolpaw EW (2012) Brain-computer interfaces: principles and practice. OUP, USA

Towards Online Functional Brain Mapping and Monitoring During Awake Craniotomy Surgery Using ECoG-Based Brain-Surgeon Interface (BSI) L. Yao, T. Xie, Z. Wu, X. Sheng, D. Zhang, N. Jiang, C. Lin, F. Negro, L. Chen, N. Mrachacz-Kersting, X. Zhu and D. Farina

1 Introduction Starting from its basic functions of brain-initiated communication and control [1, 2], Brain-computer Interface (BCI) has begun to focus intensively on neurorehabilitation [3, 4, 5], in particular for its online, real-time, and active-involvement features [6, 7, 8]. In this project, the concept of BCI will be further extended to bridge the gap between the patient’s brain and the surgeon, with the clinical applications for online functional brain mapping and monitoring during awake craniotomy brain surgery. We have named the proposed system the “Brain-Surgeon Interface (BSI)”, fully providing the online interaction between the patient’s brain and the surgeon,

L. Yao
N. Jiang
X. Zhu (&) Department of Systems Design Engineering, University of Waterloo, Waterloo, Canada T. Xie
X. Sheng
D. Zhang State Key Lab of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai, China Z. Wu
L. Chen (&) Department of Neurosurgery, Huashan Hospital, Fudan University, Shanghai, China C. Lin Shenzhen Institutes of Advanced Technology Chinese Academy of Science, Beijing, China F. Negro Universita Di Brescia, Brescia, Lombardy, Italy N. Mrachacz-Kersting Department of Health Science and Technology, Center for Sensory-Motor Interaction, Aalborg University, 9220 Aalborg, Denmark D. Farina (&) Department of Bioengineering, Imperial College London, London, UK e-mail: [email protected] © The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_9

91

92

L. Yao et al.

with the purpose of improving traditional brain surgery, especially avoiding errors in the removal of important functional brain tissues. Brain surgery is performed to remove lesions in the brain tissue, mainly due to brain tumor or abnormal discharge region causing epilepsy [9, 10, 11, 12]. Besides identifying to a brain the lesions, other regions responsible for important functions, such as moving, sensation, and language, need also to be identified [13, 14, 15]. Therefore, fMRI is routinely performed to identify these regions by pre-surgical functional brain mapping [16, 17]. Traditionally, the off-line fMRI brain maps will be used to guide the brain surgery when the patients are fully anaesthetized without any conscious activity. Although this “pre-surgical fMRI” is applicable, it has several limitations due to its low-selectivity and the brain shifts after craniotomy. Since the latest developments in anesthesiology, awake brain surgery is now much safer. During surgery, the surgeon could remap the brain using cortical electrical stimulation (ES) [18], together with the “pre-surgical fMRI”. But the cortical ES may cause seizures, which can be dangerous for the subject and the surgery procedure. Moreover, it is usually very time-consuming to get a full mapping. Compared to the “pre-surgical fMRI”, a full mapping is usually very time-consuming, the “intraoperative fMRI” technique has been developed to provide the surgeon with a real-time mapping of the brain areas, and also to check the brain activity when the surgery is finished [19]. In case of partial removal of the lesion, the brain surgery can continue immediately. The development of our ECoG-based BSI overcomes the limitations of the current techniques, fully integrating the interactive nature of BCI system, with fast, online, easy preparation features. The communication between the patient’s brain and the surgeon will be built in a novel way for the first time, with the ultimate goal of a high quality brain surgery. In this study, we present preliminary results of an innovative BSI system. This concept has the potential to be one of the new exciting applications developed from the traditional BCI approach.

2 Motor Cortex Mapping with ERD/S and MRCP During the awake craniotomy surgery, patients were required to perform simple wrist extension tasks, while ECoG and the EMG of the extensor digitorum muscle were concurrently recorded. The starting time of the task was identified by the Teager-Kaiser energy operator from EMG signal. Movement related cortical potentials (MRCP, [0.05 3] Hz) were extracted in single trials [20, 21], as shown in Fig. 1(1). This allowed the possibility of fast motor cortex mapping within one trial. The MRCP signals across all channels are shown in Fig. 1(4). Moreover, the corresponding Event related (De)synchronization (ERD/S) activation [22, 23] is shown in Fig. 1(2), and also across channels in Fig. 1(5). These two signal modalities provide a fast and reliable way for the motor cortex mapping.

Towards Online Functional Brain Mapping and Monitoring …

93

Fig. 1 Neural correlate of movement. 1 Movement related cortical potential (MRCP) with respect to wrist flexion at channel No. 18. 2 Event related spectrum perturbation (ERSP) at channel No. 18, non-significant parts were wiped out under bootstrap significance level of P = 0.01. 3 Electrode array on the cortex. 4 MRCP

3 Sensory Cortex Mapping with ERD/S and SSSEP Mechanical stimulation was applied to the index finger, using a 175 Hz sine carrier wave modulated by a 27 Hz sine wave. Within each trial, two seconds after the onset of each trial, the subject was alerted with a vibration that lasted 200 ms. Then, 2 s later, a stimulation was applied for 5 s. The power spectrum in channel 17 (localized on the sensory cortex) with respect to 27 Hz sustained stimulation is shown in Fig. 2(2). ERD/ERS across all channels are shown in Fig. 2(3). The stimulation evoked SSSEP response has a frequency specific feature [24, 25], complementarily the induced ERD/ERS oscillatory dynamics, which also reflect somatosensory processing [26, 27], have a non-stimulation frequency specific feature for a real-time decoding [28, 29, 30 31]. Therefore, the combination of ERD/S and SSSEP provides a novel sensory cortex mapping.

94

L. Yao et al.

Fig. 2 Neural correlate of tactile sensation. 1 Electrode array on the cortex and number order. 2 Power spectrum at channel No. 17 with respect to 27 Hz tactile stimulation. 3 Event related spectrum perturbation across all channels, non-significant parts were wiped out under bootstrap significance level of P = 0.01. Time zero

4 Online BCI System Between the Brain and the Surgeon Extending the traditional concept of BCI system designed for interaction between the brain and external devices, our current BSI system establishes an interactive channel between the brain and the surgeon, for assistance in precise brain surgery. The system works in an online scenario, and extracts the neural activation information with respect to the given task, and feedback as a dynamic activation map to the surgeon. Moreover, the learning algorithm can help the surgeon decide which lesioned brain tissue to remove. Using motor cortex mapping, with two to three repeated motor tasks (duration of 10–15 s), the activation region can be precisely identified together with its associated brain regions. Using sensory cortex mapping, with sensory stimulations triggered by the surgeon, the activation region can be identified by both the SSSEP response and oscillatory dynamics. This information can then be provided to the surgeon. The surgeon will interact with the patient’s brain through the BSI system online, improving the awake craniotomy surgery by monitoring the brain activation using advanced BCI techniques.

Towards Online Functional Brain Mapping and Monitoring …

95

5 Discussion and Long-Term Perspectives We propose a novel concept of a BCI system for interaction between the brain and the surgeon, with the ultimate purpose of assisting brain surgery, significantly reducing the time needed during surgery mapping and reducing the medical costs. Our pilot studies showed that MRCPs and oscillatory dynamics can be utilized for motor cortex mapping, at a single-trial level. Besides, the sensory cortex can be mapped by SSSEP and oscillatory decreasing when applying a sustained rhythmic sensory stimulation for a dozen seconds. Interestingly, we found an interesting phenomenon related to the ERS response on the motor cortical area (Fig. 2(3), Channel 6), indicating that the motor cortex is suppressed or stayed in an idle state during sensation tasks. This may provide a new way to perform motor cortex mapping using only sensory stimulation. The concept of BCI for neurosurgical purposes will be of great value in actively engaging the patient in the surgical procedure, unlike current methods. Our next challenge will be the translation of this pilot experimental setting into a full clinical system.

References 1. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain-computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791 2. Wolpaw JR, McFarland DJ (2004) Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc Natl Acad Sci U. S. A. 101(51):17849– 17854 3. Daly JJ, Wolpaw JR (2008) Brain–computer interfaces in neurological rehabilitation. Lancet Neurol 7(11):1032–1043 4. Kübler A, Nijboer F, Mellinger J, Vaughan TM, Pawelzik H, Schalk G, Mcfarland DJ, Birbaumer N, Wolpaw JR (2005) Patients with ALS can use sensorimotor rhythms to operate a brain 5. Pichiorri F, Morone G, Petti M, Toppi J, Pisotta I, Molinari M, Paolucci S, Inghilleri M, Astolfi L, Cincotti F, Mattia D (2015) Brain-computer interface boosts motor imagery practice during stroke recovery. Ann Neurol 77(5):851–865 6. Xu R, Jiang N, Mrachacz-Kersting N, Lin C (2014) A closed-loop brain-computer interface triggering an active ankle-foot orthosis for inducing cortical neural plasticity 61(7):2092–2101 7. Jiang N, Gizzi L, Mrachacz-Kersting N, Dremstrup K, Farina D (2014) A brain-computer interface for single-trial detection of gait initiation from movement related cortical potentials. Clin Neurophysiol 8. Mrachacz-Kersting N, Jiang N, Stevenson AJT, Niazi IK, Kostic V, Pavlovic A, Radovanovic S, Djuric-Jovicic M, Agosta F, Dremstrup K, Farina D (2015) Efficient neuroplasticity induction in chronic stroke patients by an associative brain-computer interface. J Neurophysiol. doi: 10.1152/jn.00918.2015 9. Behrens E, Schramm J, Zentner J, König R (1997) Surgical and neurological complications in a series of 708 epilepsy surgery procedures. Neurosurgery 41(1):1–9 10. Jacobs J, Zijlmans M, Zelmann R, Chatillon C-É, Hall J, Olivier A, Dubeau F, Gotman J (2010) High-frequency electroencephalographic oscillations correlate with outcome of epilepsy surgery. Ann Neurol 67(2):209–220 11. Jeha LE, Najm I, Bingaman W, Dinner D, Widdess-Walsh P, Lüders H (2007) Surgical outcome and prognostic factors of frontal lobe epilepsy surgery. Brain 130(2):574–584

96

L. Yao et al.

12. Salanova V, Markand O, Worth R (2002) Temporal lobe epilepsy surgery: outcome, complications, and late mortality rate in 215 patients. Epilepsia 43(2):170–174 13. Picht T, Schmidt S, Brandt S, Frey D, Hannula H, Neuvonen T, Karhu J, Vajkoczy P, Suess O (2011) Preoperative functional mapping for rolandic brain tumor surgery: comparison of navigated transcranial magnetic stimulation to direct cortical stimulation. Neurosurgery 69(3):581–589 14. Duffau H (2005) Lessons from brain mapping in surgery for low-grade glioma: insights into associations between tumour and brain plasticity. Lancet Neurol 4(8):476–486 15. Schiffbauer H, Berger MS, Ferrari P, Freudenstein D, Rowley HA, Roberts TPL (2002) Preoperative magnetic source imaging for brain tumor surgery: a quantitative comparison with intraoperative sensory and motor mapping. J Neurosurg 97(6):1333–1342 16. Shinoura N, Yamada R, Kodama T, Suzuki Y, Takahashi M, Yagi K (2005) Preoperative fMRI, tractography and continuous task during awake surgery for maintenance of motor function following surgical resection of metastatic tumor spread to the primary motor area. min-Minimally Invasive Neurosurg 48(2):85–90 17. Adcock JE, Wise RG, Oxbury JM, Oxbury SM, Matthews PM (2003) Quantitative fMRI assessment of the differences in lateralization of language-related brain activation in patients with temporal lobe epilepsy. Neuroimage 18(2):423–438 18. Szelényi A, Bello L, Duffau H, Fava E, Feigl GC, Galanda M, Neuloh G, Signorelli F, Sala F (2010) Intraoperative electrical stimulation in awake craniotomy: methodological aspects of current practice. Neurosurg Focus 28(2):E7 19. Lu J-F, Zhang H, Wu J-S, Yao C-J, Zhuang D-X, Qiu T-M, Jia W-B, Mao Y, Zhou L-F (2013) Awake intraoperative functional MRI (ai-fMRI) for mapping the eloquent cortex: is it possible in awake craniotomy? NeuroImage Clin 2:132–142 20. Niazi IK, Jiang N, Tiberghien O, Nielsen JF, Dremstrup K, Farina D (2011) Detection of movement intention from single-trial movement-related cortical potentials. J Neural Eng 8(6): 66009 21. Xu R, Jiang N, Lin C, Mrachacz-kersting N, Dremstrup K, Farina D (2014) Enhanced low-latency detection of motor intention interface applications 61(2):288–296 22. Pfurtscheller G, da Silva FH (1999) Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin Neurophysiol 110(11):1842–1857 23. Neuper C, Wörtz M, Pfurtscheller G (2006) ERD/ERS patterns reflecting sensorimotor activation and deactivation. Prog Brain Res 159:211–222 24. Müller-Putz GR, Scherer R, Neuper C, Pfurtscheller G (2006) Steady-state somatosensory evoked potentials: Suitable brain signals for brain-computer interfaces? IEEE Trans Neural Syst Rehabil Eng 14(1):30–37 25. Breitwieser C, Kaiser V, Neuper C, Müller-Putz GR (2012) Stability and distribution of steady-state somatosensory evoked potentials elicited by vibro-tactile stimulation. Med Biol Eng Comput 50(4):347–357 26. Houdayer E, Labyt E, Cassim F, Bourriez JL, Derambure P (2006) Relationship between event-related beta synchronization and afferent inputs: Analysis of finger movement and peripheral nerve stimulations. Clin Neurophysiol 117(3):628–636 27. Houdayer E, Degardin A, Salleron J, Bourriez JL, Defebvre L, Cassim F, Derambure P (2012) Movement preparation and cortical processing of afferent inputs in cortical tremor: an event-related (de)synchronization (ERD/ERS) study. Clin Neurophysiol 123(6):1207–1215 28. Yao L, Meng J, Zhang D, Sheng X, Zhu X (2013) Selective sensation based brain-computer interface via mechanical vibrotactile stimulation. PLoS One 8(6) 29. Yao L, Meng J, Zhang D, Sheng X, Zhu X (2014) Combining motor imagery with selective sensation toward a hybrid-modality BCI. IEEE Trans Biomed Eng 61(8):2304–2312 30. Yao L, Meng J, Sheng X, Zhang D, Zhu X (2015) A novel calibration and task guidance framework for motor imagery BCI via a tendon vibration induced sensation with kinesthesia illusion. J Neural Eng 12(1):16005 31. Yao L, Sheng X, Zhang D, Jiang N, Farina D, Zhu X (2016) A BCI system based on somatosensory attentional orientation. IEEE Trans Neural Syst Rehabil Eng 4320(c):1–1

A Sixteen-Command and 40 Hz Carrier Frequency Code-Modulated Visual Evoked Potential BCI Daiki Aminaka and Tomasz M. Rutkowski

1 Introduction A brain computer interface (BCI) is a system that utilizes brain activity to provide direct communication between the mind and an external environment, without involving any muscles or peripheral nervous system fibers [1]. Patients suffering from locked-in-syndrome (LIS) [2] can use BCIs to communicate with their caretakers or complete various simple daily tasks (such as typing text messages, controlling their environments or robotic applications, using the Internet and other mainstream technologies, etc.) [3–6]. BCI can provide practical real-world communication tools for people with amyotrophic lateral sclerosis (ALS) or even some disorders of consciousness (DOCs) that do not require movement—only properly classified brainwaves [1, 7, 8]. We present a new BCI that can use EEG activity elicited by a code-modulated visual evoked potential (cVEP) [4–6]. The work presented here is based on our earlier cVEP approach that we hereby extend to utilize 16 light sources. The cVEP is a natural response that the brain generates when the user focuses attention on avisual stimulus with specific code-modulated sequences, and several groups have used the cVEP in BCIs [9–15]. The cVEP-based paradigm is a type of stimulus-driven BCI, which relies on voluntary attention to specific stimuli to produce distinct patterns of brain activity. BCIs based on cVEP and other stimulus-driven attentional approaches typically require less training than BCIs D. Aminaka
T.M. Rutkowski (&) BCI-Lab, University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki 308-8577, Japan e-mail: [email protected] URL: http://bci-lab.info/ D. Aminaka Intel, Tsukuba, Japan T.M. Rutkowski Cogent Labs Inc, Tokyo, Japan © The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_10

97

98

D. Aminaka and T.M. Rutkowski

based on motor imagery, and work for nearly all healthy users. cVEP-based BCIs can also provide a particularly high information transfer rate (ITR) with reduced concerns about user annoyance and epileptic seizure induced by visual stimulation. A problem that was raised in our previous research was related to a biased classification accuracy caused by the CCA towards some of the BCI commands. The training dataset for a classifier was created from cVEP responses when user gazed at the first flashing LED pattern (the top location in Fig. 1). The remaining training patterns were created by applying circular shifts of the first LED cVEP’s response. This method was responsible for the possible accuracy drop due to a limited number of training examples. We propose in this paper to use the linear SVM-based classifier to improve the cVEP BCI accuracy and to minimize any biases related to potential overfitting problems. We use the RGB light-emitting diodes (LEDs) in order to evoke four types of cVEPs. We also utilize the higher flashing pattern carrier frequency of 40 Hz and compare our results with the classical setting of 30 Hz. This refresh rate was chosen previously due to a limited

Fig. 1 The cVEP BCI experimental set-up. The user wears an EEG cap with eight electrodes covering the visual cortex. The g.USBamp connected to a g.TRIGbox captures EEG and trigger signals, which are then preprocessed by OpenVibe on a first laptop. Segmented and filtered signals are then streamed via UDP to a second laptop running Python-based linear SVM classifier and a speech synthesis application that announces classification results as feedback

A Sixteen-Command and 40 Hz Carrier Frequency …

99

computer display refresh rate of 60 Hz [16]. Moreover, we propose to use the chromatic green-blue stimulus [17] as a further extension in our project and we compare results with the classical monochromatic (white-black) arrangement. The approach presented here is an extension of our previously reported cVEP BCI paradigms [13]. This time, we have reached the 16-command benchmark with a very small visual angle of five degrees between the flashing light sources, as shown in Fig. 1. Our main objective here was to explore the performance of this system in a real-world scenario. In particular, we sought to assess accuracy across nine healthy participants.

2 Methods Brainwave responses in these online BCI experiments were recorded using eight active g.LADYbird EEG electrodes connected to a g.USBamp portable amplifier from g.tec medical instruments GmbH, Austria. Stimulus triggers from 16 cVEP’s generating ARDUINO DUE are captured by g.TRIGbox connected to the g. USBamp. The Ethical Committee of the Faculty of Engineering, Information and Systems at the University of Tsukuba, Tsukuba, Japan approved the experiments. The number sequence spelling (1–16) experiments are conducted based on cVEP responses [13]. The experimental paradigm is implemented in an OpenVibe environment, which sends via UDP 5 * 60 Hz bandpass filtered EEG signals (also with 48 * 52 Hz notch applied) to a linear support vector machine (SVM) program implemented by our team in Python. Classification results are announced by a synthetic voice, as shown in a YouTube video [18].

2.1

Visual Stimulus Generation

In this study, we use m–sequence encoded flashing patterns [9] to create sixteen commands for the cVEP BCI. The m–sequence is a binary pseudorandom code, which is generated using a procedure as follows, xðnÞ ¼ xðn  pÞ  xðn  qÞ; ðp [ qÞ;

ð1Þ

where xðnÞ is the nth element of the m–sequence obtained by the exclusive or (XOR) operation, denoted by  in the Eq. (1), using the two preceding elements indicated by their positions (n − p) and (n − q) in the string. In this project p = 5 and q = 2 are used. An initial binary sequence is decided to create the final m–sequence that used in the following Eq. (1):

100

D. Aminaka and T.M. Rutkowski

xinitial ¼ ½0; 1; 0; 0; 1:

ð2Þ

Finally, the 31 bit–long sequence is generated based on the above initial sequence, as shown in Eq. (2). The interesting feature of the m–sequence approach, which is very useful for the cVEP-based BCI paradigm design, is an unique autocorrelation function. The autocorrelation function has only a single peak at the period sample value. If the m– sequence period is N, the autocorrelation function will result with values equal to 1 at 0; N; 2N; . . . and 1/N otherwise. It is also possible to introduce a circular shift of the m–sequence denoted by s, to create a set of m–sequences with shifted autocorrelation functions, respectively. In this study, the shifted time length has been defined as s ¼ 2 bits. Fifteen additional sequences have been generated using the above shifting procedure, respectively. During the online cVEP-based BCI experiments, the sixteen LEDs continued to flash simultaneously using the time-shifted m–sequences as explained above.

2.2

Classification of cVEP Responses

A linear SVM classifier was used to compare accuracies with a previously successfully implemented CCA method [15]. In the training session, a single dataset containing the cVEP responses to this first flashing LED was used. The remaining sixteen cVEP responses were constructed by circularly shifting the first LED responses. We used the linear SVM classifier to identify the user’s intended target based on the EEG activity elicited by the flickering patterns. The cVEP response processing and classification steps were as follows: 1. For the classifier training purposes, capturing the EEG cVEP y1 obtained in response to the first m–sequence. A procedure to construct the remaining training patterns yi ; ði ¼ 2; 3; . . .; 15Þ, based on the original recorded y1 sequence was as follows: yi ðtÞ ¼ y1 ðt  ði  1ÞsÞ;

ð3Þ

where s was the circular shift and t indicated a position in the sequence. 2. Averaging the captured j cVEPs as yi;j ðtÞ for each target i separately. The averaged responses yi were used for the linear SVM classifier training with cross-validation to avoid overfitting. In this study, there were N = 60 training datasets and the number of averaged responses was M = 10. The averaging procedure was as follows: yi;l ¼

M1 1 l þX y ; M j¼l i;j

where l = 1, 2,…,N – M +1 was the dataset number.

ð4Þ

A Sixteen-Command and 40 Hz Carrier Frequency …

101

3. For test classification purposes, cVEPs during BCI sessions were used. The 10 single cVEP sequences (each around 380 ms long) averaging procedure in online experiments to remove the non-cVEP related noise in EEG has been necessary so far on order to maintain a reasonable final accuracies. Thus, each command could be generated in about 3.8 s with additional one second break added for a comfortable eye-saccade execution between flashing targets. The total single command generation time of 4.8 s allows for a communication rate of 12.5 commands/minute.

2.3

Experimental Procedure

The users were asked to execute only micro eye saccades to gaze directly at one of the 16 LEDs flashing 31-bit long m-sequences with two bits circular shifts applied to differentiate the patterns (only commands #1 and #16 differ by a single bit). The target LEDs are arranged in 4  4 matrix with 15 cm distances (visual angle between LEDs of 5°) and 1.6 m away from user’s eyes, as shown in Fig. 1. To avoid neighboring LEDs to flash similar patterns, the following matrix placement is used: 0

1 B3 B @5 7

9 11 13 15

2 4 6 8

1 10 12 C C: 14 A 16

ð5Þ

3 Results Results from three test sessions are presented in Fig. 2. The results show that all nine healthy participants could successfully use this cVEP BCI with sixteen commands, which was not annoying and presented flicker at the border of human perception. Only one user, in the second trial, scored at a chance level of 6.5%. Also, a single user in a final session achieved 100% accuracy. The majority of the results were well above chance level. The grand mean average accuracy of all experiments was of 51%. A video demonstrating a completely successful result— accurate spelling of sixteen digits—is available on YouTube [18].

102

D. Aminaka and T.M. Rutkowski 16 cVEP BCI user accuracies obtaind with SVM classifier (non-signficant) 100.00

90.00

80.00

Accuracy [%]

70.00

60.00

50.00

40.00

30.00

20.00

10.00 chance = 6.25 0.00

First trial

Second trial

Third trial

Fig. 2 The cVEP BCI accuracy results from three test sessions conducted by each subject in the project. The boxplots visualize mean and standard error intervals. Additionally, all results are depicted in form of small dots (nine subjects for each trial). No significant differences were observed among the test trials

4 Conclusions Based on results with healthy users, we expect that this cVEP BCI application could also have significant potential for clinical applications to support different users in need. We plan to extend the cVEP BCI with sixteen commands that correspond to different applications for musical performances and instrumental applications. During this study, we also observed that the participating users were more motivated to practice, since the random sequence-based visual stimulation had a very user-friendly appearance. This could be especially helpful for patients who find some displays confusing, and could also increase appeal to healthy users. The effects of practice have not been well explored with cVEP BCIs, and might lead to

A Sixteen-Command and 40 Hz Carrier Frequency …

103

improved performance. The approach presented here could conceivably further incorporate an airborne ultrasonic display, such as our system that won the 2014 BCI Research Award [19]. Other promising future research directions include additional communication and control applications, other hybridization with other BCI approaches, new ways to improve ITR, and validation with specific patient groups.

References 1. Wolpaw J, Wolpaw EW (eds) (2012) Brain-computer interfaces: principles and practice. Oxford University Press, New York, USA 2. Plum F, Posner JB (1966) The diagnosis of stupor and coma. FA Davis, Philadelphia, PA, USA 3. Fazel-Rezai R, Allison BZ, Guger C, Sellers EW, Kleih S, Kuebler A (2012) P300 Brain-computer interface: current challenges and emerging trends. Front Hum Neuroeng 5:14. doi: 10.3389/fneng.2012.00014 4. Rutkowski TM (2015) Brain-robot and speller interfaces using spatial multisensory braincomputer interface paradigms. Front Comput Neurosci Conf Abstr 14. http://www.frontiersin. org/10.3389/conf.fncom.2015.56.00014/event_abstract 5. Rutkowski TM, Shinoda H (2015) Airborne ultrasonic tactile display contactless braincomputer interface paradigm. Front Hum Neuroscience 16:3–1. http://www.frontiersin.org/ human_neuroscience/10.3389/conf.fnhum.2015.218.00016/full 6. Rutkowski T (2016) Robotic and virtual reality BCIs using spatial tactile and auditory odd-ball paradigms. Front Neurorobotics 10:20. http://journal.frontiersin.org/article/10.3389/ fnbot.2016.00020 7. Guger C, Spataro R, Allison BZ, Heilinger A, Ortner R, Cho W, LaBella V (2017) complete locked-in and locked-in patients: command following assessment and communication with vibro-tactile P300 and motor imagery brain-computer interface tools. Front Neurosci 11:251. http://journal.frontiersin.org/article/10.3389/fnins.2017.00251/full 8. Lesenfants D, Chatelle C, Saab J, Laureys S, Noirhomme, Q Chapter 6: Neurotechno- logical communication with patients with disorders of consciousness. Neurotechnology and Direct Brain Communication: New Insights and Responsibilities Concerning Speechless But Communicative Subjects, p 85 9. Bin G, Gao X, Wang Y, Li Y, Hong B, Gao S (2011) A high-speed BCI based on code modulation VEP. J Neural Eng 8(2):025015 10. Waytowich N, Krusienski D (2015) Spatial decoupling of targets and flashing stimuli for visual brain-computer interfaces. J Neural Eng 12(3):036006. doi: 10.1088/1741-2560/12/3/036006 11. Reichmann H, Finke A, Ritter H (2016) Using a cVEP-based brain-computer interface to control a virtual agent. IEEE Trans Neural Syst Rehabil Eng 24(6):692–699. doi: 10.1109/ TNSRE.2015.2490621 12. Kapeller C, Kamada K, Ogawa H, Prueckl R, Scharinger J, Guger C (2014) An electrocorticographic BCI using code-based VEP for control in video applications: a single-subject study. Front Syst Neurosci 8:139. http://journal.frontiersin.org/article/10.3389/fnsys.2014. 00139/full 13. Aminaka D, Makino S, Rutkowski TM (2015) Classification accuracy improvement of chromatic and high-frequency code-modulated visual evoked potential-based BCI. In: Guo Y, Friston K, Aldo F, Hill S, Peng H (eds) Brain informatics and health. Lecture Notes in Computer Science, vol 9250. Springer International Publishing, London, UK, pp 232–241. http://dx.doi.org/10.1007/978-3-319-23344-4_23

104

D. Aminaka and T.M. Rutkowski

14. Aminaka D, Shimizu K, Rutkowski TM (2016) Multiuser spatial cVEP BCI direct brain-robot control. In: Proceedings of the Sixth International Brain-Computer Interface Meeting: BCI Past, Present, and Future. Asilomar Conference Center, Pacific Grove, CA USA, Verlag der Technischen Universitaet Graz, 2016, p 70 15. Aminaka D, Makino S, Rutkowski TM (2015) Chromatic and high-frequency cVEP-based BCI paradigm. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE Engineering in Medicine and Biology Society. Milan, Italy. IEEE Press, p 1906–1909. http://arxiv.org/abs/1506.04461 16. Aminaka D, Makino S, Rutkowski TM (2014) Chromatic SSVEP BCI paradigm targeting the higher frequency EEG responses. In: Asia-Pacific Signal and Information Processing Association, 2014 Annual Summit and Conference (APSIPA), Angkor Wat, Cambodia, p 1– 7. http://dx.doi.org/10.1109/APSIPA.2014.7041761 17. Sakurada T, Kawase T, Komatsu T, Kansaku K (2014) Use of high-frequency visual stimuli above the critical flicker frequency in a SSVEP-based BMI. Clin Neurophysiol 18. Rutkowski TM cVEP BCI with 16 commands and 40 Hz carrier frequency, YouTube video available online. https://youtu.be/stS3Qz6ln9E 19. Hamada K, Mori H, Shinoda H, Rutkowski TM (2015) Airborne ultrasonic tactile display BCI. In: Brain-computer interface research. Springer International Publishing, pp 57–65

Trends in BCI Research I: Brain-Computer Interfaces for Assessment of Patients with Locked-in Syndrome or Disorders of Consciousness Christoph Guger, Damien Coyle, Donatella Mattia, Marzia De Lucia, Leigh Hochberg, Brian L. Edlow, Betts Peters, Brandon Eddy, Chang S. Nam, Quentin Noirhomme, Brendan Z. Allison and Jitka Annen

1 Introduction Brain-computer interface (BCI) technology analyzes brain activity to control external devices in real time. In addition to communication and control applications, BCI technology can also be used for the assessment of cognitive functions of patients with disorders of consciousness (DOC) or locked-in syndrome (LIS) [1, 2, 3]; (Ortner et al., in press). The top-right corner of Fig. 1 reflects healthy persons with normal motor responses and cognitive functions. On the bottom-left corner are coma patients without these functions. Patients in the unresponsive wakefulness C. Guger (&)
B.Z. Allison g.tec Guger Technologies OG, Herbersteinstrasse 60, 8020 Graz, Austria e-mail: [email protected] D. Coyle Faculty of Computing and Engineering, School of Computing and Intelligent Systems, Magee Campus, Ulster University, Northland Road, Derry, Northern Ireland BT48 7JL, UK D. Mattia Neuroelectrical Imaging and BCI Lab, Fondazione Santa Lucia, IRCCS, Via Ardeatina, 306, 00179 Rome, Italy M. De Lucia Laboratoire de Recherche en Neuroimagerie (LREN), Department of Clinical Neurosciences, Lausanne University Hospital (CHUV) and University of Lausanne, chemin de Mont-Paisible 16, 1011 Lausanne, Switzerland e-mail: [email protected] L. Hochberg
B.L. Edlow Department of Neurology, Massachusetts General Hospital, 175 Cambridge Street, Boston, MA 02114, USA

© The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_11

105

106

C. Guger et al.

Fig. 1 Motor responses and cognitive functions for coma, unresponsive wakefulness state (UWS), minimally consciousness state (MCS), locked-in syndrome (LIS), complete locked-in syndrome (CLIS)

state (UWS) and minimally consciousness state (MCS) may have conscious awareness but no way to convey their awareness through any kind of movement. These patients should be carefully assessed to make sure that physicians, families and caregivers are aware of their cognitive functions. Cognitive assessment is also important for individuals with LIS, particularly CLIS (complete LIS), to understand which cognitive functions are remaining. Assessment may reveal whether patients understand instructions and conversations, and whether they may be able to communicate.

B. Peters
B. Eddy Oregon Health & Science University, 707 SW Gaines St, #1290, Portland, OR 97239, USA C.S. Nam Brain-Computer Interface (BCI) Lab, North Carolina State University, Raleigh NC 27695, USA Q. Noirhomme
J. Annen Coma Science Group, GIGA Research and Neurology Department, University Hospital of Liège, Liège, Belgium

Trends in BCI Research I: Brain-Computer Interfaces …

107

People with locked-in syndrome (LIS) exhibit quadriplegia and anarthria, but may retain some voluntary movement of the eyes, eyelids, or other body parts. LIS is not a DOC, as persons with LIS are both conscious and aware. However, people with CLIS have no voluntary motor function and are thus unable to communicate or respond to behavioral testing, leading to frequent and often prolonged misdiagnosis [2, 4]. While people with CLIS may retain relatively normal cognitive functioning, shown in Fig. 1, their cognitive abilities and conscious awareness may also be impaired for various reasons. Furthermore, since people with CLIS cannot move or communicate, they may be unable to inform doctors, family and friends that they are in fact able to understand them and wish to play an active role in decisions affecting their lives. The potential of BCI technology to support more accurate and detailed differential diagnosis among DOC and LIS patients is also apparent from the strong recent interest from the BCI community. In addition to numerous publications from different groups (reviewed in 2), there was considerable interest in this topic at the Sixth International BCI Meeting in 2016, including a workshop and day-long Satellite Event that presented the latest advances. In 2017 alone, this topic has been or will be presented in at least a dozen major conferences to our knowledge, including the Seventh International BCI Conference, Society for Neuroscience annual conference, and Human-Computer Interaction International (HCII) annual conference. This research direction was also recognized in our most recent book in this series [3]. Thus, the use of BCI technology for improved diagnosis and related goals for persons with DOC and LIS has become a prominent trend within the BCI research community. The primary goal of this article is to summarize new research results from several top groups in this field, along with commentary and future directions. First, we describe a commonly used platform for DOC assessment and communication called mindBEAGLE.

2 DOC Assessment and Communication Platform from g.tec Some of the results presented here used the mindBEAGLE system. mindBEAGLE is an electro-physiological test battery for DOC and LIS patients that can use four approaches to assess conscious awareness: (i) auditory evoked potentials (AEP); (ii) vibro-tactile evoked potentials with 2 tactors—VT2; (iii) vibro-tactile evoked potentials with 3 tactors—VT3 and (iv) motor imagery (MI). The system consists of a biosignal amplifier, an EEG cap with active electrodes, the BCI software that analyzes the data in real-time, in-ear phones for the auditory stimulation, and 3 vibro-tactile stimulators (tactors). In the AEP approach, a sequence of low (non-target) and high (target) tones is presented to the patient and evoked potentials are calculated. The BCI classifier attempts to identify the target tone based on EEG

108

C. Guger et al.

data, leading to accuracies between 0 and 100%. Chance accuracy in this task is 12.5%, and the threshold for significant communication depends on the number of trials, but high accuracy may reflect conscious awareness. In the VT2 approach, one tactor is mounted on the right hand (target) and receives 10% of the stimuli and one tactor is mounted on the left hand and receives 90% of the stimuli (non-target). Then the patient has to silently count the right hand stimuli to elicit a P300 response that the BCI system can detect. In the VT3 approach, one tactor is mounted on the left hand (10% of stimuli), one tactor is mounted on the right hand (10% of stimuli) and one tactor is mounted on the spine or leg (80% of the stimuli) [5, 6]. Now the patient can count the stimuli on the left hand to say YES and can say NO by counting right hand stimuli. The motor imagery paradigm verbally instructs the patient to imagine either left or right hand movements and the BCI system classifies the data [7]. The top of Fig. 2 shows results for an UWS patient with no reliably discriminable Evoked Potentials (EPs) and a very low BCI accuracy for AEP and VT2 testing. Although this patient shows some differences in the EPs, the intertrial variability was very high. The bottom of Fig. 2 shows the results for an MCSpatient, which look like results from a healthy control. These results indicate that this MCS- patient could follow the experimenter’s instructions, and thus is able to understand conversations. After a successful assessment run, mindBEAGLE can also be used for communication. In this case, the patient is asked a question and can answer YES or NO by attending to vibrations of either the left or right tactor. Similarly, the patient can use the MI approach by imagining a left or right hand movement to say YES and NO. The testing battery gives important information about a patient’s cognitive functions and ability to follow conversations. Furthermore, it can allow communication and identify fluctuations in cognitive function. Of special importance is that mindBEAGLE provides a standardized approach for testing patients. Currently the system is being validated in 10 centers in China, Germany, Austria, Italy, Belgium, France, Spain and the USA. One related direction that was very recently published extended mindBEAGLE technology to provide communication for persons with complete locked-in syndrome (CLIS). We showed that two of three patients with CLIS could communicate using the mindBEAGLE system [8]. This is an exciting development, because BCI technology had not yet been well validated with persons with CLIS. Consistent with the results presented above, the MI approach was not effective in the CLIS patients, but vibrotactile approaches were. We are now working with additional patients and considering new paradigms to improve communication.

Trends in BCI Research I: Brain-Computer Interfaces …

109

UWS patient

MCS patient

Fig. 2 AEP and VT2 results for one UWS (unresponsive wakefulness state) and one MCS (minimally conscicousness state) patient. The top curve shows the classification accuracy on the y-axis and the number of target stimuli on the x-axis. The bottom curve shows the EPs for target (green) and non-target (blue) stimuli. Green shaded areas reflect a significant difference between target and non-target EPs

110

C. Guger et al.

3 DOC Assessment at Ulster Initial research at Ulster [9] reported successful results with BCI-based motor imagery (MI) training in a patient who had MCS using sensorimotor rhythm (SMR) feedback. This result suggested that feedback could raise patients’ awareness about the potential for BCI technology to impact their conditions, and could be effective in a detection of awareness protocol involving motor imagery BCIs. Subsequently, four MCS patients (3 male; age range, 27–53 yr; 1–12 yr after brain injury) participated in multiple sessions with sensorimotor rhythm (SMR) feedback, to determine whether BCI technology can be used to increase the discriminability of SMR modulations [10, 11]. The study had three objectives: (1) To assess awareness in subjects in MCS (initial assessment); (2) To determine whether these subjects may learn to modulate SMR with visual and/or stereo auditory feedback (feedback sessions) and (3) To investigate musical feedback for BCI training and as cognitive stimulation/interaction technology in disorders of consciousness (DOC). Initial assessment included imagined hand movement or toe wiggling to activate sensorimotor areas and modulate SMR in 90 trials, following the protocol described in [12]. Within-subject and within-group analyses were performed to evaluate significant brain activations. A within-subject analysis was performed involving multiple BCI training sessions to improve the user’s ability to modulate sensorimotor rhythms through visual and auditory feedback. The sessions took place in hospitals, homes of subjects, and a primary care facility. Awareness detection was associated with sensorimotor patterns that differed for each motor imagery task. BCI performance was determined from mean classification accuracy of brain patterns using a BCI signal processing framework with a leave-one out cross-validation [10]. All subjects demonstrated significant and appropriate brain activation during the initial assessment without feedback. SMR modulation was observed in multiple sessions with auditory and visual feedback. Figure 3 shows results for subject E (19 sessions), showing that accuracy improves over time with auditory but not visual feedback. In conclusion, the EEG-based assessment showed that patients who had MCS may have the capacity to operate a simple BCI-based communication system, even without any detectable volitional control of movement. All EEG-based awareness detection studies prior to this research did not provide real-time feedback to the patient during the assessment. This research was the first to demonstrate stereo auditory feedback of SMR in MCS patients, allowing the patient to hear the target and feedback, which could be useful in patients who cannot use visual feedback. As many DOC patients have limited eye gaze control and/or other visual system impairments, visual feedback is often unsuitable for them. We used musical auditory feedback in the form of a palette of different musical genres. This enabled us to open a dialogue with the care teams/families on musical preference, discussed in the presence of the patient, to enhance attentiveness and engagement. Anecdotal evidence indicates that musical feedback could help engage DOC sufferers during BCI training and improve BCI performance. A quote from one of the families of participants in our study is published in a recent report [13].

Trends in BCI Research I: Brain-Computer Interfaces …

111

Fig. 3 BCI accuracy for patient E with MCS. Top row BCI accuracy with visual feedback (moving ball or computer game) and baseline accuracy without feedback. Middle and bottom row BCI accuracy with auditory feedback (pink noise, reggae, jazz, hip hop, electronic music, classical music, rock, country,…) and baseline accuracy without feedback. The number of trials in each run after artifact rejection are indicated after the type of feedback (pink noise, hip hop,…). Significant differences between baseline and feedback is indicated with the following notation: ***P  .005; **P  .05; *P  .1

4 DOC Neurophysiological Assessment at FSL The stability of Event-Related Potentials (ERPs) is essential for efficient and effective ERP-based BCI systems, especially when BCIs is applied in a challenging clinical condition such as DOC. In this regard, there are several factors that can limit (if not prevent) the use of BCI technology in patients diagnosed with DOC such as fluctuations of vigilance, attention span and abnormal brain activity due to brain damage (Giacino et al. [21]) to name few. In a recent study conducted at Fondazione Santa Lucia (Rome), Aricò and colleagues [14] showed a significant correlation between the magnitude of the jitter in P300 latency and the performance achieved by healthy subjects in controlling a visual covert attention P300-based BCI. In particular, the higher the P300 latency jitter, the lower the BCI accuracy. We speculated that the covert attention modality increases the variability of the time needed to perceive and categorize the visual stimuli.

112

C. Guger et al.

Currently, we are conducting a neurophysiological (EEG) screening in patients with DOC or functional looked-in syndrome (LIS) who are consecutively admitted at the Post-Coma Unit of the Fondazione Santa Lucia for their standard care rehabilitation. As part of this neurophysiological screening, patients are presented with a simple auditory P300 oddball paradigm, which consists of a binaural stream of 420 standard high tones (440 + 880 +1760 Hz) and 60 deviant complex low tones (247 + 494 + 988 Hz) pseudo-randomly interspersed (50 ms stimulus duration; 850 ms inter stimulus interval). Stimuli are first presented in a passive condition (just listening to auditory stimuli) and then in an active condition (mentally counting the deviant tones). EEG signals are recorded from 31 electrode positions (512 Hz sample rate) with a commercial EEG system. A preliminary (retrospective) analysis of the morphological features (amplitude and latency) of the main ERP waveforms (on Cz) was performed on a convenient sample of 13 admitted DOC patients (9 males; mean age = 47 ± 16; mean time from event = 24 ± 33.5; 5 unresponsive wakefulness state - UWS; 8 Minimally Conscious State - MCS) in their subacute and chronic stages. A wavelet transform method was applied to identify the P300 waveform peak in single trials and thus to assess the magnitude of latency jitter phenomenon [14]. We found significantly higher values of P300 jittering in UWS and MCS patients compared to a control (12 healthy subjects; 6 males; mean age = 30.3 ± 6.5) data set (p < .01), for both active and passive paradigms. Moreover, UWS patients showed significantly higher jitter values compared to MCS patients (p < .01) and to the control group (p < .001) in the active condition. The MCS data also exhibited a significantly higher jitter (p < .05) compared to control data set. A representative case is illustrated in Fig. 4. These preliminary findings prompted us to apply this analysis in a larger cohort of DOC patients to validate this measurement as indicative of different DOC states.

Fig. 4 Data from a representative MCS patient (male, 54 years old, 11 months after a traumatic brain injury). a Average of epochs related to deviant (red) and standard (blue) stimuli. Solid lines and dotted lines reflect the wavelet filtered and non-filtered potentials, respectively. b Single trial epochs associated with deviant stimuli filtered with the wavelet based method. In this case, the P300 peaks exhibited a range of latencies between 350 and 500 ms

Trends in BCI Research I: Brain-Computer Interfaces …

113

Promisingly, we also found a significant negative correlation (r = −.055, p < .05) between the jitter values observed during the active listening condition in both UWS and MCS patients and the relative JFK Coma Recovery Scale-Revised (CRS-r) scores, that is, patients with lower CRS-r scores had higher jitter values [15].

5 DOC Prediction CHUV Early prediction of comatose patients’ outcome is currently based on a battery of clinical examinations that are repeatedly performed during the first days of coma (Rossetti et al., 2010). This includes the evaluation of brain stem reflexes, the motor response, and the electroencephalographic recordings while stimulating patients with arousing stimuli. All these examinations are highly predictive of poor outcome, i.e. death or vegetative state. In this context, the development of markers identifying patients with good outcomes remains challenging. Recently, the neural responses to auditory stimuli as measured by electroencephalography (EEG) over the first days of coma provided promising results for predicting patients’ chance of surviving (Tzovara et al., 2013). This test consists of recording EEG responses to auditory stimuli during the first and second days of coma using a classic mismatch negativity (MMN) paradigm, in which a sequence of identical sounds is rarely (30% of the time) interrupted by a sound that differs from the standard stimulus in terms of pitch, location or duration. The differential response to standard and deviant sounds is measured via a single-trial decoding algorithm, and its performance is evaluated using the area under the Receiver Operating Characteristic (AUC) (Tzovara et al., 2012). The higher the value of the AUC, the more accurate the auditory discrimination between standard and deviant sounds. The test showed that an improvement in auditory discrimination between the first and second days of coma is only observed in survivors. Remarkably, the auditory discrimination per se during the first or second recording was not as predictive as the progression. The test has been extensively validated in a cohort of postanoxic comatose patients treated with therapeutic hypothermia, including 94 individuals (Tzovara et al., 2016). Results (see Fig. 5) showed a positive predictive power of 93%, with 95% confidence interval 5 0.77–0.99 when excluding comatose patients with status epilepticus either during the first or the second day of coma. In addition to the prediction of awakening, recent results revealed that the progression of auditory discrimination during coma provides early indication of future recovery of cognitive functions in survivors (Juan et al., 2016). Current validation is ongoing in other comatose patients treated with different therapeutic strategies and at multiple hospital sites. This test will be further used in a longitudinal study targeting patients who exhibit an improvement in auditory discrimination but do not wake up within the first days or weeks after coma onset. These patients could first regain a minimal level of consciousness before waking up, and

114

C. Guger et al.

Fig. 5 Schematic representation of the EEG based test for predicting comatose patients’ chance of awakening. a. Neural responses to standard and deviant sounds during an MMN paradigm are recorded through a clinical EEG at comatose patient’s bedside. EEG measurements are represented as a vector of voltage measurements across the whole electrode montage. b. Time-point by time-point voltage topographies are modeled based on a mixture of Gaussians distribution, and the corresponding posterior probabilities are used for labeling EEG single-trials as belonging to standard or deviant sounds’ responses. c. The performance of the decoding algorithm is quantified using the area under the Receiver Operating Characteristic (AUC), performed separately for each recording and patient. The AUC value is indicative of the auditory discrimination at a neural level. d. Based on the decoding performance obtained from the two recordings of the first two days of coma, one can compute the progression of the auditory discrimination and predict patients’ chances of awakening, as an improvement is typically observed in survivors (positive predictive power 93%, with 95% confidence interval 5 0.77–0.99)

could be considered for further EEG based evaluation using the MindBEAGLE system and related experimental protocols.

6 Acute DOC Assessment at Massachusetts General Hospital (MGH) Researchers at MGH have launched a pilot study to test the feasibility of using the mindBEAGLE BCI device in the Neurosciences Intensive Care Unit (NeuroICU). In addition to demonstrating the feasibility of implementing BCI in the acute NeuroICU setting, this pilot study (ClinicalTrials.gov Identifier NCT02772302) aims to determine if mindBEAGLE neurophysiological markers of cognitive function correlate with bedside behavioral assessments of consciousness. The MGH

Trends in BCI Research I: Brain-Computer Interfaces …

115

team recently enrolled its first patient (MGH1), a 72-year-old man with a history of hypertension who was admitted to the NeuroICU with a cerebellar hemorrhage that caused brainstem compression and coma. His NeuroICU course was complicated by intraventricular hemorrhage and hydrocephalus requiring bilateral external ventricular drains, as well as renal failure requiring hemodialysis. At the time of the BCI study, which was performed on his 39th day in the NeuroICU, his Glasgow Coma Scale score was 6T (Eyes = 4, Motor = 1, Verbal = 1T) and his behavioral evaluation with the CRS-R indicated a diagnosis of UWS (Auditory = 1, Visual = 1, Motor = 0, Oromotor/Verbal = 1, Communication = 0, Arousal = 2). EEG electrodes were placed manually since the presence of a left frontal external ventricular drain and a right frontal surgical wound from a recent endoscopic third ventriculostomy prevented application of an EEG cap. During the study, which was performed without complication and without any increase in intracranial pressure, the patient remained on mechanical ventilation via tracheostomy. No sedation was administered during or prior to the study. The mindBEAGLE device detected P300 responses with 70% accuracy during the VT2 paradigm, an observation that suggests that the patient was able to attend to salient stimuli. Although this VT2 result may not definitively prove conscious awareness, it suggests that the patient may be cable of higher-level cognitive processing. Notably, the mindBEAGLE device detected only 30% accuracy during the AEP task, 0% during the VTP3 task, and chance accuracy during a motor task, suggesting the possibility that the patient’s level of responsiveness may have been fluctuating. Within one month of the mindBEAGLE NeuroICU evaluation, the patient began to track visual stimuli, indicating transition from UWS to MCS.

7 LIS Assessment at Oregon Health and Science University (OHSU) Researchers at OHSU conducted a small pilot study (N = 2) investigating the effects of custom MI prompts on assessment and communication accuracy with the mindBEAGLE MI paradigm for people with LIS. It was hypothesized that custom prompts based on well-rehearsed movements [16], using a first-person perspective, and incorporating visual, auditory, and tactile sensations associated with the movement [17], would improve performance compared to a generic prompt. Patient P1 had incomplete LIS secondary to a brainstem stroke, and could communicate using eye movements. P2 had CLIS or possible DOC secondary to advanced amyotrophic lateral sclerosis. Both completed 12 weekly mindBEAGLE MI sessions, each including a 60-trial assessment run and a communication trial of 10 yes/no questions with known answers (e.g. “Is your name Bob?”). In a multiple-baseline AB design, participants were given a generic MI prompt (imagine touching the fingers to the thumb on the left or right hand, as described in the mindBEAGLE manual) in the first 6 or 7 sessions, and a custom prompt (e.g.

116

C. Guger et al.

imagine picking guitar strings with the right hand and moving between chord positions with the left) in the remaining sessions. Custom prompts were based on activities participants had enjoyed when able-bodied, as reported by the participant himself (P1) or a family member (P2), and consisted of a guided imagery script with sensory elements (e.g. the feel of the guitar strings or the sound of the notes). During assessment, participants were given auditory prompts to imagine either the left- or right-sided movement for each trial. To answer questions, they were instructed to imagine the left-sided movement for YES and right-sided for NO. Results are presented in Fig. 6. Participants’ assessment accuracy stayed near chance levels and was similar for the generic (P1: mean = 51.8 ± 4.15%, P2: mean = 41.7 ± 17.22%) and custom (P1: mean = 51.2 ± 4.92%, P2: mean = 50.0 ± 10.95%) prompt conditions. Neither participant demonstrated a significant assessment accuracy level (  66.2%) in any session, and performance did not significantly improve with repeated practice. Accuracy in responding to YES/NO questions was more variable, perhaps due to the smaller number of trials, and again stayed near chance levels. Interestingly one patient reached 90% accuracy in one YES/NO run which shows awareness during this experiment. The custom prompt did not appear to improve performance on either task, as accuracy scores under that condition remained within the expected range based on scores achieved with the generic prompt. The small sample size in this study precludes generalization of results to other potential BCI candidates. The poor performance of P1, who is known to be

Fig. 6 Results for two participants with LIS using the mindBEAGLE MI paradigm for assessment (left) and answering yes/no questions (right). Dashed lines represent calculated trendlines within each condition. Shaded areas represent the degree of data overlap from the generic prompt phase. Small dotted lines represent the expected range of responses using the custom MI prompt based on performance with the generic prompt

Trends in BCI Research I: Brain-Computer Interfaces …

117

conscious and cognitively intact, reminds us that a negative result on a BCI-based assessment is not conclusive evidence of impairment. Additional research with larger participant samples is necessary to determine the utility and appropriateness of MI BCI as a means of assessment and communication for individuals with LIS.

8 DOC Assessment at North Carolina State University (NCSU) The research team at North Carolina State University (NCSU, Raleigh, USA) took a tactile-based hybrid BCI approach to assess consciousness and establish communication with behaviorally non-responsive patients. Tactile-based BCIs are a relatively new and upcoming research topic in the BCI area, which have the potential to help visually-impaired and blind groups. Steady-State Somatosensory Evoked Potentials (SSSEPs) can be elicited on the contralateral areas of the brain with vibrational stimuli [18]. Only recently have tactile-based BCIs been hybridized with SSSEP and tactile-P300 to increase the number of usable classes and improve BCI classification accuracy [19, 20]. In this study, we investigate how different spatial attention affects recorded brain signals, and which spatial patterns provide better SSSEP responses. The stimulation equipment used was the same solenoid tactor setup presented in our previous study [20]. Five healthy volunteers were subjects for the experiment. Vibrational stimuli were presented on subjects’ fingertip, wrist, forearm, and elbow of the dominant side. One tactor presented random pulses on one of four positions, with SSSEP stimulation presented on the other three positions (see Fig. 7a). Each subject conducted 100 pseudo-randomly distributed trials by locations and pulse patterns. To generate a random pulse, a 100 Hz sine wave was presented for 250 ms, while SSSEP stimulation was generated by modulating a 27 Hz square wave atop a 100 Hz sine wave. Each trial consisted of a 5 s rest period, 2 s reference period, and 8 s stimulation period, during which the subjects were asked to focus only on counting the number of random pulses. EEG signals were recorded with a g.USBamp biosignal amplifier using a large Laplacian montage around sites C3 and C4. BCI2000 was used for data acquisition and stimulus presentation, and EEG signals were sampled at 512 Hz and band-pass filtered between 20 Hz and 56 Hz, then analyzed using Canonical Correlation Analysis (CCA) from 20–29 Hz. The average CCA values showed higher Pearson’s correlation (r-value) on the contralateral brain area for 27 Hz, while there were no differences on the ipsilateral brain area at the same SSSEP stimulus frequency (see Fig. 7b). ANOVA of different positions at 27 Hz on C3 for each subject showed that S1 had a significantly higher r-value on the fingertip than other positions (p < .0001), while S2 showed a significantly lower r-value on the fingertip than other positions (p < .0001) (see Fig. 7c). S3 (p = .0619) and S4 (p = .0763) showed marginal significance, and the r-value of fingertip was lower than that of the elbow for S3. There were no

118

C. Guger et al.

Fig. 7 a Conceptual diagram for tactor positions and stimuli when presenting random pulse b Averaged r-values of C3 and C4 areas for all subjects c Averaged r-value of each position at 27 Hz on C3 for S1 and S2

significant differences for S4 in post hoc Tukey tests. S5 showed no significant difference on positions. The CCA value showed that unattended flutter sensation can elicit SSSEPs in the contralateral brain area by simply attending to random pulses presented on the same nerve pathway. Moreover, there were individually different effects of spatial-selective attention on the nerve pathway. We have validated a new approach that evokes SSSEP through off-site attention, which may be used to reduce the mental workload needed to focus on SSSEP stimulation with random pulses and could be combined with P300 stimulation for a hybrid BCI system. In addition, these results can potentially improve the performance of a tactile-based BCI system by utilizing user-specific stimulation sites for improved SSSEP responses. These SSSEP features will be used for future research to develop a hybrid BCI for behaviorally non-responsive patients. SSSEP BCI technology could complement other emerging BCI technologies for these patients.

9 DOC Assessment and Communication in Liege Correct diagnosis of patients with DOC is vital for realistic perspectives on revalidation and outcome. The gold standard for diagnosis is still behavioral bedside assessment, preferably using the Coma Recovery Scale-revised, as it has been proven to be the most sensitive tool that can detect the smallest non-reflexive signs of consciousness [21]. Patients could perform worse than their mental function permits during this kind of testing due to motor dysfunction, aphasia, sensorimotor deficits and other causes. Metabolism as measured with glucose positron emission tomography, and functional connectivity of the default mode network as measured with functional MRI during resting state, are objective ways to assess if consciousness is remaining in DOC patients (see Fig. 8). Active functional magnetic resonance imaging (fMRI) paradigms can be employed to assess command following via MI of playing tennis or navigating through a house, which, if used successfully, can be employed for BCI-based communication.

Trends in BCI Research I: Brain-Computer Interfaces …

119

Fig. 8 From [31]. Glucose metabolism as measured with FDG positron emission tomography, BOLD resting state default mode network activity and BOLD mental imagery as measured with functional MRI in different states of consciousness. The UWS patient on the left shows the neuroimaging of a typical UWS patient with minimal residual brain function. The next panel shows a UWS patient who was unconscious on the behavioral level but who revealed signs of consciousness during the neuroimaging tests, such as command following during the active MRI paradigm. The next panel presents an MCS patient who shows residual metabolism and default mode network connectivity, albeit less than in a healthy subject. The rightmost column depicts normal brain function

Relative to MRI-based assessments, EEG-based BCIs have the advantages of being more affordable, portable and more robust to movement artifacts and metal implants. The tests are easily repeatable, making them very suitable for in clinical practice and during rehabilitation. Previous EEG based BCIs have proven useful to assess awareness and command following in this patient group. Auditory [22, 6] P300 oddball paradigms, and MI experiments [12] have been used successfully. Ethical considerations play an important role for the use of BCIs in this patient population. The outcome of the BCI assessment might influence the medical team and the patient’s loved ones [23]. If the test results show less cognitive function than expected, the patient’s family might cope better with the decision to withdraw life supporting treatment, or lose hope. If the tests show more cognitive abilities than with neurological examination, the clinical management of the patient should be improved so that the chance of recovery increases, but this outcome could also give false hope to families. If the tests show the same level of cognitive abilities as the behavioral assessments, this affirms the decision of the medical team. The patient’s physical and mental disabilities might make it hard to believe that patients can have a good quality of life, whereas healthcare professionals mainly aspire to help their patients attain and maintain a good quality of life. DOC patients cannot communicate whether they feel their life is enjoyable. LIS patients are classically able to communicate by means of eye movement, and when they compare their current well-being to the best and worst periods in their lives, the majority of patients is rather happy [24]. The feeling of well-being in the LIS subjects is comparable to the normal population, indicating that the level of physical (and possibly mental) disability does not significantly influence quality of life.

120

C. Guger et al.

Furthermore, a BCI could give the patient a level of autonomy that would be life changing and most likely increase their quality of life.

10

Alternative Approaches and Directions

As alternatives to neuroimaging and electrophysiological paradigms, non-brain-based approaches, such as measurement of subclinical electromyography signals (Bekinschtein et al., [25, 26], pupil dilation during mental calculation [27], changes in salivary pH [28, 29] or changes in respiration patterns [30] have been proposed to identify covert voluntary cognitive processing in patients with disorders of consciousness. Recently, an electromyographic paradigm detected muscle activations in response to ‘move your left/right hand’ command in 14 patients with MCS (Lesenfants et al., submitted A). Six of them only behaviorally responded to commands on the day of the recording, while all of them showed behavioral responses to commands while assessed repeatedly on multiple days. This approach could be an alternative to BCI inspired paradigms in patients with some residual voluntary muscle control. These results open the door to the development of hybrid paradigms, looking jointly for subclinical electromyography signals and voluntary brain function in response to motor command. Monitoring fluctuations of the level of vigilance can improve the detection of residual signs of consciousness by helping to select the best recording time and by tracking changes across a recording session. Attention itself can also serve as a marker of voluntary cognitive process. Tracking attention during a BCI task in 6 patients with LIS, Lesenfants and colleagues (submitted B) showed that they could track changes in attention with the EEG. They showed that the patients increased their attention during each trial in comparison to the resting periods between trials. While only two patients were successful with the BCI task, all six patients showed fluctuations of attention that could be distinguished from a rest period with more than 90% accuracy.

11

Discussion

The promising results with BCI technology for patients with DOC exhibit several trends. First, results show that new paradigms are emerging that are initially promising, but generally require broader validation with more patients over longer periods. The mindBEAGLE system could make such validation faster and easier while facilitating standardization. The system has standardized paradigms, is used in multi-center studies, is tested with VS, MCS, LIS, CLIS and healthy persons, is used at home, research centers, care facilities and intensive care units and has a standard approach to evaluation. Second, results support the hybrid BCI concept, in which one type of BCI is combined with another BCI and/or another means of communication to provide improved performance and move flexibility for users.

Trends in BCI Research I: Brain-Computer Interfaces …

121

Results have shown that different paradigms, including MI, MMN, and visual and auditory P300 s, can be effective assessment and/or communication tools for these patients, and that SSSEPs could potentially provide another type of BCI for patients. FSL showed that the P300 jitter is larger for VS than MCS and for healthy control subjects, which could further facilitate new improvements. Furthermore, the jitter was negatively correlated with the CRS-R. In some patients, non-EEG signals based on eye, muscle, or other activity could also be useful. Providing a suite of different assessment and communication options could lead to more decisive and detailed assessment and more effective communication while providing users some choice in the approach they wish to use. Third, results with MI BCIs are mixed. MI training can be effective with MCS patients, whereas MI training without feedback did not lead to effective communication in two ALS patients who explored different mental strategies. Interestingly, MCS patients could learn to modulate their SMR with auditory feedback. Fourth, persons with CLIS resulting from ALS, and perhaps other causes, could also benefit from BCI technology that has until now been focused on DOC patients. Fifth, there is a strong trend toward non-visual BCIs, which are often needed for this target group. Sixth, the joint workshops at different major conferences with different groups, and the very nature of this book chapter, show a trend toward dissemination and collaboration among researchers from different regions, disciplines, and sectors. However, this is still a new technology that requires substantial further research, development, and validation with patients in field settings. Future systems could improve existing approaches based on MI, P300 s, and other paradigms, and add additional EEG and non-EEG based tools. New software could improve classifier accuracy, facilitate user interaction, and allow improved communication and control of devices such as fans or music players. New hardware could provide better quality data in noisy settings via more comfortable and practical electrodes. Additional background research is needed to better interpret data from this challenging population, develop and test new paradigms, explore improved classifier algorithms, and explore different patient groups. Research could also explore related tools to help target patients, such as methods to predict recovery (such as the new method from CHUV) or systems for cognitive and motor rehabilitation. Another important future direction is public awareness—very few medical experts are aware of BCI-based options for DOC and other patients. Although very extensive work is still needed to develop methods and systems that are more informative, precise, flexible, and helpful, the results presented here show that BCI for consciousness assessment and communication has advanced beyond laboratory demonstrations, with successful validations of different approaches in different settings. The next several years should see significant improvements in this technology, improved quality of life for many patients, and more informative and reliable assessment tools that will help provide options and crucial information to medical staff, patients, and families. Acknowledgements The work of g.tec was supported by the H2020 grant ComaWare and ComAlert (project number E! 9361 Com-Alert). Q. Noirhomme has received funding from the European Community’s Seventh Framework Program under grant agreement n° 602450

122

C. Guger et al.

(IMAGEMEND). Research at OHSU was supported by NIH grant R01DC014294 and NIDILRR grant 90RE5017. Research at MGH was supported by NIH grant K23NS094538 and the American Academy of Neurology/American Brain Foundation. Research at NCSU was supported by NSF grant IIS1421948. Marzia De Lucia’s research at Lausanne University Hospital is supported by the “EUREKA-Eurostars” grant (project number E! 9361 Com-Alert). The work was partially supported by the Italian Ministry of Healthcare and the French Speaking Community Concerted Research Action (ARC-06/11-340). This paper reflects only the authors’ view and the funding sources are not liable for any use that may be made of the information contained therein.

References 1. Guger C, Noirhomme Q, Naci L, Real R, Lugo Z, Veser S, Sorger B, Quitadamo L, Lesenfants D, Risetti M, Formisano R, Toppi J, Astolfi L, Emmerling T, Erlbeck H, Monti MM, Kotchoubey B, Bianchi L, Mattia D, Goebel R, Owen AM, Pellas F, Müller-Putz G, Kübler A (2014) Brain-computer interfaces for coma assessment and communication. In: Ganesh RN (ed) Emerging theory and practice in neuroprosthetics. IGIGLOBAL Press 2. Lesenfants D, Habbal D, Chatelle C, Schnakers C, Laureys S, Noirhomme Q Electromyographic decoding of response to command in disorders of consciousness, submitted A 3. Coyle D, Stow J, McCreadie K, Sciacca N, McElligott J, Carroll Á (2017) Motor imagery BCI with auditory feedback as a mechanism for assessment and communication in disorders of consciousness. In: Brain-Computer Interface Research. Springer International Publishing, pp 51–69 4. Laureys S, Pellas F, Van Eeckhout P, Ghorbel S, Schnakers C, Perrin F, Berre J, Feymonville ME, Pantke KH, Damas F, Lamy M, Moonen G, Goldman S (2005) The locked-in syndrome: What is it like to be conscious but paralyzed and voiceless? Prog Brain Res 150:495–511 5. Ortner R, Lugo Z, Prückl R, Hintermüller C, Noirhomme Q, Guger C (2013) Performance of a tactile P300 speller for healthy people and severely disabled patients. In: Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2013), Osaka, JP. 3–7 July 2013 6. Lugo ZR, Rodriguez J, Lechner A, Ortner R, Gantne IS, Laureys S, Guger C (2014) A vibrotactile p 300-based brain-computer interface for consciousness detection and communication. Clin EEG Neurosci 45:14–21 7. Guger C, Kapeller C, Ortner R, Kamada K (2015) Motor imagery with brain-computer interface neurotechnology. In: Garcia BM (ed) Motor imagery: emerging practices, role in physical therapy and clinical implications, pp 61–79 8. Guger C, Spataro R, Allison BZ, Heilinger A, Ortner R, Cho W, La Bella V (2017) Complete locked-in and locked-in patients: command following assessment and communication with vibro-tactile P300 and motor imagery brain-computer interface tools. Front Neurosci 11 9. Coyle D et al (2012) Enabling control in the minimally conscious state in a single session with a three channel BCI. In: 1st International Decoder Workshop, April, pp 1–4 10. Coyle D et al (2015) Sensorimotor modulation assessment and brain-computer interface training in disorders of consciousness. Arch Phys Med Rehabil 96(3):62–70 11. Coyle D et al (2013) Visual and stereo audio sensorimotor rhythm feedback in the minimally conscious state. In: Proceedings of the Fifth International Brain-Computer Interface Meeting 2013, pp 38–39 12. Cruse D, Chennu S, Chatelle C, Bekinschtein TA, Fernández-Espejo D, Pickard JD, Owen AM (2011) Bedside detection of awareness in the vegetative state: a cohort study. Lancet 378(9809):2088–2094

Trends in BCI Research I: Brain-Computer Interfaces …

123

13. Nuffield Council on Bioethics Report (2013) Novel neurotechnologies : intervening in the brain. http://nuffieldbioethics.org/wp-content/uploads/2013/06/Novel_neurotechnologies_ report_PDF_web_0.pdf 14. Aricò P, Aloise F, Schettini F, Salinari S, Mattia D, Cincotti F (2014) Influence of P300 latency jitter on event related potential-based brain-computer interface performance. J Neural Eng 11(3):035008 15. Schettini F, Risetti M, Arico P, Formisano F, Babiloni F, Mattia D, Cincotti F (2015) P300 latency Jitter occurrence in patients with disorders of consciousness: toward a better design for brain computer interface applications. In: Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE, 2015, pp 6178–6181 16. Olsson C-J, Nyberg L (2010) Motor imagery: If you can’t do it, you won’t think it. Scand J Med Sci Sports 20:711–715 17. Bovend’Eerdt TJH, Dawes H, Sackley C, Wade DT (2012) Practical research-based guidance for motor imagery practice in neurorehabilitation. Disabil Rehabil 34(25):2192–2200 18. Snyder AZ (1992) Steady-state vibration evoked potentials: description of technique and characterization of responses. Electroencephalogr Clin Neurophysiol Potentials Sect 84 (3):257–268 19. Severens M, Farquhar J, Duysens J, Desain P (2013) A multi-signature brain-computer interface: use of transient and steady-state responses. J Neural Eng 10(2):026005 20. Choi I, Bond K, Krusienski D, Nam CS (2015) Comparison of stimulation patterns to elicit steady-state somatosensory evoked potentials (SSSEPs): implications for hybrid and SSSEP-based BCIs. In: IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2015 21. Giacino JT, Kalmar K, Whyte J (2004) The JFK Coma Recovery Scale-Revised: Measurement characteristics and diagnostic utility. Arch Phys Med Rehabil 85(12):2020– 2029 22. Laureys S, Perrin F, Faymonville M-E, Schnakers C, Boly M (2004) Cerebral processing in the minimally conscious state. Neurology 63:916–918 23. Jox RJ, Bernat JL, Laureys S, Racine E (2012) Disorders of consciousness: responding to requests for novel diagnostic and therapeutic interventions. Lancet Neurol 11(8):732–738 24. Bruno MA, Bernheim JL, Ledoux D, Pellas F, Demertzi A, Laureys S (2011) A survey on self assessed well-being in a cohort of chronic locked-in syndrome patients: happy majority, miserable minority. BMJ Open 1(1):1–9 25. Bekinschtein TA, Coleman MR, Niklison J, Pickard JD, Manes FF (2008) Can electromyography objectively detect voluntary movement in disorders of consciousness? J Neurol Neurosurg Psychiatry 79(7):826–828 26. Habbal D, Gosseries O, Noirhomme Q, Renaux J, Lesenfants D, Bekinschtein TA, Majerus S, Laureys S, Schnakers C (2014) Volitional electromyographic responses in disorders of consciousness. Brain Inj 28(9):1171–1179 27. Stoll J, Chatelle C, Carter O, Koch C, Laureys S, Einhäuser W (2013) Pupil responses allow communication in locked-in syndrome patients. Curr Biol 23(15):R647–R648 28. Ruf CA, DeMassari D, Wagner-Podmaniczky F, Matuz T, Birbaumer N (2013) Semantic conditioning of salivary pH for communication. Artif Intell Med 59(2):91–98 29. Wilhelm B, Jordan M, Birbaumer N (2006) Communication in locked-in syndrome: effects of imagery on salivary pH. Neurology 67(3):534–535 30. Charland-Verville V, Lesenfants D, Sela L, Noirhomme Q, Ziegler E, Chatelle C, Plotkin A, Sobel N, Laureys S (2014) Detection of response to command using voluntary control of breathing in disorders of consciousness. Front Hum Neurosci 8 31. Laureys S, Schiff ND (2012) Coma and consciousness: paradigms (re)framed by neuroimaging. NeuroImage 61:1681–1691

124

C. Guger et al.

Author Biographies Christoph Guger is actively running international research projects in the BCI domain, and is the CEO of g.tec medical engineering GmbH, Guger Technologies OG, g.tec neurotechnology USA Inc. and g.tec medical engineering Spain SL. Damien Coyle is professor at the University of Ulster and active in BCI research for many years. He is a specialist for signal processing and has experience with DOC patients. Donatella Mattia MD, PhD, Neurologist, Neurophysiologist. Lab. Director. Her main research interests are EEG-based BCI design and validation in neurorehabilitation and advanced signal processing methods for feature extraction and outcome measure. Marzia De Lucia is principal investigator at the Laboratoire de Recherche en Neuroimagerie of Lausanne University Hospital and the Faculty of Biology and Medicine at the University of Lausanne, Switzerland. Her research focuses on consciousness detection and comatose patients’ outcome prediction. Leigh Hochberg is a vascular and critical care neurologist and neuroscientist. His research focuses on the development and testing of novel neurotechnologies to help people with paralysis and other neurologic disorders, and on understanding cortical neuronal ensemble activities in neurologic disease. Dr. Hochberg has appointments as Professor of Engineering, School of Engineering and Institute for Brain Science, Brown University; Neurologist, Massachusetts General Hospital, where he attends in the NeuroICU and on the Acute Stroke service; Director, VA Center for Neurorestoration and Neurotechnology, Providence VAMC; and Senior Lecturer on Neurology at Harvard Medical School. He also directs the Neurotechnology Trials Unit for MGH Neurology, where he is the IDE Sponsor-Investigator and Principal Investigator of the BrainGate pilot clinical trials (www.braingate.org) that are conducted by a close collaboration of scientists and clinicians at Brown, Case Western Reserve University, MGH, Providence VAMC, and Stanford University. Dr. Hochberg is a Fellow of the American Academy of Neurology and the American Neurological Association. Dr. Hochberg’s BrainGate research, which has been published Nature, Science Translational Medicine, Nature Medicine, Nature Neuroscience, the Journal of Neuroscience, and others, is supported by the Rehabilitation R&D Service of the U.S. Department of Veterans Affairs, NCMRR/NICHD, NIDCD, and NINDS. Brian L. Edlow received his B.A. from Princeton University and M.D. from the University of Pennsylvania School of Medicine. He completed an internal medicine internship at Brigham and Women’s Hospital, followed by neurology residency and neurocritical care fellowship at Massachusetts General Hospital and Brigham and Women’s Hospital. He is currently a critical care neurologist at Massachusetts General Hospital, where he is Associate Director of the Neurotechnology Trials Unit and Director of the Laboratory for NeuroImaging of Coma and Consciousness. Dr. Edlow’s research is devoted to the development of advanced imaging techniques for detecting brain activity and predicting outcomes in patients with severe traumatic brain injury. The goals of this research are to improve the accuracy of outcome prediction and to facilitate new therapies that promote recovery of consciousness. Dr. Edlow receives support from the National Institutes of Health, Department of Defense, and American Academy of Neurology/American Brain Foundation. Betts Peters is a speech-language pathologist specializing in augmentative and alternative communication, and works on BCI research with REKNEW Projects at Oregon Health & Science University.

Trends in BCI Research I: Brain-Computer Interfaces …

125

Brandon Eddy is a clinical fellow in speech-language pathology at the Oregon Health & Science University Child Development and Rehabilitation Center in Portland, Oregon. Chang S. Nam is currently an associate professor of Edward P. Fitts Industrial and Systems Engineering at North Carolina State University (NCSU), USA. He is also an associated professor of the UNC/NCSU Joint Department of Biomedical Engineering, as well as Department of Psychology. He is director of BCI Lab at NCSU. His research interests center around brain-computer interfaces and neurorehabilitation, smart healthcare, neuroergonomics, and adaptive and intelligent human-computer interaction. Currently, Nam serves as the Editor-in-Chief of the journal Brain-Computer Interfaces. Quentin Noirhomme is senior scientist at Brain Innovation BV, where he works on the development of BCIs for clinical applications. He collaborates with the University of Maastricht and the University of Liege. Brendan Z. Allison PhD was a Senior Scientist with g.tec and is a Visiting Scholar with the UCSD Cognitive Science Department. He has been active in BCI research for over 20 years, and is active with mindBEAGLE research for different groups. Jitka Annen Jitka Annen is interested in multimodal analysis of consciousness and BCI applications in patients with DOC, and is PhD student in the Coma Science Group headed by Steven Laureys.

Recent Advances in Brain-Computer Interface Research—A Summary of the BCI Award 2016 and BCI Research Trends Christoph Guger, Brendan Z. Allison and Mikhail A. Lebedev

1 The 2016 Winners The previous chapters should help to show the high quality of the nominated projects, and thus the jury had a very difficult task. With 52 projects submitted, identifying twelve nominees and the winners was not easy, and many good submissions were not nominated, often due to a low score on one or more criteria. After tallying the scores across the scoring criteria from the different judges, the nominees were posted online and invited to our Gala Awards ceremony to learn which nominees would win first, second, and third place in 2016. The Gala Awards ceremony was part of the largest conference for the BCI community in 2016, which was the Sixth International BCI Meeting. This conference was held at the Asilomar Conference Grounds in Pacific Grove, CA, like the previous two International BCI Meetings. The ceremony was a suspenseful event, with hundreds of BCI aficionados in the audience to watch history being made. At the ceremony, Dr. Guger (the organizer) and Dr. Allison (the emcee)

C. Guger (&) Graz, Austria e-mail: [email protected] B.Z. Allison San Diego, USA M.A. Lebedev Durham, USA © The Author(s) 2017 C. Guger et al. (eds.), Brain-Computer Interface Research, SpringerBriefs in Electrical and Computer Engineering, DOI 10.1007/978-3-319-64373-1_12

127

128

C. Guger et al.

reviewed the nominated projects and asked representatives from each group to come onstage to receive a certificate. Next, the three winners were announced and handed their prizes. Without further ado, the three winners were: The BCI Award 2016 Winner Is Gaurav Sharma1, Nick Annetta1, Dave Friedenberg1, Marcie Bockbrader2, Ammar Shaikhouni2, W. Mysiw2, Chad Bouton1, Ali Rezai2 1

Battelle Memorial Institute, 505 King Ave, Columbus, OH 43201

2

The Ohio State University, Columbus, OH, USA 43210)

An Implanted BCI for Real-Time Cortical Control of Functional Wrist and Finger Movements in a Human with Quadriplegia Mikhail A. Lebedev, chair of the 2016 jury, called the winning idea “A fascinating demonstration of how spinal cord injury can be bypassed by a neural prosthesis connecting the motor cortex directly to a functional electrical stimulation device that activates the muscles of the paralyzed hand, allowing the patient to volitionally execute wrist and finger movements—the system that will find a broad range of clinical applications for restoration of motor control to paralyzed people, and rehabilitation of their neurological deficits.” The BCI Award 2016 2nd Place Winner Is Sharlene Flesher2,3, John Downey2,3, Jennifer Collinger1,2,3,4, Stephen Foldes1,3,4, Jeffrey Weiss1,2, Elizabeth Tyler-Kabara1,2,5, Sliman Bensmaia6, Andrew Schwartz2,3,8, Michael Boninger1,2,4, Robert Gaunt1,2,3 >1,2,5,8Departments of Physical Medicine and Rehabilitation, Bioengineering, Neurological Surgery, Neurobiology, University of Pittsburgh, Pittsburgh, PA, USA 3 Center for the Neural Basis of Cognition, Pittsburgh, PA, USA 4 Department of Veterans Affairs Medical Center, Pittsburgh, PA, USA 6 Department of Organismal Biology and Anatomy, University of Chicago, Chicago, IL, USA. Intracortical Microstimulation as a Feedback Source for Brain-Computer Interface Users The 3rd Place Winner Is Thomas J. Oxley, Nicholas L. Opie, Sam E. John, Gil S. Rind, Stephen M. Ronayne, Clive N. May, Terence J. O’Brien Vascular Bionics Laboratory, Melbourne Brain Centre, Departments of Medicine and Neurology, The Royal Melbourne Hospital, The University of Melbourne, Parkville, Victoria, Australia.

Recent Advances in Brain-Computer Interface …

129

Minimally Invasive Endovascular Stent-Electrode Array for High-Fidelity, Chronic Recordings of Cortical Neural Activity The first, second, and third place winners received cash awards of $3000, $2000, and $1000, respectively. These three winners also received a special bread knife, and all nominees won other prizes. Interestingly, all three winners presented work with invasive BCIs, even though the submissions were mostly non-invasive. Whether this reflects a fluke for one year or an emerging trend remains to be seen (Figs. 1 and 2). At the Gala Award Ceremony, Dr. Guger also thanked the experts in the 2016 jury: • • • • • •

Mikhail A. Lebedev (chair of the jury 2016), Alexander Kaplan, Klaus-Robert Müller, Ayse Gündüz, Kyousuke Kamada, Guy Hotson (winner 2015).

Fig. 1 Christoph Guger (left, organizer), Gaurav Sharma (First place winner, 2016), and Kyousuke Kamada (jury member), all standing onstage during the Gala Awards Ceremony at the BCI Meeting 2016 in Asilomar, CA

130

C. Guger et al.

Fig. 2 Sharlene Flesher, Jennifer Collinger, Robert Gaunt, and John Downey are delighted with their Award! The image behind them encourages people to join the BCI society. Ironically, the BCI society recently voted to add Jennifer Collinger as a board member. (Two editors of this book series, Drs. Guger and Allison, are also board members.) The next BCI meeting hosted by the BCI society, scheduled for 2018, will also feature an awards ceremony for the 2018 BCI award

Table 1 Type of input signal for the BCI system Property

2016% (N = 52)

2015% (N = 63)

2014% (N = 69)

2013% (N = 169)

2012% (N = 68)

2011% (N = 64)

2010% (N = 57)

EEG fMRI ECoG NIRS Spikes Other signals Electrodes

71,2 3,8 11,5 1,9 7,7 1,9 1,9

76,1 4,8 9,5 – 4,8 4,8 –

72,5 2,9 13,0 1,4 8,7 4,3 –

68,0 4,1 9,4 3,0 7,1 13,0 6,5

70,6 1,5 13,3 1,5 10,3 2,9 1,5

70,3 3,1 4,7 4,7 12,5 1,6 1,6

75,4 3,5 3,5 1,8 – – –

2 Directions and Trends Reflected in the Awards The Annual BCI Award shows trends in BCI technology and allows us to identify the most important directions.

Recent Advances in Brain-Computer Interface …

131

Table 2 Real-time BCIs and off-line algorithms in projects submitted to the BCI awards Property

2016% (N = 52)

2015% (N = 63)

2014% (N = 69)

2013% (N = 169)

2012% (N = 68)

2011% (N = 64)

2010% (N = 57)

Real-time BCI

94,2

96,8

87,0

92,3

94,1

95,3

65,2

Off-line applications

5,8

3,2

8,7

5,3

4,4

3,1

17,5

The following four tables summarize different characteristics of submitted projects since the award began in 2010. In each table, N reflects the number of submissions, and numbers in different cells present the percentage or submissions that that characteristic. We present one table for each of the four general BCI components presented in the introduction. Sensors Table 1 explores the different types of input signals used in the submitted projects. As with previous years, the 2016 submissions focused primarily on EEG-based systems, similar to most BCI articles. The submissions also reflected other non-invasive sensor systems, such as fMRI and NIRS, and invasive methods like ECoG and neural spikes. Signal Processing The second table analyzes the percentage of submissions that presented offline vs. real-time BCI applications. Nowadays, most of the BCI systems work in real-time and only a few projects improve off-line algorithms or hardware components (Table 2). Output/Application The third essential component of any BCI is the output. Table 3 summarizes the different outputs, and related applications, that have been submitted since 2010. The applications have varied over the years, but generally show a strong interest in control, BCI platform tools and algorithms, monitoring and assessment and control of robotic devices such as prosthetics, robots and wheelchairs. Environment/Interaction Finally, Table 4 summarizes the type of control signal that was used to influence BCI operation. This is a key component of the BCI’s operating environment and interaction with each user. Most of the BCI systems are controlled with motor imagery, P300 or SSVEP paradigms. Some systems also use face imagination or navigation in houses. Another promising trend worth mentioning is the emergence and development of the BCI Society. At a plenary session of the 2013 BCI Meeting, the attendees unanimously voted in favour of establishing an official society to represent the BCI community. Since then, the BCI Society was officially formed, including by-laws and administrative aspects. It has recruited several hundred members, held Board Member elections, launched a website, organized the 2016 BCI Meeting, and managed other activities. The Board Members and other members include most of the most respected figures in the BCI community, as well as many top experts from

132

C. Guger et al.

Table 3 Type of output system and application Property

2016% (N = 52)

2015% (N = 63)

2014% (N = 69)

2013% (N = 169)

2012% (N = 68)

2011% (N = 64)

2010% (N = 57)

Control Platform technology algorithm Stroke neural plasticity Wheelchair robot prosthetics Spelling Internet or VR game Learning Monitoring, DOC Stimulation Authentication speech assessment Connectivity Music, art Sensation Vision Epilepsy, parkinson, tourette’s, autism Depression, fatigue, ADHD, pain Neuromarketing, emotion Ethics Mechanical ventilation Roadmap

15,4 26,9

11,1 15,9

17,4 13,0

20,1 16,6

20,6 16,2

34,4 9,4

17,5 12,3

5,8

4,8

13,0

13,7

26,5

12,5

7

7,7

15,9

13,0

11,8

8,8

6,2

7

3,8 1,9

12,7 4,8

8,7 2,9

8,3 5,9

25 2,9

12,5 3,1

19,3 8,8

1,9 9,6 3,8 3,8

1,6 4,8 1,6 4,8

5,8 1,4 1,4 13,0

5,3 4,7 3,6 3

1,5 4,4 1,5 –

3,1 1,6

– –

9,4

–

1,9 3,8 – 1,9 3,8

– 1,6 – 3,2 3,2

– 1,4 – 1,4 2,9

2,4 1,8 1,2 1,2 1,2

1,5 – 1,5 -

1,6

–

-

-

1,9

4,8

1,4

–

1,5

–

–

–

–

1,4

–

1,5

–

–

– –

– –

1,4 –

– –

– –

– 1,6

– –

1,9

–

–

–

–

–

–

related fields. The BCI Society is now widely recognized as the central entity that represents the BCI community. The BCI Society has kindly allowed us to organize past and upcoming BCI Awards ceremonies at BCI Meetings, and to post announcements relating to the BCI Award on their website. As two of the editors of this book series are BCI Society Board members, we strongly support the BCI Society and look forward to ongoing friendly interaction with them. We also consider the nascent success of the BCI Society very promising in terms of reducing fragmentation and miscommunication. These problems have long been recognized in the published BCI literature and elsewhere, and we hope the BCI Society reflects a trend toward amity and accord.

Recent Advances in Brain-Computer Interface …

133

Table 4 Type of control signal used to interact with the BCI Property

2016% (N = 52)

2015% (N = 63)

2014% (N = 69)

2013% (N = 169)

2012% (N = 68)

2011% (N = 64)

2010% (N = 57)

P300/N200/ERP

11,5

28,6

11,6

11,8

30,9

25

29,8

SSVEP/SSSEP/cVEP

11,5

14,3

11,6

14,2

16,2

12,5

8,9

Motor imagery

32,7

36,5

37,7

25,4

30,9

29,7

40,4

ASSR

–

–

1,8

–

1,6

–

Fig. 3 The flyer for the 2017 BCI-research award

There do remain concerns with misrepresentation and other issues from some relatively unskilled and unscrupulous BCI researchers, companies, and media entities. Regrettably, there are notorious recent examples of this across all three of these categories. Misrepresentation of what BCIs can do, especially to patients, could be very damaging to patients and other buyers/users, the public, and the BCI community as a whole. We hope this trend does not continue, and that the BCI Society, BCI Awards and books, and other mechanisms can help refocus where it belongs: on new, promising, top-quality BCI achievements.

3 Conclusion and Future Directions The Annual BCI-Research Awards, along with this book series, have sought to recognize and identify the newest and best developments in BCI research. As these efforts continue over the years, we have more and more data we can use to explore

134

C. Guger et al.

different trends, and we may consider a specialized chapter or other article that just focused on trends and a retrospective. In the short term, however, we are focused on the next award. The 2017 BCI-Award flyer was posted online (see Fig. 3), and the deadline of June 15, 2017 has expired. The jury is currently scoring the submissions, and the awards ceremony will occur with the Seventh International BCI Conference in Graz, Austria in September 2017. We are proud to announce the jury for 2017: Natalie Mrachacz-Kersting (chair of the jury 2017), Gaurav Sharma (winner 2016), Reinhold Scherer, Jose Pons, Femke Nijboer, Kenji Kansaku, Aaron Batista, Jing Jin. This is a particularly large jury, and contains even more breadth than usual. The jury includes specialists in different imaging, signal processing, and output methods, rehabilitation, ethics, robotics, virtual reality, and numerous other BCI-related fields. The 2017 jury also has very good representation from BCI groups around the world. The chair comes from a top Danish BCI institute. Mrachacz-Kersting is a professor in the Neural Engineering and Neurophysiology lab of Aalborg University. The jury also includes experts who work in different European countries, the USA, China, and Japan. In summary, the 2016 BCI Awards and the resulting chapters have introduced and recognized many of the most innovative and promising new projects in the BCI research community. Most of the nominees come from well-known, established groups that are currently exploring even newer directions based on their nominated projects. We have also explored different trends in BCI research by analysing different characteristics of the submissions. We hope and expect that the 2017 BCI Awards will highlight another group of new and fascinating ideas, and further recognize new and developing trends.