Advances in Service and Industrial Robotics: Results of RAAD [1st ed.] 9783030489885, 9783030489892

Table of contents :
Front Matter ....Pages i-xv
Front Matter ....Pages 1-1
Optimal Design of a Rehabilitation Four Cable-Driven Parallel Robot for Daily Living Activities (Ferdaws Ennaiem, Abdelbadiâ Chaker, Juan Sebastián Sandoval Arévalo, Med Amine Laribi, Sami Bennour, Abdelfattah Mlika et al.)....Pages 3-12
Pneumatic Equipment for Ankle Rehabilitation by Continuous Passive Motion (Andrea Deaconescu, Tudor Deaconescu)....Pages 13-21
Simulation Platform for Crane Visibility Safety Assistance (Tanittha Sutjaritvorakul, Axel Vierling, Jakub Pawlak, Karsten Berns)....Pages 22-29
Design of Robotic Braces for Patients with Scoliosis (Rahul Ray, Laurence Nouaille, Briac Colobert, Gérard Poisson)....Pages 30-37
Comparison Between Two Methods of Cables Tensions Optimization of a Cable Driven Parallel Robot (Hajer Ben Amor, Sami Bennour, Abdelfattah Mlika)....Pages 38-45
Computational Method for Muscle Forces Estimation Based on Hill Rheological Model (Olfa Jemaa, Sami Bennour, David Daney, Lotfi Romdhane)....Pages 46-53
Design of a Variable Stiffness Joint for a Five-Bar-Mechanism (Maria Guadalupe Contreras-Calderón, Eduardo Castillo-Castañeda)....Pages 54-63
Visual Control Based Musculotendon Force Estimation in the Human Upper-Limb (Souha Baklouti, Olfa Jemaa, Sami Bennour)....Pages 64-72
Design and Simulation of an Underactuated Exoskeleton Mechanism for a User-Oriented Leg Exercising (Yao Shuangji, Giuseppe Carbone)....Pages 73-81
Front Matter ....Pages 83-83
Remote Center of Motion for Redundant Robotic-Assisted Ultrasound Guided Regional Anesthesia (Mohammad Alkhatib, Cyril Novales, Laurence Nouaille, Adel Hafiane, Pierre Vieyres)....Pages 85-92
Reference Frame Identification and Distributed Control Strategies in Human-Robot Collaboration (Alberto Borboni, Giuseppe Carbone, Nicola Pellegrini)....Pages 93-102
Static Balancing of a 5-DoF Mechanism with Manual Reconfiguration Feature (Terence Essomba, Zi-Wei Zhao)....Pages 103-111
Novel Safety Criterion for Synthesis of Cable Driven Parallel Robots (Ines Ben Hamida, Med Amine Laribi, Abdelfattah Mlika, Lotfi Romdhane, Saïd Zeghloul, Giuseppe Carbone)....Pages 112-120
Human Activity Recognition Enhanced Robot-Assisted Minimally Invasive Surgery (Hang Su, Wen Qi, Chenguang Yang, Jiehao Li, Xuanyi Zhou, Giancarlo Ferrigno et al.)....Pages 121-129
Online Payload Identification of a Franka Emika Robot for Medical Applications (Souad Salah, Juan Sandoval, Moncef Ghiss, Med Amine Laribi, Saïd Zeghloul)....Pages 130-136
Robot Motion Control Using EMG Signals and Expert System for Teleoperation (George Pagounis, Panagiotis Koustoumpardis, Nikos Aspragathos)....Pages 137-148
PRSX: An End-Effector for Pronation and Supination Adaptable to Arm Rehabilitation Devices (Maria Guadalupe Contreras-Calderón, Eduardo Castillo-Castañeda)....Pages 149-158
Criteria for the Design and Application of Socially Assistive Robots in Interventions for Children with Autism (Ilias A. Katsanis, Vassilios C. Moulianitis)....Pages 159-167
Front Matter ....Pages 169-169
Generation of Smooth Cartesian Paths Using Radial Basis Functions (Leon Žlajpah, Tadej Petrič)....Pages 171-180
Combining Virtual and Physical Guides for Autonomous In-Contact Path Adaptation (Tadej Petrič, Leon Žlajpah)....Pages 181-189
A Framework for the Study of Human-Robot Collaboration in Rehabilitation Practices (Giorgia Chiriatti, Giacomo Palmieri, Matteo Claudio Palpacelli)....Pages 190-198
Dyadic Human-Human Interactions in Reaching Tasks: Fitts’ Law for Two (Rebeka Kropivšek Leskovar, Jernej Čamernik, Tadej Petrič)....Pages 199-207
Maximizing the End-Effector Cartesian Stiffness Range for Kinematic Redundant Robot with Compliance (Branko Lukić, Kosta Jovanović, Nikola Knežević, Leon Žlajpah, Tadej Petrič)....Pages 208-217
Comparison of Model-Based Simultaneous Position and Stiffness Control Techniques for Pneumatic Soft Robots (Maja Trumić, Kosta Jovanović, Adriano Fagiolini)....Pages 218-226
Shaking Force Balancing of the Orthoglide (Jing Geng, Vigen Arakelian, Damien Chablat, Philippe Lemoine)....Pages 227-234
A Boosted Decision Tree Approach for a Safe Human-Robot Collaboration in Quasi-static Impact Situations (Nemanja Kovincic, Hubert Gattringer, Andreas Müller, Mathias Brandstötter)....Pages 235-244
Combining the Robot Operating System with Building Information Modeling for Robotic Applications in Construction Logistics (Camilla Follini, Michael Terzer, Carmen Marcher, Andrea Giusti, Dominik Tobias Matt)....Pages 245-253
Front Matter ....Pages 255-255
Design of a Thunniform Swimming Robot in a Multiphysics Environment (Daniele Costa, Giacomo Palmieri, Matteo Palpacelli, David Scaradozzi)....Pages 257-265
Analysis of Tire Temperature Influence on Vehicle Dynamic Behaviour Using a 15 DOF Lumped-Parameter Full-Car Model (Michele Perrelli, Flavio Farroni, Francesco Timpone, Domenico Mundo)....Pages 266-274
Motion Sickness Minimisation in Autonomous Vehicles Using Optimal Control (Zaw Htike, Georgios Papaioannou, Efstathios Siampis, Efstathios Velenis, Stefano Longo)....Pages 275-282
A Tire/Road Interaction Tool for the Evaluation of Tire Wear for Commercial Vehicles (M. De Martino, Argemiro L. A. Costa, F. Timpone, A. Sakhnevych)....Pages 283-293
Multi Actuation Scheme for Path-Following Control of Autonomous Vehicles (Javad Ahmadi, Efstathios Velenis, Heimoud El Vagha, Chenhui Lin, Boyuan Li, Stefano Longo et al.)....Pages 294-301
Front Matter ....Pages 303-303
A Saffron Spice Separation System with Computer Vision (A. Manuello Bertetto, A. Prete)....Pages 305-314
Prismatic Delta Robot: A Lagrangian Approach (Federico Colombo, Luigi Lentini)....Pages 315-324
Italian Contributions to RAAD (Marco Ceccarelli)....Pages 325-333
Minimum-Energy Trajectory Planning for Industrial Robotic Applications: Analytical Model and Experimental Results (Lorenzo Scalera, Giovanni Carabin, Renato Vidoni, Alessandro Gasparetto)....Pages 334-342
Front Matter ....Pages 343-343
Conceptual Design and Control of a Sitting-Type Lower-Limb Rehabilitation System Established on a Spatial 3-PRRR Parallel Manipulator (Santhakumar Mohan, Parvathi Sunilkumar, Larisa Rybak, Dmitry Malyshev, Sergey Khalapyan, Anna Nozdracheva)....Pages 345-355
The First Type of Singularity of a 3-PRRS Parallel Manipulator (Zhumadil Baigunchekov, Med Amine Laribi, Myrzabai Izmambetov, Zhadyra Zhumasheva, Rustem Kaiyrov)....Pages 356-363
Inverse Kinematics of a 3-PRPS Type Parallel Manipulator (Zhumadil Baigunchekov, Said Zeghloul, Abzal Kassinov)....Pages 364-370
Structurally Parametric Synthesis of a RoboMech Class Parallel Manipulator with Three DOF (Zhumadil Baigunchekov, Sobh Tarek, Sarosh Patel, Azamat Mustafa)....Pages 371-379
Functional Design of a Novel Over-Actuated Mobile Robotic Platform for Assistive Tasks (Luca Carbonari, Andrea Botta, Paride Cavallone, Giuseppe Quaglia)....Pages 380-389
Mechanism for the Locomotion Layout Reconfiguration of the Agri_q Mobile Robot (Carmen Visconte, Paride Cavallone, Luca Carbonari, Andrea Botta, Giuseppe Quaglia)....Pages 390-399
Design and Simulation of a Parallel-Mechanism Testbed for Head Impact (José Luis Rueda Arreguín, Marco Ceccarelli, Christopher René Torres San Miguél)....Pages 400-407
A New RCM Mechanism for an Ear and Facial Surgical Application (Guillaume Michel, Durgesh Haribhau Salunkhe, Damien Chablat, Philippe Bordure)....Pages 408-418
Design of Serial Link Structure-Parallel Wire System for Virtual Reality Rehabilitation and Assessment (Vishal Ramadoss, Mohamed Sadiq Ikbal, Dimiter Zlatanov, Matteo Zoppi)....Pages 419-427
Mathematical Model and Experimental Validation for a Four Bar Mechanism with a Flexible Coupler Link (Daniel Rodríguez Flores, Héctor Cervantes Culebro, Carlos-Alberto Cruz-Villar)....Pages 428-437
Some Quality Measures in Designing Compliant Mechanisms for Robotic Devices (Stefan Havlik, Jaroslav Hricko)....Pages 438-447
Prototype Design and Testing of TORVEastro, Cable-Driven Astronaut Robot (Francesco Samani, Marco Ceccarelli)....Pages 448-455
Design and Lab Experience for a Pipeline Service Robot in Space Orbital Stations (Tao Zheng, Hui Li, Qiang Huang, Xiang Wang, Marco Ceccarelli, Zhihong Jiang et al.)....Pages 456-465
Nonlinear Dynamics of a Spatial Two Link Chain (Dan B. Marghitu, Dorian Cojocaru)....Pages 466-475
Front Matter ....Pages 477-477
Autonomous Navigation in Vineyards with Deep Learning at the Edge (Diego Aghi, Vittorio Mazzia, Marcello Chiaberge)....Pages 479-486
Iterative Learning Control of Hard Constrained Robotic Manipulators (Kaloyan Yovchev, Lyubomira Miteva, Kamen Delchev, Evgeniy Krastev)....Pages 487-495
Generalization Based Database Acquisition for Robot Learning in Reduced Space (Zvezdan Lončarević, Rok Pahič, Mihael Simonič, Aleš Ude, Andrej Gams)....Pages 496-504
Analysis of Different Methods to Close the Reality Gap for Instance Segmentation in a Flexible Assembly Cell (Nawal Hafez, Vincent Dietrich, Stefan Roehrl)....Pages 505-515
Data-Driven Synthesis of Perception Pipelines via Hierarchical Planning (Vincent Dietrich, Bernd Kast, Sebastian Albrecht, Michael Beetz)....Pages 516-524
Crashing to Learn, Learning to Survive: Planning in Dynamic Environments via Domain Randomization (Sarath S. Menon, R. Sakthi Vignesh, Santhakumar Mohan)....Pages 525-534
Target Manipulation in Nuclear Physics Experiment with Ion Beams (Diego Sartirana, Daniela Calvo, Vittoria Capirossi, Carlo Ferraresi, Felice Iazzi, Federico Pinna et al.)....Pages 535-543
Front Matter ....Pages 545-545
SLIP-Based Concept of Combined Limb and Body Control of Force-Driven Robots (Patrick Vonwirth, Atabak Nejadfard, Krzysztof Mianowski, Karsten Berns)....Pages 547-556
Wizard of Botz: A Novel Approach to the Wizard of Oz Experiment (Kim Wölfel, Dominik Henrich)....Pages 557-564
Cloud-Based Digital Twin for Robot Integration in Intelligent Manufacturing Systems (Florin Anton, Theodor Borangiu, Silviu Răileanu, Silvia Anton)....Pages 565-573
Evaluation of Serial Metamorphic Manipulator Structures Considering Inertia Characteristics (N. A. Stravopodis, L. Katrantzis, V. C. Moulianitis, C. Valsamos, N. A. Aspragathos)....Pages 574-587
Vision-Way Testing in Design of Small Compliant Mechanisms (Jaroslav Hricko, Stefan Havlik)....Pages 588-595
Determination of Object Location for Robotic Grasping Using Depth Vision Sensor (Denis Chikurtev, Kaloyan Yovchev, Ava Chikurteva, Nayden Chivarov)....Pages 596-605
Back Matter ....Pages 607-609

Citation preview

Mechanisms and Machine Science

Saïd Zeghloul Med Amine Laribi Juan Sebastian Sandoval Arevalo   Editors

Advances in Service and Industrial Robotics Results of RAAD

Mechanisms and Machine Science Volume 84

Series Editor Marco Ceccarelli, Department of Industrial Engineering, University of Rome Tor Vergata, Roma, Italy Editorial Board Alfonso Hernandez, Mechanical Engineering, University of the Basque Country, Bilbao, Vizcaya, Spain Tian Huang, Department of Mechatronical Engineering, Tianjin University, Tianjin, China Yukio Takeda, Mechanical Engineering, Tokyo Institute of Technology, Tokyo, Japan Burkhard Corves, Institute of Mechanism Theory, Machine Dynamics and Robotics, RWTH Aachen University, Aachen, Nordrhein-Westfalen, Germany Sunil Agrawal, Department of Mechanical Engineering, Columbia University, New York, NY, USA

This book series establishes a well-defined forum for monographs, edited Books, and proceedings on mechanical engineering with particular emphasis on MMS (Mechanism and Machine Science). The final goal is the publication of research that shows the development of mechanical engineering and particularly MMS in all technical aspects, even in very recent assessments. Published works share an approach by which technical details and formulation are discussed, and discuss modern formalisms with the aim to circulate research and technical achievements for use in professional, research, academic, and teaching activities. This technical approach is an essential characteristic of the series. By discussing technical details and formulations in terms of modern formalisms, the possibility is created not only to show technical developments but also to explain achievements for technical teaching and research activity today and for the future. The book series is intended to collect technical views on developments of the broad field of MMS in a unique frame that can be seen in its totality as an Encyclopaedia of MMS but with the additional purpose of archiving and teaching MMS achievements. Therefore, the book series will be of use not only for researchers and teachers in Mechanical Engineering but also for professionals and students for their formation and future work. The series is promoted under the auspices of International Federation for the Promotion of Mechanism and Machine Science (IFToMM). Prospective authors and editors can contact Mr. Pierpaolo Riva (publishing editor, Springer) at: [email protected] Indexed by SCOPUS and Google Scholar.

More information about this series at http://www.springer.com/series/8779

Saïd Zeghloul Med Amine Laribi Juan Sebastian Sandoval Arevalo •

•

Editors

Advances in Service and Industrial Robotics Results of RAAD

123

Editors Saïd Zeghloul SP2MI - Site du Futuroscope University of Poitiers Poitiers Cedex 9, France

Med Amine Laribi SP2MI - Site du Futuroscope University of Poitiers Poitiers Cedex 9, France

Juan Sebastian Sandoval Arevalo SP2MI - Site du Futuroscope University of Poitiers Poitiers Cedex 9, France

ISSN 2211-0984 ISSN 2211-0992 (electronic) Mechanisms and Machine Science ISBN 978-3-030-48988-5 ISBN 978-3-030-48989-2 (eBook) https://doi.org/10.1007/978-3-030-48989-2 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

This book is the collection of the accepted papers submitted to the 29th International Conference on Robotics in Alpe-Adria-Danube Region, RAAD 2020. The conference offers an international and cooperative platform to exchange and discuss recent research results and future ideas about robotic technology, engineering, and science. Researchers from 16 countries, mostly affiliated to the Alpe-Adria-Danube Region, as well as researchers from extra-European Countries (India, Mexico, USA, Taiwan, Tunisia) participated to the present Conference. RAAD represents an event on Robotics that is organised yearly by a Community of robotic specialists in the Alpe-Adria-Danube Region of Europe. The first event was held in 1992, organized by the Jozef Stefan Institute, Ljubljana, Slovenia. It was named the 1st International Meeting on Robotics in Alpe Adria Region. In the following years the event was named International Workshop on Robotics in Alpe-Adria Region (RAA) until 1996, when the denomination was changed into International Conference on Robotics in Alpe-Adria-Danube Region (RAAD). According to its tradition, RAAD 2020 covers all major areas of research, development, and innovation in Robotics, including new applications and trends related to new theories, advanced design of robot mechanics and control architectures, and development of intelligent robotic applications. In addition, seven accepted special sessions in RAAD2020 offer deep insight into today’s hot research topics with a strong influence on application areas. For proposing, promoting, and collecting the related papers, a special acknowledgment is given to: Marco Ceccarelli and Alessandro Gasparetto (Italian Advances in RAAD Robotics), Theodor Borangiu, Aleksandar Rodić and Florin Anton (Cloud-based Robot Systems), Basilio Lenzo, Efstathios Velenis, Flavio Farroni, Francesco Timpone and Domenico Mundo (Vehicle dynamics and control), Tadej Petrič and Kosta Jovanović (Advances in Human-Robot Interaction), Giuseppe Carbone and Med Amine Laribi (Safety related devices and applications), Abdelbadiâ Chaker, Juan Sandoval and Med Amine Laribi (Assistive robotic devices, Fun2Rob project) and Andreas Müller and Michael Hofbaur (Industrial human-robot cooperation and collaboration).

v

vi

Preface

After a careful and accurate peer review process, with at least three reviews for each paper, 64 papers, authored by researchers from the RAAD community but also from other European and extra-European Countries, were considered suitable for publication in this book. The evaluation process has considered the relevance, novelty and clarity of the papers which guaranteed the high-quality level of this collection. We would like to thank all the reviewers for allocating their valuable time and effort for their outstanding work in reviewing the assigned papers. We are also very grateful to all the authors for their effort and excellent work to improve their papers. According to the planned conference sessions, this book is organized in 8 chapters, covering the major research areas of the conference: I. II. III. IV. V. VI. VII. VIII.

Assistive robotic devices, Fun2Rob project Safety related devices and applications Advances in Human-Robot Interaction Vehicle dynamics and control Italian Advances in RAAD Robotics Mechanical Design and Kinematics Learning Robot Control and Vision

We thank the University of Poitiers, in particular the Fundamental and Applied Science Faculty, for its availability to host the RAAD2020 event. We also acknowledge the support of the International Federation for the Promotion of Mechanism and Machine Science (IFToMM). The conference received generous support from local sponsors, namely the University of Poitiers, The Grand Poitiers and The Nouvelle Aquitaine region, which were crutial to make this Conference possible. Due to the epidemic situation of COVID-19 and because the safety as well as health of our attendees was our main concern, the International Scientific Committee of RAAD and the local organizing committee decided to postpone the conference to 2021 in Poitiers. We thank the publisher Springer and its Editorial staff for accepting and helping in the publication of this Proceedings volume within the book series on Mechanism and Machine Science (MMS). Poitiers, 2020

Saïd Zeghloul Med Amine Laribi Juan Sebastian Sandoval Arevalo

Organization

Organization Committee (Local) Saïd Zeghloul Med Amine Laribi Juan Sebastian Sandoval Arevalo Marc Arsicault Cyril Brèque

Univ. Univ. Univ. Univ. Univ.

of of of of of

Poitiers, Poitiers, Poitiers, Poitiers, Poitiers,

France France France France France

vii

Contents

Assistive Robotic Devices, Fun2Rob Project Optimal Design of a Rehabilitation Four Cable-Driven Parallel Robot for Daily Living Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ferdaws Ennaiem, Abdelbadiâ Chaker, Juan Sebastián Sandoval Arévalo, Med Amine Laribi, Sami Bennour, Abdelfattah Mlika, Lotfi Romdhane, and Saïd Zeghloul Pneumatic Equipment for Ankle Rehabilitation by Continuous Passive Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Andrea Deaconescu and Tudor Deaconescu

3

13

Simulation Platform for Crane Visibility Safety Assistance . . . . . . . . . . Tanittha Sutjaritvorakul, Axel Vierling, Jakub Pawlak, and Karsten Berns

22

Design of Robotic Braces for Patients with Scoliosis . . . . . . . . . . . . . . . Rahul Ray, Laurence Nouaille, Briac Colobert, and Gérard Poisson

30

Comparison Between Two Methods of Cables Tensions Optimization of a Cable Driven Parallel Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hajer Ben Amor, Sami Bennour, and Abdelfattah Mlika

38

Computational Method for Muscle Forces Estimation Based on Hill Rheological Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Olfa Jemaa, Sami Bennour, David Daney, and Lotfi Romdhane

46

Design of a Variable Stiffness Joint for a Five-Bar-Mechanism . . . . . . . Maria Guadalupe Contreras-Calderón and Eduardo Castillo-Castañeda Visual Control Based Musculotendon Force Estimation in the Human Upper-Limb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Souha Baklouti, Olfa Jemaa, and Sami Bennour

54

64

ix

x

Contents

Design and Simulation of an Underactuated Exoskeleton Mechanism for a User-Oriented Leg Exercising . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yao Shuangji and Giuseppe Carbone

73

Safety Related Devices and Applications Remote Center of Motion for Redundant Robotic-Assisted Ultrasound Guided Regional Anesthesia . . . . . . . . . . . . . . . . . . . . . . . . . Mohammad Alkhatib, Cyril Novales, Laurence Nouaille, Adel Hafiane, and Pierre Vieyres Reference Frame Identification and Distributed Control Strategies in Human-Robot Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alberto Borboni, Giuseppe Carbone, and Nicola Pellegrini

85

93

Static Balancing of a 5-DoF Mechanism with Manual Reconfiguration Feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Terence Essomba and Zi-Wei Zhao Novel Safety Criterion for Synthesis of Cable Driven Parallel Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 Ines Ben Hamida, Med Amine Laribi, Abdelfattah Mlika, Lotfi Romdhane, Saïd Zeghloul, and Giuseppe Carbone Human Activity Recognition Enhanced Robot-Assisted Minimally Invasive Surgery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Hang Su, Wen Qi, Chenguang Yang, Jiehao Li, Xuanyi Zhou, Giancarlo Ferrigno, and Elena De Momi Online Payload Identification of a Franka Emika Robot for Medical Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 Souad Salah, Juan Sandoval, Moncef Ghiss, Med Amine Laribi, and Saïd Zeghloul Robot Motion Control Using EMG Signals and Expert System for Teleoperation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 George Pagounis, Panagiotis Koustoumpardis, and Nikos Aspragathos PRSX: An End-Effector for Pronation and Supination Adaptable to Arm Rehabilitation Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 Maria Guadalupe Contreras-Calderón and Eduardo Castillo-Castañeda Criteria for the Design and Application of Socially Assistive Robots in Interventions for Children with Autism . . . . . . . . . . . . . . . . . . . . . . . 159 Ilias A. Katsanis and Vassilios C. Moulianitis

Contents

xi

Advances in Human-Robot Interaction Generation of Smooth Cartesian Paths Using Radial Basis Functions . . . 171 Leon Žlajpah and Tadej Petrič Combining Virtual and Physical Guides for Autonomous In-Contact Path Adaptation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Tadej Petrič and Leon Žlajpah A Framework for the Study of Human-Robot Collaboration in Rehabilitation Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 Giorgia Chiriatti, Giacomo Palmieri, and Matteo Claudio Palpacelli Dyadic Human-Human Interactions in Reaching Tasks: Fitts’ Law for Two . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 Rebeka Kropivšek Leskovar, Jernej Čamernik, and Tadej Petrič Maximizing the End-Effector Cartesian Stiffness Range for Kinematic Redundant Robot with Compliance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 Branko Lukić, Kosta Jovanović, Nikola Knežević, Leon Žlajpah, and Tadej Petrič Comparison of Model-Based Simultaneous Position and Stiffness Control Techniques for Pneumatic Soft Robots . . . . . . . . . . . . . . . . . . . 218 Maja Trumić, Kosta Jovanović, and Adriano Fagiolini Shaking Force Balancing of the Orthoglide . . . . . . . . . . . . . . . . . . . . . . 227 Jing Geng, Vigen Arakelian, Damien Chablat, and Philippe Lemoine A Boosted Decision Tree Approach for a Safe Human-Robot Collaboration in Quasi-static Impact Situations . . . . . . . . . . . . . . . . . . . 235 Nemanja Kovincic, Hubert Gattringer, Andreas Müller, and Mathias Brandstötter Combining the Robot Operating System with Building Information Modeling for Robotic Applications in Construction Logistics . . . . . . . . . 245 Camilla Follini, Michael Terzer, Carmen Marcher, Andrea Giusti, and Dominik Tobias Matt Vehicle Dynamics and Control Design of a Thunniform Swimming Robot in a Multiphysics Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 Daniele Costa, Giacomo Palmieri, Matteo Palpacelli, and David Scaradozzi Analysis of Tire Temperature Influence on Vehicle Dynamic Behaviour Using a 15 DOF Lumped-Parameter Full-Car Model . . . . . . 266 Michele Perrelli, Flavio Farroni, Francesco Timpone, and Domenico Mundo

xii

Contents

Motion Sickness Minimisation in Autonomous Vehicles Using Optimal Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 Zaw Htike, Georgios Papaioannou, Efstathios Siampis, Efstathios Velenis, and Stefano Longo A Tire/Road Interaction Tool for the Evaluation of Tire Wear for Commercial Vehicles . . . . . . . . . . . . . . . . . . . . . . . . . . 283 M. De Martino, Argemiro L. A. Costa, F. Timpone, and A. Sakhnevych Multi Actuation Scheme for Path-Following Control of Autonomous Vehicles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 Javad Ahmadi, Efstathios Velenis, Heimoud El Vagha, Chenhui Lin, Boyuan Li, Stefano Longo, and Efstathios Siampis Italian Advances in RAAD Robotics A Saffron Spice Separation System with Computer Vision . . . . . . . . . . . 305 A. Manuello Bertetto and A. Prete Prismatic Delta Robot: A Lagrangian Approach . . . . . . . . . . . . . . . . . . 315 Federico Colombo and Luigi Lentini Italian Contributions to RAAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Marco Ceccarelli Minimum-Energy Trajectory Planning for Industrial Robotic Applications: Analytical Model and Experimental Results . . . . . . . . . . . 334 Lorenzo Scalera, Giovanni Carabin, Renato Vidoni, and Alessandro Gasparetto Mechanical Design and Kinematics Conceptual Design and Control of a Sitting-Type Lower-Limb Rehabilitation System Established on a Spatial 3-PRRR Parallel Manipulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 Santhakumar Mohan, Parvathi Sunilkumar, Larisa Rybak, Dmitry Malyshev, Sergey Khalapyan, and Anna Nozdracheva The First Type of Singularity of a 3-PRRS Parallel Manipulator . . . . . 356 Zhumadil Baigunchekov, Med Amine Laribi, Myrzabai Izmambetov, Zhadyra Zhumasheva, and Rustem Kaiyrov Inverse Kinematics of a 3-PRPS Type Parallel Manipulator . . . . . . . . . 364 Zhumadil Baigunchekov, Said Zeghloul, and Abzal Kassinov Structurally Parametric Synthesis of a RoboMech Class Parallel Manipulator with Three DOF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 Zhumadil Baigunchekov, Sobh Tarek, Sarosh Patel, and Azamat Mustafa

Contents

xiii

Functional Design of a Novel Over-Actuated Mobile Robotic Platform for Assistive Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380 Luca Carbonari, Andrea Botta, Paride Cavallone, and Giuseppe Quaglia Mechanism for the Locomotion Layout Reconfiguration of the Agri_q Mobile Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 Carmen Visconte, Paride Cavallone, Luca Carbonari, Andrea Botta, and Giuseppe Quaglia Design and Simulation of a Parallel-Mechanism Testbed for Head Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400 José Luis Rueda Arreguín, Marco Ceccarelli, and Christopher René Torres San Miguél A New RCM Mechanism for an Ear and Facial Surgical Application . . . . Guillaume Michel, Durgesh Haribhau Salunkhe, Damien Chablat, and Philippe Bordure

408

Design of Serial Link Structure-Parallel Wire System for Virtual Reality Rehabilitation and Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . 419 Vishal Ramadoss, Mohamed Sadiq Ikbal, Dimiter Zlatanov, and Matteo Zoppi Mathematical Model and Experimental Validation for a Four Bar Mechanism with a Flexible Coupler Link . . . . . . . . . . . . . . . . . . . . . . . . 428 Daniel Rodríguez Flores, Héctor Cervantes Culebro, and Carlos-Alberto Cruz-Villar Some Quality Measures in Designing Compliant Mechanisms for Robotic Devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438 Stefan Havlik and Jaroslav Hricko Prototype Design and Testing of TORVEastro, Cable-Driven Astronaut Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448 Francesco Samani and Marco Ceccarelli Design and Lab Experience for a Pipeline Service Robot in Space Orbital Stations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456 Tao Zheng, Hui Li, Qiang Huang, Xiang Wang, Marco Ceccarelli, Zhihong Jiang, Pingli Wang, Sanliang Wang, Yanyan Zhao, and Xiao Liu Nonlinear Dynamics of a Spatial Two Link Chain . . . . . . . . . . . . . . . . . 466 Dan B. Marghitu and Dorian Cojocaru Learning Autonomous Navigation in Vineyards with Deep Learning at the Edge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 Diego Aghi, Vittorio Mazzia, and Marcello Chiaberge

xiv

Contents

Iterative Learning Control of Hard Constrained Robotic Manipulators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487 Kaloyan Yovchev, Lyubomira Miteva, Kamen Delchev, and Evgeniy Krastev Generalization Based Database Acquisition for Robot Learning in Reduced Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496 Zvezdan Lončarević, Rok Pahič, Mihael Simonič, Aleš Ude, and Andrej Gams Analysis of Different Methods to Close the Reality Gap for Instance Segmentation in a Flexible Assembly Cell . . . . . . . . . . . . . . . . . . . . . . . 505 Nawal Hafez, Vincent Dietrich, and Stefan Roehrl Data-Driven Synthesis of Perception Pipelines via Hierarchical Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516 Vincent Dietrich, Bernd Kast, Sebastian Albrecht, and Michael Beetz Crashing to Learn, Learning to Survive: Planning in Dynamic Environments via Domain Randomization . . . . . . . . . . . . . . . . . . . . . . . 525 Sarath S. Menon, R. Sakthi Vignesh, and Santhakumar Mohan Target Manipulation in Nuclear Physics Experiment with Ion Beams . . . 535 Diego Sartirana, Daniela Calvo, Vittoria Capirossi, Carlo Ferraresi, Felice Iazzi, Federico Pinna, and for NUMEN collaboration Robot Control and Vision SLIP-Based Concept of Combined Limb and Body Control of Force-Driven Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547 Patrick Vonwirth, Atabak Nejadfard, Krzysztof Mianowski, and Karsten Berns Wizard of Botz: A Novel Approach to the Wizard of Oz Experiment . . . 557 Kim Wölfel and Dominik Henrich Cloud-Based Digital Twin for Robot Integration in Intelligent Manufacturing Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565 Florin Anton, Theodor Borangiu, Silviu Răileanu, and Silvia Anton Evaluation of Serial Metamorphic Manipulator Structures Considering Inertia Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574 N. A. Stravopodis, L. Katrantzis, V. C. Moulianitis, C. Valsamos, and N. A. Aspragathos Vision-Way Testing in Design of Small Compliant Mechanisms . . . . . . . 588 Jaroslav Hricko and Stefan Havlik

Contents

xv

Determination of Object Location for Robotic Grasping Using Depth Vision Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596 Denis Chikurtev, Kaloyan Yovchev, Ava Chikurteva, and Nayden Chivarov Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 607

Assistive Robotic Devices, Fun2Rob Project

Optimal Design of a Rehabilitation Four Cable-Driven Parallel Robot for Daily Living Activities Ferdaws Ennaiem1,2(&), Abdelbadiâ Chaker2, Juan Sebastián Sandoval Arévalo1, Med Amine Laribi1, Sami Bennour2, Abdelfattah Mlika2, Lotfi Romdhane2,3, and Saïd Zeghloul1 1

Department of GMSC, Pprime Institute CNRS, ENSMA, UPR 3346, University of Poitiers, Poitiers, France {ferdaws.ennaiem,juan.sebastian.sandoval.arevalo, med.amine.laribi,said.zeghloul}@univ-poitiers.fr 2 Mechanical Laboratory of Sousse (LMS), National Engineering School of Sousse, University of Sousse, Sousse 4000, Tunisia {abdelbadia.chaker,sami.bennour.meca, abdelfattah.mlika}@eniso.u-sousse.tn 3 American University of Sharjah, PO Box 26666, Sharjah, United Arab Emirates [email protected]

Abstract. In this paper, three rehabilitation movements of the human upper extremity, which were suggested by physiotherapists, are analyzed. The geometric and kinematic characteristics of each movement are identified using a motion capture system. Based on these measurements and design constraints, a preliminary architecture of a cable-driven parallel robot (CDPR), capable of reproducing the movements, is presented. An optimization of the CDPR is then conducted, using the Genetic Algorithm method, where the objectives are the simultaneous minimization of the robot size and the tensions in the cables. The dexterity and the elastic stiffness of the resulting robot are also investigated. Keywords: Rehabilitation movements
Motion capture system
Cable-driven parallel robot
GA method

1 Introduction The typical rehabilitation is based on the manual assistance performed by a physiotherapist. Its goal is to help patients with motor impairments regain totally or partially their functional abilities. To ensure the effectiveness of this therapy and improve its quality, the physiotherapist needs to perform repetitive motions. However, the availability of specialists and the duration of sessions are limited. In addition, the physiotherapist’s performance decreases with time and the consistency of reproducing the movements is not guaranteed. Rehabilitation robots are known to have good repeatability and do not suffer from fatigue, which makes them a good alternative to manual © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 3–12, 2020. https://doi.org/10.1007/978-3-030-48989-2_1

4

F. Ennaiem et al.

rehabilitation [1–3]. Indeed, robots can assist and move the impaired member accurately and in a controlled manner. Moreover, robot sensors could be helpful in monitoring the patient movement and the responsiveness to the treatment. This paper proposes an optimized structure of a cable-driven-parallel-robot (CDPR) able to move the patient impaired member along some paths, prescribed by the physiotherapist. This paper is structured as follows: Sect. 2 details the rehabilitation movements, identified by the physiotherapist. The experimental protocol uses a motion capture system to collect kinematic data. Section 3 proposes the design of the robot. Section 4 formulates the optimization problem as well as the criteria and the constraints selected to obtain the optimal solution. The validation of the results is illustrated in Sect. 5. The last section concludes this paper.

2 Rehabilitation Tasks The rehabilitation where the physiotherapist assists the patients to perform daily living activities gives better outcomes compared with the rehabilitation of each joint separately [4]. With the help of physiotherapists from Poitiers University Hospital, three daily activities illustrated in Fig. 1, were identified. They consist of moving the patient hand from an initial position to a final position, where the patient touches the mouth, the head or the shoulder then returns to the starting position. These movements are involved in basic daily tasks, where the patient is eating, brushing his hair, or wearing clothes.

Fig. 1. Movements prescribed by ergo-therapists.

Optimal Design of a Rehabilitation Four Cable-Driven Parallel Robot

5

In order to study the gestures of these three movements and characterize each elementary activity, an experimental measurement campaign is carried out, using a motion capture system. All prescribed movements have been recorded using a Qualisys motion capture system composed of 8 high-resolution infrared cameras and 19 reflective markers fixed in anatomical locations as shown in Fig. 2. Six volunteers were asked to perform 5 cycles of each movement. Since the patient’s hand will be attached to the end-effector of the robot, the prescribed trajectories are obtained by tracking the positions taken by the marker located at the center of the patient’s hand (H_3). A comparative study was carried out in order to choose the largest recorded trajectories to be selected as the task workspace of the robot which allows to cope with the maximum number of patients. The data given by the other markers are exploited to identify the workspace of the patient upper extremity during the movements.

Fig. 2. Markers locations.

3 Selection of the Structure The design of a robot for rehabilitation has to take into account several requirements such as the ergonomics, the stiffness of the structure, the low cost and the ability to reach the task workspace [5]. In this paper, a CDPR with 3 translational degrees of freedom (dof) is considered. To fully constrain this robot 4 cables are needed. Several structure designs can be proposed as candidates for the CDPR. A comparison among these structures is performed in [6] and a rating is given as illustrated in Table 1.

6

F. Ennaiem et al. Table 1. Rating of different candidate structures.

The comparison showed that the Triangular structure is the most suitable candidate for the rehabilitation task. Actually, circular profiles manufacturing is costly. Regarding the rectangular structure, it can be a suitable choice if more than 4 cables are used. In fact, to fully constrain a spatial robot with 4 cables, no more than 3 vertices of one face are needed for the cable exit points. Thus, the size, the cost and the weight of the rectangular structure will be overestimated since it provides 4 vertices in each face. Regarding the workspace, a structure with several vertical bars prevents the patient upper limb from moving freely. The design of the adopted triangular structure is given in Fig. 3. In this paper, the end-effector is assumed to be a point mass. An appropriate orthosis will be designed in a future work. The workspace of the proposed structure is the volume formed by the intersection between 4 spheres, expressed mathematically by Eq. (1) (the attainable workspace) and the set of points satisfying Eq. (2) (the static equilibrium workspace). The center of each sphere is located at the cable exit point and its radius is the maximum length identical for all the cables. ðx  xMi Þ2 þ ðy  yMi Þ2 þ ðz  zMi Þ2 ¼ l2max X Fext=EE ¼ 0

ð1Þ ð2Þ

Optimal Design of a Rehabilitation Four Cable-Driven Parallel Robot

7

Fig. 3. CAD of the proposed structure.

Where x, y, and z are the coordinates of the center of the mass of the platform, xMi, yMi, and zMi are cable exit points’ coordinates, lmax is the maximum length identical for all the cables and Fext/EE is the external forces acting on the end-effector including the weight force and the cable tensions.

4 Formulation and Optimization Process The CDPR structure can be described using 9 design parameters a1, b1, c1, a2, b2, a3, b3, a4, and b4, as shown in Fig. 4. To fully constrain the robot, we fixed the two variables a4 and b4 so that the point M4 is the projection of the trajectories starting position onto the table plane. The other parameters represent the design vector I of the robot. The location of the robot in the reference frame R0 is defined using the

8

F. Ennaiem et al.

coordinates of the point M1. The coordinates of the points M2 and M3 are computed according to the point M1 as follows: M1ða1; b1; c1Þ

ð3Þ

M2ða1 þ a2; b1 þ b2; c1Þ

ð4Þ

M3ða1 þ a3; b1 þ b3; c1Þ

ð5Þ

Fig. 4. The structure and design vector variables.

The aim is to find the optimal design vector I* of the CDPR with the smallest size including the three studied movements trajectories in its workspace. In order to reduce the energy consumption, each motor must provide a minimum torque, in other words, the minimum cable tensions need to be selected. In addition, due to the fact that cables can only pull and not push, the tensions should be positive inside the workspace. The mathematical formulations of these criteria are given by Eqs. (6) and (7). To compute the cable tensions vector Tc, a quasi-static assumption was made. TðIÞ ¼ SðIÞ ¼

qffiffiffiffiffiffiffiffiffiffi TcT Tc

ð6Þ

X4   ðx  xMi Þ2 þ ðy  yMi Þ2 þ ðz  zMi Þ2 l2  max

i¼1

ð7Þ

T(I) represents the norm of the cable tensions vector. S(I) computes the distance between the end-effector position and the bounding surface of each sphere delimiting the workspace. The smallest size of the structure is obtained when S(I) takes the minimum possible value for each position. The cables tensions are considered to be bounded between the minimum and the maximum acceptable tensions for each position. Finally, the optimization problem with penalty formulation can be summarized as follows: Min ðFðIÞÞ Subject to 0  Tmin  Tc  Tmax

ð8Þ

Optimal Design of a Rehabilitation Four Cable-Driven Parallel Robot

9

Where FðIÞ ¼ b1

TðIÞ SðIÞ þ b2 2 þ P1 þ P2 Tmax lmax

I ¼ ½a1; b1; c1; a2; b2; a3; b3  0 if Tc ðiÞ  Tmin P1 ¼ w if Tc ðiÞ  Tmin  P2 ¼

0 if w if

Tc ðiÞ  Tmax Tc ðiÞ  Tmax

ð9Þ ð10Þ ð11Þ ð12Þ

W is a large scalar, Tc(i) denotes the ith component of the cable tensions vector, Tmin and Tmax are the minimum and the maximum values of Tc, respectively. Two penalty functions, P1 and P2, are used to handle the optimization problem constraints and eliminate candidates that do not respect mandatory cable tension boundaries. In order to simplify the optimization process, the multi-objective problem is formulated as a mono objective problem using two coefficients b1 and b2 that satisfy Eq. (13). By changing the coefficient values between 0 and 1, the importance for each criterion can be modified and then as a consequence the quality of the final solution. b1 þ b2 ¼ 1

ð13Þ

5 Results and Discussions The Genetic Algorithm method [7] implemented under MATLAB is used to solve the optimization problem. This algorithm is initialized using the parameters listed in Table 2. The upper and the lower boundaries of the design variables, used to generate the initial population, are given in Table 3.

Table 2. Initialization values. Parameters Population size Tmin [N] Tmax [N] a4 [m]

Values 150 0.5 15 0.21

Parameters Values 1 lmax [m] Platform weight [kg] 1.5 b4 [m] 0.011

10

F. Ennaiem et al. Table 3. The optimal solution, the lower and the upper boundaries of the design vector. I Lower bounds I* (b1 = b2 = 0.5) I* (b1 = 0.7, b2 = 0.3) I* (b1 = 0.3, b2 = 0.7) Upper bounds

a1 −1.5 −0.1 −0.23 −0.21 1.5

b1 −1.5 −0.3 −0.2 −0.23 1.5

c1 0 1.48 1.43 1.45 1.5

a2 −1 −0.35 −0.53 −0.4 1

b2 0.5 1.03 1.16 1.13 1.5

a3 0.5 0.86 0.83 0.92 1

b3 0 0.23 0.22 0.17 1.5

In this paper, we consider that the two criteria mentioned above have the same importance ðb1 ¼ b2 ¼ 0:5Þ: The optimal solution obtained, after 25 generations with a fitness function equal to 2.65, is given in Table 3. A spatial representation of the CDPR, as well as the prescribed trajectories, are given in Fig. 5. The proposed structure presented in Sect. 3 will be adjusted so that the cable exit points positions are those resulting from the optimization.

Fig. 5. The optimal positions of the cable exit points and the prescribed trajectories.

To have a better judgment about the optimization results, the elastic stiffness and the dexterity of the resulting robot are investigated. Their distributions, over the workspace, for z equal to 0.5 m are illustrated in Fig. 6. For each pose, the dexterity index is defined as the inverse of the condition number of the Jacobian matrix [8] and the elastic stiffness index is defined as the ratio of the largest and the smallest eigenvalues of the elastic stiffness matrix. These two kinematic properties can be integrated into the optimization process in future works, in order to have better performances. The variations of the cable tensions for each movement are given in Fig. 7. The role of wire 4 is to keep all the cables taut. Its tension is equal to Tmin.

Optimal Design of a Rehabilitation Four Cable-Driven Parallel Robot

(a)

11

(b)

Fig. 6. (a) The dexterity distribution, (b) the elastic stiffness distribution.

(b)

(a)

(c) Fig. 7. The variation of the cable tensions for (a) the hand-head movement, (b) the hand-mouth movement and (c) the hand-shoulder movement.

12

F. Ennaiem et al.

6 Conclusion A preliminary design of a cable-driven parallel robot, with 4 cables and 3 degrees of freedom, intended for helping patients with motor impairments to move their upper member along specific trajectories is proposed in this paper. The trajectories are recorded using a motion capture system and then analyzed to determine the task workspace. An optimization approach based on a Genetic Algorithm was formulated considering the cable exit points coordinates as the design variables. The optimal solution is selected according to two criteria, the minimum cable tensions distribution and the smallest structure size. The dexterity and the elastic stiffness of the resulting robot were then investigated. Acknowledgments. This work was financially supported by the “PHC Utique” program of the French Ministry of Foreign Affairs and Ministry of higher education, research and innovation and the Tunisian Ministry of higher education and scientific research in the CMCU project number 19G1121.

References 1. Lum, P.S., Burgar, C.G., Van der Loos, M., Shor, P.C.: MIME robotic device for upper-limb neurorehabilitation in subacute stroke subjects: A follow-up study. J. Rehabil. Res. Dev. 43 (5), 631–642 (2006) 2. Nef, T., Riener, R.: ARMin-design of a novel arm rehabilitation robot. In: 9th International Conference on Rehabilitation Robotics, ICORR 2005, pp. 57–60. IEEE (2005) 3. Reinkensmeyer, D.J., Kahn, L.E., Averbuch, M., et al.: Understanding and treating arm movement impairment after chronic brain injury: progress with the ARM guide. J. Rehabil. Res. Dev. 37(6), 653–662 (2000) 4. Langhammer, B., Stanghelle, J.K.: Bobath or motor relearning programme? a comparison of two different approaches of physiotherapy in stroke rehabilitation: a randomized controlled study. Clin. Rehabil. 14(4), 361–369 (2000) 5. Riener, R., Nef, T., Colombo, G.: Robot-aided neurorehabilitation of the upper extremities. Med. Biol. Eng. Comput. 43(1), 2–10 (2005) 6. Kozisek, A.: Cable driven parallel robot for lower limb rehabilitation tasks (master dissertation), France: Poitiers University. Italy: Rome Tor Vergata University (2019) 7. Shorman, S.M., Pitchay, S.A.: Significance of parameters in genetic algorithm, the strengths, its limitations and challenges in image recovery. ARPN J. Eng. Appl. Sci. 10(2), 585–593 (2006) 8. Gosselin, C.: Kinematic Analysis, Optimization and Programming of Parallel Robotic Manipulators. McGill University, Montréal (1988)

Pneumatic Equipment for Ankle Rehabilitation by Continuous Passive Motion Andrea Deaconescu and Tudor Deaconescu(&) Transilvania University, Brasov, Romania {deacon,tdeacon}@unitbv.ro

Abstract. Patients suffering from affections of the lower limb bearing joints account for a significant number of the total of post-traumatic conditions. It is the task of rehabilitation medicine to expedite the recovery of such patients, one of the recovery techniques being Continuous Passive Motion (CPM). Applying recovery-aimed motions to the lower limb joints – known as passive motions requires the development of new high-performance equipment. The paper presents and discusses the kinematics, construction and actuation of a novel rehabilitation device actuated by a pneumatic muscle developed for the recovery of the ankle by continuous passive motion. Due to the used energy source compressed air - the end-of-stroke shocks are completely absorbed and user discomfort is thus minimized. Keywords: Rehabilitation equipment Pneumatic muscle


Continuous Passive Motion


1 Introduction Maintaining the functions of the human body and/or the recovery of such is achieved by kinetic and orthotic techniques specific to rehabilitation medicine. At present, dysfunctions of the lower limb joints are ameliorated by means of passive mobilization exercises of the joints, of exercises conceived to strengthen the patient’s muscles as well as by means of adaptive equipment (prostheses, orthoses, walking aids). Kinesitherapy is one of the main recovery methods of the diminished functions of lower limb joints, and it includes techniques that involve motion as well as various forms of relaxation or immobilisation. The numerous studies published to date agree on the fact that the posttraumatic repose of the joints, while necessary in many cases, needs to be limited in time because of its negative effects on the patient body that can be inter alia of neuro-muscular, osteo-articular, cardio-vascular, or respiratory nature. Although bed rest alleviates pain, it has to be short, not exceeding 2 or 3 days. The use as part of the recovery process of equipment capable of mobilising the affected joints can reduce rehabilitation costs by up to approximately 50% compared to traditional non-mechanized procedures. In conclusion, as set out above, any short period of repose has to be followed by rehabilitation motions, whether equipmentassisted or not. Research conducted by the National Institute of Physical Medicine, Balneology and Medical Recovery of Bucharest, Romania on a representative sample of patients have © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 13–21, 2020. https://doi.org/10.1007/978-3-030-48989-2_2

14

A. Deaconescu and T. Deaconescu

revealed that patients with affections of lower limb bearing joints (hip, knee or ankle fractures) represent 54.25% of the total recorded post-traumatic disabilities [1]. Within the studied lower limb affections most cases were hip fractures (43.33%), followed by knee and ankle fractures, representing 30% and 26.7% respectively [1]. Figure 1 illustrates the incidence of the diagnosis groups by affection and sex.

Fig. 1. Incidence of the diagnosis groups by sex.

The research conducted on the sample of patients that benefitted from kinesitherapy in comparison to patients who had not performed rehabilitation exercises has yielded the following conclusions [1]: • • • • •

improvement of the pain score by 30.02%; improvement of the physical dysfunctions by 24.22%; improvement of disabilities by 24.22%; reduction of pain medication consumption by 21.82%; improvement of the general mean score of the entire sample by 24.12%, the hip fracture group featuring the highest values – 27.74%, followed by the knee fracture (22.85%) and ankle fracture groups (19.76%).

The data presented above lead to the idea of implementing kinesitherapy as a mandatory means for the post-traumatic treatment of the bearing joints. One of the techniques used in kinesitherapy is continuous passive mobilisation, which is an optimum instrument within the therapeutic toolkit of rehabilitation professionals. This technique entails setting the affected joints into motion by mechanized means without requiring any force generated by the patient’s muscles. Continuous passive motion (CPM) is the first step of the rehabilitation process. CPM aims at controlling post-surgical pain, reducing inflammation and setting the affected joint into slow and uninterrupted motion according to a plan devised according to each patient. CPM medical recovery exercises can be carried out with the help of a professional, a kinesitherapist or by means of specially conceived equipment. CPM rehabilitation devices are medical systems that allow the mobilisation of the joints and muscles at various velocities, following the recommendations of kinesitherapists. Such equipment includes pluri-articular (multi-joint) devices that allow – by means of accessories – the testing and rehabilitation of the key joints, or mono-articular (single-joint) devices dedicated to a single specific joint. The ample marketplace of lower limb rehabilitation equipment is dominated by electrically actuated ones. Figure 2 shows examples of such pluri-articular rehabilitation

Pneumatic Equipment for Ankle Rehabilitation by Continuous Passive Motion

15

devices with the commercial names Fisiotek 3000 TS and Kinetec Spectra Knee CPM, respectively [2, 3]. These pieces of equipment allow the performance of rehabilitation motions within the limits imposed by the biomechanics of the hip, knee and ankle, respectively.

Fig. 2. Examples of electrically actuated rehabilitation equipment.

The advantage of such equipment is that they can be used not only in specialised rehabilitation centres, but also in the patient homes. Medical rehabilitation equipment needs to have a compliant behaviour, that is the capacity of adapting quickly to the patient’s momentary state, often different than initially envisaged. Electrically actuated equipment meets this requirement to a smaller degree, as ensuring their compliant behaviour calls for drastically increasing the sensorization and adequate design of the control diagrams [4]. Excessive sensorization can be avoided by using adjustable compliance actuators (ACAs), characterized by safe interaction with human operators and by the ability of storing and releasing energy into passive elastic elements [5]. The pneumatic muscle is such an actuator of inherently compliant behaviour due to air compressibility. The few existing pneumatic muscle actuated rehabilitation devices of the lower limb are still in prototype phase (patents). A number of such designs can be mentioned here, like the intelligent transtibial prosthesis [6] or rehabilitation devices actuated by elastic rotating chambers (rotating pneumatic muscles) described in [7] (Fig. 3).

Fig. 3. Pneumatic muscle actuated rehabilitation equipment.

16

A. Deaconescu and T. Deaconescu

Given the significant share held by ankle fractures (26.7%) of the total of affections of the lower limb joints, this paper puts forward and discusses a pneumatic muscle actuated rehabilitation device developed for this joint.

2 Design Requirements The functional requirements to be met by such a device resulted from a thorough study of the ankle biomechanics. Thus, the limits for the motions carried out by each component of the joint were determined as well as the forces and couples that are necessary for performing such motions. The ankle links by means of the talus the bones of the lower leg (the tibia and the fibula) to the foot skeleton. It is joint that is essential for walking, as it ensures the statics and motion of the lower limb in bipedal position. Further, the ankle allows the plantar flexion and dorsiflexion of the foot. Plantar flexion is the motion that decreases the angle between the sole and the back of the foot, while dorsiflexion is the opposite motion by that the toes are brought towards the shin. Given the known angles achievable by the ankle motions (Fig. 4), further on proposed is a constructive solution of a rehabilitation device capable of meeting the imposed requirements.

Fig. 4. Motions of the ankle.

3 Construction of the Ankle Rehabilitation Equipment Given the requirement of ensuring a compliant behaviour of the ankle rehabilitation device, its actuation is achieved by a linear pneumatic muscle. Figure 5 shows the diagram of principle and the constructive diagram of the equipment. The system developed for mobilizing the ankle consists of two identical, parallelconnected mechanisms that support the patient’s leg. Each of these mechanisms

Pneumatic Equipment for Ankle Rehabilitation by Continuous Passive Motion

17

Fig. 5. Diagram of the ankle rehabilitation equipment.

consists of two bars (OB and BD) and three rotation couples (O, B and D) and a translation couple (D). Angle u3 is of interest, as it describes the rotation of the ankle. The actuation motor is a pneumatic muscle manufactured by Festo (Germany), of 20 mm interior diameter and 750 mm initial length. Using the pneumatic muscle solves some of the requirements that have to be met by the rehabilitation equipment. Thus, the price of a muscle is significantly lower than that of an actuation system consisting of an electric motor plus a screw and nut mechanism, what eventually yields a less costly device. Further, by using a pneumatic muscle endof-stroke shocks are avoided. The following quantities are determined for this device: • • • •

the the the the

limits reached by the slide; limits of the hip rotation angle u1 ; limits of the ankle rotation angle u3 evolution of these angles over the complete stroke of the slide.

The bar lengths used in the calculation of these key quantities of the rehabilitation are OB = 1000 mm; BD = 168 mm. Further, from Fig. 5 also follow the imposed limits of the ankle rotation angle, namely +20° and −40°, respectively, in relation to a vertical axis (an angular amplitude of 60°). Given the known geometry of the device the necessary displacement of the slide (joint D) can be calculated, namely x = 161 mm. As the maximum possible stroke of the free end of the muscle is of about 20% of its length in relaxed state (150 mm), between the muscle and the slide a mobile pulley mechanism was introduced in order to amplify the slide displacement to the necessary value. While the benefit of the pulley system is the doubling of the slide stroke, its disadvantage consists in halving the force supplied by the pneumatic muscle to the slide. Figure 6 shows the variation of angles u1 and u3 , as well as the limits of the stroke carried out by slide D upon feeding compressed air to the pneumatic muscle.

18

A. Deaconescu and T. Deaconescu

Fig. 6. Kinematic diagram of the rehabilitation equipment.

Using the notations of Fig. 6, the hip rotation angle (u1 ) is calculated with Eq. (1): 2  l3 þ l21  l22 2
l3
l1

ð1Þ

2  l1 þ l22  l23 u3 ¼ arccos 2
l1
l2

ð2Þ

u1 ¼ arccos and angle u3 is:

The angles u1 and u3 corresponding to the limit positions of the slide (in points D and D’) will have the following values: • corresponding to point D: u1 = 9.08° and u3 = 100.916° • corresponding to point D’: u1 = 7.39° and u3 = 42.605°. For an intermediary position of the slide, when the bar of length BD is perpendicular on the x-axis, the maximum calculated value of angle u1 is u1 = 9.67°.

Fig. 7. Variation of angle u1 versus the Fig. 8. Variation of angle u3 versus the position of the slide (from point O). position of the slide (from point O).

Pneumatic Equipment for Ankle Rehabilitation by Continuous Passive Motion

19

Figures 7 and 8 present the variations of the two angles versus the position of the slide (joint D). It can be noticed that for the completely withdrawn position of the slide (l3 = 883.7 mm) angle u1 is of 7.39°, then increases to 9.67° for an intermediary position of the slide, and eventually decreases to 9.08° for the completely expanded state of the slide when l3 = 1044.9 mm. Of interest in the of the ankle is the angle between the foot and the tibia. This is determined as: u3 supplementary = 180° − u3 . According to calculations this angle varies between 137.395° and 79.084°, the amplitude of the variation being of 58.311°, that is approximately equal to the value proposed for the equipment (60°). Figure 9 shows the graph describing the variation of this angle versus the displacement of the slide.

Fig. 9. Variation of angle u3

supplementary

versus the position of the slide.

4 Actuation of the Rehabilitation Equipment Figure 10 presents the pneumatic actuation diagram. The system includes: the pneumatic muscle actuated by an MPYE proportional directional control valve manufactured by Festo; a resistive position transducer attached to the slide; an SPC 200 controller for programming and saving the working positions, the types of motion and their sequence; electronic elements for the transmission of information from the transducer to the controller and of the commands issued by the controller to the proportional valve (electronic elements included by the SPC-AIF interface). The measured quantity is a displacement related to the origin of the frame of reference and is provided by the position transducer. The system allows the programming of one or more cycles of rehabilitation exercises, depending on the degree of mobility of each individual patient.

20

A. Deaconescu and T. Deaconescu

Fig. 10. Pneumatic actuation diagram of the rehabilitation equipment.

The velocity of the free end of the pneumatic muscle is adjustable in both directions of motion, thus allowing the rehabilitation of joints in various stages of healing. Further, at the right-hand end of muscle stroke (completely relaxed leg), a certain delay of the motion can be adjusted, thus preventing overstraining of the patient.

5 Conclusion The paper presents and discusses a pneumatic muscle actuated rehabilitation device developed for the ankle. The mechanical structure of the equipment is described, as well as the structures of the actuation and positioning systems. The proposed rehabilitation equipment benefits from a cost efficient, simple and robust construction, being easy to use by persons affected by dysfunctions of the ankle. Another major advantage of the proposed system is provided by the utilized source of energy, namely air that completely absorbs all shocks occurring at the ends of the stroke.

References 1. Dimulescu, D.M.: Methodological study of the role of postural therapy in the recovery of patients with affections of the Myo-Arthro-Kinetic and Neuromotor System [in Romanian], PhD Thesis, “Carol Davila” University of Medicine and Pharmacy, Bucharest (2007) 2. Fisiotek 3000 TS. https://www.rimec.it/es/prodotti/fisiotek-3000-ts/. Accessed 21 Nov 2019 3. Kinetec Spectra Knee CPM Machine. https://www.healthproductsforyou.com/p-kinetecspectra-knee-cpm-machine.html. Accessed 21 Nov 2019

Pneumatic Equipment for Ankle Rehabilitation by Continuous Passive Motion

21

4. Deaconescu, T., Deaconescu, A.: Pneumatic muscle-actuated adjustable compliant gripper system for assembly operations. Strojniški vestnik J. Mech. Eng. 63(4), 225–234 (2017) 5. Van Ham, R., Sugar, T.G., Vanderborght, B., Hollander, K.W., Lefeber, D.: Compliant actuator designs. IEEE Robot. Autom. Mag. 16, 81–94 (2009) 6. Versluys, R., Vanderborght, B., Lefeber, D., Naudet, J.: IPPAM: Intelligent Prosthesis actuated by pleated Pneumatic Artificial Muscles. In: Proceedings of the 9th International Conference on Climbing and Walking Robots, CLAWAR 2006, Belgium, pp. 1–9 (2006) 7. Wilkening, A., Baiden, D., Ivlev, O.: Assistive control of motion therapy devices based on pneumatic soft-actuators with rotary elastic chambers In: Proceedings of IEEE International Conference on Rehabilitation Robotics, ICORR, Zurich, pp. 1–6 (2011)

Simulation Platform for Crane Visibility Safety Assistance Tanittha Sutjaritvorakul(B) , Axel Vierling , Jakub Pawlak, and Karsten Berns Technische Universit¨ at Kaiserslautern, 67663 Kaiserslautern, Germany {tanittha,vierling,j pawlak14,berns}@cs.uni-kl.de

Abstract. Visibility assistance can increase the situational awareness of the operator. Person detection using deep learning methods shows promising outcomes in consequence of having a great amount of sample data. In the construction domain, the number of public datasets remains low. We propose the simulation system for improving person detection from the load view using deep learning. The dataset generated from the simulation system can be used to magnify the data. It can be promptly generated with a speed of half a second. Two generated datasets are evaluated by the state-of-the-art detector with fair accuracy. Keywords: Crane safety construction

1

· Visibility assistance · Autonomous

Introduction

The visibility assistance system is essential to support operator visibility. According to visibility-related fatalities, struck-by accidents contribute to 87.7% of all construction equipment accidents [1]. The crane operator cannot notice the load or the nearby workers clearly while hoisting. Simulation is introduced to the construction industry in order to increase safety. One example of simulation usage is data gathering for load-view object detection using deep learning. We first provide a literature review of visibility assistance regarding load-view detection and later how the simulation is used for data generation in deep learning. The crane load view provides a top view from the camera mouted at the boom end. This camera provides an object viewpoint, invisible from the cabin especially in blind lift situations. Several studies present the vision-based solutions for operator’s visibility such as load sway monitoring [2], hazard zone access alert [3], worker tracking [4] and automatic zoom load-view camera based on crane working zone [5]. It is worth to mention the literature of person detection from unmanned aerial vehicles (UAVs) because the aerial view and crane load view are alike. For deep learning object detection, Zhu et al. [6] improve Faster R-CNN to detect small-size person. Barekatain et al. [7] propose an action detection model to detect person using Single Shot MultiBox Detector (SSD). c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 22–29, 2020. https://doi.org/10.1007/978-3-030-48989-2_3

Simulation Platform for Crane Visibility Safety Assistance

23

The authors train the detector with different daily actions e.g. lying, walking, pushing, and calling. However, the worker activities in the construction site are quite different than the actions of a pedestrian. In deep learning object detection, there is a big data shortage in the construction domain. Mentioned learning methods require a lot of sample data to yield good results. There are a great amount of self-driving car datasets publicly available [8]. Unfortunately, these datasets are not applicable to load-view object detection due to the frontal viewpoint. UAV dataset could be possibly used as an alternative for the load view from a crane. In contrast to the construction area, the background of UAV datasets such as VCI [9] and Stanford Drone [10] are uncluttered and not dynamically changed over time. Okutama dataset [7] contains one object class, person and is mainly used for action detection. Although the action is pedestrian-based, it has some common activities with the construction worker e.g. carrying, pushing/pulling wheelbarrow and lying. Nevertheless, the data remains insufficient for data-hungry learning methods. Simulation can help to augment data while reducing localization error and time from the manual labeling. Soltani et al. propose an automated annotation using the synthetic image is able to reduce the annotating time while improving the detection accuracy [11]. Vierling et al. exploit ray tracing techniques of a game engine to extract annotated bounding boxes [12]. Eventually, not all simulated data always look identical to the real world. People dedicated themselves to travel over the world and capture the texture assets from the entire real environment and bring them into the simulated world. This work does not aim in producing perfect simulation and detector but merely demonstrates the feasibility of the generated synthetic data for loadview worker detection. Our proposed system consists of two following steps. First, we develop a simulation as testing platform for a construction machine which resembles our real test environment. Second, we initially investigate the flexibility between synthetic and real datasets for object detection improvement. We made use of our simulation platform to gather the data for worker detection using deep learning in a fast manner. We later evaluate the synthetic data on the state-of-the-art detector which is trained using only the real dataset. By feeding the synthetic data over the detector, we can check how much the detector can recognize then the synthetic data can be verified for detection improvement.

2

Proposed System

Our simulated system is developed in Unreal Engine1 . The experimental setup in this work is based on project safeguARd [13]. The aim of the project is to early detect struck-by load hazard from the mobile crane. Industrial zoom camera is used to detect workers from the load view. The testing vehicle is a telescopic crane. The location took place in Trier, Germany.

1

A game engine developed by Epic Games (www.unrealengine.com).

24

T. Sutjaritvorakul et al.

(a) Real Trier

(b) Virtual Trier

Fig. 1. Comparison of real and simulation view.

Simulation Environment: We obtain the alike experimental environment in a simulation, see Fig. 1. The 3D model landscape can be achieved from OpenStreetMap (OSM). The illumination intensity can be adjusted naturally based on the weather change feature, see Fig. 2. The load-view zoom camera pointed down to the ground from the boom end allows us to record and observe objects in the same view as from the real one. The virtual crane can operate in basic manner as a standard crane. The virtual worker behavior can be simulated as in the real scenario e.g. activities, appearance, and interaction. The different postures that can be seen in the construction site such as working with device and bending, see the second row of Fig. 4.

(a) Rain

(b) Day

(c) Day-Night Cycle

Fig. 2. Sample synthetic data in different light conditions.

Data Collection: The data collection user interface in Fig. 3 allows us to examine the current state of the crane movement and the risk of accidents in the workplace. In order to generate clean data (i.e. not having purely background image), the interface visualizes both crane broad view and the view from the overhead camera while recording. In the end, the final standard annotation output can be abruptly accomplished after the data record. The annotation estimation can be found in [12]. The bounding boxes visualize in the last row of Fig. 4 are accurately fit to the object even with occlusion, see Fig. 4e. Our system recorded and annotated the data with a speed of 0.5 s per frame with no limitation of object

Simulation Platform for Crane Visibility Safety Assistance

25

Fig. 3. Overview simulation interface of collecting data using a load-view camera.

number or class. On the contrary, the one class annotation, which is done by an experienced person, took approximately 14–20 s per frame. Each frame has three objects on average.

3

Experiments

In this section, we evaluate our simulated data on state-of-the-art person detector using deep learning. The objective of the experiment is to initially investigate if the synthetic data is applicable for the detector which is used with real world data. For this reason, the synthetic data can be later on used to support data augmentation. Besides the synthetic data, we additionally provide the evaluation of the real data collected from the real crane and similar crane setup. The hardware used in experiments is Intel(R) Xeon(R) Gold 6126 CPU, 2.60GHz, 48 cores, 188G memory. The following subsections shortly describe datasets that are used in experiments, the choice of person detector and then results. 3.1

Dataset Summary

Table 1 summarizes all datasets, which are used in Sect. 3.2. The sequence names with prefix U are generated from Unreal Engine while sequences with prefix R are collected data from the real world. The distance of the camera to the ground (Dcam ) ranges from 20 to 26 m, which is relatively equivalent to a 4- to 6-floor building. Snapshot example of each sequence can be found in Fig. 5. Dataset U are taken from the virtual crane with 30◦ boom pitch angle and R1 was taken during winter from the real crane. Lastly, R2 mimics the crane load view by collecting from the 6th floor of a building.

26

T. Sutjaritvorakul et al.

(a) Bending

(b) Carrying object

(c) Working with device

(d) Bending

(e) Carrying object

(f) Working with device

Fig. 4. Sample annotation output. The first row is segmented images and the second row is RGB images visualized the bounding boxes of each object from generated annotation file.

3.2

Person Detection

According to the requirement of visibility assistance, the operator should be warned about the nearby object in order to recognize the accident risk in (near) real-time. We choose SSD for our experiments. SSD is a one-stage detector and introduced to address the problem of multi-scales. The SSD architecture model is based on VGGNet [14]. Unlike the two-stage detectors such as Faster R-CNN [15], SSD is relatively fast, but less accuracy [16]. We directly adopt the pre-trained model from [7]. The authors [7] retrain the model with Okutama dataset, pedestrian action from aerial view. The average precision (AP) of the model, which the authors originally evaluate on the Okutama dataset is 72.3%. 3.3

Results and Analysis

We use the detection evaluation benchmark from [17] with Intersection over Union (IoU) threshold of 0.5. Figure 5 shows selected frames from the detector. Table 1 reports AP of each sequence. For the synthetic dataset, the detector results in fair performance. The detector is able to achieve AP of 49.03% and 39.24% for sequence U. As mentioned earlier in Sect. 1, Okutama dataset which is used in the pre-trained model has some common activities to the worker. The detector, therefore, produces a correlative result with the load view. On the other hand, we are aware that training purely synthetic dataset would be insufficient because the training model could be biased to the synthetic data. For further supplemental analysis of the real dataset, the detector achieves good

Simulation Platform for Crane Visibility Safety Assistance

(a) U1

(b) U2

(c) R1

(d) R2

27

Fig. 5. Sample of detected results. The green bounding box is the detected target with class and confidence label.

result in sequence R1 with 75.23% AP. In sequence R2, the detector suffers from shadow of the object which is strongly akin to a person dimension itself, see Fig. 5d. It is a challenging sequence because it is very sunny. Regarding the small angle between the sun and the targets, this results in a short shadow. The detector considers the body and its shadow as one single object. Consequently, this creates a lot of false positives. Despite the fact that this SSD detector precisely detects a person with the bigger bounding box, this causes IoU to become extremely low and eventually lower AP. The difference of inference time between sequence U and sequence R is derived from the larger image size in sequence U whose takes longer to process. Table 1. Dataset summary and detection results. Seq name Frames Resolution

Dcam (m) Average object instances per frame

[email protected] (%) Average inference time (ms per frame)

U1

209

1600 × 1200 24

11

49.03

217.24

U2

403

1600 × 1200 20

4

39.24

217.33

R1

713

720 × 480

25

3

75.23

206.52

R2

400

800 × 600

26

4

12.0

208.08

28

4

T. Sutjaritvorakul et al.

Conclusion

In this paper, we propose the simulation system for developing the crane operator visibility assistance. A similar construction environment to the real test location is built up in simulation. The load view allows us to observe the distance between workers and the crane hook. Moreover, the annotation user interface enables both load view and overall simultaneously. With our proposed system, the data recording and annotation can be done in a rapid manner and more accurate compared to a human. Finally, we do the preliminary experiments using SSD based detector to recognize workers from the load view camera. Due to the lack of public datasets and difficulties in data collection on the crane, the synthetic dataset can complement the real world dataset in order to improve worker detection from load view camera. As future work, we would like to further use this simulation system to generate more data from different environments and altitudes including the zoom level using the proposed system. The camera noise effect can add to the synthetic data such as interlacing in order to make it look similar to the industrial camera. A short shadow can be generated in order to let the network learn as a negative sample. After data collection, we will input these data and fine-tune the adopted pre-trained network model [7] with both synthetic and real datasets and later do an evaluation. Acknowledgement. This work is funded from the Federal Ministry of Education and Research (BMBF) under grant agreement number 01|16SV7738 and name safeguARd.

References 1. Hinze, J.W., Teizer, J.: Visibility-related fatalities related to construction equipment. Saf. Sci. 49(5), 709–718 (2011) 2. Fang, Y., Chen, J., Cho, Y.K., Kim, K., Zhang, S., Perez, E.: Vision-based load sway monitoring to improve crane safety in blind lifts. J. Struct. Integr. Maint. 3(4), 233–242 (2018) 3. Yang, Z., Yuan, Y., Zhang, M., Zhao, X., Zhang, Y., Tian, B.: Safety distance identification for crane drivers based on mask R-CNN. Sensors 19(12), 2789 (2019) 4. Neuhausen, M., Teizer, J., K¨ onig, M.: Construction worker detection and tracking in bird’s-eye view camera images. In: Proceedings of the 35th ISARC, Berlin, Germany (2018) 5. Vierling, A., Sutjaritvorakul, T., Berns, K.: Crane safety system with monocular and controlled zoom cameras. In: Proceedings of the International Symposium on Automation and Robotics in Construction (ISARC), vol. 35, pp. 1–7. IAARC Publications (2018) 6. Zhu, H., Qi, Y., Shi, H., Li, N., Zhou, H.: Human detection under UAV: an improved faster R-CNN approach. In: 2018 5th International Conference on Systems and Informatics (ICSAI), pp. 367–372. IEEE (2018)

Simulation Platform for Crane Visibility Safety Assistance

29

7. Barekatain, M., Mart´ı, M., Shih, H.-F., Murray, S., Nakayama, K., Matsuo, Y., Prendinger, H.: Okutama-action: an aerial view video dataset for concurrent human action detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 28–35 (2017) ´ 8. Laflamme, C.-E.N., Pomerleau, F., Gigu`ere, P.: Driving datasets literature review. arXiv preprint arXiv:1910.11968 (2019) ¨ uner, U.: ¨ Top-view trajectories: a pedestrian 9. Yang, D., Li, L., Redmill, K., Ozg¨ dataset of vehicle-crowd interaction from controlled experiments and crowded campus. arXiv preprint arXiv:1902.00487 (2019) 10. Robicquet, A., Sadeghian, A., Alahi, A., Savarese, S.: Learning social etiquette: human trajectory understanding in crowded scenes. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) European conference on computer vision (ECCV 2016). LNCS, vol. 9912, pp. 549–565. Springer (2016) 11. Soltani, M.M., Zhu, Z., Hammad, A.: Automated annotation for visual recognition of construction resources using synthetic images. Autom. Constr. 62, 14–23 (2016) 12. Vierling, A., Sutjaritvorakul, T., Berns, K.: Dataset generation using a simulated world. In: Berns K., G¨ orges D. (eds.) Advances in Service and Industrial Robotics, RAAD 2019. Advances in Intelligent Systems and Computing, vol. 980, pp. 505– 513. Springer, Cham (2019) 13. Bremer Institut f¨ ur Produktion und Logistik (BIBA): safeguARd, Augmented Reality-based assistance system for commercial vehicles to raise the safety level, no. D040011, November 2015 14. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556 (2014) 15. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28, Curran Associates, Inc., pp. 91–99 (2015). http://papers.nips.cc/paper/5638-faster-r-cnn-towards-real-time-objectdetection-with-region-proposal-networks.pdf 16. Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., Murphy, K.: Speed/accuracy trade-offs for modern convolutional object detectors. CoRR, vol. abs/1611.10012 (2016). http:// arxiv.org/abs/1611.10012 17. Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results (2007). http:// www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html

Design of Robotic Braces for Patients with Scoliosis Rahul Ray1,2(&), Laurence Nouaille1, Briac Colobert2, and Gérard Poisson1 1

PRISME Laboratory, University of Orléans, INSA CVL, Orléans, France [email protected] 2 Proteor, Dijon, France

Abstract. This paper presents a new robotic device, designed for the treatment of idiopathic scoliosis (IS). Scoliosis is a complex 3D spine deformity. Hard braces are proved to be effective for its treatment but still have more shortcomings which needed to be overcome. Braces cannot realize specific control over vertebra and they also limit the daily work routine. They can cause pain, skin breakdown and bone deformations. The work performance of brace does not intend as per the users need. To solve these problems, we have designed a new robotic brace exerting “three-point pressure” with considerations of human biomechanics propertie. The robot is described from mechanical to control point of view. The robotic brace, based on a double Stewart-platform, has been designed, produced by rapid prototyping and then equipped with 12 linear actuators. Position control and force control approaches are nowadays available and implemented on an electronics/informatics device. This robotics brace has been used on a healthy person and then validated. The prototype version allows adjust dynamical force applied on human body to rectify the scoliosis. Keywords: Scoliosis


3D spine torsion
Robotic brace

1 Introduction The development of scoliosis is a complex and dynamic process on sagittal, coronal and transverse planes. Idiopathic Scoliosis (IS) is [1, 2] the most common form with unknown etiology, accounting for about 80% of all scoliosis [3, 4]. The harmful impact of scoliosis on the body is more than just spinal curvature. Current studies reveal that scoliosis deteriorates other systems of human body, such as digestive, nervous and skeletal systems. Scoliosis deteriorates not only the spine but also ribs and pelvis. Moreover, it results in some other boring problems, including reduced the quality of daily life, disability, pain, functional limitations and possible progression during adulthood. Scoliosis can be informed in infancy or early childhood but Adolescent Idiopathic Scoliosis is the most common and affects 2 to 3% of people. The probability of the progression for this pathology in females is 8 times higher than that of the male, which need to be treated. To implement 3D brace for treatment of spine deformity stiffness of the human, torso must be realized. Force and moment exerted by the brace on the body calculation plays major role for the brace to be more precise and effective © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 30–37, 2020. https://doi.org/10.1007/978-3-030-48989-2_4

Design of Robotic Braces for Patients with Scoliosis

31

for the treatment of the spine deformity. Motion of the actuators with the body motion to measure the force and torque should be realized with real-time implementation to obtain complex movement with jointed appendages [5]. About 0.5% of population need scoliosis treatment [6]. There are three main methods for the treatment of scoliosis: observation, bracing and surgery. Observation and bracing are the two nonoperative treatments for scoliosis, which can greatly reduce the costs of its therapy. However, a number of researches have shown that brace treatments are more effective than observation. The development of new braces fit under the arm improves the effect of the treatment and more particularly the patient’s quality of daily life [7]. In [8, 9] authors reported that brace treatments decrease the progression of high-risk curves to threshold for surgery in patients with adolescent idiopathic scoliosis. In their investigations, successful treatments are 72% with brace and 48% with simply observation over time. Consequently, investigate and propose an effective brace is very important and necessary for all patients with scoliosis. Furthermore, exploiting the interests of robotics is also an opportunity that can improve service to patients.

2 Robotic Brace Design In literature review, few robotic devices have been designed for the treatment of scoliosis: Atlas commercialized by Japet Medical Devices society and RoSE developed in Columbia University [10] are the best known robots. Our active robotic spinal brace, for patients with scoliosis, is designed based on the 6 degrees-of-freedom (6-DoFs) “three-point pressure” treatment (Fig. 1.c) [11]. A double Stewart-platform composed of upper, middle and low corrective rings (Fig. 1 b) is the kinematic structure of the developed robot. Each ring is fixed to another by buckles. The three rings are elliptical structures which are similar to human trunk shape. The two adjacent rings are driven by two sets of six electric linear actuators and the robot offers 6-DoFs corrective forces to realize the 3D dynamic correction for the scoliosis treatment.

(a)

(b)

(c)

Strap Linear actuator Brushless motor Polyamide 12 Position sensor Multiaxis force sensor

Fig. 1. The 3D printed brace prototype: (a) Front view (b) Back view (c) The 3 Pressure-points

32

R. Ray et al.

A potentiometer and a force sensor (Fig. 1 a) for measuring the position and force have been added on each actuator [12]. The two Stewart-platforms are connected in series and can be controlled in position or force mode independently. Besides, the robotic spinal brace is equipped with a 6D force sensor to detect the corrective forces applied in real time on the patients spine. Therefore, the brace system can adjust the rehabilitation scheme along with the rehabilitation status of patients by controlling the proposed robotic spinal brace flexibly. For the convenience of the patient to wear, each corrective ring was divided into two symmetrical parts which were armpit and the upper ring has two grooves matched with patients’ arm. The middle corrective ring fixed on the waist applies corrective forces on patients’ spine. Besides, the ring can adjust position along the trunk through the elongation and shortening of actuators to adapt to different stress regions according to different scoliosis curves. The lower corrective ring fixed on the crotch keeps immoveable when exerting corrective forces. Besides, actuators can elongate and shorten in response to motions of the wearer which ensure that the three corrective rings don’t prevent the wearer from moving Thus, the comfort and convenience of this brace are greatly improved without losing treatment effect. In order to control the prototype, the next section presents its kinematics model.

3 Robotic Modelling The architecture of each parallel platform follows the kinematic structure of the Stewart- platform where all the limbs share identical kinematic chain from Fig. 2.

Fig. 2. Geometry and parameterization of Stewart platform: Agrawal et al. 2018 [15]

Design of Robotic Braces for Patients with Scoliosis

33

The parameterization will be presented for only the lower parallel platform, which can also be extended to the top one. Each limb connecting the fixed base denoted A to the moving platform denoted B forms a kinematic loop which can be expressed in the vector form as: qi ¼ pB  ai + RB bi ; for i ¼ 1; . . .:; 6

ð1Þ

with pB = [PxB, PyB, PzB]T a position vector and RB a rotation matrix formed by the 3 angles (wB, hB, uB), and where qi is a vector with magnitude qi along a unit vector along the leg. q ¼ ½q1 ; q2 . . .:q6 T is the actuated joint coordinates vector and XB = [PxB, PyB, PzB, wB, hB, uB]T the moving platform motion variables vector. The rotation of the moving platform around the base one is defined by pitch-rollyaw angles (w, h, u) around x, y, z axes in Fig. 2 [13–15]. For inverse kinematic analysis, the moving platform position pB and orientation RB are given and the platform is to solve for the joint variables, here: L = [l1, l2, l3, l4, l5, l6]T. The length of each limb li (Fig. 2) can be expressed as a norm of vector qi l2i ¼ qTi qi ¼ ½pB  ai + RB bi T ½pB  ai + RB bi 

ð2Þ

Hence, each limb length can be uniquely determined for given position and orientation of the moving platform. For forward kinematic analysis, the joint variables li are given and the problem is to solve for XB of the moving platform. In this model, an iterative numerical solver is used with screw axis representation of the rotation matrix. XB is redefined with screw coordinates as [16]. XB = [PxB, PyB, PzB, sx, sy, sz, ϴ]T in which sx, sy, sz and ϴ are obtained from rotation matrix and these equations are to be solved for forward kinematics. Ei ¼ l2i + qTi qi ¼ 0

ð3Þ

P The nonlinear least-square optimization was used to minimize (½ Ei). The Jacobian matrices are computed from velocity loop closures which are directly obtained by differentiating as follows: p:B ¼ q:i  xB x RB bi

for i ¼ 1; . . .6

ð4Þ

in which xB, the angular velocity of platform B relative to platform A, is calculated on the static wrench of the moving platform. Dynamics of the system has been determined using Lagrange’s method applied to the limb and the platform. The corresponding developments are not detailed in this paper.

34

R. Ray et al.

4 Controller Two control modes were implemented to articulate the system: motion and force control modes. Each limb has built in sensors to record its joint displacement or force applied by the joint actuators. This information is used by the closed loop controller in both modes. The upper and lower parallel platforms have their own motion and force controller which can be operated independently. The control architecture scheme for motion control is shown in Fig. 3 left. Since direct measurement of motion of the platform requires additional Cartesian motion sensors, the controller was developed in joint space using active joint position feedback. The Xd desired motion of the platform is mapped into the joint space variable qd using inverse kinematics. The error is measured as the difference between the desired joint motion and the actual one. A PID controller generates the joint torques necessary to derive each actuator. The control architecture for force control is shown in Fig. 3 right. It was implemented in joint space and the Jacobian is used to compute the desired joint torque based on desired output force/moment of the platform which was real time computed from limb position feedback and forward kinematics. A quasi-static control approach was implemented as the bandwidths of both motion and force control are fairly low.

Fig. 3. A PID controller implemented in joint space for motion control and force control

5 Simulation and Evaluation In order to evaluate the force performance of the robotic brace prototype, some experiments have been performed with it. The classical PID control strategy is adopted in force control mode. Twelve Autonix® miniature linear actuators, an ATI 6-axis force/torque sensor, a NI-sbRIO real-time controller and a computer. This computer provides the force and the position reference signals and the force and motion controller are running in sbRIO hardware. The sampling time for the experimental system is set to 20 ms. In a first time, desired constant force signals are exerted on the robotic brace to evaluate performance of the corrective system. These signals are along six directions (30 N for surge, sway and heave; 1.5 Nm for roll and pitch and 2 Nm for yaw), responses to the reference constant force signals. So, the robotic brace generates six orientations corrective forces to treat IS and track the desired corrective force signals. The brace is an active one and the size and direction of corrective forces can be controlled easily based on the type and degree of scoliosis. In order to further evaluate the brace performance, desired sine force signals will be applied on the brace in six

Design of Robotic Braces for Patients with Scoliosis

35

directions surge (12 N/0.03 Hz), sway (10 N/0.04 Hz), heave (15 N/0.05 Hz), roll (2 Nm/0.08 Hz), pitch (2 Nm/0.1 Hz), yaw (2 Nm/0.08 Hz). After these possible tests for the controller we have chosen controls that can drive the model to best for its assistance and compatible with the system. Various parameters were measured in instant of time, force, torque, motion, trajectory tracking and all movements and rotations. On another hand, the brace was experimented on a healthy human being while he was in movement (Fig. 4). The range of motion was to determine the motion which was allowed by the brace transparent mode: the “zero desired force”. Different movements will be performed depending upon the scoliosis type of the patient in a future use. Some of the place to be checked for scoliosis on interior superior iliac spines, sacrum, sternoclavicular joints and the inferior border of the manubrium, where lateral bending (Fig. 4 a), flexion/extension and transverse (Fig. 4 b) with vertical rotation is done with response of trajectory.

Fig. 4. The robot on a human: (a) Lateral bending (b) Flexion extension

6 Limitation and Future Work The current prototype is not as low profile as currently used braces made out of sheets of thermomolded plastics. This was unavoidable in the first iteration where off the shelf commercial motors were used. It is however much lighter and more comfortable than the devices mentioned in the literature. Compliance with bracing regiments continues to be a hurdle as many individuals do not comply with the prescribed 12 h of wear per day. While a bulkier brace would seem to be more visible and therefore less likely to be adopted, it is hoped that through appropriate control strategies the time the brace needs to be worn can be reduced without negatively impacting outcomes. If wearer can receive the same quality of care while only wearing the brace for a majority of the day, then compliance may increase and individuals may be more likely to benefits from bracing, as it has been shown by Hasler that flexible bracing does not improve compliance if still requiring long duration wear [17].

36

R. Ray et al.

The dynamic present brace will hopefully pave the way for future dynamic braces in this area. There are a number of improvements that can be made to this brace in the future. One of the next steps will be to move to sheet formed polyethylene in-line with current fabrication techniques [18]. With this, the control board and other electronics should be made compact so that the system becomes wearable and transparent for the patient. Further testing needs to be done to evaluate the effectiveness of the proposed control strategies compared to traditional bracing techniques. The brace should also be utilized to do pilot studies for spinal correction as a part of future evaluations.

7 Conclusion Scoliosis is usually treated by three ways: observation, bracing or surgery. Braces are efficient but their wearing during several hours cause pains to the patient. For this reason, it was interesting to design a robotic spine brace which can follow the patient’s movements. For improving the treatment effect for scoliosis, the new designed robot, presented in this paper, is based on three-point pressure treatment. This robotic brace offers active dynamic adjustable corrective forces. Moreover, it can make movements in slant and axial rotation for improving the effective force to be applied on the patient’s body. A first prototype based on two serial Stewart-platforms was produced. Several experiments carried out have validated the kinematic architecture of the prototype and its control strategies. The prototype will be improved by choosing other materials and components to provide compliance to do pilot studies for spinal corrections [18, 19]. Further, testing needs will be done to evaluate the effectiveness of the proposed control strategies compared to traditional bracing techniques. Acknowledgements. The work described in this paper was supported by Proteor Company, Dijon, France.

References 1. Fayssoux, R., Cho, R., Herman, M.: A history of bracing for idiopathic scoliosis in North America. Clin. Orthop. Relat. Res. 468(3), 654–664 (2010) 2. Weinstein, S.L., Dolan, L.A., Wright, J.G., Dobbs, M.B.: Effects of bracing in adolescents with idiopathic scoliosis. N. Engl. J. Med. 369(16), 1512–1521 (2013). PMID: 24047455 3. Wynne, J.H.: The Boston brace system philosophy, biomechanics, design & fit. Stud. Health Technol. Inform. 135, 370–384 (2007) 4. Rigo, M., Weiss, H.: The Chêneau concept of bracing-biomechanical aspects. Stud. Health Technol. Inform. 135, 303 (2008) 5. Blount, W.P.: The milwaukee brace in the treatment of the young child with scoliosis. Arch. Orthop. Trauma Surg. 56(4), 363–369 (1964) 6. Price, C.T., Scott, D.S., Reed Jr., F.E., Riddick, M.: Night-time bracing for adolescent idiopathic scoliosis with the Charleston bending brace: preliminary report. Spine 15(12), 1294–1299 (1990)

Design of Robotic Braces for Patients with Scoliosis

37

7. Coillard, C., Vachon, V., Circo, A.B., Beauséjour, M., Rivard, C.H.: Effectiveness of the SpineCor brace based on the new standardized criteria proposed by the scoliosis research society for adolescent idiopathic scoliosis. J. Pediatr. Orthop. 27(4), 375–379 (2007) 8. Wong, M.S., Cheng, J.C., Lam, T.P., Ng, B.K., Sin, S.W., Lee-Shum, S.L., Chow, D.H., Tam, S.Y.: The effect of rigid versus flexible spinal orthosis on the clinical efficacy and acceptance of the patients with adolescent idiopathic scoliosis. Spine 33(12), 1360–1365 (2008) 9. Mac-Thiong, J.-M., Petit, Y., Aubin, C.-E., Delorme, S., Dansereau, J., Labelle, H.: Biomechanical evaluation of the boston brace system for the treatment of adolescent idiopathic scoliosis: relationship between strap tension and brace interface forces. Spine 29(1), 26–32 (2004) 10. Ophaswonge, C., Murray, R.C., Agrawal, S.K.: Design of a parallel architecture robotic spine exoskeleton. https://doi.org/10.1115/DETC2017-67842 11. Chang, C.L., Kelly, D.M., Sawyer, J.R., Diangelo, D.: Mechanical testing of a novel fastening device to improve scoliosis bracing biomechanics for treating adolescent idiopathic scoliosis. Appl. Bionics Biomech. 18 (2018) 12. Lou, E., Venkateswaran, S., Hill, D.L., Raso, J.V., Donauer, A.: An intelligent active brace system for the treatment of scoliosis. IEEE Trans. Instrum. Meas. 53(4), 1146–1151 (2004) 13. Park, J.-H., Stegall, P., Agrawal, S.K.: Dynamic brace for correction of abnormal posture of human spine (2015). 978-1-4799-41 14. Park, J.-H., Stegall, P., Agrawal, S.K.: Wrench capability of Stewart platform with series elastic actuators. J. Mech. Robot. 10/021002-1 (2018) 15. Park, J.-H., Stegall, P., Agrawal, S.K., Roye Jr., D.P.: Robotic spine exoskeleton, pp. 1534– 4320 (2018) 16. Innocenti, C.: A novel numerical approach to the closure of the 6-6 stewart platform mechanism. In: Proceedings of the Fifth International Conference on Advanced Robotics, pp. 852–855 (1991) 17. Hasler, C.C., Wietlisbach, S., Buchler, P.: Objective compliance of adolescent girls with idiopathic scoliosis in a dynamic SpineCor brace. J. Child. Orthop. 4(3), 211–218 (2010) 18. Cafolla, D., Ceccarelli, M., Wang, M.F., Carbone, G.: 3D printing for feasibility check of mechanism design. Int. J. Mech. Control. 17(1), 3–12 (2016) 19. Cafolla, D., Ceccarelli, M.: An experimental validation of a novel humanoid torso. Robot. Auton. Syst. 91(C), 299–313 (2017)

Comparison Between Two Methods of Cables Tensions Optimization of a Cable Driven Parallel Robot Hajer Ben Amor(&), Sami Bennour, and Abdelfattah Mlika Mechanical Laboratory of Sousse (LMS), National Engineering School of Sousse, University of Sousse, 4000 Sousse, Tunisia [email protected], [email protected], [email protected]

Abstract. This manuscript presents a three-DOF four cable driven parallel robot with four cables (CDPR). The robot structure consists of a parallel redundant manipulator actuated by a set of cables. The presented robot presents one degree of actuation redundancy. In this document, the kinematic analysis of such manipulator is illustrated in first. Then, the dynamic model is developed in order to calculate the tensions in robot cables. These tensions must be positive. However, a comparison between two methods of cables tensions optimization is presented and developed in detail. The first method is based on the minimizing the Euclidean norm of the cable tensions while the second one consists in the calculating of the pseudo inverse of the Jacobian matrix. Finally, an example of trajectory is given in order to illustrate a comparison between the two presented methods. Keywords: Cables Optimization


Robot
Tensions
Redundancy
Dynamic


1 Introduction Robots and a lot of kinds of manipulators are frequently used now. Cable-Driven Parallel Robots (CDPR) represent a particular class of parallel robots. In fact, CDPRs are mechanisms composed of a base and a moving-platform connected by a set of cables in a given number of points. CDPRs have attracted the attention of researchers and industrials due to their several advantages compared to the serial robots. Moreover, they are characterized by a large workspace, high dynamics, an important payload capacity, an easy reconfiguration… [1, 4–7]. Thereby, several studies have proposed prototypes for different applications, e.g., the famous Skycam robot [2]. Dynamic modelling of a planar CDPR without considering of elastic effects of cables has been developed [3]. Besides; there are other works considering cable as elastic [8, 10]. Vibration of elastic cables have been studied also [11]. Besides, because cables can only exert tensions (they cannot push), the optimization of the cable tensions is rather challenging. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 38–45, 2020. https://doi.org/10.1007/978-3-030-48989-2_5

Comparison Between Two Methods of Cables Tensions Optimization

39

This paper addresses the developing of the distribution of the tension in the cables of the robot. However, the kinematic and the dynamic models of a planar parallel cable robot are presented in detail. The developed models are used to develop two different methods to maintain cables in tension. To illustrate, a numerical example is addressed and presented in the end of the paper.

2 Kinematic Modelling In this paper, a CDPR is composed of moving plate-form which is connected to motors shafts through pulleys by four cables. As shown in the Fig. 1, the moving platform is defined by the Bi ði ¼ 1; . . .; 4Þ points and a base defined by the Ai ði ¼ 1; . . .; 4Þ points. The reference frames for the moving platform and the base are respectively: RP = (P, x, y) and Rb = (O, x0, y0).

g

Fig. 1. Planar model of the robot

The displacement of the moving platform is established by the variation of the cables lengths Li ði ¼ 1; . . .; 4Þ. The kinematic modeling consists in developing the motors angular position qi ði ¼ 1; . . .; 4Þ as a function of the platform pose. The pose vector X ¼ ½x y aT denotes the position and the orientation of the platform in relation to Rb. On the other hand, the motors angular position can be written as: qi ¼

DLi Li  Li0 ¼ ri ri

ði ¼ 1; . . .; 4Þ

ð1Þ

where Li0 ði ¼ 1; . . .; 4Þ is the initial length for each cable and ri ði ¼ 1; . . .; 4Þ is the pulley radius.

40

H. Ben Amor et al.

Moreover, the cable length vector can be written as follows: li ¼ Li ui ¼ r  Qrbi  rai

ði ¼ 1; . . .; 4Þ

ð2Þ

where ½Ai Bi Rb kAi Bi k r ¼ ½OPRb

ui ¼

ði ¼ 1; . . .; 4Þ

ð3Þ

rbi ¼ ½Bi PRp rai ¼ ½OAi Rb

0

cosðaÞ Q ¼ @ sinðaÞ 0

sinðaÞ cosðaÞ 0

1 0 0A 1

Therefore, the motors angular velocity is given as follows [9]: q_ ¼ R1 J ðX ÞX_

ð4Þ

where is q_ ¼ ½ q_ 1 q_ 2 q_ 3 q_ 4 T the motors angular velocity vector, R1 is a diagonal matrix containing the ri coefficients, X_ is the plate-form angular velocity and J ðXÞ is the Jacobian matrix of the robot. 0

uT1 B J ðX Þ ¼ @ ... uT4

1 ðQrb1  u1 ÞT C .. A . T ðQrb4  u4 Þ

ð5Þ

3 Dynamic Modelling In this paper, we suppose that the cables elasticity can be neglected, and cables are massless. Dynamics modeling is concerned with relating the Cartesian translational motion of the moving platform to the required active joint torques. The dynamic model can be divided into two expressions the first one consists in the dynamic of the platform where the second expression presents the actuator dynamic model. Then the global model is calculated by the coupling of the two expressions through the T vector that presents the cables tensions vector. • Cartesian Dynamics Model The cartesian dynamic model for the robot platform can be written as follow: € þ CX_ ¼ WT þ fg MX

ð6Þ

Comparison Between Two Methods of Cables Tensions Optimization

41

where M denotes the mass matrix. C denotes the Coriolis/centripetal matrix [9]. fg denotes the gravity vector. W ¼ J T . X_ denotes the moving platform velocity vector • Actuator Dynamics Model The Actuator Dynamics Model is given by: Cm ¼ I m €q þ RT

ð7Þ

where Cm denotes the actuator torques vector. q€ is the actuator angular acceleration vector. R denotes the cable pulley matrix radii. Im denotes actuator moments of inertia matrix. 3.1

Tension Distribution

The dynamic modeling is used to maintain positive cable tension for all wrenches. For CDPRs with actuation redundancy, Eq. (6) is under constrained which means that there are infinite solutions to the vector T = [t1 t2 t3 t4]T. In the next sections two different methods of maintaining positive cable tension are developed. • Pseudo inverse method For CDPRs with one degree of actuation redundancy, the positive cable tension method of Robert. L is adapted [3]. The optimal solution, which must be inferior or equal to a specified maximum, of the positive tension is given by the following expression tension tmax: Top ¼ T þ bN

ð8Þ

  € þ CX_  fg T ¼ W þ MX

ð9Þ

Where

W+is the Moore-Penrose pseudo inverse of the Jacobian matrix. N 2 R4 denotes the null space of JT. N can be physically interpreted as internal forces. It means that term does not contribute into motion of the platform and only provides positive tension in the cables [8]. b is an arbitrary scalar. However, for each tension component that is negative, calculate bi ¼ ðtmax  ti Þ=ni ; then select the smallest magnitude of these to be the b.

42

H. Ben Amor et al.

• Euclidean norm optimization Because the degree of freedom of the platform is inferior to the number of cables, there may exist infinitely solutions to the cable tensions. To ensure continuity of cable tensions along a continuous path not passing by singularities of the W matrix, one generally uses the norm p, 1 < p < ∞. Specifically, the norm 2 of T (also called Euclidean norm), it refers to quadratic programming [9], and the problem is given by: minT

rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi! X4 t2 kT k2 ¼ i¼1 i

ð10Þ

€ þ CX_  fg Subject to WT ¼ MX

ð11Þ

and timin  ti  timax

ð12Þ

ði ¼ 1; . . .4Þ

4 Case Study To show the performance of the presented methods to maintain positive cable tension, a simulation study in MATLAB has been performed. The dynamic characteristic of the robot is illustrated in Table 1. The desired trajectory of the moving platform is a circle in 1 s. To produce a smooth motion, we require that the platform velocity and acceleration start and end at zero. Then the desired trajectory can be written as follows: 8   x ¼ 0:15cos12pt5  30pt4 þ 20pt3  > > < y ¼ 0:15 sin 12pt5  30pt4 þ 20pt3 > a¼0 > : 0t1

Table 1. Parameters of the planar cable-robot Actuator moments of inertia Im (kg
m2 ) Cable pulley radii ri (m) Platform moment of inertia (kg
m2 ) Platform mass (kg) a (m) b (m) c (m) timin (N) timax (N)

0.00081 0.026 0.0015 0.91 0.35 0.06 0.05 0 150

ð13Þ

Comparison Between Two Methods of Cables Tensions Optimization

43

Considering Eq. (2) and Eq. (13), the cables tensions expression using the MoorePenrose pseudo inverse can be written as:   €  fg T ¼ W þ MX

ð14Þ

where fg = [0 −mg 0] in which m and g present, respectively, the platform mass and the gravity acceleration constant. Figure 2 shows the simulation results in term of cable tension of the four cables for the circular trajectory using the developed pseudo inverse method. Figure 2(a) illustrates the cables tensions where it can be observed that the tensions have negative values, these results are obtained by applying Eq. (9). Besides, in the Fig. 2(b), tensions are positive and they are calculated using Eq. (8). Based on the desired position of the platform the problem of the Euclidean norm optimization is given by: 8 > >
> :

€  fg Subject to WT ¼ MX and timin  ti  timax ði ¼ 1; . . .; 4Þ

ð15Þ

Figure 3 shows the simulation results in term of cable tensions of the four cables for circular trajectory using the Euclidean norm optimization.

(a) Cables’ tension using the pseudo inverse matrix

(b) Optimal cables’ tension

Fig. 2. Simulation results using the first method

44

H. Ben Amor et al.

Fig. 3. Simulation results using the second method

As shown in Fig. 2b, and Fig. 3 for the simulation with the pseudoinverse method and with the Euclidean norm optimization, cables tensions are positive and they are inferior to the maximum tension for each cable (150 N). We can observe that the cables tensions presented in the Fig. 3 are greater than those presented in the Fig. 2. Also, we can note that the simulation results for the second methods takes one hour to calculate while for the resulting simulation for the first method is obtained in a few of seconds. Then we can conclude that the first method is the satisfying method if we want to control the robot in real time using a strategy of tensions distribution.

5 Conclusion This paper studies a planar parallel cable robot. A generic dynamic model has been introduced. The objective is to compare between two algorithms able to maintain cables in tension (positive tensions). An example of a desired trajectory was given to show the effectiveness of the presented positive cables tensions methods.

References 1. Antoine, M., Stéphane, C., Philippe, C.: Design of a cable-direct-driven robot with grasping device. Procedia 70, 290–295 (2018) 2. Cone, L.: Skycam: an aerial robotic camera system. Byte 10(10), 122 (1985) 3. Williams II, R.L., Gallina, P., Vadia, J.: Planar translational cable-direct-driven robots. J. Robot. Syst. 20, 107–120 (2003) 4. Lamine, H., Bennour, S., Romdhane, L.: Design of cable-driven parallel manipulators for a specific workspace using. Adv. Robot. 30, 585–594 (2016) 5. Lorenzo, P., Paolo, G.: Cable-direct-driven-robot (CDDR) with a 3-link passive serial support. Robot. Comput. Integr. Manuf. 30, 265–276 (2016)

Comparison Between Two Methods of Cables Tensions Optimization

45

6. Saeed, B., Amir, K.: Cable-based robot manipulators with translational degrees of freedom. In: Industrial Robotics. Modelling and Control, pp. 211–236 (2006) 7. Trevisani, A.: Underconstrained planar cable-direct-driven robots: a trajectory planning method ensuring positive and bounded cable tensions. Mechatronics 20, 113–127 (2010) 8. Khosravi, M.A., Taghirad, H.D.: Dynamic analysis and control of fully-constrained cables robots with elastic cables: variable stiffness formulation. Springer, Cham (2015) 9. Lamaury, J.: Contribution à la commande des robots parallèles à câbles à redondance d’actionnement, Université Montpellier II - Sciences et Techniques du Languedoc (2013) 10. Najafi, F., et al.: Development a fuzzy PID controller for a parallel cable robot with flexible cables. In: International Conference on Robotics and Mechatronics (2016) 11. Yuan, H., et al.: Vibration analysis of cable-driven parallel robots based on the dynamic stiffness matrix method. J. Sound Vib. 394, 527–544 (2017)

Computational Method for Muscle Forces Estimation Based on Hill Rheological Model Olfa Jemaa1(&), Sami Bennour1, David Daney2, and Lotfi Romdhane1,3 1

Laboratory of Mechanical of Sousse, University of Sousse, Bp. 264 Erriadh, 4023 Sousse, Tunisia [email protected] 2 National Institute for Research in Computer Science and Automation, 33405 Bordeaux, France 3 College of Engineering, American University of Sharjah, Sharjah, UAE

Abstract. The aim of this paper is to propose a computationally efficient method at combining a direct dynamics approach and musculoskeletal system in order to generate muscle forces. The estimation method is essentially decomposed into three main parts. The first part is the development of a biomechanical model of upper limb allows to determine the musculotendon lengths. The second part is the processing of electromyography signals (EMG). The last part consists of estimating the musculotendon forces based on a Hill rheological model allowing to represent the elastic behavior of muscles. This study selects the motion of elbow flexion as the research object. The obtained results have confirmed the feasibility of forward approach for estimating muscle forces during dynamic contraction. They have shown a good overall correlation between the estimated muscle forces and the measured EMG data. These estimated muscle forces can be exploited in future experimental work as effective information to design and control exoskeletons. Keywords: Musculoskeletal model
Dynamics activation contraction
Hill rheological
Muscle forces


Dynamics

1 Introduction Muscle forces quantification is one of the major topics in various fields such as rehabilitation [1], ergonomics [2], design and control of exoskeletons [3]. For instance, several studies have shown that the intelligent control of the exoskeleton robot and the active assistant training according to human intent are realized based on the predicted muscle forces. However, direct measurements of muscle forces are generally not feasible; therefore, several indirect methods have been developed. Firstly, muscle forces can be predicted in vitro using inverse dynamics approach [4]. Inverse dynamics is an indirect method that utilized known kinematic data and external forces to calculate the resultant muscle forces. A major limitation of inverse dynamics is the muscular redundancy problem [5]. Indeed, it is usually difficult to determine the forces of the individual muscles that result from an equilibrium equation, due to high number of © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 46–53, 2020. https://doi.org/10.1007/978-3-030-48989-2_6

Computational Method for Muscle Forces Estimation

47

unknown muscle forces compared to the obtained equilibrium equations. To address the redundancy, a method called forward dynamics is proposed [6]. The forward approach uses biological signals (EMG) as the main inputs in order to determine which muscles were used during a specific movement. Forward approach is widely used in the literature to predict muscle forces for the shoulder [7], the elbow [8], and the wrist [9]. The main goal of this paper is to propose a computationally method for calculating muscle forces of the upper limb based on the forward dynamics approach. The paper is made of three main parts. In the first section, musculotendon lengths of Biceps and Triceps muscles crossing the elbow joint were determined from joint angles using a three-dimensional anatomical model of the upper limb. Moreover, EMG signals are filtered and transformed to muscle activation, which is presented in the second section. Finally, the last section proposes to the transformation of the muscle activations and musculotendon lengths into musculotendon forces using Hill rheological model. This computational method is shown in (Fig. 1).

Fig. 1. Process estimating muscle forces using EMG signals and joint angles

2 Models and Methods 2.1

Musculoskeletal Model

The musculoskeletal model of upper limb, used in the present study, is a threedimensional model developed on OpenSim platform [10]. The model is composed of the following skeletal elements: thorax, sternum, scapula, clavicle, humerus, radius, ulna, and wrist. It has 7 degrees of freedom representing: the shoulder elevation angle, shoulder plane of elevation, shoulder internal-external rotation, elbow flexionextension, forearm pronation-supination, wrist flexion-extension and wrist radialulnar deviation. Ranges of motion for each joint angle are determined from the OpenSim. The biomechanical model includes also 30 musculotendon actuators. The musculoskeletal model is shown in (Fig. 2).

48

O. Jemaa et al.

Fig. 2. (A) Musculoskeletal model of upper extremity (B) Schematic of musculotendon actuator

Based on this biomechanical model, we have expressed the length of each musculotendon actuator LMT , which is represented by a polynomial equation according to the joint angles “Eq. (1)”: LMT ðQ1 ; . . .; Qk Þ ¼ c þ a1 f1 ðQ1 ; . . .; Qk Þ þ a2 f2 ðQ1 ; . . .; Qk Þ þ . . . þ an fn ðQ1 ; . . .; Qk Þ ð1Þ

Where ðQ1 ; . . .; Qk Þ are the joint angles and ðf1 ; . . .; fn Þ are non-linear generic functions. To determine the coefficients ða1 ; . . .; an Þ of the polynomial equation, least squares fit was used. 2.2

EMG Process and Muscle Activation

After the EMG signal recording, several filters are applied in order to filter, rectify and normalize the signal. The initial processing of the raw signals consists of removing artifacts by band-pass filter. The last processing is used to form a linear envelope using a low-pass filter. Next, EMG signals are normalized by dividing them by the EMG values obtained during a maximal voluntary contraction. The next step is the dynamics activation, which transforms muscular excitation into muscle activation. Physiologically, there is a delay between the time of muscular excitation and that of the muscular activation. This electromechanical delay must be included in the differential equations that describes dynamics activation [12]. The first relationship, which relates muscle excitation to neural activation, was modeled by a first-order differential equation “Eq. (2)”. A nonlinear relationship between neural activation and muscle activation occurs when forces were below 40% maximum voluntary contraction MVC, whereas the relationship becomes linear at levels above 40% MVC “Eq. (3)”.
 duðtÞ sacti þ sdesc 1 ¼ ðeðtÞ  uðtÞÞ eðtÞ þ dt sdesc sacti :sdesc ( aðtÞ ¼ a lnðbuðtÞ þ 1Þ 0  u  u0 u0  u  1 aðtÞ ¼ muðtÞ þ c

ð2Þ

ð3Þ

Computational Method for Muscle Forces Estimation

49

where aðtÞ is the muscle activation, eðtÞ is the neural excitation, u(t) is the neural activation, a; b; m and c are constant coefficients, and sact and sdesc are, respectively, the activation and deactivation times constants [11]. 2.3

Computational Method to Estimate Muscle Forces

Hill Rheological Model. To obtain more realistic forces, it is essential to use the correct rheological model of the physical behavior of the actuators. In this study, we used Hill’s type model, shown in Fig. 2. Each musculotendon actuator is represented by two main elements: tendon and muscle fibers. Firstly, the tendon is modeled by simple elastic model. Secondly, the muscle fibers are represented by three elements: a contractile element controlled by the nervous system producing the active muscle force in parallel with a passive element that produces the passive force, which is represented by a damper and an elastic element. Then, the force exerted by a muscle is the sum of the passive and active forces: F M ¼ FA þ FP

ð4Þ

The active and passive forces are calculated according to several physiological parameters of muscles such as: maximum isometric muscle force F0M , force-length relationships fla and flp , force-velocity relationship flv and the activation of muscle aðtÞ. For these relationships, we have chosen to use the analytical expressions developed by [13]. Indeed, the force-length relationship for the passive element is determined by an exponential function and the active force-length relationship of contractile element is represented by a Gaussian function. Additionally, active muscle force is also dependent on muscle velocity. Previous studies show that when a muscle actively shortens, it produces less force than it would under isometric conditions. They have quantified this result by an empirical hyperbolic relationship when a muscle is shortening. Then, active and passive forces are expressed, respectively, by “Eq. (5) and Eq. (6)”.

Where LM and V M are, respectively, the length and velocity of muscle fibers. LMT is the musculotendon length which is determined by OpenSim software. Based on these equations, we have determined the analytical expression of muscle fibers velocity as a function of the physiological parameters of the muscles [13]. Dynamics Contraction. Once the muscle activation and the musculotendon lengths are obtained, the next step is to compute the musculotendon forces. According to Hill-type

50

O. Jemaa et al.

model, muscle fibers are in series and off-axis by a pennation angle a with the tendon. The total length of the musculotendon complex is denoted then by LMT , expressed by: LMT ¼ LT þ LM cosðaÞ

ð8Þ

Where LT is the tendon length and LM is the length of muscle fibers. The dynamic contraction of the each actuator is characterized by a first order differential equation “Eq. (9)” that will be integrated numerically to ensure the calculation of the musculotendon force. @F F_ MT ¼ @t

MT

  ¼ K T V MT  V M cosðaÞ

ð9Þ

Where V MT is the musculotendon velocity, V M is the muscle fibers velocity and K T is the stiffness coefficient of the tendon. Finally, to estimate the musculotendon forces during a dynamic contraction, each musculotendon is modeled by a system of three differential equations. The first equation represents the dynamics activation, the second equation represents the dynamics contraction, and the last allows the integration of the muscle velocity to explicitly produce the muscle length. The dynamics system is: 8 : u > < ¼ ðe  uÞðK1 e þ K2 Þ  F_ MT ¼ K T V MT  V M cosðaÞ > : _M L ¼ VM

ð10Þ

Where ðu; F MT ; LM Þ are the outputs of this mathematical model, and e is the input variable.

3 Results and Discussion The aim of this study was to explore the feasibility of direct dynamics approach for estimation the exerted muscle forces under dynamic contraction. This article selects the motion of elbow flexion as the research object, where flexion angle varies from 0° to 120°. As a first step, the EMG signals have been collected for two muscles involved in the elbow flexion, which are Biceps and Triceps muscles. Indeed, Biceps is the main flexor of the elbow, it is an agonist muscle. Triceps is the main extensor of elbow; it is an antagonist muscle. EMG signal for each muscle was measured with two surface electrodes. Surface electrodes are generally more practical than fine wire electrodes to detect muscle activation during motion analysis. However, random electrical disturbances, electrode misplacement may reduce the quality of the EMG data.

Computational Method for Muscle Forces Estimation

51

Fig. 3. Musculotendon length for Biceps (in red) and Triceps (in blue) muscles

After the experimental data collection, raw EMG signals are filtered, rectified and normalized by the Maximum Voluntary Contraction (MVC) values. In a second step, these normalized signals, called muscular excitation, were converted into the muscular activation. A musculoskeletal model of the upper limb was developed on OpenSim software. It has 7 degrees of freedom and it is controlled by 30 musculotendon units. Based on this biomechanical model, the lengths and velocities of the muscles were determined. Figure 3 shows the lengths for Biceps and Triceps muscles as a function of the flexion angle of the elbow. Next, the third step of direct dynamics method is to estimate muscle forces from muscle activities and lengths. The Hill model is considered as an idealized mechanical representation of the muscle. The dynamic of each muscle is modelled by a system of three differential equations. The first equation represents the muscle activation, the second is the muscle contraction, and the last differential

Raw EMG Fig. 4. Estimated forces from direct dynamics approach for Biceps and Triceps muscles

52

O. Jemaa et al.

equation expresses the variation of muscle length during motion. Finally, these differential equations were solved to obtain the musculotendon forces. Figure 4 shows a reasonably good correlation between the estimated muscle forces from direct dynamics method and measured EMG signals for the Biceps and Triceps muscles. This confirms the effectiveness of the direct dynamics approach in estimating the muscle forces.

4 Conclusion The aim of this paper has addressed a computational method to predict muscle forces of upper limb using EMG signals. The procedure includes three main steps: biomechanical model, dynamics activation and dynamics contraction. The biomechanical model of the upper extremity enables the determination of musculotendon lengths for Biceps and Triceps muscles. The dynamics activation method allowed to filter and convert the raw EMG signals to muscle activations. Finally, the dynamics contraction was developed to predict muscle forces based on a rheological model allows modeling the elastic behavior of the muscle.

References 1. Ai, Q., Ding, B., Liu, Q., Meng, W.: A subject-specific EMG-driven musculoskeletal model for applications in lower-limb rehabilitation robotics. Int. J. Hum. Robot. 13(3), 1–22 (2016) 2. Pontonnier, C., Dumont, G.: From motion capture to muscle forces in the human elbow aimed at improving the ergonomics of workstations. Virtual Phys. Prototyp. 5(3), 113–122 (2010) 3. Li, Z., Huang, Z., He, W., Su, C.Y.: Adaptive impedance control for an upper limb robotic exoskeleton using biological signals. IEEE Trans. Ind. Electron. 64(2), 1664–1674 (2017) 4. Erdemir, A., McLean, S., Herzog, W., van den Bogert, A.J.: Model-based estimation of muscle forces exerted during movements. Clin. Biomech. 22(2), 131–154 (2007) 5. An, K.N.: Graphical Interpretation of tie Solution to tie Redundant Problem in Biomechanics (1978) 6. Wang, L., Buchanan, T.S.: Prediction of joint moments using a neural network model of muscle activations from EMG signals. IEEE Trans. Neural Syst. Rehabil. Eng. 10(1), 30–37 (2002) 7. Bélaise, C., Dal Maso, F., Michaud, B., Mombaur, K., Begon, M.: An EMG-marker tracking optimisation method for estimating muscle forces. Multibody Syst. Dyn. 42(2), 119–143 (2018) 8. YuWei, W., HanWu, H.: Research on the motion characteristic of elbow joint angle based on the sEMG of single muscle. Cogent Eng. 3(1), 1–12 (2016) 9. Mobasser, F., Eklund, J.M., Hashtrudi-zaad, K.: Estimation of elbow-induced wrist force with EMG signals using fast orthogonal search. IEEE Trans. Biomed. Eng. 54(4), 683–693 (2007)

Computational Method for Muscle Forces Estimation

53

10. Delp, S.L., et al.: OpenSim: open-source software to create and analyze dynamic simulations of movement. IEEE Trans. Biomed. Eng. 54(11), 1940–1950 (2007) 11. Lloyd, D.G., Besier, T.F.: An EMG-driven musculoskeletal model to estimate muscle forces and knee joint moments in vivo. J. Biomech. 36(6), 765–776 (2003) 12. Manal, K., Buchanan, T.S.: A one-parameter neural activation to muscle activation model: estimating isometric joint moments from electromyograms. J. Biomech. 36(8), 1197–1202 (2003) 13. Thelen, D.G.: Adjustment of muscle mechanics model parameters to simulate dynamic contractions in older adults. J. Biomech. Eng. 125(1), 70–77 (2003)

Design of a Variable Stiffness Joint for a Five-Bar-Mechanism Maria Guadalupe Contreras-Calderón(&) and Eduardo Castillo-Castañeda Instituto Politécnico Nacional, CICATA-Unidad Querétaro, Querétaro, Mexico [email protected], [email protected]

Abstract. Variable stiffness actuators have been developed as an alternative to rigid actuators and have been implemented in human-robot interaction tasks such as industrial or assistance tasks to improve the dynamic performance of robots. This paper presents a new design of a variable stiffness joint to generate resistance to movement in the end-effector of a mechanism. The design is based on the cantilever beam principle, where the beam presents a flexion when applying a force and that keeps its stiffness constant. Movable support is implemented, which slides through the beam and allows to vary its flexion according to the movable support distance that is proportional to the stiffness. Also, tests have been performed with the rotational joint, where the end of the beam has been fixed on the rotational joint axis and the link is rotated to a specific angle to study the contribution of stiffness in the resistance to movement of the end-effector link. Keywords: Variable stiffness joint Resistance force


Five-bar mechanism
Cantilever beam


1 Introduction The mechanical devices that interact with humans must be safe and consider all types of possible accidents. The traditional robotics systems have rigid structures and they are powered used rigid active joints such as: DC motors, AC motors, stepping motors, pneumatic or hydraulic pistons, frequently with a reduction gear. These rigid active joints are convenient in industrial applications to achieve high accuracy in position, fast operation and high stiffness. However, since more and more robotics systems working in cooperation with humans are developed, high demand for safety is required. This is a paradigm shift from rigid robots to flexible robots, that can be implemented, for example, using flexibles elements or flexible joints [1]. It has been reported in [2–4] that robots used in human interaction environments using systems with compliance elements such as joints or actuators, reduce injuries caused by accidental collision between them, due to the reduction of inertia during a dynamic impact, thanks to the mechanical decoupling between the inertia of the motor and the inertia of the links. In general, the actuators are designed to be as rigid as possible to avoid the uncertainties generated by flexible or soft links since its automatization is easier in an environment structured. Recently, a major research effort has been centered in develop variable © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 54–63, 2020. https://doi.org/10.1007/978-3-030-48989-2_7

Design of a Variable Stiffness Joint for a Five-Bar-Mechanism

55

stiffness actuators (VSA). Numerous VSA designs have been proposed with a motivation of: Improve the safety of robots and humans (providing intrinsic compliance), add additional redundancies in control (allow it that robots be rigid and accuracy) and, improve the energetic efficiency and the dynamic range of existing actuators (for example, exploiting energy storage capabilities). When incorporating an elastic element, the new actuator offers the possibility to achieve a high dynamic range with smaller motors. However, one of the difficulties using such actuators is the considerable increase in complexity on the trajectories planning and control, particularly in the context of movements with highly dynamics [5]. Recently, some designs have been proposed for specific applications such as the variable stiffness actuator for assistance robots with decoupled control [6]. The actuator AwAS presented in [7] was implemented in ASIBOT, an assistance robot, they demonstrated that the adjustment of joint stiffness, for a given range of values, reduce the impact force during a collision between human a robot. In the work proposed in [8], the implemented control of a VAS determines the human-robot interaction speed. Stiffness references will be generated adopting an on–line sub–optimal control algorithm. The possibility of varying the stiffness of the transmission reduces the risk of injury during the execution of highspeed trajectory tracking tasks. The University of Almeria designed a VAS that they denominated MEVASTT [9] which presents a serial configuration based on cablepulley transmission, the actuator has wide ranges of angular deviation and rigidity, being able to reach a completely stiff behavior. A VAS using variable stiffness springs [10] was implemented in a high-speed five-bar mechanism to minimize input torques for the pick-and-place specific task. In this paper, we propose a variable stiffness joint (VSJ) to adjust the end-effector stiffness to vary the resistance in a five-bar-mechanism. In our case, the actuators will only be used to vary the stiffness and not to generate the movement of the five-bar mechanism. This mechanism can be used as a basis in industrial robotics applications where there is interaction between a human operator and the robot.

2 VSJ Design Proposal 2.1

Conceptual Design

The design proposal is based on the cantilever beam principle, see Fig. 1. In this case, a beam with length l is embedded in its end A and free in the opposite end B; the beam flexes when a force F is applied around B, in this case, the beam stiffness is constant. We propose the integration of a second element, named movable support (MS) that slides along the beam varying the flexion according to the distance d. With this new configuration, the beam stiffness becomes variable that depends on the distance d. M1 corresponds to the bending moment. Our proposal is to fix the beam free end to the rotational joint axis. In this way, the stiffness of this joint will also depend on the distance d, see Fig. 2. When the MS slides, the beam stiffness is modified and the resistance in C can be changed (Fig. 3).

56

M. G. Contreras-Calderón and E. Castillo-Castañeda

Fig. 1. Conceptual design diagram

Fig. 2. The beam fixed in axis of a rotational joint.

Fig. 3. The beam is deformed according a curvature radius q.

Considering that the beam is made of the linear elastic material, according with the Euler-Bernoulli principle, the bending moment M1 is given by: M1 ¼

EI q

ð1Þ

where E is the Young’s modulus, I is the moment of inertia of the beam cross-section and q is the curvature radius of the deformed beam. The term EI depends on the beam material, named “flexural stiffness module” [11]. The stiffness constant k is defined as follows:

Design of a Variable Stiffness Joint for a Five-Bar-Mechanism

k¼

2.2

Fl2 EI

57

ð2Þ

CAD Design

The main idea is to replace the active joints of a five-bar mechanism with VSJ so that the end-effector of the mechanism represents the stiffness as a resistance to the motion when performing trajectories, to eliminate motors that generate the movement of the mechanism. In this way, the only actuators that remain are those that perform the stiffness variation. Figure 4 shows the CAD design of the VSJ applied in a rotational joint. The end of the beam is fixed to the base and the opposite end is fixed in the axis of the rotational joint, therefore when the link rotates the beam flexes.

Fig. 4. CAD design of VSJ.

The proposal VSJ design varies the resistance in the end-effector (point C) of the link through the variation of the stiffness in the joint modifying the distance d. The MS slides through the base rail along the beam varying the distance from the point B, while the link rotates and the beam flexes. As the d increases the resistance in the end-effector increases if d decreases the resistance in C decreases. The MS has two spherical wheels that allow to slide along the beam and support the beam flexion. The VSJ can be controlled by a linear motion of the MS to obtain a specific resistance in the endeffector. Two VSJ’s are integrated in a five-bar mechanism, replacing the active joints, represented by h1 and h2, as shown in Fig. 5. 2.3

Prototype Manufacturing

The prototype is manufactured with PLA material with 3D printing, the beam is built with two flexible materials: PLA and flexible metal (steel alloy, 8% cobalt), both beams with length of 11 cm with a constant cross section. Figure 6 shows the final VSJ assembly. A dynamometer is placed at the end-effector to measure the resistance force.

58

M. G. Contreras-Calderón and E. Castillo-Castañeda

Fig. 5. The joint is implemented in a five-bar-mechanism

Fig. 6. Final VSJ assembly.

3 Resistance Force Measure The tests are performed for two different material: PLA and steel alloy. The test consists of manually move the end-effector to rotate the VSJ a b angle, to reach six different angles values, see Fig. 7: −45°, −30°, −10°, 10°, 30° and 45°. For each angle, the force F necessary to rotate the end-effector was measured using a RHINO digital dynamometer with an accuracy of 10 g. Five repetitions of this measurement were performed to obtain an average force. Also, the MS was displaced a distance d from the point A to the closest point to the rotational joint. Five different d values (cm) were selected: 1, 3, 5, 7 and 9. Therefore, five beam stiffness can be estimated. The resulting graphs for PLA material are shown in Fig. 8, for b and −b, with a beam length of 12 cm.

Design of a Variable Stiffness Joint for a Five-Bar-Mechanism

59

Fig. 7. Link angular position

Fig. 8. The relation between distance d and resistance force (PLA)

The diamonds, squares and triangles lines correspond to the resistance force behavior for an angle of ±10°, ±30° and ±45° respectively. The force required to move the link to a specific angle is increasing as d increasing. The maximum resistance force, 4.74 N, was obtained for a distance d of 9 cm and an angle of −45°; however, there is a symmetrical behavior since the resistance force at +45° has a similar value (4.58 N). The equilibrium point, that is, where the resistance force is equal to zero, is obtained for an angle of 0°. Also, to reach angles higher than ±45°, more force is required, and the beam material fractures. The resulting graphs for steel alloy material are shown in Fig. 9, for b and −b, with a beam length of 11 cm. It can be observed that for 10° rotations there are no large variations of the resistance force. However, for rotations of 30° and 45° the force increases for big values of distance d; as the previous case for PLA material. The maximum resistance force obtained is 16,219 N for an angle of 45° and a distance of 7 cm; from that distance values, the beam presents a permanent deformation.

60

M. G. Contreras-Calderón and E. Castillo-Castañeda

Fig. 9. The relation between distance d and resistance force (steel alloy)

4 Simulation The materials characteristics are: for Steel alloy the Young’s modulus (E) is 20.6  1010 Nm2, and PLA material is 3.5  109 Nm2, the moment of inertia (I) of the beam is 6.66  10−12 m2 and the length is 0.1 m. It obtains the bending moment and the shear force equations using 30° link position and the d = 0.01 m for movable support; the necessary force to move it is 7.4 N for steel alloy and 2.45 N for PLA. The free body diagram and bending moment for both materials is show in Fig. 10.

a)

b)

Fig. 10. a) Free body diagram b) Shear force and bending moment

The following values result, for steel alloy F = 7.4 N, vertical reaction Ry = −111.0 N, RMS = 118.4 N, k = 0.05393 and a moment about the fixed support Ma ¼ þ 0:37 Nm. For PLA material F ¼ 2:45 N, Ry = −36.75 N, RMS = 39.2, k = 1.0510 and Ma ¼ þ 0:1225 Nm. The equations for the shear force (V1, V2) for 0  x  0:1 and the bending moment (M1, M2) are following:

Design of a Variable Stiffness Joint for a Five-Bar-Mechanism Steel alloy V1 = −111 V2 = 7.4 M1 = −111x −0.37 M1 = 7.4x −0.814

61

PLA V1 = −36.75 V2 = 2.45 M1 = −36.75x −0.1225 M2 = 2.45x −0.2695

The beam deflection and the elastic curve is shown in Fig. 11, it can be seen the deflection value of a1 = 0.00008 mm, a2 = 0.07693 mm for steel alloy and a1 = 0.001555 mm a2 = 1.505 mm for PLA. The deformation in the base is almost zero. The stress results for a deflection sum shows in Fig. 12, deflection larger than one of metal, suffers the major deflection where de force F is applying. We can observe that the PLA material have a deflection major than metal alloy, and then, need less force to move the link. The MS was placed at distance d = 0.07 m, and the link move to 30° position, the necessary force for steel alloy was F = 11.3 N and F = 3.19 N for PLA. The following values result, for steel alloy, vertical reaction Ry = −9.68 N, RMS ¼ 20:986 N, k = 0.1317 and a moment about the fixed support Ma ¼ þ 0:226 Nm. For PLA material, Ry = −2.0507 N, RMS = 5.2407, k = 0.2189 and Ma ¼ 0:0478 Nm The equations for the shear force (V1, V2) for 0  x  0:1 and the bending moment (M1, M2) are following: Steel alloy V1 = −9.6857 V2 = 11.3 M1 = −9.6957x −0.226 M1 = 11.3x −1.243

PLA V1 = −2.0507 V2 = 3.19 M1 = −2.0507x −0.0478 M2 = 3.19x −0.319

The deflection beam shows in Fig. 13a, the stress results are as follows, a1 = 0.002379 mm, a2 = 0.016174 mm for steel alloy (Fig. 13b) and a1 = 0.02976 mm a2 = 0.1353 mm for PLA (Fig. 13c). Depending on the distance of the MS is the deformation in the bar. The smaller the distance d, the deformation between the MS and the point A decreases, while the deformation increases from the MS to the point B. If the distance d increases, the deformation between the MS and the point A increases, while between the MS and point B decreases. As can be seen in the stress results, PLA has greater deformation than steel.

Fig. 11. Deflection and elastic curve of the beam

62

M. G. Contreras-Calderón and E. Castillo-Castañeda

Fig. 12. Deflection on the beam a) steel alloy b) PLA

Fig. 13. a) Deflection force of the beam b) Steel alloy analysis stress c) PLA analysis stress.

5 Conclusions The design and construction of a stiffness joint variable that allows variation in the resistance force on the end-effector of a five-bar mechanism was presented. This joint is based on the principle of a cantilever beam with a support point. The resistance force obtained for two types of materials was measured by displacing of the movable support and the angle of rotation of the joint’s output link.

Design of a Variable Stiffness Joint for a Five-Bar-Mechanism

63

The beam made of PLA presented a fracture when the MS reach its maximum displacement at a joint rotation of 45°. On the other hand, the steel alloy beam only presents a permanent deformation but there is not the fracture. The material deformation is important for this kind of application, this joint can be located as an active joint in some mechanisms, such as a five-bar, which industrial application is to perform repetitive trajectories in one plane; then, the material will deform faster and the resistance characteristics will change. The next stage of this project will be to control the MS, with a linear actuator, to obtain a desired resistance force values on the end-effector for a five-bar mechanism; also, to find the corresponding dynamic model.

References 1. Hogan, N.: Impedance control: an approach to manipulation: Part I—theory. ASME. J. Dyn. Syst., Meas. Control. 107(1), pp. 1–7 (1985) 2. Flacco, F., De Luca, A.: Stiffness estimation and nonlinear control of robots with variable stiffness actuation. In: The International Federation of Automatic Control, Milano Italia, pp. 6872–6879 (2011) 3. Park, J., Lee, Y., Song, J., Kim, H.: Safe joint mechanism based on nonlinear stiffness for safe human-robot collision. In: IEEE International Conference on Robotics and Automation, Pasadena, CA, pp. 2177–2182 (2008) 4. Hyun, D., Yangm, H.S., Park, J., Shim, Y.: Variable stiffness mechanism for human-friendly robots. Mech. Mach. Theory 45(6), 880–897 (2010) 5. Braun, D.J., Howard, M., Vijayakumar, S.: Exploiting variable stiffness in explosive movement tasks. Robot. Sci. Syst. 7, 25–32 (2012) 6. Medina, J., Jardón, A., Balager, C.: Control desacoplado de un actuador de rigidez variable para robots asistenciales, RIAI - Rev. iberoamericana de automática e informática industrial 13(1), 80–91 (2016) 7. Huete, A.J., Victores, J.G., Martínez, S., Giménez, A., Balaguer, C.: Personal autonomy rehabilitation in home environments by a portable assistive robot. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 42(4), 561–570 (2012) 8. Tonietti, G., Schiavi, R., Bicchi, A.: Design and Control of a variable stiffness actuator for safe and fast physical human/robot interaction, pp 526–531 (2005) 9. López, J., Giménez, A., García, D., Jardón, A., Balaguer, C.: Diseño y simulación de un actuador de rigidez variable. An. Ing. Mecánica Rev. Asociación Española Ingenieria Mecánica 1(18), 154–161 (2012) 10. Balderas, R., Briot, S., Chriette, A., Martinet, P.: Minimizing input torques of a high-speed five-bar mechanism by using variable stiffness springs. In: CISM, vol. 584, pp. 61–68 (2019) 11. Beléndez, T., Neipp, C., Beléndez, A.: Flexión de una Barra Delgada Empotrada en un Extremo: Aproximación para Pequeñas Pendientes. Rev. Brasileira de Ensino de Física 24(4), 399–407 (2002)

Visual Control Based Musculotendon Force Estimation in the Human Upper-Limb Souha Baklouti(&), Olfa Jemaa, and Sami Bennour Laboratory of Mechanical of Sousse ‘LMS’, University of Sousse, Bp. 264 Erriadh, 4023 Sousse, Tunisia [email protected]

Abstract. This paper deals with an upper-limb muscle forces estimation method using Kinect camera. This latter provides 3-D information of the upper limb trajectory, which serves as an input of the kinetic model of the upper-limb. Estimation methods are used to determine the muscle force from the torque output of the inverse dynamics. An EMG-based forward dynamics model is used for validation purposes. Experimental results on a human being validate the proposed visual tracking upper-limb muscle forces estimation method. Keywords: Visual control


EMG
Upper-limb
Musculotendon force

1 Introduction Due to repetitive tasks and strenuous work which requires supporting combined loads, whether in industry, sports or simply for household chores, the risks of developing applied musculoskeletal disorders is high. In France, for example, the INRS admitted that the number of people suffering from musculoskeletal disorders had reached 43359 [1] and that it was constantly increasing. However, this disorder is much more serious in developing countries because they do not respect the same safety rules and do not benefit from sufficient preventive measures [2]. In this case, doctors and physiotherapists need a reliable tool allowing them to follow the evolution of the patient’s muscular activity. However, because of the complex arrangements of muscles and the number of muscles involved in a movement, Zajac and Gordon state that it may be difficult to experimentally assess the role of muscles [3] which makes it difficult in the professional practice of physiotherapy to determine the numerical value of a muscle force during an activity. This paper proposes a physiotherapy-at-a-distance system allowing the estimation of the Human upper limb muscle forces. An analysis of the patient’s movements is therefore essential to quantify the stresses applied on the musculoskeletal structure. On this ground, this paper proposes two methods to estimate the force generated by muscles. The first is a method of inverse dynamics which has as main input the data acquired by KINECT XBOX360 and includes a robotic model of the right upper limb. The second method has a validation purpose and is a forward dynamics method having an electromyography signal as input. The electromyography (EMG) signal was acquired by a wireless EMG sensor kit.

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 64–72, 2020. https://doi.org/10.1007/978-3-030-48989-2_8

Visual Control Based Musculotendon Force Estimation in the Human Upper-Limb

65

2 Upper-Limb Muscle Forces Estimation Method To estimate muscular effort, there are many methods that have been the subject of research for years. These methods can be classified into three families: forward, inverse and mixed dynamics estimation methods [4]. The forward dynamics estimation methods are based on an estimation of the efforts directly from recordings of muscle activity, most often from electromyography recordings (EMG). Even though several studies demonstrate the difficulty of estimating efforts based solely on the direct method [5–7], the direct methods are reliable, at least from a qualitative point of view of muscle contraction patterns, since they are based on a direct measurement of muscle activity. Inverse dynamics methods use movement to determine muscle effort. These methods are interesting for virtual reality applications because the estimation of the forces can be carried out from a simple measurement of the movement. Mixed muscle forces estimation methods are a combination of the physiologically relevant method, forward dynamics, and the mechanically relevant methods, inverse dynamics. Generally, mixed methods are a correction of the EMG estimates, by optimizing the values of the resulting torques to make them coincide with the values measured during the experiment. In this paper, we introduce an inverse dynamics muscle forces estimation method using a KINECT XBOX 360 sensor and an EMG-based forward dynamics muscle forces estimation method. The test rig is as shown by Fig. 1.

Fig. 1. Schematic diagram of the proposed musculotendon force estimation method test rig.

2.1

Motion and EMG Signal Acquisition

The motion and EMG signal recording are ran simultaneously. However, the data acquired by Kinect is noisy because the depth data recorded is of poor quality since the approach to structured light it adapts is not always sufficiently robust to provide a high level of completeness of the framed scene. Filtering is then essential to extract the necessary data from the noise. The filter we used is a 6-order bandpass filter with the cutoff frequencies {5 and 20} Hz. The duration of acquisition has also been saved as

66

S. Baklouti et al.

well and is the duration of the experience. The motion acquisition and duration data will serve as an input to the inverse dynamics model. The EMG sensor that we have prepared is designed, as shown in Fig. 2, to send signals via Wi-Fi in order to get rid of the wires that restrict the subject’s movements.

Batterie li-po Li-po battery Port debattery chargement pourport batterie li-po Li-po charging Module Wi-fiESP-12F ESP-12F Wi-fi module

EMG Myoware sensor Capteur EMG Myoware

Fig. 2. EMG-sensor kit 3D design and components.

The raw EMG is saved and will be used as an input to the EMG-based forward dynamics model for validation purposes. 2.2

Inverse Dynamics and Forces Estimation

To proceed with the inverse dynamics in order to calculate the joint torques, we created a robotics model that represents the right upper-limb parameters for human being. In biomechanics, body segments are seen as solids connected by articular groups or joints [8]. Therefore, the laws of solid kinematics can be applied to the upper limb segments. The upper limb model we use includes 7 degrees of freedom that define the kinematics of the shoulder, elbow and wrist [9]. Previous studies have shown strong relationships between the body’s segments properties (length, weight, center of mass, moment of inertia) and the Human height, weight and gender. The relationships between these three parameters and the body properties used in our model were obtained from [10] and [11]. The upper limb kinematics model is shown by Fig. 3 and represented by Table 1 using the Denavit-Hartenberg convention [12] with CA is the carrying angle (initial angle between arm and forearm), L1 is the length of the arm, L2 is the length of the forearm, L3 is the length of the hand. ai, ai, hi and di are the Denavit-Hartenberg parameters. h = [h1 … h7] with hi is the angular position of the ith joint, i = 1 … 7. As the input for the inverse dynamics are the joints articular parameters (trajectory, speed and acceleration), a passage through the inverse kinematics in essential. For the inverse kinematics, Paul method was used to determine the joints articular trajectories. The outputs of the inverse kinematics are the joint articular trajectory which can be derived into the joint articular speed and acceleration and serve as an input to the inverse dynamics. [13] stated that the recursive Newton-Euler formulation (RNE) is used to calculate the inverse dynamics of the manipulator, i.e. the torques of the joints required for a given set of angles, speeds and accelerations of the joint.

Visual Control Based Musculotendon Force Estimation in the Human Upper-Limb

67

Fig. 3. 7-DoF 3D Kinematics model of the upper limb. Table 1. Denavit-Hartenberg table representing the upper-limbs links. Functional motion ai p Flexion-extension (Shoulder) 2 p Adduction-abduction (Shoulder) 2 p Internal-external rotation (Shoulder) 2 Flexion-extension (Elbow)  p2 þ CA Pronation-supination of the forearm (Elbow) p2 p Flexion-extension (Wrist) 2 p Adduction-abduction (Wrist) 2

ai 0 0 0 0 0 0 −L3

hi h1 h2 h3 h4 h5 h6 h7

+ + +

p 2 p 2 p 2

+

p 2

di 0 0 −L1 0 −L2 0 0

Muscle force (the unknown of the problem) is related to the muscle torque and the muscular moment arms as shown by Eq. (1) where TMT is the musculotendon moment matrix, RðhÞ is the moment arm matrix and FMT is the musculotendon force matrix at a given instant. TMT ¼ RðhÞ
FMT

ð1Þ

However, the inversion of the equation is not possible as our system is redundant because the upper limb involves several muscles to actuate a single joint, which implies that there are not enough equations of balance between the articular torques necessary for movement and the muscular efforts generating these torques. Therefore, the muscles forces estimation problem can be solved through optimization by minimizing an objective function representing a physiological behavior of humans in order to obtain a distribution of efforts. We choose as objective function nonlinear functions allowing a more realistic theoretical distribution of forces (for example, Minimization of the sum of stresses in muscles [14], Minimization of the sum of muscular efforts [15],

68

S. Baklouti et al.

Minimization of the sum of normalized muscular efforts [16]) subjected to constraints representing the equality of the sum of the individual muscular moments compared to the couples joints calculated from the analysis of inverse dynamics [17]. The musculoskeletal model we proposed includes 50 muscles and muscle compartments and the moment arms are estimated, given the joint angle and the respective framework, from data given by Holzbaur [18]. 2.3

EMG-Based Forward Dynamics

As has been mentioned, an EMG-based forward dynamics procedure was followed in order to extract the musculotendon force from the raw EMG signal for validation purposes. The transition from the raw EMG signal to the calculation of muscle strength is not trivial and involves several steps. First, the raw EMG signal must be processed and transformed into a rectified, filtered and normalized EMG. Second, since the EMG signal is a function of its previous values, we must take into account the activation history of the muscle which leads to neuronal activation. Next, we transform neural activation into muscle activation using the non-linearization relationship [19]. Finally, muscle activation serves as an input to the Hill type muscle model. In the Hill-type model, the tendon attaches muscle fibers to the bones of the musculoskeletal model therefore the force exerted on the tendon is the force generated by the muscle model as a whole. Mathematically, each muscle is modeled by a system of three differential equations (Eq. 2 [20]. The first line of Eq. 2 represents the muscle activation dynamics, the second line represents the muscle contraction dynamics and the third line allows the determination of the muscle length by integrating the muscle velocity where u is the neural activation, k1 and k2 are constants determined from activation and deactivation time, e is the rectified, filtered and normalized EMG signal, a is the muscle activation, FT is the musculotendon force, LM is the muscle length and VM is the muscle contraction velocity. 3 3 2 ðe  uÞðk1 u þ k2 Þ u_ T M 7 4 F_ T 5 ¼ 6 4 g F ; a; u; L 5 M M d L L_ VM ¼  dt 2

ð2Þ

3 Experimental Results One of the most repetitive movements, regardless of work, age, gender or any other factor, is the movement that connects the hand to the mouth (For example: eating, brushing your teeth, etc.). For illustrative purpose, our two volunteers (subject 1: male, 23 years old, 1.86 m and 81 kg; Subject 2: female, 23 years old, 1.56 m and 70 kg) were invited to do this movement in front of Kinect XBOX 360, with the EMG sensor kit fixed to their right arm, while imitating one another, at a slow pace to ensure better results. All tests are done at zero external force, i.e. no mass was carried by the hand. The hand’s Cartesian trajectory acquired by the motion acquisition device for subject 1

Visual Control Based Musculotendon Force Estimation in the Human Upper-Limb

69

are as shown in Fig. 4. These Cartesian coordinates are implemented in the inverse kinematics model, which yields the articular trajectory, speed and acceleration of the joints. These articular parameters are then implemented in the inverse dynamics model and yields the articular torques.

Fig. 4. Cartesian trajectory of the hand on the three axes X, Y and Z over time, result of the filtering of the coordinates acquired by KINECT XBOX 360 during a hand-mouth movement, subject 2.

In this paper, we focus on muscle strength for muscle ECU (Extensor carpi ulnaris). The ECU extends the wrist, but acting alone, it inclines the hand towards the ulnar side; by its continuous action, it extends the elbow joint. Therefore, we pay particular attention to the 4th joint of the upper limb, flexion-extension movement of the forearm in order to observe the effect of the gender and physical characteristics of the person on the generation process of the muscle force. The torque around the forearm flexionextension joint resulting from the upper-limb’s movement is as shown in the Fig. 5a. This figure shows the difference between the male joint torque (subject 1) and female joint torque (subject 2). As expected, a man’s joint torque is greater than that of a woman. This is due to the physiological characteristics of men (for example, a larger size). However, women reach peak torque more often than men. This result is validated by [21] who reported that the time taken by women to reach 70% of their maximum isometric strength level was almost double that of men. The moment arms estimated from the data given by Holzbaur model show higher variation in the male lever arm than in the female. For example, the ECU moment arm from the flexion-extension forearm joint is as shown by Fig. 5b. These results can be explained by the fact that men generally have larger muscles and longer arms than women. The higher moment arm for men indicates that the tension on the muscle fibers when stretched is higher for a given change in joint angle, which partly explains the increased risk of injury for men. The muscle force estimation yields the result shown by Fig. 5c. The accuracy of the estimated muscular forces was tested by quantitative comparison of the inverse dynamics model and the EMG-based forward dynamics model.

70

S. Baklouti et al.

(b)

(a)

(c)

Fig. 5. (a) Moment arm for subject 1 and subject 2 estimated from Holzbaur model. (b) Joint torque around flexion-extension of forearm joint obtained by inverse dynamics for subject 1 and subject 2. (c) Estimated musculotendon force from the joint torques for subject 1 and subject 2.

Since the two signals have delays between them as they are not from the same source, we tried to adjust the time span to that which synchronizes the two force curves. Over a time span of 200 s (zoom of the area in red shown in Fig. 5c), the muscle force determined from inverse dynamics and EMG-based forward dynamics shows as presented by Fig. 6a, for subject 1 and Fig. 6b, for subject 2.

(a)

(b)

Fig. 6. Comparison between the musculotendon force estimated by inverse dynamics and EMGbased forward dynamics for (a) subject 1 and (b) subject 2 at an interval of 200 s.

Visual Control Based Musculotendon Force Estimation in the Human Upper-Limb

71

The root mean square (RMS) was calculated from the musculotendon force estimated by the inverse dynamics model at an interval of 200 s to be 5.06N and 2.38N compared to 3.69N and 1.99N calculated from the musculotendon force estimated by the EMG-based forward dynamics, which presents 72.95% and 83.88% from the musculotendon force estimated by the inverse dynamics, for subject 1 and subject 2, respectively. The maximum force generated by ECU at an interval of 200 s is estimated to be 23.65N and 11.34N by inverse dynamics for subject 1 and subject 2, respectively. These values are estimated to be 18.14N and 9.28N by the EMG-based forward dynamics method which presents 76.70% and 81.83% of the musculotendon forces estimated by the inverse dynamics, for subject 1 and subject 2, respectively. The coefficient of determination, R-squared, of the two estimated force data for subject 1 is 0.404. In other words, the two curves tighten around each other 40.4% of the time. On the other hand, the R-squared of the two force data estimated for subject 2 is 0.7321. That is to say, the two curves tighten around each other 73.2% of the time. We believe that the main reason for these results is that the EMG of the ECU is well amplified by the pretreatment and that the ECU muscle of subject 2 produces excessive force. Another reason is that subject 2 does not practice any type of sport while subject 1 does it regularly. Studies show that the EMG signal in upper limb is influenced by muscle fatigue resulting from strenuous exercise, which causes a decrease in the muscle activation level during the task [22], which is believed to be the case for subject 1.

4 Conclusion and Future Work This paper proposed a visual tracking based musculotendon force estimation in the Human Upper-Limb method using the inverse dynamics. An EMG-based forward dynamics model was proposed to validate the results obtained by the inverse dynamics. The results of the muscle force estimation are satisfactory, particularly for subject 2. Accordingly, a contribution of this paper deals with a satisfactory muscle force estimation using Kinect camera. Future work will deal with experimental validations for weight-lifting. To do this, other tests concerning the isometric forces for a given held weight must be carried out. Another future work is to predict musculoskeletal disorders from the data acquired and generated by our system. Therefore, a historical record of the subjects is essential and statistical methods are to be used.

References 1. INRS: Troubles Musculo-squelettique (TMS). www.inrs.fr/risques/tms-troubles-musculo squelettiques/statistiques.html 2. Koehn, E.E., Kothari, R.K., Pan, C.S.: Safety in developing countries: professional and bureaucratic problems. J. Constr. Eng. Manag. 121, 261–265 (1995) 3. Zajac, F., Gordon, M.: Determining muscle’s force and action in multi-articular movement. Exerc. Sport Sci. Rev. 17(1), 187–230 (1989)

72

S. Baklouti et al.

4. Buchanan, T.S., Loyd, D.G., Manal, K., Besier, T.F.: Neuromusculoskeletal modeling: estimation of muscle forces and joints moments and movements from measurements of neural command. J. Appl. Biomech. 20, 367–395 (2004) 5. Gagnon, D., Larivière, C., Loisel, P.: Comparative ability of EMG, optimization, and hybrid modelling approaches to predict trunk muscle forces and lumbar spine loading during dynamic sagittal plane lifting. Clin. Biomech. 16(5), 359–372 (2001) 6. Loyd, D.G., Besier, T.F.: An EMG-driven musculoskeletal model to estimate muscle forces and knee joint moments in vivo. J. Biomech. 36, 765–776 (2003) 7. Brown, S.H.M., Potvin, J.R.: Constraining spine stability levels in an optimization model leads to the prediction of trunk muscle cocontraction and improved spine compression force estimates. J. Biomech. 38(4), 745–754 (2005) 8. Cappozzo, A.: Gait analysis methodology. Hum. Mov. Sci. 3, 27–50 (1984) 9. Gopura, R.A.R.C., Kiguchi, K., Li, Y.: SUEFUL-7: a 7DOF upper-limb exoskeleton robot with muscle-model-oriented EMG-based control. In: The 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems (2009) 10. Murray, I.A.: Determining Upper Limb Kinematics and Dynamics During Every Day Tasks, in Mechanical, Materials and Manufacturing Engineering. University of Newcastle upon Tyne, Newcastle upon Tyne (1999) 11. Clauser, C.E.: Weight, volume, and center of mass of segments of the human body (1969) 12. Corke, P.I.: A simple and systematic approach to assigning Denavit-Hartenberg parameters. IEEE Trans. Robot. 23(3), 590–594 (2007) 13. Luh, J.Y.S., Walker, M.W., Paul, R.P.C.: On-line computational scheme for mechanical manipulators. ASME J. Dyn. Syst. Measur. Control 102, 69–76 (1980) 14. Crowninshield, R.D., Brand, R.A.: A physiologically based criterion of muscle force prediction in locomotion. J. Biomech. 14(11), 793–801 (1981) 15. Seireg, A., Arvikar, R.J.: Biomechanical Analysis of the Musculoskeletal Structure for Medicine and Sports. Hemisphere Publishing Corporation, New York (1989) 16. Pedotti, A., Krishnan, V.V., Stark, L.: Optimization of muscle-force sequencing in human locomotion. Math. Biosci. 38(1–2), 57–76 (1978) 17. Erdemir, A., Mclean, S., Herzog, B., Antonie, J.: Model-based estimation of muscle forces exerted during movements. Clin. Biomech. 22(2), 131–154 (2007) 18. Holzbaur, K.R.S., Murray, W.M., Delp, S.L.: A model of the upper extremity for simulating musculoskeletal surgery and analyzing neuromuscular control. Ann. Biomed. Eng. 33(6), 829–840 (2005) 19. Kurt, M., Buchanan, T.S.: A one-parameter neural activation to muscle activation model: estimating isometric joint moments from electromyograms. J. Biomech. 36, 1197–1202 (2003) 20. Menegaldo, L.L., Oliveira, L.F.: The influence of modeling hypothesis and experimental methodologies in the accuracy of muscle force estimation using EMG-driven models. Multibody Syst. Dyn. 28, 21–36 (2012) 21. Komi, P.V., Karlsson, J.: Skeletal muscle fibre types, enzyme activities and physical performance in young males and females. Acta Physiologica Scand. 103(2), 210–218 (1978) 22. Rota, S., Morel, B., Saboul, D., Rogowski, I., Christophe, H.: Influence of fatigue on upper limb muscle activity and performance in tennis. J. Electromyogr. Kinesiol. 24, 90–97 (2014)

Design and Simulation of an Underactuated Exoskeleton Mechanism for a User-Oriented Leg Exercising Yao Shuangji1(&) and Giuseppe Carbone2(&) 1

2

Yanshan University, Qinhuangdao, China [email protected] DIMEG, University of Calabria, Rende, Italy [email protected]

Abstract. This paper discusses the requirements for designing a leg exoskeleton, which is intended for human rehabilitation purposes. The proposed exoskeleton is based on an underactuated mechanism to achieve a leg walking motion with desired human-like behavior. A design solution with appropriate parameters is presented. Its main peculiarities and the operation performances are preliminarily characterized via a dynamic simulation. Keywords: Leg exoskeletons mechanism


Human rehabilitation
Underactuated

1 Introduction Rehabilitation of human limbs has attracted significant interest in the research community. Several design solutions and exoskeleton prototypes are available in research laboratories or even commercially such as reported in [1–12]. An interesting exoskeleton example is the Berkeley Lower Extremity Exoskeleton (BLEEX) that can enhance the loaded ability of wearers. BLEEX adopts bionic design, as based on wearable mechanical skeleton designed to imitate the structural characteristics of human limbs [1]. It can be used to improve the soldiers’ capability of walking loads in complex environment for long distance. Another example of a wearable exoskeleton robot named SuitX Phoenix is reported in [2]. SuitX Phoenix can support up to 4 h of walking, with a walking speed of 1.8 km/h (normal people’s walking speed is 4–7 km/h). The weight of wearable part including the battery is about 12 kg, cost about $40,000. Common drawbacks of the above mentioned examples is high costs and complexity. An underactuated leg exoskeleton that is based on linkage mechanism with spring elements is proposed for example, in [12]. Indeed, a promising low-cost solution can be based on using passive elements to constrain the kinematics of an underactuated mechanism. Accordingly, in this paper, we propose a linkage mechanism with passive elements that can adjust the position of links according to the ground contact for performing a human-like gait with a reduced number of degrees of freedom (DOFs). The proposed leg exoskeleton is modeled and simulated in dynamic conditions in order to validate its feasibility and estimate its main performances. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 73–81, 2020. https://doi.org/10.1007/978-3-030-48989-2_9

74

Y. Shuangji and G. Carbone

2 Design Requirements Human walking represents a complex dynamic activity, due to human body coordination and balancing. Main relevant tasks are: keeping stability, prevent injuries, avoiding obstacles and human body spatial motions. For example, [10] defines the human gait as cyclic motion of each human lower limb from a body support position to the next one with ground reactions high enough on each leg for supporting the rest of the human body. The motion cycle consists of two main phases: stand phase and swing phase. Each phase is made of a number of steps such as initial contact or heel strike, loading response in mid-distance, terminal stance, propulsion, [3]. Given the complexity of the walking one can propose an user-friendly exoskeleton prototype, which is designed by considering an underactuated linkage-based leg mechanism solutions such as proposed in [8, 9]. The exoskeleton mechanism design requirements aims at disabled medical recovery training and exercise. It should have a simple structure, be lightweight, easy to wear and to adapt to human locomotion. Moreover, the proposed mechanism should: 1) perform human gait cycle; 2) have less actuators than degrees of freedom; 3) be safe for any user.

3 The Proposed Underactuated Leg Exoskeleton Mechanism The proposed underactuated leg exoskeleton mechanism is shown in Fig. 1.

a)

b)

Fig. 1. The proposed underactuated leg mechanism exoskeleton.

Design and Simulation of an Underactuated Exoskeleton Mechanism

75

The mechanism is designed with only 1 actuator and 3 DOFs in hip, knee and ankle joint respectively. It contains spring elements to constrain the kinematics and three joints limitation in each joint. This kinematic design for a new underactuated leg exoskeleton is based on four-bar mechanism and linkage-slider mechanism with torsional springs and linear springs. It can perform human gait walking motion. Elements in Fig. 1 are described as: O—actuator joint (rotation joint); A—exoskeleton hip joint; B—exoskeleton knee joint; C—exoskeleton ankle joint; AB—exoskeleton human-like thigh link; BC—exoskeleton human-like shank link; CF—exoskeleton human-like foot link; OD—actuator link for leg exoskeleton mechanism; DE—middle link for leg exoskeleton mechanism; EF—lower link for leg exoskeleton mechanism; E—linear slider; There are two spring systems in this mechanism, which is: S1—torsional spring S2—linear spring In order to limit the rotation angle in a reasonable range to prevent injured like human being, three joint limitations are set as: AL—rotation limitation in hip joint; BL—rotation limitation in knee joint; CL—rotation limitation in ankle joint; The proposed mechanism is a closed chain linkage with underactuated joints. It can rely on the sole driving source on hip joint to drive the motion in three joints at hip, knee and ankle. Thus, the mechanism can greatly reduce the number of drivers and the weight of exoskeleton. When the underactuated joint works in the foot landing process, the energy storage elements (spring system) can absorb the gravitational potential energy of the wearable exoskeletons. The exoskeleton weight’s gravitational potential energy can be transformed into elastic potential energy when it is standing on the ground. In the process of leg swing, spring systems release the stored elastic energy and provide part of the energy for the exoskeleton leg lifting and swing. The energy consumption can be reduced when the leg in the swing and standing cycle, and the energy efficiency is improved. The basic elements of mechanical structure are joints, links and a slider, so it can be designed according to the physiological structure as human legs. Some design solutions with suitable dimension in lengths and angles can meet the requirements of the walking rehabilitation assistance treatment. There are joint rotation limitations in the exoskeleton joints. In the standing situation, the knee and ankle joint limitation will work. Thus, the linkage mechanism can perform high rigidity and stably when exoskeleton stands on the ground. There are two spring elements set in the thigh and

76

Y. Shuangji and G. Carbone

shank links, as shown in Fig. 1b). In order to constrain the kinematics in underactuated joints, a torsional spring and a linear spring are designed. These torsion spring and linear spring will produce elastic deformation and damping in the spring tension and compression process. Thus, the springs can reduce the impact force and vibration on leg when the gait motion transfers from swing to standing.

4 Performance Analysis via Simulation The design parameters of a new underactuated mechanism for this leg exoskeleton design solutions are shown in Fig. 1a). In order to show the relationship of each links, the mechanism link vectors closed chain are shown in Fig. 1b). The Li (i = 1, …, 8) are the length of each links in the mechanism, The hi (i = 1, …, 8) are the angle of each links relative to coordinate axis X. hS1 is the angle of the torsional spring, S2 is the length of the linear spring. Link sizes of the designed new leg exoskeleton mechanism are obtained by kinematics tests in 3D software and listed in Table 1. Values in Table 1 have been obtained by implementing a dimensional synthesis procedure to fulfill the desired walking pattern with the desired overall dimensions.

Table 1. Main parameters for the underactuated leg exoskeleton mechanism in Fig. 1. Design parameters 1 2 3 4 5 6 7 8 122.0 550.0 260.0 678.0 283.0 515.0 413.0 74.0 Li/(mm) hi/(°) 7.3 266.5 352.0 250.5 260.8 265.3 279.0 328.0

Numerical simulation for the two legs exoskeleton mechanism walking is run in SolidWorks Motion environment. It has been computed with the input angle initial value of h3 as h3(0) = 352°, and the initial values of spring parameters are assumed as some other design parameters are: hS1 = 78.5°; S2 = 135 mm; hS2 = 270°; The value of coefficients k1, k2, c1, c2 of springs and dampers in simulations have been determined with values k1 = 25  10−3 Nm/deg, k2 = 15  103 N/m, c1 = 20  10−3 Nms/deg and c2 = 5103 Ns/m, respectively. The walking ground is flat and the material of contact surface between shoes and ground is set as rubber. For the disabled people rehabilitation walking exercise function, the movement of the waist in horizontal direction is restricted by two frame bars as shown in Fig. 2. An actuated cyclic rotating motion is located at link L3, whose input motion formulation of link L3 for left and right leg is given by  Left leg: Dh3L ðtÞ ¼

A sinð2p=TÞt 0

if sinð2p=TÞt  0; if sinð2p=TÞt\0;

ð1Þ

Design and Simulation of an Underactuated Exoskeleton Mechanism

 Right leg: Dh3R ðtÞ ¼

0 A sinð2p=TÞt

if sinð2p=TÞt  0; if sinð2p=TÞt\0;

77

ð2Þ

where t is the time variable, A is the maximum swing rang for link L3. In order to move the linkage mechanism smoothly, sine function is selected in Eqs. (1) and (2) as the movement function with the features of cyclic and intermittent motion. The variable T in Eq. (1) and (2) is the cyclic time for the movement. other higher order cyclic motions might be considered if additional boundary conditions are available. Considering the human walking cycle time, set the cyclic time T = 2 s in Eq. (1) and (2) for this simulation. It means the exoskeleton walking process spends 1 s by a leg per step. Considering the human thigh rotated range in walking, set the maximum swing rang of link L3 A = 30° in Eq. (1) and (2). It means the input link L3 will rotate 30° and back in a walking cycle. The input movement curves for link L3 for left leg are shown as Fig. 3a). The simulation lasts 12 s. The walking simulation involved gravity and the inertial force of the underactuated leg exoskeleton. The proposed exoskeleton mechanism can perform human-like walking gait with suitable kinematics design in Table 1. The joints’ rotation in hip, knee and ankle joints in the mechanism coupled with the linkages can carry out humanlike walking motions by two legs. The link AB, which is thigh link of the exoskeleton, is 0° while it is standing. Link AB rotated about 35° forward in leg swing forward phase. When the leg is landing, link AB rotated about 7° backward for the thigh backward swing. The simulation result in Fig. 3b) express link BC (which is shank link of the exoskeleton) rotated angle according link AB. The knee rotation angle begins from 0° while the leg is standing as shown in Fig. 4a). Link BC rotated about 56° according link AB in thigh link swing forward process. When the thigh link swings backward the angle between link AB and link BC become small. The minimum angle is −8° before foot landing, which reached the angle limitation of knee joint. The knee joint angle increases slowly while the gait is from standing phase to back swing phase, as shown in Fig. 4a). The ankle joint rotated range is 20° totally which is from −8° to 12° in the walking cycle.

Fig. 2. Two legs exoskeleton mechanism in a simulation environment.

78

Y. Shuangji and G. Carbone 40 H i p jo in t r o t a t e d a n g le ( d e g )

I n p u t r o t a t io n ( d e g )

30 20 10 0 0.00 1.00

2.00 3.00

4.00

5.00

6.00 7.00 Time (sec)

8.00

30 20 10 0 -10 0.00

9.00 10.00 11.00 12.00

a)

1.00

2.00

3.00

4.00

5.00

6.00 7.00 Time (sec)

8.00

9.00 10.00 11.00 12.00

b)

K n e e jo in t r o t a t e d a n g le ( d e g )

60 50 40 30 20 10 0 -10 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00 12.00 Time (sec)

a)

A n k le j o in t r o t a t e d a n g l e ( d e g )

Fig. 3. Simulation data: a) Input movement at link L3 on left leg; b) rotation angle in hip joint.

14 10 5 0 -5 -10 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00 12.00 Time (sec)

b)

Fig. 4. Simulation results: a) Rotation angle in knee joint; b) Rotation angle in ankle joint.

The ankle joint limitation is reached when the leg is swing forward and backward as shown in Fig. 4b). Figure 5 a) and b) show the motion of torsional spring and linear spring in a walking cycle, respectively. Because the first step is begin from two legs standing state, the springs performance are different in the first second by comparing with their performance in other time. The torsional spring angle rotated from −15° to 13° and the curve is smooth. The initial length of linear spring is 150 mm. It is compressed to 140 mm and extended to 157 mm in walking cycle. It can be seen in Fig. 5 that there are small disturbance in the curve of linear spring motion at the moment of foot landing on the ground. It means the contact process has some vibration between foot and ground. Further investigations can aim at a careful selection of spring coefficient and damping for reducing this effect. Figure 6 shows the horizontal directional position (in X axis) of foot mass center in walking cycle. The foot performs step by step moving in the walking cycle simulation. The step length can be carried out in a suitable length by kinematic design. The waist part of two legs exoskeleton mechanism is limited in the horizontal bars. Figure 6 b) shows the moving speed of the waist part in horizontal direction. Because the leg

Design and Simulation of an Underactuated Exoskeleton Mechanism

79

160

15

L in e a r s p r i n g m o t io n ( m m )

T o r s i o n s p r in g r o t a t e d a n g le ( d e g )

exoskeleton is walking from standing, the waist moving speed is accelerated from 0 mm/s. The waist moving speed becomes to a stable cycle when the walking cycle become regularly. The leg exoskeleton mechanism can walk in a constant speed in this regular cyclic process. Figure 7 shows the reaction force on the foot by the ground. It can be seen that the contact force at the moment of foot landing is more than 1500 N. Foot landing and contact the ground is a collision process in very short time, thus the impact force is very large at that moment. The contact force is less than 100 N when the foot is standing on the ground, which is much smaller than in usual landing phase. The above results preliminarily demonstrate the effectiveness of the proposed exoskeleton for lower limb exercising. Further investigations will be required in future aiming at the design and construction of a prototype. This should include more simulations with a more detailed simulation model.

10

0

-10

-20 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00 12.00 Time (sec)

155 150 145 140 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00 12.00 Time (sec)

a)

b)

3463 2619 1774 929 85 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00 12.00 Time (sec)

a)

W a i s t s p e e d i n X a x is ( m m / s e c )

F o o t p o s it io n in X a x i s ( m m )

Fig. 5. Simulation results: a) Torsion spring rotation angle; b) Linear spring motion.

600 500 400 300 200 100 0 -100 0.00 1.00 2.00 3.00 4.00 5.00 6.00 7.00 8.00 9.00 10.00 11.00 12.00 Time (sec)

b)

Fig. 6. Simulation results: a) Foot position in walking (in X axis); b) Waist speed in horizontal direction (X axis).

80

Y. Shuangji and G. Carbone

Contact force (newton)

2000 1500 1000 500 -0 0.00

1.00

2.00

3.00

4.00

5.00

6.00 7.00 Time (sec)

8.00

9.00

10.00 11.00 12.00

Fig. 7. Simulated contact force on the foot by the ground.

5 Conclusion In this paper we have discussed the design requirements for the low-cost easy operation leg exoskeleton mechanism. An underactuated mechanism of leg exoskeleton based on closed chain linkage bar is proposed. The proposed leg exoskeleton mechanism has 3 DOFs with only 1 actuator. It can simulate human hip joint, knee joint and ankle joint and the normal physiological structure of thigh, shank, foot size. The rotation range in each joint can be limited by joint limitation design. Simulation of walking is carried out to characterize the solution performance. The simulation results show the proposed underactuated leg exoskeleton mechanism can obtain feasible human like walking cycle gait with design results.

References 1. Kazerooni, H.: The Berkeley lower extremity exoskeleton project. In: Experimental Robotics IX, pp. 9–15. Springer, Heidelberg (2006) 2. Phoenix [EB/OL]. http://www.suitx.com/phoenix.html. Accessed 27 Dec 2019 3. Copilusi, C., Ceccarelli, M., Carbone, G.: Design and numerical characterization of a new leg exoskeleton for motion assistance. J. Robotica 33(5), 1–16 (2015) 4. Carbone, G., Ceccarelli, M.: Legged robotic systems. In: Cutting Edge Robotics. InTech (2005) 5. Tao, L., Ceccarelli, M.: Design and simulated characteristics of a new biped mechanism. Robotica 33(7), 1–21 (2014) 6. Liu, J., Tan, M., Zhao, X.: Legged robots—an overview. Trans. Inst. Measur. Control 29(2), 185–202 (2007) 7. Walsh, C.J., Paluska, D., Pasch, K., et al.: Development of a lightweight, underactuated exoskeleton for load-carrying augmentation. In: IEEE International Conference on Robotics and Automation, pp. 3485–3491. IEEE (2006) 8. Lim, H.O., Takanishi, A.: Mechanism and control of anthropomorphic biped robots. In: Mobile Robotics, Moving Intelligence. InTech (2006) 9. Wang, M., Ceccarelli, M.: Design and simulation of walking operation of a Cassino biped locomotor. In: New Trends in Mechanism and Machine Science, pp. 613–621. Springer International Publishing (2015)

Design and Simulation of an Underactuated Exoskeleton Mechanism

81

10. Giesbrecht, D.: Design and optimization of a one-degree-of freedom eight-bar leg mechanism for walking machine, Ph.D. thesis. The University of Manitoba (2010) 11. Mcgeer, T.: Passive walking with knees. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 3, pp. 1640–1645. IEEE Xplore (1990) 12. Cherry, M.S., Choi, D.J., Deng, K.J., et al.: Design and fabrication of an elastic knee orthosis —preliminary results. In: International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, ASME 2006, pp. 565–573. American Society of Mechanical Engineers (2006)

Safety Related Devices and Applications

Remote Center of Motion for Redundant Robotic-Assisted Ultrasound Guided Regional Anesthesia Mohammad Alkhatib(B) , Cyril Novales, Laurence Nouaille, Adel Hafiane, and Pierre Vieyres Laboratoire PRISME EA 4229, Universit´e d’Orl´eans, 18000 Bourges, France [email protected], {cyril.novales,laurence.nouaille,pierre.vieyres}@univ-orleans.fr, [email protected]

Abstract. Ultrasound-guided regional anesthesia (UGRA) becomes a standard procedure in surgical operations and contributes to pain management; it offers the advantages of the needle and targeted nerve detection and provides the visualization of regions of interest such as anatomical structures. In UGRA, the remote center of motion (RCM) constitutes an essential issue as the anesthetist has to manipulate a needle inside the human body. However, RCM imposes a very challenging task, where it is important to ensure that the needle should move within the constraints of the insertion point in order to prevent patient harm. To respond to this need, this paper proposes a control framework for robot-assisted UGRA for physical human-robot collaboration using 7 degrees of freedom robot manipulator (Franka Emika). This paper shows a geometric method computing the intended robot’s end-effect position with respect to the RCM constraints. This method helps the anesthetist to execute a more sophisticated motion within the patient’s body with high accuracy. Keywords: Regional anesthesia · Needle insertion kinematics Medical robotics · Remote center of motion constraints

1

·

Introduction

Regional anesthesia (RA) is performed to block the sensation of pain in a specific region of the body by stopping the nerve impulses connection between that region and the central nervous system. Nowadays RA is a well-known procedure in many operating rooms. RA is performed by the anesthetist to reduce pain scores, to facilitate earlier hospital discharge, and to improve postoperative prompt recovery. Traditionally, this technique is performed with needle blind guidance which increases the risk of block failure, nerve trauma, and local anesthetic This work is part of the DANIEAL 2 project supported by Centre Val de Loire Region (France) grant 2016 – 00108375. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 85–92, 2020. https://doi.org/10.1007/978-3-030-48989-2_10

86

M. Alkhatib et al.

toxicity [19]. Ultrasound (US) imaging is a relatively low-cost, no radiation, and real-time technique that offers a huge performance impact on the practice of UGRA in the operating room [2,7,12]. However, the UGRA procedure requires a high level of anesthetist’s skills, mainly due to the challenge of US low-resolution image and the needed experience in controlling, tracking and estimating the needle and nerve positions [13,20]. The interaction pivot point between the needle and the skin of the patient is named the insertion point or the trocar in medical term. This point restricts the anesthetist to control the needle to only four degrees-of-freedom (DoF) motion: three rotational DoF and one translational DoF. Robots are already used in medical settings and assistance and show their huge impacts [10]. Robotic-assisted UGRA is a solution that can provide the anesthetists with a better control of the needle orientation and trajectory around the insertion point, and avoid nerve trauma. Using such a robotic assisted procedure, and as the robot end-effector (i.e. the needle) must pass through this small insertion point, its movements are limited to a translation along its axis and rotations at the insertion point [8]. The manipulator motion is then constrained with respect to that point on the end-effector known as Remote Center of Motion (RCM) [9]. The basic action of robotic assistants in UGRA can be concluded as to move the needle through the insertion point and to generate a remote center of motion (RCM) as shown in Fig. 1. In theory, a high number of DoF serial robot provides a large operation workspace and dexterity. However, keeping the end-effector working through an RCM point and limiting its motions is a very challenging task which demands an accurate control strategy [4,18]. In the literature, various methods have been proposed to address RCM constraints such as Jacobian with constrained quadratic optimization [6], isotropy-based kinematic optimization [11], constrained Jacobian with Lie algebra [16], gradient projection approach [1], dual quaternionbased kinematic [14]. The previous researches discussed the modeling of trocar kinematics only or combined with trajectory following. Very few of the control strategies reported in the literature are concerned with the problem of RCM constraints and their geometry in the workspace. Mayer et al. proposed a trocar modeling with Euler angles representation for heart surgery [15]. In [3], the authors proposed using geometric constraints with stereo visual servoing for controlling the robot position from point-to-point and extended Jacobian solution for manipulating serial end-effector. The main contribution of this article is formulating a new cartesian control method to maintain RCM constraints while offering a more accurate end-effector position. The method controls the motion of the rigid needle and it describes the end-effector position with respect to the RCM constraints for a teleoperated UGRA control using a 7-DoF serial robot. It provides a safe and flexible solution for the robot and the medical staff in a shared workspace where it explicitly models the rotational and translational motions at the RCM point.

Remote Center of Motion for Redundant Robotic-Assisted UGRA

87

Fig. 1. UGRA motion around the insertion point (Pinsertion - RCM ), where the tool frame is the end-effector position and rotation. As a result, the needle can only move in 4-DoF: Rx , Ry , and Rz are the rotations around x, y and z-axis respectively, and Tz is the translation along the z-axis.

The paper is structured as follows: Sect. 2 details the proposed needle insertion kinematics, followed by experimental setup, results and discussion in Sect. 3, and conclusions are given in Sect. 4.

2

Needle Insertion Kinematic

In UGRA, the needle must have only 3-DoF for its orientations (i.e., pan, tilt, and spin rotations) corresponding to the roll, pitch, and yaw angles around the RCM point. Moreover, 1-DoF is needed for the axial translation corresponding to the depth of insertion as shown in Fig. 1. Hence, to simplify the control strategy for the robot and to help the kinematic design, we choose to define the movements of the needle with respect to these four displacements. Different strategies exist to control the needle movements inside the patient, where the most comfortable way would be the operational space using the Cartesian representation. The principal objective of this section is to propose an explicit and yet generic expression of the intended tool frame position as shown in Fig. 1. Hence, the robot command and movements are controlled using the tool frame position. The tool frame control requires guaranteeing the RCM constraints. Therefor, we choose to perform tool frame motion along a sphere centered at RCM point as shown in Fig. 2. Thus, the tool frame is moved from a “start” position on the sphere to a “final” position on the same sphere with 2D tangential constraints. The start position of the tool frame is found using the direct kinematics model assisted by the robot built-in library. This position is composed of a 3D vector of translations and a 3 × 3 rotation matrix with respect to the base frame. Let’s define the starting point of the end-effector as 3D translations (ST = [ST x ST y ST z ]T ) and 3 × 3 rotation matrix (SR ). To find the RCM position (sphere center with a radius R as shown in Fig. 2), first we have to retrieve from the rotation matrix the three rotation of the end-effector (Roll φ, Pitch θ, and Yaw ψ): (1) φ = tan−1 (SR21 /SR11 )

88

M. Alkhatib et al.

Fig. 2. A Spherical representation of end-effector position. Where α and β are the sphere representation angles and R is the radius of the sphere.

 2 2 ) θ = tan−1 (SR31 / SR32 + SR33

(2)

ψ = tan−1 (SR32 /SR33 )

(3) T

Using these rotations, the RCM position ([RCMx RCMy RCMz ] ) is found by: RCMx = ST x + (cos(φ)sin(θ)cos(ψ) + sin(φ)sin(ψ))R

(4)

RCMy = ST y + (sin(φ)sin(θ)cos(ψ) − cos(φ)sin(ψ))R

(5)

RCMz = ST z − cos(θ)cos(ψ)R

(6)

As shown in Fig. 1 and Fig. 2, the tool frame (end-effector) movements from one position to another (such as start position to final position in Fig. 2) depend on the applied rotations on the end-effect (Rx and Ry ). Hence for finding the final position (translation), the two spherical rotations are: π − Rx (7) 2 π β = − Ry (8) 2 We now have to consider how to compute the desired final position. Thanks to the spherical representation and having the RCM position with the two rotation angles, the final position translation coordinates (FT = [FT x FT y FT z ]T ) are: α=

FT x = RCMx + cos(α)R

(9)

FT y = RCMy + cos(β)R

(10)

FT z = RCMz + sin(α)sin(β)R

(11)

To guarantee the RCM constraints, the tool frame position should always face the RCM point (the tool frame zaxis should cross the RCM point); to comply with this requirement, the final rotation should respect the RCM position.

Remote Center of Motion for Redundant Robotic-Assisted UGRA

89

Thereby, the rotation matrix of the final end-effector position (FR ) is represented by: (12) FR = [Xaxis Yaxis Zaxis ]3×3 where Xaxis , Yaxis , and Zaxis are the axes that represent the final tool-frame orientation. To find these axes, if we can initialize a 3D coordinate where one axis points toward the RCM position, it will simplify the problem to use axis perpendicularity to find the others. Using this property, these axes are found by: Zaxis = normalize(ST − RCM )

(13)

Xaxis = normalize((0, 1, 0) − Zaxis )

(14)

Yaxis = Zaxis × Xaxis

(15)

where × is the cross product and each axis represented as a 3D vector in space. All the introduced above allows to control the 2 rotation constraints (Rx and Ry ). For the 3rd rotation (rotation on the z-axis), the end-effect self-rotation around its axis is just applying the Rz rotation directly on the end-effect rotation matrix. To control translation on the z-axis, the needle insertion is equivalent to the decrease of the sphere radius. The position and rotation of the end-effector depend on the RCM position, where the translation on its z-axis will produce the end-effector position changes but keeping the RCM position in space. The orientation of the end-effector will be the same but the new translation position of the end-effector after applying a translation Tz on its z-axis will be: FT x = ST x + Tz (RCMx − ST x )

(16)

FT y = ST y + Tz (RCMy − ST y )

(17)

FT z = ST z + Tz (RCMz − ST z )

(18)

and the radius (R) will be updated to:  R = R ± (FT x − ST x )2 + (FT y − ST y )2 + (FT z − ST z )2

(19)

where ± is + or − operator which depends on the sign of Tz (insertion or extraction of the needle). With the orientation and the position of the end-effector, and using the inverse kinematics model provided by the robot built-in library, the positions of the joint angles of the robot can be defined. The redundancy of our 7-DoF robot in front of the 4-DoF necessary for the robotic-assisted UGRA is not an issue. On one hand, the alignment singularities are already well managed by the built-in Franka library; on the other hand, we have chosen an initial configuration of the robot that allows us to stay in the same configuration (as our workspace is reduced to 20 × 20 × 20 cm3 ), and where the dexterity is maximum.

M. Alkhatib et al.

Rviz display

Real robot

90

(a) start position

(b) Rotations

(c) Rotation & translation

Fig. 3. RCM results. Where (a) is the robot starting position, (b) is the robot position after applying different rotations, and (c) is the robot position after applying different rotations and z-translations. The laser points in the first row and the red sphere in the second row represent the RCM position.

3

Experimental Setup, Results and Discussion

The experiments were conducted using a 7-DoF Franka Emika Panda robotic arm [5]. The Panda arm operates using the Robotic Operating System (ROS), a dedicated software framework for robot software development [17]. The proposed method was implemented as a ROS node where it publishes/subscribes data with other nodes via ROS messages on the ROS topics. Franka Emika research provides a built-in library (libfranka), and franka-ros that integrates libfranka into ROS and ROS control. Libfranka allows the robot to be controlled in joint or cartesian space and provides the direct and inverse kinematics models. The communication between this interface and the robot is made possible thanks to the Franka Control Interface (FCI), which provides the current robot status and enables its direct control with an external workstation PC connected via local Ethernet at a communication rate of 1 kHz. In these experiments, the robot was assisted with a tool that we designed using a 3D printer to hold the needle. To control the robot position and to avoid any discontinuity errors, the applied rotations and translation should be reachable and do not exceed the predefined limits. Therefore, while working at 1 kHz, the planning of smooth motion is essential. The considered smooth motion is an important key for controlling the robot, where the movement changes gradually until reaching the maximum predefined velocity. In this work, where the needle in the UGRA procedure moves slowly and

Remote Center of Motion for Redundant Robotic-Assisted UGRA

91

precisely, the position of the tool frame was controlled in the operational space 2 with a maximum velocity of 3 cm/s and acceleration of 2 m/s . In this paper, we used position accuracy as it is the major metric commonly used to evaluate RCM methods in medical robotics. Basically, position accuracy is the Euclidean distance between a given insertion point and the computed RCM point found by the proposed method. Here, the proposed method is validated with this metric; and it is validated using 2 orthogonal laser beams that point out at the RCM point and a sphere in Rviz visualization as shown in Fig. 3. Figure 3 shows an example of applying different rotations and translation on the Franka robot. Where Fig. 3a shows the robot at the starting position, Fig. 3b shows the robot position after applying different rotations for 10 s, and Fig. 3c shows the robot position after applying different rotations and translations for another 10 s. For these experiments and after 20 s of movements, the position accuracy of RCM constraints has been computed. These experiments have been repeated and the maximum error is less than 1 mm. In the real situation of UGRA and because of the softness and elasticity of the skin tissue, the needle has a movement of up to 5 mm around the insertion point. For that, the results suggest that with a given configuration control, the manipulator has a highly accurate performance when it is operating with respect to RCM constraints.

4

Conclusions

In this paper, we addressed a challenging problem in robotic-assisted UGRA which is needle insertion control within the so-called trocar point. We have presented an experimental system with the remote center of motion constraints. It enables full Cartesian control of the robot end-effector by implementing the needle insertion kinematics. The proposed method implements using ROS and a 7 degrees of freedom Franka Emika robot. To evaluate the accuracy of the system, a performance metric is used which is the Euclidean distance between the remote center of motion point and the insertion point. The proposed method is accurate to find the exact position and rotation of the end-effector with an error less than a 1 mm while maintaining the needle insertion kinematics. We are currently working on further control improvements of the UGRA robotic platform by developing force control and force feedback haptic control and that’s why we 2 choose commanding at 1 kHz and using 2 m/s acceleration.

References 1. Azimian, H., Patel, R.V., Naish, M.D.: On constrained manipulation in roboticsassisted minimally invasive surgery. In: 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics, pp. 650–655. IEEE (2010) 2. Bloc, S., Rontes, O., Mercadal, L., Delbos, A.: Block success rate: a question of target and definition. Reg. Anesth. Pain Med. 38(6), 553 (2013) 3. Dahroug, B., Tamadazte, B., Andreff, N.: 3D path following with remote center of motion constraints. In: ICINCO (1), pp. 84–91 (2016)

92

M. Alkhatib et al.

4. Dombre, E., Michelin, M., Pierrot, F., Poignet, P., Bidaud, P., Morel, G., Ortmaier, T., Sall´e, D., Zemiti, N., Gravez, P., et al.: Marge project: design, modeling and control of assistive devices for minimally invasive surgery. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 1–8. Springer, Heidelberg (2004) 5. Franka Emika: Panda (2018). https://www.franka.de 6. Funda, J., Taylor, R.H., Eldridge, B., Gomory, S., Gruben, K.G.: Constrained cartesian motion control for teleoperated surgical robots. IEEE Trans. Rob. Autom. 12(3), 453–465 (1996) 7. Gray, A.T.: Ultrasound-guided regional anesthesiacurrent state of the art. Anesthesiol. J. Am. Soc. Anesthesiol. 104(2), 368–373 (2006) 8. Johnson, C.R., Barr, R.C., Klein, S.M.: A computer model of electrical stimulation of peripheral nerves in regional anesthesia. Anesthesiol. J. Am. Soc. Anesthesiol. 106(2), 323–330 (2007) 9. Kuo, C.H., Dai, J.S.: Kinematics of a fully-decoupled remote center-of-motion parallel manipulator for minimally invasive surgery. J. Med. Dev. 6(2), 021008 (2012) 10. Lanfranco, A.R., Castellanos, A.E., Desai, J.P., Meyers, W.C.: Robotic surgery: a current perspective. Ann. Surg. 239(1), 14 (2004) 11. Locke, R.C., Patel, R.V.: Optimal remote center-of-motion location for roboticsassisted minimally-invasive surgery. In: Proceedings 2007 IEEE International Conference on Robotics and Automation, pp. 1900–1905. IEEE (2007) 12. Marhofer, P., Chan, V.W.: Ultrasound-guided regional anesthesia: current concepts and future trends. Anesth. Analg. 104(5), 1265–1269 (2007) 13. Marhofer, P., Willschke, H., Kettner, S.: Current concepts and future trends in ultrasound-guided regional anesthesia. Curr. Opin. Anesthesiol. 23(5), 632–636 (2010) 14. Marinho, M.M., Bernardes, M.C., B´ o, A.P.: A programmable remote center-ofmotion controller for minimally invasive surgery using the dual quaternion framework. In: 5th IEEE RAS/EMBS International Conference on Biomedical Robotics and Biomechatronics, pp. 339–344. IEEE (2014) 15. Mayer, H., Nagy, I., Knoll, A.: Kinematics and modelling of a system for robotic surgery. In: On Advances in Robot Kinematics, pp. 181–190. Springer, Heidelberg (2004) 16. Pham, C.D., Coutinho, F., Leite, A.C., Lizarralde, F., From, P.J., Johansson, R.: Analysis of a moving remote center of motion for robotics-assisted minimally invasive surgery. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1440–1446. IEEE (2015) 17. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: ROS: an open-source robot operating system. In: ICRA workshop on open source software, Kobe, Japan, vol. 3, p. 5 (2009) 18. Schneider, O., Troccaz, J.: A six-degree-of-freedom passive arm with dynamic constraints (PADyC) for cardiac surgery application: preliminary experiments. Comput. Aided Surg. 6(6), 340–351 (2001) 19. Tsui, B.C., Suresh, S.: Ultrasound imaging for regional anesthesia in infants, children, and adolescentsa review of current literature and its application in the practice of extremity and trunk blocks. Anesthesiol. J. Am. Soc. Anesthesiol. 112(2), 473–492 (2010) 20. Woodworth, G.E., Chen, E.M., Horn, J.L.E., Aziz, M.F.: Efficacy of computerbased video and simulation in ultrasound-guided regional anesthesia training. J. Clin. Anesth. 26(3), 212–221 (2014)

Reference Frame Identification and Distributed Control Strategies in Human-Robot Collaboration Alberto Borboni1(&), Giuseppe Carbone2, and Nicola Pellegrini1 1

Department of Mechanical and Industrial Engineering, Università degli Studi di Brescia, Brescia, Italy {alberto.borboni,nicola.pellegrini}@unibs.it 2 Department of Mechanical, Energy and Management Engineering, Università della Calabria, Rende, Italy [email protected]

Abstract. The reference frame definition is a significant postulate for the robot motion planning process. Reference frame selection can be classified as allocentric, egocentric and route-centric. The identification of a reference frame affects the complexity of measuring and implementing the controlling variables. The work reports this topic by referring specifically at the human robot collaboration in a workstation. Accordingly, this work proposes a methodology for the evaluation of the reference frame in a human robot communication aimed at enabling, effective and safe distributed control strategies. Keywords: Reference frame


Human-Robot interaction
Distributed control

1 Introduction The reference frame definition is a central assumption for the robot motion planning process. It admits multiple solutions with practical implications in many research domains: mobile robotics [1], cooperative and collaborative applications [2, 3], deformable mechanical systems [4], finite element models [5, 6] and haptic interfaces [7]. In this work, authors focus at the reference frame definition considering the interaction between human and robot in collaborative environment. A human being adopts three types of spatial reference frame to describe the unstructured movement: egocentric, allocentric and route-centric [8, 9], based on the reference frame origin that is fixed on the human observer, permanent on an external entity or connected on the route, respectively. The axes of the egocentric reference frame are along the vertical, longitudinal, lateral directions. The axes of the allocentric reference frame depend on the perception of the entity associated with this reference [10]. In the route-centric reference frame, an axis is corresponding to the route. During the motion gestures, the human changes from a reference frame to another according with real-time collision-free analysis. As consequence, the temporal definition of the adopted reference frame is highly discontinuous and non-linear. In addition, human represents the space according to his/her physical dimensions through a cognitive map © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 93–102, 2020. https://doi.org/10.1007/978-3-030-48989-2_11

94

A. Borboni et al.

that can be represented in a mixed geometrical-topological space, where reference frames are defined with not-homogeneous metrics. Similar concepts have been implemented in industrial and service robots by their human designers, i.e. in [11] researches the robot perceives an obstacle, it fixes an allocentric reference frame on the barriers to seeking appropriate trajectory. In [3], a perceptive variable has been defined to describe the functional task of the robot in a non-time-variant-space according to the reference frame. In fact, a set of spatial and temporal variables is not the optimal solution to describe the manipulation task; in [12], the orientation of the reference frame depends on a kinematical or on a dynamical variable. The present work is devoted to discuss the selection and the efficacy of a reference frame identification. In addition, a distributed control strategy is defined as a procedure for safety assurance of the human in collaborative applications. The literature review on reference frame identification, is characterized in Sect. 2. In Sect. 3, the problem formulation of human-robot distributed control is described. In Sect. 4, a case study related 2D-sharing strategies is presented. Discussion and conclusions about findings, limitations and possible future works are provided in Sect. 5, 6.

2 Related Work: Reference Frame Identification The reference frame selection approaches developed for research and industrial environment have been investigated. The main design specifications have been summarized according to the following criteria: (1) Spatial Reference Frame (Allocentric, Egocentric, Route-centric), (2) Application, (3) Main purpose and (4) Key characteristics that progress the innovation beyond the current state. Table 1 lists all the above information. In investigated scenarios, different sensors feedback are presented ranging from medical, to surgical and autonomous mobile robot, for a spatial mapping, for this reason, the signals management is central. Use of an adequate signal loop is the first requirement; moreover, the interface communication between human and robot needs to be accurately defined assuring coherent interpretation, in order to avoid any safety impact. Further approaches as vision-based techniques, fuzzy logic, artificial neural mapping, reactive navigation techniques, biologically inspired techniques, geometrical path planning, are used individually or combined to find the optimal collision-free path. In teleoperation (industrial or medical), the human focuses at the interface control while driving the physical end-effector. Operator training and precise instructions support the subject to complete a task without performance degradation, while the usage of a robot could enable nominal predefined task precision. Finally, the disturbance features and their representation are directly related to the complexity of model and to the risk of task accomplishment. The rudimentary approach is to define the workspace, as a structured invariant area. This assumption is not feasible. Same result by neglecting time-variant variables or limiting the access at human-robot in workstation space. All previous aspects support the online reference transformation and reference transfer strategies in global environment, in particular for collaborative applications.

Reference Frame Identification and Distributed Control Strategies

95

Table 1. Comparison of selected reference frame identification, application and usage. Reference Application frame Allocentric Autonomous highperformance flight

Egocentric allocentric

Autonomous mobile robot

Egocentric allocentric

Autonomous mobile robot

Egocentric allocentric

Flexible multibody

Allocentric Surgical medical Egocentric Human cognition

Egocentric

Human cognition

Routecentric

Autonomous mobile robot

Allocentric Prothesis medical

Allocentric Prothesis medical

Allocentric Rehabilitative medical

Main purpose

Key characteristics

Ref.

[I] Decouple the trajectory Planning long, dynamically optimization feasible, time-optimal trajectories in the presence of wind. Decouples a path optimization in ground frame and velocity optimization in airframe Human/robot Definition of perceptive action [2] distributed control reference is directly relevant to the measured sensory outputs [3] Human/robot Comparison between distributed control perceptive frame and time based reference frame [4] Dynamic modeling of Study of interaction between floating frame and absolute interconnected rigidnodal coordinate frame elastic robotic reference manipulators Graphical user interface Development of the frame of [7] reference transformation tool [8, 9] Humans perception of Navigation strategies depend spatial relations on the agent’s confidence (reference frame and sensor information) [10] Humans perception of The location is influenced by spatial relations the animacy of the reference object Optimal path planning Planar path planning based on [11] geometrical method for the navigation of mobile robot consisting of elliptical obstacles [12] Autonomous levelling Implementation of self adjusting wrist able to maintain hand’s orientation relative to the ground reference frame [13] Autonomous levelling Artificial knees that adapt stiffness parameters to provide optimal performance at each stage of walking Autonomous levelling Camera stabilization for robotic [14] rehabilitation devices (continued)

96

A. Borboni et al. Table 1. (continued)

Reference frame Routecentric

Application

Main purpose

Key characteristics

Autonomous mobile robot

Obstacle avoidance

Routecentric

Autonomous mobile robot

Routecentric

Surgical medical

Egocentric allocentric

Collaborative formulation

Review of parameter used to classify obstacle avoidance methods/algorithms Obstacle avoidance Robot is un-informed of the arena beforehand; hence the path planning of a robot is done live from sensor data Component for tracking Validation of an open-library systems for passive optical tracking systems Controller able to work Movement control in action in action oriented frame oriented reference frames

Ref. [15]

[16]

[17]

[18]

3 A Procedure to Define Proper Reference Frame for Human-Robot Interaction In this section, authors describe the problem of human-robot distributed control. In order to develop a robust procedure, the key aspect is to create a common motion reference that can be agreed by the humans and the robot involved. Human gesture is determined in a variety of task-oriented reference frames. In contrast, robotics movement control is expressed as the world frame, or a robot fixed base frame. The proposed concept introduces a perceptive action reference, a parameter that is directly related to the measured sensor outputs and the manipulation task. In addition, unstructured working space has to be considered due to the presence of unknown fixed obstacles and unexpected moving object/human during robot motion. A P configuration of robot, human, environment and task can be described as in (1), where Sri and Rri are a generic robot space and reference; Stj and Rtj are a task space and reference and Shk and Rhk are a human space and reference.


Sr1 ; Robot : Rr ;
t1 S1 ; Task : Rt ; (1 Sh1 ; Human : Rh1 ;

Sr2 ; Rr2 ; St2 ; Rt2 ;

. . . Sri ; . . . Rri ; . . . Stj ; . . . Rtj ;

Sh2 ; . . . Rh2 ; . . .

Shk ; Rhk ;

. . . Srn . . . Rrn . . . Stm . . . Rtm . . . Shp . . . Rhp

ð1Þ

Reference Frame Identification and Distributed Control Strategies

97

The P configuration can be described with a set of coordinates xri , xtj or xhk , in a space Sri , Stj or Shk with respect to a reference frame Rri , Rtj or Rhk . To perceive the P configuration and to provide the decision, a generic scalar measure dri;ii , dtj;jj , dhk;kk needs to be defined, respectively in a robot, task or human space, as shown in (2):   r dr dri;2 . . . dri;ii . . . dri;nn  xi ; Ri ! i;1 xj ; Rtj ! dtj;1 dtj;2 . . . dtj;jj . . . dtj;mm   xk ; Rhk ! dhk;1 dhk;2 . . . dhk;kk . . . dhk;pp

ð2Þ

Start

Identification physical quantities to be measured Select M [1, …, N] sensors

Measure i-th quantity

Define j-th space

Set p-th frame via Eq. (4) Perform computation Eq. (2) Adjacent discriminated?

p-th frame validated

End

Fig. 1. A flow-chart of the proposed procedure.

The term d describes a measure. Literature proposes several approaches [19–22] to compute d based on dimensionally homogeneous metric, risk management, perception, safety standards. The conventional problem consists in selecting the form of d, when it is known the space and the reference frame. Authors propose the selection of the terms composed by space, reference frame and form of d.

98

A. Borboni et al.

This work suggests two criteria to select the term: (i) the computation of d; (ii) the form of d is adequate to discriminate two adjacent configurations. The criterion (ii) is necessary to quantify the phenomenon magnitude; thus, the criterion (i) defines a global optimal problem with the constraint described in (ii). The solution can be adopted as a common framework by the different interacting agents. A reference frame should be adopted for a specific metric d by the robot and the human. Figure 1 summarizes the proposed procedure to obtain a proper reference frame selection. Starting from the identification of physical features step, to the selection of N integer number of sensors (e.g. accelerometers, cameras or sensors able to provide kinematic representation of d perception), then the definition of the space and the frame computation; and finally the frame validation phase.

4 Sharing Strategies 2D-Case Study The graphical description of the presented scenario, 2D spatial area, is shown in Fig. 2. The frame setting considers the robot end-effector (Rtj ), the SCARA robot (Rri ), an operator (Rhk ), a static obstacle and a moving object. Proposed coordinate reference frame concept is based on defining iteratively a coordinate reference frame for the all the required points of the obstacles, end-effector and human. This method uses kinematic transformations and manipulates them to find the human safe and obstacle-free path for a collaborative robot task realization. In this way, a flexible task description that can model robot movement with respect to egocentric and allocentric reference frames is formulated. The position of an arbitrary egocentric Rhk reference frame on a human can be written as: Rhk ¼ Rtj þ A u

ð3Þ

Skh ; Rkh

Sjt ; Rjt

V = [0; 0; 0; 0; 0; 0]

V ≠ [0; 0; 0; 0; 0; 0] β

α

S ir ; R ir

Fig. 2. 2D-Working space with Human-Robot interaction.

Reference Frame Identification and Distributed Control Strategies

99

where the vector Rtj defines the location of the robot end-effector coordinate frame, A = A(h) is the transformation matrix that defines the orientation of the local coordinate Rtj frame with respect to the robot frame Rrj as function of the used orientation  is the vector that defines the location of the arbitrary point with parameters h, and u respect to the end-effector coordinate frame. Reference system selection refers at minimization of computational resource utilization criteria, in future works this aspect need to be considered. Furthermore, human is considered as stationary in a workstation position to provide the work instruction task, with static head-torso location and timevariant limbs motion. The mentioned assumption is acceptable for the preliminary work, future studies need to develop the mathematical description of real environment. To validate the 2D concept, the kinematics of a human-robot chain are represented in Fig. 3a,b. The distributed control is definite by human that transfer his commands to robot and consequently it drives the end-effector, Fig. 3a. In proposed perceptive task description approach, the distributed control is transferred to robot that in real time manage all sensor signals, update the perceive parameters, monitor the safety-risks and manage the collision avoidance, Fig. 3b. The kinematic equations are expressed as follows:

Skh ; Rkh

Sjt ; Rjt

Skh ; Rkh

Sjt ; Rjt

V = [0; 0; 0; 0; 0; 0]

V = [0; 0; 0; 0; 0; 0]

V ≠ [0; 0; 0; 0; 0; 0]

V ≠ [0; 0; 0; 0; 0; 0]

β

β

α

α

S ir ; R ir

S ir ; R ir

(a)

(b)

Fig. 3. 2D-Working space of SCARA robot with Human, Global, End-effector references and fixed-moving obstacles: a) Human-centered, b) Robot-centered.

Rri ¼ Rtj  Rhk

ð4Þ

Rtj ¼ Rhk  Rri þ x1  Rtj

ð5Þ

Rtj frame related to frame Rri represents the vector Rhk . The vectors Rtj and Rhk are the vector from human to task and vector from human to robot, respectively. The outer product term of Eqs. (2, 3) is due to the rotation of robot reference Rri . The task descriptors for a segments spatial orientation can be computed in Euler (3D) or Spherical angles (2D), or as the inclination of a body axis to any other (1D).

100

A. Borboni et al.

5 Discussion The adoption of a reference frame for the task definition, the safety assurance and the obstacle avoidance allows the coordination and knowledge sharing between robot and human. As mentioned in Sect. 3, the optimal selection of the terms composed by space, reference and d measure is optimal in a definite condition, thus, when multiple criteria are considered to properly define the actions of human and robot a necessary choice can produce performance degradation. In these cases, two solutions can be implemented: (a) defining a hierarchy of different measures to control in the collaborative framework; (b) defining a general scalar measure that is composed by a combination of the basic measures. The solution (a) is easier to implement nevertheless some measures were neglected. The solution (b) consider available measures simultaneously, the limit consists in influence to different parameters that need to be calibrated. The human safety was decisive. The robot safety needs to be evaluate. Finally, under the previous hierarchical constraints, the task should be realized. Hybrid solutions can be designed, when different measures are adopted simultaneously, the term composed by space, reference frame and measure can become complex.

6 Conclusions This paper presents a procedure to perceptive reference frame distribution definition in human-robot collaborative tasks. Its scope is to analyze and evaluate the effectiveness to distributed reference frame in order to guarantee the operator safety and obstacles collision avoidance. Literature methodologies and robots usage have been analyzed and compared by considering different approaches of reference frame selection (allocentric, egocentric, route-centric). Moreover, peculiarities and key characteristics of collaborative tasks are considered. A flexible task description is mathematically formulated to model robot gesture with respect to egocentric and allocentric reference frames. A casestudy was outlined for validating the proposed concept by considering the relative kinematics of a human-robot chain in a planar case. This case-study considers the robot end-effector, a SCARA robot, an operator, a static obstacle and a moving object. Acknowledgements. The paper presents results from the research activities of the project ID 37_215, MySMIS code 103415 “Innovative approaches regarding the rehabilitation and assistive robotics for healthy ageing” co-financed by the European Regional Development Fund through the Competitiveness Operational Programme 2014-2020, Priority Axis 1, Action 1.1.4, through the financing contract 20/01.09.2016, between the Technical University of Cluj-Napoca and ANCSI as Intermediary Organism in the name and for the Ministry of European Funds.

Reference Frame Identification and Distributed Control Strategies

101

References 1. Dugar, V., Choudhury, S., Scherer, S.: A kiTE in the wind: smooth trajectory optimization in a moving reference frame. In: Proceedings of the IEEE International Conference on Robotics and Automation, Article no. 7989017, pp. 109–116 (2017) 2. Tan, J., Xi, N., Goradia, A., Sheng, W.: Coordination of multi-robot and human systems in a perceptive reference frame. Int. J. Veh. Auton. Syst. 2(3–4), 201–216 (2004) 3. Tarn, T.-J., Tan, J., Xi, N.: A perceptive reference frame for cooperative and reconfigurable multi-robot systems. IFAC Proc. Volumes (IFAC) 16, 553–558 (2005) 4. Nada, A.A., Hussein, B.A., Megahed, S.M., Shabana, A.A.: Floating frame of reference and absolute nodal coordinate formulations in the large deformation analysis of robotic manipulators: a comparative experimental numerical study. In: DETC 2009, vol. 4, pp. 889– 900 (2010) 5. Garcia-Vallejo, D., Sugiyama, H., Shabana, A.A.: Finite element analysis of the geometric stiffening effect. Part 1: A correction in the floating frame of reference formulation. J. Multibody Dyn. 219(2), 187–202 (2005) 6. Garcia-Vallejo, D., Sugiyama, H., Shabana, A.A.: Finite element analysis of the geometric stiffening effect. Part 2: Non-linear elasticity. Proc. Inst. Mech. Eng. Part K: J. Multibody Dyn. 219(2), 203–211 (2005) 7. Wickens, C.D., Keller, J.W., Small, R.L.: Left. no, right! development of the frame of reference transformation tool (FORT). In: Proceedings of the Human Factors and Ergonomics Society, vol. 2, pp. 1022–1026 (2010) 8. Oess, T., Krichmar, J.L., Röhrbein, F.: A computational model for spatial navigation based on reference frames in the hippocampus, retrosplenial cortex, and posterior parietal cortex. Front. Neurorobot. 11, 4 (2017) 9. Wolbers, T., Wiener, J.M.: Challenges for identifying the neural mechanisms that supports patial navigation: the impact of spatial scale. Front. Hum. Neurosci. 8, 571 (2014) 10. Stoltmann, K., Fuchs, S., Krifka, M.: The influence of animacy and spatial relation complexity on the choice of frame of reference in German. In: Lecture Notes in Computer Science. LNAI, vol. 11034, pp. 119–133 (2018) 11. Kalla, P., Koona, R., Ravindranath, P., Sudhakar, I.: Coordinate reference frame technique for robotic planar path planning. Materials 5(9), 19073–19079 (2018) 12. Brown, A., Uneri, A., De Silva, T., Manbachi, A., Siewerdsen, J.H.: Design and validation of an open-source library of dynamic reference frames for research and education in optical tracking. In: Progress in Biomedical Optics and Imaging - SPIE, vol. 10576, p. 105760M (2018) 13. Mäkinen, P., Dmitrochenko, O., Mattila, J.: Floating frame of reference formulation for a flexible manipulator with hydraulic actuation - modelling and experimental validation. In: BATH/ASME 2018 Symposium on Fluid Power and Motion Control, FPMC 2018 (2018) 14. James, K.B.: System for controlling artificial knee joint action in an above knee prosthesis. U.S. Patent US5571205 A, 5 Novemeber 1996 15. Pathak, A.: System and method for stabilizing unintentional muscle movements. U.S. Patent 20140052275 A1, 20 February 2014 16. Abiyev, R., Ibrahim, D., Erin, B.: Navigation of mobile robots in the presence of obstacles. Adv. Eng. Softw. 41, 1179–1186 (2010) 17. Siegwart, R., Nourbakhsh, I.R.: Introduction to Autonomous Mobile Robot, Massachusetts Institute of Technology Press, Cambridge (2011) 18. Al-Taharwa, I., Sheta, A., Al-Weshah, M.: A mobile robot path planning using genetic algorithm in static environment. J. Comput. Sci. 4(4), 341–344 (2008)

102

A. Borboni et al.

19. García De Jalón, J., Unda, J., Avello, A.: Natural coordinates for the computer analysis of multibody systems. Comput. Methods Appl. Mech. 56, 309–327 (1986) 20. Chirikjian, G.S., Zhou, S.: Metrics on motion and deformation of solid models. J. Mech. Des. 120(2), 252–261 (1998) 21. Mazzotti, C., Sancisi, N., Parenti-Castelli, V.: A measure of the distance between two rigidbody poses based on the use of platonic solids. In: Robot Design, Dynamics and Control, ROMANSY21 2016. CISM, vol 569. Springer, Heidelberg (2016) 22. Cordero, C.A., Carbone, G., Ceccarelli, M., Echávarri, J., Muñoz, J.L.: Experimental tests in human–robot collision evaluation and characterization of a new safety index for robot operation. Mech. Mach. Theory 80, 184–199 (2014)

Static Balancing of a 5-DoF Mechanism with Manual Reconfiguration Feature Terence Essomba(&) and Zi-Wei Zhao National Central University, Zhongli District, Taoyuan City 32001, Taiwan [email protected]

Abstract. This work presents the static balancing of a mechanism with 5 degrees of freedom. This specific feature allows it to manipulate a surgical tool and hold it in a certain position with no locking system required. The proposed mechanism is made of a linear parallelogram mechanism and an angular spherical decoupled architecture. Its kinematics is studied to permit the analysis of its gravitational potential energy. Several specific mechanical concepts based on tension springs are designed and implemented to generate an elastic potential energy that compensates the gravitational potential energy. This method allows the conservation of the mechanism total potential energy, which is required for the static balancing. A prototype of the statically balanced mechanism is manufactured. Its static balancing is successfully testing. Keywords: Statically balanced mechanism
Parallelogram linkage
Spherical decoupled mechanism
Kinematics

1 Introduction For certain surgical tasks, it is required to maintain a tool in a specific position and orientation. In order to dismiss the medical personnel from holding these tools for a long period of time, surgical rooms are often equipped with specific mechanical holders. These kinds of arm are operated by unlocking the mechanism and manipulating it into a desired configuration and then re-locking the mechanism joints. The major issue with these devices is the residual motion observed during the mechanism locking phase. This considerably compromises the accuracy once the mechanism is locked. This problem can be solved by the use of Statically Balanced Mechanism (SBM). They are manually operated to position their end effector while automatically compensating the effects of gravity. There is consequently no need to lock and unlock the mechanism before and after each motion, which suppresses the locking phase residual motion. In 1995, a technique using tension spring to compensate the gravity applied on passive mechanism is introduced [1]. The method relies on the conservation of the Total Potential Energy (TPE) that is the sum of the Gravitational Potential Energy (GPE) and the Elastic Potential Energy (EPE). These energies are respectively noted as variables ET, EG and EK. For a given linkage which height is modified by a variable h, the potential energy conservation formula is written as followed:

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 103–111, 2020. https://doi.org/10.1007/978-3-030-48989-2_12

104

T. Essomba and Z.-W. Zhao

d d ðET Þ ¼ ðEG þ EK Þ ¼ 0 dh dh

ð1Þ

To achieve this characteristic, the appropriate spring stiffnesses are calculated with respect of their fixation points. They remain independent from their corresponding joint coordinate. Several instances of SBM based on this concept can be found in the literature [2–12]. The objective of the present work is to design 5-DoF SBM for the positioning of lightweight instruments. A mechanical concept providing the uniformization of the static balancing will be proposed to enlarge its workspace and adapt to different geometric environment. The next Section describes the mechanism concept and will study its kinematics. The third and fourth Sections present the static balancing of the linear mechanism and the angular mechanisms. The Sect. 5 shows the mechanical design and testing of the SBM.

2 Mechanism Concept and Kinematic Analysis It is defined that the SBM must be able to manipulate a thin tool or instrument with both linear and angular motion. It will generate a total of 5 DoF divided into 3 linear DoF and 2 angular DoF. In order to simplify the mechanical concept and the static balancing study, it is proposed to use a hybrid mechanism and partially decoupled linear and angular motions. 2.1

Kinematics of the Linear Mechanism

The architecture of the linear mechanism in several working modes is shown in Fig. 1. Its kinematic is very simple and can be found using the geometric method. The position of the end effector can be written as a vector of three components, pE ¼ ½xE yE zE T .

Fig. 1. Possible working modes of the linear mechanism: (a) Horizontal, (b) Vertical Upward and (c) Vertical Downward.

Static Balancing of a 5-DoF Mechanism with Manual Reconfiguration Feature

105

For the resolution of the Forward Kinematic Model (FKM), the expression of each component of pE are written with respect of the coordinates of the mechanism’s three revolute joints given by angles h0, h1 and h2. When the linear mechanism is locked in Horizontal working mode, the FKM can be written as followed: 2

3 2 3 lðcos h1 þ cos h2 Þ sin h0 xE pE ¼ 4 yE 5 ¼ 4 lðcos h1 þ cos h2 Þ cos h0 5 zE lðsin h1 þ sin h2 Þ

ð2Þ

The specific working mode of the mechanism can be directly integrated in the kinematic model to obtain a unified model. The locking joint hL is integrated as a variable that can take three possible values: 0 for Horizontal working mode, p/2 for Vertical Upward and −p/2 for Vertical Downward working mode. The unified FKM of the mechanism can be then written by considering that the angle h0 and hL orients the two parallelogram linkages. The position of the linear mechanism end effector is therefore written as: 2

3 coshL sinh0 lðcos h1 þ cos h2 Þ þ sinhL sinh0 lðsin h1 þ sin h2 Þ pE ¼ 4 coshL cosh0 lðcos h1 þ cos h2 Þ  cosh0 sinhL lðsin h1 þ sin h2 Þ 5 sinhL lðcos h1 þ cos h2 Þ þ coshL lðsin h1 þ sin h2 Þ

ð3Þ

Where hL is −p/2, 0 or p/2, regarding the mechanism working mode. 2.2

Kinematics of the Angular Mechanism

The angular mechanism is made of a spherical architecture that provide an RCM. A Spherical Decoupled Mechanism assembled as a 5-Bar Spherical Linkage has been chosen for the mechanical architecture. It has been proposed and studied in 2018 [13] as a mechanism that merges the kinematic advantage of decoupled architecture while maintaining a simple design compared to other decoupled mechanisms. In 2019, it has been optimized for the manipulation of neuro-endoscope showing that it is well adapted for holding thin and light instruments. This mechanism is made of two serial arms of two spherical linkage referred as A and B. Their joints are denoted by A1 and A2 for the arm A and B1 and B2 for the arm B. The point O is the origin of the reference frame and the location of the center of rotation. The axes z1A, z2A, z1B, z2B and zE are respectively passing through the joints A1, A2, B1, B2 and E and are all intersecting in the point O. The axes z1A and z1B respectively are the same as x and y axes of the origin reference frame. One advantage of the SDM is the simplicity of its kinematic model compared to most spherical mechanisms. It can be resolved by simple geometric method using spherical trigonometry. The angular motion of the SDM end effector can be described by two rotations of angles uA and uB, respectively measured on plan (Oxz) and (Oyz). Since the mechanism is decoupled, each of angle is independently expressed with respect of only one joint. On the arm A for example, the angle uA is the one formed

106

T. Essomba and Z.-W. Zhao

between the axis z and the plane (OzEz1B) as shown on Fig. 2. On one hand, the angle eA has a complementarity with the angle uA on the plan (OzEz1A). The same method can be applied to the arm B.

Fig. 2. Mechanical architecture of the angular mechanism.

By applying the spherical trigonometry formula, the IKM of the mechanism is obtained by expressing the active joint variables hA2 and hB2 with respect to the platform angular coordinates uA and uB.    8 < hA2 ¼  cos1 sin uA  ðcos aÞ2 =ðsin aÞ2    : hB2 ¼  cos1 sin u  ðcos aÞ2 =ðsin aÞ2 B

ð4Þ

The passive joints of the present SDM can be calculated as followed:     8 cos a < h1A ¼ cos1 cos h2B  cos1 sin h2A sin a sin h2A     cos a : h1B ¼ cos1 cos h2A  cos1 sin h2B sin h2B sin a

ð5Þ

Based on the kinematics of the mechanism, the position of all linkages will be known for every configuration. They are required to determine the gravitational potential energy of the mechanism which is calculated using their height.

3 Static Balancing of the Linear Mechanism In order to ensure the mechanism static balancing in any working mode, it is proposed to add a linkage that is oriented by the change of configuration given by the mechanism variable hn. As the mechanism configuration is changing, the elongation of the tension

Static Balancing of a 5-DoF Mechanism with Manual Reconfiguration Feature

107

spring attached to it with evolve as well. This evolution can be changed by relocking that linkage into another position regarding the working mode of the mechanism. The mechanical concept is illustrated in Fig. 3. The rotation of the added linkage can control the mechanism working mode. The angle between this linkage and the longitudinal linkage of the parallelogram mechanism corresponds to the parameter hL. Based on this concept, the elongation of the tension spring is controlled by the angles hn and hL. While hn controls the progressive elongation of the tension spring, the angle hL defines how the elongation shall evolve. It is noted that in the case of hL = 0, the proposed concept will simulate the architecture of the classical parallelogram in horizontal working mode.

Fig. 3. Mechanical concept for the uniform static balancing of the linear mechanism.

Based on the expression given by Eq. (1), the potential energy conservation formula is applied to the second linkage and is developed as followed:    d l  g  mL2 k2  2 sinðhL þ h2 Þ þ a2 þ b22 þ 2a2 b2 sinðh2 þ hL Þ ¼ 0 dh2 2 2

ð6Þ

The simplification of Eq. (6) allows finding the required spring stiffness: k2 ¼

l  g  mL2 2a2 b2

ð7Þ

The same method is applied to the first linkage and the required stiffness of its spring is written as followed: k1 ¼

l  g  ðmL1 þ 2mL2 Þ 2a1 b2

ð8Þ

108

T. Essomba and Z.-W. Zhao

4 Static Balancing of the Angular Mechanism The drawing of the conceptual design is shown on Fig. 4-(a). The first linkage of the SDM is made of two parallel linkages that present a right angle separating two straight sections. While the distal sections of both linkages form a rectangle, the proximal section of both linkages form a parallelogram architecture with the same kinematics as those of the linear mechanism.

Fig. 4. Static balancing concept for the SDM first linkage (a) and second linkage (b).

In the present design, this parallelogram is used to accommodate the tension spring that is elongated the same way as the horizontal parallelogram. The total potential energy of the linkage A is differentiated with its associated and developed as followed:    d kA  2 2 a þ a2  2a1 a2 sinh1A mA  g  ay  sin h1A þ ¼0 dh1A 2 1

ð9Þ

The simplification of Eq. (9) allows isolating the required spring stiffness and writing its expression with respect of the other linkage mechanical parameters as followed: kA ¼

mA  g  az a1 a2

ð10Þ

The mechanical concept illustrated in Fig. 4-(b) is proposed for the elongation of the spring of the second linkage. While the second linkage is oriented by the two different angles h1A and h2A, the parallelogram design of the first linkage allows the second one to be connected to a platform which orientation remains unchanged.

Static Balancing of a 5-DoF Mechanism with Manual Reconfiguration Feature

109

The expressions of the total potential energy of linkage C is written as followed: @2 @h1A @h2A



mC  g  cy ðcosh1A sinh2A  sinh1A Þ. . . . . . þ k2C c21 þ c22  2c1 c2 sinh1A cosh2A

 ¼0

ð11Þ

After differentiating the Eq. (11) with and h1A and h2A, the expression of the spring stiffness can be isolated as followed: kC ¼

mC  g  cy c1 c 2

ð12Þ

5 Mechanical Design of the Statically Balances Mechanism Although the relationships between the mechanical parameters to ensure the mechanism static balancing have been found, these parameters cannot be numerically. For this, a highly realistic CAD model (at least) is required to determine parameters such as the exact mass of the linkages, the linkage sizes, etc. 5.1

Parameters Identification

Based on a realistic CAD model, all mechanical parameters of the mechanism used for the static balancing study have been identified and used to calculate the required tension springs based on Eqs. (7), (8), (10) and (12). The results are given in Table 1.

Table 1. Tension spring stiffnesses for the mechanism. Spring stiffness (N/mm) k1 k2 kA kB kC kD Theoretical values 3.7013 2.4159 0.1281 0.1281 0.0132 0.0132 Available values 3.73 2.39 0.13 0.13 0.02 0.02

5.2

Prototype Testing

The entire 5-DoF SBM mechanism has been manufactured. The prototype has been mounted and tested by manipulating it in several configurations. The linear mechanism was tested with the spherical mechanism mounted at its end. It can be seen on Fig. 5 that it is able to hold itself while supporting the angular mechanism in any position, in any working mode (vertical or horizontal).

110

T. Essomba and Z.-W. Zhao

Fig. 5. Prototype of the SMB in several configuration in vertical and horizontal mode.

The angular mechanism was also tested by manipulating its end effector in several orientations. It is shown on Fig. 6 that the mechanism will conserve its orientation in any angle.

Fig. 6. Prototype of the statically balanced SDM in several configuration for testing.

6 Conclusion A statically balanced mechanism has been designed based on a 5-DoF architecture. The parallelogram mechanism is integrated with a specific mechanical concept that ensure the static balancing in any working mode: horizontal, upward and downward. The angular mechanism is made of a spherical decoupled mechanism. Its static balancing is ensured by a mechanical concept that allows the appropriate spring elongation with two different angles. The mechanical parameters associated with the mechanism static balancing are found and used for the design of a prototype. It has been successfully tested as it can preserve its balance is any direction and in any working mode.

Static Balancing of a 5-DoF Mechanism with Manual Reconfiguration Feature

111

References 1. Rahman, T., Ramanathan, R., Seliktar, R., Harwin, W.: A simple technique to passively gravity-balance articulated mechanisms. J. Mech. Des. 117(4), 655–658 (1995) 2. Agrawal, A., Agrawal, A.K.: Design of gravity balancing leg orthosis using non-zero free length springs. Mech. Mach. Theory 40(6), 693–709 (2005) 3. Banala, S.K., Agrawa, S.K., Fattah, A., Krishnamoorthy, V., Hsu, W.-L., Scholz, J., Rudolh, K.: Gravity-balancing leg orthosis and its performance evaluation. IEEE Trans. Rob. 22(6), 1228–1239 (2006) 4. Kuo, C.-H., Lai, S.-J.: Design of a novel statically balanced mechanism for laparoscope holders with decoupled positioning and orientating manipulation. ASME J. Mech. Rob. 8(1), 015001 (2016) 5. Cho, C.-H., Lee, W.: Design of a static balancer with equivalent mapping. Mech. Mach. Theory 101, 36–49 (2016) 6. Chen, Y., Liu, Y., Li, C., Liu, G., Zhu, Y., Zhao, J., Cai, H.: Gravity balance mechanism for a spatial robotic manipulator. J. Mech. Sci. Technol. 30(2), 865–869 (2016) 7. Gosselin, C.M., Wang, J.: On the design of gravity-compensated six-degree-of-freedom parallel mechanisms. In: Proceeding of IEEE International Conference of Robotics and Automation, vol. 3, pp. 2287–2294 (1998) 8. Wang, J., Gosselin, C.M.: Static balancing of spatial four-degree-of-freedom parallel mechanisms. Mech. Mach. Theory 35, 563–592 (2000) 9. Russo, A., Sinatra, R., Xi, F.: Static balancing of parallel robots. Mech. Mach. Theory 40, 191–202 (2005) 10. Lessard, S., Bigras, P., Bonev, I.A.: A new medical parallel robot and its static balancing optimization. ASME J. Med. Devices 1(4), 272–278 (2007) 11. Lin, P.-Y., Shieh, W.-B., Chen, D.-Z.: A stiffness matrix approach for the design of statically balanced planar articulated manipulators. Mech. Mach. Theory 45, 1877–1891 (2010) 12. Lin, P.-Y., Shieh, W.-B., Chen, D.-Z.: Design of statically balanced planar articulated manipulators with spring suspension. IEEE Trans. Rob. 28(1), 12–21 (2012) 13. Essomba, T., Nguyen Vu, L.: Kinematic analysis of a new five-bar spherical decoupled mechanism with two-degrees of freedom remote center of motion. Mech. Mach. Theory 119, 184–197 (2018)

Novel Safety Criterion for Synthesis of Cable Driven Parallel Robots Ines Ben Hamida1,2(&), Med Amine Laribi1, Abdelfattah Mlika2, Lotfi Romdhane2,3, Saïd Zeghloul1, and Giuseppe Carbone4 1

Department of GMSC, Pprime Institute, CNRS - University of Poitiers, ENSMA - UPR 3346, Poitiers, France {ines.ben.hamida,med.amine.laribi}@univ-poitiers.fr 2 Mechanical Laboratory of Sousse (LMS), National Engineering School of Sousse, University of Sousse, 4000 Sousse, Tunisia [email protected] 3 Department of Mechanical Engineering, American University of Sharjah, P.O. Box 26666, Sharjah, UAE 4 Department of Mechanical, Energy and Management Engineering, University of Calabria, 87036 Rende, Italy [email protected]

Abstract. This paper deals with the synthesis of a cable driven parallel robot intended for rehabilitation tasks. The objective is essentially to enhance the safety of patients using rehabilitation robotic devices. A novel safety criterion is proposed. It consists in maximizing the distance between the robot’s cables and the user. A dimensional synthesis of LAWEX robot considering the proposed safety criterion is carried out and optimal solutions are provided. Keywords: Cable driven parallel robot Rehabilitation device


Dimensional synthesis
Safety


1 Introduction Physical rehabilitation is one of the most effective ways to treat disabilities that appears post strokes. Robotic devices intended for rehabilitation attracted the interest of many researchers due to their efficiency in helping patients to restore. Cable driven parallel robots (CDPRs) is a special class of parallel robots that have been widely proposed for rehabilitation tasks due to their flexibility, ease of assembly, large workspace, etc. [1–4]. Thanks to the cables structure, CDPRs are considered as a relatively safe devices compared to rigid robots. However, some risks can be identified since the user interacts directly with the robot. A previous work of Carbone et al. [5] investigated on safety issues of rehabilitation devices. Motion ranges of upper limb for different people of different age were identified for the design of an inherently safe robotic rehabilitation device. Another safety measure that was investigated in literature, is the detection of cable failure. In fact, an emergency stop is often used with classical rigid robots however due to the flexibility of cables, the braking of motors could be not enough to stop the robot. Boschetti et al. [6] proposed an operational strategy for cable © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 112–120, 2020. https://doi.org/10.1007/978-3-030-48989-2_13

Novel Safety Criterion for Synthesis of Cable Driven Parallel Robots

113

failure that consists in generating an opposite wrench in case of failure which leads the end-effector to a safe position. This approach was tested and validated with LAWEX robot; a CDPR intended for rehabilitation of upper arm. Pott [7] addressed the safety issue related to the cable breaking load. To guarantee the user safety, the breaking load was proposed to be twice of the nominal charge of the manipulator. This factor has been considered for determination of upper and lower cables tensions. This work aims to reinforce the safety of rehabilitation CDPR with a new criterion. It consists in maximizing the distance between cables and the patient during the physiotherapy task. The proposed criterion was applied for optimal design of LAWEX. This paper is organized as follows: Sect. 2 introduced a problem formulation for the design of a rehabilitation safe cable driven robot. A novel safety criterion is developed to be implemented of the synthesis problem. Section 3 presents a case of study of the LAWEX robot; first, geometric and kinematic study of the robot is performed then the optimization problem is solved using Genetic Algorithms method [8]. Finally, some conclusions are given in the last section.

2 Problem Formulation of an Inherent Safe Cable Driven Robot 2.1

Formulation of the Synthesis Problem

The objective of this work is to provide a safe robotic device for rehabilitation of upper arm. The desired task is the flexion of the upper arm as shown in Fig. 1. An optimization problem is proposed to assure the safety of the patient during the rehabilitation task with the smallest cable driven robot having a compact workspace as close as possible to a desired trajectory. The mathematical formulation of the optimization problem is given as follow. 8
F 1 ðIÞ > > Minimize F ðIÞ ¼ > > F 2 ðIÞ < Subject to Ci ðIÞ  0 > > I ¼ ½x1 ; ::; xn  > > : xj min \xj \xj max

ð1Þ

Where F1 ðIÞ and F2 ðIÞ are the objective functions defining the size and safety criteria, respectively. Ci ðIÞ are the constraint functions. I ¼ ½x1 ; ::; xn  is the design vector defined by the set of design variables and xj min , xj max the search domain for each design variable xj . The desired trajectory presented in Fig. 2 defines the position of the wrist fixed on the robot’s end effector during the movement of upper arm flexion.

114

I. Ben Hamida et al.

Fig. 1. Flexion of upper arm

Fig. 2. Desired trajectory

The constraints of the optimization problem, Ci ðIÞ, guarantee that all points from the desired trajectory (Fig. 2) belongs to the feasible workspace. In the case of cable driven robot, the reachable space of one cable is determined by functions of two spheres (according to the minimum and maximum lengths of cables) called power point functions [3, 9, 10]. fi min ¼ lmin 2  ðx  xAi Þ2 þ ðy  yAi Þ2 þ ðz  zAi Þ2 fi max ¼ ðx  xAi Þ2 þ ðy  yAi Þ2 þ ðz  zAi Þ2 lmax 2

ð2Þ

Where: xAi ; yAi ; zAi define the position of the motor of the ith cable. x; y; z are the coordinates of a point from the desired trajectory (see Fig. 2) lmin and lmax are the minimum and maximum lengths of cables respectively. A feasible workspace of a CDPR with m cables requires: – A negative power point function fi min  0 and fi max  0 where i ¼ 1 :: m

ð3Þ

– A positive cables tensions since cables can only pull not push. The cables tension can be examined through the relation between the transpose of jacobian matrix and the total wrench applied by the end-effector.  1 s [ 0m1 ; s ¼ J T Wp

ð4Þ

Where: J: is the Jacobian matrix s ¼ ½s1 . . .sm T is the vector of all cables tensions T Wp ¼ ½f T mT  is the wrench applied by the end-effector on the external environment.

Novel Safety Criterion for Synthesis of Cable Driven Parallel Robots

2.2

115

A Novel Safety Criterion

A safety criterion that can be considered to reinforce the security of patient is the minimum distance between the cables and the patient during the rehabilitation exercise as presented in Fig. 3. The approach consists in discretizing the cables lines into “n” points and calculate the distance between each point and the cylinder surrounding the patient. The objective of this method is to maximize this distance to avoid any risk of collision between the patient and the robot. The patient is considered to be located inside a cylinder with diameter D, the shoulder width, and centered on its spine, as shown in Fig. 4.

Fig. 3. Safety criterion

Fig. 4. Cylinder surrounding the patient

  The distance dj between a given point Pj xj ; yj ; zj on the cable and the cylinder is given by the following equation dj ¼

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi  2  2 D xj  xc þ yj  yc  2

ð5Þ

Where xc ; yc , zc are the coordinates of a point from the cylinder axis (zc ¼ zj Þ. An objective function for safety criterion can be implemented as follows, where the idea is to maximize the distance between cables and user. F2 ðI Þ ¼

1   ; j ¼ 1 :: n min dj

Where: dj is the distance between a given point of a cable and the user n: number of discretized points on cables.

ð6Þ

116

I. Ben Hamida et al.

3 Dimensional Synthesis of LAWEX Robot 3.1

LAWEX Robot

This section deals with a specific CDPR intended for rehabilitation of upper arm: LAWEX. The mechanism is composed of 3 rotative motors controlling the lengths of 3 non-flexible cables attached on the same point to a circular end-effector as shown in Fig. 5. The 3 actuated cables generate 3 translational motions in the space.

Fig. 5. LAWEX robot

Fig. 6. Design parameters

The positions of motors are given in Eq. (7). The inverse kinematic model of LAWEX robot is defined by the lengths of cables according the end-effector position given by the vector p (Eq. (8)). ! A1 ¼ ½d1 ; 0; h where A1 ¼ OA1 ! A2 ¼ ½d; d; h where A2 ¼ OA2 ! A3 ¼ ½0; d 2 ; h where A3 ¼ OA3

ð7Þ

! p ¼ OP ¼ ½x y zT

ð8Þ

The inverse kinematic model (IKM) is obtained by the closed-loop equation of each cable defined in the following equation. p ¼ A1 þ l1 ¼ A2 þ l2 ¼ A3 þ l3 Where: li is the vector from the ith motor to the end effector.

ð9Þ

Novel Safety Criterion for Synthesis of Cable Driven Parallel Robots

117

The IKM is given by the lengths of the 3 cables qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 8 > l ¼ k l k ¼ k p  A1 k ¼ ðx  d1 Þ2 þ y2 þ ðz  hÞ2 > 1 > < 1 qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi l2 ¼ kl2 k ¼ kp  A2k ¼ ðx  dÞ2 þ ðy  d Þ2 þ ðz  hÞ2 > qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi > > : l3 ¼ kl3 k ¼ kp  A3k ¼ x2 þ ðy  d2 Þ2 þ ðz  hÞ2

ð10Þ

The cables tensions are determined according to Eq. (4), where: 0 xd B ½ s1 ; s2 ; s3  T ¼ @

1

l1 y l1 zh l1

xd l2 yd l2 zh l2

11 0 1 x 0 l3 yd2 C @ 0 A l3 A zh Mg l3

ð11Þ

M: end-effector mass, g: gravitational acceleration. 3.2

Optimization of LAWEX Robot

For optimal dimensioning of LAWEX robot, the proposed formulation for safety optimization is applied. Considering the rehabilitation exercise described in Sect. 2.1, the prescribed trajectory is an arc of center the position of the patient’s elbow E ðxE ; yE ; zE Þ and radius the length of the forearm. The design vector I is given by 6 parameters shown in Fig. 6. I ¼ ½d1 ; d2 ; d; h; r; zE T pffiffiffi pffiffiffi Where: r ¼ 2 xE ¼ 2 yE and d1 ; d2 ; d; h define the position of motors. The multi-objective optimization problem is based on the minimization of two objective functions as introduced in Sect. 2.1. F1 ðIÞ is the power function evaluating the distance between the prescribed trajectory and workspace boundaries and F2 ðIÞ is the safety function of distance between patient and cables. We fix the minimum and maximum length of cables as lmin ¼ 100 mm and lmax ¼ 1000 mm. For the safety criterion, we discretize each cable into 6 points and we evaluate the distance between each point and the cylinder surrounding the patient. 8 P3 P3 > i¼1 fi min þ i¼1 fi max > > > F1 ðIÞ ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P P3 < 2 3 2 i¼1 ðfi min Þ þ i¼1 ðfi max Þ F ðIÞ ¼ > > 1 > >   ; j ¼ 1 :: 6; i ¼ 1; 2; 3 : F2 ðIÞ ¼ min dij

ð12Þ

118

I. Ben Hamida et al.

The problem is subject to two constraints: – The prescribed trajectory (Eq. (13)) – The positive tensions of cables (Eq. (14))


fi max ¼ ðx  xAi Þ2 þ ðy  yAi Þ2 þ ðz  zAi Þ2 lmax 2  0 ; i ¼ 1; 2; 3 fi min ¼ lmin 2  ðx  xAi Þ2 þ ðy  yAi Þ2 þ ðz  zAi Þ2  0 si [ 0; i ¼ 1; 2; 3

ð13Þ ð14Þ

The resolution of the optimization problem is carried out using the Multi-Objective Genetic Algorithm solver. The fitness function is given by the following equation. The variation ranges of the design parameters are given in Table 1. f ¼ ðF k ðIÞ þ F 0 ðIÞÞ

ð15Þ

Table 1. Variation ranges of design parameters d2 d h r zE d1 Min (mm) 20 20 10 30 500 0 Max (mm) 1000 1000 500 1500 1000 1000

Where Fk ðIÞ for k ¼ 1; 2 is the objective function (Eq. (12)). F0 ðIÞ: is ( a penalty function defined as follow: 0 if the constraints are satisfied ðEqs. ð13Þ; ð14ÞÞ F0 ðIÞ ¼ 100 3.3

Results and Discussion

The obtained Pareto front is presented in Fig. 7. The two considered criteria are contradictory; the improvement of one generates the degradation of the other. The safety criterion is given by the inverse of minimal distance between cables and patient. This distance ranges between 14 mm (1=0:07) and 50 mm (1=0:02), as shown on the Pareto front. The representation of both extreme solutions (see Fig. 8) given in Table 2 shows that the variation of the robot size is too small compared to the safety criterion.

Novel Safety Criterion for Synthesis of Cable Driven Parallel Robots

Fig. 7. Pareto front

(a)

(b)

Fig. 8. Extreme solutions: (a) smallest workspace, (b) safer structure

Table 2. Design parameters of extreme solutions (mm) d1 d2 d h r zE Solution 1 970 800 193 1041 543 691 Solution 2 945 945 235 878 668 521

119

120

I. Ben Hamida et al.

4 Conclusion This paper addresses the safety issue of robotic rehabilitation devices by taking into account the distance between cables and patient during the therapy. An optimization problem was formulated to enhance safety and compactness for a prescribed trajectory. A case study has been carried out on LAWEX rehabilitation cable driven robot. The obtained results show safety and compactness are competitive design objectives. Often, an improvement of safety is obtained at cost of a bulky structure. The proposed formulation allows to find the best tradeoff solution within a Pareto front. Acknowledgement. This work was developed in part during a research stay of the first author within the ERASMUS+ program 2019–2020 at the university of Calabria, under the supervision of Prof. Carbone.

References 1. Cafolla, D., Russo, M., Carbone, G.: CUBE, a cable-driven device for limb rehabilitation. J. Bionic Eng. 16, 493–494 (2019) 2. Mao, Y., Agrawal, S.K.: Design of a cable-driven arm exoskeleton (CAREX) for neural rehabilitation. IEEE Trans. Robot. 28(4), 922–931 (2012) 3. Amine, L.M., Giuseppe, C., Saïd, Z.: On the optimal design of cable driven parallel robot with a prescribed workspace for upper limb rehabilitation tasks. J. Bionic Eng. 16, 503–504 (2019) 4. Lamine, H., Laribi, M.A., Bennour, S., Romdhane, L., Zeghloul, S.: Design study of a cablebased gait training machine. J. Bionic Eng. 14(2), 232–244 (2017) 5. Giuseppe, C., Gherman, B., Ulinici, I., Vaida, C.: Design issues for an inherently safe robotic rehabilitation device. In: International Conference on Robotics in Alpe-Adria Danube Region RAAD, pp. 1–8 (2017) 6. Boschetti, G., Carbone, G., Passarini, C.: Cable failure operation strategy for a rehabilitation cable-driven robot. Robotics 8(17), 1–12 (2019) 7. Pott, A.: On the limitations on the lower and upper tensions for cable-driven parallel robots. In: Lenarčič, J., Khatib, O. (eds.) Advances in Robot Kinematics, pp. 243–251. Springer, Cham (2014) 8. Kelaiaia, R., Company, O., Zaatri, A.: Multiobjective optimization of a linear Delta parallel robot. Mech. Mach. Theory 50, 159–178 (2012) 9. Ben Hamida, I., Laribi, M.A., Mlika, A., Romdhane, L., Zeghloul, S.: Geometric based approach for workspace analysis of translational parallel robots. In: ROMANSY 22 – Robot Design, Dynamics and Control, vol. 584, pp. 180–188 (2019) 10. Laribi, M.A., Mlika, A., Romdhane, L., Zeghloul, S.: Geometric and kinematic performance analysis and comparison of three translational parallel manipulators. In: The 14th IFToMM World Congress (2015)

Human Activity Recognition Enhanced Robot-Assisted Minimally Invasive Surgery Hang Su1 , Wen Qi1 , Chenguang Yang2(B) , Jiehao Li1 , Xuanyi Zhou1 , Giancarlo Ferrigno1 , and Elena De Momi1 1

2

Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, 20133 Milan, Italy [email protected] Bristol Robotics Laboratory, University of the West of England, Bristol BS16 1QY, UK [email protected]

Abstract. This paper presents a human activity recognition enhanced robot-assisted minimally invasive surgery (HAR-RAMIS) for the redundant robot, which aims to bridge the human activity recognition techniques and intelligent robot control strategies in the operating room. It allows the robot to follow human activities in the surgical room, and change behavior intelligently and cooperatively. In this study, multiple sensors are combined to enable human activity recognition, and a deep learning technique is applied to understand human behavior and intention. The robot control strategies can be switched automatically according to the different surgical tasks and the different human activities in the operating room. The experimental study was performed in a lab setup environment using the KUKA LWR4+ to demonstrate the efficiency of the developed strategy, indicating that the future surgical robot systems will be able to work with medical staff in a reasonable collaboration manner. Keywords: Human activity recognition · Minimally invasive surgery Redundant manipulator · Event-based control

1

·

Introduction

Robot-assisted minimally invasive surgery (RAMIS) represents a significant advancement compared to the traditional open surgery for a large number of interventions. RAMIS offers excellent potential to increase the benefits of MIS further. Robotic surgical platforms on the market, such as da Vinci, Senhance, Revo-i, Versius, etc., facilitate the surgical operation with high-definition (HD) 3D vision and a wide range of dexterous surgical tools [14]. However, specialized custom-developed surgical robots are expensive, and limiting their acquisitions among hospitals. Industrial robots with redundant manipulators such as KUKA c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 121–129, 2020. https://doi.org/10.1007/978-3-030-48989-2_14

122

H. Su et al.

robot have been successfully adopted and further developed in precise automation processes for several decades. Their lower cost concerning specialized surgical robots has increased interest in their application in the medical field, in particular in RAMIS. Human activity recognition (HAR) using multisensor fusion provides significant guidance for RAMIS. Machine learning (ML) and deep learning (DL) techniques are widely utilized for the scientific study of the statistical models of human behaviors. In previous works, compared to the recognition rate for the identification of human activities by using different combined sensors, a deep convolutional neural network (DCNN) is applied to the HAR system [4]. Moreover, in order to overcome time-consuming issues, an adaptive recognition and real-time monitoring system for HAR are enhanced with ML methods. These classifiers are proved to identify more human motions in dynamic situations for improvement in accuracy enhancement, robustness, and time-saving [9]. Furthermore, a hybrid hierarchical classification algorithm combining DL and thresholdbased method is presented to distinguish complex activities for fast computation [3]. Although our previous studies have presented many effective frameworks, most of the acceptable results are achieved under limited assumptions and conditions, which cannot meet the complex environments in the operating room. Hence, a novel multisensor fusion-powered HAR using the DL algorithm for RAMIS is discussed in this paper. Human activities will be comprehensively acquired and preprocessed using a multimodal sensor fusion system. Then deep learning technique is applied to understand human behavior and intention. The robot control strategies will be switched automatically according to the different surgical tasks and the different human activities in the operation space of the robot. The proposed HAR-RAMIS approach is taking one step forward in exploring the higher-level surgery-related knowledge empowered by AI techniques in order to enable more surgical robots for both intelligent and efficient operation and cooperation. This article is organized as follows: Methodology is presented in Sect. 2. In Sect. 3, the proposed approach is implemented, followed by a performance evaluation with discussion. Section 4 concludes the achievements and delineates avenues for further work on this topic.

2

Methodology

Figure 1 depicts the detailed scenarios of DL based structure for HAR using multiple sensors, i.e., KINECT and Inertial Measurement Unit (IMU), which instead of complex instrument operation procedures. Multiple data, such as depth vision, acceleration, orientation, and angular velocity, are collected for accuracy enhancement [2]. In the RAMIS operation scenario, the surgeon’s activities are essential to be understood in order to enable the robot with intelligent behavior. For example, to avoid possible collision, interference, and artifact, we use depth cameras, EMG sensors, and IMU sensors to collect multiple data for training the DNN model. Meanwhile, the position, direction, even action intention can be corrected and predicted by using multiple data from these sensors.

Human Activity Recognition Enhanced RAMIS

123

Fig. 1. The scenario of multisensor fusion-powered human activity recognition in the operating room.

The details of data processing and modeling are described in Fig. 2. The collected data from multiple sensors will be processed by different signal processing algorithms, such as wavelet denoising, baseline drifting removal, Kalman filter, and Particle filter [8]. The adopted approaches aim to adjust the raw data/signals to the usable range. Furthermore, the depth vision data can be calibrated with the real positions and directions of the operator, which increases the accuracy of the collected multiple data. Continuously, all of the data will be divided into several segments using the sliding window mechanism. The DL technology has been verified as one of the most robust methods to establish models for regression and classification. After labeling these segments, we adopt the DL approach to build the classifier for recognizing activities using multiple sensor data, as shown in Fig. 3. More details, the DL-based classifier is trained using both depth vision and IMU data, such as acceleration, angular velocity, and direction. In a similar scenario, the operator carries IMU sensors positioned at different positions, which can provide more information for identifying more complex activities, even perceiving action intention so that it can reduce the unnecessary interference during the surgical operation [13].

Fig. 2. The procedures of multiple data processing and modeling.

124

H. Su et al.

Fig. 3. Human activity recognition using multiple sensors.

Except for sensing and rearrangement of the priority tasks, the smoothness and stability should be considered as a guarantee of safety. A continuous adaptive control method will be utilized to ensure the security and levelness of switching of the tasks. Figure 4 shows the final control diagram. The general event-driven scheme for HAR-RAMIS includes two primary modules, namely event-driven and hierarchical control architecture. The DCNN model aims to identify human activity for further event-based decision-making by adopting multiple data fusion techniques.

Fig. 4. The proposed event-driven scheme for HAR-RAMIS.

Utilizing the redundancy of the serial robot to achieve different control objectives, such as achieving human-like behavior, manipulability optimization, compliant safety enhanced strategy, and remote center of motion in MIS has been

Human Activity Recognition Enhanced RAMIS

125

developed and discussed in the past decades according to the-state-of-arts techniques [1,5]. In the RAMIS, the control objectives can be defined with three levels based on the priority, as follows: (I) Surgical Operation [9]: The priority task should be the surgical operation during the RAMIS, which is of vital importance for the success of the surgery; (II) RCM Constraint [10,11]: The surgical tool should respect the small incision on the abdominal wall, which produces a kinematic constraint, known as a remote center of motion (RCM); (III) Other robot behaviors [6,7,12]: The other redundancies of the robot arm could be utilized to achieve extra tasks.

3

Multisensor Based HAR Implementation in a Lab Setup Environment

The experiments were designed to demonstrate the performance of the multisensor based HAR-RAMIS system. Hence, we compared the classification accuracy and the number of activities by using different types of sensors. The first experiment aimed to prove the classification ability of the multisensor (IMUs and Kinect) based HAR system. The second experiment was designed to prove that using multiple sensors (Myo armband and Kinect) not only can increase the recognition rate but also identify more activities, even hand gestures. The experiments have implemented these methods in MATLAB 2018b with the hardware platform of Intel(R) i7 Core, 2.80 GHz CPU, and 16.0 GB RAM. 3.1

IMUs and Kinect Based HAR

We asked ten subjects (five females and five males, age range 18–35 years old) to do five possible activities in the surgery room, namely sitting, standing, walking, bend over forwards, and a series of transfer motions (e.g., from sitting to standing) by carrying two IMU sensors on the waist and in the left pant’s pocket. Meanwhile, a Kinect camera was used to capture depth data for calibrating the results of the sensor-based HAR model. Each activity had been done one minutes. Due to the same sampling frequency was 30 Hz, the collected datasets had about 3000 samples including IMU signals and 3D joints depth data. Six classification models were compared to prove that the DL method could get a higher accuracy. They are two types of feed-forward neural network (FFNN), multi-classes support vector machine (SVM), and other three kinds of DCNN models. Both single layer and two layers FFNN algorithms were used, which could be expressed as single neural network (SNN) and multiple neural network (MNN). The number of nodes of SNN was 60, while the hidden layers’ nodes of MNN were 40 and 80, respectively. The multi-classes SVM model adopted one versus one strategy. The proposed DCNN model was described in Fig. 5 with three deep convolutional modules, a dropout layer, a full connection layer, and a classification layer. To improve the comprehensive of the

126

H. Su et al.

comparison experiments, we reconstruct the DCNN model by removing the last two convolutional modules layer by layer. We use “2L-DCNN” and “1L-DCNN” to represent these two DCNN models.

Fig. 5. The schematic diagram of Deep Convolutional Neural Networks.

The details of the DCNN frame can be described as: – Inputs: a matrix with d dimensions and fixed time-length, namely ˜s∗d×Ld . The value of d depends on the selected sensors. – Deep Convolution Modules: All of the three deep convolutional modules consist of 2D convolution layer, batch normalization (BN) layer and Rectified Linear Unit (ReLU) activation function. The filter size of all of these three CNN modules is 5 × 5 with the following window sizes, i.e., 4, 8, and 16. – Max-pooling: The max-pooling was performed to reduce the dimensionality by down-sampling and return the maximum features in each local patch. The max-pooled result acquired from the last layer of Conv. Module #3 reshape to create a feature vector for the input matrix. – Dropout: Due to the convolved and max-pooled feature vectors usually too large to cause over-fitting, it needs a dropout layer to avoid this problem and reduce the training time. In this article, we set 0.5 as the percentage of the dropout. – Output: The fully-connected layer place a SoftMax layer as an output layer. The mentioned five activities would be identified by computing the probability of each input of the node in the SoftMax layer. We adopt the leave-one-out cross-validation strategy to compare the performance among these six classifiers by using different sensors. More detail, the classifier is built on the collected nine subjects’ data, and tested on the last one. All of the results are run 20 times for avoiding overfitting or underfitting. We combine different sensors’ data to verify the best combination of sensors. Table 1 shows the comparison accuracy among the mentioned six ML and DL algorithms. To prove the multisensor based HAR model can obtain higher accuracy, we compare the results by selecting the collected data from one IMU sensor, two IMUs, and adding the depth data. By comparing the accuracy, the DCNN model with three convolutional modules is the best method to classify the mentioned five activities. It also can be

Human Activity Recognition Enhanced RAMIS

127

Table 1. The comparison accuracy of classifying five activities among SVM, SNN, MNN, DCNN, 1L-DCNN, and 2L-DCNN using different sensors. Sensors

Accuracy (%) SVM

IMU (waist)

SNN

MNN

1L-DCNN

2L-DCNN

DCNN

65.36 ± 5.69 71.28 ± 4.77 75.82 ± 4.18 79.58 ± 2.01 82.46 ± 2.38 84.56 ± 2.62

IMU (pocket)

64.11 ± 5.85 70.49 ± 3.67 74.24 ± 3.55 78.70 ± 2.35 81.99 ± 3.16 83.57 ± 2.29

2 IMUs

69.47 ± 5.22 75.89 ± 5.25 78.77 ± 4.00 82.11 ± 2.09 83.97 ± 2.58 86.01 ± 2.48

2 IMUs + Kinect 70.49 ± 5.14 76.36 ± 5.38 79.34 ± 4.89 83.07 ± 2.58 85.06 ± 2.96 87.18 ± 2.59

find that putting the IMU sensor on the waist will obtain higher accuracy than that in the pocket. However, combining both of them will increase the performance of classifying these five activities (the results are shown on the third row). In order to improve the accuracy, we add the depth data to train the classifier. The accuracy which are shown in the forth row prove that the performance of classifier will be enhanced by using more sensors. Moreover, to avoid any ambiguity, we can use the recognition success rate because the depth data provide the position information, which can improve the classification accuracy. 3.2

Myo Armband and Kinect Based Hand Gesture Recognition

The same ten participants (five females and five males, age range 18–35 years old) were asked to do three hand gestures (shown in Fig. 6, which express two, five, and ten) by wearing the Myo armband on the forearm. Similarly, the designed five classification methods were chosen to train the classifier based on the collected data, including eight-dimensions (8D) EMG signals and depth vision. To prove the mentioned claim in Sect. 1, we also compared the six models by combining different sensors. The leave-one-out cross-validation strategy was also adopted, and all of the results were also run 20 times in this experiment. Table 2 shows the acquired results (accuracy) by comparing the different classifiers and using sensors. The first two rows are the results of adopting Myo armband and Kinect, while the last two rows use IMU sensors.

Fig. 6. The demonstrated three figure gestures recognition using Myo armband in surgery room.

128

H. Su et al.

Table 2. The comparison accuracy for hand gesture recognition among SVM, SNN, MNN, DCNN, 1L-DCNN, and 2L-DCNN using different sensors. Sensors

Accuracy (%) SVM

Myo

SNN

MNN

1L-DCNN

2L-DCNN

DCNN

76.48 ± 3.99 77.12 ± 6.16 77.49 ± 5.98 82.44 ± 4.59 84.82 ± 3.74 86.71 ± 3.27

Myo + Kinect 84.87 ± 2.77 85.59 ± 4.57 85.99 ± 3.72 89.79 ± 2.98 90.73 ± 1.27 93.83 ± 1.09 IMU

Not work

2 IMUs

Not work

It can be seen that the IMU sensors cannot be used to recognize the hand gestures, while EMG signals and depth data can. Moreover, by combining the data collected from Myo armband and Kinect improve the accuracy for hand gesture recognition. The designed DCNN model is also the best method by comparing with the other four approaches. The comparison results prove that a multisensor based HAR system not only increases the accuracy for classifying activities but also can recognize more motions, such as hand gestures, which play important roles in the operating room.

4

Conclusion

In this paper, we proposed a HAR-RAMIS using the DL algorithm. Human activities are comprehensively acquired and preprocessed using a multimodal sensor fusion system. It takes one step forward in exploring the higher-level surgery-related knowledge empowered by AI techniques in order to enable the robots for intelligent and efficient operation and cooperation. The designed two experiments prove that using the DCNN approach and multisensor obtains a higher recognition rate than other methods. It shows its potential in the future operating room. Acknowledgment. This work was supported in part by the European Unio’s Horizon 2020 Research and Innovation Program through SMARTsurg Project under Grant 732515, and in part by the Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/S001913.

References 1. Atawnih, A., Papageorgiou, D., Doulgeri, Z.: Reaching for redundant arms with human-like motion and compliance properties. Robot. Auton. Syst. 62(12), 1731– 1741 (2014) 2. Li, Z., Huang, B., Ajoudani, A., Yang, C., Su, C.Y., Bicchi, A.: Asymmetric bimanual control of dual-arm exoskeletons for human-cooperative manipulations. IEEE Trans. Rob. 34(1), 264–271 (2017) 3. Qi, W., Aliverti, A.: A multimodal wearable system for continuous and realtime breathing pattern monitoring during daily activity. IEEE J. Biomed. Health Inform. PP, 1 (2019)

Human Activity Recognition Enhanced RAMIS

129

4. Qi, W., Su, H., Yang, C., Ferrigno, G., De Momi, E., Aliverti, A.: A fast and robust deep convolutional neural networks for complex human activity recognition using smartphone. Sensors 19(17), 3731 (2019) 5. Sandoval, J., Su, H., Vieyres, P., Poisson, G., Ferrigno, G., De Momi, E.: Collaborative framework for robot-assisted minimally invasive surgery using a 7-DoF anthropomorphic robot. Robot. Auton. Syst. 106, 95–106 (2018) 6. Su, H., Enayati, N., Vantadori, L., Spinoglio, A., Ferrigno, G., De Momi, E.: Online human-like redundancy optimization for tele-operated anthropomorphic manipulators. Int. J. Adv. Rob. Syst. 15(6), 1729881418814695 (2018) 7. Su, H., Li, S., Manivannan, J., Bascetta, L., Ferrigno, G., De Momi, E.: Manipulability optimization control of a serial redundant robot for robot-assisted minimally invasive surgery. In: 2019 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2019) 8. Su, H., Ovur, S.E., Zhou, X., Qi, W., Ferrigno, G., De Momi, E.: Depth vision guided hand gesture recognition using electromyographic signals. Adv. Robot. 15, 1–13 (2020). https://doi.org/10.1080/01691864.2020.1713886 9. Su, H., Qi, W., Yang, C., Aliverti, A., Ferrigno, G., De Momi, E.: Deep neural network approach in human-like redundancy optimization for anthropomorphic manipulators. IEEE Access 7, 124207–124216 (2019) 10. Su, H., Sandoval, J., Makhdoomi, M., Ferrigno, G., De Momi, E.: Safety-enhanced human-robot interaction control of redundant robot for teleoperated minimally invasive surgery. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6611–6616. IEEE (2018) 11. Su, H., Sandoval, J., Vieyres, P., Poisson, G., Ferrigno, G., De Momi, E.: Safetyenhanced collaborative framework for tele-operated minimally invasive surgery using a 7-DoF torque-controlled robot. Int. J. Control Autom. Syst. 16(6), 2915– 2923 (2018) 12. Su, H., Yang, C., Ferrigno, G., De Momi, E.: Improved human-robot collaborative control of redundant robot for teleoperated minimally invasive surgery. IEEE Robot. Autom. Lett. 4(2), 1447–1453 (2019) 13. Yang, C., Chen, C., Wang, N., Ju, Z., Fu, J., Wang, M.: Biologically inspired motion modeling and neural control for robot learning from demonstrations. IEEE Trans. Cogn. Dev. Syst. 11(2), 281–291 (2019) 14. Yang, C., Luo, J., Liu, C., Li, M., Dai, S.L.: Haptics electromyography perception and learning enhanced intelligence for teleoperated robot. IEEE Trans. Autom. Sci. Eng. 16(4), 1512–1521 (2018)

Online Payload Identification of a Franka Emika Robot for Medical Applications Souad Salah2, Juan Sandoval1(&), Moncef Ghiss2, Med Amine Laribi1, and Saïd Zeghloul1 1

2

Department of GMSC, Pprime Institute CNRS, ENSMA, University of Poitiers, UPR 3346, Poitiers, France {juan.sebastian.sandoval.arevalo,med.amine.laribi, said.zeghloul}@univ-poitiers.fr National Engineering School of Sousse, LMS, University of Sousse, BP 264 Sousse Erriadh, 4023 Sousse, Tunisia {Salah.Souad,Moncef.Ghiss}@eniso.u-sousse.tn

Abstract. Torque-control approaches implemented in collaborative robots include the compensation of static and dynamic effects, such as the gravity compensation approach. Therefore, it is required to know the inertial parameters of each robot’s link. Robot controllers generally allow to get access to these inertial parameters and even directly provide the compensation torques. However, these torques must be recalculated when attaching a new tool to the endeffector, by providing the controller with the inertial parameters of the added payload. In this paper, we briefly mention four methods for the identification of the inertial parameters and present one in depth to identify the inertial parameters of a payload held by the end effector of a collaborative robot, i.e. Franka Emika robot, in order to improve its dynamic compensation in the context of medical applications. Keywords: Collaborative Robots
Model identification
Inertial parameters
Dynamic compensation
Medical cobots

1 Introduction Collaborative Robots (Cobots) are knowing an increasing interest when it comes to the field of medicine. In fact, several cobot-based solutions have been developed nowadays in order to assist surgeons and doctors and help them practice their work under the best possible circumstances. Many robots have revolutionized the field of medicine by their flexibility and precision, such as the Stryker Mako robot used in the total knee arthroplasty [1] and the Kuka medical robot [2]. In the context of a teleoperation application such as the doppler echography, [3, 4] for instance, the ultrasound probe is held by a collaborative robot controlled by a vascular doctor through a haptic interface. Consequently, the robot reproduces the same movements executed by the interface. Depending on the type of the examination, the vascular doctor is constraint to use different types of probes with different mass and form for which the inertial parameters are generally unknown. Adding the probe on the robot’s end effector leads to a bad © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 130–136, 2020. https://doi.org/10.1007/978-3-030-48989-2_15

Online Payload Identification of a Franka Emika Robot

131

gravity compensation that could affect negatively the trajectories’ accuracy and disturb the force feedback felt by the doctor. Therefore, problems can be encountered while executing the examination. The same problem is met in other medical applications, such as in robot-assisted laparoscope holder [5]. Although some torque-controlled robots include a payload identification procedure, e.g. kuka iiwa, this is not the case for the Franka Emika robot used by the PPRIME Institute for the medical applications [3–5]. Therefore, after having studied the main existing payload estimation methods based on the proprioceptive sensors data, a suitable method has been implemented. This paper presents three experimental study cases allowing to validate the effectiveness of the implemented method. The paper is organized as follows. In Sect. 2, we mention several methods used to estimate the inertial parameters. Section 3 deals with the deep description of the chosen method along with the Franka Emika robot’s dynamic modeling. In Sect. 4, we describe the experimental validations with the different adopted case studies. Finally, a brief conclusion about the obtained results is provided.

2 Methods to Estimate the Inertial Parameters A payload is generally characterized by 10 inertial parameters: the mass, the coordinates of the center of mass and the parameters of the symmetric inertial matrix. These parameters can be estimated through the joint sensors measures (position, velocity, acceleration and torque) during the execution of suitable trajectories. According to [6–8], multiple identification methods were developed. The first one consists in a new identification of the robot parameters with the payload. The inertial parameters of the payload can be computed using the variation on some estimated inertial parameters with and without payload. The second one consists in a global identification of the robot parameters and the payload parameters in one step. Another method used to determine the inertial parameters consists in identifying the dynamic model of the robot using a reverse engineering approach [9, 10]. Also, according to [6–8], the inertial parameters of the payload can be estimated using the joint torques data measured for a same trajectory twice, with and without the payload attached to the robot. For this paper, only the last method is adopted since it is unnecessary to identify the overall dynamic parameters of the robot, as in the case of the other methods, which considerably minimizes the calculation time.

3 Identification Modeling The inverse dynamic model of a n-DoF (degrees of freedom) robot could be written as follows: _ €qÞ
u s0 ¼ K ðq; q;

ð1Þ

_ €q; s0 2 > < qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðsy ðm1 =2 þ m2 Þ L2  p2x  p2z þ ð2m1 þ m2 þ m3 Þpy þ m2 lÞ=M ¼ yðtÞ > > > > > > > > qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi > > : ðsz ðm1 =2 þ m2 Þ L2  p2  p2 þ ð2m1 þ m2 þ m3 Þpz þ m2 lÞ=M ¼ zðtÞ x y

ð12Þ

Note that the input displacements ðqx ; qy ; qz Þ of the manipulator Orthoglide can be found via inverse kinematic Eq. (1), (2) and (3).

232

J. Geng et al.

3 Illustrative Example and Simulation Results To create a CAD model and carry out simulations via ADAMS software, the following parameters of the Orthoglide are used [19]. These parameters correspond to the parameters of the prototype developed in LS2N (Fig. 5). The geometric parameters are: L ¼ B1 C1 ¼ B2 C2 ¼ B3 C3 ¼ 0:31 m, off pat ¼ 0:1 m, sx ¼ sy ¼ sz ¼ 1. The masses of links are: m1 ¼ 0:396 kg, m2 ¼ 0:248 kg and m3 ¼ 0:905 kg. The trajectory of the output axis P of the platform is given by its initial position Pi with the coordinates: xi ¼ 0, yi ¼ 0, zi ¼ 0 and the final position Pf with the coordinates: xf ¼ 0:1 m, yf ¼ 0:07 m, zf ¼ 0:11 m. The corresponding input displacements are determined via inverse kinematics: qxi ¼ 0:31 m; qxf ¼ 0:1812472 m, qyi ¼ 0:31 m, qyf ¼ 0:3420294 m, qzi ¼ 0:31 m, qzf ¼ 0:1749561 m. The coordinates of the common COM of the manipulator have been found: xSi ¼ 0:0360334 m, ySi ¼ 0:0360334 m, zSi ¼ 0:0360334 m, xSf ¼ 0:0440614 m, ySf ¼ 0:0853467 m, zSf ¼ 0:0513056 m. The traveling time of this trajectory is tf ¼ 1 s. Fig. 3 and Fig. 4 show the variations of shaking forces and the shaking moments for two studied cases: 1) the displacement of the platform of the unbalanced manipulator by the straight line with fifth order polynomial profile and 2) the generation of the motion via the displacement of the manipulator centre mass by “bang-bang” profile.

Fig. 3. Variations of shaking forces for two studied cases.

Fig. 4. Variations of shaking moments for two studied cases.

The simulation results show that the shaking force has been reduced up to 31%. Compared to the increase of the shaking moment of the balancing by adding counterweights, the shaking moment has a reduction of 30%. Another advantage of this method is its simplicity and versatility. In the case of changing trajectory, it is just necessary to provide the initial and final coordinates of the end-effector, calculate the input parameters according to the proposed method and implemented in the manipulator control system.

Shaking Force Balancing of the Orthoglide

233

4 Conclusion and Future Works It is known that the shaking force balancing by counterweights mounted on the links is more appropriate for serial and planar parallel manipulators. It is much more difficult for spatial parallel manipulators. Therefore, in this paper, an alternative method based on optimal acceleration control of the common COM is discussed. The object of study is the spatial 3-DOF parallel manipulator known as Orthoglide. The suggested balancing technique consists in the fact that the Orthoglide is controlled not by applying platform trajectories but by planning the displacements of the total mass centre of moving links. The trajectories of the total mass centre of the manipulator are defined as straight lines and are parameterized with “bang-bang” profile. Such a control approach allows the reduction of the maximum value of the centre of mass. The numerical simulations show the efficiency of the proposed method. Future works concern now the experimental Fig. 5. The prototype of the Orthovalidation of the suggested balancing technique via glide (LS2N). tests that will be carried out on the prototype of the Orthoglide developed in LS2N (Fig. 5).

References 1. Filaretov, V.F., Vukobratovic, M.K.: Static balancing and dynamic decoupling of the motion of manipulation robots. Mechatronics 3(6), 767–783 (1993) 2. Bayer, A., Merk, G.: Industrial robot with a weight balancing system. EP Patent 2 301 727 (2011) 3. Agrawal, S.K., Fattah, A.: Reactionless space and ground robots: novel design and concept studies. Mech. Mach. Theory 39(1), 25–40 (2004) 4. Fattah, A., Agrawal, S.K.: Design and modeling of classes of spatial reactionless manipulators. In: 2003 IEEE International Conference on Robotics and Automation (ICRA), Taipei, pp. 3225–3230. IEEE (2003) 5. Fattah, A., Agrawal, S.K.: Design arm simulation of a class of spatial reactionless manipulators. Robotica 23(1), 75–81 (2005) 6. van der Wijk, V., Herder, J.L.: Dynamic balancing of Clavel’s Delta robot. In: Kecskeméthy, A., Müller, A. (eds.) Computational Kinematics, pp. 315–322. Springer, Heidelberg (2009) 7. Alici, G., Shirinzadeh, B.: Optimum force balancing with mass distribution and a single elastic element for a five-bar parallel manipulator. In: 2003 IEEE International Conference on Robotics and Automation (ICRA), Taipei, pp. 3666–3671. IEEE (2003) 8. Ouyang, P.R., Zhang, W.J.: A novel force balancing method for real-time controllable mechanisms. In: ASME 2002, 27th Biennial Mechanisms and Robotics Conference, Montreal, pp. 183–190 (2002) 9. Briot, S., Arakelian, V., Le Baron, J.-P.: Shaking force minimization of high-speed robots via centre of mass acceleration control. Mech. Mach. Theory 57, 1–12 (2012)

234

J. Geng et al.

10. Briot, S., Arakelian, V., Sauvestre, N., Baron, J.-P.L.: Shaking forces minimization of highspeed robots via an optimal motion planning. In: ROMANSY 18 Robot Design, Dynamics and Control, pp. 307–314. Springer, Vienna (2010) 11. Arakelian, V.: Design of partially balanced 5R planar manipulators with reduced centre of mass acceleration (RCMA). In: Parenti-Castelli, V., Schiehlen, W. (eds.) ROMANSY 21 Robot Design, Dynamics and Control, vol. 569, pp. 113–122. Springer, Cham (2016) 12. Arakelian, V., Geng, J., Fomin, A.S.: Minimization of shaking loads in planar parallel structure manipulators by means of optimal control. J. Mach. Manuf. Reliab. 47(4), 303–309 (2018) 13. Geng, J., Arakelian, V.: Design of partially balanced planar 5R symmetrical parallel manipulators via an optimal motion planning. In: Uhl, T. (ed.) Advances in Mechanism and Machine Science, IFToMM 2019, vol. 73, pp. 2211–2220. Springer, Cham (2019) 14. Wenger, P., Chablat, D.: Kinematic analysis of a new parallel machine tool: the orthoglide. In: Lenarčič, J., Stanišić, M.M. (eds.) Advances in Robot Kinematics, pp. 305–314. Springer, Dordrecht (2000) 15. Chablat, D., Wenger, P.: Architecture optimization of a 3-DOF translational parallel mechanism for machining applications, the orthoglide. IEEE Trans. Robot. Autom. 19(3), 403–410 (2003) 16. Pashkevich, A., Chablat, D., Wenger, P.: Kinematics and workspace analysis of a three-axis parallel manipulator: the orthoglide. Robotica 24(1), 39–49 (2006) 17. Arakelian, V., Briot, S.: Balancing of Linkages and Robot Manipulators: Advanced Methods with Illustrative Examples. Springer, Cham (2015) 18. Khalil, W., Dombre, E.: Modeling, Identification and Control of Robots. Hermès, Paris (2003) 19. Guegan, S., Khalil, W., Lemoine, P.: Identification of the dynamic parameters of the orthoglide. In: 2003 IEEE International Conference on Robotics and Automation (ICRA), Taipei, pp. 3666–3671. IEEE (2003)

A Boosted Decision Tree Approach for a Safe Human-Robot Collaboration in Quasi-static Impact Situations Nemanja Kovincic1 , Hubert Gattringer1 , Andreas M¨ uller1(B) , and Mathias Brandst¨ otter2 1

Institute for Robotics, Johannes Kepler University, Linz, Austria {nemanja.kovincic,hubert.gattringer,a.mueller}@jku.at 2 JOANNEUM RESEARCH, Klagenfurt, Austria [email protected]

Abstract. According to the ISO/TS 15066, human safety in quasi-static impact situations in human-robot collaboration is assessed first by identifying all high-risk impact situations and then by measuring maximal and steady-state values the impact force and pressure at these possibly critical situations. This means that if something is changed in a collaborative application, the ISO/TS 15066 requires that the risk analysis and the force measurements must be redone, which severely limits the flexibility of a robotic system. In this paper, a physics guided boosted decision tree is proposed as a tool to assess human safety. The basic hypothesis is that a physics guided boosted decision tree can be trained to estimates the peak impact force for a given impact velocity, robot configuration, an impact point on the robot and a human body part. Based on experimental measurements done with the Universal Robots UR10e and on a simple mathematical model of an impact between a point on a robot and a point on a human body part, a feature vector is generated as an input to the boosted decision tree. After the training using Matlab’s Least-squares boosting algorithm, the boosted decision tree can predict the measured peak impact force with a relative error of less than 9% thus supporting the basic hypothesis. However, the predictions of the trained boosted decision tree are valid only for the case of a quasi-static impact in a vertical direction between a robot’s end-effector and a back of human’s non-dominant hand. Keywords: Human-robot collaboration · Safety learning · Boosted decision tree · ISO/TS 15066

1

· Robotics · Machine

Introduction

Over the last two decades, human-robot collaboration has gained a research interest since more and more robots are starting to share their workspace with This work was supported by the DR.KORS - Dynamic reconfigurability of collab¨ orative robot systems project, funded by FFG - Osterreichische Forschungsf¨ orderungsgesellschaft, project number 864892. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 235–244, 2020. https://doi.org/10.1007/978-3-030-48989-2_26

236

N. Kovincic et al.

humans, see [2]. To achieve such collaborative work, human safety must be guaranteed, which gave rise to ISO/TS 15066 (the Technical Specification of the International Organization for Standardization) that provides limiting values of forces and pressures a human body can withstand without being harmed, see [1]. According to the specification, human safety can be guaranteed either by avoiding any possible impact situation between a robot and a human and guarantying that an impact will not happen or by so called power and force limiting that ensures non-harmful unintended contact situations. In the power and force limiting case, the impact forces and pressures must less than the maximally allowed ones prescribed by the ISO/TS 15066. These maximal allowed values are obtained by experimental measurements and are dissimilar for different human body parts. If an impact between a robot and a human is allowed, ISO/TS 15066 also distinguishes between two types of impact: a transient impact, or “dynamic impact”, where a person’s body part is impacted by a moving part of a robot but can recoil or retract from it without clamping or trapping the body part, and a quasi-static impact that includes clamping or crushing of the person’s body part between part of the robot and another fixed or moving part of the collaborative work cell, see [1]. To assess human safety in a case of a quasi-static impact, all critical impact situations must be identified through risk analysis and the corresponding impact forces and pressures must be experimentally measured and compared to their limit values prescribed by the ISO/TS 15066. To simplify this safety assessment task, in [15], a safety map concept is introduced that relates, for each robot pose, the impact velocity and the concept of reflected mass to a type of human injury. However, the bio-mechanical injury data used to generate safety map is obtained thorough literature survey and not by in situ experimental measurement required by ISO/TS 15066. Also, in [9], an algorithm that utilizes injury knowledge for generating safe robot motions is proposed. However, the injury knowledge used was obtained using free-fall gravity drop-test experiment which does not correspond to an impact scenario between a collaborative robot and a human. Namely, all collaborative robot must be able to detect the impact and to react to it in order to minimize possible damage to a human body. Further, as noted in [7], classical Severity Indices established in the automobile industry cannot be used in safety analysis in human-robot collaboration due to big differences in impact velocities. For that reason, in [3], a new safety index for robots, called NIR, is proposed. However, as noted in [3], these indices are not applicable when a human body part is trapped between a robot and a rigid constraint, and thus they are not applicable to quasi-static impact situations. This paper addresses the safety assessment problem of collaborative robots and provides a strategy on how to guaranty human safety the power and force limiting case described in ISO/TS 15066. Namely, the goal is to determine the maximum velocity for which, if an un-intentional impact happens, the impact force and pressure can be considered as safe assuming that the impact point on the robot, a human body part and a robot configuration are known. In particular, we are interested in assessing safety for the case of a quasi-static impact,

A Boosted Decision Tree Approach for a Safe Human-Robot Collaboration

237

in a vertical direction, between a robot’s end-effector and a back of human’s non-dominant hand. The presented strategy is based on the principle to exploit physics as much as possible and then to learn what is unknown in the dynamical process. The need to learn what is unknown is driven by the fact that in industrial robots the controller and the impact detection and reaction mechanisms are in most cases proprietary and thus un-specified for the end-user. Namely, all collaborative robots must be able to detect an unintended impact and to react appropriately. In doing so, as describe in [8], they go through five phases of the so called collision detection pipeline where they have to detect, isolate and identify the impact and the impact force and then to classify the impact and to react to it. This pipeline makes a collaborative robot a hybrid system, i.e. a system having both a continuous and discrete behavior since its controller changes its work regime when the impact is detected. Although there are several known contact detection mechanisms, of which some are controller and trajectory dependent, see [5,8], stopping and retracting mechanisms are usually robotbrand specific. Thus, the hybrid nature and the fact that not all components of the robotic system are known make modeling and identification of the whole robotic system much harder. Furthermore, the influence of a human body part, with its complex structure, must be properly taken into account when modeling the impact dynamics. All of this motivates the use of machine learning (ML) approaches together with physical knowledge to estimate all unknown dynamical processes during an impact between a robot and a human body part. There are several different ways to incorporate physical knowledge in ML models, see [12]. The most common way is to use physics either to generate features or to guide feature selection. The other way is to use residual modeling where a ML model predicts an error made by a physics based model, see [4, 13]. Note that the ML models obtained in these two ways must be trained with labeled data. One more way of incorporating physics is to use physics based constraints in a ML model, see [10,11,13,16,18]. These constraints are introduced by generalizing the loss function to include not just the prediction error and the regularization term but also the constraints representing physical consistency. The ML models obtained in this way can be trained with a small number of labeled data or even with no labels at all, see [18]. Note that the previous approaches can be combined as in [13] where the output of a physics based model is used as an additional feature in a neural network model, and constraints on the output are used to enable label free learning. The basic hypothesis of this work is that a theory guided boosted decision tree (BDT) can be used to predict the peak impact force for a given impact velocity, robot configuration and an impact point on the robot. If one has such BDT, it can be used to predict peak impact forces for a whole set of different impact velocities, from which it is easy to evaluate the safe ones and to determine the maximal impact velocity such that the impact is safe for a human. The problem of predicting the peak impact force is a regression problem for which several supervised ML algorithms exist. For example, one can treat regression

238

N. Kovincic et al.

using linear or polynomial functions, neural networks, support vector machines or random decision trees and random forests, see [6]. In this paper, the regression problem is treated by training a boosted decision tree using Matlab’s Leastsquares boosting (LSBoost) algorithm.

2

Impact Modeling

Following [14] and the assumptions therein, in order to exploit physics as much as possible, a mathematical model describing an impact between a robot and a human body part is taken in a simple form as ¨ + Dx x˙ + Kx x = F, Mx x

x(0) = 0,

x(0) ˙ = vc ,

(1)

where x denotes a deformation of the spring representing human body part [1], vc stands for the impact velocity and where T −1 Mx = uT (Jc M−1 u, cs Jc )

Dx = uT (Jc D−1 JTc )−1 u,

Kx = uT (Jc K−1 JTc )−1 u,   T Qres + Rc sign(−J−1 F = uT (J−1 c ) c u) In the previous equations Qres = g(qcs ),

Mcs = M(qcs ),

D=

(2) for x˙ > 0.

 ˙  ∂ QR,v (q) = const,  ∂ q˙ ˙ cs =0 qcs , q

 ∂ g(q)  + k JTc,tra Jc,tra , K = K(qcs ) = ∂q qcs

(3)

where qcs denotes a vector of generalized coordinates at the contact configuration, q˙ cs is a vector of generalized velocities at the contact configuration, uT = nT 0T with n denoting unit vector of the impact force, or impact orientation, M(q) stands for the symmetric positive definite mass matrix and g(q) denotes the gravity vector. Furthermore QR,v denotes viscous friction, Rc stands for a diagonal matrix whose elements are dry friction coefficients, and T  Jc (q) = Jc,tra (q) Jc,ang (q) denotes Jacobian matrix corresponding to the contact point with Jc,tra and Jc,ang denoting respectively translator and angular part of the Jacobian. Note that the Jacobian matrix is assumed constant during the impact. Note that the damping matrix D represents influence of the viscous friction and it does not change with the impact configuration of the robot while the stiffness matrix K represents influence of the impact and gravity forces and it changes with the change of the impact configuration of the robot.

3

Physics Guided Boosted Decision Tree

To train a BDT to predict the peak impact force for a given impact velocity, robot configuration and an impact point on the robot, training examples must be provided first. In this work, based on experimental peak impact force measurements

A Boosted Decision Tree Approach for a Safe Human-Robot Collaboration

239

performed with the Universal Robots UR10e and published in [17], the training examples are generated following an information flow shown in Fig. 1. As can be seen in the mentioned figure, based on the known impact configuration and impact velocity, the input feature vector of the BDT is obtained by taking coefficients of the differential equation (1) together with the velocity initial condition, while the desired output is the measured peak impact force. With a total of 1501 training examples prepared, the BDT is trained using Matlab’s LSBoost algorithm.

qcs , vc

Mx x ¨ + Dx x˙ + Kx x = F

 Mx

x(0) = 0, x(0) ˙ = vc

Dx

Kx

F

vc

T

Fc,max BDT

Fig. 1. Physics guided boosted decision tree information flow

3.1

Experimental Measurements

In [17], experimental measurements done on a line in a horizontal plane, on a distance from a robot base ranging from 0.35 m to 1.15 m, with a step of 0.01 m are reported. In these experiment, the impact was in the direction of the normal

Fig. 2. Experimentally measured peak impact force taken from [17] (red cross) with the corresponding mean values (blue square) and the data used for the data augmentation (black circle).

240

N. Kovincic et al.

to the horizontal plane, i.e. in a vertical direction, while the impact velocities were 50, 100, 150, 200, and 250 mm/s. For more details regarding experimental measurements and experimental setup, one can refer to [17]. Note that due to the rotational symmetry around the base joint, the line experiment is enough to have impact force measurements, for a fixed orientation, in the whole horizontal plane, see [17]. Also, note that the maximum force value that the measuring device used in the experiments can measure is 300 N. The data corresponding to the force of 300 N are purposely omitted. Also, note that the measuring device was set to correspond to the back of a non-dominant hand and thus all experimental measurements correspond to this human body part [1]. In Fig. 2, experimentally measured peak impact forces are shown together with the corresponding mean values. As in [17], the data is shown in the logarithmic scale using a natural logarithm. Although the measurements are available at every 0.01 m, in practice, it is impractical to perform measurements at such a high resolution. Thus, the data set for training a boosted decision tree is formed using data interpolation. Namely, from the set of mean values, the data set for the interpolation was formed by selecting points at every 0.05 m starting from 0.35 m. The interpolation was done in a logarithmic scale using a linear function. The interpolation of distance with a step of 0.01 m is done first, and then velocity interpolation is done with a step of 0.01 m/s. The results of interpolations are shown in Fig. 3. Note that the measured data set, shown in Fig. 2 in red color, contains data points that are not included in the augmented data set. However, the BDT will be trained using the augmented data set and then tested using both data sets.

Fig. 3. The interpolated data.

A Boosted Decision Tree Approach for a Safe Human-Robot Collaboration

3.2

241

Training Results

The BDT is trained using 1501 training examples and with the LSBoost algorithm with a maximum of 100 learning cycles. The trained BDT has relative training error of 3.95%, and it was tested on the whole available data set giving the total prediction error of 8.63%. In Fig. 4 comparison of the measured and the peak impact forces predicted by the BDT is shown together with the biggest relative prediction error. From this figure it can be seen that most of the BDT predictions are in good agreement with the measured peak impact forces, while the biggest relative error is due to inability of the BDT to extrapolate data. Namely, classical decision trees, and also boosted decision trees, can not predict values outside of the historically observed range of values. Thus, they do not have extrapolating capabilities. In Fig. 5, the BDT relative error histogram, obtained using the whole available data set and divided in 20 bins is shown. From this figure, it can be seen that most of the measured peak impact forces are predicted with less then 5% relative error, while as in the previous case, bigger relative errors arise form inability of the BDT to extrapolate predictions.

Fig. 4. Comparison of measured peak impact force (red cross) and the peak impact force predictions of the boosted decision tree (blue circle). The biggest relative prediction error is indicated by the black diamond.

242

N. Kovincic et al.

Fig. 5. Relative error histogram divided in 20 bins.

4

Conclusion

In this paper, a methodology for evaluating the safety of a quasi-static impact in human-robot collaboration is proposed. The basic hypothesis of this paper is that a physics guided boosted decision tree can be trained to predict the peak impact force for a given impact velocity, robot configuration and an impact point on the robot. Based on the mathematical model presented in [14] and the experimental peak impact force measurements published in [17], a physics guided boosted decision tree was trained using Matlab’s LSBoost algorithm and the results shown in Fig. 5 and 4 support the hypothesis. Further, due to the rotational symmetry around the base joint, once trained BDT can predict the peak impact force for a fixed orientation in the whole horizontal plane. However, since all experimental measurements correspond to the back of a non-dominant hand, the trained BDT can only be used for this part of a human body. Furthermore, the trained BDT can be used to estimate the maximum allowed impact velocity such that the obtained peak impact force can be considered as a safe one. Namely, the BDT can be used to predict peak impact forces for a whole set of different impact velocities. From these predictions, it is easy to evaluate the safe ones and to determine the maximal impact velocity such that the impact is safe for a human. Finally, although the trained BDT has small prediction error, classical decision trees and also boosted decision trees can not predict values outside of the historically observed range and thus do not have extrapolating capabilities. This can be seen in Fig. 4 were the biggest relative error is caused by the inability of the BDT to extrapolate data.

A Boosted Decision Tree Approach for a Safe Human-Robot Collaboration

243

References 1. Technical Specification: ISO/TS 15066:2016, Robots and robotic devices: collaborative robots. Technical report, The International Organization for Standardization (2016) 2. Ajoudani, A., Zanchettin, A.M., Ivaldi, S., Albu-Sch¨ affer, A., Kosuge, K., Khatib, O.: Progress and prospects of the human-robot collaboration. Auton. Robots 42(5), 957–975 (2018) 3. Ech´ avarri, J., Ceccarelli, M., Carbone, G., Al´en, C., Mu˜ noz, J.L., D´ıaz, A., MunozGuijosa, J.M.: Towards a safety index for assessing head injury potential in service robotics. Adv. Robot. 27(11), 831–844 (2013) 4. Forssell, U., Lindskog, P.: Combining semi-physical and neural network modeling: an example of its usefulness. IFAC Proc. Vol. 30(11), 767–770 (1997) 5. Garofalo, G., Mansfeld, N., Jankowski, J., Ott, C.: Sliding mode momentum observers for estimation of external torques and joint acceleration. In: IEEE International Conference on Robotics and Automation (ICRA) (2019) 6. G´eron, A.: Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly Media, Sebastopol (2019) 7. Haddadin, S., Albu-Sch¨ affer, A., Hirzinger, G.: Safety evaluation of physical human-robot interaction via crash-testing. In: Robotics: Science and Systems, vol. 3, pp. 217–224. Citeseer (2007) 8. Haddadin, S., De Luca, A., Albu-Sch¨ affer, A.: Robot collisions: a survey on detection, isolation, and identification. IEEE Trans. Rob. 33(6), 1292–1312 (2017) 9. Haddadin, S., Haddadin, S., Khoury, A., Rokahr, T., Parusel, S., Burgkart, R., Bicchi, A., Albu-Sch¨ affer, A.: On making robots understand safety: embedding injury knowledge into control. Int. J. Robot. Res. 31(13), 1578–1602 (2012) 10. Hoo, K.A., Sinzinger, E.D., Piovoso, M.J.: Improvements in the predictive capability of neural networks. J. Process Control 12(1), 193–202 (2002) 11. Jia, X., Willard, J., Karpatne, A., Read, J., Zwart, J., Steinbach, M., Kumar, V.: Physics guided RNNs for modeling dynamical systems: a case study in simulating lake temperature profiles. In: Proceedings of the 2019 SIAM International Conference on Data Mining, pp. 558–566. SIAM (2019) 12. Karpatne, A., Atluri, G., Faghmous, J.H., Steinbach, M., Banerjee, A., Ganguly, A., Shekhar, S., Samatova, N., Kumar, V.: Theory-guided data science: a new paradigm for scientific discovery from data. IEEE Trans. Knowl. Data Eng. 29(10), 2318–2331 (2017) 13. Karpatne, A., Watkins, W., Read, J., Kumar, V.: Physics-guided neural networks (PGNN): an application in lake temperature modeling. arXiv preprint arXiv:1710.11431 (2017) 14. Kovincic, N., Gattringer, H., M¨ uller, A., Weyrer, M., Schlotzhauer, A., Kaiser, L., Brandst¨ otter, M.: A model-based strategy for safety assessment of a robot arm interacting with humans. PAMM 19(1), e201900247 (2019) 15. Mansfeld, N., Hamad, M., Becker, M., Marin, A.G., Haddadin, S.: Safety map: a unified representation for biomechanics impact data and robot instantaneous dynamic properties. IEEE Robot. Autom. Lett. 3(3), 1880–1887 (2018)

244

N. Kovincic et al.

16. Ren, H., Stewart, R., Song, J., Kuleshov, V., Ermon, S.: Learning with weak supervision from physics and data-driven constraints. AI Mag. 39(1), 27–38 (2018) 17. Schlotzhauer, A., Kaiser, L., Wachter, J., Brandstotter, M., Hofbaur, M.: On the trustability of the safety measures of collaborative robots: 2D Collision-force-map of a sensitive manipulator for safe HRC. In: IEEE International Conference on Automation Science and Engineering (CASE) (2019) 18. Stewart, R., Ermon, S.: Label-free supervision of neural networks with physics and domain knowledge. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

Combining the Robot Operating System with Building Information Modeling for Robotic Applications in Construction Logistics Camilla Follini1,2(&), Michael Terzer2, Carmen Marcher1,2, Andrea Giusti2, and Dominik Tobias Matt1,2 1

Faculty of Science and Technology, Free University of Bozen-Bolzano, Bolzano, Italy [email protected] 2 Fraunhofer Italia Research s.c.a.r.l., Bolzano, Italy

Abstract. Logistics in construction sites represents one of the main causes of musculoskeletal disorders for workers in the long run. For this reason, logistics represents a task that is highly suitable for automation. However, the application of robotics in construction is hindered by the on-site environment, which is often unstructured and subject to frequent change. To tackle this problem, the system proposed in this paper is a collaborative robotic platform aimed at humancentered applications on-site. The platform follows an operator while it carries heavy loads, such as materials and equipment, and it stops when it is near to the assisted worker. The system does not only avoid obstacles that are detected by its sensors, but it also navigates thanks to its knowledge of geometric and semantic information of the building project. This is achieved through a bridge between the Robot Operating System and the project data contained in the Building Information Model. The proposed system was developed and tested in laboratory, where it followed an operator while avoiding dynamic obstacles and areas marked in the building project file. This system represents a step forward towards the application of on-site collaborative construction robots. Keywords: Industry 4.0
Robot Operating System
Building information modeling
Logistics
Collaborative robotic platform
Construction robotics

1 Introduction The construction industry represents one of the main economies worldwide. In the European Union, it generates about the 9% of the gross domestic product [1]. Nonetheless, construction is often plagued by several problems such as inefficiency, low productivity, a high number of accidents, low attractivity, and a general lack of innovation [2]. This becomes even more evident when compared to other industries

C. Follini and M. Terzer—Contributed equally to this work. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 245–253, 2020. https://doi.org/10.1007/978-3-030-48989-2_27

246

C. Follini et al.

pushing towards Industry 4.0 [3], such as manufacturing, where collaborative robotic solutions are arising [4]. Logistics can be considered one of the harshest jobs on construction sites. In indoor tasks, it consists almost entirely of handling operations of heavy loads, such as lowering, lifting, and carrying. These operations are usually boring, repetitive, fatigue-intensive, and amongst the riskiest factors of musculoskeletal disorders in construction [5]. Moreover, logistics is considered work to be minimized according to lean construction standards. That is, because it represents necessary work that produces no added value in the construction supply chain [6]. For these reasons, automation of logistics operations on-site is a promising research area. To this end, some solutions have explored the use of Mobile Robotic Platforms (MRPs) navigating the construction site while carrying loads. Japanese construction companies, in particular, have been developing systems ranging from one to several hundred kilograms in payload [7]. A notable example is Shimizu’s Robo-Carrier, a forklift that autonomously navigates the construction site to transport material to other robotic systems developed by the company in charge of the assembling [8]. The knowledge of the building geometry is particularly important for robotic navigation in construction sites and for localization of relevant logistics areas (such as temporary storage points and site warehouse). Among the different software tools for project design, Building Information Modeling (BIM) is steadily spreading in the construction industry [9]. Its main advantage over more traditional tools is that it enables the linking of information to geometric 3D objects. For instance, a 3D object can store information about its material, cost, producer, and construction scheduling time. When connected to scheduling information, the model is referred to as “4D BIM” [9]. Given this, BIM has been used as basis for more than its original purpose as project planning tool, and it has been researched as enabler for autonomous mobile robots on-site. For instance, research has been carried out on algorithm-based route planning in 2D [10] and 3D [11], selecting the most convenient path in buildings. BIM has also been linked with computer vision in view of object recognition. For example, in [12] it is suggested that recognition from site point clouds is effective with objects with a high Level of Detail (LOD). Based on these advancements, the use of BIM has been extended to robotic navigation. For instance, object-recognition-based navigation has been proposed in [13]. In this research, real-life pictures are segmented and compared with a digital 3D object repository. Upon recognition of the object and its real-world scale, the system updates the BIM file of the building with the library object. The connection of BIM to the Robot Operating System (ROS) [14] is explored in [15]. In that research, a 4D-BIM file is used to retrieve waypoints where the construction progress must be monitored. After planning a route comprising all the waypoints, the MRP navigates the site autonomously to collect data. Obstacle detection has been implemented to avoid collisions due to unforeseen changes or mapping misalignments. Compared to the literature above, our solution focuses on a system for site logistics that is aimed at collaborating with human workers. In particular, our proposed approach tackles two relevant barriers for employment of MRPs on-site: safety and acceptance [16]. Concerning the latter, the interface of the system has been developed to be intuitive and useable by workers with little experience of robotic devices. Therefore, the interface requires very little input from the workers’ side. The system reads their position, and it follows them autonomously while it carries heavy materials. Safety is

Combining the Robot Operating System with Building Information Modeling

247

ensured by the system stopping when it finally reaches the operator. The robustness of the system is increased through dynamic obstacle avoidance and fusion of data detected by the sensing capabilities of the robot and BIM.

Fig. 1. Conceptual representation of the system main goals

2 Proposed Approach A barrier for the implementation of MRPs in construction is the site environment, which is often unstructured and subject to continuous change. At the same time, a complete description of the geometry of the building and the scheduling of tasks exists before the start of construction works, sometimes under a common repository that is the BIM file. We aim at the development of a robotic system that takes advantage of 3D and 4D BIM digital data to navigate the construction site while it follows an operator and it carries heavy materials or equipment (see Fig. 1). The data is overlapped to the map already generated by the system’s LIDAR sensors, and it helps the system avoid obstacles that cannot be recognized by them, such as holes in the ground and overhanging objects. The data is transmitted to the platform in the form of a ROS 2D costmap. Here, elements are filtered not only by their potential of collision with the system, but also by their scheduled construction time. To simplify the problem, the following assumptions were considered: 1) a BIM file exists which has parameters for scheduling and whose level of detail is high enough for navigation; 2) the MRP has means to determine the current date; 3) the MRP can locate its real-world starting point in the BIM file. The proposed method has been conceptually divided into three main parts: 1) the follow-me function, 2) the dynamic obstacle detection, and 3) the extraction of data from the BIM file to ease navigation.

3 Implementation The follow-me function on the MRP has been developed using an indoor tracking system. Indoor tracking systems deliver highly accurate measurements of position and orientation of the tracked bodies. Therefore, they are suited for this preliminary setup,

248

C. Follini et al.

and the data obtained from them can represent a ground truth for further development. The desired trajectory of the follow-me function gets calculated by tracking the position of both the MRP and the operator. These two are tracked thanks to spherical reflective markers that have been fixed respectively on the platform and on a construction helmet. The relative positions are then translated to a world coordinate system, which is defined by the tracking system. The implementation of dynamic obstacle avoidance has been realized through two planar LIDAR sensors. These sensors provide the possibility to measure distances on a 2D plane around them. BIM is used to retrieve information on objects unidentifiable or difficult to identify by the LIDAR sensors, and on planned building geometries as well as metadata. The metadata considered concern scheduling information, in particular the date in which construction of objects would start and end. To achieve this, objects are extracted from an Industry Foundation Classes (IFC) file, which is the open format that can be generated by every BIM software tool. The data extracted from this file is translated to a costmap that can be used inside of the ROS navigation stack. The map is then transmitted and superimposed to the one generated by the LIDAR sensors. Before the real system was tested, several simulations have been performed. The setup consisted of a PC running ROS and Gazebo [17], a ROS-enabled software for robotic simulation. Since the producer of the MRP delivers CAD models and Unified Robot Description Format (URDF) descriptions for their product, a realistic simulation within Gazebo suited the authors’ needs. The system has then been tested in a protected indoor environment, in view of creating a solid basis to further tailor it to the use in unstructured environments. 3.1

Hardware

The tracking system employed is produced by the company ART1. It consists of four tracking cameras, a computing unit in form of a rack, and several marker configurations. The MRP is a Husky A200 from the company Clearpath2. This vehicle is specifically designed for research activities in harsh outdoor environments. The user bay of the MRP is equipped with a computing unit running ROS and a network switch. The MRP is equipped with two LIDAR Sensors, respectively pointing forward and backwards, with a 270° planar field of view each. In addition, the MRP is equipped with an Inertial Measurement Unit (IMU) that provides acceleration and angular velocity measurements in the cartesian coordinate system. The control PC is a Dell Latitude E6420 running Linux Ubuntu 16.04 and ROS Kinetic. This PC is needed to launch the overall control architecture, as well as the bridge to the tracking system. The LIDAR sensors are connected over an ethernet interface to the computer of the MRP. The MRP includes a local network, that has a wireless connection to the external network. The external network consists of a router that connects the computing unit of the tracking system and the ROS-led PC. Since the system includes two computer running ROS, a time synchronization of the two systems is required. This

1 2

ART company website: https://ar-tracking.com/. Clearpath company website: https://clearpathrobotics.com/.

Combining the Robot Operating System with Building Information Modeling

249

synchronization is realized over the NTP deamon chrony3, running both on the MRP and the control PC. An overview of the different hardware components can be seen in the set-up shown in Fig. 2.

Fig. 2. Picture of the laboratory set-up for the testing phase (Image adapted from: live-style.it)

3.2

System Architecture in ROS

The overall software architecture of the system is shown in Fig. 3. Here, nodes have been distinguished between those that have been developed by the authors and those available as open-source nodes or ROS packages. The PC running on the MRP runs one of the master nodes named roscore. This node interfaces the LIDAR-scannergenerated messages (/laser) and a node that controls the MRP (husky_ctrl). A second roscore is run by the control PC. It enables: 1) the art_bridge node, which interfaces the data coming from the tracking system to tf-messages [18] in ROS, 2) a controller node for the follow-me algorithm, 3) the nav2d4 package, visualized as Operator-node in Fig. 2, that includes nodes for various purposes: (such as mapping managing, trajectory planning, and obstacle avoidance), and 4) a node for the map server, which feeds the static map coming from the BIM-to-ROS toolchain into the global costmap.

3 4

Chrony official website https://chrony.tuxfamily.org/. Nav2d page on ROS wiki: http://wiki.ros.org/nav2d.

250

C. Follini et al.

Fig. 3. Software architecture based on ROS

3.3

Navigation and Integration of IFC Data

The art_bridge node receives the data from the ART tracking system over a User Datagram Protocol (UDP) interface, and it composes a tf message according to each coordinate frame received. This use of tf-frames within ROS enables their visualization (e.g. in RViz [19]), as well as the use of related coordinate system transformation tools. The tracking system uses a custom data format for each tracked coordinate frame. The follow_me_ctrl node enables the follow-me algorithm. The inputs of this node are the art_bridge tf messages, while the output is a custom message type cmd, defined by the nav2d package. The node registers the current position of the vehicle, and it computes the vector to the target, which is defined by the coordinate frame of the tracked helmet. Thanks to that, the desired turn value and speed get calculated. When the MRP is close to the target, the system sets its velocity to 0 to stop the MRP in the near proximity of the operator. The BIM_to_ROS node is the one used to superimpose the BIM-generated map to the one based on LIDAR information. The node is based on the IFCOpenShell5 library. It first parses an IFC file to read and store the geometry and the metadata of BIM objects, grouped under their building floor. It then filters objects relevant to the navigation based on their IFC type (e.g. IfcDoor, IfcWall, etc.). Filtering is also dependent on objects geometry (i.e. whether it imposes a risk of collision with the MRP), and their planned starting and ending construction date. Thereafter, the objects that are not filtered out are sectioned by a plane. The result is a series of meshes representing the intersection between the 3D objects and the sectioning plane. This section is finally printed as a picture, where white pixels represent the empty space

5

IFCOpenShell library website: http://ifcopenshell.org/.

Combining the Robot Operating System with Building Information Modeling

251

whereas black pixels the obstacles. This picture is translated to a ROS costmap by the creation of a yaml file, which contains further mapping information, such as the map scale and origin. The costmap is sent to the navigation control node through the map_server6, and it can be activated by enabling the static map on RViz. The BIMbased map is superimposed to the map dynamically generated by the LIDAR sensors. The result of this superimposition is shown in Fig. 4. Here, a simple IFC file was used, which contained only the lab boundaries (i.e. walls and access door) and an element in the center of the room representing a hole in the ground. This element has been modeled in the BIM file as a skylight that was planned to remain uncovered in a certain timeframe, which contained the experiment date. The hole on the floor level has been chosen because it is an obstacle that cannot be detected by the 2D LIDAR sensors. The same process can be used to highlight small obstacles as well as to create off-limits areas, where undetectable elements have been posed on the ground and should not be moved by the MRP (e.g. electrical cables, measuring tapes). We consider other short obstacles that can be commonly found on construction sites, such as debris or unlevelled floor parts, to be negligible. That is, because we expect that the MRP can drive over them, since the LIDAR is positioned under the plane passing through the center of the wheels. As shown in Fig. 4, there is no distinction between the BIM-generated obstacles and the ones dynamically detected by the LIDARs. Therefore, the MRP adjusts its trajectory to avoid both.

Fig. 4. Visualization on RViz of the laboratory costmap without (a) and with (b) the static IFC map overlapping. As shown in Fig. 2, the obstacle in the center of the room is strictly virtual, and its position on the lab floor has been marked with tape.

6

Map_server page on ROS wiki: http://wiki.ros.org/map_server.

252

C. Follini et al.

4 Discussion and Conclusion The paper presents the development of a human-centered assistive system to support with logistics on the construction site. It enables an outdoor-capable MRP to follow an operator while avoiding obstacles both detected by its sensors and extracted from a BIM file of the building. Further development will foresee the use of sensors that can be mounted on the MRP, without requiring a fixed tracking system to identify the position of the user. In this work we started from a bottom-up research strategy. Indoor tracking systems deliver highly accurate measurements of position and orientation of the tracked bodies. Therefore, they are suited for a preliminary setup, and the data obtained from them can represent a ground truth for further development. However, this will introduce uncertainty in the detection of the relative position between operator and MRP and requires further development of the follow-me function that assists the system in path planning in case the operator disappears from the MRPs field of view. Additionally, future work will evaluate the feasibility of a bi-directional connection to IFC files, where information on geometries and metadata can be updated to be readable on a BIM software tool. Further work will also address the evaluation of the actual applicability of the system on a real construction site. This environment is characterized by many more challenges for robotic systems compared to manufacturing factories. Among these, the most relevant are the unstructured space subject to weather conditions, the very rough tolerance of operation (often on a scale of centimeters compared to the plan), the collaboration with workers not used to robotic systems, and finally the economic competition with cheap manual labor. To this end, the applicability potential and the actual decision of adopting this type of advanced system on a construction site can be supported with a dedicated decision support system (DSS) for equipment selection as described in [20].

References 1. European Commission: Construction. https://ec.europa.eu/growth/sectors/construction_en. Accessed 06 Dec 2019 2. European Construction Sector Observatory: Analytical Report - Improving the human capital basis (2017) 3. Kagermann, H., Wahlster, W., Helbig, J.: Recommendations for implementing the strategic initiative INDUSTRIE 4.0 (2013). https://www.din.de/blob/76902/e8cac883f42bf28536e7e8 165993f1fd/recommendations-for-implementing-industry-4-0-data.pdf 4. Di Cosmo, V., Giusti, A., Vidoni, R., Riedl, M., Matt, D.T.: Collaborative robotics safety control application using dynamic safety zones based on the ISO/TS 15066:2016. In: Berns, K., Görges, D. (eds.) Advances in Service and Industrial Robotics, pp. 430–437. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-19648-6_49 5. CPWR: Chart Book (6th edition): Fatal and Nonfatal Injuries - Musculoskeletal Disorders in Construction and Other Industries. https://www.cpwr.com/chart-book-6th-edition-fatal-andnonfatal-injuries-musculoskeletal-disorders-construction-and-other. Accessed 09 Dec 2019

Combining the Robot Operating System with Building Information Modeling

253

6. Pérez, C.T., Fernandes, L.L.A., Costa, D.B.: A Literature Review on 4D Bim for Logistics Operations and Workspace Management. Presented at the 24th Annual Conference of the International Group for Lean Construction, Boston (2016) 7. Bock, T., Linner, T.: Site logistics robots. In: Construction Robots. Elementary technologies and Single-Task Construction Robots (2016) 8. Shimizu Corporation: A Production System for a New Era. https://www.shimz.co.jp/en/ topics/construction/item12/. Accessed 07 Apr 2018 9. Harris, B.N.: Building Information Modeling: A Report from the Field. Building Information Modeling (2016) 10. Schlette, C., Roßmann, J.: Sampling-based floor plan analysis on BIMs. Presented at the 33rd International Symposium on Automation and Robotics in Construction, 21 July (2016). https://doi.org/10.22260/ISARC2016/0004 11. Lin, W.Y., Lin, P.H., Tserng, H.P.: Automating the generation of indoor space topology for 3D route planning using BIM and 3D-GIS techniques. Presented at the 34th International Symposium on Automation and Robotics in Construction, 1 July (2017). https://doi.org/10. 22260/ISARC2017/0060 12. Chen, J., Fang, Y., Cho, Y.K.: Performance evaluation of 3D descriptors for object recognition in construction applications. Automat. Constr. 86, 44–52 (2018). https://doi.org/ 10.1016/j.autcon.2017.10.033 13. Ferguson, M., Law, K.: A 2D-3D object detection system for updating building information models with mobile robots. In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa Village, HI, USA, pp. 1357–1365. IEEE (2019). https://doi.org/ 10.1109/WACV.2019.00149 14. Quigley, M., Conley, K., Gerkey, B.P., Faust, J., Foote, T., Leibs, J., Wheeler, R.C., Ng, A. Y.: ROS: an open-source robot operating system. In: ICRA 2009 (2009) 15. Ibrahim, A., Sabet, A., Golparvar-Fard, M.: BIM-driven mission planning and navigation for automatic indoor construction progress detection using robotic ground platform. Presented at the 2019 European Conference on Computing in Construction, 10 July (2019). https://doi. org/10.35490/EC3.2019.195 16. Czarnowski, J., Dąbrowski, A., Maciaś, M., Główka, J., Wrona, J.: Technology gaps in human-machine interfaces for autonomous construction robots. Automat. Constr. 94, 179– 190 (2018). https://doi.org/10.1016/j.autcon.2018.06.014 17. Koenig, N., Howard, A.: Design and use paradigms for Gazebo, an open-source multi-robot simulator. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), vol. 3, pp. 2149–2154 (2004). https://doi.org/10.1109/ IROS.2004.1389727 18. Foote, T.: tf: The transform library. In: 2013 IEEE Conference on Technologies for Practical Robot Applications (TePRA), Woburn, MA, USA, pp. 1–6. IEEE (2013). https://doi.org/10. 1109/TePRA.2013.6556373 19. Kam, H.R., Lee, S.-H., Park, T., Kim, C.-H.: RViz: a toolkit for real domain data visualization. Telecommun. Syst. 60, 337–345 (2015). https://doi.org/10.1007/s11235-0150034-5 20. Marcher, C., Giusti, A., Schimanski, C.P., Matt, D.T.: Application of decision support systems for advanced equipment selection in construction. In: Cooperative Design, Visualization, and Engineering, pp. 229–235. Springer, Cham (2019). https://doi.org/10. 1007/978-3-030-30949-7_26

Vehicle Dynamics and Control

Design of a Thunniform Swimming Robot in a Multiphysics Environment Daniele Costa1(&), Giacomo Palmieri1, Matteo Palpacelli1, and David Scaradozzi2 1

DIISM, Polytechnic University of Marche, Ancona, Italy {d.costa,g.palmieri,m.palpacelli}@univpm.it 2 DII, Polytechnic University of Marche, Ancona, Italy [email protected]

Abstract. Bio-inspired solutions devised for Autonomous Underwater Robots are currently investigated by researchers worldwide as a source of propulsive improvement. Despite the efforts made to pursue the substantial potential payoffs of marine animals’ locomotion, the performances of biological systems are still far to reach. In order to address this very ambitious objective, the authors of this work have designed and manufactured a series of ostraciiform swimming robots in the last three years. However, the aim to pursue the highest propulsive efficiency to maximize the robot autonomy, has driven them to improve their design by moving from ostraciiform to thunniform locomotion. In order to properly size the robot bio-inspired thruster – i.e. the caudal fin – and its actuation system, the performances of a flapping foil have been deeply investigated by means of computational fluid dynamics techniques. The numerical predictions led to the optimal design of a transmission mechanism capable to convert the continuous rotation of a single motor in the harmonic rototranslation of the robot thruster, a motion law closely resembling the fin kinematics in thunniform locomotion. Furthermore, in order to compute the robot resulting motion, the propulsive forces and torque generated by the flapping thruster have been integrated in a multibody model which accounts both for the mass distribution of the robotic fish and the hydrodynamic forces due to the relative motion between its segmented body and the surrounding fluid. The performed dynamic analysis allowed to compute the robot total efficiency in the cruising condition. Keywords: Biomimetics
Efficient thrusters Multibody systems
Underwater robots


Multiphysics simulations


1 Introduction Autonomous Underwater Vehicle, or AUVs, are robotic systems capable to move in immersion and complete a planned mission autonomously, thus without the presence of human operators nearby or on a ship close to the working site. While the sensors allow AUVs to examine the surrounding environment and compute their position, the propulsive system allows them to move and thus perform their mission. Besides a hull shape optimization, which in turn reduces the resistance forces that the water exercise © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 257–265, 2020. https://doi.org/10.1007/978-3-030-48989-2_28

258

D. Costa et al.

on the AUV, a more efficient propulsive system allows saving power and thus increases the vehicle autonomy under the same onboard battery payload. In order to improve the propulsive performances of AUVs, bio-inspired solutions are currently investigated as a source of efficiency and manoeuvrability. Although the comparative biomechanics of moving through water has attracted the attention of biologists and engineers for almost a century, the last two decades have witnessed a significant growth in the study of aquatic animal locomotion. The attempts to design AUVs capable of moving like marine animals are due to the superior performances of biological swimmers. Hence, several prototypes of bio-inspired robots have been manufactured worldwide and an extensive review is given in [1]. The possibility to replicate the swimming modes matured over thousands of years of evolution relies on the understanding of the fluid mechanics principles of aquatic locomotion. According to swim mechanics, marine animals propel themselves by undulating their bodies, tails and fins following specific patterns called swimming modes. The highest maneuverability is achieved when the whole body undulates with respect to the surrounding water, like in anguilliform locomotion. On the other hand, propulsive efficiency is maximal when the body motion is confined to the rigid caudal fin of thunniform swimmers, like tunas and scombrids. Here, the fin performs a rototranslation motion called flapping, tracing an undulating path as the fish moves forward, in order to adjust its angle of attack and thus prevent the flow separations which are commonly associated to efficiency loss [2]. Therefore, since the aim of the authors is to pursue the highest propulsive efficiency, thunniform locomotion has been adopted to design the bio-inspired robot presented in this paper. First, in order to properly size the vehicle propulsive system, the thrust force generated by its bio-inspired thruster – i.e. the flapping caudal fin – must be accurately quantified. To this end, computational fluid dynamics analysis (CFD) represents an invaluable tool to estimate the hydrodynamic forces and torque resulting from the fluidstructure interactions occurring during swimming [3]. In this paper, the CFD numerical predictions led to the design of a transmission mechanism capable to convert the continuous rotation of a single motor in the harmonic roto-translation of the robot thruster, as in thunniform locomotion. However, in order to size the vehicle propulsive system, the robot dynamics must be properly computed. In this paper, the propulsive forces obtained by CFD analysis have been integrated in a multibody model which accounts both for the mass distribution and for the hydrodynamic effects, like added mass and viscous damping, applied to the vehicle body. As a matter of fact, multibody techniques can be easily adapted to a large class of bio-inspired underwater vehicles, due to their rigid forebody and multijoint robotic tails. In other words, CFD and multibody simulations actually represent the backbone of a Multiphysics Framework usable to size and optimize the propulsive system and the mass distribution of a given robot, once a certain number of requirements and conditions has been defined [4]. The rest of this paper is organized as follows: Sect. 2 details the CFD analysis performed on the flapping fin, while Sect. 3 is dedicated to the transmission system designed to generate the flapping motion. Finally, Sect. 4 shows the results of the multibody simulations which led to the sizing of the propulsive system. Conclusions and future work are discussed in Sect. 5.

Design of a Thunniform Swimming Robot in a Multiphysics Environment

259

2 Computational Fluid Dynamics Analysis The numerical analysis have been performed using MIGALE, a research code based on the Discontinuous Galerkin space discretization [5]. In this work, a moving reference frame has been employed to account for the fin flapping motion. In this way, the computational complexity of the solving algorithm is reduced when compared to the dynamic mesh boundary condition commonly used in commercial codes. Fifth-order polynomials have been used to represent the solution within the mesh elements, resulting in a sixth-order space discretization, while the time step size has been set to ensure the results to be independent of the time discretization. The motion law of a flapping fin can be described by the following expressions: s ¼ s0 cosð2pftÞ

a ¼ a0 cosð2pft þ DÞ

ð1Þ

where a is the angle of rotation while s is the translation in a direction perpendicular to the swimming velocity U, as shown in Fig. 1a; then, a0 and s0 are the amplitudes of the harmonic functions in Eq. (1), f is their common frequency and D is the phase shift between the components of the flapping motion.

Efficiency η

90 70 50 30 10 0.0

0.2

s=c s=2.5c (a)

0.4

0.6

Strouhal Number s=1.5c s=3c

0.8

s=2c (b)

Fig. 1. Thunniform locomotion: (a) kinematics of a flapping fin, (b) propulsive efficiency as a function of the Strouhal number and translation amplitude with a0 = 20° and D = 90°.

Dealing with thunniform locomotion, the fin kinematics and the resulting propulsive performances depend on five parameters: rotation and translation amplitudes, frequency and phase shift of the associated harmonic functions, plus the position of the trace of the rotation axis along the foil chord. A detailed analysis of the results obtained by varying the aforementioned parameters in the numerical simulations is beyond the scope of the present paper. However, since the aim of the authors is to pursue the highest possible propulsive efficiency, comments regarding the latter are still provided. Particularly, Fig. 1b shows its sensitivity from the Strouhal number St, a fluid dynamics

260

D. Costa et al.

non-dimensional parameter used to characterize oscillating flow phenomena, at increasing values of the translation amplitude s0, indicated as a multiple of the fin chord length c. For a thunniform swimmer, St and propulsive efficiency are defined as: St ¼ 2s0 f =U

g ¼ hT iU=hPi

ð2Þ

where and

are respectively the time-averaged generated thrust and the mechanical power required to spin the fin. By observing Fig. 1b, it results clear that the greater s0, the higher the efficiency, at least when St is lower than about 0.35. Indeed, past the optimal value of St, the oscillation frequency becomes too high for the flow to remain attached to the moving fin: then separation occurs despite the adjustment of fin attack angle due to the translation component, resulting in an efficiency loss.

3 Functional Analysis According to the CFD predictions, the optimal values of the rotation amplitude a0 and phase difference D are respectively 19° and 85°. Regarding the translation amplitude, Fig. 1b has shown that as s0 grows, the efficiency peak rises up to 86%. These results have driven the design of the propulsive system of the robot proposed in this work. In [1], the authors have presented a transmission mechanism that converts the continuous rotation of the drive, at constant angular velocity, in a harmonic angular oscillation; the core of this solution is the spatial-cam kinematic joint in Fig. 2a: here, the output angle h and the input angle u are related by the following expressions: tan h ¼ h=L cos u

h  h=L cos u ¼ k cosð2pftÞ

k ¼ h=L  h0

ð3Þ

where h and L are the geometric parameters in Fig. 2a while f is the constant revolute frequency of the drive; Eq. (3) is verified if k is sufficiently smaller than one.

Fig. 2. Transmission systems: (a) spatial-cam joint, (b) proposed mechanism (n = 3).

Design of a Thunniform Swimming Robot in a Multiphysics Environment

261

The transmission system presented in this paper, designed to generate the harmonic roto-translation for the rigid fin of a thunniform swimmer, has the same layout of a nlink serial manipulator connected through revolute joints and driven by a single actuator, as shown in Fig. 2b where n = 3. Here, the nth link of the kinematic chain is composed by the robot fin. Each joint is driven by the spatial-cam joint showed in Fig. 2a. As stated before, a single motor installed on the robot forebody drives all the links of the chain: in other words, each cam is connected to a single shaft passing through the whole transmission system. In this way, each link oscillates with the same frequency f of the motor. However, in order to allow the shaft to comply with the rigid motion of the revolute pairs, while spinning at constant angular speed through the whole mechanism, a Double Cardan joint has been inserted along the shaft in correspondence of each revolute joint. This solution represent an improvement with respect to the system presented by the authors in [6], where flexible joints were used instead: indeed, Double Cardan joints are not limited to small deflection angles – 6 or 7° tops – as flexible joints. Therefore, the proposed solution allows larger translation amplitudes under the same mechanism total length. The translation displacement s of the robot fin and its rotation angle a can then be simply computed by means of the kinematic expressions in Eq. (3), as: s¼

nP 1

di sin

i¼1

i P

! hj



j¼1

a ¼ h1 þ . . . þ hn1 þ hn

nP 1

di

i¼1

i P

! kj cos u

hj ¼ kj cos u

j¼1

ð4Þ

hn ¼ kn cosðu þ dÞ u ¼ 2pft

where di are the lengths of the links, ki are their functional parameters as in Eq. (3) and d is the initial rotation of nth cam joint; the phase shift D between the translation and rotation motion components s and a of the fin is actually due to the angular displacement d. The approximation in the translation expression in Eq. (4) is valid when the sum of the functional parameters kj is reasonably smaller than one: this condition limits the rotation between the links. By comparing Eq. (4) and Eq. (1), it is possible to compute the unknown quantities kn and d, similarly to the procedure in [6]: Þ s0 ¼ kd nðn1 2

0 sin D tan d ¼ a0 acos Dnk

k2n ¼ a20 þ n2 k2  2na0 k cos D

ð5Þ

where it has been assumed that all the links have the same length and that all the cam joints, save the last one, have the same functional parameter k. In order to compute the optimal configuration for the proposed transmission mechanism, the following considerations, requirements and constraints need to be stated: A. considering the space necessary to house the shaft bearings, the cam and the Double Cardan joints, the minimum link length d is 100 mm; B. the optimal values of a0 and D have been already computed by CFD; C. the fin aspect ratio AR, meaning the ratio of its span b to its mean chord c, must be greater than three in order to match the two-dimensional flow condition in the numerical simulations [2];

262

D. Costa et al.

D. the greater s0, the higher the propulsive efficiency peak; E. the cruising speed U must be equal to twice the robot length per second [1]; F. the mean propulsive thrust generated by the robot fin must balance the drag force exerted by the fluid on the vehicle frontal section [1]. According to the stated considerations, the optimal configuration has been computed as a three-link (n = 3) serial mechanism where the maximal rotation of the first two revolute pairs is 19°, corresponding to k equal to 0.3; by setting the link length to the minimal value, the translation amplitude s0 is equal to 100 mm. Finally, dynamic balance plus the relation between the fin span and chord due to the aspect ratio condition allow to compute c = 52 mm and b = 156 mm. Therefore, the translation amplitude is twice the fin chord, which corresponds to an efficiency peak equal to 83.2% as shown in Fig. 1b. Figure 3 shows the assembly of the proposed transmission system.

Fig. 3. Assembly of the transmission system (tail and fin) and the robot forebody.

4 Multibody Analysis The design method employed in the previous Section is based on a force balance in the surge direction. While its main advantage is represented by its simplicity, the sizing procedure does not take into account the hydrodynamic forces acting on the robot body nor the damping effect due to recoil. Therefore, the computed values of the fin span and chord can be only considered as a preliminary result. However, those values can be used as the initial guess of the optimization method presented in this Section: here, the computational fluid dynamics characterization of the fin is integrated in a multibody model which accounts both for the vehicle mass distribution and for the hydrodynamic effects applied to the swimmer forebody and tail. As the first step of the sizing and optimization procedure, the numerical trends of the propulsive forces and torque coefficients due to the fin flapping motion have been parametrized as a function of the Strouhal number: in this way, the performances of the robot thruster are expressed by a single parameter which take into account both the cruising velocity U and the motor revolute frequency f. As stated in the previous Section, the translation amplitude here has been set equal to three times the fin chord.

Design of a Thunniform Swimming Robot in a Multiphysics Environment

263

Then, the multibody model has been created: in this work, the dynamic analysis has been performed by using MSC Adams View. First, the robot CAD model showed in Fig. 3 has been imported in the multibody modelling environment; particularly, the links of the transmission system have been covered by rigid shells shaped like the tail of a thunniform swimmer, as already done with the forebody. Since the aim of the author is to optimize the vehicle propulsive system in the cruising condition, a planar joint has been applied to the robot center of mass in order to constraint its motion to the horizontal plane. Next, the hydrodynamic loads due to the fluid-structure interaction have been applied to the robot forebody and tail links. Namely, two distinct contributions have been considered: added mass and damping effects. A rigorous analysis of an incompressible flow would require the solution of the Navier-Stokes equations. However, since the robot velocity is of the order of its length per second, most of the hydrodynamic effects have no significant influence on the resulting motion. Consequently, the added mass and damping loads can be expressed by the linearized expression presented in [7]. Finally, the propulsive forces and torque vectors have been applied to the robot fin using run-time expressions to compute their modulus during the simulations: particularly, the coefficients predicted by CFD analysis, whose amplitude depends on the square of the Strouhal number, have been multiplied by the dynamic pressure and to the fin reference surface, i.e. its span times its chord. Consequently, the resulting expression of the propulsive forces is a linear function of the fin span b and a quadratic function of the motor revolute frequency f. This feature has been exploited in the following sizing step, whose final objective is to compute the optimal values of f and b which allow the robot to cruise at speed U equal to twice its body length per second, i.e. 1 m/s, when St corresponds to the propulsive efficiency peak, as shown in Fig. 1. Thus, starting by the value of b chosen in the previous Section, the vehicle dynamics has been solved in order to compute the average-steady cruising speed, achieved after the acceleration phase, at different frequency values. Then, the corresponding values of St have been calculated. The stated procedure has been iterated by varying the span length b, until the aforementioned optimization condition has been matched: average cruising speed equal to 1 m/s when the Strouhal number is equal to 0.2. Figure 4a shows the average velocity (black lines) as a function of the motor revolute frequency, computed at three growing values of the fin span. The corresponding trends of St (grey lines) are also presented. As expected, the cruising speed rises as b grows beyond the first guess value calculated in the previous Section. The charts shows that optimization condition is matched when the fin span is 170 mm: in this configuration, when the motor spins at 0.85 Hz, the cruising velocity reaches the design value while the Strouhal number matches the value corresponding to the maximal efficiency showed in Fig. 1b. Once the optimal fin geometry and kinematics have been properly computed, the driving torque can be calculated in order to choose a proper motor for the robot: the corresponding trend inside a fin oscillation period is shown in Fig. 4b. Here, the average mechanical efficiency, in the optimized cruising condition, is 67%.

D. Costa et al. 0.24

b=156 b=170 b=185

1.3 1.2

0.23 0.22

1.1

0.21

1

0.2

0.9

0.19

0.8

0.18 0.7

0.8

0.9

Frequency [Hz]

1

1.1

(a)

Motor Torque [Nm]

Cruising Speed [m/s]

1.4

Strouhal Number

264

1.2 1.0 0.8 0.6 0.4 0.2 0.0 -0.2

(b)

0.0

0.3

0.6 0.9 Time [s]

1.2

Fig. 4. (a) Optimization procedure, (b) Motor torque in the optimized configuration.

5 Conclusions Bio-inspired propulsive solutions have been investigated in the last two decades as a source of improvement for AUVs. In this paper, the authors used the numerical predictions on a flapping foil to design a transmission mechanism that converts the continuous rotation, at constant angular velocity, of a single motor, in the harmonic rototranslation of the caudal fin of a swimming robot, a motion law which corresponds to the highest propulsive efficiency in biological underwater locomotion. The adoption of the Double Cardan joints allows the transmission shaft to spin with the same speed through the whole mechanism, which in turn results in the required constant phase shift between the rotary and translational components of the fin motion. At the same time, Double Cardan joints allow greater angular deflections with respect to flexible joints, thus reducing the number of links required to obtain the same fin translation displacements and consequently, the mechanical efficiency loss due to the friction in the cam joints. Furthermore, when compared to a robotic tail driven by multiple servomotors, the proposed solution reduces the system inertia and encumbrance, while waterproofing issues are minimal because a single motor needs to be sealed. Finally, multibody technique has allowed to compute the optimal values of the fin geometry and its flapping frequency, as well as to size the system motor. The project is now at the beginning of the prototyping phase: a suitable CAD model has been drawn while Double Cardan joints of the required size have been identified among commercial components. A physical prototype of the proposed vehicle is currently being printed by means of high-resolution stereolithography. At the same time, the multibody model is undergoing extensive improvements in order to simulate attitude variations and depth changing manoeuvres.

References 1. Scaradozzi, D., Palmieri, G., Costa, D., Pinelli, A.: BCF swimming locomotion for autonomous underwater robots: a review and a novel solution to improve control and efficiency. Ocean Eng. 130, 437–453 (2017)

Design of a Thunniform Swimming Robot in a Multiphysics Environment

265

2. Sfakiotakis, M., Lane, D.M., Davies, J.B.C.: Review of fish swimming modes for aquatic locomotion. IEEE J. Oceanic Eng. 24(2), 237–252 (1999) 3. Chao, L.M., Pan, G., Zhang, D., Yan, G.X.: On the drag–thrust transition of a pitching foil. Ocean Eng. 192, 106564 (2019) 4. Amodio, D., Callegari, M., Castellini, P., Crivellini, A., Palmieri, G., Palpacelli, M., Paone, N., Rossi, M., Sasso, M.: Integrating advanced CAE tools and testing environments for the design of complex mechanical systems. In: Longhi, S., Monteriù, A., Freddi, A., Frontoni, E., Germani, M., Revel, G. (eds.) The First Outstanding 50 Years of “Università Politecnica delle Marche”, pp. 247–258. Springer, Cham (2019) 5. Bassi, F., Rebay, S.: Numerical evaluation of two discontinuous Galerkin methods for the compressible Navier-Stokes equations. Int. J. Numer. Meth. Fluids 40(1–2), 197–207 (2002) 6. Costa, D., Palmieri, G., Callegari, M., Scaradozzi, D.: Functional design of a biomimetic flapper. In: 4th IFToMM Symposium on Mechanism Design for Robotics (2018) 7. Fossen, T.I.: Marine Control System-Guidance, Navigation and Control of Ships, Rigs and Underwater Vehicles. Marine Cybernetics (2002)

Analysis of Tire Temperature Influence on Vehicle Dynamic Behaviour Using a 15 DOF Lumped-Parameter Full-Car Model Michele Perrelli1(&), Flavio Farroni2, Francesco Timpone2, and Domenico Mundo1 1

2

Department of Mechanical, Energy and Management Engineering, University of Calabria, Rende, Italy {michele.perrelli,domenico.mundo}@unical.it Department of Industrial Engineering, University of Naples “Federico II”, Naples, Italy {flavio.farroni,francesco.timpone}@unina.it

Abstract. The vehicle dynamic behaviour analysis is a crucial step for the evaluation of performance in terms of stability and safety. Tires play an important role by generating the interaction forces at each road-tire contact patch. The longitudinal and lateral dynamics are analysed by using instrumented vehicles with expensive high precision sensors to get a measurement of estimates of physical parameters of interest. This paper deals with the evaluation of vehicle under/oversteering behaviour and of braking performance using a Realtime (RT) simulator. The simulations were performed by using an efficient 15 Degrees of Freedoms (DOFs) Lumped-Parameter Full Vehicle Model (LPFVM), comprising a tire model with temperature-dependent properties. A virtual Driver-in-the-Loop (vDiL) scheme was used to perform test manouvers. The virtual driver is based on two PID regulators for speed and steering control. Finally, this paper reports the results of constant radius tests as defined by standard ISO4138 and of a braking manoeuvre. In both tests, a type-A road profile as defined by ISO 8608 standard was simulated. Keywords: Real-time simulation dynamics
Safety


Lumped-parameter modelling
Vehicle

1 Introduction Nowadays, the reduction of road accidents is one of the main challenges for vehicle designers. In the past decades, many ADASs (Advanced Driver Assistance Systems) have been developed to improve the safety and comfort of drivers, passengers and pedestrians [1, 2]. Modern ADASs are capable of detecting possible accidents and actuating, automatically, emergency procedures, such as automatic car stopping and lane-keeping [3]. Technologies such as active steering, differential braking, active suspension, Anti-Lock Braking System (ABS) and others, are based on the processing of the signals coming from dedicated sensors. Based on sensor feedback, ADAS can actuate automatic safety actions [4] acting on suspensions, brakes, throttle, steering © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 266–274, 2020. https://doi.org/10.1007/978-3-030-48989-2_29

Analysis of Tire Temperature Influence on Vehicle Dynamic Behaviour

267

wheel. Despite ADAS allow to improve the driving safety, their effectiveness depends on the friction forces arising at road-tires contact patches [5, 6]. Therefore, the knowledge of mechanisms of interaction between tires and the road surface is fundamental for the analysis of vehicle safety, handling and comfort performance. With reference to vehicle dynamics topic, computer simulations represent a powerful tool for researchers and designers. In literature, there are several studies focused on the integration of different control strategies (e.g., based on Artificial Neural Network, adaptive linear Kalman filters, machine learning models) with physics-based vehicle models of diversified complexity, from simple 2 DOF lumped parameter models for lateral dynamics up to detailed Multi-Body simulation models [7–11]. In this paper, the effects of tires temperature changes on lateral and longitudinal dynamics of a vehicle are evaluated using a 15 DOFs full vehicle model implemented with a PID-based virtual Driver-in-the-Loop (vDiL) scheme. The details of the proposed simulation environment can be founded in [12, 13]. To improve the accuracy of simulation results, all test manoeuvres were executed using a type-A road profile as defined by ISO 8608 standard [14]. The outline of the paper is the following: Sect. 2 describes the mathematical formulation used for tire-road contact force modelling; Sect. 3 provides an explanation of the procedure used to identify Pacejka Steady-State Magic Formula (MF) [15] parameters; Sect. 4 discusses the results of a braking manoeuvre and of a constant-steer test according to ISO4138 standard; Sect. 5 provides concluding remarks and an overview on future developments.

2 Road-Tire Contact Forces Computation for 15DOF Car Model The vehicle simulation platform is based on the following closed-loop scheme (Fig. 1).

Fig. 1. Information flow between submodels

268

M. Perrelli et al.

The simulation software, implemented in C++, enables the interaction between a human or a virtual driver with the LPFVM, to which the vehicle control inputs (throttle, gear, brake, clutch and steering wheel angle) are sent by using the UDP communication protocol. The LPFVM is based on a set of Ordinary Differential Equations (ODEs) governing the dynamic equilibrium of the vehicle chassis (three translational and three rotational equilibrium conditions) and of each wheel (translational equilibrium in the vertical direction and rotational equilibrium around the spindle axis). The LPFVM is divided into four main sub-models, namely car, powertrain, tire and steer. Each model computes a subset of vehicle state variables and exchanges information with the other sub-models (Fig. 1) to provide a prediction of the vehicle response to the driver’s control inputs. A description of the overall simulation platform is provided in [12], while the focus of this section is on tire submodel characteristics. The tire submodel uses inputs coming from internal computations of car submodel and computes vertical, longitudinal and lateral forces, longitudinal and lateral slip values, and torque around the spindle axis for each wheel. At each simulation step, the car submodel communicates to the tire the current value of the vertical position and velocity, the angular velocity of the four spindles, the longitudinal and the lateral velocity of each wheel. The tire subsystem sends back to car submodel the vertical, longitudinal and lateral forces acting on each spindle as well as the driving/braking torque values. The vertical force estimation is achieved by a simple spring-damper model. For each tire, the vertical force is calculated with the following equation: Fz ¼ Pk ðDz  PPL Þ þ Pc D_ z

ð1Þ

where Pk, PPL and Pc are, respectively, the tire vertical stiffness, the static tire deformation in the vertical direction and the tire damping coefficient. Dz, it is computed as: Dz ¼ Ds  Dr

ð2Þ

where Ds is the vertical displacement of the spindle and Dr is the road profile elevation. The longitudinal slip r and lateral slip a are computed for longitudinal and lateral force estimation as: xPr  Vx xPr    Vy a ¼ f
d  atan Vx r¼

ð3Þ ð4Þ

where, Vx, Vy, x, Pr and d are, respectively, the longitudinal and lateral velocity of the spindle, the angular velocity of the tire as estimated by the car submodel, the tire radius and the steering angle. In Eq. (4) f is a hyperbolic function of the longitudinal velocity defined as:

Analysis of Tire Temperature Influence on Vehicle Dynamic Behaviour

269

tanhð10Vx  8Þ þ 1 2

ð5Þ

f¼

This function was introduced to make the lateral slip negligible when the driver rotates the steering wheel in case of a stationary vehicle. Finally, the longitudinal and lateral forces are computed using a simplified version of Pacejka’s Magic Formula, as follows: Fx ¼ Fz Dx sinfCx  atan½ðBx r  Ex ðBx rÞ  atanðBx rÞg  

 Fy ¼ Fz Dy sin Cy  atan ðBy a  Ey By a  atan By a

ð6Þ ð7Þ

After forces have been calculated it is possible to calculate the torques on the tires due to road contact as follow: Troad ¼ Fx Pr

ð8Þ

The procedure used, to calculate MF coefficients, is described in Sect. 3.

3 Pacejka Coefficient Estimation Method The tire, as said, has been modelled adopting a simplified version of the MF, conceiving the sole pure tangential interactions and the macro-parameters. The reference tire parameters have been identified employing a sedan vehicle experimental data, processed by and innovative version of TRIP-ID tool [16], able to provide MF coefficients depending on temperature, replicating the variations on grip, stiffness and shape variations due to the effects of thermal phenomena on tread polymers and carcass behaviour [17].

Fig. 2. Location of the modelled tires performance on the grip/temperature diagram

As shown in Fig. 2, the reference tire has been hypothesized as working for the whole simulated run at an average temperature corresponding to a friction level located in the increasing part of the grip/temperature performance curve. The so-called “cold

270

M. Perrelli et al.

tire” is considered at a lower temperature, characterized by a friction level about 7% lower, and the “hot”, on the other side, at about 7% higher. At the same time, the braking stiffness variation for longitudinal interaction curve, and the cornering stiffness one for lateral, have been modelled as decreasing of about 15% from the reference for the cold tire, and increasing of the same amount for the hot one, obtaining the final results showed in Fig. 3. In such a preliminary stage, as said, the tire has been considered as running at different constant temperatures in each case, and neglecting the combined interaction and the effects due to camber variations. Next steps in the activity will concern the implementation of a real-time thermal model, useful to take into account the local temperature distribution in the radial direction across the different materials of the tire composite structure, responsible for the different behaviour of carcass in terms of stiffness, and of tread, mainly affecting frictional phenomena. Moreover, the further discretization along lateral direction will allow replicating the rib-by-rib adherence variations linked to footprint area modifications according to camber and dynamic stress acting during severe manoeuvres.

Fig. 3. Pure tangential tire interaction curves implemented at different temperature

Finally, the implementation of combined interactions of the self-alignment moment and MF micro-parameters will lead to the achievement of a highly reliable tire simulation, useful for vehicle setup optimization and to feed lap time prediction algorithms [18].

4 Implemented Test Manouvers The simulation environment described above was used to simulate two different test manouvers: a constant-steer test and a braking manoeuvre from 100 km/h to 0 km/h. The constant-steer test manoeuvre was defined according to the ISO4138 standard which specifies open-loop test methods for determining the steady-state circular driving behaviour resulting from several specific types of control inputs. The selected test method requires driving the vehicle at several speeds while keeping the steering-wheel

Analysis of Tire Temperature Influence on Vehicle Dynamic Behaviour

271

angle fixed. The details of the test are explained in Sect. 4.1. In both implemented manouvers, a type-A random road profile (Fig. 4), as defined by ISO 8608, was generated considering also a spatial offset between the front and rear tire-road contact patches, which corresponds to the value of the car wheelbase L.

Fig. 4. ISO 8608 Type-A random road profile

The ISO 8608 provides a classification of longitudinal road profiles based on power spectral density (PSD) of the vertical displacement [14]. 4.1

Constant Steer Test

To perform the constant steer test, the steering wheel angle d was set equal to the kinematic Ackerman value dAck_ref for a reference path radius of 100 m and the speed s of the virtual vehicle was steadily increased from 10 km/h to 250 km/h. The speed was increased with a step of 5 km/h, only after the vehicle travelled a full-circular path in steady-state conditions for each speed value. By keeping the steering angle fixed, as the longitudinal speed increases, the car travels on circular trajectories with an increasing radius. This test was conducted both using the reference, the hot and the cold tire parameters of MF. As shown in Fig. 4a, for the same speed and a fixed steering angle, using a warm tire allows driving the vehicle on circular paths having a smaller radius with respect to the reference and cold tire. If a cold tire is used instead, a higher radius for circular paths is obtained during the drive test simulations. Such effects are mainly linked to the frictional behaviour of the different tire specifications, highlighting the importance of the definition of the optimal thermal working range for adherence optimization. To evaluate the handling performance of the vehicle, once the path radii were calculated, the understeering gradient K was determined for each case using the following equation:

K¼

  L d  180 p RðsÞ ay ð s Þ

ð9Þ

272

M. Perrelli et al.

where R(s) was measured for each speed after steady-state conditions were reached and, then, lateral acceleration was calculated as follow: ay ðs) ¼

s2 R(s)

ð10Þ

Fig. 5. (a) Radius vs speed – (b) Understeer gradient

Figure 5b shows the trend of K as a function of lateral acceleration for hot, cold and reference tire. It is to be noticed that the car model shows an understeering behaviour (K > 0) in case of reference tires. The understeering gradient is almost two times as high as the one estimated for the case of cold tires in the whole velocity range. Using the hot tire, instead, due to the different balance between the two axles sideslip angles, the vehicle shows an oversteering behaviour as long as the lateral acceleration is lower than 0.43 g (K < 0) after which the vehicle becomes understeering (K > 0). 4.2

Braking Manoeuvre

Braking to a full stop is a driving manoeuvre regularly used in experiments to investigate driver behaviour or to test acceleration feedback. During a typical braking manoeuvre, the car is accelerated up to target speed and then decelerated until full stop is achieved. To perform this test, for all tire temperature levels, the car sped up to 100 km/h before start braking on a straight path. With reference to Fig. 6, during the acceleration phase, no relevant differences between the three temperatures of the tire were observed, so the car reached the target speed after about 13.4 s in all cases. A different car behaviour was observed instead during the braking manoeuvre. In fact, with respect of the reference tire, the full stop was achieved about 0.1 s earlier in case

Analysis of Tire Temperature Influence on Vehicle Dynamic Behaviour

273

Fig. 6. Speed vs time

of hot tires, characterized, as expected, by higher frictional performance in longitudinal interaction. In case of cold tires, we can observe a time delay about 0.4 s for the full stop achievement with respect to the reference tire, validating the effectiveness of the whole simulation architecture. For the reference, hot and cold tire, the braking distances were, respectively, about 58.3, 57.8 and 63.3 m with the corresponding braking times of 17.51, 17.41 and 17.91 s.

5 Conclusion The preliminary analysis carried out feeding a 15DOF lumped parameter full car model with different tires, modelled with a simplified version of MF formulation and replicating road interaction at three different temperature levels, and consequentially at different friction and stiffness conditions, allowed to verify the reliability of the simulation system, able to produce results, for ISO testing manoeuvres, agreeing with the expected physical responses. In particular, both longitudinal and lateral interactions provided trajectories and vehicle handling behaviour able to validate the effectiveness of the motion equations also in unexpected conditions. This results will lead to the following steps consisting in a more complex tire modelling both from the interaction and thermodynamic points of view. Acknowledgement. This work was supported by the project “FASTire (Foam Airless Spoked Tire): Smart Airless Tyres for Extremely-Low Rolling Resistance and Superior Passengers Comfort” funded by the Italian MIUR “Progetti di Ricerca di Rilevante Interesse Nazionale (PRIN) call 2017 - grant 2017948FEN.

References 1. Ragesh, N.K., Rajesh, R.: Pedestrian detection in automotive safety: understanding state-ofthe-art. IEEE Access 7, 47864–47890 (2019) 2. Bengler, K., Dietmayer, K., Farber, B., et al.: Three decades of driver assistance systems: review and future perspectives. IEEE Intell. Transp. Syst. Mag. 6(4), 6–22 (2014)

274

M. Perrelli et al.

3. Stark, L., Düring, M., Schoenawa, S., et al.: Quantifying Vision Zero: crash avoidance in rural and motorway accident scenarios by combination of ACC, AEB, and LKS projected to German accident occurrence. Traffic Inj. Prev. 20(sup1), 126–132 (2019) 4. Kukkala, V.K., Tunnell, J., Pasricha, S., Bradley, T.: Advanced driver-assistance systems: a path toward autonomous vehicles. IEEE Consum. Electron. Mag. 7(5), 18–25 (2018) 5. Edunyah, I.: Causes of tyre failure on road traffic accident; a case study of Takoradi Township. Int. J. Sci. Res. Publ. 6(2), 30–35 (2016) 6. Osueke, C.O., Uguru-Okorie, D.C.: The role of tire in car crash, its causes, and prevention. Int. J. Emerg. Technol. Adv. Eng. 2(02), 54–57 (2012) 7. Candreva, S., Mundo, D., Gubitosa, M., Toso, A.: On the correlation between a 3D highfidelity multi-body vehicle model and a 1D 15-DOFs vehicle concept model. In: Proceedings of ISMA 2014, pp. 1615–1627 (2014) 8. Sharifzadeh, M., Timpone, F., Senatore, A., et al.: Real time tyre forces estimation for advanced vehicle control. Int. J. Mech. Control 18(2), 77–84 (2017) 9. Jahnke, M.D., Cosco, F., Novickis, R., et al.: Efficient neural network implementations on parallel embedded platforms applied to real-time torque-vectoring optimization using predictions for multi-motor electric vehicles. Electron 8(2), 1–27 (2019) 10. Cosco, F., Naets, F., Desmet, W.: Use of concept modelling for online input force estimation. In: Proceedings of ISMA 2014, pp. 1639–1651 (2014) 11. Carpinelli, M., Gubitosa, M., Mundo, D., Desmet, W.: Automated independent coordinates’ switching for the solution of stiff DAEs with the linearly implicit Euler method. Multibody Syst. Dyn. 36, 67–85 (2016) 12. Mundo, D., Gencarelli, R., Dramisino, L., Garre, C.: Development, validation and RT performance assessment of a platform for driver-in-the-loop simulation of vehicle dynamics. In: Mechanisms and Machine Science, pp. 130–138 (2018) 13. Perrelli, M., Francesco, C., Giuseppe, C., Mundo, D.: Evaluation of vehicle lateral dynamic behaviour according to ISO-4138 tests by implementing a 15-DOF vehicle model and an autonomous virtual driver. Int. J. Mech. Control 20(2), 31–38 (2019) 14. Loprencipe, G., Zoccali, P.: Use of generated artificial road profiles in road roughness evaluation. J. Mod. Transp. 25, 24–33 (2017) 15. Pacejka, H.B.: Tire and Vehicle Dynamics. Butterworth-Heinemann, Oxford (2006) 16. Farroni, F., Lamberti, R., Mancinelli, N., Timpone, F.: TRIP-ID: a tool for a smart and interactive identification of Magic Formula tyre model parameters from experimental data acquired on track or test rig. Mech. Syst. Signal Process. 102, 1–22 (2018) 17. Farroni, F., Russo, M., Sakhnevych, A., Timpone, F.: TRT EVO: advances in real-time thermodynamic tire modeling for vehicle dynamics simulations. Proc. Inst. Mech. Eng. Part D J. Autom. Eng. 233(1), 121–135 (2019) 18. Farroni, F., Sakhnevych, A., Timpone, F.: A three-dimensional multibody tire model for research comfort and handling analysis as a structural framework for a multi-physical integrated system. Proc. Inst. Mech. Eng. Part D J. Autom. Eng. 233(1), 136–146 (2019)

Motion Sickness Minimisation in Autonomous Vehicles Using Optimal Control Zaw Htike, Georgios Papaioannou(B) , Efstathios Siampis, Efstathios Velenis, and Stefano Longo Cranfield University, College Road, Cranfield, Bedfordshire MK43 0AL, UK {zaw.htike,g.papaioannou}@cranfield.ac.uk Abstract. This paper presents the application of optimal control theory in order to minimise motion sickness in autonomous vehicles. By formulating an optimal control problem, the optimum velocity profile is sought for a predefined road path from a specific starting point to a final one. Specific and given boundaries and constrains are applied in order to minimise the motion sickness without compromising the journey time. For the representation of the motion sickness as a cost function to our optimal control problem, the illness rating is used. Different case studies are investigated by changing the cost functions of our problem for a fixed road trajectory of the vehicle in three road routes by varying the road curvature. According to the results, the road with flexible lateral manoeuvrability is found to provide lower sickness as well as shorter journey time when both motion sickness and journey time are considered in the cost function.

Keywords: Motion sickness

1

· Optimal control · Autonomous vehicles

Introduction

Autonomous vehicles are expected to push towards the evolution of mobility environment in the near future, by decreasing traffic accidents, traffic conjunction and vehicle fuel consumption. One of the main limitations they face is motion sickness (MS) and it seems capable of putting in risk fully automated vehicles (AVs) wide impact, as well as their acceptance by the public [1]. Based on experimental studies, the occurrence of motion sickness in road vehicles is most closely related to low-frequency fore-and-aft, lateral and yaw accelerations [2,3] and [4]. In particular, yaw acceleration increases when traversing corners or roundabouts, while fore-and-aft and lateral acceleration vary due to vehicle braking and centripetal effects, respectively. Also, the severity of the motion sickness depends on the exposure time and the intensity of these motions. In reality, a human driver is thought to drive in a manner that prevents motion sickness occurrence, because the driver, who does not want to get carsick, c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 275–282, 2020. https://doi.org/10.1007/978-3-030-48989-2_30

276

Z. Htike et al.

functions as a type of motion sickness detector or predictor [5]. The above argument depicts the main difference between passengers riding on an autonomous vehicle and a taxi. With the autonomous vehicles not having this driver’s function, the consideration of the minimisation of motion sickness in their implementation and design frameworks, is crucial in order to fully replace the driver. Moreover, in self-driving vehicles, the loss of controllability (handing over the vehicle controls to automation) will free up the occupant’s time, and then finally the complete transformation of the entire interior into a more leisurely and economically design could take place [3] and [5]. As a result of the above, the passengers will be able to engage more and more in either leisurely or economically productive (non-driving) tasks (i.e. reading, watching movies, working and playing games), which will make them unable to anticipate on the direction of travel increasing their susceptibility to motion sickness, its severity and its frequency [1] and [6]. In this respect, research into motion sickness on autonomous vehicles has just started to get attention. On contrary to the past when the cause, the function, the symptoms, and the estimation of motion sickness were the main issues, nowadays the answer to the question of how to minimise it has arisen. In this work, the attention will be turned in aspect of minimising motion sickness and journey time with optimal control theory. More specifically, for a predefined road path from a specific starting point to a final one, the optimum velocity profile is sought within specific and given boundary and constrains in order to minimise the motion sickness without compromising the journey time.

2 2.1

Methods and Material Vehicle and Road Tracking Model

For motion planning, the simplest vehicle model and most commonly used is the point mass model. The vehicle is simplified as a point mass model that can

Fig. 1. Curvilinear coordinates for road tracking

Motion Sickness Minimisation

277

accelerate within bounds, i.e. friction circle, which is also chosen in this work as a constraint [7]. The equations describing the point mass model are given below (1):  S¨x = x ¨ = ax , S¨y = y¨ = ay , ax 2 + ay 2  amax (1) where S denotes the position distance, ax , ay are the longitudinal and lateral accelerations, while amax is the maximum absolute acceleration that the vehicle is constrained to reach. As presented in Fig. 1, the curvilinear coordinates approach has been used in many research works and is the most effective way to describe road centreline using only the line curvature (κ) as a function of length (s), [8,9]. The road heading angle θ as well as x, y coordinates may be calculated by integrating the curvature as follows (2): dx = cosθ, ds

dθ = κ(s), ds

dy = sinθ ds

(2)

The curvilinear coordinates is given as: vx cosα − vy sinα 1 − sn κ s˙n = vx sinα + vy cosα α˙ = ψ˙ − sκ ˙ s˙ =

(3) (4) (5)

where vx is the vehicle forward velocity, vy is the lateral velocity, ψ˙ is the yaw rate, s and sn are the longitudinal and lateral position on the road strip while α is the vehicle relative heading to the road. In summary, the vehicle dynamics is described by means of two control inputs plus eight state variables: u = [ax , ay ]T

x = [vx , vy , s, sn , α, x, y, θ]T

(6)

The equations of motion can be summarised into: A(x)x˙ = f [x(t), u(t)]

(7)

where matrix A is invertible provided that u > 0, i.e. the vehicle never stops, and sn < 1/κ i.e. the vehicle never passes over the local curvature centre of the road [8]. 2.2

Motion Sickness Evaluation and Illness Rating

British Standard (BS 6841-1987)[10] provides guidelines for evaluation of human exposure to whole-body mechanical vibration and repeated shock and an empirical approximation describing the method of evaluating motion sickness is suggested, its suitability was examined by Turner and Griffin [3]. The total motion sickness dose value resulted from lateral and longitudinal motion is given as:  M SDV = 0

T

(ax,w (t))2 dt

 12

 + 0

T

(ay,w (t))2 dt

 12 (8)

278

Z. Htike et al.

where M SDV is the motion sickness dose value (in ms−1.5 ), ax,w (t) and ay,w (t) are the frequency weight acceleration in the longitudinal and lateral direction. ax,w (t) = ax (t) × Wf ay,w (t) = ay (t) × Wf

(9) (10)

where ax (t) and ay (t) are the longitudinal and lateral acceleration. Wf is the weighting factor and the weighting curve can be found in [11]. T is the total period (in seconds) during which motion could occur. From the standards, a simple linear approximation between motion sickness dose and mean passenger illness rating is given as: IR = K × M SDV

(11)

where IR is predicted illness rating and K is an empirically derived constant (=1/50) based on data from studies of motion sickness in ships and road [3] and [12]. The illness rating value is divided into four levels; 0 indicates feeling fine, 1 indicates slightly unwell, 2 indicates quite ill, and 3 indicates absolutely dreadful. For example, if the person’s illness rating is 0.9 which could indicate that the person is feeling slightly unwell. In this work, illness rating will be used to represent motion sickness. Despite the fact that IR will continue to increase as motion sickness dose value is the integral over time of squared accelerations, this work is to focus on the minimisation for any given illness rating.

3

Optimal Control Problem Formulation

The optimal control problem presented in this work is to find the appropriate vehicle control inputs, that can drive the vehicle along a predefined path from the initial position (s0 ) to the final position (sf ), such that both the journey time (F 1) and the motion sickness (F 2) to be minimized. This could be solved as an optimal control problem (OCP), by setting specific cost function which represent the objectives set and satisfying a set of differential equality and algebraic inequality constraints. Following the approach used in many research works in minimum lap time simulation problem [8] and [9], it is convenient to change the independent variable from time (t) to distance (s) in the equations of motion (7). This transformation is based on the following derivation rule (12): dx dt dx = x = = x˙ s˙ −1 ds dt ds

(12)

As mentioned above, our objectives will be the minimisation of both the journey time (F1 ) and the motion sickness (F2 ). Thus, they will be combined in a costfunction J = F (F1 , F2 ) for the optimal control problem that will be formulated.  sf 1 ds F2 = IR (13) F1 = s0 s˙

Motion Sickness Minimisation

f ind :

subject

to :

minimize u∈U

J

Ax = f (x, u, s) φ(x, u, s)  0 b(x(s0 ), x(sf )) = 0

279

(14)

(15) (16) (17)

In addition, two inequality constrains will be considered, such that the vehicle  will be able to accelerate within the bounds set by the friction circle; ax 2 + ay 2  amax and never exceed the road borders considering left-width (Lw ) and right-width (Rw ) from the centreline of the road; −Lw  sn  Rw . Last but not least, boundary conditions have been added, since illness rating is a function of acceleration. Thus, its minimisation could lead to the vehicle to not move at all. In this respect, to overcome this problem, minimum speed is introduced at umin = 5 [m/s] during the whole journey to ensure that the motion planner would move the vehicle and force it to complete the journey. Moreover, in order to achieve the best possible optimal solution, the maximum boundary condition for velocity is set as “Free”, and in this work maximum speed is set as umax = 40 [m/s]. The optimal control problem has been formulated as described above and solved using GPOPS II solver [13] with MATLAB suite.

4

Results

In this work, the optimal control technique is applied to vehicle dynamics in order to minimise motion sickness and journey time. Therefore, three cases have been set up to investigate the compromise of motion sickness with minimum journey time in various route scenarios. , for minimum time – Case 1 : J = F1 – Case 2 : J = w1 F1 + w2 F2 , for combined objectives – Case 3 : J = F2 , for minimum motion sickness where w1 and w2 are the weighting factors. Three different road routes are presented to investigate the effect of road paths on motion sickness. The selected routes have the same length for all the straight and curvature sections (i.e. A, B, C, D and E), however the road curvature κ(s) at B and D sections are gradually decrease from K1, K2 to K3 shown in Fig. 2a and the road trajectory can be seen in Fig. 2b. All the above cases are tested with the condition where the road doesn’t allow any lateral manoeuvrability to the vehicle by setting the road width at zero (i.e. Lw , Rw = 0m). In this way, the vehicle travels on the fixed road path. The simulation results for each case with three road route are tabulated in Table 1, where the journey time and illness rating for each case are presented. Similarly, the optimum velocity profile for each case is illustrated in Fig. 3. According to Table 1 for all the road routes. Case 1 yields the highest illness rating with shortest journey time as its objective is to achieve minimum time

280

Z. Htike et al.

(a) Road curvature profile

(b) Road Trajectory

Fig. 2. Trajectory of the three roads

only. On the other hand, Case 3 achieves lowest illness rating but with highest journey time, as its cost function represents motion sickness. In addition, Case 2 where both time and illness rating are considered, lower motion sickness is achieved compared to Case 1 and also shorter journey time compared to Case 3. Table 1. Optimal solution for journey time and illness rating K1 Time [s] IR

K2 Time [s] IR

K3 Time [s] IR

Case 1 15.80

0.641 15.07

0.591 14.22

0.535

Case 2 17.31

0.437 16.56

0.394 15.66

0.348

Case 3 25.23

0.163 24.13

0.147 22.85

0.130

Taking minimum time solution i.e. Case 1 to be the baseline, it can be seen that the time taken for the vehicle to complete the journey increases significantly by about 10% for Case 2 while it is 60% for Case 3. However, when comparing their illness rating, IR significantly reduces to more than 30% for Case 2, and 75% for Case 3. It can be concluded that when both journey time and motion sickness are taken into consideration, better result could be obtained. Regarding Fig. 3, in all the three roads, the velocity profile of the vehicle in Case 1 decelerates and accelerates rapidly along the path whenever possible to decrease journey time, which increases illness rating. On the other hand for Case 2, the vehicle travels as fast as it can but avoid rapid change and maintaining the minimum illness rating. Finally in Case 3, the vehicle accelerates very slowly and smoothly as rapid change in the acceleration would result in higher illness rating. This results in longer journey time. Similarly, the journey time and illness

Motion Sickness Minimisation

281

rating decreases as the road curvature decreases, as higher κ in road K1 would first result in higher lateral acceleration and hence required to further slow down compares with lower κ in road K2.

Fig. 3. Comparison of the velocity profile

5

Conclusions

The application of optimal control technique for motion sickness minimisation in autonomous vehicles has been successfully implemented. Motion sickness is minimised by taking the optimum trajectory and velocity profile for any given road with flexible road width. The journey time has also shorten when illness rating and minimum time are considered in the cost function. However, in order to achieve optimal solution for both minimum sickness as well as journey time, further work will required to investigate the weighting factor between the two and their trade off. Acknowledgement. The research is partially funded by Innovate UK through the AIDCAV project and the EPSRC under Grant EP/N509450/1.

282

Z. Htike et al.

References 1. Sivak, M., Schoettle, B.: Motion sickness in self-driving vehicles (2015) 2. Cheung, B., Nakashima A.: A review on the effects of frequency of oscillation on motion sickness. No. DRDC-TR-2006-229. Defence research and development Toronto (Canada) (2006) 3. Turner, M., Griffin, M.J.: Motion sickness in public road transport: the effect of driver, route and vehicle. Ergonomics 42(12), 1646–1664 (1999) 4. Elbanhawi, M., Simic, M., Jazar, R.: In the passenger seat: investigating ride comfort measures in autonomous cars. IEEE Intell. Transp. Syst. Mag. 7(3), 4–17 (2015) 5. Wada, T.: Motion sickness in automated vehicles. In: Advanced Vehicle Control: Proceedings of the 13th International Symposium on Advanced Vehicle Control (AVEC 2016), 13–16 September, Munich, p. 2016. CRC Press, Germany (2016) 6. Griffin, M.J.: Handbook of Human Vibration. Academic press, Cambridge (2012) 7. Althoff, M., Koschi, M., Manzinger, S.: CommonRoad: composable benchmarks for motion planning on roads. In: 2017 IEEE Intelligent Vehicles Symposium (IV) 11 Jun 2017, pp. 719–726. IEEE (2017) 8. Lot, R., Biral, F.: A curvilinear abscissa approach for the lap time optimization of racing vehicles. IFAC Proc. Volumes 47(3), 7559–7565 (2014) 9. Smith, E.N., Velenis, E., Tavernini, D., Cao, D.: Effect of handling characteristics on minimum time cornering with torque vectoring. Vehicle Syst. Dyn. 56(2), 221– 248 (2018) 10. British Standards Institution: British standard guide to measurement and evaluation of human exposure to whole-body mechanical vibration and repeated shock. British Standards Institution, London (1987) 11. Donohew, B.E., Griffin, M.J.: Motion sickness: effect of the frequency of lateral oscillation. Aviat. Space Environ. Med. 75(8), 649–656 (2004) 12. Lawther, A., Griffin, M.J.: Motion sickness and motion characteristics of vessels at sea. Ergonomics 31(10), 1373–1394 (1988) 13. Patterson, M.A., Rao, A.V.: GPOPS-II: A MATLAB software for solving multiplephase optimal control problems using HP-adaptive Gaussian quadrature collocation methods and sparse nonlinear programming. ACM Trans. Math. Softw. (TOMS) 41(1), 1 (2014)

A Tire/Road Interaction Tool for the Evaluation of Tire Wear for Commercial Vehicles M. De Martino1, Argemiro L. A. Costa1, F. Timpone2,3, and A. Sakhnevych2,3(&) 2

1 Prometeon Tire Group, Milan, Italy Università degli Studi di Napoli Federico II, Naples, Italy [email protected] 3 MegaRide – UniNa Spin-Off Company, Naples, Italy

Abstract. The understanding and control of tire wear, preventing tread degradation and irregular wear has been a challenge to tire product engineers, and an important issue for fleet management. There is not a simple equation to analyze and predict it. The optimal wear, and consequent mileage performance, depends not only on the tire, but also on its interaction with the vehicle and the road. It varies with operational conditions and, furthermore, with vehicle and tire maintenance. This paper aims to show the preliminary analysis useful for the development of a tool able to predict tire wear performances in the truck & bus environment, anticipating the results coming from a standard field evaluation. A predictive tool is important for both tire manufacturer and final customers: the advantage for tire manufacturer is the possibility to drastically reduce timeto-market and to have reliable and controlled results; for OEMs, the possibility to receive tire models based on outdoor tests using their own vehicle as a moving lab; for final customer, the advantage of having smart-tire able to predict wear performances, generating valuable advices for maintenance and fleet control. Such aspects will be explained in this paper covering the first results obtained during an outdoor session where a reference vehicle has been instrumented in order to evaluate longitudinal behavior. Keywords: Tire wear


Smart-tire
Truck & bus
Fleet control
Trick tool

1 Introduction Vehicle suspension system and, generally speaking, vehicle dynamics have a fundamental influence on the tire contact patch forces and as a results the tire characteristics can be seen as having an important effect on vehicle behavior. The strong impulse to the research in the field of vehicle systems analysis and modeling comes, of course, from the improvement of tire technological development mainly in motorsport and automotive industries. This, with a certain delay, is happening also in the industrial tire business (Truck & Bus, OTR, and Agro). The definition of a standard testing activity © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 283–293, 2020. https://doi.org/10.1007/978-3-030-48989-2_31

284

M. De Martino et al.

able to estimate tire forces and slip indices is a crucial task for many, and it is one of the pillar (together with thermodynamics and adherence studies) needed to reach the final goal: a tools to predict the wear. The pneumatic tire is one of the few mechanical structures subjected to large deflections and deformations. It is constructed of non-linear anisotropic composite materials. It is a complex mechanical structure, which presents modeling difficulties like the contact between tread surface and roadway, and mathematical complications due to rubber incompressibility and hyperelasticity. The models that intend to simulate the tire behavior must take all these characteristics, consequence of how product engineer develop the product, into account [1].

2 Understanding the Tire 2.1

Tire Structure

The tread compound (Fig. 2) is the most external material that covers the belts, on which grooves are stamped to form the tread pattern. The tread compound must attend different requisites: grip on wet and dry pavements, abrasion and cut resistance, comfort, and a high mileage guarantee. For these reasons, it is natural to focus the attention on the tread compounds when studying tire wear. The tread pattern is the one of the most important parts of the tire talking about wear performance. Its shape, how groove and lug have been designed, the void ratio, the rib proportioning, each part of the tread pattern design is studied to obtain the best rolling resistance, wet grip, wear resistance balance (Fig. 1).

Fig. 1. The magic triangle.

The reinforcement of the structure is obtained with the carcass and belt plies, which will determine the geometry of inflated tire (Fig. 2). The tire belts and carcass, among other available materials, can be made of nylon, polyester, rayon or steel cords. The carcass is anchored at the beads, wrapping it on. The beads are generally steel wires disposed in the circumferential direction.

A Tire/Road Interaction Tool for the Evaluation of Tire Wear

285

Fig. 2. The pneumatic tire structure complexity.

One good reference, for more details about tire mechanics and its particularities of process and project, is the book edited by S. K. Clark [2], printed by the American Department of Transportation. 2.2

Rubber Compounds

The basic ingredients of the rubber compounds are the elastomers, materials constituted by entangled polymeric chains (Fig. 3). The elastomers are the materials responsible for its physical characteristics: elasticity, flexibility and impermeability to the air and water.

Fig. 3. Illustration of molecular chain structure of rubbers linked by the atoms of sulfur, the crosslinks [2].

The elastomers, generically named as rubbers, are defined as being any polymeric material that has the same elastic behavior of natural rubber at room temperature, which, by definition, is a natural elastomer. The rubber behavior depends on the environment and on the operational conditions that will modify the physical properties and determine its aging. This influence begins

286

M. De Martino et al.

at the manufacturing process and goes until the end of product life on the market. The rubber compounds are viscoelastic materials by nature, and hence with behavior depending on the deformation frequency and temperature. On the other hand, when excited dynamically, rubber compounds will generate heat that modifies the mechanical properties itself. The rubber compounds are vulcanized in the pneumatic industries. Such process consists in one or more atoms of sulfur linking the polymer chains each other (Fig. 3), enhancing the rubber elasticity and stability at high temperatures and deformations. While metals have a plastic behavior when submitted to deformations greater than 1%, rubbers can remain elastic for strains in excess of 500%. Due to this characteristic, rubbers are known as hyperelastic materials. 2.3

Wear

All the dynamic forces acting on the commercial vehicles for the different maneuvers have to be supported by the contact patch of tire, a small area not greater than a notebook paper. The contact pressure distribution and the resulting footprint shape will depend on the operational conditions, tire loads, accelerations, tire inflation pressure, vehicle suspension kinematics [3]. The pavement interaction will be influenced also by the temperature, soil roughness and wet or dry conditions. When the local shear forces, at each point of the tire contact patch, exceed the limit of frictional coefficient, local

Fig. 4. Finite Element simulation of tread wear: (a) and (b) tire model; (c) experimental contact pressure; (d) simulated contact pressure and frictional energy obtained by multiplying the contacting shear forces by the sliding on dynamic conditions.

A Tire/Road Interaction Tool for the Evaluation of Tire Wear

287

sliding will start, resulting in abrasion and tread wear. The level of the contact forces and disposition of local slips will determine if the tire tread will be submitted to a homogeneous wear or uneven wear. Higher contact pressure prevents the slippage and abrasion, but if the contact pressures are too high, the thermomechanical degradation of the rubber compounds will result also in irregular wear. To optimize the tread wear a finite element model has been used to evaluate the footprint shape and the respective forces (Fig. 4). The boundary conditions for the tire wear simulation are obtained by the tools proposed below. The usual methodology to get the tire dynamic forces as input to calculate the tire wear are the multibody simulations, using commercial codes like TruckSim®. But the characterization of the tires for models like FTire [3] is very expensive and time consuming [4, 5], constraining the activity for product development. In the next paragraphs is described a new methodology, under development for trucks and bus vehicles, that will be valuable to get not only the dynamic forces, but also the vehicles dynamics on-line, in different pavements and operational conditions, permitting to construct a real time map the circulating fleet, optimizing fuel consumption and maintenance.

3 The Tool All the analysis and results explained in the paper have been obtained using TRICK4TRUCK [6, 7]. The software is a customized version for industrial application of TRICK tool (Tire/Road Interaction Characterization & Knowledge). It comprises of a vehicle model which process experimental signals acquired from vehicle CAN bus or from dedicated instrumentation. The output of the tool is several extra “virtual telemetry” channels containing force and slip estimations, useful to provide tire interaction characteristics. The results coming from the tool will be integrated with other physical models for the prediction and the simulation of specific performances. 3.1

Input Parameters and Test Procedure

For a correct employment of TRICK4TRUCK, a reliable description of the vehicle is essential together with information about the vehicle under test, described in [6]. All the data for the identification of tire interactions have to be acquired during dedicated test sessions: the aim is to have a test routine to investigate tire behavior in the widest possible range of working conditions. In particular, the tire road interactions to be highlighted are: • Pure longitudinal: start maneuver with high wheel-spin and braking maneuver with (if possible) wheel blocked on straight road. • Pure lateral: curves performed at null longitudinal forces • Combined: a series of test able to keep tire at high exertion levels, adopting an aggressive driving style (high sliding and high slip angle values)

288

M. De Martino et al.

The first step of test has been done in December in Nardò technical center. Considering the space available on the truck dynamic platform, has been decided to proceed with maneuver indicated below (Table 1). Table 1. Test procedures. 5 s standing vehicle 5 s at constant speed Braking manoeuvre from 100 km/h Full throttle acceleration (0–100 km/h) + braking (100–0 km/h) Full throttle acceleration (0–100 km/h) + braking (100–0 km/h) Circle clockwise at 15 km/h Circle clockwise at 30 km/h Circle clockwise at max speed Circle counter-clockwise at 15 km/h Circle counter-clockwise at 30 km/h Circle counter-clockwise at max speed 8 circuit at 15 km/h 8 circuit at 30 km/h 8 circuit at max speed Full throttle acceleration (0–100 km/h) + braking (100–0 km/h)

4 First Results Preliminary results showed in the paper will focus on longitudinal interaction. Furthermore, in this phase-zero some additional hypotheses have been added because of the unavailability of the whole vehicle parameters set. In particular: • • • •

Suspension system has been considered rigid Cx, Cz have been neglected Camber/Toe have been considered constant during acceleration/braking maneuver Center of Gravity position has been estimated since the information is missing from vehicle manufacturer. The Vehicle used is an IVECO STRALIS 500 MY 2011 (Fig. 5).

Fig. 5. IVECO STRALIS 500 MY 2011 – 4  2 vehicle.

A Tire/Road Interaction Tool for the Evaluation of Tire Wear

289

This vehicle from Prometeon test-fleet has been chosen for its easiness in loading/unloading. Trying to analyze all the range of working condition, 3 sets of load have been tested (Table 2). Table 2. Load conditions. Front axle [N] Rear axle [N] Unloaded 5500 3450 Loaded 6100 5850 Overloaded 6850 8500

4.1

Input Parameters

The channels needed for the TRICK have been acquired through DEWESOFT (Fig. 6). In detail: – – – – – – – – –

d*: driver’s steering angle - analogical output [rad] ay*: vehicle lateral acceleration - IMU output [m/s2] ax*: vehicle longitudinal acceleration - IMU output [m/s2] X*LF: left front wheel speed - CAN bus output [m/s] X*RF: right front wheel speed - CAN bus output [m/s] X*LR: left rear wheel speed - CAN bus output [m/s] X*RR: right rear wheel speed - CAN bus output [m/s] r*: vehicle yaw rate – IMU output [rad/s] V*: vehicle speed – IMU output

Fig. 6. Acquisition system.

To validate the output, two KISTLER dynamometric wheels have been mounted. IMU was mounted behind the driver cabin (Fig. 7).

290

M. De Martino et al.

Fig. 7. Front and Rear axle dynamometric wheels (left) and IMU (right).

4.2

Center of Gravity

To estimate the position of center of gravity in the various loading condition, two maneuvers have been evaluated. For front and rear wheelbase, the values can be easily estimated starting from a zero static test. Weighing the front and rear axle the formulation to be used is a force balance (Fig. 8).

Fig. 8. Results of force balance to estimate front and rear wheelbases.

For the evaluation of center of gravity height, signals used come from a braking maneuver from 100 km/h. Acceleration in braking maneuver has the shape shown below. For the calculation of center of gravity height, it is needed to evaluate the vehicle in a stationary phase (highlighted in the graph) (Fig. 8), otherwise it is not possible to evaluate the center of gravity with a simple rotational equilibrium around one of the two centers of tyre/road contact. hCG ¼ ðW1  Fz1 Þ
l=ðm
ax Þ

4.3

ð1Þ

Rolling Radius

Constant speed test was used to evaluate rolling radius. To this purpose, slip angle has to be the smallest (pure rolling conditions) and constant. Constant speed test has been

A Tire/Road Interaction Tool for the Evaluation of Tire Wear

291

Fig. 9. Longitudinal speed and the part extracted for the Rolling Radius estimation.

chosen because of its constant longitudinal acceleration close to zero (Fig. 9). A certain range of time has been extracted to evaluate rolling radius (the most constant part). Knowing that slip ratio is almost zero (consequence of the maneuver), and knowing the speed and the angular speed from instrumentation and CAN bus, it is possible to evaluate rolling radius, as: slip ratio ¼ ðVGPS  x
RÞ=VGPS

ð2Þ

where: V_GPS is the speed acquired by GPS; x is the angular speed acquired by CAN bus; R is the rolling radius evaluated by constant speed test In the following figures, rolling radius of tires has been calculated as an average value of ratio between the speed acquired by GPS and tires’ angular speed (Fig. 10).

Fig. 10. Reference tire in unloaded (left) and loaded (right) conditions.

4.4

Slip Ratio

Slip ratio has been evaluated either in braking (Eq. 2) or acceleration (Eq. 3) phases. It is necessary to highlight that the best method to evaluate tire/road interaction is through a blocked wheel maneuver and high slip acceleration. While acceleration with

292

M. De Martino et al.

high slip is almost impossible since torque power and weight of commercial vehicle make this type of maneuver almost impossible, blocked wheel braking would be possible after some modification on braking system. In the first test, the vehicle has not received any modification (Fig. 11).

Fig. 11. Slip ratio and longitudinal braking (positive = braking).

In this case the slip ratio of front tire is less than zero. This is expected because front tire in an acceleration maneuver is a free rolling tire where rolling resistance has to be considered. Rear wheel has, of course, positive slip ratio. This is in line with expectation considering that rear axle is the tractive axle. 4.5

Dynamometric Wheel Output

When TRICK will be launched and first results will be available, it will be needed to compare results with measured forces. Instrumenting the vehicle with KISTLER dynamometric wheels has allowed us to have “force vs slip ratio” graph (Fig. 12, left). In the graph is clear the weight transfer from rear to front that affects the longitudinal characteristics (Fig. 12, right).

Fig. 12. Longitudinal force vs slip ratio (left) and the experimentally acquired forces (right).

A Tire/Road Interaction Tool for the Evaluation of Tire Wear

293

5 Conclusions In this paper part of the starting evaluation for the customization of TRICK tool for Commercial Vehicle has been described. The final aim is to have a tool able to process all the information obtained from specific testing session using more vehicles as a moving lab. This would be the weapon to predict one of the performances most timeconsuming to predict: the vehicle behavior on real pavements, without the requirement of expensive, complex, and often not fully representative benches. What has been explained in the paper is a summary of preliminary studies carried out on the data coming from a dedicated testing activity. The focus was dedicated to first step of pure longitudinal interaction. Information about pure lateral and combined interaction obtained in the same testing activity will be evaluated and presented in order to have complete set of data.

References 1. Costa, A.LA.: Vibrational behavior of tire-suspension system for commercial vehicles regarding comfort and tread wear. In: Segundo Colloquium Internacional de Suspensões e Primeiro Colloquium de Implementos Rodoviários, 2002, Caxias do Sul. Proceedings. Caxias do Sul: SAE Brasil - Seção Caxias do Sul - RS (2002) 2. Gent, A.N., Walter, J.D.: The Pneumatic Tire, University of Akron, National Highway Traffic Safety Administration (NHTSA) (2005) 3. Gipser, M.: FTire – the tire simulation model for all applications related to vehicle dynamics. Vehicle Syst. Dyn. 45(Supplement), 1 (2005) 4. Pacejka, H.B.: Tyre and Vehicle Dynamics, Butterworth-Heinemann (2006) 5. DELFT-TYRE, Tyre Models Users Manual, TNO Automotive (2002) 6. Farroni, F.: T.R.I.C.K.-Tire/Road Interaction Characterization & Knowledge – a tool for the evaluation of tire and vehicle performances in outdoor session, Elservier 7. Farroni, F., Russo, M., Russo, R., Timpone, F.: A physical–analytical model for a real-time local grip estimation of tyre rubber in sliding contact with road asperities. Proc. Inst. Mech. Eng. Part D J. Automobile Eng. 14 February 2014. https://doi.org/10.1177/095440701452 1402 8. Farroni, F., Russo, M., Sakhnevych, A., Timpone, F.: TRT EVO: advances in real-time thermodynamic tire modeling for vehicle dynamics simulations. Proc. Inst. Mech. Eng. Part D J. Automobile Eng. 233(1), 121–135 (2019)

Multi Actuation Scheme for Path-Following Control of Autonomous Vehicles Javad Ahmadi1,2(&), Efstathios Velenis1, Heimoud El Vagha1, Chenhui Lin1, Boyuan Li1, Stefano Longo1, and Efstathios Siampis1 1

Centre for Automotive Engineering, School of Aerospace, Transport and Manufacturing, Cranfield University, Cranfield MK43 0AL, UK {javad.ahmadi,e.velenis,heimoud.el-vagha,chenhui.lin, boyuan.li,s.longo,Efstathios.Siampis}@cranfield.ac.uk 2 Department of Mechanical Engineering, Payame Noor University, 19395-3697, Tehran, Iran

Abstract. This paper presents a Multi actuation scheme for path following control of autonomous vehicles. Steering angle of front and rear axles accompanied with wheel torques of front axle and individual left and right wheels of the rear axle are the five control inputs of the system. The required feedback outputs of the system consist of lateral offset, heading angle error and forward speed of the vehicle and their time derivative. In the high-level layer of the controller, zeroing the two path states: lateral offset and heading angle error, and tracking in the longitudinal velocity are targeted by decentralization of proportional-derivative controllers. The high-level part outputs the desired generalized forces and moment on the vehicle body-fixed coordinate. These desired values are tackled by the control inputs through an allocation scheme. The capability of the proposed closed-loop system is analyzed through nonlinear numerical simulations. Keywords: Autonomous vehicle


Path following
Multi actuation

1 Introduction Autonomy of ground vehicles has become an active research area in the context of ITS due to its potential developments it can have on human-being life. Trajectory following control (TFC) as a main task of an autonomous vehicles has been under focus by many investigators [1–4]. Actuator redundancy with the aim of increased maneuverability and fault tolerance of the system have been employed in many dynamic systems including vehicular applications [5–7]. While use of torque vectoring, front steering control and their combination for path following of autonomous vehicles are well researched, their augmentation with rear wheel steering control is not well addressed. In this paper, a multi actuation scheme for path following control of autonomous vehicles is presented. Steering angle of front and rear axles accompanied with wheel torques of front axle and individual left and right wheels of the rear axle are the five control inputs of the system. The required feedback outputs of the system consist of lateral offset, heading angle error and forward speed of the vehicle and their time derivative. In the high-level layer of the controller, zeroing the two path states: lateral offset and heading © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 294–301, 2020. https://doi.org/10.1007/978-3-030-48989-2_32

Multi Actuation Scheme for Path-Following Control of Autonomous Vehicles

295

angle error, and tracking in the longitudinal velocity are targeted by decentralization of proportional-derivative controllers. The high-level part outputs the desired generalized forces and moment on the vehicle body-fixed coordinate. These desired values are tackled by the control inputs through an allocation scheme. The capability of the proposed closed-loop system is analyzed through nonlinear numerical simulation. This rest of the paper is organized as follows. In Sect. 2, the dynamic modeling is given. Proposed control system which is made up of higher and lower levels is deduced in Sect. 3 respectively. Section 4 describes simulations results for evaluating the developed controller in Sect. 3. Summary and conclusions are given in Sect. 5.

2 Modelling Based on the free body diagram in Fig. (1), using Newton-Euler formulation for plane rigid body motion, the following dynamics are obtained, P

Fx m P Fy _Vy ¼ rVx þ m V_ x ¼ rVy þ

Fig. 1. Tire friction forces.

ð1Þ ð2Þ

296

J. Ahmadi et al.

P r_ ¼

Mz Iz

ð3Þ

Where, Vx is the longitudinal velocity, Vy is the lateral velocity and r is the yaw rate of the vehicle. In the Fig.P 1, the P friction forces acting on the vehicle Fxi ’s and Fyi ’s (for i = 1 to 4) are depicted. Fx , Fy are the vectorial P summation of friction forces in the longitudinal and lateral direction of the vehicle. Mz is the summation of the yaw moments produced by the tire friction forces. m and Iz , are the mass and mass moment of inertia of the vehicle. By kinematic modelling of vehicle motion, we have [1], z_ e ¼ Vx sinwe þ Vy coswe

ð4Þ

w_ e ¼ r  jd Vx

ð5Þ

where ze and we are lateral offset and heading angle error of the vehicle, respectively and jd is the road curvature.

3 Controller Design 3.1

High-Level Layer

For path-following problem, the main two state variables that are to be controlled are the lateral deviation offset and heading angle error of the vehicle. Assuming the generalized lateral force acting on vehicle is the most effective variable on the vehicle lateral offset channel can result in the following proportional-derivative control law, X

Fydes ¼ kd1 z_ e  kp1 ze

ð6Þ

where, kd1 and kp1 are the derivative and proportional gains wrt vehicle lateral offset. Similarly, assuming the generalized yaw moment acting on vehicle is the most effective variable on the vehicle heading angle channel can result in the following proportionalderivative control law, X Mzdes ¼ kd2 w_ e  kp2 we ð7Þ where, kd2 and kp2 are the derivative and proportional gains wrt vehicle heading angle error. Furthermore, to achieve tracking in the vehicle forward speed of the vehicle it is proposed that the generalized longitudinal force acting on vehicle manipulated as by the following proportional-derivative control law, X

  Fxdes ¼ kd3 V_ x  V_ xdes  kp3 ðVx  Vxdes Þ

ð8Þ

where, kd3 and kp3 are the derivative and proportional gains wrt the errors in desired vehicle forward speed.

Multi Actuation Scheme for Path-Following Control of Autonomous Vehicles

3.2

297

Low-Level Layer

To achieve the outputs of the high-level layer explained in previous section, a low level-layer is used which is explained in this section. In the longitudinal direction of the vehicle, we have, Xfdes þ Xrldes þ Xrrdes ¼ Xdes

ð9Þ

Where, Xfdes ; Xrldes and Xrrdes stand for desired longitudinal forces acting on the front axle, left and right wheels of the rear axle, respectively. Furthermore, the following two relation are considered to address the normal load transfer due to longitudinal and lateral acceleration as, Xfdes Fzf ¼ ¼ a1 ! Xfdes  a1 ðXrldes þ Xrrdes Þ ¼ 0 Xrldes þ Xrrdes Fzr

ð10Þ

Xrldes Fzrl ¼ ¼ a2 ! Xrldes  a2 Xrrdes ¼ 0 Xrrdes Fzrr

ð11Þ

Where, Fzf ; Fzr ; Fzrl and Fzrr are the normal forces acting on the front and rear axles, and left and right wheels of the rear axle, respectively. While the last relation is considered for lateral acceleration, the former one is for longitudinal acceleration. Combining the above three equations in the matrix form, we have 2

1 41 0

1 a1 1

32 3 2 3 1 Xfdes Xdes a1 54 Xrldes 5 ¼ 4 0 5 0 a2 Xrrdes

ð12Þ

equivalently, 2

1 CM1 ¼ 4 1 0

1 a1 1

3 2 3 2 3 Xdes Xfdes 1 a1 5 ! 4 Xrldes 5 ¼ CM11 4 0 5 0 a2 Xrrdes

ð13Þ

Calculating the yaw moment generated by Xfdes ; Xrldes and Xrrdes , MzX ¼

tr ðXrrdes  Xrldes Þ 2

ð14Þ

The complementary yaw moment is calculated by, MzY ¼ Mzdes  MzX

ð15Þ

The above complementary yaw moment is to be produced by Yfdes and Yrdes . Therefore we have

298

J. Ahmadi et al.



 Ydes ¼ Yfdes þ Yrdes 1 ! lf MzY ¼ Yfdes lf  Yrdes lr

1 lr



Yfdes Yrdes





Ydes ¼ MzY

 ð16Þ

equivalently,  CM2 ¼

1 lf



1 lr

 !

Yfdes Yrdes



¼ CM21



Ydes MzY

 ð17Þ

Based on Fig. 1, assuming linear tire behavior and small steering angle simultaneously,   Yfdes ¼ Fxf sindf þ Fyfldes þ Fyfrdes cosdf ! Yfdes  Fyfldes þ Fyfrdes  Ca afl þ Ca afr ð18Þ Based on the definition for side-slip angle of front tires as, afl ¼ df  tan1

Vy þ Lf r Vx 

Tf 2

r

! ; afr ¼ df  tan1

Vy þ Lf r Vx þ

Tf 2

!

r

ð19Þ

We have 1

Ca afl þ Ca afr ¼ Ca 2df þ tan

Vx 

Tf 2

r

! þ tan

1

Vy þ Lf r Vx þ

!!

Tf 2

r !  1 1 Vy þ Lf r 1 Vy þ Lf r Yfdes þ 0:5tan þ 0:5tan T T 2Ca Vx  2f r Vx þ 2f r

 ! df ¼

Vy þ Lf r

!

ð20Þ

Similar to the above approach used for front wheels, based on the definition of sideslip angle for rear tires, the rear steering angle is calculated as,  dr ¼

! !  1 1 Vy  Lr r 1 Vy  Lr r Yrdes þ 0:5tan þ 0:5tan 2Ca Vx  T2r r Vx þ T2r r

ð21Þ

4 Simulation Result To assess the performance of the proposed controller system a numerical simulation has been carried out. In the simulation, a seven degree of freedom vehicle handling model and a Dougoff nonlinear tire model is used. In the simulation the actuator fault tolerance capability of the feedback system is examined. In this regards, two faulty scenarios are compared against fault-free scenario. In the first faulty scenario, the actuator of the rear left wheel can produce no wheel torque. In the second faulty scenario, the actuators on the front wheels and rear left wheels can produce no wheel torque. It is intended that the

Multi Actuation Scheme for Path-Following Control of Autonomous Vehicles

299

Fig. 2. Responses of the vehicle-cornering with 0:6 g constant acceleration on a circular path with radius 15 m and with V 0 ¼ 6:11 ms and V f ¼ 13:61 ms.

Fig. 3. Control inputs during cornering with 0:6 g constant acceleration on a circular path with radius 15 m and with V 0 ¼ 6:11 ms and V f ¼ 13:61 ms.

300

J. Ahmadi et al.

vehicle performs a cornering on a circular path with radius 15 m. While cornering, it is desired that the vehicle with initial forward speed V0 ¼ 6:11 ms and 0:6 g constant longitudinal acceleration reaches the final forward speed Vf ¼ 13:61 ms. The desired path and the state variables of the vehicle are shown in Fig. 2. As can be seen path tracking is well achieved even in the faulty scenarios which shows the fault tolerance capability of the system. The five control inputs are given in Fig. 3 which are all practical. While the steering angle of the both axles are not much different for the faulty cases in compare with fault free case, the wheel torques of the rear right wheel is considerably larger for the faulty cases in compare with the fault free case.

5 Conclusion A multi actuation scheme for path following control of autonomous vehicles is addressed in this paper. The actuators employed are the steering angle of front and rear axles, wheel torques of front axle and individual torques of left and right wheels of the rear axle. The required feedback outputs of the system consist of lateral offset, heading angle error and forward speed of the vehicle and their time derivative. In the high-level layer of the controller, zeroing the two path states: lateral offset and heading angle error, and tracking in the longitudinal velocity are targeted by decentralization of proportional-derivative controllers. The high-level part outputs the desired generalized forces and moment on the vehicle body-fixed coordinate. These desired values are tackled by the control inputs through an allocation scheme. The fault tolerance capability of the proposed closed-loop system is analyzed through nonlinear numerical simulation. Acknowledgment. Research supported by Innovate UK through AID-CAD project.

References 1. Marzbani, H., Khayyam, H., Ching Nok, T.O., Jazar, R.N.: Autonomous vehicles- autodriver algorithm and vehicle dynamics. IEEE Trans. Veh. Technol. 68(4), 6379–6390 (2019) 2. Norouzi, A., Masoumi, M., Barari, A., Sani, S.F.: Lateral control of an autonomous vehicle using integrated backstepping and sliding mode controller. Proc. Inst. Mech. Eng. Part K J. Multi-body Dyn. 233(1), 141–151 (2019) 3. Yan, W., Wang, L., Zhang, J., Li, F.: Path following control of autonomous ground vehicle based on nonsingular terminal sliding mode and active disturbance rejection control. IEEE Trans. Veh. Technol. 68(7), 6379–6390 (2019) 4. Yan, W., Wang, L., Zhang, J., Li, F.: Should the desired heading in path following of autonomous vehicles be the tangent direction of the desired path. IEEE Trans. Veh. Technol. 16(6), 6379–6390 (2015)

Multi Actuation Scheme for Path-Following Control of Autonomous Vehicles

301

5. Bagheri, A., Azadi, S., Soltani, A.: A combined use of adaptive sliding mode control and unscented Kalman filter estimator to improve vehicle yaw stability. Proc. Inst. Mech. Eng. Part K J. Multi-body Dyn. 231(2), 388–401 (2016) 6. Aria Noori Asiabar and Reza Kazemi: A direct yaw moment controller for a four in-wheel motor drive electric vehicle using adaptive sliding mode control. Proc. Inst. Mech. Eng. Part K J. Multi-body Dyn. 233(3), 549–567 (2019) 7. Ahmadi, J., Sedigh, A.K.: Adaptive vehicle lateral-plane motion control using optimal tire friction forces with saturation limits consideration. IEEE Trans. Veh. Technol. 58(8), 4098– 4107 (2009)

Italian Advances in RAAD Robotics

A Saffron Spice Separation System with Computer Vision A. Manuello Bertetto1 and A. Prete2(&) 1

Department of Mechanical and Aerospace Engineering, Politecnico di Torino, Turin, Italy [email protected] 2 Faculty of Engineering, Politecnico Di Torino, Turin, Italy [email protected]

Abstract. A separation device is described to obtain saffron spice after harvesting. To set the separation device, a vertical wind tunnel is developed which measures the terminal velocities of parts of saffron flower, with variations for different regions and years of production. The separation device is equipped with a vision system, which evaluates the separation and the percentage of purity of the saffron spice obtained after separation. In this paper, the design and the realization of the wind tunnel and of the separation device are described and the separation of the flower is performed and discussed, evaluating the separation efficiency. Keywords: Saffron spice


Spice separation
Terminal velocity

1 Introduction In recent years, the interest in the “organic” sector is growing up. In particular, in Europe and the USA, the importance of bio approach is increasing in order to emphasise the ecosystem management rather than external agricultural inputs, with the aim of maintaining the site fertility in the long term. Organic has become a way of working in many countries, mainly in EU and the USA. Consumers in general perceive these products of higher quality or healthier than those from conventional agriculture, while others prefer to believe to correspond to good practices towards the environment or the human workforce. Organic farming is a system that supports the health of ecosystems, people, processes, biodiversity and cycles by respecting local conditions. Organic agriculture focuses on innovating within tradition for environment preservation [1, 2]. Saffron is a commonly organic product. The production of saffron spice in the EU is about seven tons, four percent of world production, although ninety percent of the global commercialization of saffron, is led by European companies [3, 4]. The production of saffron has enormous labour and financial costs: one kilogram of dried spice comes from one hundred and fifty thousand flowers. Saffron production has different phases that, nowadays, are carried out completely manually. Duration percentages of production phases are shown in (Fig. 1): duration is directly linked to the costs of the work necessary for executing each phase and to the cost of one hour of work, by dedicated personnel, in the region where this work is © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 305–314, 2020. https://doi.org/10.1007/978-3-030-48989-2_33

306

A. Manuello Bertetto and A. Prete

carried out [5, 6]. Harvesting and separating phases are the most time-consuming of the entire production cycle, and therefore the most cost affecting. The separation phase is the longest, while the harvesting phase is the most labour intensive [7]. Mechanisation of these phases would have a great impact in reducing these specific production costs. 10% 10%

planting bulbs

packaging

40% separation

10% drying 30% harvesting

Fig. 1. The percentage of human work time to produce saffron spice.

Nowadays the most expensive and labour-intensive phases, flower harvesting and spice separation are carried out by hand, even if there are many proposals for mechanization and automation, some automatically controlled, of these production phases. A professional picker harvests about one thousand flowers per hour, considering also resting time, since harvesting is such a tiring activity. To separate the spice, one hundred minutes are on average necessary to process one thousand flowers. If an efficient robotic system had been proposed for the automation of the collection and separation phases, the production of saffron spice would have been competitive again in Europe where, today, it is marginal. Automated separation methods could bring future intensive saffron production. To this aim, many methods have been suggested to perform the separation of saffron spice. For a mechanized saffron spice separation, the most proposed method is based on the aerodynamics properties of flowers and part of it. To separate the saffron flower components, mainly the stigmas, air flows are generated to exploit the diversity of the aerodynamic characteristics of the components to be treated, in relation to their mass properties. Researchers in [8, 9] describe and use a device where gravity and aerodynamic forces act on parts, allowing the separation in a vertical column where an air flow is generated. In [10], the authors illustrate mechanical devices to support human labour in harvesting and saffron spice separation. They describe a portable device to facilitate harvesting in the field and an air separator and also discuss the results of their method compared to the completely manual method and report the results in the field. In [11] the authors make a deep economic analysis of the saffron cultivation and the profit margin for farmers and for the brokers. In line with that analysis in [12] is shown the importance of a mechanical harvesting machine to improve the economy of a country, the efficiency and the profit margin for the farmers. In [13] a portable facilitating machine is shown and mathematically modelled, the harvesting device has a simple design capable of performing the cutting procedure using only one actuated degree of freedom. In the same paper separation of the stigmas was performed comparing three airflow systems for the separation by generating aerodynamic forces on the flower parts

A Saffron Spice Separation System with Computer Vision

307

in the presence of gravity forces. The work reported in [14] is based on the different terminal velocities of saffron components: a separator is described to separate the stigma from the other parts of saffron flower. In [15] a new mechanised method to orientate the flowers to allow a successive separation of stigmas from saffron flowers is described. In addition, a model was validated by experimental tests, which defined the terminal velocity of the saffron flowers with given masses and geometries. The model was defined using the drag coefficient of wedges and cones vs. a half vertex angle measured and referred to in [16]. To have a more complete mechanised system to produce saffron spice, in [17] the research describes the design and characterization of a saffron flower harvesting and spice separation system. The main parts of the system are a harvesting portable device and a double flow, three-way controlled cyclone separator, where the weight, the mass centrifugal forces and the aerodynamic actions act on saffron parts for separation. The working principle is based on the difference of weight, terminal velocities and aerodynamic forces of different parts of the saffron flower. Many authors have offered systems to mechanise saffron spice production; authors also have proposed evaluating methods, using image analysis to optimise production strategies and to evaluate the efficiency of the proposed mechanised systems. In [18], using image analysis, a computer vision system is developed. The aim is to find the optimal cutting point of saffron flowers, in order to obtain stigmas. The identified cutting point is then sent to the cutting end effector. To recognize and locate saffron flowers in the field, an algorithm was developed in [19], based on image processing techniques, to enhance saffron quality by automated harvesting, with the idea of designing a saffron harvester robot for use in the fields. A separation device is here described to obtain saffron stigmas after harvesting. To set the separation device a vertical wind tunnel is developed for measuring the terminal velocities of the different saffron parts; terminal speed that varies with the different regions and years of production. The separation device is equipped by a vision system, to evaluate the separation phase production and the obtained percentage of purity of the saffron spice, after separation.

2 Materials and Methods The system was designed to be used in two configurations: as a wind tunnel for measuring terminal velocity or as a separator to divide stigmas from the rest of the flower. 2.1

Vertical Wind Tunnel for Terminal Velocity Measurement

The vertical wind tunnel is created using a 110 mm diameter tube (COES blue power) of 1000 mm length. At half height a window is opened in order to insert the sample which needs to be measured. The opening is sealed with a straight coupling with a transparent window. The sleeve was created by machining the straight coupling from the same line of product. At the top of the tube is inserted an unequal double branch with main diameter of 110 mm and secondary diameters of 50 mm, jointed with an angle of 45°. At the top of the branch a cap is inserted, specially modified to house a

308

A. Manuello Bertetto and A. Prete

light to enlighten the test chamber. The tube is supported in vertical position by a custom-made support in wood (Fig. 2). Inside the bottom part of the tube, a 3D-printed, honeycomb-shaped, flow addresser (Fig. 3) is placed to reduce as much as possible the turbulence. The flow addresser is designed to have the characteristic length ten time greater than the diameter of the opening. Also, the wind tunnel is designed to have the diameter greater than ten times the maximum diameter of the tested object. To help the flow inlet from the bottom without creating a vortex, a 3d-printed convergent is inserted. At the bottom of the test chamber’s opening there is a tray to support the material which is being tested (Fig. 4). In the higher part of the wind tunnel (near the branch), a fine wire mesh is fitted in to prevent the tested part being aspirated by the aspirator motor. Two aspirators with 1000 W power each are connected to the branch. The two motors are controlled by an electronic board (Fig. 5) to change the supply voltage using a TRIAC. The two aspirators are controllable independently. With this configuration the wind tunnel can achieve fluid speed in the chamber from 1,0 m/s to 8.5 m/s. The fluid velocity is measured with a hot-wire anemometer (Testo 405i – Range from 0 to 30 m/s – Resolution 0.01 m/s – Precision ± (0,1 m/s + 5% of m.v.)) mounted slightly above the test chamber, using a 3d printed support, specifically designed to provide a tight and sealed fit.

Fig. 2. Wind tunnel configuration, at the centre the main tube, with the windows opened, at the sides the two aspirators, at the bottom the control panel, at half-height (in orange) the anemometer.

Fig. 3. Honeycomb-shaped flow addresser; its length is more than ten times greater than the characteristic size of the alveoli.

A Saffron Spice Separation System with Computer Vision

Fig. 4. Particular of the test chamber with a complete flower ready to be tested. In the figure can be seen the grid to support the flower and the reference to identify uniquely every flower.

2.2

309

Fig. 5. The control panel with the voltmeter of the mains voltage, two rotary switches to turn on/off the aspirators, the potentiometers to set the speed, and two voltmeters/ammeters indicating the voltage and current supplied to the aspirators.

Vertical Separator with Computer Vision to Evaluate Separation

For this purpose, the system uses parts of the wind tunnel. The flow addresser is removed. At the end of the main tube, an unequal single branch with main diameter of 110 mm and secondary diameter of 50 mm, is connected (main diameter and secondary diameter are positioned at 45°). At the end of the branch there is a tube of same diameter with a length of 500 mm. The bottom part of the tube is inserted in the base to hold all the assembly in vertical position. Under the base, where the separated stigmas will accumulate after the separation, a camera (Panasonic NV-GS15) is set up directly connected to a pc with the RoboRealm software (Fig. 6). In the software the image is processed with some filter to reduce flickering and to improve the contrast and brightness of the image. The image is then processed by some blobs filters in order to remove the background,

Fig. 6. Diagram showing the operating principle of the separator

310

A. Manuello Bertetto and A. Prete

then it is processed by some colour filter to identify anthers and stigmas from dimensions and colour. Then, after the software has correctly recognised stigmas and anthers, a custom-written script is used to identify uniquely each anther and stigma and to count them despite the movement of the conveyor belt (Fig. 7). In this configuration, the concept is to aspire air from the bottom part of the tube and to move the fluid from down to up. In this case the fluid speed is set slightly above anther’s maximum terminal velocity. In this system the airflow is aspirated from the top part, unlike the system discussed in [13], where the flow was generated from the bottom with a rotating fan. So, the system showed in [13] suffers from more turbulence in the air flow while the system illustrated in this paper has a steady and regular flow. Worth noting also that in this system stigmas are collected on a conveyor belt with computer vision, and the system is continuous, while the system showed in [13] is discontinuous because to collect the separated stigmas is necessary to stop the machine. If this concept is right, the stigmas should fall and drop on the conveyor belt which moves them under the camera to evaluate the correct separation. Anthers should, instead, be aspirated and caught by the net positioned above the upper window. Fig. 7. Image of the PC screen with the computer vision output

3 Results The research was done using four different batches of flower with ten flowers each: batch one and two came from the same farm and batch three and four were from different farms. Every batch had results that were congruent and collimating with the other batch. Table 1. Results of the measurement done with the machine in the wind tunnel configuration Min velocity Petals 1.65 m/s Anthers 1.6 m/s Stigmas 2.7 m/s

Mean velocity 1.82 m/s 2.07 m/s 3.13 m/s

Max velocity 1.98 m/s 2.43 m/s 3.5 m/s

To keep the data of every batch as much as consistent as possible, measurements were performed, when possible, immediately after the flower picking; otherwise the flower has been kept in controlled temperature and humidity conditions, the time necessary to carry out the tests. From the collected and elaborated amount of data (it must be remembered that for each saffron flower there is a stigma, three anthers and six petals), it clearly emerges that each part of the flower has a very specific range of speed, with stigmas with the highest speed followed by anthers and petals (Table 1). This

A Saffron Spice Separation System with Computer Vision

311

trend agrees also with [8], although speeds in this research are uniformly higher for the bulbs of the high productive type and therefore producing larger flowers with higher speeds. Data show clearly that the lowest speed of stigmas is always higher than the highest speed of anthers; this allows a single-stage separation to separate stigmas from other parts of the flower, and, if separation is correctly implemented, the process should give a complete separation without the need to add more stages of separation in order to increase the purity of the separated material. It is also important the fact that, from the distribution of the terminal velocity (Fig. 8), there are not any anomalies in the trend of value, and terminal velocity almost follows a normal distribution.

Fig. 8. Graph showing the distribution of the flower parts terminal velocity.

Comparing the flowers terminal velocity here measured, with those measured in [15] for flowers having the same half vertex angle, the values are in good agreement; the maximum error is about 15%. For a better understanding of result found in the separator configuration, it is necessary to explain in detail which objective is pursued and define some indexes to better evaluate the performance of the separator. For every flower inserted in the machine, only one stigma is expected to be dropped on the conveyor belt, and six petals and three anthers are expected to be aspirated and caught by the net. If the stigma is aspirated or anthers and petals are dropped on the conveyor belt, the separation is not correct. The index Pam (Percentage of aspirated material) is calculated for each part of the flower and is defined by number of parts aspirated divided by number of flower parts inserted. Pam ¼

NP NP

aspirated

ð1Þ

inserted

The separation is done correctly if Pam index is equal to 0% for stigmas and to 100% for petals and anthers.

312

A. Manuello Bertetto and A. Prete

Table 2. Table showing the goal and the result obtained from the test for the Pam index. Pam Petals Anthers Stigmas

Goal 100% 100% 0%

Result 100% 96.67% 0%

The Pam index is helpful to understand whether the machine separates too much (eliminating valuable anthers) or too little (leaving anthers and petals between the anthers), but does not tell us anything about the quality of the separated product; to better understand this, a Purity index is created. The purity index is defined by the number of correctly separated stigmas, divided by the total of anthers, stigmas and petals introduced in the machine, multiplied by a coefficient due to the number of each component for every flower. Purity ¼

NInserted

NSeparated Stigmas þ  NInserted 1 3

Stigmas Anthers

þ 16  NInserted

ð2Þ Petals

Table 3. Table showing the goal and the result obtained from the test for the Purity index. Goal Result Purity 100% 96.77%

Results referred in Tables 2 and 3 show that the separator can remove easily all petals and almost all anthers, without aspirating any stigmas. The most challenging task is to correctly remove anthers due to the closeness of the anther’s maximum terminal velocity with the minimum terminal velocity of stigmas. The difference between these velocities is about 8% referred to the mean stigma’s terminal velocity and the difference between maximum petals terminal velocity and minimum terminal velocity of stigmas, referred to the mean stigmas terminal velocity is about 25%. Due to the above consideration, the separation of anthers is very high but not complete.

4 Conclusion The system here designed and realized yielded the expected results both for the dynamic and aerodynamic characterization of flower parts and for the separation process; authors consider the wind tunnel to be essential to set the correct separation threshold. In addition, results clearly show the effectiveness of this type of separation system: separated material has a high grade of purity and, above all, there is not any loss or damage to the stigmas which are the most important part due to their low quantity and high value. The vision system is effective to evaluate the efficiency of the realized separation system. Authors are confident to be able to further improve separation efficiency, mainly by improving the

A Saffron Spice Separation System with Computer Vision

313

geometry of the tunnel and choosing other types of materials to build the separation machine. To improve production, the authors believe that a mechatronic controlled handling system for supplying the incoming material and one for the handling of the separate outgoing material may be effective. Acknowledgments. Thanks for the support of MIUR and AGRIS Sardegna referring to “L.R. n. 6/2012 articolo 3, comma 34 – Interventi regionali per l’attuazione della strategia comunitaria in agricoltura – Attuazione interventi relativi alla ricerca”.

References 1. Willer, H., Lernoud, J.: The world of organic agriculture statistics and emerging trend. Research Institute of Organic Agriculture FiBL, IFOAM – Organics International, Bonn, Germany (2018) 2. Bolan, N.S., Kunhikrishnan, A., Choppala, G.K., Thangarajan, R., Chung, J.W.: Stabilization of carbon in composts and biochars in relation to carbon sequestration and soil fertility. Sci. Total Environ. 424, 264–270 (2012) 3. Douglas, M., Perry, N.: Growing saffron - the world’s most expensive spice. Crop & Food Research, New Zealand Institute for Crop & Food Research (2003) 4. Gresta, F., Lombardo, G.M., Siracusa, L., Ruberto, G.: Saffron, an alternative crop for sustainable agricultural systems. A review. Agron. Sustain. Dev. 28(1), 95–112 (2008) 5. https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Wages_and_labour_costs. Accessed 02 Dec 2019 6. http://www.europeansaffron.eu/archivos/White%20book%20english.pdf. Accessed 29 Nov 2019 7. Concu, A., Garau, M., Manuello, A., Ricciu, R.: Human energy involved in manual and mechanically facilitate harvesting of saffron flowers. In: International Conference on Robotics in Alpe-Adria Danube Region (2018) https://doi.org/10.1007/978-3-319-61276-8_99 8. Emadi, B.: Separating saffron flowers parts using vertical air column. World Acad. Sci. Eng. Technol. 37, 25–28 (2009) 9. Moghanizadeh, A.: Detaching of saffron flower parts based on aerodynamic properties. Ama, Agric. Mech. Asia Africa Latin America 48(3), 14–19 (2017) 10. Gambella, F., Paschino, F., Bertetto, A.M.: Perspectives in the mechanization of saffron (Crocus Sativus L.). Int. J. Mech. Control 14(2), 3–8 (2013) 11. Saraf, S.A., Wani, M.H., Wani, S.A., Rauf, S.A., Baba, S.H., Shaheen, F.A., Mir, G.H., Dar, K.R., Chattoo, M.A.: Economics of saffron cultivation in Kashmir. Acta Hortic. 1200, 165– 176 (2018) 12. Kumar, V., Nirbhavane, S., Parihar, N.: Scope of mechanical harvesting of Saffron for economic development of farmers in Kashmir Region of Jammu & Kashmir, India. Indian J. Hill Farming (2019) 13. Gambella, F., Paschino, F., Bertetto, A.M., Ruggiu, M.: Application of mechanical device and airflow systems in the harvest and separation of saffron flowers (crocus sativus l). Trans. ASABE 56(4), 1259–1265 (2013) 14. Moghanizadeh, A., Neshat, M.: A study on the performance of the saffron separator for different air flows. Turk. J. Agric. Food Sci. Technol. 3(3), 148–153 (2015)

314

A. Manuello Bertetto and A. Prete

15. Manuello Bertetto, A., Ricciu, R., Obinu, M.: Mechanized trim of saffron flowers to prepare the stigma separation. Int. J. Mech. Control 18(2), 129–134 (2017) 16. Hoerner, S.F.: Fluid - Dynamic drag – theoretical, experimental and statistical information presented by Dr. Ing. S. F. Hoerner. Pratical Information on Aerodynamic drag and Hydrodynamic resistance, Published by the Author (1965) 17. Manuello Bertetto, A., Ricciu, R.: Mechanical harvester and double flow ciclone separator: prototypes to improve saffron spice productions. Int. J. Mech. Control 13(1), 9–14 (2012) 18. Perez-Vidal, C., Gracia, L., Gracia, C.: Computer vision applied to saffron flore (Crocus sativus L.) processing. Spanish J. Agric. Res. 9(4), 1176–1181 (2011) 19. Jafari, A., Bakhshipour, A., Hemmatian, R.: Integration of color features and artificial neural networks for in-field recognition of saffron flore. Iran Agric. Res. 33(1), 1–14 (2014)

Prismatic Delta Robot: A Lagrangian Approach Federico Colombo(&)

and Luigi Lentini

Department of Mechanical and Aerospace Engineering, Politecnico di Torino, Turin, Italy {federico.colombo,luigi.lentini}@polito.it

Abstract. This paper shows the kinematic and dynamic model of a prismatic Delta robot designed for a 3D printer. After description of the direct and inverse kinematic algorithms, the forces on the actuated prismatic joints are computed with the lagrangian approach. Examples of actuators forces computed with the lagrangian approach are shown in case of Lissajous type path imposed on the end effector (inverse kinematics) and traveled at constant speed. The results are checked with the energy conservation principle. The effect of inertia forces is evaluated at different printing speeds. Keywords: Prismatic delta robot Dynamic modelling


Lagrangian approach
Parallel robots


1 Introduction Delta robot structure is employed is several industrial applications, such as pick and place machines and 3D printers. An evident advantage respect to other structures, such as the cartesian one, is the higher acceleration that can be reached, due to the small moving masses. This robot was invented in 80s by Clavel [1]. It was then introduced into the market, initially for pick and place operation and packaging, then also for haptic controllers and other purposes (e.g. medical and pharmaceutical), last but not least for 3D printing [2, 3]. The kinematics of the delta robot with rotating actuators is proposed in [4], while in [5] the kinematic analysis is performed for the delta robot with prismatic actuators, with particular attention to singularities. The kinematic study for this structure is also carried out in [6]. About the dynamics, there are three methods to determine the actuators forces in dynamic conditions [7]: the Newton-Euler procedure, the principle of virtual works and the Lagrange’s equation with multipliers formalism. The principle of virtual work has been applied in [8] to calculate the dynamics of the prismatic delta robot. The lagrangian approach was employed to study the dynamics of a delta robot with rotational actuators [9], but to the best of author’s knowledge, this approach has not yet been employed to the delta robot with prismatic actuators. This paper is aimed at covering this literature lack. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 315–324, 2020. https://doi.org/10.1007/978-3-030-48989-2_34

316

F. Colombo and L. Lentini

2 The Prismatic Delta Robot The architecture of the delta robot with prismatic joints is depicted in Fig. 1. The robot is a parallel machine composed by a platform and three legs each of them is a serial kinematic chain of type P(SS)2. Due to the presence of the parallelograms in each leg, the platform can only translate along the three directions. The prismatic actuated joints are vertical and connect the three carts to the fixed basement. Points A1, A2 and A3 are defined in the middle of the short sides of the parallelograms in correspondence of the carts, while points P1, P2 and P3 are defined in the middle of the opposite sides, in correspondence of the platform. Points B1, B2 and B3 are the vertical projections of points A1, A2 and A3 on the basement plane. A fixed reference system OXYZ is located in the center of the basement, with X axis parallel to segment B1B2 and oriented from B1 to B2 and Y axis oriented towards point B3. Both triplets B1B2B3 and P1P2P3 identify equilateral triangles attached to the basement and to the platform respectively. The side lengths of these triangles are indicated as sB (triangle on basement) and sP (triangle on platform). Point P is located in the center of the platform.

Fig. 1. Architecture of the prismatic Delta robot

The dofs associated to the prismatic joints are indicated with q1, q2 and q3 and represent the coordinates along Z axis of points A1, A2 and A3 respect to fixed reference frame.

Prismatic Delta Robot: A Lagrangian Approach

317

3 Kinematics The direct kinematics study involves the calculation of position, velocity and acceleration of the platform when the input trajectories in prismatic joints are known. Vice versa, the inverse kinematics involves the calculation of the joints trajectories when the motion of the platform is supposed to be known. The closure equations are the following: OB1 þ B1 A1 þ A1 P1 þ P1 P ¼ OP OB2 þ B2 A2 þ A2 P2 þ P2 P ¼ OP OB3 þ B3 A3 þ A3 P3 þ P3 P ¼ OP

ð1Þ

Imposing that the length of segments A1 P1 , A2 P2 and A3 P3 is constant in time and equal to l, the length of the long side of the parallelograms, the following non-linear equations are derived:


2


 SB SP 2 þ yP þ pffiffiffi  pffiffiffi þ ðzP  q1 Þ2 ¼ l2 2 3 2 3

 SB SP 2 SB SP 2 þ xP  þ yP þ pffiffiffi  pffiffiffi þ ðzP  q2 Þ2 ¼ l2 2 2 2 3 2 3
 SB SP 2 x2P þ yP  pffiffiffi þ pffiffiffi þ ðzP  q3 Þ2 ¼ l2 3 3

3.1

SB SP  xP þ 2 2

ð2Þ

Direct Kinematics

Known q1, q2 and q3 and their first and second time derivatives, the direct kinematics allows to compute the coordinates xP, yP and zP of point P and their time derivatives. As system (2) is non-linear, it can be solved with an iterative method such as Newton Raphson or the three spheres intersection algorithm [6]. The velocities are easily computed solving a linear system resulting from the differentiation of system (2). These equations can be written in a more compact way using the matrix notation: 8 9 8 9 < x_ P = < q_ 1 = Jx y_ P ¼ Jq q_ 2 : ; : ; z_ P q_ 3

ð3Þ

where Jx and Jq are the Jacobian matrixes. It results 8 9 8 9 < x_ P = < q_ 1 = y_ P ¼ Jx1 Jq q_ 2 : ; : ; z_ P q_ 3

ð4Þ

318

F. Colombo and L. Lentini

The accelerations are obtained differentiating once more equations system (4) and expressing in matrix notation. It results 0 8 9 8 1 8 9 29 < €xP = < €q1 = < x_ 2P þ y_ 2P þ ðz_ P  q_ 1 Þ = B C €y ¼ Jx1 @Jq €q2  x_ 2P þ y_ 2P þ ðz_ P  q_ 2 Þ2 A : P; : ; : 2 ; €q3 €zP x_ P þ y_ 2P þ ðz_ P  q_ 3 Þ2

3.2

ð5Þ

Inverse Kinematics

The inverse problem is simpler, as the joint parameters q1, q2 and q3 can be expressed in explicit way as a function of the platform coordinates: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
2
2ffi S S S S B P B P q1 ¼ z P  l 2  x P þ   yP þ pffiffiffi  pffiffiffi 2 2 2 3 2 3 sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
2
 SB SP SB SP 2 2 q2 ¼ z P  l  x P  þ  yP þ pffiffiffi  pffiffiffi 2 2 2 3 2 3 sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
 SB SP 2 q3 ¼ zP  l2  x2P  yP  pffiffiffi þ pffiffiffi 3 3 The velocities are computed solving system 8 9 8 9 < q_ 1 = < x_ P = q_ 2 ¼ Jq1 Jx y_ P : ; : ; z_ P q_ 3

ð6Þ

ð7Þ

while the accelerations from system 0 8 9 8 1 8 9 29 < €q1 = < €xP = < x_ 2P þ y_ 2P þ ðz_ P  q_ 1 Þ = B C €q ¼ Jq1 @Jx €yP  x_ 2P þ y_ 2P þ ðz_ P  q_ 2 Þ2 A : 2; : ; : 2 ; €q3 €zP x_ P þ y_ 2P þ ðz_ P  q_ 3 Þ2

ð8Þ

4 The Lagrangian Approach   The generalized coordinate vector is defined as q ¼ q1 ; q2 ; q3 ; xp ; yp ; zp . This vector can be partitioned in the vector of the independent parameters q1 ; q2 ; q3 and the vector   of the dependent parameters xp ; yp ; zp : q ¼ qind ; qdep . The mass of each cart is mcart , the mass of each element between the spherical joints is mlink (note that there are 6 of these elements), while the mass of the platform is mplat . Figure 2 depicts a simplified lumped mass distribution, in which the mass of each link

Prismatic Delta Robot: A Lagrangian Approach

319

is splitted in equal parts at the two ends of the link. The extruder nozzle Q is located in point Q at distance PQ below point P. In this work all dissipations are neglected.

Fig. 2. Simplified lumped mass distribution

The forces Q1, Q2, Q3 on the actuators can be computed from the lagrangian Eq. (9), valid for j = 1 to 6:
 3 d @L @L X @Ci ¼ ki þ Qj  dt @ q_ j @qj @qj i¼1

ð9Þ

where ki (i = 1 to 3) are the lagrangian multipliers, which can also be computed solving system (9) and Ci (i = 1 to 3) are the closure equations. Qj are the generalized external forces, assumed to be null on platform: Q ¼ ½Q1 ; Q2 ; Q3 ; 0; 0; 0. System (9) can be written in the following matrix form ½ Af xg ¼ fbg

ð10Þ

320

F. Colombo and L. Lentini

where the unknown parameters vector is f x g ¼ f k1

k2

k3

Q1

Q2

Q 3 gT ¼



fkg fQg

 ð11Þ

and the constant vector fbg involves both inertia terms and the terms due to gravity: fbg ¼ ½M f€qg þ fF g

ð12Þ

Partitioning properly system (10), it is possible to obtain the expressions of the lagrangian multipliers and of the actuators forces:

4.1

8 9 

mplat þ 3mlink g  T 1 < 0 = mplat þ 3mlink  T 1 €qdep þ Jx Jx 0 fkg ¼ : ; 2 2 1

ð13Þ

8 91 08 9 > > = =

< €q1 > B C fQg ¼ ðmlink þ mcart Þ@ €q2 þ g 1 A > > > > : ; : ; 1 q€3 8 91 08 9 €xp > > > = =

<    T 1 B C þ mplat þ 3mlink Jq Jx @ €yp þ g 0 A > > ; ; : > : > €zp 1

ð14Þ

Static Solution

In case the accelerations are null the static solution is obtained: 8 9 8 9 l1 +l0 ; l0 +l3 > l1 +l2 , there is a lock in the movement of the crank link when it is aligned with the flexible coupler link as it can be seen in Fig. 6. This type of behavior is not noticeable in the case of the rigid coupler link because the deformation in this case are much lower than in the flexible case.

5

Conclusions

In this paper a four-bar mechanism with a flexible coupler link has been modelled and validated by simulation and hardware studies. It has been observed that for this particular case there is no significant difference between using the Timoshenko or Bernoulli beam theory from the viewpoint of precision in the state variables. However, the computational time is greater in the case of Timoshenko than in the case of Bernoulli beam theory. From experiments it has been observed that the use of vision would not be possible to control vibration mechanism because of the computing time for processing images. Hence the importance of having an accurate model.

References 1. Dogan, M., Morgul, O.: Nonlinear PDE control of two-link flexible arm with nonuniform cross section. In: American Control Conference Minneapolis, Minnesota, USA, pp. 400–405 (2006) 2. Low, K.H., Vidyasagar, M.: A lagrangian formulation of the dynamic model for flexible manipulator systems. J. Dyn. Syst. Meas. Control 110(2), 175–181 (1988) 3. Wenhuan, Y.: Mathematical modelling for a class of flexible robot. Appl. Math. Model. 19(9), 537–542 (1995)

Four Bar Mechanism

437

4. Bayo, E.: Timoshenko versus Bernoulli beam theories for the control of flexible robots. In: Proceeding of IASTED International Symposium on Applied Control and Identification, pp. 178–182 (1986) 5. Theodore, R.J., Ghosal, A.: Modeling of flexible-link manipulators with prismatic joints. IEEE Trans. Syst. Man Cybern. B Cybern. 27(2), 296–305 (1997) 6. Zhu, G., Ge, S.S., Lee, T.H.: Simulation studies of tip tracking control of a singlelink flexible robot based on a lumped model. Robotica 17(01), 71–78 (1999) 7. Megahed, S.M., Hamza, K.T.: Modeling and simulation of planar flexible link manipulators with rigid tip connections to revolute joints. Robotica 22(03), 285– 300 (2004) 8. Martins, J.M., Mohamed, Z., Tokhi, M.O., Da Costa, J.S., Botto, M.A.: Approaches for dynamic modelling of flexible manipulator systems. IEEE Proc. Control Theory Appl. 150(4), 401–411 (2003) 9. Mohamed, Z., Tokhi, M.O.: Command shaping techniques for vibration control of a flexible robot manipulator. Mechatronics 14(1), 69–90 (2004) 10. Karkoub, M., Yigit, A.S.: Vibration control of a four-bar mechanism with a flexible coupler link. J. Sound Vib. 222(2), 171–189 (1999) 11. Yang, K.-H., Park, Y.-S.: Dynamic stability analysis of a flexible four-bar mechanism and its experimental investigation. Mech. Mach. Theory 33(3), 307–320 (1998) 12. Ha´c, M.: Dynamics of planar flexible mechanisms by finite element method with truss-type elements. Comput. Struct. 39(1), 135–140 (1991) 13. Madenci, E., Guven, I.: The finite element method and applications in engineering R Springer, Boston (2015) using ANSYS. ¨ 14. Ulker, H.: Dynamic analysis of flexible mechanisms by finite element method. PhD ˙ thesis, Izmir Institute of Technology (2010) 15. Rao, S.S., Fook Fah, Y.: Mechanical Vibrations. Prentice Hall, Indianapolis (2011) 16. Tokhi, M.O., Azad, A.K.: Flexible Robot Manipulators: Modelling, SImulation and Control, vol. 68. IET, London (2008) 17. Wang, J., Gosselin, C.M., Cheng, L.: Modeling and simulation of robotic systems with closed kinematic chains using the virtual spring approach. Multibody Sys. Dyn. 7(2), 145–170 (2002) 18. Brasey, V.: A half-explicit runge-kutta method of order 5 for solving constrained mechanical systems. Computing 48(2), 191–201 (1992) 19. Kumpanya, D., Thaiparnat, S., Puangdownreong, D.: Parameter identification of bldc motor model via metaheuristic optimization techniques. Proc. Manuf. 4, 322– 327 (2015)

Some Quality Measures in Designing Compliant Mechanisms for Robotic Devices Stefan Havlik(&) and Jaroslav Hricko Institute of Informatics, Slovak Academy of Sciences, Banská Bystrica, Slovakia {havlik,hricko}@savbb.sk

Abstract. This paper deals with design problems of mechanically compliant mechanisms for robotic devices, as multi-component force and displacement sensors, or fast moving positioning devices. It presents a systematic view to quantifying the quality of design via criteria that should be taken into account. The problems of reachable sensitivity and accuracy in static and dynamic modes are analyzed. When designing any compliant structures the modeling and simulation techniques play most important role in evaluation of expected performance characteristics. The design criteria related to requirements are discussed in more details and two illustrative examples of compliant devices with different requirements are given. Keywords: Compliant mechanisms Accuracy


Compliance
Modeling
Stiffness


1 Introduction Compliant mechanical structures and mechanisms represent a broad class of mechanical system where displacements are reached by elastic deformation of their flexural parts/segments or the whole flexural body. The desired motion is the mechanical response on forces/torques applied on input ports. Such mechanisms are main functional parts of various small/micro positioning devices, or sensors of mechanical quantities (force) [1, 2]. Naturally, designing complex compliant structures that include elastic and relatively rigid elements suppose using techniques for force and compliance analysis, modeling and simulation of flexible structures, as well. The design techniques: the choice of structure, geometry and parameters that satisfy some optimal/compromise solution are widely elaborated and published [4–10]. Any design process includes evaluation of the quality; i.e. there are several criteria related to forces, displacement, dynamical performance and constraints that should be considered. Then, the multi-criteria optimization procedure should be applied to choose the best solution [3]. The goal of this paper is to point out that using very careful modeling and simulation procedures in the design process is useful and really need. The attention is devoted to mechanisms for multi-component force, or, displacement sensors and multid.o.f. positioning devices. The main parts: the compact compliant mechanisms exhibit several unwanted features, especially cross sensitivity/deflection effects that deteriorate © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 438–447, 2020. https://doi.org/10.1007/978-3-030-48989-2_47

Some Quality Measures in Designing Compliant Mechanisms for Robotic Devices

439

the performance quality of the device [6, 9, 11]. For this reason, any design should consider expected quality measures, as discussed below.

2 Design Criteria Robotic devices, we have in mind, can work in two modes: static (quasi static), or dynamic within a given frequency range. Another point of view is flexural motions of mechanisms, i.e. number of the d.o.f. in case of positioning devices, or number of force/displacement components for sensors. For both modes one can state the range of operations, as well as technical and operational constraints. There are: – Operational: dimensions, range of forces/motions/frequency, power limits of actuators… etc. – Technical constraints: material and technological constraints, limits of interfaces, HW/SW limits… etc. It is clear that any design should obey functional parameters that specify static and dynamic performance of the sensing, or positioning system. In the initial conceptual design the accuracy and the dynamical range of operation should be taken into consideration. 2.1

Accuracy

In cases of sensors, their compliant mechanical structure due to its finite stiffness is potential source of positional oscillations, or, force disturbances. So, the mechanism for a sensory device which works within a given frequency range should have a satisfactory high limit of natural frequency of mechanical oscillation [8]. To have the realistic view on possible performance of a sensory device the preliminary calculation concerning achievable accuracy and characteristics of mechanics should be made. For simplicity of next engineering approach we suppose a onedimensional case and linear model of the sensory function. Then, the input - output relation can be expressed by the second order differential equation _ þ kT uðtÞ ¼ f ðtÞ m€uðtÞ þ buðtÞ

ð1Þ

where f(t) denotes the unknown time variable force, u(t) is the deflection (mechanical strain) which corresponds to the output sensory signal and m, b, kT are mass, damping and stiffness/sensitivity parameters respectively. The sensor Laplace sensitivity will be SðpÞ ¼

UðpÞ 1 ¼ 2 FðpÞ p þ 2bx0 þ x20

ð2Þ

where b = g/2mx0 is the damping factor and x0 = kT/m is the frequency of free oscillations.

440

S. Havlik and J. Hricko

Assuming harmonic force changes the frequency characteristics for k = x/x0 is 1 x20 ð1  k2  2jbkÞ

ð3Þ

1 qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x20 ð1  k2 Þ2 þ 4b2 k2

ð4Þ

SðjkÞ ¼ Its amplitude part is then jSðkÞj ¼

and phases uðkÞ ¼ tan1

2bk 1  k2

ð5Þ

Let us divide the amplitude frequency response into the frequency independent part So(k = 0) that represents measurements of the static loads and the frequency dependent one Sk(k > 0) expressing the influence of the load frequency jSðkÞj ¼ S0 ðk ¼ 0Þ
Sk ðk  0Þ, where S0 ¼

1 m ¼ x20 k1

1 Sk ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2 ð1  k Þ þ 4b2 k2

ð6Þ

Introduce the relative amplitude error of the output signal by the relation nðkÞ ¼

S  S0 ¼ Sk  1 S0

ð7Þ

Then, after substitution of (6), the errors due to sensor dynamics will be 1 nðkÞ ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi  1 2 2 ð1  k Þ þ 4b2 k2

ð8Þ

Because of in real sensor constructions the damping is considered to be very weak (so called structural damping), neglecting friction, one can assume a worst possible case with the damping factor value of zero (b = 0). In practical applications one can also consider xo  x. Then the values of parameters k4, b2, k2, in above relations, comparing with the other values, can be neglected. After such a simplification one can get nðkÞ ¼ k2

ð9Þ

Some Quality Measures in Designing Compliant Mechanisms for Robotic Devices

441

At the stage of the conceptual design the requirement for the sensor dynamics can be formulated in the following way: Let x be the supposed maximal frequency of the force/displacement oscillations. In order to get the proposed (sensing/positioning) accuracy within the whole frequency range 0 – x, the minimal frequency of free oscillations of mechanisms should be x x0  pffiffiffi n

ð10Þ

The above relation which represents the dynamical quality measure enables a brief of expected performance by relatively simple engineering way. 2.2

Sensitivity (Cross Sensitivity/Cross Displacement)

In cases of mechanisms for multi-component force sensors or multi d.o.f. positioning apparatus the vectors of actuating loads and displacements are related. As introduced, for simplicity, the elastic deflections are small (within the validity of Hooke’s low) and the flexure behaves as a linear force – displacement system with validity of the superposition principle. Then, the mechanisms can be described by the linear model representation. The above conclusion that minimal frequency of natural mechanical oscillations must be verified, it results in solving the following equation i.e. solving the eigenvalue problem, ðK  x2 MÞ/ ¼ 0

ð11Þ

where K and M are stiffness and mass matrices of the mechanical system and / are the mode shapes. For the correct function of the sensing/positioning mechanism the condition (10) should be satisfied for all frequencies from (11). In an ideal case the best quality of the mechanisms will be reached when each direction/component exhibits the same sensitivity. The general method used for examination characteristics of the linear transformation matrix is the singular value decomposition (SVD) [4, 8, 11]. Using SVD the K matrix can be described in form K ¼ G:U:HT

ð12Þ

where G 2 ℜmm, H 2 ℜ66 are orthogonal matrices, U = diag[u1, u2, …, uk, 0, …, 0]; u1,  u2  …  uk are singular values of the K matrix. Solving the Eq. (12) is then very well conditioned if the condition number will be j¼

u1 ¼1 _ uk

ð13Þ

442

S. Havlik and J. Hricko

This means that all components will have approximately the same sensitivity. The geometric interpretation of this analysis is in the form of a generalized ellipsoid, where the lengths of its main axes are singular values with orientation given by columns of the G matrix. The meaning of singular values/form of ellipsoid consists in two main inferences: – It gives a global view on the mechanism as to the deflections and errors propagation. – It allows comparing the “kinematic/flexural quality” of various structures/ configurations. 2.3

Dynamics of Multi-D.O.F. Positioning Mechanism

For slowly moving mechanisms the errors are mainly due to cross deflection/ compliance effects and represent quasi-static departures from desired positions. They can be easily compensated by feed-forward ways using compliance models of the mechanisms [4, 8, 11]. Much more complex situation arises in cases of the fast moving mechanisms, where the stiffness and damping coefficients in particular d.o.f. are different. To show this dependence the brief and simple simulation of the two d.o.f. mechanisms will be made. Consider a linear system m€y þ By_ þ Ky ¼ Bx_ þ Kx

ð14Þ

where K, B are the diagonal stiffness and damping matrices with k1, k2 and b1, b2 - the diagonal coefficients. Solving (14) will give the transfer functions in form YðsÞ ¼ GðsÞ
XðsÞ

ð15Þ

where G(s) is the matrix of transfer functions 

G11 GðsÞ ¼ 0

0 G22

"

 ¼

b 1 s þ k1 ms2 þ b1 s þ k1

0

0 b2 s þ k2 ms2 þ b2 s þ k2

# ð16Þ

In order to see how different stiffness and damping parameters will result in the final accuracy in particular directions the amplitude errors are calculated. For calculations within given frequency range the following values of parameters were stated: m = 1.10−3 kg; k1 = 2.5; b1 = 2.5. The errors in amplitudes are then calculated for five ratios of stiffness kp = k1/k2 and damping coefficients bp = b1/b2 (Fig. 1). As can be seen in Fig. 1, the difference of stiffness and damping coefficients results in amplitude errors, especially in phases at higher frequencies.

Some Quality Measures in Designing Compliant Mechanisms for Robotic Devices

443

Fig. 1. The amplitude and phase errors for different stiffness and damping

3 Some Design Examples 3.1

Linear Motion Reducer

The task was to design an elastically compliant mechanism - motion reducer/force amplifier for very precisely adjustable hinge for heavy weight platforms [13]. This is a typical example of compliant mechanism for quasi static mode of operation with the given reduction ratio [5, 6, 9, 14–16, 18]. The first proposal considered rigid arms and elastic joints (Fig. 2a).

Fig. 2. Linear motion/force amplifier

As showed calculations using FEM analysis in Comsol Multiphysics the mechanical stress is mainly concentrated in joints and over-crossed the allowable values. For this reason the final solution is in distributed compliance i.e. in two arms structure shown in Fig. 2b. Final configuration of the whole positioning mechanisms fixed on the test-stand for experimental measurements is shown in Fig. 3a. The experiments have been made in

444

S. Havlik and J. Hricko

specialized metrological laboratory. The positioning mechanism was loaded by several mass of 0, 300, 600 kg and position of the mass body was measured. It was confirmed that the positioning accuracy of some 10−4 mm can be reached (Fig. 3b). Base

M

LVDT sensor 10-3 mm

Load

Laser distance sensor, 10-4 mm

±3mm

a)

b)

Fig. 3. a) Configuration of whole positioning mechanism, b) Errors of output motion

3.2

Two D.O.F. Linear Motion Mechanism for Fast Moving Apparatus

The task is to design the planar mechanism for the X-Y micro-positioning table working within the range of displacement ±0.5 mm with frequency up to hundred motioncycles per second. External dimensions of this mechanism should be approx. 20  20 mm. The device should be actuating by PZT actuators. The design concept and proposed structure of 2 d.o.f. compliant mechanisms of sensor/positioning device are shown in Fig. 4a, b [17]. Considering previous study it was need to respect the symmetry and minimal cross deflection effects within the whole range of motions [12, 17]. Three different variants were simulated with results in Table 1. Consequently, the best configuration (c) was chosen. Then, for chosen variant, the flexural analysis and motion simulations were made (Fig. 5).

Fig. 4. The design concept and compliant structure of the positioning mechanisms/sensor

Some Quality Measures in Designing Compliant Mechanisms for Robotic Devices

445

Table 1. Comparing cross deflections for various configurations of the flexure (a)

(b)

(c)

Motion x [mm]

0,4814

0,7416

0,5300

Cross motion y [mm]

0,0808

0,0881

0,0034

Cross error x

16,8064

11,8836

0,6507

13,707

15,617

15,599

Kinematic configuration

Max. stress [MPa]

y [%]

Fig. 5. a) Surface stress, b) Visualization of flexural displacements

4 Conclusion Flexural characteristics of compliant mechanisms determine final performances of such devices the design procedure should include building mathematical models that enable to optimize the geometry and parameters of the mechanisms. For this reason flexural characteristics of the mechanical structure should be carefully analyzed. The above brief discussion gives the more detailed view on criteria that should be taken into account when designing such mechanisms. It is shown that available simulation methods are very useful tools for design of devices that exhibit desired performance characteristics. Acknowledgment. This work was supported by the national scientific grant agency VEGA under project No.: 2/0155/19 – “Processing sensory data via Artificial Intelligence methods”.

446

S. Havlik and J. Hricko

References 1. Howell, L.L.: Compliant Mechanisms. Wiley, New York (2001) 2. Lobontiu, N.: Compliant Mechanisms: Design of Flexure Hinges. CRC Press, Boca Raton (2003). ISBN 0849313678 3. Carbone, G., Ceccarelli, M., Penisi, O.: An optimum design of a mechanism for microgrippers. In: 16th Brazilian Congress of Mechanical Engineering, COBEM 2001, Uberlandia, pp. 194–203 (2001) 4. Havlík, Š., Carbone, G.: Design of compliant robotic micro-devices. In: Proceedings of the 15th International Workshop on Robotics in Alpe-Adria-Danube Region, RAAD 2006, Balatonfured, Hungary, 15–17 June 2006 (2006). ISBN 963-7154-48-5 5. Li, Y., Xu, Q.: Optimal design of a novel 2-dof compliant parallel micromanipulator for nanomanipulation. In: Proceedings of the 2005 IEEE International Conference on Automation Science and Engineering, Edmonton, Canada, 1–2 August 2005 (2005) 6. Li, Y., Xu, Q.: Design and analysis of a totally decoupled flexure-based XY parallel micromanipulator. IEEE Trans. Robot. 25(3), 645–657 (2009) 7. Nashrul, M., Zubir, M., Shirinzadeh, B.: Development of a high precision flexure-based microgripper. Precis. Eng. 33, 362–370 (2009) 8. Havlík, Š.: Modeling, analysis and optimal design of multi-component force/displacement sensors. In: Proceedings of the First ECPD International Conference on Advanced Robotics and Intelligent Automation, Athens, Greece, 6–8 September 1995, pp. 283–289 (1995) 9. Awtar, S., Slocum, A.H.: Design of parallel kinematic x-y flexure mechanisms. In: Proceedings of IDETC/CIE 2005, ASME 2005 International Design Engineering Technical Conferences, Computers and Information in Engineering Conference, Long Beach, California, USA, 24–28 September 2005, pp. 89–99 (2005) 10. Hricko, J., Havlík, Š.: Design and optimization of mechanics for two axes force/displacement E-M sensor. In: Proceedings of the 11th International Conference on Measurement, MEASUREMENT 2017, pp. 99–102. IEEE Xplore, SCOPUS (2017). ISBN 978-80-972629-0-7 11. Havlík, Š.: Improving accuracy of compliant robotic (micro) devices. In: Szakál, A. (ed.) Proceedings of the RAAD 2010, pp. 385–389. IEEE, Budapest (2010). ISBN 978-1-42446884-3 12. Havlík, Š., Hricko, J.: About the accuracy of fast moving robotic devices based on compliant mechanisms. In: Advances in Robot Design and Intelligent Control. AISC, vol. 540, pp. 162–170. Springer, Wroclaw (2016/2017). ISBN 978-3-319-49057-1. ISSN 2194-5357 13. Havlík, S., Hricko, J., Prada, E., Jezny, J.: Linear motion mechanisms for fine position adjustment of heavy weight platforms. In: Berns, K., Görges, D. (eds.) Advances in Service and Industrial Robotics, RAAD 2019. AISC, vol. 980, pp. 19–25. Springer, Cham (2020). ISBN 978-3-030-19648-6 14. Chen, J., Zhang, C., Xu, M., et al.: Rhombic micro-displacement amplifier for piezoelectric actuator and its linear and hybrid model. Mech. Syst. Signal Process. 50, 580–593 (2015) 15. Ling, M., et al.: Enhanced mathematical modeling of the displacement amplification ratio for piezoelectric compliant mechanisms. Smart Mater. Struct. 25 075022 (2016). ISSN 1361665X

Some Quality Measures in Designing Compliant Mechanisms for Robotic Devices

447

16. Qi, K.-Q., Xianga, Y., Fang, Ch., Zhangb, Y., Yu, Ch.-S.: Analysis of the displacement amplification ratio of bridge-type mechanism. Mech. Mach. Theory 87, 45–56 (2015) 17. Havlik, Š.: Design of smart robotic mechanisms. In: Proceedings of the 14th International Workshop on Robotics in Alpe-Adria-Danube Region, RAAD 2014, Smolenice - Slovakia, June 2014. ISBN 978-1-4799-6798-8 18. Kang, B.H., Wen, J.T.Y., Dagalakis, N.G., Gorman, J.J.: Analysis and design of parallel mechanisms with flexure joints. In: Proceedings of IEEE International Conference on Robotics and Automation, pp. 4097–4102 (2004)

Prototype Design and Testing of TORVEastro, Cable-Driven Astronaut Robot Francesco Samani(&)

and Marco Ceccarelli

LARM2: Laboratory of Robot Mechatronics, University of Rome Tor Vergata, Rome, Italy [email protected], [email protected]

Abstract. A prototype of TORVEastro robot is designed and built for testing activity to check the soundness of the robot design and to characterize its performance. A CAD model is used to design and built a prototype with 3D printed parts and commercial components. The built prototype is tested to verify the operation efficiency and to evaluate the performance characteristics. Keywords: Space robotics Astronaut robots


Experimental robotics
Design
Testing


1 Introduction Space Robotics is considered one of the most promising approaches for On-Orbit Servicing (OOS) missions such as docking, berthing, refueling, repairing, upgrading, transporting, rescuing, and orbital debris removal [1]. With the development of space technology, more complex space tasks will be carried out in the future. Space robots are increasingly playing a key role in on-orbit servicing operations as service robots [2]. Typical Robots are already developed for medical care, space exploration, demining operation, surveillance, entertainment, museum guide. According to the International Federation of Robotics (IFR), “a service robot is a robot, which operates semi or fully autonomously to perform services useful to the wellbeing of human and equipment, excluding manufacturing operations” [3]. Currently an astronaut robot in-operation is Robonaut (NASA), a humanoid robot, that has been developed by the Robotics Laboratory at the Lyndon B. Johnson Space Center (JSC) of NASA in Houston, Texas [4]. The astronaut robot from ESA is Rollin’ Justin has four-finger hands with a mobile base for autonomous operation over a long range [5]. Kirobo is Japan first robot astronaut (JAXA) [6] that was developed by University of Tokyo. Skybot F-850 is a Russian service space humanoid robot, which have been launched from Kazakhstan for the International Space Station [7]. The European Space Agency (ESA) and the Italian Space Agency (ASI) are currently developing a space robot system called JERICO that is planned to be installed in the SPEKTR module of the Mir station. JERICO consists of a seven-axes robot with a sensorized end-effector [8]. Several key technologies have made ROTEX robot technology experiment: multisensory gripper technology, local

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 448–455, 2020. https://doi.org/10.1007/978-3-030-48989-2_48

Prototype Design and Testing of TORVEastro, Cabledriven Astronaut Robot

449

sensory feedback control concepts, and the powerful delay-compensating 3D-graphics simulation (predictive simulation) in the telerobotic ground station [9]. Human and Robot Interaction (HRI) plays an important role in future planetary exploration mission, where astronauts with extravehicular activities (EVA) have to communicate with robot assistants [10]. The application of a service robot in a space orbital station have to consider characteristics and constrains of space environment as outlined in [11]. Microgravity makes motion easy but the stability of mechanisms can be very sensitive to little vibrations. Radiation in the space can be dangerous for actuators, controllers, drivers, sensors and in general for the electronic devices [12]. Moreover, the space is characterized by strong temperature changes. In orbital stations, energy source is very limited and therefore energy consumption must be kept at the lowest level [13]. It is necessary to consider that it is hard to transport objects and in order to make the transportation of the space service robot to space station easy, a robot should be as light and small as possible, as stressed in [14]. In this paper TORVEastro, a three-link astronaut robot, is presented with a conceptual design that is used also in a CAD design for performance simulations. The TORVEastro robot is designed for service applications in space stations and characteristics are discussed with the aim to identify design problems and operation features. A study of feasibility is discussed through performance evaluation by using kinematics and dynamic simulation results as reported in [15]. The paper reports experiences with a built prototype and its testing for performance evaluation.

2 TORVEastro CAD Design The proposed TORVEastro service space robot is designed with three arm-leg as in Fig. 1 with a central body and each arm-leg of two links (Fig. 1a). There are 3 symmetric arm-legs and each of them has 3 DoFs and three rotary joints that allow the rotation along h1, h2 and h3, Fig. 1b. The central body of the robot has cylindrical dimensions of 20 cm of radius and of 20 cm of height. The total mass of the space robot is less than 5 kg and its robot arm length is of 70 cm. The inside cross-section of the link allows the use of cables for sensors and actuators. The symmetrical geometry of the legs makes them interchangeable thanks to the possibility of adopting a structure with multi-functional end-effectors. The CAD design in Fig. 1b shows a section of the cylindrical body of the robot with only 1 arm-leg. The robot has three leg-arms and everyone consists in two links, L11 and L21. The first link has two DoFs whereas the first DoF gives the possibility to rotate the angle h1 and it has an angular velocity direction orthogonal to the lateral surface of the central body. The second link has 1

450

F. Samani and M. Ceccarelli

DoF and it is able to rotate h2 angle. An IMU sensor (inertial measurement unit) is used to monitor position, velocity and acceleration data. The second DoF refers to the direction orthogonal to the plane surface of the central body and it allows the possibility to rotate the angle h2. The second link L21 can rotate of h3 angle with an angular velocity respect to the first link. To have low inertia and to not expose the engines to harsh weather conditions, the motors are located inside the central body and they move the links by cables.

Fig. 1. A CAD model for mechanical design of TORVEastro: a) overall design; b) a detail of an arm structure with parameters

The point H1 represents the wrist position of the end-effector. Referring to [10] the maximum benefit in relation to mass and mechanical characterization is obtained when the thickness of the first link is 2,5 mm and the thickness of the second link is 1 mm. TORVEastro offers the possibility of facilitated movements in the outdoor space of orbital stations in order to be able to reach objects. The service robot can move along on the rods and handrails by using the 3 arm-legs. All electronic and sensitive parts can be repaired easily since the structure is easily accessible in each of its parts.

Prototype Design and Testing of TORVEastro, Cabledriven Astronaut Robot

451

Fig. 2. Details of actuating solutions for the arm-leg design in Fig. 1b

In Fig. 2 when the link of the robot does not moving the F01L = F01R (where F01L is the actuating force by servomotor to the link in the left side and F01R in the right side). These two forces give firm configuration to the link. Since diameter of the steel cable is 0,05 mm and section is 0,02 mm2, the stress pulling is 2 N with a tension of 100 Pa, well lower that the admitted. The servomotors 1A and 3A in Fig. 3 can rotate alternately counterclockwise and clockwise toward and this creates a torque motor that give rotation to the link L21 along h2 and h3.

3 A Prototype and Testing Layout Figure 3a shows the prototype of TORVEastro built in LARM2 at University of Rome Tor Vergata. The structure of the robot is 3D printed in PLA filament (Polylactic Acid). In position A, servomotors actuating cables are fixed inside the central body of the

a)

b)

Fig. 3. TORVEastro: a) A built prototype at LARM2 in Rome; b) solution for mechanical transmission

452

F. Samani and M. Ceccarelli

space robot. In B the Arduino Mega 2560 is 5 cm above the lipo battery that supplies 7,4 V and 6000 mAh. Cables and motherboards are positioned between battery and Arduino. Servomotor 1 allows the rotation of link L11 for the angle h1 by a gear transmission between motor and link in Fig. 3b. Servomotor 1 moves a spur gear that has module 1 and 19 teeth. Spur gear is linked with internal spur gear that has the same module 1 and 45 teeth thus it is possible to have proper torque and a proper capability of movement with h1 angle. The links L11 and L21 are controlled by tensioned and lubricated cables. Figure 4 summarizes the testing layout at LARM2. Using Arduino program is possible to control properly the movement of the links. A lipo battery gives power to the servomotors linked to Arduino by electric cables. A capacitor is used to reduce electric noise and stabilize the voltage. A capacitor with 1 lF is used to decouple servomotors of electrical network circuit from other parts. Noise caused by other circuit elements is shunted through the capacitor. A breadboard is used host all the devices together.

Fig. 4. A testing layout of TORVEastro at LARM2 Rome

4 Test Results Test are carried out at LARM2 by using the built prototype in Fig. 3 within a testing setup as in Fig. 4 to check the feasibility of the TORVEastro operation and to characterize its performance. A snapshot during a test in Fig. 5 shows the movements of an arm-leg by checking its operation in basic motion.

Prototype Design and Testing of TORVEastro, Cabledriven Astronaut Robot

453

Fig. 5. A snapshot during testing of TORVEastro prototype of Fig. 3: a) and b) rotation in h1; c) and d) simultaneous rotation in h2 and h3

Experimental tests of performance evaluation have been repeated 5 times to characterize the repeatability of a movement. Only one arm-leg is analyzed for operation characterization since the symmetric geometry of TORVEastro design. The experimental results were acquired in Fig. 6 by using IMU sensor in terms of leg joint angle position, velocity and acceleration to stress the feasibility of the operation of TORVEastro space robot in basic operations. The end-effector moved from start point to end point using same angular velocity for all servomotors. Max angle range in terms of rotation is 100-degree, max angular velocity is 40 rad/s and max angular acceleration is 15 rad/s2. The results show that the robot can be considered feasible for a practical implementation in service space tasks since the proper motions and actions can be experienced.

454

F. Samani and M. Ceccarelli

Fig. 6. Experimental results of a test like in Fig. 5 in terms of motion of leg-arm extremity: a) orientation angle; b) angular velocity; c) angular acceleration

Prototype Design and Testing of TORVEastro, Cabledriven Astronaut Robot

455

5 Conclusions A study of feasibility is discussed through performance evaluation by using kinematics and dynamics simulations also in Ref. [15] with results showing feasibility of TORVEastro robot operation and its peculiarities in order to define a proper full design. By analyzing operation problems and requirements of space stations, a new kind of 3 arm-leg cable driven robot has been proposed and simulated for helping astronauts. The conceptual design of TORVEastro as a cable-driven astronaut robot with three arm-leg limbs of same design is characterized with a lab prototype that is presented with a compact light weighted solution. The feasibility of its operation is validated with lab testing to check the operation efficiency and basic performance in the motion of the arm-leg limb design. TORVEastro robot is proposed for service applications to help astronauts in assembling and other tasks like repairing and monitoring works outside space orbital stations.

References 1. Flores-Abad, A., Ma, O., Pham, K., Ulrich, S.: A review of space robotics technologies for on-orbit servicing. Prog. Aerosp. Sci. 68, 1–26 (2014) 2. Ceccarelli, M. (ed.): Service Robots and Robotics: Design and Application, Engineering Science Reference, pp. 1–14 (2012) 3. IFR: Service Robot. https://ifr.org/service-robots/. Accessed 14 Feb 2020 4. Csorba, K., Varga, D., Tevesz, G., Vajk, I.: The RobonAUT autonomous mobile robot construction contest. IFAC Proc. Volumes 9(PART 1), 360–365 (2012) 5. Sagath, D., Papadimitriou, A., Adriaensen, M., Giannopapa, C.: Space strategy and governance of ESA small member states. Acta Astronautica 142, 112–120 (2018) 6. Tsuchida, A., et al.: Acta astronautica KIBO (Japanese experiment module in ISS) operations—2008/3–2009/10. Acta Astronautica 68(7–8), 1318–1324 (2011) 7. ITMO: Physical sciences. https://news.itmo.ru/en/science/cyberphysics/news/8710/. Accessed 02 Dec 2020 8. Didot, E., Dettmann, J., Losito, S., Torfs, D., Colombina, G.: JERICO: a demonstration of autonomous robotic servicing on the Mir space station. Robot. Auton. Syst. 23, 29–36 (1998) 9. Hirzinger, G., Brunner, B., Dietrich, J.: ROTEX-the first remotely controlled robot in space. In: Proceedings of the 1994 IEEE International Conference on Robotics and Automation, San Diego, CA, USA, vol. 3, pp. 2604–2611 (2002) 10. Liu, J., Luo, Y., Ju, Z.: An interactive astronaut-robot system with gesture control. Comput. Intell. Neurosci. 1–11 (2016) 11. Yoshida, K., Wilcox, B.: Space robots and systems. In: Handbook of Robotics, pp. 1031– 1063. Springer, Heidelberg (2008) 12. Ceccarelli, M., Li, H., Carbone, G., Huang, Q.: Conceptual kinematic design and performance evaluation of a chameleon-like service robot for space stations. Int. J. Adv. Robot. Syst. 12, 17 (2015) 13. Yoshida, K.: Achievements in space robotics. IEEE Robot. Autom. Mag. 16, 20–28 (2009) 14. Sun, Z., Li, H., Jianga, Z., Song, Z., Mo, Y., Ceccarelli, M.: Prototype design and performance tests of beijing astronaut robot. Appl. Sci. 8, 1342–1353 (2018) 15. Samani, F., Ceccarelli, M.: Design and performance simulation of TORVEastro three-link astronaut robot. In: IOP Conference Series Materials Science and Engineering, vol. 659, p. 012010 (2019)

Design and Lab Experience for a Pipeline Service Robot in Space Orbital Stations Tao Zheng1, Hui Li1(&), Qiang Huang1, Xiang Wang1,3, Marco Ceccarelli1,2, Zhihong Jiang1, Pingli Wang1, Sanliang Wang1, Yanyan Zhao1, and Xiao Liu1 1

2

Key Laboratory of Biomimetic Robots and Systems, Beijing Institute of Technology, Beijing 100081, China [email protected] Lab of Robot Mechatronics, University of Rome Torvergata, Rome, Italy [email protected] 3 China Academy of Space Technology, Beijing 100081, China

Abstract. The inspection and maintenance of ventilating pipelines in space orbital stations is difficult and time-consuming for astronauts. In this paper, a novel pipeline robot with three parts is proposed to replace astronauts in inspecting ventilating pipeline. An experimental testbed is designed to test the operation feasibility of pipeline robot in pipelines with different cross-sections and shapes, including cases with obstacles and ruptures. Ground test results show that the proposed BIT pipeline robot is able to move smoothly in most of pipelines with different cross-sections and shapes. Keywords: Space robotics


Pipeline robot
Design
Lab testing

1 Introduction A space orbital station provides a long-term service spacecraft to astronauts with a properly livable environment for astronauts [1]. The ventilation system in space orbital stations is a crucial device for keeping the air quality and temperature constant at proper values [2]. Environmental parameters inside of space orbital station are closely related to astronaut health, and therefore keeping the ventilating pipeline unobstructed and clear is vital importance. Currently, there are many different kinds of pipeline robots, but most of them are used in gravity field rather than in micro-gravity field. In 2002, Hyouk Ryeol Choi et al. from Sungkyunkwan University designed a pipeline robot as an articulated structurelike a snake with a tether cable to move in complicated structures of the pipeline networks [3]. In 2005, Segon Roh et al. from Sungkyunkwan University published a paper about pipeline robot control algorithm [4], with wheeled motion to move in different shaped pipelines. But it was experienced difficult to go through the pipeline joint and was only able to apply to pipeline with diameter from 85 mm to 109 mm [5]. In [6], a pipeline robot is presented with three legs and three compliance active joint connecting tracks as reliable and simple in structure. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 456–465, 2020. https://doi.org/10.1007/978-3-030-48989-2_49

Design and Lab Experience for a Pipeline Service Robot

457

In this paper, a novel pipeline robot with three parts is presented which is able to move smoothly in pipelines with diameter from 70 mm to 180 mm and different shapes. A pipeline testbed is designed to test the motion of the pipeline robot and the motion information is captured by suitable sensors. Results are discussed to show the feasibility of the proposed design and to characterize its operation performance.

2 Design Issues and Requirements Due to the limited space in a space orbital station, the arrangement of ventilation pipeline should be completed according to the internal shape of space orbital station. In order to save space and transport air to different room, pipelines are designed with different cross-sections and shapes, as in the example in Fig. 1. 300mm The maximum inner diameter

2mm Thickness

ø100mm

ø 100mm

ø100mm ø 180mm

The minimum inner diameter

a)

b) 20mm Pipeline

Maximum clearances

Pipeline

ø100mm

Inner diameter

c)

d)

Fig. 1. Examples of pipeline shapes encountered by pipeline robot, a) with continuously varying diameters, b) T-shaped pipeline, c) S-shaped pipeline, d) Pipeline joint

The BIT pipeline robot is able to move in pipeline with diameter range from 70 mm to 180 mm within a testbed summarizing the situations in Fig. 1. Figure 1 a) shows the pipeline with continuously varying diameter from 100 mm to 180 mm, and the length of continuously varying diameter pipeline is 300 mm. Figure 1 b) shows the T-shaped pipeline, the diameter of pipeline is 100 mm and the angle between two pipelines is 90 degree. Figure 1 c) represents S-shaped pipeline whose diameter is 100 mm and the radius of curvature is 100 mm. Figure 1 d) represents a pipeline joint whose distance between the two pipelines is 20 mm.

458

T. Zheng et al.

The velocity of air flow in a ventilating pipeline is from 3 m/s to 5 m/s and keeping a pipeline robot stationary against such an air flow is one of the basic features for operation efficiency. The length of ventilating pipeline is longer than 30 m, therefore, a pipeline robot should be able to move autonomously along more than 30 m, with a duration of more than one hour. With the above characteristics, the purpose of monitoring whole pipeline environment can be completed by such a pipeline robot.

3 The BIT Pipeline Robot The BIT pipeline robot aims to monitor and clean obstacles inside of pipeline mounted in the space orbital station, so moving smoothly inside of pipeline is an essential function to monitor and clean obstacles. In Fig. 2, the Beijing Institute of Technology (BIT) pipeline robot is conceived as six units. The pipeline robot is a snake-like structure with head, waist, and tail and is represented in Fig. 3. Unit 1 is energy supply unit and the number of energy supply method is two. One kind of energy supply method is battery-powered, another kind of energy supply method is wired-powered. Unit 2 is motion sensor and consists of hall sensors and optical encoders. Unit 3 is motion control unit which is a motor control circuit. Unit 4 is internal drive control unit which is the micro control units (MCU). Unit 5 is vision unit and used to shoot video inside of pipeline. To make full use of space resource and to decrease the energy loss in the wires, battery and fine wires are used to supply power at same time. Unit 6 is attitude measuring unit which consists of a group of potentiometers.

U6:Attitude measuring unit

U5: Vision unit

Pipeline Robot

U4:MCU

U3: Motion control unit

U1: Energy supply unit

U2: Motion sensor unit

Fig. 2. A conceptual design of BIT pipeline robot for Space Orbital Stations

Figure 3 a) represents a CAD design of pipeline robot and Fig. 3 b) shows a prototype. In Fig. 3 a), T-represents tracks, S-represents supporting arm controllers, Rrepresents hinges and C-represents cameras. The mechanical design of pipeline robot head and tail is the same. The head has four tracks and four supporting arms. A supporting arm is composed by two parallelogram mechanisms with connections between tracks and pipeline robot body. The angle between two supporting arms is 90°. The

Design and Lab Experience for a Pipeline Service Robot

459

Fig. 3. The BIT pipeline robot for space orbital stations; a) a CAD design; b) a built prototype at BIT in Beijing

structure with four supporting arms and four tracks is more reliability than the structure with three supporting arms. The diamond crossing construction is able to make the supporting arm to stretch and shrink, so that tracks of pipeline robot are able to contact the pipeline wall with different diameters. The size range of supporting arm is from 70 mm to 180 mm. One reason why we choose tracks rather than wheels is that tracks can provide larger contact area than wheels and they can keep the pipeline robot body more stable. In addition, tracks can cross a pipeline joint, while wheels cannot. The two group of diamond crossing constructions can be controlled independently, so that the angle between tracks and pipeline robot axis can be adjusted. Under the driving of pipeline robot waist, the relative attitude between head and tail can be adjusted.

460

T. Zheng et al.

4 Test-Bed Layout A BIT pipeline robot prototype has been built, and a testbed was designed to characterize its operation performance in proper testing. Figure 4 a) is a CAD design of pipeline robot testbed, and Fig. 4 b) is a test bed made by acrylic resin. The acrylic resin has good physical properties such as high rigidity and high transparency, so that the kinetic characteristic of pipeline robot can be easily observed. This testbed consists of S-shaped pipelines, T-shaped pipelines, continuously varying diameters pipelines, and pipelines with different diameters.

Fig. 4. A design of a testbed for BIT pipeline robot; a) a CAD design; b) a built testbed

5 Test Results In order to test the kinetic characteristic of the BIT pipeline robot, experiments have been carried out using the properly designed testbed simulating characteristic operations. The BIT pipeline robot was placed inside of testbed pipelines with different shapes and cross-sections to test the motion characteristics as reported in Fig. 5, Fig. 6 and Fig. 7 shows, with the feedback data of motors captured during tests. As the Fig. 5 c) and Fig. 6 c) shows, m11, m12, m13 represent track driving motors installed at the head of pipeline robot and m21, m22, m23 represent track driving motors installed at the tail of pipeline robot. In order to keep the same rotating speed of tracks, the control mode of track driving motors is speed control mode. The speed curve of tracks shown in Fig. 8, 9 and 10 are overlap and the speed is constant. In Fig. 8, 9 and 10, the speed curve of tracks shown in figure c) and figure d) are overlap and the rotating position is zero. From Fig. 8, 9 and 10, the purple curve coincides with blue curve at the beginning. The purple curve denotes screw driving motor rotating speed curve or relative rotating position curve. The blue curve denotes roll motion driving motor rotating speed curve or relative rotating position curve. ms1 and ms2

Design and Lab Experience for a Pipeline Service Robot

461

m13 m12

m14

a)

b)

m11 mp2

c)

Fig. 5. The snapshots of the BIT pipeline robot moving inside of testbed pipelines with diameter is 100 mm; a) at the beginning of pipeline testbed; b) in the middle of pipeline; c) at the end of pipeline with indication of main components

m14

m13

m12

m11

ms3 ms4 mp2

mr2

mr1

ms1 ms2 m24

a)

b)

m23

m22

m21

c)

Fig. 6. The snapshots of the BIT pipeline robot moving inside of testbed pipelines with diameter of 180 mm; a) at the beginning of pipeline testbed; b) in the middle of pipeline; c) at the end of pipeline with indication of main components

462

T. Zheng et al.

Fig. 7. The snapshots of the BIT pipeline robot moving inside of S-shaped testbed pipelines with diameter of 150 mm

Fig. 8. The acquired data of the BIT pipeline robot moving in testbed straight pipelines with diameter of 100 mm; a) speed of motors in the robot head; b) speed of motors in the robot tail; c) angle of motors in the head of robot; d) angles of motors in the robot tail

Design and Lab Experience for a Pipeline Service Robot

463

represent screw driving motor installed at head and tail of pipeline robot respectively and they are all position control model. The purple curve in figure c) and figure b) shows that the changing of relative position of ms1 and ms2 is similar and the position of screw installed at head and tail are approximately the same, that means the length of the support arm is approximately equal. We can get the same result from the rotating speed curves of screw. mp1 and mr1 represent pitch motion driving motor and roll driving motor respectively installed at head of pipeline robot. mp2 and mr2 represent pitch motion driving motor and roll driving motor respectively installed at tail of pipeline robot. mp1 and mp2 represent pitch motion driving motors installed at head and tail of pipeline robot respectively. They are not controlled, so pipeline robot is able to adopt the changes in the shape of the pipeline and the feedback data of this motor as figure show.

Fig. 9. he acquired data of the BIT pipeline robot moving in testbed straight pipelines with diameter of 180 mm; a) speed of motors in the robot head; b) speed of motors in the robot tail; c) angle of motors in the head of robot; d) angles of motors in the robot tail

464

T. Zheng et al.

Fig. 10. The acquired data of the BIT pipeline robot moving in testbed S-shaped pipelines with diameter of 150 mm; a) speed of motors in the robot head; b) speed of motors in the robot tail; c) angle of motors in the head of robot; d) angles of motors in the robot tail

6 Conclusions The BIT pipeline robot is presented with its features in monitoring applications of pipelines in space orbital stations with light-weight compact design and autonomous durable operation. Testing in a lab setup is reported to show the adaptation of a designed prototype in a pipeline network with different cross-sections and shapes. The design is characterized with operation performance that are discussed also from testing results as satisfactory for the servicing tasks in space orbital stations. Acknowledgments. This work was supported in part by the National Key Research and Development Program of China under Grant 2018YFB1305300 and in part by the National Natural Science Foundation of China under Grant 61733001, Grant 61873039, Grant U1713215, and U1913211.

References 1. Zhu, X., Liu, J., Sun, H.: Numerical simulation method of air distribution in cabin of space station. Space Med. Med. Eng. 5(28), 378–383 (2015)

Design and Lab Experience for a Pipeline Service Robot

465

2. Wu, H., Li, N., Liang, X.: Numerical study of the effects of size of inlets and outlets of ventilation system in experimental space station cabin. Space Med. Med. Eng. 14(4), 268–271 (2001) 3. Choi, H.R., Ryew, S.M.: Robotic system with active steering capability for internal inspection of urban gas pipelines. Mechatronics 12(5), 713–736 (2002) 4. Roh, S.G., Choi, H.R.: Differential-drive in-pipe robot for moving inside urban gas pipelines. IEEE Trans. Rob. 21(1), 1–17 (2005) 5. Roh, S.G., Kim, D.W., Lee, J.S., et al.: In-pipe robot based on selective drive mechanism. Int. J. Control Autom. Syst. 7(1), 105–112 (2009) 6. Park, J., Hyun, D., Cho, W.H., et al.: Normal-force control for an in-pipe robot according to the inclination of pipelines. IEEE Trans. Ind. Electron. 58(12), 5304–5310 (2011)

Nonlinear Dynamics of a Spatial Two Link Chain Dan B. Marghitu1 1

and Dorian Cojocaru2(B)

Auburn University, Auburn, AL 36849, USA [email protected] 2 University of Craiova, Craiova, Romania [email protected]

Abstract. A spatial two link chain with revolute joints is considered. The equations of motion for the system are developed using Kane’s dynamical equations. Phase plane and return maps of the kinematic chain for different initial condition are developed. The motion of the mechanical system in non-periodic.

Keywords: Kinematic chain

1

· Spatial motion · Return maps

Introduction

The kinematic and dynamic equations of motion for a spatial open kinematic chain are deduced using Kane’s method. The equations of motion are numerically solved with MATLAB functions. Lagrange and Euler used for the first time in analytical dynamics the concept of quasi-coordinates [1–3]. Huston and Passerello developed equations of motion for a model of the human body using the principles of classical mechanics [4] Tavakoli et al., proposed a method to derive the dynamic equations recursively using Kane’s equations for a snake-like robot [5]. The constrained and unconstrained multi-body systems dynamics using the Euler-Lagrange approach was studied in [6]. He derived the partial derivatives in closed form and the equations of motion resulting in Kane’s dynamical equations. Kane’s dynamical equations of motion were used to analyze a wheeled mobile manipulator subject to nonholonomic constraints [7]. A robotic R-R arm is simulated with Kane’s method and MATLAB [8]. The paper presented the advantages of the Kane’s dynamical equations and did not elaborate on the complexity of the nonlinear equations for different scenarios. For a spatial system with two degrees of freedom the use of Kane’s methods leads to a simpler form of the equations of motion. A tutorial introduction to Kane’s method and a detailed derivation of Kane’s equations with a geometric interpretation and examples with changing constraints are presented in [9]. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 466–475, 2020. https://doi.org/10.1007/978-3-030-48989-2_50

Nonlinear Dynamics of a Spatial Two Link Chain

467

A swinging babyboot with revolute joints was considered in [10–12]. The solution to the differential equations reveals that the mechanical model has a non-intuitive motion. For certain initial values the motion of is orderly. For different conditions a small variation in the initial condition lead to different results. The system is sensitive to initial conditions. In this paper we study a two link spatial mechanical system with non-periodic motions using phase plane and return maps. The system is analyzed with MATLAB and combines symbolical and numerical computations to find and solve the equations of motions.

2 2.1

Equations of Motion Kinematics

A spatial kinematic chain with revolute joints is shown in Fig. 1. The mass of the link 1 is m1 and the mass of the link 2 is m2 . The length of link 1 is OA = L1 and the length of link 2 is AB = L2 . The distance from O to C1 is OC1 = L1 /2 and the distance from A to C2 is AC2 = L2 /2. The mass centers of links 1 and 2 are C1 and C2 , respectively. The “fixed” reference frame 0 (RF0) has unit the vectors [ı0 , j0 , k0 ].

Fig. 1. Spatial kinematic chain with revolute joints.

468

D. B. Marghitu and D. Cojocaru

Figure 1 shows the rotating reference frame 1 (RF1) of unit vectors [ı1 , j1 , k1 ] that is attached to link 1 and the rotating reference frame 2 (RF2) of unit vectors [ı2 , j2 , k2 ] attached to link 2. Link 1 rotates about a vertical axis ı0 at the origin O. Link 1 is connected to link 2 with a revolute joint at point A. The link 2 rotates with respect to link 1 about an axis fixed in both 1 and 2, passing through A, and along the axis of link 1. Link 2 can rotate freely about the axis of link 1. Link 1 has a moment of inertia I1 about any line normal to 1 and passing through C1 and link 2 has a moment of inertia I2 about any line normal to 2 and passing through C2 . The simple rotation between the axes of RF1 and RF0 is defined by the first generalized coordinate q1 . The rotation from RF0 to RF1 is defined by ⎤⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ 1 0 0 ı0 ı0 ı1 ⎣ j1 ⎦ = ⎣ 0 c1 s1 ⎦ ⎣ j0 ⎦ = R10 ⎣ j0 ⎦ , (1) k1 0 −s1 c1 k0 k0 where s1 = sin q1 and c1 = cos q1 . The simple rotation between RF1 and RF2 is defined by the second generalized coordinate q2 and the unit vectors ı2 , j2 and k2 are expressed as ⎤⎡ ⎤ ⎡ ⎤ ⎡ ⎤ ⎡ ı1 c2 s2 0 ı1 ı2 ⎣ j2 ⎦ = ⎣ −s2 c2 0 ⎦ ⎣ j1 ⎦ = R21 ⎣ j1 ⎦ , (2) k2 k1 k1 0 0 1 where s2 = sin q2 and c2 = cos q2 . The angular velocity of link 1 in RF0 expressed in terms of RF1 is ω10 = q˙1 ı1 .

(3)

The angular velocity of the link 2 with respect to the link 1 expressed in terms of RF1 is ω21 = q˙2 k1 .

(4)

The angular velocity of the link 2 with respect to the fixed reference frame, RF0, expressed in terms of RF1 is ω20 = ω10 + ω21 = q˙1 ı1 + q˙2 k1 , = cos q2 q˙1 ı2 − sin q2 q˙1 ı2 + q˙2 k2 .

(5) (6) (7)

The angular acceleration of the link 1 in RF0 is (1) (1) d d d ω10 = ω10 + ω10 × ω10 = ω10 dt dt dt = q¨1 ı1 ,

α10 =

(8) (9)

Nonlinear Dynamics of a Spatial Two Link Chain

469

(1) d represents the derivative with respect to time in reference frame RF1. where dt The angular acceleration of the link 2 in RF0 is

α20 =

(2) (2) d d d ω20 = ω20 + ω20 × ω20 = ω20 , dt dt dt

(10)

(2) d represents the derivative with respect to time in reference frame RF2. where dt The position vector of C1 , the mass center of link 1 is

rC1 = −0.5 L1 k1 .

(11)

The position vector of the mass center of link 2 is rC2 = −L1 k1 − 0.5 L2 j2 = −0.5 L2 j2 − L1 k2 .

(12)

The velocity of C1 in RF0 is d rC1 + ω10 × rC1 . dt

(13)

d rC2 + ω20 × rC2 . dt

(14)

d vC1 + ω10 × vC1 . dt

(15)

d vC2 + ω20 × vC2 . dt

(16)

(1)

vC 1 = The velocity of C2 in RF0 is

(2)

vC 2 = The acceleration of C1 in RF0 is

(1)

aC 1 = The acceleration of C2 in RF0 is

(2)

aC 2 = 2.2

Generalized Active Forces

The external gravitational forces exerted on the links 1 and 2 are Gi = −mi g k0 ,

i = 1, 2.

(17)

The generalized active force Q1 of all the forces and torques acting on the links 1 and 2 are Q1 =

∂vC1 ∂vC2 · G1 + · G2 . ∂ q˙1 ∂ q˙1

(18)

Q2 =

∂vC1 ∂vC2 · G1 + · G2 . ∂ q˙2 ∂ q˙2

(19)

470

2.3

D. B. Marghitu and D. Cojocaru

Generalized Inertia Forces

The central principal axes of the slender rod 1 are parallel to ı1 , j1 , k1 . The central principal moments of inertia associated with the principal axes are IC1x = I1 , IC1y = I1 , IC1z = 0, respectively. The inertia matrix associated with the central inertia dyadic of rod 1 is ⎤ ⎤ ⎡ ⎡ I1 0 0 0 IC1x 0 I¯C1 → ⎣ 0 IC1y 0 ⎦ = ⎣ 0 I1 0 ⎦ . (20) 0 0 IC1z 0 0 0 The central principal axes of the slender rod 2 are parallel to ı2 , j2 , k2 . The central principal moments of inertia associated with the principal axes are IC2x = I2 , IC2y = 0, IC2z = I2 , respectively. The inertia matrix associated with the central inertia dyadic of rod 1 is ⎤ ⎡ ⎤ ⎡ 0 I2 0 0 IC2x 0 I¯C2 → ⎣ 0 IC2y 0 ⎦ = ⎣ 0 0 0 ⎦ . (21) 0 0 IC2z 0 0 I2 The inertia force of link j = 1, 2 is Fin j = −mj aCj ,

(22)

where mj is the mass of link j. The inertia moment of link 1 in RF0 is Min 1 = −α10 · I¯C1 − ω10 × (I¯C1 · ω10 ),

(23)

The inertia moment of rod 2 in RF0 is Min 2 = −α20 · I¯C2 − ω20 × (I¯C2 · ω20 ).

(24)

The generalized inertia forces are Kin r =

2   ∂vCj j=1

∂ur

· Fin j

 ∂ωj0 + · Min j , ∂ur

(25)

r = 1, 2, or Kin r =

∂vC1 · Fin 1 + ∂ur ∂vC2 · Fin 2 + ∂ur

∂ω10 · Min 1 + ∂ur ∂ω20 · Min 2 , ∂ur

(26)

where u1 and u2 are the generalized speeds defined by the kinematical equations of motion u1 = q˙1

and

u2 = q˙2 .

(27)

Nonlinear Dynamics of a Spatial Two Link Chain

471

The Kane’s differential dynamical equations for the kinematic chain are Kin r + Qr = 0, r = 1, 2.

(28)

The system of ordinary differential equations of motion, Eqs. (27), (28), can be solved with numerical techniques and the generalized coordinates qr and the generalized speeds ur are determined.

3

Numerical Application and Results

For the chain shown in Fig. 1 the mass of the link 1 is m1 = 1 kg and the mass of the link 2 is m2 = 0.02 kg. The length of slender link 1 is L1 = 0.2 m and the length of slender link 2 is L1 = 0.14. The gravitational acceleration is g = 9.81 m/s2 . The differential equations of motion were solve using MATLAB function ode45. The first simulations are conducted with the initial condition q1 (0) = 30◦ and q˙1 (0) = q˙2 (0) = 0 rad/s. For q2 (0) two close different values are chosen. Figure 2 shows the variation in time, t = 3 s, for the generalized coordinates q1 (t) and q2 (t). The initial conditions are q1 (0) = 30◦ and q2 (0) = 5◦ for the solid line plot and q1 (0) = 30◦ and q2 (0) = 5.5◦ for the dashed line style plot. The two plots are similar for different initial conditions for q2 (0). The angle q1 (t) experiences a regular behavior. The phase plane curve for q1 (t) and q˙1 (t) for the two different initial conditions q2 (0) = 5◦ and q2 (0) = 5.5◦ are represented in Fig. 3. From Fig. 3 the motion of q1 (t) is not periodic. Figures 4 and 5 shows the return map of the maxima of q1 (t) for the two different initial conditions q2 (0) = 5◦ and q2 (0) = 5.5◦ . The motion is not periodic and is not quasi-periodic.

Fig. 2. Generalized coordinates q1 and q2 for q1 (0) = 30◦ .

472

D. B. Marghitu and D. Cojocaru

Fig. 3. Phase plane q1 − q˙1 for q1 (0) = 30◦ .

Fig. 4. Return map, q2 (0) = 5◦ .

Fig. 5. Return map, q2 (0) = 5.5◦ .

The next simulations are conducted with the initial condition q1 (0) = 75◦ and q˙1 (0) = q˙2 (0) = 0 rad/s. For q2 (0) the same two close different values are chosen: q2 (0) = 5◦ and q2 (0) = 5.5◦ . Figure 6 shows the generalized coordinates q1 (t) and q2 (t) for q2 (0) = 5◦ (solid line) and q2 (0) = 5.5◦ (dashed line). For this case the values of q2 (t) at t = 3 s are very divergent. The phase plane curves for q1 (t) − q˙1 (t) and q2 (t) − q˙2 (t) for the two different initial conditions q2 (0) = 5◦ and q2 (0) = 5.5◦ are shown in Figs. 7 and 8. From Figs. 7 and 8 the motions of

Nonlinear Dynamics of a Spatial Two Link Chain

473

Fig. 6. Generalized coordinates q1 and q2 for q1 (0) = 75◦ .

Fig. 7. Phase planes for q2 (0) = 5◦ .

Fig. 8. Phase planes for q2 (0) = 5.5◦ .

q1 (t) and q2 (t) are not periodic. Figures 9, 10, 11 and 12 shows the return map of the maxima of q1 (t) and q2 (t) for the two different initial conditions q2 (0) = 5◦ and q2 (0) = 5.5◦ . The motions are not periodic and are not quasi-periodic.

474

D. B. Marghitu and D. Cojocaru

Fig. 9. Return map for q1 , q2 (0) = 5◦ .

Fig. 10. Return map for q1 , q2 (0) = 5.5◦ .

Fig. 11. Return map for q2 , q2 (0) = 5◦ .

Fig. 12. Return map for q2 , q2 (0) = 5.5◦ .

4

Conclusions

For the dynamics of a spatial kinematic chain, with two revolute joints, a theoretical model was developed. The chain is released from rest. The nonlinear equations of motion are numerical solve with MATLAB for two different initial conditions: q1 (0) = 30◦ and q1 (0) = 75◦ . The initial conditions for the second generalized coordinate are very close and q2 (0) = 5◦ , q2 (0) = 5.5◦ . From the phase plane plots and the return maps the motions are not periodic and are not quasi-periodic. Acknowledgment. The results presented on this paper were partially supported from the funds of complex research project Intelligent and distributed control of 3 complex autonomous systems integrated into emerging technologies for personal medical assistance and precise flexible manufacturing service (CIDSACTEH), financed by PN III scheme, program 1 Growth of national research and development system, sub-program 1.2 Institutional performance. The research project code is PN-III-P1-1.2-PCCDI-2017-

Nonlinear Dynamics of a Spatial Two Link Chain

475

0290, and the financing contract for execution in RDI Complex Projects is registered with no. 78PCCDI 2018 at UEFISCDI.

References 1. Whittaker, E.T.: Analytical Dynamics of Particles and Rigid Bodies. Dover Publications, Mineola (1944) 2. Meirovitch, L.: Methods of Analytical Dynamics. McGraw Hill, New York (1970) 3. Baruh, H.: Analytical Dynamics. WCB/McGraw Hill, New York (1999) 4. Huston, R.L., Passerello, C.E.: On the dynamics of a human body model. J. Biomech. 4(5), 369–378 (1971) 5. Tavakoli Nia, H., Pishkenari, H.N., Meghdari, A.: A recursive approach for the analysis of snake robots using Kane’s equations. Robotica 24, 251–256 (2006) 6. Parsa, K.: The Lagrangian derivation of Kane’s equations. Trans. CSME 31(4), 407–420 (2007) 7. Tanner, H.G., Kyriakopoulos, K.J.: Modeling of a mobile manipulator with the use of Kane’s dynamical equations. In: IFAC Intelligent Autonomous Vehicles, Madrid, Spain, pp. 611–617 (1998) 8. Purushotham, A., Anjeneyulu, M.J.: Kane’s method for robotic arm dynamics: a novel approach. J. Mech. Civil Eng. 6, 7–13 (2013) 9. Gillespie, R.B.: Kane’s equations for haptic display of multibody systems. Hapticse, Electron. J. Haptics Res. 2(3) (2003). http://www.haptics-e.org 10. Kane, T.R., Levinson, D.A.: Dynamics: Theory and Applications. McGraw-Hill, New York (1985) 11. Kane, T.R.: Mechanical demonstration of mathematical stability and instability. Int. J. Eng. Educ. 2(4), 45–47 (1974) 12. MotionGenesis Kane Tutorial (2016). www.MotionGenesis.com

Learning

Autonomous Navigation in Vineyards with Deep Learning at the Edge Diego Aghi1,2(B) , Vittorio Mazzia1,2,3 , and Marcello Chiaberge1,2 1

2

DET, Politecnico di Torino, Turin, Italy {diego.aghi,vittorio.mazzia,marcello.chiaberge}@polito.it PIC4SeR - Politecnico Interdepartmental Centre for Service Robotic, Turin, Italy 3 SmartData@PoliTo - Big Data and Data Science Laboratory, Turin, Italy https://pic4ser.polito.it/, https://smartdata.polito.it/

Abstract. With the rapid growth of the world population over the past years, the agriculture industry is asked to respond properly to the exponential augmentation of global demand for food production. In the past few years, autonomous agricultural field machines have been gaining significant attention from farmers and industries in order to reduce costs, human workload, and required resources. Nevertheless, achieving sufficient autonomous navigation capabilities requires the simultaneous cooperation of different processes; localization, mapping, and path planning are just some of the steps that aim at providing to the machine the right set of skills to operate in semi-structured and unstructured environments. In this context, the presented research exploits later advancement in deep learning and edge computing technologies to provide a robust and fully integrable local planner for autonomous navigation along vineyards rows. Moreover, the devised and tested platform necessitates only of low range and low-cost hardware with minimal power and bandwidth requirements. The machine learning algorithm has been trained and tested with acquired images during different field surveys in the north region of Italy. Then, after performing an optimization process, the overall system has been validated with a customized robot platform in the appropriate environment.

1

Introduction

Nowadays, with the continuous growth of the human population, agriculture industries and farmers have been facing the exponential augmentation of global demand of food production. According to the projections of growth established in 2017 by the United Nations [1], by 2050, the global population will be around 9.7 billion and it is expected to reach 11.1 billion in 2100. So, there is an incremental need of new techniques and technologies aimed at maximizing efficiency and productivity of every single land sustainably. Over the years, precision agriculture [2] and digital farming [3] have gradually contributed with autonomous robotic machines and information collection c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 479–486, 2020. https://doi.org/10.1007/978-3-030-48989-2_51

480

D. Aghi et al.

to improving crops yield and resources management. Research on applications of mobile robotic systems in agricultural tasks has been increasing vastly [4]. However, despite the rising in investments and research activities on the subject, many implementations remain experimental and far from being applied on a large scale. Indeed, most of the proposed solutions require a combination of real-time kinematic GPS (RTK-GPS) [5,6] and costly sensors like threedimensional multi-channel Light Detection and Ranging (LIDAR) [7]. Those solutions, besides being very expensive, are unreliable and prone to failure and malfunction due to their complexity. In [8], Riggio et al. proposed a low-cost solution based only on a single-channel LIDAR and odometry, but it is greatly affected by the type of canopy and condition of the specific vineyard. Instead, in [4], they proposed a vision based-control system using a clustering algorithm and Hough Transform in order to detect the central path between two rows. However, it is extremely sensitive to illumination conditions and intra-class variations. On the other hand, several recent works [9,10] focus their efforts on finding an affordable solution for the generation of a global map with related way-points. However, path following inside vineyard rows is still a challenging task due to localization problems and variability of the environment. Indeed, GPS receivers require to function in an open area with a clear view of the sky [11] so, expensive sensors and solutions are needed in order to navigate through vineyards rows. In this context, we present a low-cost solution for an autonomous navigation in vineyards rows using only a camera sensor and an edge computing platform. We have exploited latest advancement in deep learning and model optimization techniques to obtain a robust and power-efficient local planner to effectively navigate between vineyards canopies with only the raw RGB images coming from the front view of the robot. The remainder of this paper is organized as follows. Section 2 introduces the materials and data used for this research. Section 3 and 4 give a detailed overview of the proposed methodology with the obtained experimental results followed by the conclusion and future works.

2

Materials and Data

In order to acquire a dataset for training and testing the network, we performed field surveys in two distinct rural areas in the north part of Italy: Grugliasco near the metropolitan city of Turin in the Italian region of Piedmont and Valle San Giorgio di Baone in the Province of Padua in the Italian region Veneto. The collected video samples present different types of terrains, wine quality and were acquired at different time of the day, with diverse meteorological conditions. Videos were shot at 1080p with a 16:9 ratio in order to have more flexibility during the data processing process. On the other hand, as edge AI embedded computational platform, we selected NVIDIA Jetson Nano; it has been presented in June 2019 and targets applications where reducing the board size, power consumption, and price is important. For the hardware acceleration, it features a NVIDIA Maxwell GPU with peak performance of 472 GFLOPs and it can work in two power modes: 5 W or 10 W.

Autonomous Navigation in Vineyards with Deep Learning at the Edge

3

481

Proposed Methodology

Our goal is to develop a real-time local motion planner with an ultra-light computational load able to overcome the problems faced by the GPS device when carrying out an autonomous navigation along a vineyard row. Greatly inspired by Giusti et al. [12], we applied transfer learning [13] to train, with a custom dataset, a convolutional neural network (CNN) to classify the view of the front camera of the field machine in three distinct classes: left, center and right. Where, in a vineyard scenario, the class center describes the view of the camera when the vehicle is pointing at the end of the vineyard row, whereas the classes left and right are needed to indicate whether the vehicle is pointing at the left side or at the right side of the vineyard row respectively. Successively, using the trained network predictions, we design a simple control system to route the path of the robot through vineyards rows. Moreover, we exploited latest advancement in model optimization techniques in order to obtain an efficient and lightweight network able to inference in real-time on a low-cost edge AI platform. Finally, we integrated the deep learning model and the derived controller with Robot Operating System (ROS) framework. We evaluated the overall system in a vineyard near the city of Turin using a mobile platform with an ultrasound sensor to avoid possible collisions, a 720p USB camera and the Jetson Nano board to perform all required computations. The resulting system is a low-cost, powerefficient, and connection free local path planner that can be easily integrated with a global system achieving fully autonomous navigation in vineyards. 3.1

System Workflow

The system workflow can be divided into four main phases as shown in Fig. 1. Initially, a camera acquires images from the environment in front of the robot. Successively, those images are processed by the designed and optimized CNN that performs a multi-class classification locally and in real-time. After that, the control values are computed according to the class in which the classified image belongs to in order to properly drive the robot along vineyard rows. Finally, these values are sent to the robot platform using ROS that manages the signal communication to the actuators to perform the required motion. 3.2

Network Architecture

We have carefully selected a deep learning known architecture that reaches high performance by also containing hardware costs. MobileNet [14] network, due to its efficient design, works reasonably fast on mobile devices and embedded systems without too much memory allocation. This choice allows us to meet our requirements of low weight and computational load and, therefore, to have a wider range of possible hardware to be employed.

482

D. Aghi et al.

Fig. 1. An overview of the workflow of the presented system. Principal steps are reported in order with explanations of their role inside the global algorithm.

We have modified the original final fully connected layers of the MobileNet by substituting them with two fully connected layers of 256 and 3 neurons, respectively. The resulting model is a CNN network with an overall depth of 90 layers and with just 3.492.035 parameters. Moreover, we optimized the network model and we sped up the inference procedure by using TensorRT.

4 4.1

Experimental Results and Discussion Dataset Creation and Pre-processing

We used a dataset of 33.616 images equally balanced along with the three previously introduced classes. To create the dataset, we took several videos in a variety of vineyards rows with a 1080p resolution camera in order to have more flexibility during the pre-processing phase. In particular, for the first video of the center class, we recorded rows with the camera pointing at its center. Whereas, for the other two videos, classes left and right, we registered with the camera rotated of 45◦ with respect to the longitudinal axis of the row towards the left and the right side, respectively. Subsequently, we took each video as a streaming of images and we selected the best frame every six consecutive ones. Additionally, before being used to train the network, the images have been normalized and resized to the expected dimensions of the input images of the network.

Autonomous Navigation in Vineyards with Deep Learning at the Edge

483

Fig. 2. Three samples of the dataset used to train the network, one for each class. Figure 2(a) is an example of the left class, Fig. 2(b) of the center class, and Fig. 2(c) of the right class. Dataset samples have been collected with different weather conditions and at a different time of day. The resulting heterogeneous training set is aimed at giving generality and robustness to the model.

4.2

Training

As already introduced, we trained the network using a technique known as transfer learning [13]; instead of starting to train with weights randomly initialized, we used variables obtained with an earlier training session. In particular, we exploited weights obtained fitting MobileNet with the ImageNet classification dataset. Subsequently, we removed the last two fully connected layers and replaced them with a number of neurons equal to our number of classes. Using this technique, we were able to take advantage of previous low-level features, learned by the network, greatly reducing the number of images and epochs required for the training. Indeed, edges, contours, and basic textures are generalpurpose features that can be reused for different tasks. In order to properly train, validate and test the model, we randomly divided the dataset into three subsets as follow: 70% for the training set, 15% for the validation set, and the remaining 15% for the test set. We trained the resulting network for only six epochs with a batch size of 64. To increment the robustness of the network and to overcome possible problems of overfitting we used different techniques such as dropout, weight decay and data augmentation with changes in zoom and brightness. Finally, we used RMSprop as an optimizer, accuracy as a metric, and cross entropy as a loss function. 4.3

Model Results

The implemented model has been trained and tested with the subdivision of the dataset introduced in Sect. 4.2, giving an accuracy of 1.0 over the test set. Therefore, this model is the one employed for the navigation. Moreover, in order to evaluate the robustness of the network over new scenarios, we performed an experimentation training the model only with a small part of the available dataset. So, we trained the architecture with just a vineyard type and tested the resulting model with five completely different scenarios with diverse wine quality and weather conditions. In particular, in Table 1, it

484

D. Aghi et al.

is depicted how the dataset has been partitioned for training, validation, and testing for this second experimentation. The resulting split percentages (18, 8, 74) are due to the amount of images available for each region of the available dataset. Table 1. Dataset division for the second evaluation of the model. Subset

% Number of items

Training set

18

6.068

8

2.681

Validation set Test set

74 24.867

As shown in Table 2, an accuracy of 0.94 is achieved by the re-trained model in this second case. This is an optimal result considering the fact that the network has been trained with a very small dataset and it has been tested with a completely different vineyard scenario. This clearly demonstrates how transfer learning, for this specific task, is very effective at providing good generalization capabilities with also a small training set. Table 2. Results of the second evaluation of the proposed model.

4.4

Class right left center

precision 0.850 1.000 1.000

micro avg macro avg weighted avg

0.941 0.950 0.950

recall f1-score 1.000 0.919 0.899 0.947 0.924 0.961 0.941 0.941 0.941

0.941 0.942 0.942

Optimization and Deployment

As previously introduced, the employed network has been optimized using tensorRT. It allows us to tune the portion of available GPU memory dedicated to TensorFlow with a parameter called per process gpu memory fraction (PPGMF). In particular, after many attempts, in order to find the best configuration, we set the PPGMF to 0.75. TensorRT, besides not affecting the accuracy of the prediction, it gives a significant increment to the number of frames elaborated per second by our model. In fact, the control frequency using Tensorflow with a frozen graph was 21.92 Hz, whereas, with the performed optimization, we reach 47.15 Hz.

Autonomous Navigation in Vineyards with Deep Learning at the Edge

485

Fig. 3. Test on the relevant environment with the Clearpath Robotics, Jackal model platform.

As far as the deployment is concerned, the system has been implemented in a ROS-oriented robot platform. The robot in which the local planner has been tested is an unmanned ground vehicle: the model Jackal from Clearpath Robotics (Fig. 3). The tests have been carried out in a new vineyard scenario, and the system proved to be able to perform an autonomous navigation along the vineyards row autonomously even with a low-resolution camera.

5

Conclusion and Future Works

We proposed a local motion planner for autonomous navigation along vineyards rows. We exploited latest advancement in deep learning optimization techniques in order to obtain a lightweight, power-efficient model able to run locally on a low-cost edge AI platform. The proposed system provides a considerably high control frequency that meets the requirements of a real-time autonomous navigation task, the predictions of the network are robust to changes in brightness, in vineyards dimensions and, eventually, to missing grapevines along the vineyard rows. Finally, the local planner that we proposed demonstrated to work exceptionally well in real conditions even if the images given as input have a very low resolution. However, as future works, an additional system with a proportional controller may be implemented in order to improve the controllability of the robot during the navigation. Acknowledgments. This work has been developed with the contribution of the Politecnico di Torino Interdepartmental Centre for Service Robotics PIC4SeR (https:// pic4ser.polito.it) and SmartData@Polito (https://smartdata.polito.it).

References 1. UN DESA: World population prospects, the 2017 revision, volume I: comprehensive tables. United Nations Department of Economic & Social Affairs, New York (2017)

486

D. Aghi et al.

2. Mulla, D.J.: Twenty five years of remote sensing in precision agriculture: key advances and remaining knowledge gaps. Biosyst. Eng. 114(4), 358–371 (2013) 3. Shamshiri, R.R., et al.: Research and development in agricultural robotics: a perspective of digital farming. Chinese Society of Agricultural Engineering 4. Sharifi, M., Chen, X.Q.: A novel vision based row guidance approach for navigation of agricultural mobile robots in orchards. In: 2015 6th International Conference on Automation, Robotics and Applications (ICARA), pp. 251–255 (2015) 5. Guzm´ an, R., et al.: Autonomous hybrid GPS/reactive navigation of an unmanned ground vehicle for precision viticulture-VINBOT. In: 62nd German Winegrowers Conference, Stuttgart (2016) 6. Santos, D., Neves, F., et al.: Towards a reliable robot for steep slope vineyards monitoring. J. Intell. Robot. Syst. 83(3–4), 429–444 (2016) 7. Astolfi, P., et al.: Vineyard autonomous navigation in the Echord++ GRAPE experiment. IFAC-PapersOnLine 51(11), 704–709 (2018) 8. Riggio, G., Fantuzzi, C., Secchi, C.: A low-cost navigation strategy for yield estimation in vineyards. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 2200–2205 (2018) 9. Santos, L., et al.: Path planning approach with the extraction of topological maps from occupancy grid maps in steep slope vineyards. In: 2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 1–7 (2019) 10. Zoto, J., et al.: Automatic path planning for unmanned ground vehicle using UAV imagery. In: International Conference on Robotics in Alpe-Adria Danube Region, pp. 223–230 (2019) 11. Ma, C., et al.: GPS signal degradation modeling. In: Proceedings of International Technical Meeting of the Satellite Division of the Institute of Navigation (2001) 12. Giusti, A., et al.: A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robot. Autom. Lett. 1(2), 661–667 (2015) 13. Long, M., et al.: Deep transfer learning with joint adaptation networks. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 2208–2217 (2017) 14. Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. Preprint at https://arxiv.org/abs/1704.04861 (2017)

Iterative Learning Control of Hard Constrained Robotic Manipulators Kaloyan Yovchev(&), Lyubomira Miteva, Kamen Delchev, and Evgeniy Krastev Faculty of Mathematics and Informatics, Sofia University, Sofia, Bulgaria {k.yovchev,kkdelchev,eck}@fmi.uni-sofia.bg, [email protected]

Abstract. This paper considers the hard constraints in the movement of the joints of robotic manipulators executing repetitive tasks in the presence of measurement noise in the dynamic model. In our research work we have established that subject to these conditions the Iterative Learning Control (ILC) is one of the best methods performing such tasks with high precision. The existing applications of ILC don’t take in consideration the inherent nonlinearity of the dynamic model of a robotic manipulator neither the hard constraints in state space. This paper presents a robust and convergent Constrained Output ILC (COILC) method for non-linear hard constrained systems like industrial robots. Unlike known ILC methods the COILC employs a novel algorithm to cancel the currently executing iteration before the occurrence of a violation in any of the state space hard constraints. This way COILC resolves both the hard constraints in the state space and the transient growth problem, which is a major obstacle in applying ILC to non-linear systems. The convergence and the performance of the proposed numerical procedure are experimentally evaluated through computer simulation of the well-known SCARA-type robot TT-3000. The obtained results are discussed and demonstrate that COILC is suitable for solving hard constrained state space problems of non-linear systems in robotics with certain inaccuracies in the dynamic model. Keywords: Hard constrained systems manipulators
Computer simulation


Iterative Learning Control
Robotic

1 Introduction The basic idea of the Iterative Learning Control (ILC) is to compensate the tracking error for systems that have to perform a repetitive job with high precision. This is done by tracking multiple consequent iterations of the job execution, and between each one the input signals are corrected on the basis of the already collected data in order to reduce the tracking error. Thus, in a natural iterative process of self-learning, the input signals for achieving the highest precision are computed. ILC is introduced by Uchiyama in 1978 [1] and six years later this method began to be widely employed for high precision tracking control of systems that execute tasks in repetitive mode. The publications of Arimoto et al. [2, 3] have major contribution to © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 487–495, 2020. https://doi.org/10.1007/978-3-030-48989-2_52

488

K. Yovchev et al.

ILC research and applications. According to Arimoto the ILC method executes a sequence of iterations of the same assignment, where each iteration is used to learn and correct errors in the mathematical model as well as in the determined system disturbances. This ILC study is mainly related to its applications in the field of robotics. The convergence analysis is important for the synthesis of an ILC method. For nonlinear systems the robustness and convergence are proven in 1989 by Heizinger et al. [4]. One of the main problems of the ILC is the growth of the transient error (Fig. 1). The essence of the problem consists in the possibility of several iterations, in which the error increases many times before it starts to decrease again and to converge to zero. This problem is addressed by Longman, R.W. and Huang, Y. in [5]. A rather limited number of solutions to the problem of transient error is known in the scientific literature concerning ILC - the slow learning rate ILC [6], the monotone ILC [7], or the Bounded-Error Algorithm (BEA) [8, 9]. In research [10] a variation of ILC is used for learning the task-specific torques, that preserves a given criterion. This variation aborts any iteration before violation of the selected criterion. Other major disadvantage of the ILC methods is that they cannot be applied directly to systems in which there are constraints, e.g. constraints of input (control) signals, state space constraints or velocity constraints [11, 12]. Most studies mainly address restrictions on input signals or speeds [12–14]. The convergence of these methods has not been proven when used to solve the problem with hard constraints in the state space coordinates [15]. The robotic manipulators have constant mechanical design constraints due to the bearing’s construction, the shape of the links and the permissible angle of rotation [16]. These constraints make ILC methods unusable for manipulation robots. For example, in cases where the trajectory is planned close to the constraints of the generalized coordinates (Fig. 1). The transient error and the hard constraints are serious problems when ILC methods are used in real-world conditions, because such a deviation is beyond the real robot’s running capabilities without disturbing the hard constraints of the joint angles.

Fig. 1. Representation of the transient error growth problem when ILC method is applied to hard constrained systems.

Iterative Learning Control of Hard Constrained Robotic Manipulators

489

The main objective of this paper is to propose a new general solution for applying ILC method within a constrained state space. This new solution will modify the learning update law in order to allow the executed output trajectory to be into the whole area defined by the state space constraints. This paper is organized as follows. Section 2 formulates the problem of applying ILC to hard constrained robotic manipulator. Section 3 proposes as a solution a new general algorithm. Section 4 validates the proposed algorithm through a computer simulation.

2 Problem Formulation Let us consider the well-studied SEIKO Instruments SCARA-type robot TT-3000. The kinematics of this robot is shown in Fig. 2.

Fig. 2. Kinematic of SEIKO Instruments TT-3000 robot.

The Lagrange’s formulation of dynamical equations of motion of this robot is as follows [17]: a11 ðq2 Þ€q1 þ a12 ðq2 Þ€q2  2bsinq2 q_ 1 q_ 2  bsinq2 q_ 22 þ d1 q_ 1 þ fc1 ¼ u1 ; a12 ðq2 Þ€q1 þ a11 ðq2 Þ€q2  2bsinq2 q_ 21 þ d2 q_ 2 þ fc2 ¼ u2 ;

ð1Þ

a11 ðq2 Þ ¼ a þ 2bcosq2 a11 ðq2 Þ ¼ v þ bcosq2 a22 ¼ v
a ¼ I1 þ m1 s21 þ I2 þ m2 I12 þ s22 b ¼ m2 l1 s2 v ¼ I2 þ m2 s22 ;

ð2Þ

where:

and di ; i ¼ 1; 2 is viscous damping, fci is Coulomb’s friction, li is the specified length of the link of the robot, si is the position of the center of the mass, mi is the total mass of the link, and Ii is the inertia of the link about its center of mass. The standard ILC control scheme is shown in Fig. 3, where P represents the robot arm; L is the feed-forward controller; M is the memory of the control system; the input trajectory ul is the feed-forward term of the control law u in (3), and l ¼ 1; . . .; N is the

490

K. Yovchev et al.

current iteration number; N is the total number of iterations; ql is the actual output trajectory; and qd is the desired output trajectory. The offline computed feed-forward term ul þ 1 decreases the tracking error of the robot’s motion on the next iteration.

Fig. 3. Iterative Learning Control scheme.

The learning update law of the standard ILC scheme can be synthetized as follows of the considered ILC scheme requires the specification of feed-forward and feedback controllers, respectively. The feed-forward controller is based on the update control law that improves the feed-forward control term: ul þ 1 ¼ ul þ Lðql Þð€qd  € ql Þ

ð3Þ

where Lðql ðtÞÞ; l ¼ 0; 1; . . .; N is a learning operator; u0 ðtÞ  0 is the initial feedforward control input; t 2 ½0; T  denotes time, where ½0; T  is the robot tracking time interval. ^ ðqÞ is The robustness and convergence of the ILC update law (3) if LðqÞ  A ^ proven in [18], where A is the corresponding estimate of the  inertia matrix A. For the considered TT-3000 robot with Eqs. (1) and (2): A ¼ aij ; i; j ¼ 1; 2. However, this type of ILC does not take into consideration the joint constraints of the robotic manipulator. Now, let’s consider system with the following output state space constraints for yð t Þ 2 R m :   max ; i ¼ 1; 2; . . .; m qðtÞ ¼ ðq1 ðtÞ; q2 ðtÞ; . . .; qm ðtÞÞ : qi ðtÞ 2 Qmin i ; Qi

ð4Þ

For the TT-3000 robot m ¼ 2. Let us consider the transient growth error problem in ILC. In the computer simulations of ILC it is common for the learning process to start with a fast increase of the trajectory error before it converges to a small value. It will actually stop the learning process because the robot will not be able to execute the desired trajectory due to its hard constraints (4). In the following section we present a novel ILC algorithm for hard constrained robotic manipulators. The output trajectories will be bounded by those state space constraints.

Iterative Learning Control of Hard Constrained Robotic Manipulators

491

3 Iterative Learning Control of Hard Constrained Robotic Manipulators The idea of this new algorithm is to constrain the output trajectory at each iteration l, so that the state space constraints cannot be violated during the ILC procedure. Let’s consider the robotic manipulator described by (1) and constrained by (4). Then, the following subset of the attainable desired output trajectories defined by the ILC tracking accuracy l:
Qd ¼ fqd ðtÞ : qd ðtÞ 2 Rm ; qd ðtÞ ¼ qd1 ðtÞ; qd2 ðtÞ; . . .; qdm ðtÞ ; qdi ðtÞ 2 ðQmin þ l; Qmax  lÞ; t 2 ½0; T ; i ¼ 1; 2; . . .; mg: i i

ð5Þ

Constrained Output ILC (COILC) Algorithm For the robotic system defined in (1) with the hard constraints defined in (4) we can apply the following ILC procedure for any attainable desired trajectory qd 2 Qd in (5): a. Set the initial iteration number l ¼ 0 and begin the iterative procedure. b. Starting from the initial position ql ð0Þ the system is tracking the desired trajectory under the control ul ðtÞ until the first moment Sl for which there exists or qlj ðSl Þ ¼ Qmax or the end position ql ðT Þ j : 1  j  m and either qlj ðSl Þ ¼ Qmin j j is reached. When t ¼ Sl ; Sl 2 ð0; T , the tracking process stops. c. After the current tracking performance has finished, the learning controller updates the feed-forward control term according to the following learning update law: 

ul þ 1 ðt; Sl Þ ¼ ul ðt; Sl1 Þ þ ul ðt; Sl Þ; u0 ðt; S1 Þ u0 ðtÞ; Lðql ðtÞ; tÞðq_ d ðtÞ  ^ql ðtÞÞ; t 2 ½0; Sl ; Sl 2 ð0; T ;  ul ðt; Sl Þ ¼ 0; 8t 2 ðSl ; T 

ð6Þ

d. If the output error dql ðtÞ1 is less than or equal to an acceptable tracking accuracy, then exit from the learning procedure, else set l ¼ l þ 1 and go to step (b). From [19] it can be proven that the COILC algorithm is convergent and (6) can be applied for constrained output systems. Furthermore, for such systems COILC also solves the transient growth error problem without requiring knowledge for additional parameters other than the constraints of the system. COILC is supposed to also have a better convergence rate. The actual performance of the proposed COILC algorithm is evaluated in the next section through a computer simulation.

4 Simulation Results Computer simulation of TT-3000 robot is used for validation of the proposed in the previous section algorithm for ILC application on hard-constrained robotic manipulator (COILC). For proper simulation two different sets of the parameters from Eqs. (1) and (2) are used. Those sets are presented and compared in Table 1. These sets are

492

K. Yovchev et al.

estimated in [17] through identification methods. The standard set is used for the simulated robotic arm and the other set of experimental estimations are used for cal^ ðqÞ. culating the learning operator of the ILC procedure LðqÞ  A

Table 1. TT-3000 parameters for the computer simulation. a b 6596.2 1505.6 ^ Experimental estimations [kg.cm2] ^a b 6555.3 1471.8 Standard parameters [kg.cm2]

v d1 1353.4 17.29 ^v d^1 1847.8 17.29

d2 fc1 12.49 6.46 ^fc1 d^2 12.49 8.13

fc2 3.44 ^fc2 5.51

The computer simulation of the algorithm is done on MATLAB. For this computer simulation the desired trajectory of the robotic manipulator is as defined in Eq. (7) and the hard constraints of the robot’s joints are as defined in Table 2. The accuracy l of the ILC method is set to 0.02 rad. In this experiment we are applying COILC algorithm with learning update law (6) with 25 iterations. qd;1 ðtÞ ¼ 0:25 cosð2tÞ qd;2 ðtÞ ¼ 0:2sinð2tÞ; t 2 ½0; 1:1p

ð7Þ

Figure 4 shows the recorded trajectory of both links of the simulated SCARA-type robot during different iterations. As shown iterations 1 and 2 are stopped before violation of the hard constraints of both first and second joints.

Table 2. TT-3000 hard constraints for the computer simulation. Qmax Qmax Joint constraint [rad] Qmin Qmin 1 2 1 2 –0.57 0.57 –0.37 0.37

Fig. 4. Recorded trajectories of selected iterations of a. first link, b. second link

Iterative Learning Control of Hard Constrained Robotic Manipulators

493

Figure 5.a shows the norm of the maximum tracking error of all iterations. The error is limited due to the existence of hard joint constraints. This solves the transient error growth problem. Figure 5.b shows the duration of the different iterations. The first three iterations are shorter. However, the ILC is still convergent process. The correct execution of the desired trajectory completed in 9 iterations. Several executed trajectories and the link limits of both links of the robotic manipulator are shown in Fig. 6. The experiment shows that the COILC approach prevents violation of the constraints of the robotic manipulator and allows application of ILC method within constrained state space.

Fig. 5. a. Maximum tracking error of the ILC iterations, b. Time duration of each iteration.

Fig. 6. COILC approach for hard constrained robotic manipulator.

494

K. Yovchev et al.

5 Conclusion This paper presents a novel and computationally efficient general solution of the problem of using ILC method for hard constrained robotic manipulators. The proposed Constrained Output ILC algorithm (COILC) takes into account the joint constraints of the system and successfully overcomes the major deficiency of the ILC methods when they are used within the control of constrained systems. This method also solves the transient growth problem. The COILC uses modified learning update law and introduces a special stop condition. This results in a computationally effective method. The COILC is validated through a computer simulation and implicates that the COILC is expected to be of a considerable interest for robotic applications. Acknowledgement. This research is supported by the Fund for Scientific Research at Sofia University “St. Kliment Ohridski” under grant 80-10-7/2019 and 80-10-23/18.03.2020.

References 1. Uchiyama, M.: Formation of high-speed motion pattern of a mechanical arm by trial. Trans. SICE (Soc. Instr. Control Eng.) 14(6), 706–712 (1978) 2. Arimoto, S., Kawamura, S., Miyazaki, F.: Iterative learning control for robot systems. In: Proceedings of IECON, Tokyo, Japan (1984) 3. Arimoto, S., Kawamura, S., Miyazaki, F.: Bettering operation of robots by learning. J. Robot. Syst. 1(2), 123–140 (1984) 4. Heizinger, D., Fenwick, B., Paden, B., Miyazaki, F.: Robust learning control. In: Proceedings of 28th Conference on Decision and Control, Tampa, FL (1989) 5. Longman, R., Huang, Y.: The phenomenon of apparent convergence followed by divergence in learning and repetitive control. Intell. Autom. Soft Comput. 8(2), 107–128 (2002) 6. Bristow, D.A., Singler, J.R.: Robustness analysis of slow learning in iterative learning control systems. In: 2011 American Control Conference on O’Farrell Street, San Francisco, CA, USA, 29 June–01 July 2011 7. Park, K.-H., Bien, Z.: A study on iterative learning control with adjustment of learning interval for monotone convergence in the sense of sup-norm. Asian J. Control 4(1), 111–118 (2002) 8. Delchev, K.: Iterative learning control for nonlinear systems: a bounded-error algorithm. Asian J. Control 15(3), 1–8 (2013) 9. Yovchev, K., Delchev, K., Krastev, E.: Computer simulation of bounded error algorithm for iterative learning control. In: Advances in Intelligent Systems and Computing (Proceedings of RAAD 2016), vol. 3, pp. 136–143. Springer (2017) 10. Petrič, T., Gams, A.: Task space torque profile adaptations for dynamical human-robot motion transfer. In: International Conference on Robotics in Alpe-Adria Danube Region, pp. 44–52. Springer, Cham (2018) 11. Mishra, S., Topcu, U., Tomizuka, M.: Optimization-based constrained iterative learning control. IEEE Trans. Control Syst. Technol. 19(6), 1613–1621 (2011) 12. Chu, B., Owens, D.: Iterative learning control for constrained linear systems using projection method. In: International Workshop on Human Adaptive Mechatronics, Loughborough, GB, 13–14 May 2010

Iterative Learning Control of Hard Constrained Robotic Manipulators

495

13. Tan, Y., Xu, J.X.: On iterative learning control for nonlinear time-varying systems with input saturation. In: Symposium on Learning Control at IEEE CDC 2009, Shanghai (2009) 14. Schöllig, A., D’Andrea, R.: Optimization-based iterative learning control for trajectory tracking. In: European Control Conference, Budapest, Hungary, pp. 1505–1510 (2009) 15. Oriolo, G., Panzieri, S., Ulivi, G.: An iterative learning controller for nonholonomic mobile robots. Int. J. Robot. Res. 17(9), 954–970 (1998) 16. Chavdarov, I., Nikolov, V., Naydenov, B., Boiadjiev, G.: Design and control of an educational redundant 3D printed robot. In: 2019 International Conference on Software, Telecommunications and Computer Networks (SoftCOM), Split, Croatia, pp. 1–6 (2019) 17. Shinji, W., Mita, T.: A parameter identification method of horizontal robot arms. Adv. Robot. 4(4), 337–352 (1990) 18. Delchev, K.: Simulation-based design of monotonically convergent iterative learning control for nonlinear systems. Arch. Control Sci. 22(4), 371–384 (2012) 19. Yovchev, K., Delchev, K., Krastev, E.: State space constrained iterative learning control for robotic manipulators. Asian J. Control 20(3), 1145–1150 (2018)

Generalization Based Database Acquisition for Robot Learning in Reduced Space Zvezdan Lonˇcarevi´c(B) , Rok Pahiˇc, Mihael Simoniˇc, Aleˇs Ude, and Andrej Gams Joˇzef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenia [email protected]

Abstract. In order to increase the autonomy of the modern, high complexity robots with multiple degrees of freedom, it is necessary for them to be able to learn and adapt their skills, for example, using reinforcement learning (RL). However, RL performance greatly depends on the task dimensionality. Methods for reducing the task dimensionality, such as deep autoencoder neural networks, are often employed. Such neural network based dimensionality reduction approaches require a large example database for training, but obtaining such a database for a real robot is a complex and tedious process. This paper proposes a method of obtaining a database for the training of a deep autoencoder network, which serves for the dimensionality reduction of robot learning, and thus accelerates the robot’s ability to adapt to the real world. The presented method is based on a few real-world examples and statistical generalization. A comparison to using a simulated-only database on the use-case of robot throwing shows that the proposed approach achieves better realworld performance. Keywords: Deep autoencoder · Reinforcement learning database · Generalization · Robotic throwing

1

· Obtaining

Introduction and Related Work

Humanoid robots require continuous learning and adaptation of their actions and skills [1] in order to become part of people’s everyday lives. Most popular ways of obtaining new skills in nowdays robotics are Learning by Demonstration (LbD) and Reinforcement Learning (RL). Motions obtained by LbD often need to be improved or adapted to new situations. RL offers a framework and a set of tools for the design of sophisticated and hard-to-engineer behaviors [2]. Learning actions from scratch using RL only is usually not possible because the search space is too large. That is why LbD and RL approaches are combined so that the human teacher demonstrates the task to the robot and provides initial parameters before RL algorithms are used to improve or adapt performance. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 496–504, 2020. https://doi.org/10.1007/978-3-030-48989-2_53

Database Acquisition

497

In RL, different variations of the action are executed by varying the parameters (search space) describing the learned knowledge. At the end of each iteration, the quality of an action is measured by reward. When enough variations of the task are performed (exploration), this newly obtained knowledge is exploited in order for the agent to maximize the reward. In an episodic task, such as throwing, only the terminal reward is available. Estimation-maximization algorithms, such as Policy Learning by Weighting Exploration With the Returns (PoWER) can be used in such cases. Combining PoWER with movement primitives (MPs) was already successfully demonstrated, for example, for the ball-in-a-cup game [3] and pancake flipping [4]. Trajectory encoding methods based on the movement primitives (MPs) are in general suitable for learning purposes, because they describe actions with a reduced number of parameters. Most widely spread MPs are Dynamical Movement Primitives (DMPs [5,6]), Compliant Movement Primitives (CMPs [7]) and Probabilistic Movement Primitives (ProMPs [8]). To even more reduce the number of parameters, dimensionality reduction techniques, such as principal component analysis [9], or autoencoders (AE) are often employed. Deep autoencoders are special types of neural networks that are often used to train movements in a low dimensional latent space [10]. In our previous work, we ran RL algorithms in the latent space of autoencoder networks that autoencode DMPs [11]. For the autoencoder parameters to be learned properly, a big database of trajectories (actions) is needed. In this paper we investigate and compare methods of obtaining the database for training of deep autoencoder networks for dimensionality reduction. We compare two methods: one, where are the data for deep AE training is obtained in simulation, and one where the data for deep AE training is obtained through statistically generalizing a small set of real-world example trajectories. We performed our comparison on the use-case of robotic throwing since it is nonlinear task with only terminal reward available and the success of the action can be easily defined. We used DMPs for encoding the trajectories because they are deterministic movement primitives, meaning that only one demonstration is sufficient, and they can be easily temporally and spatially rescaled. Our proposed method of generalizing a small set of trajectories is based on the already shown approach that goals and other characteristics of the action can be used as a queries for creating a control policy for the previously unknown movements on the set of already demonstrated movements [6]. As the quality of the initial movement heavily influences the number of required iterations for RL algorithm, generalization also helped us to choose better initial movement and reduce the time needed for successfully learning the action.

2

Learning Space and Learning

We encoded all the trajectories into Cartesian Dynamic Movement Primitives (CDMPs) and from this representation into the latent space of a deep autoencoder network. Latent space of the autoencoder network was used as the search space for PoWER algorithm. The deep AE network was trained with two

498

Z. Lonˇcarevi´c et al.

datasets: a simulated dataset, and a dataset composed of a few real-world actions, combined with many actions generalized from that real-world dataset. In the following subsections, brief descriptions of CDMPs, deep autoencoder network, PoWER and generalization algorithm are given. 2.1

Cartesian Space Dynamic Movement Primitives

CDMPs consist of two parts: position and orientation parts. The position part is the same as with regular (joint space) DMPs but orientation part is different because it is represented in unit quaternions. Detailed CDMP description and auxiliary math are explained in [12]. The important for our work was that they are able to present trajectories with the following parameters: – Weights w pk , w ok ∈ R3 , k = 1, . . . , N , which represent the position and orientation parts of the trajectory, respectively – Trajectory duration τ – Final desired, goal position g p and orientation g o of the robot This parameters were inputs/outputs of our deep autoencoder network. 2.2

PoWER

In order to avoid learning complex models in robotics, we have focused on modelfree methods. Algorithm of our choice was stochastic RL method PoWER [3]. In this method, policy parameters θn at current iteration n are updated to new parameters using: (θk − θn )Rk wk , (1) θn+1 = θn + Rk wk where k is the index of the trial, and Rk the corresponding reward of the k-th trial. Importance sampling, denoted by ·wk , minimizes the number of required trials (rollouts). Importance sampler is usually implemented by taking the best m rollouts comprised of the reward Rk and corresponding parameters θk , where m is the length of the importance sampler. θk is selected using stochastic exploration policy θk∗ = θn∗ + k ,

(2)

where probability of i is given as Gaussian zero noise: i ∼ N (0, σ 2 ). Variance of the noise (σ 2 ) is the only parameter that needs to be set. 2.3

Deep Autoencoder Network

A deep autoencoder is a special type of neural network, trained so that input matches the output data. This neural network consists of two parts: an encoder and a decoder part. Encoder part encodes the input parameters into latent space values while decoder part reconstructs latent space values into the output values of the network. Training of autoencoder parameters (θ  ) is described by

Database Acquisition

θ  = arg min  θ

499

n

 1 L(θ (i) , θ (i) ) n i=1

(3)

1 L(θ (i) , h(g(θ (i) ))), n i=1

(4)

n

= arg min  θ

where L is the Euclidean distance between the input and output vectors and n is the number of samples [11]. 2.4

Generalization

As it is not feasible for us to record all the trajectories needed for the required database, we need some method that will allow us to obtain the training database from only a few demonstrations. Generalization was used for this purpose. We can compute generalized trajectories from a set of example trajectories together with the parameters characterizing the task. The set is defined as: Z = {ydk (tk,j ), y˙ dk (tk,j ), y¨dk (tk,j ); qk | k = 1, ..., M ; j = 1, ..., Tk },

(5)

where yd , y˙ d , y¨d are the measured position, velocity and accelerations and q is the query point describing the movement (in our case it was the landing spot of the ball). The generalization allows us to determine the trajectory based on the new query point that is not in the set. This means that we need to learn the function that is capable of returning the DMP parameters for the corresponding query point: (6) G(Z) : q → [WT , τ, g]T The method that we used for that purpose is Gaussian Process Regression and it is described in details in [13].

3

Experimental Evaluation

In order to test the approach of learning a throwing action in the latent space of deep autoencoder network on the real system, we used the humanoid robot Talos that was equipped with a throwing spoon both in simulation (in Gazebo) and in the real-world. Simulation database was acquired by executing 500 random shots in the simulation, and measuring the landing spot in xz-plane on the five different heights as shown by black circles in the Fig. 2. We then generalized these into 625 equally distributed shots, so that the simulated database matches the realworld database.

500

Z. Lonˇcarevi´c et al.

Fig. 1. Experimental setup for obtaining the database in simulation (left) and on the real system (right)

Real world database was obtained by manually tweaking the parameters of the initial shot until having 5 × 5 (2) successfully performed hits on different pairs of heights (z-axis) and lengths (x-axis) and applying Gaussian Process Regression to extend the database into the 625 generalized landing spots. 0.7 0.65 0.6 0.55 0.5 0.45 0.4 0.35 0.3 0.25 0.2 0.5

1

1.5

2

2.5

3

3.5

Fig. 2. Graph showing initially executed shots for the simulation obtained database (black circles), for the generalized database (not shown) equally distributed between the 5 × 5 real-world database (red diamonds) and targets used for reinforcement learning (green circles).

As the distance from the target in the initial shot heavily influences the overall number of iterations needed for reinforcement learning to achieve desired behaviour, we tested the quality of initial shot for both databases. Setups for measuring the landing spots in simulation and real system are shown in Fig. 1. Landing spots used for generalizing the corresponding databases are shown with the black circles in the Fig. 4. All the trajectories were encoded using Cartesian Dynamic Movement Primitives (CDMPs) and those parameters were inputs and outputs of autoencoder

Database Acquisition

501

network similar to [14]. As we were testing a planar problem (only in xz-plane) we excluded CDMP parameters describing the orientations around x and z axes and position in y axis, initial position and goal of the throwing spoon so that the line of the shot stays the same as for the initial shot when latent space variables are changed. Excluding that parameters and taking into the account that number of weights used for describing the trajectory shape was N = 25 we simplified the learning problem into the learning of 76 CDMP parameters: T  T , τ , (7) θ = wxT , wzT , wrot y that were encoded into the only 2 latent space parameters using the fully connected, 5-layered deep AE neural network with 76-50-2-50-76 neurons. We performed two experiments, comparing the error of the initial shot by using either the simulated or real-world generalized database, and comparing the number of required iterations needed for the successful convergence to target by using PoWER. When we compared the number of iterations with PoWER algorithm, we performed three learning experiments for three different targets, together 9 learning instances for each database (marked with green circles in Fig. 2). In order to determine the error in the distance we used experimental setup shown in Fig. 1-right and a high speed camera.

4

Results

Figure 3 shows absolute error of the initial shot for the simulated database. Results shows that this approach is good for targets in the middle-range but inaccurate for long or short throws. This is because of the inaccuracy in the dynamic model, lack of air resistance and simulated models of friction in Gazebo simulation. We performed measurements by throwing the ball at 15 different targets, three times at each target. Targets are presented by purple diamonds and the contour plot graph is obtained by interpolating between the values. 0.65

0.18 Measured Values 0.16

0.6

0.14

0.55

0.1

0.45 0.08

0.4

Error in initial shot

0.12

0.5

0.06

0.35

0.04

0.3

0.25 1.2

0.02

0

1.4

1.6

1.8

2

2.2

2.4

2.6

Fig. 3. Error of the generalized initial shot for the case of simulation-obtained database.

502

Z. Lonˇcarevi´c et al. 0.65

0.6

0.18 Measured Values Shots for creating database

0.16

0.14

0.55

0.1

0.45 0.08

0.4

Error in initial shot

0.12

0.5

0.06

0.35

0.04

0.3

0.25 1.2

0.02

0

1.4

1.6

1.8

2

2.2

2.4

2.6

Fig. 4. Error of the initial shot for the case of the database obtained by shooting on the 5 × 5 different targets marked with the black circles and then generalizing using GPR.

Fig. 5. Error convergence of PoWER or the case of the databases obtained by generalization over 5 × 5 shots (blue graph) and for the simulated database (red graph). Shaded areas show standard deviation in error.

To obtain a relevant comparison between databases obtained by simulation and by performing 25 throws on the real robot (black circles in Fig. 4) and then generalizing to other targets, we performed the same series of experiments for this second database. It turned out that there is no significant difference in maximal error of the first shot. However, the results on the generalized database show more realistic behavior, where targets that are further and higher (longest shots) have the biggest error. This corresponds to the system noise of the robot, starting placement of the ball, position of the robot, etc. For the task to be successfully learned, we ran reinforcement learning algorithm in latent space of a deep autoencoder network, as described in Subsect. 2.3. By changing the latent space parameters, the throwing converges towards the target. We used three targets (depicted by green circles in Fig. 2) and performed three epochs of reinforcement learning for each of them and for both databases. Figure 5 shows the convergence graph where error for each target is given in

Database Acquisition

503

percentage. Shaded areas on the graph show the standard deviation in error. Although from graphs it is obvious that there is no big difference in the number of required iterations (simulation database needed at most 14 and 5 × 5 needed at most 13 shots to successfully converge towards target), simulation database needed bigger exploration noise than the database made from generalized examples of the real system. The higher noise is problematic, because the learning algorithm might want to execute trajectories which are not feasible, or even not safe for the robot.

5

Conclusion

The results of the experiments have shown the possibility of using generalization for the purpose of obtaining the training database. As already shown in [6], RL and generalization methodologies are complementary because the generalization can provide a good initial approximation for RL thus speeding up the convergence. We have shown that a databases obtained by generalization of only few examples can perform better than a huge databases acquired only in simulation. This is caused by error in the model itself, dynamical models of simulation and the fact that every robot has its own system noise. In the future we will investigate how we can reduce the required real-world entries into database even further, for example by first using AEs to encode the actions, and then use generalization in the real-world.

References 1. Peters, J., Kober, J., M¨ ulling, K., Kr¨ amer, O., Neumann, G.: Towards robot skill learning: from simple skills to table tennis. In: Machine Learning and Knowledge Discovery in Databases, pp. 627–631 (2013) 2. Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. (IJRR) 32(11), 1238–1274 (2013) 3. Kober, J., Peters, J.: Policy search for motor primitives in robotics. Mach. Learn. (MLJ) 84(1), 171–203 (2011) 4. Kormushev, P., Calinon, S., Saegusa, R., Metta, G.: Learning the skill of archery by a humanoid robot iCub. In: 10th International Conference on Humanoid Robots, pp. 417–423, December 2010 5. Ijspeert, A.J., Nakanishi, J., Hoffmann, H., Pastor, P., Schaal, S.: Dynamical movement primitives: learning attractor models for motor behaviors. Neural Comput. 25, 328–373 (2013) 6. Ude, A., Gams, A., Asfour, T., Morimoto, J.: Task-specific generalization of discrete and periodic dynamic movement primitives. IEEE Trans. Robot. 26, 800–815 (2010) 7. Deniˇsa, M., Gams, A., Ude, A., Petriˇc, T.: Learning compliant movement primitives through demonstration and statistical generalization. IEEE/ASME Trans. Mechatron. 21, 2581–2594 (2016) 8. Paraschos, A., Daniel, C., Peters, J., Neumann, G.: Probabilistic movement primitives. In: Neural Information Processing Systems, pp. 2616–2624 (2013)

504

Z. Lonˇcarevi´c et al.

9. Jolliffe, I.T., Cadima, J.: Principal component analysis: a review and recent developments. Philos. Trans. Roy. Soc. A: Math. Phys. Eng. Sci. 374(2065), 20150202 (2016) 10. Chen, N., Bayer, J., Urban, S., van der Smagt, P.: Efficient movement representation by embedding dynamic movement primitives in deep autoencoders. In: 15th International Conference on Humanoid Robots, pp. 434–440, November 2015 11. Lonˇcarevi´c, Z., Simoniˇc, M., Ude, A., Gams, A.: Reduction of trajectory encoding data using a deep autoencoder network: robotic throwing. In: Advances in Service and Industrial Robotics, pp. 86–94 (2020) 12. Ude, A., Nemec, B., Petriˇc, T., Morimoto, J.: Orientation in Cartesian space dynamic movement primitives. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 2997–3004, May 2014 13. Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. The MIT Press, Cambridge (2005) 14. Pahiˇc, R., Lonˇcarevi´c, Z., Ude, A., Nemec, B., Gams, A.: User feedback in latent space robotic skill learning. In: 18th International Conference on Humanoid Robots, pp. 270–276, November 2018

Analysis of Different Methods to Close the Reality Gap for Instance Segmentation in a Flexible Assembly Cell Nawal Hafez1 , Vincent Dietrich2(B) , and Stefan Roehrl1 1 2

Technical University of Munich, Arcisstraße 21, 80333 Munich, Germany [email protected] Siemens Corporate Technology, Otto-Hahn Ring 6, 81739 Munich, Germany [email protected]

Abstract. The training of deep learning models for perception requires large annotated datasets, which are expensive and tedious to generate for real world applications. Small job shops with very low production volumes can often not spend their resources on such data generation tasks. One possible solution is the use of simulated data. But there always exists a discrepancy between simulated and real data, which may severely decrease the real world performance of models trained only on simulated data. Therefore, in this publication, we investigate different methods to account for this: photo-realistic rendering, domain randomization and domain adaptation. We analyze the individual and combined effectiveness of these approaches for an instance segmentation model. The target setting is a flexible assembly cell for low volume production with limited resources for data generation and training. We critically discuss the results and show that even simple rendering techniques, when combined with domain randomization, can lead to good results.

1

Introduction

Object detection is relevant in various fields, such as assembly and robotics. Learning based object detection generalizes effectively and in most cases eclipses more traditional approaches in efficiency and accuracy rates [12]. A major hurdle for the use of object detection and instance segmentation networks is the data generation. Often, large datasets like ImageNet [1] or COCO [9] are required that are versatile in terms of lighting and noise levels to allow for generalization. Synthetic data can be a solution despite the reality gap in performance between simulated and real data. Therefore, we evaluate different methods to close this reality gap: domain randomization, domain adaptation and photo-realistic rendering. Photo-realistic rendering tries to capture the physics of the world, so that realistic images are rendered. In domain adaptation, a mapping transformation between two domains is learned. The goal is to transform images created in a simulation to a more realistic domain. Domain randomization applies different types of noise to emulate the versatility of the real world. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 505–515, 2020. https://doi.org/10.1007/978-3-030-48989-2_54

506

N. Hafez et al.

Fig. 1. Our approach relies on generating data using three methods. We use these datasets to train a deep neural network and compare the performance on a real dataset.

The contribution of this work is an analysis and comparison of the suitability of these data generation methods for the training of an instance segmentation network, which is used in a real world assembly scenario.

2 2.1

Related Work Domain Randomization (DR)

The idea behind domain randomization is to force the training procedure to learn general scene features that possibly transfer better to real world data. In previous work, the poses of objects are randomized and shapes are added to the scene [17,18]. Additionally, arbitrary textures are added to the objects and camera positions as well as lighting settings are randomized. Finally, the background is replaced with a random image from the COCO dataset [9]. Other methods also randomize the textures of background objects in the scene, such as the table where the objects are placed, the robot and floor [16]. 2.2

Domain Adaptation (DA)

In DA, images are transformed from one domain to another. Some of the most popular DA methods are based on Generative Adversarial Networks (GAN). These networks are split into two parts. The generator tries to imitate the initial domain and the discriminator tries to determine if it is fake [4]. CycleGAN [19] utilizes unpaired images with the idea of minimizing adversarial loss. The cycle consistency ensures that the image is transferable from domain A to domain B and back to domain A [19]. It has been shown that Conditional Adversarial Networks achieve the best results [8]. The difference between conditional and regular GANs is that the conditional network learns the loss function that trains the mapping.

Analysis of Different Methods to Close the Reality Gap

2.3

507

Photo-Realistic Rendering (PR)

Rendering engines such as OpenGL cannot create highly realistic images yet. Therefore, advanced and usually slower renderers from tools such as Mitsuba1 and Blender2 can be used to increase the photo-realism [16]. A recent method utilizes a mix of domain randomization and photo-realistic rendering for object instant detection [6]. They replace the background of every image and add a lot of clutter. In the work of [7,18], 3D objects are placed in a 3D scene with realistic materials and lighting in a physically plausible manner. PR has been shown to be effective in simple environments and in combination with real data [15]. However, it always requires a lot of fine tuning. In general, it has mixed results [10,11,13,14].

3

Data Generation Approaches

In this section, we explain the data generation approaches that we investigate, which are instances of the following categories: real image acquisition, photo-realistic rendering, domain randomization, and domain adaptation. After generating the data we train an instance segmentation network to evaluate its effectiveness. We also combine the methods to see if it can improve performance. An overview of the approach is visualized in Fig. 1. 3.1

Real Images (RI)

An existing model-based object detection pipeline was used to annotate the data in a semi-automatic fashion. In the first step, a point cloud is acquired from an ASUS Xtion RGBD sensor and the plane is removed. Then, object clusters in the point cloud are extracted and classified based on their dimensions. In the final step, an RGB camera at the robot end-effector is placed in close range above the target object and the operator find shape model 3d from the Halcon library is used for pose estimation3 . The data generation for one image requires a few seconds, which may already be too long for job shops with high product variability. It also requires manually moving the objects for creating new scenes. The semi-automatic approach decreased some of the effort connected with data acquisition, but it is not viable in arbitrary applications. Alternatively manual data annotation services could be used. Overall, 130 annotated images are available, of which 105 are used for training, the rest for validation.

1 2 3

http://www.mitsuba-renderer.org. https://www.blender.org. https://www.mvtec.com/de/produkte/halcon/.

508

3.2

N. Hafez et al.

Photo-Realistic Rendering (PR)

We use renderers from Sect. 2.3 and obtain CAD models by scanning objects with the EinScan-Pro Scanner4 . Blender Rendering (BR). As first variant we use the Cycles renderer from Blender (see footnote 2), which is known for its photo-realism. Images are rendered in a simple scene as depicted in Fig. 2a. The robot is omitted, as no direct import of the robot URDF model is possible. Bullet OpenGL Renderer with Scanned Models (BORS) + Real Background Images (RBI). We render the scene of the textured models with the Bullet OpenGL5 renderer and replace the background with images of the real empty scene of our setup, see Fig. 2b. No manual data annotation is required because the objects’ positions are placed in simulation.

(a) Gen. with BR

(b) Gen. with BORS (c) Gen. with BORS + RBI

Fig. 2. Rendering with scanned models

3.3

Domain Randomization (DR)

The domain randomization is applied as a post-processing step on previously rendered data from Bullet or Blender using the imgaug library6 . The most relevant augmentations are: gaussian blur, additive gaussian noise, sharpness kernel, average blur, pixel addition/removal, median blur, contrast normalization and brightness manipulation. The augmentation is applied at every iteration, meaning that the training images are different. We also randomize the background for a subset of the data with images from the VOC2012 dataset [3]. For the rendering we use the scanned models, see Fig. 2c, as well as texture-less models, see Fig. 4a. We also apply random textures to the models in one experiment.

4 5 6

https://www.einscan.com/handheld-3d-scanner/einscan-pro/. https://github.com/bulletphysics/bullet3. https://github.com/aleju/imgaug.

Analysis of Different Methods to Close the Reality Gap

(a) Robot arms and static camera

(b) TM object

(c) Switch 1 object

(d) Switch 2 object

(e) Rail object

(f) Image of our scene

509

Fig. 3. The scene and the different objects

3.4

Domain Adaptation (DA)

We use two DA methods to train models that transform the simulated images into the target domain: pix2pix [8] and CycleGAN [19]. Both methods are trained on the annotated dataset with real images as introduced in Sect. 2.2. After training the model we generate images by transforming simulated images into the target domain and use these images to train the segmentation method, see Fig. 4. Each approach has several models from different times of the training procedure. We use a Pytorch implementation for CycleGAN and pix2pix methods7 .

4

Experiments

An overview of all the experiments is listed in Table 1. It includes the information about the method used to generate the data, as well as precision and recall recorded on the test data. 4.1

Experimental Setup

In Fig. 3, we show the test application of an electrical cabinet assembly scenario which includes four different target objects: a hat rail (Rail), a SIMATIC ET 200S terminal module (TM) and two Siemens circuit breakers (Switch 1 and Switch 2). The setup is a real robotic research demonstrator8 . The data generation, training and inference pipeline is used within the robotic assembly process [2]. All images are obtained from a static ASUS Xtion PRO LIVE camera, mounted between two KUKA IIWA robot arms. 7 8

https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix. https://youtu.be/cCIUPZi0Dk4.

510

N. Hafez et al.

(a) Simulated image

(b) Inferred with CG model 1

(c) Inferred with CG model 2

(d) Simulated image 2

(e) Inferred with P2P model 1

(f) Inferred with P2P model 2

Fig. 4. A simulated image and its transformation into the target domain using CycleGAN and pix2pix models

Using one or multiple of our data generation options we generate data and use transfer learning for the Mask-RCNN9 network with the pretrained weights of the COCO [9] dataset. The training-validation-split for Mask-RCNN is 90%– 10%. We use a subset of a real annotated dataset that was not used at any stage of the training for the evaluation. Confusion matrices as well as mask overlapping errors and precision-recall metrics are the main analysis tools for the effectiveness of each approach. We set a threshold of 0.5 for the minimum overlap. We limit the number of images to 500 simulated ones because training with significantly more images did not alter the results. If domain randomization is applied for data generation, we apply random effects at each epoch, which leads to a higher number of pictures effectively being used for training. 4.2

Domain Adaptation Experiments

Our DA experiments include training CycleGAN and pix2pix models. The models are subsequently used to generate datasets for the instance segmentation training. Examples of inferred images with the CycleGAN and pix2pix models are depicted in Fig. 4. An instance segmentation result is shown in Fig. 5. The results of DA experiments clearly show that the precision is better than recall for all but one experiment. The confusion matrix for experiment 1 is shown in Table 2a and it highlights the model’s struggle with detecting the switches. 9

https://github.com/matterport/Mask RCNN.

Analysis of Different Methods to Close the Reality Gap

511

Table 1. Overview of experiments: The methods used to generate data for each experiment, the number of images, precision and recall are specified. The following list of abbreviations is necessary to understand the experiments: DA: domain adaptation, RI: real images, VOCB: replacing backgrounds with VOC images, DR: domain randomization, TR: mesh texture randomization, BOR: Bullet OpenGL renderer, CG: CycleGAN, PR: photo-realistic rendering, BORS: BOR with scanned CAD models, BR: Blender renderer, RBI: real background images, NR: noise randomizations, P2P: pix2pix. Experiments with ∗∗ indicate DA models that required real data for training and those with ∗ indicate the direct use of real annotated data in the training. Exp. # Methods

Details

#images

1∗∗

DA

CG model 1

400

77.6

83.9

2∗∗

DA

CG model 2

400

63.0

70.2

3∗∗

DA

CG model 3

400

70.2

79.8

4∗∗

DA

P2P model 1

400

87.2

76.6

5∗∗

DA

P2P model 2

400

86.8

79.8

6∗∗

DA

P2P model 3

400

75.4

76.6

7∗

RI

RI

105

98.2

91.1

8

PR

BORS + RBI

300

93.5

81.5

9

DR

BOR + VOCB + NR

400

84.7

84.7

10

DR

BORS + VOCB + NR

300

74.7

87.9

11

PR + DR BR + VOCB + NR

300

72.9

75.8

12

PR + DR BR + BORS + NR + VOCB 300 + 150

78.4

79.0

13

PR + DR BR + BORS + NR + VOCB 150 + 150

75.6

87.9

14

DR

91.1

63.7

15∗

RI + DR RI + BOR + VOCB

105 + 200

100.0

94.4

16∗

RI + DR RI + BOR + VOCB + NR

105 + 200

100.0

95.2

17∗

RI + DA RI + CG

105 + 200

100.0

90.3

18∗∗

DR + DA BOR + VOCB + NR + CG

200 + 200

82.8

85.5

BR + VOCB + TR + NR

400

Precision [%] Recall [%]

The TM has a very high recall but low precision which is due to the fact that a lot of background objects are predicted as TMs. Experiments 2–6 lead to similar conclusions. In general, Pix2pix models have better precision than CycleGAN models. 4.3

Experiments with Real Images

Even though we use as few as 105 images, we achieve the best precision and recall when relying solely on one source of data. Looking closer at the results, it is clear that this model detects TMs and rails perfectly but the only mistakes are caused by the switches.

512

N. Hafez et al.

(a) Input test image

(b) Segmented Image

Fig. 5. Exemplary input and output after the Mask R-CNN [5] is trained with pix2pix [8] data

4.4

Photo-Realistic Rendering Experiments

In experiment 8, we recreate realistic images by using scanned models with background images of the scene, see Fig. 2b. In this case, we improve the precision in comparison to any stand-alone method except real images—but the recall is not as high. Rendering with Blender using scanned models experiment does not deliver better results even in combination with DR methods such as background replacement, as shown in experiment 12 and 13. One possible explanation is that we do not model the robotic arms in the scene. 4.5

Domain Randomization Experiments

DR experiments rely on simplified simulations, which is why we use untextured models and color each object mesh in a uniform manner in the URDF10 file (exp. 9). In this case, we achieve the same recall as the best DA model and a far higher precision. The main issue in this case is the detection of switches, see Table 2b. The false detection of TMs is usually the camera attached to the robot hand which is not modeled precisely in the simulation. Exemplary images with textured models are depicted in Fig. 2c. Similarly generated images are used for training in experiment 10. We could not scan the rail because of its reflective surface which is incompatible with the scanner. It is clear that the recall for switches improves tremendously. However, the precision for TMs decreases. When we add the TM texture, it is possible that the network does not need to learn the exact shape of the object because the robot hands are brighter in the simulation. Therefore, anything that is similarly sized to parts of the robot hands is recognized as a TM. This supports the idea of randomizing mesh texture. This is done in experiment 14 and it does improve precision. 4.6

Combinations

There are a few things we notice when combining multiple data generation methods. Combining RI with any method improves precision in all cases. This has 10

http://wiki.ros.org/urdf.

Analysis of Different Methods to Close the Reality Gap

513

Table 2. Confusion Matrices gt predicted rail switch TM background

rail switch TM background 24 0 2 0

0 37 10 1

0 3 43 4

3 7 16 0

gt predicted rail switch TM background

(a) Experiment 1

rail switch TM background 24 0 2 0

0 33 13 2

0 0 48 2

4 1 12 0

(b) Experiment 9 gt

predicted rail switch TM background

rail switch TM background 26 0 1 0

0 44 4 0

0 0 49 1

0 0 0 0

(c) Experiment 16

been shown in previous research too [15]. It is also notable that when we combine real images with augmented simulated data we achieve the best results (exp. 15 and 16). These are also the only experiments which improve recall and precision in comparison to just training with real images. The confusion matrix for experiment 16 is in Table 2c.

5

Discussions

First of all, DA does not produce good results with the low amounts of data in our experiments. Training the instance segmentation model with DA-generated data performed the worst out of the three possibilities even when combined with the other methods. Increasing data is a lot easier for CycleGAN because it does not require paired images, but only images of the two domains. Pix2pix outperforms CycleGAN which is not surprising given the visual quality of its images. Photo-realistic rendering resulted in better precision than DA methods. But in general the effort connected in terms of scene creation and attention to detail is not reflected in the results. If no real data is available, the best performance for our setup was using DR. The performance of DR with a simple simulation engine, Bullet, is better than DA and does not require any real annotated data. It achieves a recall of 85%. The OpenGL renderer from Bullet combined with DR achieves very good results especially when used with uniform colored meshes. Including scanned objects improved recall but not precision even when combined with random backgrounds.

514

6

N. Hafez et al.

Conclusion

Using neural networks for object recognition is a widely researched topic. However, it is associated with the gathering and annotation of large amounts of data. Annotating real data is time-consuming, expensive and sometimes complicated. Therefore, we compared photo-realistic rendering, domain adaptation and domain randomization to generate simulated data for the training of an instance segmentation network in order to achieve good real-world performance in an assembly scenario. In summary, surprisingly simple rendering techniques combined with domain randomization produced the best simulation-only results. Domain adaptation did not perform well for the low amounts of data that we targeted. A final conclusion is that the generation and composition of training data requires attention and might even vary across applications.

References 1. Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009) 2. Dietrich, V., Kast, B., Fiegert, M., Albrecht, S., Beetz, M.: Automatic configuration of the structure and parameterization of perception pipelines. In: ICAR. IEEE (2019) 3. Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. IJCV 88(2), 303–338 (2010) 4. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: NIPS (2014) 5. He, K., Gkioxari, G., Doll´ ar, P., Girshick, R.: Mask R-CNN. In: ICCV (2017) 6. Hinterstoißer, S., Pauly, O., Heibel, T.H., Marek, M., Bokeloh, M.: An annotation saved is an annotation earned: using fully synthetic training for object instance detection. arXiv:1902.09967 (2019) 7. Hodan, T., Vineet, V., Gal, R., Shalev, E., Hanzelka, J., Connell, T., Urbina, P., Sinha, S.N., Guenter, B.: Photorealistic image synthesis for object instance detection. arXiv:1902.03334 (2019) 8. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR (2017) 9. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ ar, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: ECCV (2014) 10. Mitash, C., Bekris, K.E., Boularias, A.: A self-supervised learning system for object detection using physics simulation and multi-view pose estimation. In: IROS (2017) 11. Movshovitz-Attias, Y., Kanade, T., Sheikh, Y.: How useful is photo-realistic rendering for visual learning? In: ECCV (2016) 12. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: CVPR (2016) 13. Richter, S.R., Vineet, V., Roth, S., Koltun, V.: Playing for data: ground truth from computer games. In: ECCV (2016) 14. Su, H., Qi, C.R., Li, Y., Guibas, L.J.: Render for CNN: viewpoint estimation in images using CNNs trained with rendered 3D model views. In: ICCV (2015)

Analysis of Different Methods to Close the Reality Gap

515

15. Sundermeyer, M., Marton, Z.C., Durner, M., Brucker, M., Triebel, R.: Implicit 3D orientation learning for 6D object detection from RGB images. In: ECCV (2018) 16. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P.: Domain randomization for transferring deep neural networks from simulation to the real world. In: IROS (2017) 17. Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., Birchfield, S.: Training deep networks with synthetic data: bridging the reality gap by domain randomization. In: CVPRW (2018) 18. Tremblay, J., To, T., Sundaralingam, B., Xiang, Y., Fox, D., Birchfield, S.: Deep object pose estimation for semantic robotic grasping of household objects. arXiv:1809.10790 (2018) 19. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV (2017)

Data-Driven Synthesis of Perception Pipelines via Hierarchical Planning Vincent Dietrich1(B) , Bernd Kast1 , Sebastian Albrecht1 , and Michael Beetz2 1

2

Siemens Corporate Technology, Munich, Germany {vincent.dietrich,bernd.kast,sebastian.albrecht}@siemens.com Institute for Artificial Intelligence, University Bremen, Bremen, Germany [email protected]

Abstract. Robotic systems in production environments have to adapt quickly to new situations and products to enable customization and short product cycles. This is especially true for the robot perception, which is sensitive to changes in environment and task. Therefore, we present an approach to quickly synthesize perception pipelines based on hierarchical planning. We calibrate the hierarchical model, such that it reflects condensed experience from historical data. We validate our approach in a simulated assembly scenario with objects from the Siemens Robot Learning Challenge, taking into account different possible sensor types and placements. Keywords: Perception system engineering Hierarchical planning

1

· Robot perception ·

Introduction

Robotic systems, for instance in assembly applications, require reliable perception in order to take meaningful actions in partially unstructured environments. Unfortunately, changes in task or environment can degrade the perceptual performance, for instance when the product changes. This requires expensive and time consuming engineering effort to find a new system con‘figuration that fulfills the task requirements of accuracy and speed. The engineering of perception systems is challenging and depends on task, environment, sensor, sensor placement and the available set of algorithms and their parameterization. A perception engineer uses a combination of knowledge and experience in order to steer a trialand-error procedure to find a suitable configuration. For instance, as depicted in Fig. 1, different algorithms have distinct output characteristics that additionally depend on the quality of the input data. The engineer implicitly knows these characteristics and uses this knowledge to choose promising pipelines. Additionally, a human engineer can perform a hierarchical task decomposition and start with high level decisions such as the sensor choice, while roughly estimating the expected performance given the sensor’s data quality. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 516–524, 2020. https://doi.org/10.1007/978-3-030-48989-2_55

Data-Driven Synthesis of Perception Pipelines via Hierarchical Planning

517

Fig. 1. Kernel density estimation of the positional output error with respect to ground truth of different perception and state estimation operators for a set of 200 perception pipelines. The operators have distinct error characteristics, which have to be taken into account in order to design a perception pipeline. Also, the inherent uncertainty and noise in the operator output is clearly visible.

Therefore, we present and investigate an approach to encode the engineering knowledge and experience in a hierarchical model, such that a hierarchical planner can perform a knowledge enabled search for suitable configurations. The main contributions of this publication are: – a hierarchical model that enables hierarchical planning for the synthesis of perception pipelines, – a systematic handling of metrics that enables online planning, and – an approach to encode engineering knowledge and experience in the hierarchical model via model calibration.

2

Related Work

In the computer vision community, the problem of automatic configuration or synthesis has a long standing history. For instance, Radig et al. [9] proposed a toolbox in 1992 for automated design of image understanding systems that uses the FIGURE system for knowledge based synthesis of pipelines by Messer [7]. The approach is based on logical rules and descriptions implemented in PROLOG, which are used to infer suitable pipelines. The more recent work of Nagato et al. [8] applies genetic programming and hierarchical program structuring in order to quickly adapt an image processing pipeline to a changing production environment. The approach was successfully demonstrated in a real production environment. Irgendfried et al. [4] address the automation of the design of entire inspection systems using accurate sensor simulation and uncertainty quantification. Another notable approach is the RoboSherlock framework by Beetz et al. [1] which leverages unstructured information management and ontologies in order to generate perception pipelines based on semantic queries.

518

V. Dietrich et al.

Our own prior work includes automatic configuration of perception systems using single level planning and factor graphs for uncertainty representation [3] and the joint optimization of pipelines and parameters [2]. This work is grounded in robotics research and builds upon a hierarchical modeling and planning system that has successfully been demonstrated for task and motion planning [6]. This allows a tight integration between task planning and perception planning. Additionally, the target domain is not restricted to computer vision. General sensor fusion and state estimation algorithms, for instance using physics simulation, are targeted as well. Furthermore, a condensed model is trained that encodes experience and can be used for relatively fast planning of pipelines.

3

Approach Hierarchical Planning

Dataset

Model Calibration

Knowledge Layer New Belief

Current Belief

Goal

Execution Layer

Pipeline

Fig. 2. Schematic overview of the approach. The core is a hierarchical planning system which acts on a 2-layer hierarchical model with a knowledge layer and an execution layer. The knowledge layer needs model calibration which is performed with a ground truth annotated dataset of pipeline executions. This planning system is able to determine a pipeline that converts a given belief to a belief with a target quality specified by a given goal.

Our approach is based on a hierarchical model combined with hierarchical planning and model calibration as depicted in Fig. 2. The hierarchical model is based on prior publications [2,5,6] and allows to model algorithms, denoted as operators, and instance classes, denoted as concepts, within the perception domain at different abstraction layers. This allows the hierarchical planner to search for solutions on cheap abstract domains and verify or refine the plans up to a real execution [6]. The model contains concepts such as pose, belief, point cloud and depth image. In this publication we employ only two different layers, the knowledge layer and the execution layer. The former encodes symbolic knowledge and experience available for the domain. The latter provides the ability to actually execute the available perception algorithms on simulated or on real data.

Data-Driven Synthesis of Perception Pipelines via Hierarchical Planning

519

The key aspect of this publication is the systematic use and calibration of so called metrics m ∈ M. The metrics allow to assess properties, especially the quality of instances, in a compact manner. E.g., when a perception engineer looks at a point cloud he indirectly assesses its quality and determines by experience which algorithm he should try next in order to get the desired result. The operators on the knowledge layer can be parameterized such that they encode this experience and model the relationship of the input metrics and output metrics. We summarize the metric model as follows: – – – –

A metric is single valued A metric can be associated with a distribution Multiple metrics can be associated with an instance We differentiate between computable and estimated metrics.

Computable metrics can be calculated at runtime based on the available data, as for instance the size of a point cloud. Estimated metrics can not be determined in an exact manner at runtime and need to be approximated. E.g., the positional distance with respect to ground truth is not known at pipeline runtime. Still, it has to be estimated if the planning goal is formulated as positional distance. The set of input and output metrics of an operator are denoted as MIN and MOUT respectively.

Point Cloud

Belief Point Delta mbel

Point Delta mpcd 0.002 m** # Points mnum 1280 x 960* Knowledge layer (K)

poseEstimation K

0.0015 m** Pos. Delta mpos

tOP = fT (MIN )

0.0012 m**

MOUT = fIO (MIN ) subset

Execution layer (E)

subset *computable metric **estimated metric

Parameter Set Point Cloud

Belief

Point Delta mpcd 0.0012 m* # Points mnum 1280 x 960*

Point Delta mbel poseEstimation E

0.0013 m* Pos Delta mpos 0.0009 m**

Fig. 3. Hierarchical view on a pose estimation operator on knowledge layer and execution layer. Detailed explanation in Sect. 3.

In order to work within the hierarchical modeling and planning approach, the metrics require a representation on different levels of abstraction. The role of the metrics within the hierarchy is therefore shown exemplary in Fig. 3. A pose estimation operator on the knowledge as well as the execution layer is displayed alongside with input and output instances. Each instance is annotated with different metrics that describe the quality of the instance. On the knowledge layer these are mostly estimated metrics that approximate the actual value.

520

V. Dietrich et al.

On the execution layer actual values can be calculated for some of the metrics. Due to uncertainties in data and execution the operators on the execution layer not always yield results as predicted on the knowledge layer. A subset relationship is defined for all concepts, which is used during planning in order to identify such deviations and perform backtracking if the relationship does not hold. Therefore, model inaccuracies at the knowledge layer can be handled. The approximations on the knowledge layer and for estimated metrics on the execution layer is encoded in the functions fT and fIO that model the runtime and the relationship between inputs and outputs of the operator, see Fig. 3. Both can be arbitrary functions that require parameterization. This step is performed initially based on a dataset of already executed perception pipelines, cf. Fig. 2. In this work we use linear models for the transfer functions and linear regression for the calibration. For the hierarchical planning we use the planner presented in [6]. The framework is designed such that different planning or search algorithms can be used at different layers. We use a scheduler at the knowledge level in order to find the fastest pipeline that fulfills the application requirements. On the execution layer breadth first search is used. On the knowledge layer no distinction between different operator parameterizations is made. The parameterization is chosen based on a sub-problem planned within the execution layer. Additionally, only metrics are used during planning that can be calculated or estimated during runtime. Therefore, the presented system can be used out of the box for online decision making and active perception.

4

Evaluation

We address two core questions regarding the presented approach: – Is the model and the calibration approach suitable for the given application? – Is the planning system capable of synthesizing working pipelines in a reasonable amount of time? The target application is an assembly scenario, where the robot perception system should give accurate 6D poses of the objects base plate and shaft 1 from the Siemens Robot Learning Challenge1 , see Fig. 4. The setup is simulated and two different depth cameras are used, an Intel D435 consumer sensor equivalent and a Photoneo PhoXi M industrial sensor equivalent. Distance dependent longitudinal noise is added based on camera characteristics according to manufacturer data2 . Additionally, the sensors are placed at six different poses in order to cover a range of different hardware setups, see Fig. 4. 1 2

https://new.siemens.com/us/en/company/fairs-events/robot-learning.html. https://www.intel.com/content/dam/support/us/en/documents/emergingtechnologies/intel-realsense-technology/BKMs Tuning RealSense D4xx Cam. pdf and http://wiki.photoneo.com/index.php/PhoXi 3D scanners family.

Data-Driven Synthesis of Perception Pipelines via Hierarchical Planning

521

d = 75 cm, α = 90◦

d = 50 cm, α = 90◦ d = 75 cm, α = 45◦ d = 25 cm, α = 90◦

Object: base plate

d = 50 cm, α = 45◦

d = 25 cm, α = 45◦

Object: shaft 1

Target Object

Fig. 4. Left: Application setup with 6 different view poses. Right: Target objects from the Siemens Robot Learning Challenge.

We use the following computable metrics on the execution layer: – mnum : Number of points of the point cloud – mpcd : average distance between points that can be a associated with known objects in the scene, such as a plane, and those objects – mbel : average distance between a depth rendering of the object estimate and the point cloud The position distance mpos between belief and ground truth can not be computed at runtime and has to be estimated. The mapping between mbel and mpos is therefore as well approximated with a linear model during the model calibration. The available operator set within the experiment is summarized in Table 1. Table 1. Overview of used operators and their model calibration. (Operators marked with * are implemented using Bullet (https://pybullet.org/), otherwise using Open3D (http://www.open3d.org/)) Operator

Comment

# Param. Sets fT

fIO

getPointCloud*

point cloud rendering

0

d → mpcd

–

0.030 s

prepareModel

create object point cloud

1

1

0.050 s

–

removePlane

plane estimation and removal 3

1

0.013 s

mnum → mnum

smoothTSDF

using Signed-Distance Functions

1

0.288 s

2



mnum mpcd





→

poseEstimationFPFH using Fast Point Feature Histograms

8

2

mnum → t mpcd → mbel

refineICP

Iterative Closest Point

2

2

mnum → t mbel → mbel

refinePhysics*

achieve physical plausibility with plane

6

1

2.060 s

mnum



mpcd

mbel → mbel

The model calibration is performed on a dataset of an exhaustive exploration of pipelines on 2 scenes, 2 cameras and 6 viewpoints of the object base plate .

522

V. Dietrich et al.

The result of the calibration are parameterized models of the runtime behavior fT and the input to output relation fIO . In this work we use single-input, single-output linear models, but the approach supports models of arbitrary complexity and arbitrary number of inputs and outputs. The choice of input and output types for the models is done by manual examination of the correlation. The relationship between given and ground truth metrics is calibrated in addition to the operator behavior. In Table 1 the calibrated input-output relationships for the different operators are listed. Approximately constant values are directly shown. In the first row of Fig. 5 actual linear approximations for runtime and input-output relationship are displayed.

Fig. 5. Top left: Linear model fit for runtime depending on input cloud size for poseEstimationFPFH. Top right: Linear model fit for the output mbel depending on the input mbel . Bottom left: Comparison between baseline and planned pipelines for sensor D435. Bottom right: Comparison between baseline and planned pipelines for sensor Phoxi M.

In order to assess the approach we compare the planning results with the results of a greedy search, as shown in the bottom row of Fig. 5. The calibration has been performed only on base plate and results are displayed for both

Data-Driven Synthesis of Perception Pipelines via Hierarchical Planning

523

objects. Furthermore, the planning was performed with two different goal settings: 0.005 m and 0.0025 m as indicated by the colors and the horizontal lines. We can make a few observations. First of all, longer runtimes lead to lower errors, which is coherent with engineering intuition. Although this is not a general rule. For instance, the removal of the plane can lead to higher accuracy and shorter runtime due to a lower number of points for the pose estimation. Furthermore, we can observe that the ground truth of the majority of the planned results fulfills the goal requirements. During planning only runtime metrics are used, which can only approximate the ground truth. Therefore, the result may sometimes deviate from the ground truth due to uncertainty and insufficient calibration. The goal is reached with high success rates for the calibration object as well as the unseen object, which indicates that the model generalizes and is sufficiently complex. Most of the outliers are associated with the unseen and non calibrated shaft 1 object, which conforms with our expectations. A further observation regards the runtime of the planned pipelines. On the knowledge level a scheduler is used in order to find the fastest pipelines that reach the goal. The experimental data shows that the planning system is actually capable of identifying a suitable compromise between speed and accuracy. This is achieved with a median planning time of about 51 s. Finally, we compare the results between the consumer camera and the industrial camera. The more expensive industrial camera can achieve higher accuracy overall. But while there are no outliers for 2.5 mm the model still leaves room for improvement as the outliers for a goal of 5 mm imply. In summary, both questions formulated at the beginning of this section can be generally answered positively. However, there is still potential within further investigation to improve the model accuracy and decrease the synthesis times for actual use for active perception.

5

Conclusion

We presented an approach to synthesize perception pipelines using hierarchical planning, which allows to find a sequence of operators with sufficient accuracy for the task at hand. The hierarchical model consists of a knowledge layer, which encodes engineering experience from data and steers the search procedure, and an execution layer where actual operators are executed on the given data. Scheduling on the top level ensures that pipelines with short runtime are preferred. Experiments in simulation with different objects, sensor types and sensor placements show promising results. The knowledge layer is calibrated for one object and performs well for a unseen test object. The duration of the synthesis process is mostly less than a minute and therefore allows rapid adaptation, e.g., for a product change. Further speedup would also enable running the synthesis online during operation.

524

V. Dietrich et al.

References 1. Beetz, M., B´ alint-Bencz´edi, F., Blodow, N., Nyga, D., Wiedemeyer, T., M´ arton, Z.C.: RoboSherlock: unstructured information processing for robot perception. In: International Conference on Robotics and Automation. IEEE (2015) 2. Dietrich, V., Kast, B., Fiegert, M., Albrecht, S., Beetz, M.: Automatic configuration of the structure and parameterization of perception pipelines. In: International Conference on Advanced Robotics. IEEE (2019) 3. Dietrich, V., Kast, B., Schmitt, P., Albrecht, S., Fiegert, M., Feiten, W., Beetz, M.: Configuration of perception systems via planning over factor graphs. In: International Conference on Robotics and Automation. IEEE (2018) 4. Irgenfried, S., Worn, H., Bergmann, S., Mohammadikaji, M., Beyerer, J., Dachsbacher, C.: CAD based workflow for semi-automatic design of optical inspection systems. AT-AUTOMATISIERUNGSTECHNIK 65(6), 426–439 (2017) 5. Kast, B., Albrecht, S., Feiten, W., Zhang, J.: Bridging the gap between semantics and control for industry 4.0 and autonomous production. In: International Conference on Automation, Science and Engineering. IEEE (2019) 6. Kast, B., Dietrich, V., Albrecht, S., Feiten, W., Zhang, J.: A hierarchical planner based on set-theoretic models: towards automating the automation for autonomous systems. In: International Conference on Informatics in Control, Automation and Robotics (2019) 7. Messer, T.: Model-based synthesis of vision routines. In: Advances In Machine Vision: Strategies and Applications, pp. 79–97. World Scientific (1992) 8. Nagato, T., Koezuka, T.: Automatic generation of image-processing programs for production lines. Fujitsu Sci. Tech. J. 52(1), 27–33 (2016) 9. Radig, B., Eckstein, W., Klotz, K., Messer, T., Pauli, J.: Automatization in the design of image understanding systems. In: International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Springer (1992)

Crashing to Learn, Learning to Survive: Planning in Dynamic Environments via Domain Randomization Sarath S. Menon1(B) , R. Sakthi Vignesh1 , and Santhakumar Mohan2 1

PSG College of Technology, Coimbatore, India [email protected] 2 IIT Palakkad, Palakkad, India

Abstract. Autonomous robots in the real world should avert collisions with pedestrians and react quickly to sudden changes in the environment. Most local planners rely on static environment maps that cannot capture such critical elements of uncertainty. Learning based methods for end-toend navigation have become popular recently, but it is still unclear how to incorporate collision avoidance in a simple, safe and quick manner. We propose a reinforcement learning curriculum based on domain randomization to train a high performing policy entirely in simulation. The policy is first trained in a simple obstacle-free environment to quickly learn point to point navigation. The learned policy is transferred to a dynamic environment to learn collision avoidance. The key idea is to randomize the obstacle dynamics to obtain a robust planner that can be directly deployed to the real world. The resultant policy outperforms conventional planners in dynamic real world environments, even when the robot is intentionally obstructed. Keywords: Mobile robot navigation Domain randomization

1

· Reinforcement learning ·

Introduction

The game of American football best illustrates human dexterity in real world navigation. Players execute complicated strategies with split-second timing to dodge 11 aggressive competitors, finally reaching a goal position. Unfortunately, this sophisticated skillset that comes naturally to human beings sets a high bar for autonomous robots navigating in human crowds. At minimum, the robot is expected not to hinder pedestrian movement in the environment. Also, it should react quickly to changes in environment configuration such as reorientation of furniture. This issue is especially relevant for service robots operating in factory floors, where the configuration space is constantly changing. Widely used collision avoidance techniques such as DWA often fail to meet this standard [6]. Please visit https://www.youtube.com/watch?v=FSj-Lmh3fVE to see a video of our work. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 525–534, 2020. https://doi.org/10.1007/978-3-030-48989-2_56

526

S. S. Menon et al.

This is a direct consequence of assuming a static environment: planning in fixed maps that cannot capture moving objects. Conventional robot navigation systems are modular: systems for perception, state estimation, planning and control are developed independently. Although each of these aspects is well understood individually, merging and finetuning these modules to obtain a complete system demands significant domain experience and expensive manual labor. Furthermore, rich environment maps with accurate obstacle representations are not always available. Typically, maps are generated through SLAM, a time consuming process that relies on manual control for efficient environment exploration. However, map building is not practically feasible for applications such as search and rescue. Another key concern is that conventional planners cannot leverage experience to self-improve performance over time. In this work, we propose an end-to-end trainable planner for point to point navigation in dynamic environments. We use reinforcement learning to train a policy that generates robot velocities given sensory input and relative goal position. Our approach does not depend on rich environment maps – only high level building floorplans are required to specify the goal position. The navigation stack consists of the high level planner trained via reinforcement learning and a low level kinematic controller. The high level planner maps a combination of raw sensory input and relative goal position to robot velocity commands. The robot specific kinematic model maps the robot velocities to low level wheel velocities. Aim for the sky and you’ll reach the ceiling. Aim for the ceiling and you’ll stay on the floor. — Douglas Adams Although originally used in a completely different context, the above quote summarizes key the motivation behind our approach – Domain Randomization. Instead of training in a fixed simulation environment, the obstacle dynamics and number of obstacles are randomized to obtain a robust planner. The learned policy can be directly deployed since it is invariant to the discrepancies between simulation and the real world. 1.1

Contributions

– Domain randomization for robust obstacle avoidance – outperforms DWA (Dyamic Window Approach) and TEB (Time Elastic Band) in avoiding collisions with dynamic obstacles including human agents and moving cartons, even when the robot is blocked intentionally. – Generalizes to previously unseen environments – trained in a simple bounded space but generalizes to offices, hallways and factory floors. – Entire training happens in simulation – learning is safe (no harm to human beings or the robot platform), inexpensive and accelerated by a factor of 10. – The learned planner is platform independent – training in simulation and real world deployment happens on mobile platforms with different kinematic constrains.

Crashing to Learn, Learning to Survive: Planning in Dynamic Environments

527

– Our approach does not rely on expert demonstrations speed up training – human demonstrations are inherently sub-optimal and tedious to collect. Instead, we use forward transfer learning to reduce training time by 90%. Learning Robot Navigation. Approaches for learning mobile robot navigation using reinforcement learning dates back to the original work of [18] and [16]. Several independent works have discussed end-to-end local planners for robot navigation [10,11,17,20] (in particular, see the foundational work by [17]). Faust et al. proposed PRM-RL, pioneering approach for long range navigation using reinforcement learning [5], followed by an extension to automate the hyperparameter search [1]. However, they do not account for dynamic obstacles. Everett et al. [4] proposed an inspiring local planner for motion planning in dynamic environments, which is similar to our work in many aspects. However, it depends critically on accurate state estimation of dynamic agents, which is error prone. In contrast, our approach doesn’t rely on accurate state estimation. Consequently, we could omit the three expensive depth cameras used for state estimation. Domain Randomization. Learning based systems trained in simulation do not transfer directly to the real world in most cases: The real world contains a variety of error sources that are lost in simulation. [9] and [3] have discussed this issue in detail. One approach to bridge this gap is to train the model in a simulator that closely resembles real world environments. However, it is a time consuming and expensive process. [14] have presented an inspiring solution to this problem in the context of UAV collision avoidance. Their method, introduced by [2] is called Domain Randomization. They randomized the environment colors and textures extensively to account for sources of variability in the real world. Similarly, [19] achieved robust localization for robot grasping by randomizing visual characteristics of the object to be grasped. However, such nuisance factors are already abstracted away by the 2D LiDAR in our case. Instead, we use domain randomization to learn robust control strategies – collision avoidance in dynamic environments irrespective of the dynamics and number of obstacles.

2

System Overview

We address the issue of motion planning in dynamic environments using Reinforcement Learning (RL). The agent should navigate to a specified goal while avoiding collisions at the same time. The policy maps directly from sensory input to motor commands: perception, cognition and planning are considered to be integrated into the policy. However, separate modules for input preprocessing and low level control are required to complete the navigation stack. The overall architecture of our navigation system is illustrated in the framework of the sense act cycle in Fig. 1.

528

S. S. Menon et al. Laser scan

xt | 10

vt | 2

Relative position

pt | 2

Linear and Angular velocities

Previous action

vt−1 | 2

End to end planner

Kinematic model

2D LiDAR

Sense

Robot

Act

Fig. 1. An overview of our approach in the sense-act framework. The policy accepts a 14-dimensional state vector and directly outputs 2-dimensional robot velocities.

We model the navigation process as a Markov Decision Process, M = where S is the state space, A is the action space, P(st+1 |st , at ) is the transition probability distribution, R is the reward function and γ ∈ [0, 1] is the discount factor. Perception. We use a 2D LiDAR for sensory input since it abstracts visual nuisance factors in the environment such as surface texture and color. The policy accepts as input a tuple {xt , vt−1 , pt } where xt is the 10 dimensional laser scan vector, pt is the two dimensional vector encoding the robot position relative to the goal and vt−1 is the planner output from the previous time step. Although commercially laser scanners provide 360–720 dimensional distance measurements, we do not feed the raw distance measurements since it would limit the planner’s utility to the particular model of LiDAR used. Instead, 10 distance measurements are uniformly sampled at each timestep and normalized to the range [0, 1]. The relative position is expressed in terms of the Euclidean distance and polar angle between the robot and goal position. This term is analogous to the error that is minimized in feedback control systems. Naturally, direct usage of the absolute robot pose yields the best results in simulation. However, Gaussian noise is added to obtain a planner that is robust to real world localization error. The previous output is included to account for the sequential nature of robot motion planning.

Crashing to Learn, Learning to Survive: Planning in Dynamic Environments

529

Planning Policy. We use reinforcement learning to train a neural network policy in simulation that can be directly transferred to the real world. We use a variant of DDPG [8], an off-policy actor-critic algorithm since it is widely used and benchmarked in the reinforcement learning community. Reinforcement learning aims to find a policy π that maximizes the expected sum of long term reward (1): T   γ t · r(st , πθ (st )) J(θ) = E

(1)

t=0

The reward indicates whether the action taken by the policy is desirable or not. This feedback is obtained from a manually engineered reward function (2). To ensure that the robot learns to move successively closer to the goal, the reward magnitude should be proportional to the euclidean distance from goal position dt . The reward is positive if the robot moves closer to the goal compared to the previous timestep and negative if it moves away. We have designed a linear reward function similar to [17]. By observing [state, reward] sequences, the policy learns to choose actions in such a way that the long term reward is maximized. ⎧ ⎪ if success ⎨0.95, (2) R(d) = −1, if crash ⎪ ⎩ 0.2 ∗ (dt − dt−1 ), if otherwise Instead of low level wheel velocity control [5], the planner predicts linear and angular robot velocities at each timestep. This relieves the policy of the added complexity of learning rigid body kinematics, which is already well understood [15]. Also, this enables generalization across mobile robot platforms since the learned planner is independent of any particular drive assembly. Low Level Control. We use the kinematic controller included in the ROS framework [12] to transform linear and angular velocities to individual wheel velocities. Considering the performance limitations imposed by physical robot platforms, the maximum linear and angular velocities should be limited to 2.2 m/s and 1 rad/s respectively. However, the policies trained via DDPG output unconstrained velocities. Therefore, we run the 2-dimensional output through Sigmoid and Tanh functions respectively to constrain robot velocities to the range [0, 1] and [−1, 1]. This output is scaled by coefficients 2.2 and 1 to obtain suitable linear and angular velocities.

3

Method

From a pure RL perspective, the agent has to learn two things: (a) Knowledge that reaching the goal state yields maximum positive reward and undesirable states yield negative reward. (b) Skills to reach the state which yields the highest positive reward. In the context of navigation, it translates to (a) Knowledge that

530

S. S. Menon et al. Simulation

Obstacle Free

Random obstacle dynamics

Real World

and Sensor noise

Fig. 2. The key idea is to use domain randomization to learn robust control strategies for collision avoidance in dynamic environments. First, the planner is trained in a simple obstacle free environment to quickly learn point to point navigation. Second, random static obstacles are introduced to obtain a policy that is robust to obstacle dynamics.

reaching the goal position yield maximum positive reward and collisions yields maximum negative reward. (b) Selecting linear and angular velocities to reach the goal position without crashing. Figure 2 illustrates the training process. It is impractical to train the planner directly in real world environments since deep RL algorithms are highly sample inefficient and unstable during the initial exploratory phase. Therefore, we train the policy in a controlled, simulated environment and transfer directly to the real world. The most intuitive approach to train the policy directly in a dynamic environment. However, it is time inefficient due to a fundamental challenge in RL: environment exploration to find out which states yield maximum positive reward. Since the neural network policy is initialized with random weights, the robot executes random trajectories in the initial training phase. Fortunately, this element of randomness aids environment exploration – some trajectories happen to lead to the goal position. If random exploration happens in a dynamic environment, the robot is more likely to crash into a moving obstacle rather than reach the goal position. Hence, a relatively large number of training episodes are required for complete environment exploration. Therefore, we decompose the navigation problem into two subtasks: point to point navigation and collision avoidance. First, the policy is trained in an obstacle free environment to quickly learn point to point navigation. Second, the learned policy is transferred to a dynamic environment to learn collision avoidance. Collision Avoidance via Domain Randomization. RL agents learn from failures – the robot has to crash multiple times to learn that collisions are undesirable. However, it is debatable how the obstacles should be introduced into the environment. The most intuitive approach is to model the obstacle the dynamics to closely resemble human dynamics [7]. However, a policy trained in such an environment cannot avoid collisions with other moving objects. Instead, we use the simple but promising concept of domain randomization. The key idea is

Crashing to Learn, Learning to Survive: Planning in Dynamic Environments

531

Task completion time Domain randomization

Success rate [%]

Static

No

26

Static

Yes

100

Dynamic

Yes

93

100

Time duration [s]

Obstacle type

Ours Dynamic Window

50

Timed Elastic band 0

20

40

60

80

100

Success Rate[%] (a)

(b)

Fig. 3. (a) Training results in simulation. As expected, the first case demonstrates the worst performance. A success rate of 100% is achieved if the robot navigates to the goal position without crashing in all 10 trials. Counterintuitively, the second case outperforms the third by 7%. The third policy is used for real world experiments (b) Comparison of task completion time in dynamic environments – Our policy is 40% faster

to introduce multiple obstacles at random positions in the environment during training, even in unrealistic configurations. We argue that a policy trained in such a highly variable environment learns to generalize across all possible obstacle configurations. The policy cannot differentiate virtual and physical obstacles since the real world seems just like another unrealistic simulation. This approach is powerful because it doesn’t assume knowledge about the target environment in advance – the model is already invariant to obstacle dynamics (Fig. 3).

4

Results

Success Rate[%]

Social Navigation

Static Obstacles

Carton Pushing

100

Ours Dynamic Window

50

Timed Elastic band 0 0

1

2

3

0

1

2

3

0

1

2

3

Number of Obstacles (a)

(b)

(c)

Fig. 4. Results of real world experiments (10 trials). A success rate of 100% is achieved if the robot navigates to the goal position without crashing in all 10 trials. (a) The robot has to reach the goal position in the presence of 3 human agents navigating randomly. (b) The robot path is obstructed intentionally by moving cartons. (c) Static obstacles are introduced at random positions after map building. Our policy outperforms conventional planners in all 3 cases.

We compare our planner against two widely used local planners: Dynamic Window Approach (DWA) [6] and Timed Elastic Band (TEB) [13]. Quantitative

532

S. S. Menon et al.

evaluation is in terms of the mean success rate: the fraction of successful attempts over 10 trials. A success rate of 100% is achieved if the robot navigates to the goal position without crashing in all 10 cases. Social Navigation. First, we evaluate the ability to navigate in crowded real world scenarios (see Fig. 4 (a)). Our policy outperforms both DWA and TEB in all our tests. When obstructed by a human agent, the conventional planners get stuck and abort planning 8 out of 10 times. In some cases, the TEB planner waits for the agent to pass, DWA can replan if the agent waits or takes a detour. However, such delays are unacceptable in social navigation – trajectory recomputation takes at least 15 s. Our the end-to-end policy is computationally much lighter than modular pipelines. Intentional Obstruction. We evaluate performance in two separate scenarios: (a) 3 human agents try to obstruct the robot path throughout the course of navigation. The conventional planner gets stuck and aborts the planning sequence in all test cases, whereas our planner takes reliable and safe detours to reach the goal position. (b) a human agent obstructs the robot path by pushing cartons. Figure 4(b) summarizes our test results. We find that the performance of both planners drops to less than 40% in case of a single carton. Either the robot collides with the incoming carton or planning is aborted. In case of multiple cartons, the success rate drops to zero whereas our planner maintains 100% and 80% success rate respectively. Generalization to New Environments Although the policy is trained a 10 × 10 m bounded space, it generalizes to hallways, offices and even factory floors. Our policy achieves more than 90% success rate in all cases.

5

Conclusion

We present reproducible and robust techniques for robot navigation in dynamic environments. Domain randomization is used to learn robust collision avoidance strategies instead of just robust perception or localization. The policy is first trained in a simulated obstacle free environment to learn point to point navigation. Second, random static obstacles are introduced to obtain a policy that is robust to obstacle dynamics. The resulting policy outperforms DWA and TEB in case of social navigation, even when the robot is intentionally obstructed. Although the policy is trained in a 10 × 10 m bounded space, it generalizes to offices, hallways and even factory floors. Since the entirety of the learning process happens in simulation, the learning process is cheap, safe and accelerated.

Crashing to Learn, Learning to Survive: Planning in Dynamic Environments

533

References 1. Chiang, L., et al.: Learning navigation behaviors end-to-end with AutoRL. IEEE Robot. Autom. Lett. 4, 2007–2014 (2019) 2. Dosovitskiy, A., Koltun, V.: Learning to act by predicting the future. CoRR (2016). http://arxiv.org/abs/1611.01779 3. Dulac-Arnold, G., Mankowitz, D.J., Hester, T.: Challenges of real-world reinforcement learning. CoRR (2019). http://arxiv.org/abs/1904.12901 4. Everett, M., Chen, Y.F., How, J.P.: Motion planning among dynamic, decisionmaking agents with deep reinforcement learning. IROS (2018). http://arxiv.org/ abs/1805.01956 5. Faust, A., Ramirez, O., Fiser, M., Oslund, K., Francis, A., Davidson, J., Tapia, L.: PRM-RL: long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning. CoRR (2017). http://arxiv.org/abs/1710.03937 6. Fox, D., Burgard, W., Thrun, S.: The dynamic window approach to collision avoidance. IEEE Robot. Autom. Mag. 4, 23–33 (1997) 7. Helbing, D., Moln´ ar, P.: Social force model for pedestrian dynamics. Phys. Rev. E 51, 4282–4286 (1995) 8. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. CoRR (2015). https://arxiv.org/abs/1509.02971v4 9. Mahmood, A.R., Korenkevych, D., Vasan, G., Ma, W., Bergstra, J.: Benchmarking reinforcement learning algorithms on real-world robots. CoRR (2018). http://arxiv. org/abs/1809.07731 10. Pfeiffer, M., Schaeuble, M., Nieto, J.I., Siegwart, R., Cadena, C.: From perception to decision: a data-driven approach to end-to-end motion planning for autonomous ground robots. CoRR (2016). http://arxiv.org/abs/1609.07910 11. Pfeiffer, M., Shukla, S., Turchetta, M., Cadena, C., Krause, A., Siegwart, R., Nieto, J.I.: Reinforced imitation: sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations. CoRR (2018). http://arxiv. org/abs/1805.07095 12. Quigley, M., Conley, K., Gerkey, B.P., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: ROS: an open-source robot operating system (2009). http://www. willowgarage.com/sites/default/files/icraoss09-ROS.pdf 13. Quinlan, S., Khatib, O.: Elastic bands: connecting path planning and control, vol. 2, pp. 802–807, May 1993. http://www8.cs.umu.se/research/ifor/dl/Control/ elastic%20bands.pdf 14. Sadeghi, F., Levine, S.: CAD2RL: real single-image flight without a single real image. CoRR (2016). http://arxiv.org/abs/1611.04201 15. Siegwart, R., Nourbaksh, R., Scaramuzza, D.: Introduction to Autonomous Mobile Robots, 2nd edn. The MIT Press, Cambridge (2011) 16. Smart, W.D., Pack Kaelbling, L.: Effective reinforcement learning for mobile robots. In: Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. CH37292) (2002). http://people.csail.mit.edu/lpk/papers/ 2002/SmartKaelbling-ICRA2002.pdf 17. Tai, L., Paolo, G., Liu, M.: Virtual-to-real deep reinforcement learning: continuous control of mobile robots for mapless navigation. CoRR (2017). http://arxiv.org/ abs/1703.00420 18. Thrun, S.: An approach to learning mobile robot navigation. Robot. Autom. Syst. 14, 301–319 (1995)

534

S. S. Menon et al.

19. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P.: Domain randomization for transferring deep neural networks from simulation to the real world, pp. 23–30, September 2017. http://arxiv.org/abs/1703.06907 20. Zhang, J., Springenberg, J.T., Boedecker, J., Burgard, W.: Deep reinforcement learning with successor features for navigation across similar environments. CoRR (2016). http://arxiv.org/abs/1612.05533

Target Manipulation in Nuclear Physics Experiment with Ion Beams Diego Sartirana1,2(B) , Daniela Calvo1 , Vittoria Capirossi1,3 , Carlo Ferraresi1,2 , Felice Iazzi1,3 , Federico Pinna1,3 , and for NUMEN collaboration 1

INFN, Section of Turin, 10125 Turin, Italy [email protected] 2 DIMEAS, Polytechnic of Turin, 10129 Turin, Italy 3 DISAT, Polytechnic of Turin, 10129 Turin, Italy Abstract. The NUMEN project at Laboratori Nazionali del Sud of INFN studies the nuclear matrix elements of Double Charge Exchange reactions between some particular nuclei. This information will be helpful in studying the Neutrinoless Double Beta decay. Because of the very low cross-sections of such reactions, high statistics must be acquired using very intense ion beams coupled with thin targets in order to get a good resolution in energy measurements. Since the target is irradiated with low energy and intense ion beams the crucial problems are the large amount of released heat and the activation of the material. To cope with the first problem isotopes are evaporated on a graphite substrate with high thermal conductivity that will be cooled down using a cryocooler. The second issue demands for an automatic system for the target handling during the data taking of the experiment. Positioning of the target-holder with appropriate precision and contact force is a special requirement, while all materials have to be selected to suit both the large temperature range and the foreseen radiation level. This work focuses on the automatic system for target manipulation, describing its design characteristics to cope with such particular working condition and to be integrated in the experiment.

Keywords: Automatic manipulator Hostile environment

1

· Nuclear physics experiment ·

Introduction

One of the hot topics in modern Nuclear Physics is the neutrino particle nature; Neutrinoless Double Beta decay fits in this context. Double Charge Exchange (DCE) reactions and Neutrinoless Double Beta decay share many similarities. The NUMEN project [1] aims to study some DCE nuclear reactions. First data have been already taken at Laboratori Nazionali del Sud (a INFN facility in Catania, Italy), using the spectrometer MAGNEX. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 535–543, 2020. https://doi.org/10.1007/978-3-030-48989-2_57

536

D. Sartirana et al.

Since high statistics is required to get good precision measurement, a more powerful superconducting cyclotron has been designed, together with an upgrade of the whole MAGNEX spectrometer. These ion beams will impinge on targets made of thin layers of specific isotopes (up to some hundreds of nm thick) generating heat, which has to be dissipated to avoid damaging the target itself [2]. To cope with the heat dissipation, the isotopes are evaporated on a HOPG (Highly Oriented Pyrolytic Graphite) substrate of about 2 µm featuring a high thermal conductivity. A target-holder (TH) made of copper clamps the target disk. In the current setup, the target is mounted on a sample-holder contained in a cylindrically shaped vacuum chamber (scattering chamber). The chamber is mounted on a rotating platform that hosts MAGNEX, a quadrupole and the detectors (Fig. 1). The rotating axis coincides with the target, which remains still. A new chamber has been designed to fit the target cooling system, which is foreseen to be a cryocooler able to dissipate about 20 W by keeping the target at less than 40 K.

Fig. 1. Plan view of the principal structures of MAGNEX spectrometer. The ion beam first hits the target, then the results of this interaction are driven by a quadrupole magnet and a dipole, and reach the focal plane detector (FPD).

To guarantee a good thermal conductivity, the base of the target-holder must be accommodated to the cold finger of the cryocooler with an adequate coupling force of about 15 N. Moreover, the contact surfaces will be lapped to reduce the thermal resistance. With the above value of the coupling force, the temperature difference between the surfaces of cryocooler and target-holder has been estimated to be less than 0.3 K [3].

Target Manipulation in Nuclear Physics Experiment with Ion Beams

537

In addition, the target-holder must be shaped to accommodate a graphite sample disk for reference measurements, an empty location to study the background eventually generated by the copper frame itself around the target, and an alumina disk used to verify the alignment and the squeezing of the ion beam (Fig. 2). The alignment of each of these elements to the beam line asks for a linear vertical motion of 100 mm of the target-holder, which is achieved by using a linear actuator under the sustain of the cryocooler. Besides, the angular error in the target-holder alignment to the ion beam must be less than 0.5◦ to avoid different path length of the particles inside the isotope layer and therefore affect the resolution in energy of the spectrometer.

Fig. 2. Image of the target-holder (TH) exploded in all its components. The total height is 95 mm and the diameter of the circular base is 50 mm.

The ions will be transported to the NUMEN experimental hall by two different beam lines, the High Intensity (HI) line, that provides high intensity ion current, and the Low Intensity (LI) line, that allows using ion beams featuring lower currents. A relative 70◦ rotation of MAGNEX aligns the axis of the quadrupole along the two lines respectively, bringing it alternatively in the HI or in the LI configuration. Therefore, the connection between target-holder and cryocooler must respect these two different configurations by securing two different positions of coupling system, in order to maintain the target always perpendicular to the beam line. Since the scattering chamber is built-in with the rest of MAGNEX, each configuration requires proper connection to the corresponding beam line. The high intensity of the ion beams together with the planned values of energy per nucleon poses a not negligible problem of radiation dose inside the chamber. The activation of the materials is not excluded; therefore, the replacement of an exhausted target cannot be done manually. An automated system capable of handling the target must be designed and built. The following sections will present the design details of the whole target manipulation and storage system, as well as a description of the handling procedure.

538

2

D. Sartirana et al.

The Target Handling and the Manipulator

The Fig. 3 shows the procedure that shall be adopted for placing the targets inside the scattering chamber, once the latter has been posed in communication with the ambient by opening a proper gate valve. The sequence of events and phases is: 1) entry of manipulator end effector along a horizontal trajectory, with TH in horizontal position; 2) rotation of TH around a transversal axis, reaching the vertical position; 3) lifting of Cold Finger CF of the cry-ocooler to contact the base of TH; 4) rotation of TH to complete the bayonet coupling with CF; this last movement places the target perpendicular to the ion beam. Subsequently, the manipulator will retract and the chamber will be sealed and de-pressurized for performing the experiment. The same movements, with different sequence, will be performed by the manipulator for extracting the target after the experiment.

1)

2) W

3) W

4) W

W

TH G G

G

TH

TH CF

CF

G

TH CF

CF

Fig. 3. Sequence for placing the target inside the scattering chamber. At the end, the target is placed perpendicular to the ion beam.

The design of the automatic system for target handling and storing takes into account the specifications required by the experiment. The system is composed by two main devices: the manipulator and the storage system. The Fig. 4 shows an overall image of the manipulator, which is supported by a rigid structure TF fixed to the MAGNEX platform. The system has in the complex four degrees of freedom. The first DoF is a linear horizontal motion, provided by the pneumatic cylinder C (SMC mod. C96S-B40-800CJJ, 40 mm bore and 800 mm stroke), coupled with two linear guides LG (MISUMI mod. BSH2M-D30-L1150M12) devoted to support any radial component of the external forces acting on the end effector. Two further DoFs are provided by the wrist unit W (IAI mod. WUM), which can control the pitch and yaw motion of the end effector, consisting of the pneumatic gripper G (SCHUNK mod. KGG 60-20), devoted to grasp the targetholder TH by means of two specially shaped fingers F.

Target Manipulation in Nuclear Physics Experiment with Ion Beams

539

A series of adjustable limiters allows regulating with high precision the several movements of the manipulator, minimizing any possible positioning error. The manipulation system is placed at a certain distance from the scattering chamber, i.e. far away from the target-holder during the beam collision: this is necessary to prevent damaging of the system from the radiations produced by the experiment. To this regard, the dose of radiations to this region is estimated to be around 2 × 106 gray as upper limit (as evaluated from Radioprotection Office of LNS). This value was an important reference for the choice of the system’s components and materials. Particular attention was paid to the CERN documentation about mechanical behaviour in activated environments [4]. In particular, the most critical materials in these conditions are the silicone rubbers for the pneumatic components, which are not forbidden but require periodical check and maintenance.

Fig. 4. Overall view of the manipulator.

3

Connection Between Target-Holder and Cryocooler

The connection between the target-holder TH and the cold finger CF of the cryocooler is achieved by a bayonet coupling, presented in Fig. 5. The base of the target-holder is provided with two pairs of wedge-shaped protrusions that mate with two proper seats obtained in a positioning cup PC attached to the CF. Since the MAGNEX must be able to operate both in LI and HI configuration, which correspond to two different orientations of the whole spectrometer shifted by 70◦ around the target vertical axis, the manipulation system shall couple the TH to the CF in one of both angular position required by the foreseen configuration, using the corresponding couple of wedges.

540

D. Sartirana et al.

In order to provide the desired forcing between the contact surfaces of TH and CF, the rotary coupling motion forces the wedges under two flexural springs S, which provide the desired pre-load. The springs are fixed to the CF by a positioning cup provided with two equidistant protrusions. These latter also have the function of limiting the coupling movement, ensuring the precision of positioning. The flexural springs have been dimensioned respecting the specific requirements of this application. The first request is a contact force greater than 15 N; the second condition is the maximum grip torque exerted on the target-holder: the chosen gripper in fact can provide a 50 N force for each jaw, which corresponds to a 20.7 N force to each wedge in use. It has also to be considered the friction between the two copper surfaces to couple (that shall be lapped to obtain a surface roughness of 0.25 µm) and between the steel spring and the copper wedge.

Fig. 5. The bayonet coupling between target-holder TH and the cold finger CF of the cryocooler. After lifting of CF to contact the base of TH, a 25◦ rotation engages one couple of TH wedges with the flexural springs S, bringing them in contact with the limiters on the positioning cup PC.

From the free-body diagram of Fig. 6, it is possible to derive the mathematical model of coupling between one spring and the wedge, expressed by Eq. (1):     F · l2 F · l2 K = F · sin + (µ1 + µ2 ) · F · cos (1) 2·E·I 2·E·I Where: K : is the maximum force on a single wedge given by the gripper (20.7 N); F : is the normal force acting on the spring (greater than 7.5 N by requests); L : is the length of the flexible part of the spring; E : is the elastic modulus of the spring material; I : is the moment of inertia of the spring cross section;

Target Manipulation in Nuclear Physics Experiment with Ion Beams

μ1 · F spring

FO θ

K

θ

F

FV

wedge

541

θ μ2 · FV

Fig. 6. Model of coupling between pre-load spring and TH wedge.

µ1 : is the friction coefficient between the wedge and the spring; µ2 : is the friction coefficient between the target holder and the cold finger. A special problem to consider is the operation at a cryogenic temperature around 40 K. The materials in the TH-CF joint, in fact, could undergo important variations in the characteristics: this could represent a problem in particular for the flexing springs, which are typically made of harmonic steel. While the variation of the Young modulus, which increases slightly at such a low temperature, is not a problem, more important is the considerable embrittlement that these materials undergo at such a low temperature [5]. This has led to the use of AISI 316 stainless steel, which maintains its characteristics constant even in cryogenic environments [6]. The model of Eq. (1) has been implemented in Matlab with an iterative algorithm, which led to an optimal solution for the system characteristics. The calculation considering springs made of AISI 316 stainless steel and a contact force increased to 25 N corresponds to spring cross section of 0.6 × 4 mm and to 0.5 mm of spring deformation imposed by the wedge, the angle of the wedge is 2.8◦ . This solution limits to a minimum the quantity of material inside the chamber during the use of the beam, that is convenient to reduce the activation of material inside the chamber. Another important design feature is the precision in angular alignment of the TH to the ion beam, determined by the geometric tolerances of the lower cup of the system that define the position of the limiters: the present solution limits the alignment error to a maximum of 0.3◦ , that is lower than the requested value.

4

The Storage System

The storage system is devoted to provide a given number of new target-holders for performing the tests as well as to collect, after the experiment, the exhausted and activated THs that must be handled and stored with particular precautions. The Fig. 7 shows an image of the system, which is completely automated in order to control from remote the manipulation of the target-holders. A frame SF, which is in turn mounted on the triangular frame TF of the manipulator,

542

D. Sartirana et al.

supports the device. The latter is composed by a main plate SP, provided with a given number of slots, guided by a slide rail (Misumi mod. BJKSW-H20L590) and driven by a linear actuator SA (IAI mod. RCA-SA6D-A-30-12-350). The movements to position the target-holders TH on the plate are: i) a vertical stroke of 5 mm, regulated by the limiters L and performed by two membrane actuators MC (Festo EV-20x75+DP) positioned under the sustain plate; and ii) a 90◦ rotation around the wrist pitch axis.

Fig. 7. Image of the automatic storage system.

5

Conclusion

A whole automatic manipulation system for targets that must be used in a nuclear physics experiment has been completely designed. The design had to cope with very particular conditions, which determined the search of non-usual solutions. The main issues are the operating temperature, around 40 K, the extreme precision and care required in handling the target of the experiment, and the activation generated during the tests. The low operating temperature called for special attention to the materials of some elements, like the pre-load springs devoted to provide appropriate contact force between the target-holder and the cryocooler. The precision and care requested in the manipulation are guaranteed by adopting accurate machining of all structural parts and by special elements for adjusting and regulating the motion of the several degrees of freedom. To face the activation problem, it has been necessary to evaluate in all elements of the system the presence of materials possibly subjected to this phenomenon, and study their placement at safe distance from the source of radiation.

Target Manipulation in Nuclear Physics Experiment with Ion Beams

543

References 1. Cappuzzello, F., et al.: The NUMEN project: NUclear Matrix Elements for Neutrinoless double beta decay. Eur. Phys. J. A 54, 72 (2018) 2. Capirossi, V., et al.: Study, fabrication and test of a special cooling system for targets submitted to intense ion beams. Nuclear Inst. Methods Phys. Res. A 954, 161122 (2018) 3. Xian, Y., et al.: Re-estimation of thermal contact resistance considering near-field thermal radiation effect. Appl. Therm. Eng. 157, 113601 (2019) 4. Beynel, P., Maier, P., Sch¨ onbacher, H.: Compilation of radiation damage test data. PART III: Materials used around high-energy accelerators (1982) 5. Roesler, J., Harders, H., Baeker, M.: Mechanical Behaviour of Engineering Materials. Springer, Heidelberg (2007) 6. Ekin, J.W.: Experimental Techniques for low temperature Measurements. Oxford University Press, Oxford (2006)

Robot Control and Vision

SLIP-Based Concept of Combined Limb and Body Control of Force-Driven Robots Patrick Vonwirth1(B) , Atabak Nejadfard1 , Krzysztof Mianowski2 , and Karsten Berns1 1

Department of Computer Science, Technische Universit¨ at Kaiserslautern, 67663 Kaiserslautern, Germany {vonwirth,nejadfard,berns}@cs.uni-kl.de 2 Faculty of Power and Aeronautical Engineering, Warsaw University of Technology, 00-665 Warsaw, Poland [email protected] Abstract. Many different approaches in hard- and software have been investigated to achieve bipedal locomotion. They can be partitioned into two separate classes: Mathematical approaches, offering provable stability, and bio-inspired ones, reproducing natural observations. The paper at hand presents a new concept to overcome this separation. By generalizing several SLIP variations (Spring Loaded Inverted Pendulum), a new type of hardware abstraction, the so-called Central Mass Model (CMM), is introduced. The CMM is designed to directly support the execution of bio-inspired control approaches, while its physical simplicity still allows for mathematical proofs. A controller, implementing the CMM abstraction on a force-driven robot, is derived and described in detail for the bipedal robot Carl. Keywords: Bipedal robot

1

· Force control · Inverted pendulum

Introduction

Both, artificial bipedal locomotion control, as well as observed human walking modeling has been studied for decades and centuries. Introducing several gait phases and events, biological locomotion studies nowadays offer a deep, structured insight into human-like gait. Joint motions, related torques, power emittance or absorption and hence individual control purposes during the gait phases have been discovered. Simplified mechanical walking models, mostly based on the so-called Spring Loaded Inverted Pendulum (SLIP) model [2,5], have been introduced to describe that gait. On the robotics side, a lot of different control concepts have been developed independently and unrelated to each other and to the SLIP analyses. Against their natural variety, all the robotic approaches can be partitioned into two main classes: Mathematical and Biological approaches. The focus of biological approaches lies in the similarity to nature, whereas mathematical concepts c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 547–556, 2020. https://doi.org/10.1007/978-3-030-48989-2_58

548

P. Vonwirth et al.

mainly focus on proofed stability of efficiency. A detailed overview of human-like locomotion and their state of robotic implementation has been summarized by Torricelli et al. in [16]. On the mathematical side, the concept of Divergent Component of Motion (DCM) [4] approaches the balance of an upright body on two legs during locomotion. Having its main focus on provable stability, the DCM control does not address efficiency or human likeness. Especially addressing efficiency, trajectory optimization methods such as Hybrid Zero Dynamics (HZD) [17] have been introduced. However, since their pre-planned trajectories cannot handle disturbances or model inaccuracies, it is required to combine such an approach with inefficient online-adaptation strategies [10]. Biologically motivated methods focus on the reproduction of human-like locomotion. The most popular concept is to break down the complex whole-body control to a set of several simple and independent local controllers. A possible implementation is to implement one small feedback controller per artificial muscle [14]. However, without having a mathematical model, it is neither possible to prove anything, nor to reason about efficiency or other characteristics. Similar to HZD, there are also biological approaches that rely on predefined control patterns [6,13]. In contrast to HZD, these patterns locally define dynamic control values, e.g. a force. The control’s flexibility to counteract disturbances arises from the passive dynamics of the robot itself. The Bio-inspired Behavior-Based Bipedal Locomotion Control (B4lc), introduced by Luksch et al. [8] is a hybrid of all. It contains predefined control patterns, as well as local reflexes, and partially relies on simplified mathematical models, e.g. Inverted-Pendulum based balance strategies [19]. Unfortunately, since this mixture is missing any clear concept, its complexity is not overseeable and hence unmanageable. Except for B4lc, all these control approaches do not consider the SLIP based walking models, derived from natural observations, although it is known that they are able to generate major natural gait characteristics. Furthermore, these simplified models offer the possibility to mathematically proof stability criteria. E.g. Stability regions, defining where to apply different bio-inspired balance strategies [15]. Using a SLIP-based robot abstraction, therefore, seems promising to fill the gap between the two – mathematical and bio-inspired – classes of control methods. Within the paper at hand, a new concept of robot abstraction, the Central Mass Model (CMM) is introduced in Sect. 2. By generalizing existing SLIP variations, the CMM concept, on the one hand, offers direct support to execute bio-inspired control approaches. On the other hand, its physical simplicity still allows for mathematical proofs. Based on the CMM abstraction, new bio-inspired control strategies can be developed and executed that fulfill both, human-like behavior and theoretically provable stability at the same time. For example, biological, upright balancing using the hip- and ankle strategies as studies in depth within [15]. An appropriate controller to abstract a force-driven robot to

Limb and Body Control

549

a CMM is shown in Sect. 3. At the example of the planar, bipedal robot Carl [11], Sect. 4 gives a concrete implementation of the derived controller. Finally, Sect. 5 concludes the introduced concept together with a short outlook on future and ongoing work.

2

Central Mass Model

Bringing together the benefits of mathematical and biological inspired control approaches, a simple, but general, robot abstraction model is required. It has to supports both, natural control strategies as well as mathematical analyses. From natural gait studies, it is known that the SLIP model is able to reproduce the major human gait characteristics [2,5]. Furthermore, its simple dynamics allow to develop and prove stable, natural control strategies. For example, bio-inspired, upright balancing, using the so-called ankle- and hip-strategies [15,19]. With a variety of different extensions to the SLIP model, several individually characteristic details have been improved. For modeling the double support phase in walking, the spring-leg of the SLIP model is doubled by Geyer et al. [5]. Adding a flat foot that offers the opportunity to create an ankle torque, Antoniak et al. are able to not only reproduce the vertical Ground Reaction Force (GRF), but also its horizontal component [1]. With the introduction of a trunk, the single inverted pendulum becomes a double inverted pendulum. Using this extension, the problem of natural upper body balance can be addressed [7,12]. Since all these SLIP variations address different, but all crucial, gait properties, the Central Mass Model is defined as a generalization of all of them: Definition 1. The Central Mass Model (CMM) is a robot abstraction model, offering direct support to execute bio-inspired, SLIP-based control approaches. It represents a robot as a single Central Mass (CM), surrounded by several, arbitrary connected masses, the limb end effectors. These end effectors represent all the robot’s extremities like feet, hands, or a head. All the limb end effectors, as well as the central mass, are force-/torque-controlled in their position and orientation with respect to the central mass. All control values are individually weighted by their importance at the present time. An underlying, hardware-specific CMM controller is used to abstract and drive the actual robot in accordance with the user-defined, weighted CMM control commands. The torque controls of the limbs’ end effectors abstract the SLIP ankle joint. Hence, horizontal GRF control [1], as well as ankle balancing strategies [15,19] are directly supported. Instead of the polar representation of hip torque and spring leg force, the CMM is using the equivalent Cartesian representation. The trunk and the spring leg control are therefore combined within a limb’s end effector force control. Since forces and torques can only be applied between two masses, it is a physical requirement to define some end effector mass per limb. In case of a leg, during the stance face in walking, this limb mass is negligible. However, for modeling the swing phase of the same leg, it is mandatory to achieve

550

P. Vonwirth et al.

a physically valid motion. The so-called Central Mass (CM) is not required to represent the Center of Mass. Its main purpose is to define an arbitrary, but shared anchor point for all limbs. This way, the dynamic interaction between all the limbs is covered within the model’s single center. With the ability to specify control values for the central mass, the limbs’ dynamic interaction can be specified. Unfortunately, due to this additional control, the system becomes over-specified. Hence, the underlying CMM controller has to find valid limb controls that best fit all user-defined limb and central mass controls according to their appropriately defined weights. This way, the desired central mass forces and the torque are automatically distributed to unused limbs. Physical limitations (e.g. maximum force of an actuator), additional passive elements, and over-actuation can be handled implicitly. Virtual Pivot Point control [3,7] can be realized by simply transforming the desired force at the VP to the resulting force and torque at the central mass. Furthermore, due to the shared control coordinate frame of all limbs, whole-body control strategies can be implemented easily. E.g. grip control by impedance rendering between (hand) end effectors as proposed by the so-called Virtual Linkage Model [18].

3

CMM Controller

The CMM controller is required to abstract the physical platform to the simplified, but general Central Mass Model. Although there is no general rule or restriction on how to realize such a CMM controller, the following section introduces one possible concept to be applied to force-driven robots. A common solution to best fit an over-specified system is to find the solution that minimizes its weighted squared errors w · e. The minimization problem can be formulated as follows:   (1) min eT · diag(w) · e F

e = (L · F − D) lb ≤ F ≤ ub

(2)

0≤w where F = [F1 , · · · , Fn ]T is the result vector of all actuator forces, elementwise bounded between lb and ub. D = [D L1 , · · · , D Lm , D CM ]T is a vector of all desired force and torque controls for all limbs Lx and the central mass CM . Appropriately, L is the transformation matrix that calculates the control values out of the actuator forces F . diag(w) is a diagonal matrix, containing the non-negative control weights related to the desired controls. The desired control vector D together with the weighting vector w are the abstract, CMM-related inputs to the controller. The result force vector F is the platform-specific output to the real robot.

Limb and Body Control

551

By defining F  := [F T , −1]T being the force vector F extended by the constant −1, Eq. 2 can be rewritten to a single matrix multiplication.   e = L D · F (3) Using this, the initial minimization problem, stated in Eq. 1, can be reformulated:   T  min F (4) · H · F  F

 T   H = L D · diag(w) · L D   ub lb  ≤F ≤ −1 −1

(5)

0≤w By definition, H is a quadratic and symmetric matrix. Furthermore, for all nonzero error weights w > 0, the H-matrix is by definition positive semi-definite. However, with every zero weight w = 0, H might lose a rank and hence the minimization problem its solvability. Unfortunately, zero weights are a desired scenario. They indicate unused limbs (i.e. single unused limb control properties) that are intended to be used for the shared central mass control. To assure the Hmatrix to be positive definite, and hence to always guarantee a unique solution, the definition of H in Eq. 5 has to be extended to  T   H := L D · diag(w) · L D + diag() (6) >0 where diag() is a constant diagonal matrix of very small, positive values . Obviously, this extension comes at the cost of introducing errors to the result forces F . Related to the size of , the extended H-matrix is pushing all result forces independently towards zero. However, for very small , these errors become negligible for the final physical behavior of the entire system.

4

CMM Control Applied to Carl

The planar, two-legged robot Carl has been introduced as a single-leg version by Sch¨ utz et al. [11]. In the two-legged version, the single leg is duplicated without changing its kinematic structure. In Fig. 1a, a simulation model of the bipedal version of Carl is pictured. At first, the CMM abstraction of Carl has to be defined. The Central Mass is representing the trunk, the two legs are modeled as two surrounding masses. Figure 1b gives a graphical sketch of the CMM abstraction of Carl. Due to the planarity of Carl, the control vectors for both limbs and the trunk (i.e. the central mass) are 3-dimensional. They each contain a 2D-Force combined with a

552

P. Vonwirth et al. Y

Y θ

θ

X Central Mass

X

−θleg −θHip

−θKnee

dleg Limb

Limb (a) Simulation

lseg lleg

(b) CMM Abstraction

(c) Leg Kinematic

Fig. 1. The planar, bipedal robot Carl [11]. On the left side, a simulation model of Carl is pictured in figure (a). In the middle, figure (b) sketches the planar CMM abstraction of Carl as side view. The kinematic definitions of one leg are described in figure (c). Please note, that the angle arrows of the three leg angles are pointing in negative direction. T

one-dimensional torque Dx := [Fx , Fy , τ ] . Since Carl is driven by 5 actuators per leg, the CMM controller’s result vector is a 10-dimensional concatenation of the each five actuator forces per leg F := [F L , F R ]. The error Eq. 3 of the CMM controller for Carl is hence ⎤ ⎤ ⎡ ⎤ ⎡ ⎡ 0 DL LL FL eL ⎣ eR ⎦ = ⎣ 0 LR D R ⎦ (7) · ⎣F R ⎦ eCM LCML LCMR D CM 11×9 −1 On the left side, the error vector e is a composition of the individual error vectors eL/R/CM for the left leg, the right leg and the trunk respectively. Equivalently, the desired control vector D is constructed from D L/R/CM . The transformation matrix L is decomposed into four separated transformation matrices. LL and LR calculate the individual control values of the left and the right leg, LCML and LCMR calculate the   controls of the central mass, i.e. the trunk. From the structure of the L D -matrix, it is easy to see that both legs are solely coupled by the control of the central mass. Due to the identical kinematic structure of both legs, all the inner L... matrices are of the same structure for both legs. Representative for both legs, the kinematic structure and all related definitions of one leg are shown in Fig. 1c. The inner leg control matrices LL/R , can be calculated in three steps: T LL/R = R(θleg ) · J−T leg · Am

(8)

Limb and Body Control

553

where R(θleg ) is a 3 × 3 rotation matrix around the 3rd dimension. ATm is the transposed 3 × 5 lever arm Jacobian, calculating the torques of the three leg joints (hip, knee and ankle) out of the actuator forces τ = ATm · F . A detailed introduction of the lever arm matrix Am is given by Nejadfard et al. in [9]. The transposed inverse of the leg Jacobian matrix is derived from the kinematic as follows: ⎡ ⎤ ⎤ ⎤ ⎡ ⎡ ˙ x˙ θHip 0 dleg/2 0 ⎣y˙ ⎦ = R(θleg ) · ⎣lleg lleg/2 0⎦ (9) · ⎣ θ˙Knee ⎦ ˙θ ˙Ankle 1 1 1 J θ leg ⎡ ⎡ ⎤ − 1/dleg 2/dleg Fx ⎣Fy ⎦ = R(θleg ) · ⎣ 1/lleg 0 τ 0 0

− 1/dleg

⎤

− 1/lleg ⎦

1

J−T leg

⎤ τHip · ⎣ τKnee ⎦ τAnkle ⎡

(10)

with θleg = θHip +

θKnee 2

θKnee lleg = 2 · lseg · cos 2  2 2 dleg = (2 · lseg ) − lleg

(11)

 >0

(12)

≥0

(13)

where lseg is the length of the equally long thigh and shank. The presented, non-standard computation of the Jacobian matrix has been chosen due to its singularity. If the knee is fully stretched, the virtual leg width, i.e. dleg , becomes zero. To always assure a valid H-matrix, a very small  > 0 can be defined, rewriting Eq. 13 to    2 2 dleg := max , (2 · lseg ) − lleg ≥>0 (14) From the construction of J−T leg in Eq. 10, it can be seen that the singularity only effects the force acting in the direction of lleg . At the point of singularity, this force becomes infinite for every joint torque and hence, zero torque can create any demanded force. Furthermore, all miscalculated forces can be compensated by the knee torque that is acting solely in this singular direction. Hence, the error that is introduced by Eq. 14 only affects the knee torque and is always close to zero. For very small values of , although mathematically existing, the error becomes negligible for the overall physical behavior of the entire system. The trunk control matrices LCML/R are computed equivalently. As a fundamental proof of concept, the basic human-like balancing strategies [19] have been tested on the simulation model of Carl (see Fig. 1a). On one leg, the hip- and ankle-strategies have been executed while the other leg has been kept passive. With this setup, the robot was able to balance on a rotating ground

554

P. Vonwirth et al.

plate, executing a sinusoidal change of orientation at 5 Hz with an amplitude of at max 2.5◦ . A snapshot showing 10 s of the forces of all five muscle while balancing is given in Fig. 2. The 5 Hz oscillation of the ground plate is clearly visible within the oscillations of the muscle forces.

Fig. 2. A 10 s-snapshot of one leg’s muscle activities of Carl while balancing on a shaking ground plate. The plate’s orientation is executing a sinusoidal change of orientation at 5 Hz with an amplitude of 2.5◦ .

5

Conclusion and Outlook

The paper at hand proposes a new concept to abstract an arbitrary robot to a simple, standardized model, the so-called Central Mass Model (CMM). Following the well-known concept of SLIP, the CMM is designed to directly support the execution of bio-inspired, SLIP-based control approaches. At the same time, its physical simplicity allows for mathematical proofs on such control algorithms. To validate the functionality of the proposed concept, appropriate experiments have to be performed. For example, bio-inspired balancing strategies as described in [15] can deliver a wide range of proper control results. The static ankle- and the dynamic hip-strategy cover most of the CMM control opportunities. Furthermore, balancing with bent and fully stretched knees can especially focus on the problematic singular position. Beyond the fundamental proof of concept, further experiments have to be made to get a more detailed overview of the capabilities and the limitations of the proposed concept. The human-like balancing strategy is planned to be extended to a complete, bipedal balancing system and to be tested not only in simulation but on the real robot hardware of Carl. Afterward, other existing bio-inspired, SLIP-based motions like hopping, running, and walking can be evaluated.

References 1. Antoniak, G., Biswas, T., Cortes, N., Sikdar, S., Chun, C., Bhandawat, V.: Springloaded inverted pendulum goes through two contraction-extension cycles during the single-support phase of walking. Biol. Open 8(6), bio043695 (2019). https:// doi.org/10.1242/bio.043695

Limb and Body Control

555

2. Blickhan, R.: The spring-mass model for running and hopping. J. Biomech. 22(11– 12), 1217–1227 (1989) ¨ Badri-Spr¨ 3. Drama, O., owitz, A.: Trunk Pitch Oscillations for Joint Load Redistribution in Humans and Humanoid Robots. arXiv:1909.03687 [cs], September 2019 4. Englsberger, J., Ott, C., Albu-Sch¨ affer, A.: Three-dimensional bipedal walking control using divergent component of motion. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2600–2607. IEEE (2013) 5. Geyer, H., Seyfarth, A., Blickhan, R.: Compliant leg behaviour explains basic dynamics of walking and running. Proc. Roy. Soc. B: Biol. Sci. 273(1603), 2861– 2867 (2006). https://doi.org/10.1098/rspb.2006.3637 6. Kawakami, T., Hosoda, K.: Bipedal walking with oblique mid-foot joint in foot. In: 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 535–540. IEEE, December 2015 7. Lee, J., Vu, M.N., Oh, Y.: A control method for bipedal trunk spring loaded inverted pendulum model. In: The Thirteenth International Conference on Autonomic and Autonomous Systems, pp. 24–29 (2017) 8. Luksch, T., Berns, K.: Control of bipedal walking exploiting postural reflexes and passive dynamics. In: IEEE International Conference on Applied Bionics and Biomechanics (ICABB). Citeseer (2010) 9. Nejadfard, A., Sch¨ utz, S., Vonwirth, P., Mianowski, K., Karsten, B.: Moment arm analysis of the biarticular actuators in compliant robotic leg CARL. In: Conference on Biomimetic and Biohybrid Systems, pp. 348–360. Springer, Heidelberg, July 2018 10. Reher, J., Cousineau, E.A., Hereid, A., Hubicki, C.M., Ames, A.D.: Realizing dynamic and efficient bipedal locomotion on the humanoid robot DURUS. In: 2016 IEEE International Conference on Robotics and Automation, pp. 1794–1801, May 2016. https://doi.org/10.1109/ICRA.2016.7487325 11. Sch¨ utz, S., Nejadfard, A., Mianowski, K., Vonwirth, P., Berns, K.: CARL – a compliant robotic leg featuring mono- and biarticular actuation. In: IEEE-RAS International Conference on Humanoid Robots (2017) 12. Sharbafi, M.A., Maufroy, C., Ahmadabadi, M.N., Yazdanpanah, M.J., Seyfarth, A.: Robust hopping based on virtual pendulum posture control. Bioinspir. Biomim. 8(3), 036002 (2013). https://doi.org/10.1088/1748-3182/8/3/036002 13. Sharbafi, M.A., Rashty, A.M.N., Rode, C., Seyfarth, A.: Reconstruction of human swing leg motion with passive biarticular muscle models. Hum. Mov. Sci. 52, 96– 107 (2017) 14. Song, S., Geyer, H.: A neural circuitry that emphasizes spinal feedback generates diverse behaviours of human locomotion. J. Physiol. 593(16), 3493–3511 (2015). https://doi.org/10.1113/JP270228 15. Stephens, B.: Humanoid push recovery. In: 2007 7th IEEE-RAS International Conference on Humanoid Robots, November 2007. https://doi.org/10.1109/ICHR. 2007.4813931 16. Torricelli, D., Gonzalez, J., Weckx, M., Jim´enez-Fabi´ an, R., Vanderborght, B., Sartori, M., Dosen, S., Farina, D., Lefeber, D., Pons, J.L.: Human-like compliant locomotion: state of the art of robotic implementations. Bioinspir. Biomim. 11(5), 051002 (2016). https://doi.org/10.1088/1748-3190/11/5/051002 17. Westervelt, E., Grizzle, J., Koditschek, D.: Hybrid zero dynamics of planar biped walkers. IEEE Trans. Autom. Control 48(1), 42–56 (2003). https://doi.org/10. 1109/TAC.2002.806653

556

P. Vonwirth et al.

18. Williams, D., Khatib, O.: The virtual linkage: a model for internal forces in multigrasp manipulation. In: Proceedings IEEE International Conference on Robotics and Automation, vol. 1, pp. 1025–1030 (1993). https://doi.org/10.1109/ROBOT. 1993.292110 19. Zhao, J., Liu, Q., Sch¨ utz, S., Berns, K.: Experimental verification of an approach for disturbance estimation and compensation on a simulated biped during perturbed stance. In: IEEE International Conference on Robotics and Automation (ICRA 2014), Hongkong, China (2014)

Wizard of Botz: A Novel Approach to the Wizard of Oz Experiment Kim W¨ olfel(B) and Dominik Henrich Chair for Applied Computer Science III, University of Bayreuth, 95447 Bayreuth, Germany {kim.woelfel,dominik.henrich}@uni-bayreuth.de http://robotics.uni-bayreuth.de/ Abstract. Speech-based robot instruction is a promising field in private households and in small and medium-sized enterprises. It facilitates the use of robot systems for experts as well as non-experts, especially, when the user executes other tasks. To gain information on how users instruct robots or perceive robot motions, wizard of oz experiments are often used. While this is totally fine for object manipulation tasks, force based motions are hard to simulate. We introduce a new kind of the wizard of oz setup, where the robot is not controlled through a controller, but through another robot arm. By doing this, force feedback is available for the wizard and controlling the robot arm is easily possible for nonexperts. We tested this setup in a user study. The results are presented and compared to previous studies. Keywords: Human-robot interfaces

1

· Force control · Wizard of Oz

Introduction

One long term goal in current robotic research is the development of robot systems, which have approximately the same cognitive, communicational, and handling abilities as humans. As part of this ongoing development, application domains for robot systems shall be expanded, from industrial settings with separated working cells, fixed object positions, and preprogrammed motions towards a flexible usage in small or medium-sized enterprises or private households. One of the main challenges is to allow an intuitive communication between the user and the system. For example, by controlling the robot by speech [2,12], gestures [7,13] or haptic control [9]. One well used approach for testing approaches for an intuitive control is the Wizard of Oz (WoZ) experiment [10]. In this approach, the systems are evaluated, but letting a human imitate the system behaviour. This saves programming time and helps finding problems of a concept in an early stage. Controlling the robot is often done by a gaming controller or a graphical user interface [10]. While this is totally fine for object manipulation tasks, it is hard to apply it for force based robot motions, where the appropriate force is not already known. For example, if the goal of a study is to gather information about uncertain force parameters in specific situations or materials. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 557–564, 2020. https://doi.org/10.1007/978-3-030-48989-2_59

558

K. W¨ olfel and D. Henrich

Fig. 1. Example for performing a robot motion based on a spoken instruction using our Wizard of Botz setup.

We present a WoZ setup, which facilitates such studies: Wizard of Botz (WoB) (see Fig. 1). In WoB the robot arm in front of the participant is not controlled via a graphical user interface, but by a second identical robot arm. This enables the wizard to generate more natural movements and to have more control of the abilities of the controlled robot arm. Thus, the contribution of this paper is a WoZ setup which contains force feedback and also allows a robot control by experts as well as non-experts. The remainder of this paper is organized as follows: Sect. 2 contains a brief summary of related work in the area of WoZ experiments. The setup of our Wizard of Botz study is presented in Sect. 3 and a user study based on the introduced setup along with its result is presented and classified in Sect. 4. Finally, Sect. 5 contains a summary of the scientific gain achieved by our approach and an outlook on future work.

2

Related Work

The first part of this section contains a brief overview of WoZ studies. The second part of this section presents criteria from the literature that are used to classify WoZ experiments and their setups. When it comes to human robot interaction, WoZ studies enjoy great popularity in robotics and other research areas, because it allows researches to evaluate their systems in an early stage, without enormous implementation and calibration effort. The main idea is to simulate a feature of an artificial system to be able to gain information on how users interact with it or what users expect of it. If an interaction takes place, a human being controls the system while the user thinks the system is already fully implemented. Over the years many extensions of this approach were created. An extensive evaluation of 54 papers dealing with WoZ experiments and its extensions can be found in [10]. Among others the Wizard Control Type is presented to classify different WoZ studies. The Wizard Control Type describes, what component, verbal, non-verbal, navigation, manipulation, sensing, or mapping of the system is simulated by

Wizard of Botz: A Novel Approach to the Wizard of Oz Experiment

559

the wizard. In the last years, most of the studies examined the verbal interaction component, closely followed by non-verbal modalities like gaze, pointing, or nodding. Also navigation tasks have often been simulated, which was perhaps down to the fact that it is easy to manage. Less examined components are manipulation [8], sensing [14], or mapping [3]. In [10] the following criteria for a classification of WoZ studies are presented: Kelley Criteria [6], Fraser and Gilbert Criteria [4], Green Criteria [5] and Steinfeld Criteria [11]. The Kelley Criteria (C1) highlights, in which extend the wizard is in fully control of the robot. Viewing a WoZ study as an iterative process, the wizard should gradually phase out and autonomy of the system should phase in. The Fraser and Gilbert Criteria (C2) have been introduced, to specify pre-conditions that guarantee the usefulness of a WoZ study. It must be possible to simulate the future system, given human limitations, it must be possible to specify the future system’s behavior and it must be possible to make the simulation convincing. The Green Criteria (C3) are introduced to improve the process of a WoZ study by specifying user instructions, which tell the user how he has to interact with the system; behavior hypotheses of the user, which comply with the expectations of the interviewer; and robot behaviors, which define how a robot should react to a human action. The Steinfeld criteria (C4) finally classifies the kind of WoZ study depending on the more human- or more robot centric property. The different types are Wizard with Oz (use of real technology in simulated or controlled environments), Wizard and Oz (real technology in a real environment), Oz of Wizard (the user is simulated) and Oz with Wizard (the behavior of a system is measured, not the participants behavior).

3

Wizard of Botz

In this section we introduce WoB. The first part (Sect. 3.1) deals with the setup and in the second part (Sect. 3.2) strengths and limitations are mentioned. 3.1

Set Up

In the presented setup the wizard control type is manipulation, because it emerged from the idea to be gain information on how a user instructs a system that is only capable of executing motions and not able to give acoustic or non-verbal feedback. An extension to a verbal feedback are planned for future research. The main setup is similar is build as follows: The participant and the wizard are located at different places in a way that the participant is unable to recognize the wizard or at least, that the wizard robot is guided by a human. In our case, the wizard is located in the same room as the participant (see Fig. 2), but separated via a curtain. The most important component of this setup are robot arms that can be controlled in a so called gravity compensation mode, which means, that a user is able to move the robot arm in space by directly guiding it.

560

K. W¨ olfel and D. Henrich

Fig. 2. WoB study setup: The wizard with two monitors, a robot arm and a headset, and the participant with two webcams and the robot.

Here, we are working with two KUKA LWR IV robot arms [1] (R1 and R2). R1 is equipped with a Schunk two finger gripper, and R2 with a robotiq three finger gripper. In that way the wizard is able to control R1 by guiding R2. The joint positions Θ2 ∈ IR7 of R2 are sent via a local network to the R1 controller, which are there set as goal joint positions Θ1 ∈ IR7 for R1, resulting in R1 imitating R2s motion with a slight delay. To give the wizard insight into the participants workspace two Logitech HD webcams, including microphones, are installed at the participants workspace which stream their recordings to two monitors and the headset located in the wizards chamber. With this information the wizard is able to recognize speech, gestures and even haptic interactions between the participant and R1. R1 is located in the middle of a room and is able to manipulate objects on a table in front of it. The participant is sitting at the table as seen in Fig. 2. This enables the wizard to move R2 according to the user instructions, but it looks as if the system interpreted the instruction and transformed it into a robot motion. 3.2

Setup Strengths and Limitations

The main strength of this setup is the simplicity of controlling the wizard robot. No time consuming introduction is needed to move the robot and it is even possible to restrict the wizards visual feedback. In that way it is also possible to simulate robot systems which got no visual sensors. The simplicity also enables experiments, where the wizard as well as the user are participants in the same user study. This can spare time, because in some cases two studies can be merged into a single one.

Wizard of Botz: A Novel Approach to the Wizard of Oz Experiment

561

Fig. 3. User study tasks: Coining, Painting, Constructing, and Shoving.

The main limitation of WoZ is, that it is not possible to move the robot arm with high accuracy and repeatability as a robot program, which is easily possible by using a controller or a graphical user interface. Therefore WoZ could be improved by adding a hybrid approach, allowing the execution of preprogrammed motions if necessary.

4

User Study

To evaluate WoB setup we performed a user study, where participants where asked to instruct a robot to manipulate objects in a way, where different kinds of forces for the same verbs are needed. First, the experiment procedure is presented, then a classification of WoZ in this use case is performed and the user study results are discussing regarding the usability of WoZ for force based motions. 4.1

Experiment Procedure

The experiment is divided into two main phases: introduction and instruction. In the introduction phase the participants are given a short insight into the research field of speech based robot programming and its application in the future. Demographic data and data concerning the personal experience in the topics robotics, programming and speech based control together with the controlled devices are captured. The following instruction phase is explained in a written form and further explained verbally. In the instruction phase the participants are told to instruct the robot arm to perform the following four tasks (see Fig. 3) by only using their voice: coining in modelling clay (T1), painting with a brush (T2), constructing with building blocks (T3), and shoving glasses to a goal position (T4). All tasks are again divided into two phases. In the first phase the desired output of the task is shown in form of modelling clay coined with different forces in T1, a sheet of paper containing painted lines with a different thickness T2, a house structure build with three building blocks in T3 and three differently filled glasses located on goal positions in T4. The participants are then told to generate the desired output via instructing the robot arm verbally and starting from an initial setup in form of: Three not coined pieces of modelling clay and a coining tool attached to the robot arm in

562

K. W¨ olfel and D. Henrich

T1, a blank sheet of paper, paint and a brush attached to the robot arm in T2, the building blocks standing separated on the table in T3 and the three glasses standing on start position in T4. At the beginning of the coining task a modelling clay is shown, which has been coined with different depths. The participants are then asked to instruct the robot to reproduce this coins on other clays. After each task the participants had to rate the process by rating these statements on a scale from 1 (strong disagree) to 7 (strong agree) in a questionnaire: Q1 Q2 Q3 Q4 Q5 Q6 Q7

It was easy to program the robot The robot behaved the way i imagined it I liked the possibility to correct the robot A verbal feedback would have been nice The robots motions did not feel unfamiliar I can imagine controlling a robot like this in the future A speech based control sufficient

After filling all questionnaires the participants are able to give some general feedback for the user study and to explain their behaviour during the experiment. 7

3

6

2

5

1

4 0 3

-1

2

-2

1 0

-3 roboƟcs

programming

speech control

Q1

Q2

Q3

Q4

Q5

Q6

Q7

Fig. 4. Results of the user study. Box plots for the experience values (left) and the questionnaire answers (right)

4.2

Results

Altogether n = 15 users, 6 male and 9 female, participated in the user study with an average age of 28. Their mean experience in the topics robotics, programming and speech control are shown in Fig. 4. Reaching from 1 (non-expert) to 7 (expert) the mean experience regarding programming was 3.3, regarding robotics was 2.5, and regarding speech based control 2.4. While 27% of the participants never controlled a machine before, the majority of the remaining controlled at least their smartphone.

Wizard of Botz: A Novel Approach to the Wizard of Oz Experiment

563

During the instruction phase most of our observations overlap with the results described in [8], where also a WoZ study was performed, with the difference that the robot arm was controlled via a graphical user interface. To name a few, users tend to use objects in the workspace as references instead of pure left or right commands, some users commanded the robot arm and gripper like a human arm or hand and participants provided feedback if an execution went bad or well. Two additional behavior where also observed in our study. Users switch between using each instruction separately and depending on each other. Another behavior observed is the paraphrasing for force based parameters like soft or hard with distance parameters like deep or slightly deep. We also observed that experts as well as non-experts in robotics did not notice that the robot arm was not moved by a program, but by a human being. This confirms our assumption that our approach can be used for imitating force based robot motions, while preserving the illusion that the motion is generated by a robot program. 4.3

Classification

Regarding C1, we would not classify our user study as a classic WoZ study, while it is not iterative at all. Nevertheless, phasing in system autonomy, like preprogrammed motion segments, would be possible in future scenarios. Fully met are the pre-conditions contained in C2. Following the user study results we are able to simulate the future system convincingly. The wizards recognition ability was not constrained and was not allowed to produce errors intentionally. The wizards training is vanishingly low, because the wizard only reacted to predefined verbs in the same way or did not move at all. The criteria C3 are partly met by our system. User instructions were not specified, because the user should talk to the system as natural as possible. Some user behaviors were predicted and corresponding robot behaviors were set. Regarding C4 our study can be classified as a Wizard with Oz study while it took place in our robotics laboratory.

5

Conclusion and Future Work

We present Wizard of Botz, a variant of the Wizard of Oz user study type, which facilitates the consideration of force feedback during robot motions and enables even non-experts to perform as a wizard. The consideration of a force feedback extends the set of motions one is able to imitate and the control via guiding the robot allows a more flexible integration of the robot arm. By guiding the wizard robot by hand it is even possible to limit the perception of the wizard during the experiment. In that way insights in how users perform force based motions even if no visual feedback is present, is possible. We classified our experiment based on a user study setup by means of state of the art classifiers and discussed strengths and limitations compared to other Wizard of Oz variants.

564

K. W¨ olfel and D. Henrich

In future work we will investigate the possibilities to combine our approach with preprogrammed motion segments or facilitating the wizard control by limiting the robots working space and extend the information given to the wizard. We are also looking forward to test this approach for other communication channels like gestures or haptic interactions. Acknowledgements. This work has partly been supported by Deutsche Forschungsgemeinschaft (DFG) under grant agreement He2696-18.

References 1. Bischoff, R., et al.: The KUKA-DLR lightweight robot arm-a new reference platform for robotics research and manufacturing. In: ISR 2010 (41st International Symposium on Robotics) and ROBOTIK 2010 (6th German Conference on Robotics), pp. 1–8. VDE (2010) 2. Bugmann, G., Pires, J.N.: Robot-by-voice: experiments on commanding an industrial robot using the human voice. Ind. Robot: Int. J. 32(6), 505–511 (2005) 3. Fischer, K.: Interpersonal variation in understanding robots as social actors. In: 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 53–60. IEEE (2011) 4. Fraser, N.M., Gilbert, G.N.: Simulating speech systems. Comput. Speech Lang. 5(1), 81–99 (1991) 5. Green, A., Huttenrauch, H., Eklundh, K.S.: Applying the Wizard-of-Oz framework to cooperative service discovery and configuration. In: RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No. 04TH8759), pp. 575–580. IEEE (2004) 6. Kelley, J.F.: An iterative design methodology for user-friendly natural language office information applications. ACM Trans. Inf. Syst. (TOIS) 2(1), 26–41 (1984) 7. Qian, K., Niu, J., Yang, H.: Developing a gesture based remote human-robot interaction system using kinect. Int. J. Smart Home 7(4), 203–208 (2013) 8. Ralph, M., Moussa, M.A.: Toward a natural language interface for transferring grasping skills to robots. IEEE Trans. Robot. 24(2), 468–475 (2008) 9. Riedl, M., Henrich, D.: Guiding robots to predefined goal positions with multimodal feedback. In: ISR 2018; 50th International Symposium on Robotics. VDE (2018) 10. Riek, L.D.: Wizard of Oz studies in HRI: a systematic review and new reporting guidelines. J. Hum. Robot Interact. 1(1), 119–136 (2012) 11. Steinfeld, A., Jenkins, O.C., Scassellati, B.: The Oz of wizard: simulating the human for interaction research. In: Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, pp. 101–108 (2009) 12. Tellex, S., et al.: Understanding natural language commands for robotic navigation and mobile manipulation. In: AAAI Conference on Artificial Intelligence (2011) 13. Wolf, M.T., et al.: Gesture-based robot control with variable autonomy from the JPL BioSleeve. In: International Conference on Robotics and Automation (2013) 14. Yamaoka, F., et al.: A model of proximity control for information-presenting robots. IEEE Trans. Robot. 26(1), 187–195 (2009)

Cloud-Based Digital Twin for Robot Integration in Intelligent Manufacturing Systems Florin Anton, Theodor Borangiu(&), Silviu Răileanu, and Silvia Anton Department of Automation and Applied Informatics, University Politehnica of Bucharest, Bucharest, Romania {florin.anton,theodor.borangiu,silviu.raileanu, silvia.anton}@cimr.pub.ro

Abstract. The paper describes the architecture design and implementing solution for the digital twins of industrial robot, aggregated and embedded in the global health monitoring, maintenance and control system of manufacturing resources. Manufacturing scheduling and control system. The main functionalities of the digital twin are: monitoring the current status and quality of services performed by robots working in the shop floor, early detecting anomalies and unexpected events to prevent robot breakdowns and production stops, and forecasting robot performances and energy consumption. Machine learning techniques are applied in the cloud layer of the virtual twin for predictive, customized maintenance and optimized robot allocation in production tasks. The paper introduces a framework integrating the virtual robot twins in an ARTItype control architecture, proposes a solution to implement the twin on a distributed cloud platform and exemplifies the concepts in a shop floor case study with SCARA assembly robots. Keywords: Digital twin
ARTI
Industrial robot
Anomaly detection Predictive maintenance
Edge computing
Cloud computing




1 Introduction The term ‘Industry of the Future’ (IoF) indicates the new industrial revolution initiated by a new generation of manufacturing systems designed to be adaptive, fully connected, highly efficient (optimized) and robust (fault tolerant, highly available). The global IoF model describes a new stage of manufacturing fully automatized and robotized, using ever more advanced information, control and communication technologies (IC2T). IoF works with ‘intelligent factories’, characterized by digitalization and interconnection of distributed manufacturing entities in a ‘system of systems’ approach: i) new types of production resources (e.g., intelligent robots) highly interconnected and self-organizing in the shop floor, products deciding upon their own work route and resources; ii) smart decisions taken from real-time production data collected from resources and products and processed with machine learning techniques [1]. © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 565–573, 2020. https://doi.org/10.1007/978-3-030-48989-2_60

566

F. Anton et al.

The digital transformation of manufacturing through resource, product and order virtualization relies on techniques allowing for intelligent decision-making complying with the principles of flexibility and reality-awareness. These issues are addressed in the literature through three domains: big data analytics, machine learning and digital twins [2]. The holistic view of the real capabilities, status and features of an entity (e.g., robot, process, product) including its digital representation, execution context, history of behaviour, time evolution and status can be encapsulated in a digital twin (DT) defined as an extended virtual model of the robot, process, product, that is persistent even if its physical counterpart is not always on line/connected; this extended digital model can be shared in a cloud database with other shop floor entities [3]. Robot virtualization relates to the capacity of creating robot services in the cloud – a part of the Cloud Manufacturing (CMfg) technology [4]. Manufacturing control frameworks like CyberPhysical Production Systems (CPPS) that use machine learning (ML) in CMfg models. The idea of DT is currently being introduced in manufacturing control architectures to virtualize three abstract entity classes: resources (robots, machines), products (client preferences, recipes,) and orders (operation sequences, precedencies). It becomes a key element between asset role and decision making and is able to monitor, forecast and influence the behaviour of the physical twin. Two main classes of DT are of interest in robotized manufacturing [6]: • Digital twin of resources and products: a) monitoring the health, quality of services (QoS) performed by resources, detecting anomalies in their behaviour and launching predictive maintenance programs; b) performing the traceability of products. • Digital Twin of organizations: mirroring and forecasting the control system’s actions and performances for batch optimization, robustness and process resilience. The paper introduces a framework of a reality-aware digital twin for industrial robots integrated in intelligent manufacturing systems that use cloud services performed on two layers: i) a layer distributed among fog computing elements, and ii) a centralized cloud IaaS (Infrastructure as a Service). This framework enables robot virtualization, health monitoring and anomaly detection, and the coupling between behavioural robot models and multi-physical processes for real-time predictive robot maintenance and optimal allocation in production tasks based on energy consumption and QoS forecasts. The rest of the paper is organised as follows: Sect. 2 defines the multi-layered DT architecture for an industrial robot integrated in manufacturing, and the embedding principles. Section 3 describes the edge- and fog computing layers that implement the distributed DTs in multi-robot manufacturing systems. Section 4 presents the Cloud IaaS for the high-level DT layers embedded in predictive individual robot maintenance and collective optimized robot allocation tasks. Experimental results are presented and a set of conclusions are formulated.

Cloud-Based Digital Twin for Robot Integration

567

2 Digital Twin Architecture of Robots in Manufacturing The DTs of industrial robots, components of manufacturing systems, are embedded in the global resource management and batch scheduling programs with the following objectives: safe and efficient utilization of robotic resources and optimization of batch production (energy consumption, make-span, balanced resource usage, etc.). Figure 1 shows the 4-layer DT architecture of the Industrial Robot – component of an intelligent manufacturing system. The lower level Digital Twin layers (DT I and DT II) define the composition of the robot components and processes served by the robot, while the higher level ones (DT III and DT IV) implement methods for predictive robot maintenance and allocation in manufacturing tasks being executed through high performance computing in the cloud:

Fig. 1. 4-layer aggregate digital twin embedded for predictive anomaly detection, maintenance and allocation of industrial robots in manufacturing operations

DT I - Data Stream Collecting. This DT element is connected to the physical twin (the components of: the robot manipulator - arm/gripper/tools; robot controller electronic boards/backplane; machine vision - video cameras/lighting system; external sensors - vibrations/effort/temperature), to the process it serves and to the environment in which it operates, and gathers data from a multitude of shop floor entry points. DT II - Data Stream Processing and Analysis. The functions of this DT element are: sorting and aligning the data streams generated by DT I in normalized time intervals, and aggregating them using covariance techniques. There are four functional modules composing this second DT element: • Repository of behavioural robot models: it contains models of the robot operating modes that must be defined and called in order to infer correctly the various possible

568

F. Anton et al.

behaviours of the robot system (e.g., full automated/semi-automated mode, dry run etc.). In addition, some behaviour parameters must be defined off-line and eventually re-evaluated at run time (e.g., parameters of the robot’s TOOL transformation). • Repository of multi-physical process models: it contains models of the physical processes upon which the robot acts; the parameters of these models depend strongly on the robot operating parameters. • Time alignment and data aggregation: the main function consists in joining raw data streams tagged according to the two above models in application-specific streams. Distributive join/merge big data operations are executed in parallel on multiple worker nodes in map-reduce clusters. Once the information needed for an application is assembled in a ‘joined’ stream, the next operation is a ‘reduce’ type one. Multiple aggregations are possible using different keys for map-reduce, i.e. tagging messages by their shop floor location or energy consumption area or heat distribution. • Detection of deviations and unexpected events: this functional component of the DT generates immediate decisions of resource blocking in case unexpected events or significant deviations of the sensed robot parameters from the limit values specified in the behavioural models or from the controller’s planned commands are detected. DT III - Machine Learning. This high-level DT is used to extract insights in realtime from data streams aggregated at the robot workplace (resource, environment, process). For multi-robot shop floors, the computation is parallelized for all robot channels. Three machine learning (ML) techniques are used [5]: prediction with deep learning neural network to learn patterns and variations in measurements (e.g., consumed energy); the neural network is then used to make forecasts for these measurements; classification - the DT tries to find a class for every feature vector received; clustering - the DT tries to find similarities in non-labelled, multidimensional data and tag each feature vector accordingly. The usage of ML needs historical data storage to train the ML models. DT IV- Intelligent Decision Making. The decision support available on this DT layer allows embedding the robot’s networked digital twin in two scenarios: i) weighting the allocation of resources to operations on products in multi-robot manufacturing systems, based on forecasted values of QoS, performances and/or power consumption, for global optimization; ii) anomaly detection and customization of robot maintenance programs: the classification is done at multiple levels; one classification is the robot health using multiple combined KPIs in the map-reduce phase.

3 Collecting and Processing Robot Data at Shop Floor Edge In order to connect robot sensors, actuators or embedded control devices in an Industrial IoT (IIoT) framework, a dedicated type of hardware was developed – the IoT gateway [7]. This hardware connects devices to a network which is in most of the cases directly connected to Internet. Additionally, IoT gateways pre-process and filter the

Cloud-Based Digital Twin for Robot Integration

569

data from shop floor devices locally, at the edge of the CMfg, reducing thus the amount of data to be sent in the Internet with big impact on response time and network transmission costs. Apart the use of IoT gateways as concentrators of legacy devices and the access to local networks, their processing power can be used to treat more complex information originating from the robot workstation, locally and faster without overloading the cloud. Thus, decision algorithms based on high performance mathematical computation which are not available in the robot operating system can be executed on the IoT gateway and the result returned to the robot controller or to the cloud. These types of algorithms are usually associated with vision systems and with end-effector tools the control part of which is not embedded in the robot controller (in most cases these are dedicated/one of a kind systems built ‘in house’ by the application integrator). An example of such robot gripper device is a magnetic tool with collision detection capability. In order to analyse in real-time the data streams generated by robot sensors/external devices supporting multiple communication protocols for data transfer, the IoT gateway was extended in the present research to the aggregation node concept [8]. An aggregation node consists of a group of entities connected to an industrial processing unit – the node’s kernel (PC workstation, Single Board Computer - SBC, Next Unit of Computing - NUC) that is flexible enough to run big data software. A NUC is a small form factor computer kit manufactured by Intel [intel.com] which offers the same processing power of desktop PCs. The NUC uses high-end microprocessors and is usually integrated in robot-vision applications since it can be mounted near the image acquisition devices, minimizing thus the processing times due to the direct, high-speed connection. Having standard USB ports, it can be extended with modules for industrial communication and digital/analogue I/O channels. The following data stream processing of robot parameters are considered for the aggregation nodes on level II of the digital twin: • Motor temperatures, which joined with the energy consumption data are used for fault detection and preventive maintenance, Fig. 2, A. • Energy-related parameters for the entire robot manipulator and its n joint motors (voltage, current, phase), Fig. 2, A. This information is collected continuously and per-operation and used in resource scheduling, parameter adjustment (maximum speed and acceleration) and fault detection [9]. • Joint values in order to compute motor utilisation and associate the values with cruise speed, acceleration and payload, Fig. 2, D. • Additional communication modules for bidirectional interaction between the aggregation node processor and the robot controller in order to map gripper/tool information to the robot’s internal signals, Fig. 2, B (e.g., measuring the amplitude of tool vibrations and correlating these vibrations with the acceleration and speed of the robot in order to optimize its cycle time). • Evaluation of the robot base vibrations and checking the robot arm alignment for periodic evaluation of its motion accuracy, Fig. 2, C. The vibrations of the base segment are monitored; based on the computed evolution a position alignment check command would be issued. The robot is moved to a point where two sharp marks must align; the verification can be visual or automatic.

570

F. Anton et al. Digital twin: knowledge extraction, data analytics, machine learning and making intelligent decisions

CLOUD

Publish-subscribe network protocol (MQTT)

IoT gateways

Aggregation node kernel (PC/SBC/NUC)

……

Data collection (A)

Video camera with inertial sensors

Data collection (E)

Data collection and actuation (B) TCP connection (D)

Smart sensors (energy, voltage, current, temperature)

Web connection (for parameter monitoring)

Electric contact or visual check

Smart sensors (collision, weight, vibrations) TCP connection (for programming)

Mobile frame Fixed frame

Data collection (C)

Robot TOOL

Smart sensors (vibrations) Client

Resource controller

Control cable

Fig. 2. Interconnections of the robot DT II aggregation node for continuous and operation-based monitoring of QoS and anomaly detection (left); robot alignment device (right)

• Integration of external information used for robot guidance. Variations of the relative position between a fixed camera and a robot can be known by measuring the amplitude of vibrations in the robot base and camera support. Vibrations relative to a mobile frame (e.g. gripper/tool) are measured using analogue inputs collected relative to the possible movement axis. Vibrations relative to a fixed frame (e.g. robot base - camera support) are measured with inertial sensors. Sensors are connected to the aggregation node which processes data and issues parameter change commands (limit speed/acceleration) or invalidates camera-robot calibration.

4 Experiments with Cloud DT for Adept Robots. Conclusions Experiments were performed to implement a 4-layer Digital twin framework for Adept industrial robots in a private Cloud IaaS model. Robot data was acquired both from IoT gateways and using the Adept ACE framework that allows data storing into a log file. Adept ACE is based on a series of processes which run in the kernel space of the Adept robot V+ operating system and have access to robot status data. The data which was collected falls in three categories: (1) Data characterizing the robot: AC input [V]; amplifier bus voltage [V]; DC input [V]; amplifier temperature [°C]; encoder temperature [°C]; base board temperature [°C]; duty cycle [% limit]; harmonic drive usage [%]; peak velocity [RPM];peak torque [% max torque]; peak

Cloud-Based Digital Twin for Robot Integration

571

position error [% soft envelope error]; (2) Data for a robot-serviced belt: belt velocity [mm/sec]; instance count; instances per minute; active instances; latch faults; (3) Data characterizing the process: idle time [%]; processing time [%]; average total time [ms]; targets processed/not processed/per minute; parts processed/not processed/per minute. The Cygwin Linux tools package was used to exploit the data from the log file; the log file can be processed in real-time by a sequence of tools or a pipeline. The pipeline is executed in a loop based on a configured time interval by the watch command. The pipeline consists of three commands: tail which extracts the last line from the log file, cut which extracts the individual logged values from the line extracted by tail (each time when Adept ACE scans the robot status, all values are placed on a single log line and are comma delimited) and sends the output to a custom script check-and-log which is based on a configuration file. The pipeline has two functions: i) tests the read values and compares them to a predefined threshold; if the value exceeds the threshold the script can send a notification in the manufacturing cell in order to stop the production or to handle the situation; ii) generates log data for the cloud system. In order to do this the logger command is used which generate a standard log message which is sent to the rsyslog logging service (Fig. 3).

Fig. 3. Processing robot data in cloud DT III and DT IV

The check-and-log script is executed in a pipe but is also running standalone in order to obtain other information acquired by the IoT devices, the communication capabilities being offered by Net Cat (a communication tool which can be easily integrated with scripts). When the messages generated by logger reach the local rsyslog server they get a timestamp (with millisecond resolution) and then are sent over a VPN connection to the cloud where another rsyslog server gets the message and stores it into a database (MySQL or Postgres). This remote logging has the advantage that the local storage will not be exhausted and if there are delays in receiving the

572

F. Anton et al.

messages, they will already have an associated timestamp which is not affected by the connection delay [9]. The robot data from the database is analysed by a software agent that uses a heuristic parameter monitoring algorithm; this algorithm can be executed in two ways: 1) Learning a “normal” behaviour of the robot system (learning the variation of the parameters during a robot cycle). 2) Detecting anomalies of the robot based on collected data and trained model. The first execution mode when the behaviour is learned should not to generate false positive or false negative situations; the model training is restricted on a single “static” robot cycle (which is executed multiple times). A “static” robot cycle means that the cycle is composed by motions using fixed trajectories, constant robot loads, fixed robot speeds and fixed accelerations/decelerations. For each robot task a model can be trained and used to detect deviations from normal behaviour, based on 3 values defined for each parameter: standard deviation, maximum value and minimum value. During the experiments the digital model of the robot was created based on over 10 K readings. For some parameters such as voltage the model was based on the standard deviation for a robot cycle, in other cases (for example temperatures) the minimum and maximum values have been considered (for the entire robot uptime). Figure 4 presents the data samples acquired for model training.

Fig. 4. Robot parameter values used for training

The paper proposes a layered architecture for aggregate digital twins of industrial robots integrated in multi-resource manufacturing systems. These 4-layer digital twins are embedded in individual health monitoring and predictive maintenance programs and collective optimal allocation for product operations at batch level. The robot DT layers span three implementing levels of Cyber-Physical Systems: the edge computing level for data acquisition, the fog computing level for data processing and analysis and the cloud computing level for machine learning and intelligent decision making.

Cloud-Based Digital Twin for Robot Integration

573

References 1. Morariu, C., Răileanu, S., Borangiu, T., Anton, F.: A distributed approach for machine learning in large scale manufacturing systems. In: Service Orientation in Holonic and MultiAgent Manufacturing. Studies in Computational Intelligence, vol. 803, pp. 41–52. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-03003-2_3 2. Peres, R.S., Rocha, A.D., Leitao, P., Barata, J.: IDARTS – towards intelligent data analysis and real-time supervision for Industry 4.0. Comput. Ind. 101, 138–146 (2018). https://doi.org/ 10.1016/j.compind.2018.07.004 3. Oracle: Digital Twins for IoT Applications: A Comprehensive Approach to Implementing IoT Digital Twins. Redwood Shores (2017) 4. Qia, Q., Taoa, F., Zuoa, Y., Zhaob, D.: Digital twin service towards smart manufacturing. Procedia CIRP 72, 237–242 (2018). https://doi.org/10.1016/j.procir.2018.03.103 5. Puleri, M., Sabella, R., Osseiran, A.: Cloud robotics. 5G paves the way for mass-market automation. Ericsson Technology Review, 28 June 2016. ISSN 0014-0171 6. Sabella, R., Thuelig, A., Carozza, M.C., Ippolito, M.: Industrial automation enabled by Robotics, Machine Intelligence and 5G. Ericsson Technology Review, February 2018 7. Desai, N.: What is an IoT Gateway and How Do I Keep It Secure? Global Sign Internet GMO Group (2016). https://www.globalsign.com/en/blog/what-is-an-iot-gateway-device 8. Gloria, A., Cercasa, F., Souto, N.: Design and implementation of an IoT gateway to create smart environments. In: Proceedings of the 8th International Conference on Ambient Systems, Networks and Technologies, ANT 2017 (2017). https://doi.org/10.1016/j.procs.2017.05.343 9. Răileanu, S., Anton, F.A., Borangiu, T., Anton, S., Nicolae, M.: A cloud-based manufacturing control system with data integration from multiple autonomous agents. Comput. Ind. 102, 50–61 (2018)

Evaluation of Serial Metamorphic Manipulator Structures Considering Inertia Characteristics N. A. Stravopodis1(B) , L. Katrantzis1 , V. C. Moulianitis1 , C. Valsamos1 , and N. A. Aspragathos2 1

Department of Product and Systems Design Engineering, University of the Aegean, 84100 Ermoupolis, Syros, Greece {n.stravopodis,lef.katrantzis,moulianitis,balsamos}@syros.aegean.gr 2 Department of Mechanical Engineering and Aeronautics Rio, University of Patras, 26500 Achaia, Greece [email protected] https://www.syros.aegean.gr, https://www.upatras.gr

Abstract. This paper presents the dynamic evaluation of Serial Metamorphic Manipulator structures considering inertia characteristics. The basic structural components are presented and the proposed representation is discussed. In order to investigate the ability of a metamorphic structure to reduce mass-coupling errors and achieve dynamic isotropy conditions, a global performance index based on the Dynamic Conditioning Index (DCI) is introduced. The optimization results from the use of the introduced index show that a metamorphic structure expands the manipulator’s versatility of achieving anatomies with significantly improved dynamic characteristics. Additionally, the conformity of the generated optimal structures, with the general design conditions for simplifying the dynamics of fixed anatomy manipulators, is briefly discussed.

Keywords: Serial metamorphic structure evaluation

1

· Dynamics

Introduction

Manipulator dynamic performance is difficult to determine because of the highly non-linear nature of the problem. The simplification of a robotic manipulator system in order to enhance analysis and control has been on the forefront of research for the past decades. Robotic structure modification and mass-inertia distribution are two of the most common approaches for dealing with the problems regarding the dynamic performance of manipulators. Studies presented in [1–5] specified general design conditions for fixed structure manipulators. Guidelines such as eliminating the coefficients of non-linear terms in robot’s energy equation, simplifying the energy c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 574–587, 2020. https://doi.org/10.1007/978-3-030-48989-2_61

Structure Evaluation of Serial Metamorphic Manipulators

575

equation from terms generated by relative motion of articulated bodies and eliminating the configuration dependant terms in the elements of the Generalized Inertia Matrix (GIM) lead to modifications of the structural parameters. Although the physical interpretation is enhanced using these techniques, modifying the structural parameters leads to the redesign of the manipulator links and the alteration of the robot assembly. As production lines shift towards more adaptable systems, changing to a complete new manipulator for different tasks cannot be considered as the optimal solution. According to MAR [6], configurability, reconfigurability and metamorphosis have been identified as key technological aspects that will enable the evolution of robotics in the future, as a direct result of the increased complexity and diversity of the tasks to be performed by robots as they are integrated into more application domains, including the manufacturing industry. Modular Reconfigurable Manipulators (MRM) were introduced at the last decades, in order to address the growing need of adaptable, multi-purpose robotic workcells. Since the mass properties of the individual arm links and the kinematic structure of the arm linkage play a significant role in the resulting dynamic behaviour of a manipulator, it is expected that a MRM presents enhanced capabilities for optimizing its dynamic performance. Serial Metamorphic Manipulators (SMM) are a class of MRM that can reach different anatomies without reassembling or adding new components in their initial structure [7]. The initial arrangement of the main elements of the metamorphic kinematic chain plays a significant role in the resulting set of feasible anatomies. Since, the versatility of the manipulator system highly depends on the set of anatomies capable of satisfying given the user-defined specifications, the optimal synthesis of the metamorphic structure is of great importance. Consequently, an important factor in metamorphic manipulator design, considering dynamic performance optimization, is the determination of an initial structure that can generate a large set of anatomies that comply with the attribute of reduced dynamic coupling. Nevertheless, reconfigurability presents new challenges to the robot designer since the reconfigurable kinematic structure should be appropriately modeled [7] and the dynamic model of the manipulator should be efficiently computed and reevaluated depending on the system’s reconfiguration [8–10]. In this paper, dynamic isotropy refers to the physical concept discussed in [11]. It is strongly dependent on manipulator’s inertia characteristics and greatly varies within the manipulator’s workspace, while the physical interpretation can be given from different viewpoints. From the joints point of view, the moment of inertia of an isotropic manipulator remains constant, while from the end-effector point of view, uniform acceleration characteristics are achieved. A great variety of indices have been proposed for the quantification of dynamic isotropy of robotic manipulators. The most common description was accomplished by drawing a n-dimensional ellipsoid [12–14]. Under this depiction, dynamic isotropy is depicted through the axial lengths of the ellipsoids. In [15,16] isotropy is investigated under the acceleration of the end-effector given the limits of joint actuator torques. Beside the ellipsoid techniques, indices have been proposed that stem

576

N. A. Stravopodis et al.

from drawing other types of hyper-surfaces [17,18] or by depicting the variation of GIM from an ideal diagonal form [19]. The main goal of this paper is to evaluate the capability of an SMM to allow for dynamic decoupling and isotropy for the available set of anatomies of each structure. The extent of dynamic performance enhancement, in a wide area of the configuration space, through the use of different number of structural elements is also studied. A metamorphic structure optimization process is proposed for setting the optimal initial manipulator structure, considering dynamic isotropy and maximizing the optimal set of structure’s anatomies. The rest of the paper is organized as follows. In Sect. 2 the metamorphic structure representation used in this paper is discussed. In Sect. 3 the proposed methodology is presented. In Sect. 4 the performance index used for dynamic isotropy investigation and the algorithm for optimal structure determination are discussed. In Sect. 5 the optimization results for the evaluated structures are presented and the structures generated are contrasted with the design condition formulated for fixed-structure manipulators.

2

Modeling the Metamorphic Structure

In this work, Modular Structure Representation (MSR) proposed in [20], is modified in order to strictly define the modeling of metamorphic structures. The definitions of Active Joint, Passive Joint (Pseudo joint), Metamorphic Anatomy and Configuration, with the corresponding statements and the string notation formulated for the establishment of MSR are used in this work. Additionally, the definition of Assembly Sequence, as the sequence of the basic components that form a metamorphic structure, is added. Metamorphic Structure is defined as the setting of structure’s synthetic links (assembly parameters) for a given assembly sequence. The main modification lies to the removal of the three types of distinct connections between active or pseudo joint twists: perpendicular, parallel and skew. In order for the metamorphic link to be assembled to the serial structure an abstract Synthetic Link is introduced. The assembly parameters consist of the three independent rotation variables about the axes of the preceding frame and the corresponding position variables. This local parameterization, despite of not being the optimal, facilitates a user-friendly depiction of the assembly. In order to shape the metamorphic structure, the following statements are established: – Between a preceding active module and a metamorphic link an interconnection is formed through the use of a synthetic link. – Adjacent modules inside a metamorphic link are at an orthogonal reference configuration. Consequently the three types of distinct connections are modeled through the assembly parameters of the synthetic link. The synthetic link is assigned the “&” symbol to simplify the string notation of the corresponding structure. For each “&” symbol in the structure string the 6 parameters {Rs , Ps } of the synthetic link

Structure Evaluation of Serial Metamorphic Manipulators

577

should be also given. These local parameters define the transformation between the preceding active module of link i and the passive module of the next link i + 1. The modified MSR notation given above and the assembly parameters of the synthetic links are presented in Fig. 1.

3

Proposed Methodology

In Fig. 2 the methodology that permeates the current study is illustrated. In this work 3 DoF metamorphic structures are evaluated, in order to be able to achieve complete dynamic linearization of each structure. In this way the effect of anatomy metamorphosis to the enhancement of manipulator’s inertia properties becomes more evident. The first step is the determination of the design variables of the metamorphic structure. The designer can choose metamorphic links formed by one or two pseudojoints. A larger number of pseudojoints doesn’t contribute to enriching the possible instances between the connected active joints [7]. Between two adjacent links a synthetic link is defined, with the corresponding assembly parameters. For structure optimization a genetic algorithm is utilized. Using screw theory the mass matrix elements are formulated in analytical form as a function of the reference metamorphic link inertia matrix, the active and pseudo joint twists and configuration and the transformation induced by the synthetic link. The gene suitable for the structure evaluated is formed and an intrinsic dynamic performance measure is used for the formulation of the fitness function. A group of structures is determined with regard to the minimum fitness function, that depicts the minimization of mass-coupling errors and dynamic isotropy condition of the inertia matrix at the reference anatomy of the structure.

(a) Basic components of a Metamorphic Structure

(b) Synthetic Link Transform Definition

Fig. 1. Modified MSR modeling

578

N. A. Stravopodis et al.

Finally, each structure extracted is exhaustively investigated for the number of possible anatomies. The ratio of structure’s anatomies that present a better score than the reference anatomy (anatomy richness) to the total number of structure’s anatomies in conjunction with the overall best anatomy score, shape the final structure optimization result.

Fig. 2. Proposed methodology overview

4 4.1

Metamorphic Structure Optimization Generalized Inertia Matrix of a Metamorphic Manipulator

In metamorphic manipulators, structure assembly and anatomy metamorphosis should also be integrated in GIM calculation. This procedure is significantly simplified through the use of twists and the Product of Exponentials (POE) formula [21]. The local parameterization defined for the ith synthetic link corresponds to a transformation matrix gsi ki+1 ∈ SE(3). This transformation is reflected to the spatial frame through the transformation gski+1 = gssi · gsi ki+1 , as presented in Fig. 1b. The Mass Matrix elements of a metamorphic manipulator are determined as: Mij (q, qp ) =

n 

ξiT · ATli · Ml · Alj · ξj

(1)

l=max(i,j)

⎧ −1 ⎪ ⎨Ad ξj+1 ·qj+1

with

e

Aij = and

·e

ξp ·qp k k

·eξj+2 ·qj+2 ···eξi ·qi

I ⎪ ⎩ 0

i>j i=j i 0 (dynamic isotropy). Summing the weighted errors between these values observed to repel solutions that drove towards isotropic configurations. 4.3

Optimization Procedure for a 3 DoF Structure

Due to the limitation on structure’s DoF and the statements of the modified MSR the possible assembly sequences are limited to four chains. Since the goal of this work is an in depth study of these assembly sequences a genetic algorithm is utilized to search for each one. The chromosome x is formed by a floating-point number to randomly determine the configuration space points, the assembly parameters for each synthetic link of the structure and the DCI weighting factors. The DCI based objective functions are calculated for a set of points in the configuration space of the reference anatomy of each structure. The population size selected for each generation is 1000. The maximum number of generations produced is 2000. For each gene of the population two fitness values are extracted from Eq. 8 and the average Pareto distance is calculated. Tournament selection procedure is then implemented. The next generation is formed using 10% elite children from the previous one and 70% from crossover and 20% from mutation operations. Intermediate crossover and adaptive feasible mutation methods are implemented. For the best structures extracted for each assembly sequence from genetic algorithm optimization, anatomy metamorphosis is exhaustively executed in order to examine the variation of the inertial properties for the set of structure’s emerging anatomies. The score function, to determine the best structure considering anatomy richness is: −1 f2 = Paf eas · fbest

(9)

The ratio of optimal feasible anatomies to structure’s total anatomies (Paf eas ) is used as a factor that enhances the best (minimum) global DCI value (fbest ) among all feasible anatomies. The overview of the optimization procedure is illustrated in Fig. 3.

Structure Evaluation of Serial Metamorphic Manipulators

581

Fig. 3. Optimization procedure overview

5 5.1

Results and Discussion 3 DoF Metamorphic Structure Optimization Results

The optimal structures generated are presented in Table 1a. Seven structures that present f1 ≤ 0.15 are selected. The two values extracted for f1 correspond to the best global DCI values of the off-diagonal and the diagonal elements respectively. The first index achieves zero values (dynamic decoupling) while for the second one, lower values correspond to a reference anatomy closer to isotropy. For each structure the MSR representation and the extracted assembly parameters {Rs , Ps } for the abstract synthetic links are presented in each column. The global DCI values extracted for reference anatomy of structure (fref ) are given in the corresponding column. The three weighting factors acquired are identical for all structures. The first two values that correspond to the metamorphic links are given the value 0.45, while the third one that relates to the fixed link is given the lower limit value (0.1). Setting the factor of the fixed link to minimum is justified due to the lower total mass of the fixed link relative to the first two. Since a large difference exists between the total mass of the links, a fully isotropic structure cannot be achieved. The structures obtained with minimum f1 have identical and relative small values (closer to the third value) for the first two diagonal elements of the GIM. Each structure is reevaluated as described in Sect. 4.3 and the final score is given in Table 1b. The highest scoring structure (i = 4) is illustrated in Fig. 4a at reference anatomy. Reference anatomy is dynamically decoupled but nonisotropic. In Fig. 4b the same structure is presented at the best anatomy. This anatomy is closer to isotropy and the mean DCI value between best and reference anatomy is reduced by 17%. Although the reference anatomy doesn’t achieve the best score from the genetic algorithm optimization, there is an ample number

582

N. A. Stravopodis et al.

of anatomies that lead to enhanced inertia properties which contributes to the highest overall score. The rest of the structures presented in Table 1a achieve values closer to isotropy but only a limited range of optimal anatomies is extracted. According to these results, the trade-off between anatomy richness and optimal isotropic structures is displayed. In Fig. 5a, 5b, 5c mass-decoupling achieved at optimal anatomy shown in Fig. 4b is presented. Structures (i = 1, 6) are illustrated in Fig. 6, 7. For these, a DCI comparison takes place to illustrate the minimized DCI values achieved by the optimized structures in the configuration space (Fig. 8). From Table 1b information regarding the effect of using different number of pseudojoints in the assembly sequence is extracted. Structures using more pseudojoints don’t necessarily achieve the greater total number of optimal anatomies. The total number of pseudojoints used doesn’t affect the capability of the metamorphic manipulator to achieve isotropic structures. Table 1. Optimization procedure results i 1 2 3 4 5 6 7

MSR

fref

Assembly Parameters

[0, −1.57, 1.26] [0.14, 0.14, 0.14] 0&10&110# [0.03, −1.35, 0.81] [0.10, 0.09, 0.11] 0&110&10# [0.15, 0.25, −0.96] [0.03, 0.07, 0.13] 0&10&110# [−1.03, −1.57, −1.27] [0.02, 0.06, 0.10] 0&110&110# [0.01, −0.74, 1.01] [0.14, 0.10, 0.11] 0&110&110# [0.02, 0.51, −1.32] [0.03, 0.12, 0.10] 0&110&10# [0.15, 0.24, −0.96] [0.03, 0.07, 0.13] 0&10&10#

[−1.22, −1.57, 1.08] [0.11, 0.11, 0.15] [1.09, −1.46, −1.08] [0.04, 0.03, 0.05] [−0.98, −1.41, 1.01] [0.05, 0.01, 0.05] [−0.13, −0.18, −0.52] [0.08, 0.02, 0.01] [1.35, 1.06, −1.33] [0.03, 0.09, 0.10] [1.04, −1.36, 0.0.60] [0.06, 0, 0.04] [−0.97, −1.41, 1.01] [0.05, 0, 0.05]

0.0585 0

0.0550

0.0507 0

0.0585

0.0653 0

0.0692

0.1156 0

0.1295

0.0600 0.0001 0.0581 0.0540 0.0003 0.0647 0.0607 0.0084 0.0758

(a) Optimized Structures i 1 2 3 4 5 6 7

Npused 2 3 3 3 4 4 3

Nabest 11 7 15 287 22 20 1

Naf eas 9 4 8 231 12 12 1

Natot 49 343 343 343 2401 2401 343

fbest 0.0517 0.0507 0.0611 0.0959 0.0590 0.0531 0.0607

Paf eas (%) 18.37 1.17 2.33 67.35 0.50 0.50 0.29

f1

f2 3.5527 0.2301 0.3817 7.0226 0.0847 0.0941 0.0480

(b) Exhaustive Anatomy Evaluation of optimized structures

Structure Evaluation of Serial Metamorphic Manipulators

5.2

583

Integrating the General Design Conditions for Simple Dynamics of Fixed-Structure Robots in a Metamorphic Structure

As expected, the majority of the optimized structures remains in conformity with the optimal design conditions defined for the dynamic design of fixed-structure robots. Although the subject can’t be analyzed in depth here, some initial observations are stated. First, the majority of structures presented in Table 1a tend to achieve mass balancing conditions. For all structures illustrated at best anatomy, the total center of mass of links 1–3 is located near the first rotation axis. This is even more evident at Fig. 4b, 7b. The main axial direction of link 2 is parallel with the second rotation axis. Under this structure, the altering of pseudojoints 3,4 places the total center of mass of links 2 and 3 very close to the second rotation axis. In contrast with fixed-structure robots, metamorphic manipulators achieve more than one ideal anatomies for the same structure, as shown in Table 1b. Metamorphic links present a variable inertia tensor, and are capable of changing the pseudojoints’ angles to reach the optimal settings for each structure.

(a) Reference Anatomy

(b) Best Anatomy θp =(-1.5708,-1.5708,1.0472)

Fig. 4. Optimized structure (i = 4)

584

N. A. Stravopodis et al.

(a) M12

(b) M13

(c) M23

Fig. 5. Mass Matrix off-diagonal elements comparison for best anatomy of structure (i = 4) and a random structure/anatomy

(a) Reference Anatomy

(b) Best Anatomy θp =(-0.5236,-1.5708)

Fig. 6. Optimized structure (i = 1)

Structure Evaluation of Serial Metamorphic Manipulators

(a) Reference Anatomy

585

(b) Best Anatomy θp =(0,0,-0.5236,-0.5236)

Fig. 7. Optimized structure (i = 6)

(a) Optimized (i=1) vs. Random

(b) Optimized (i=6) vs Reference

Fig. 8. DCI comparison for optimized and reference/random structures

6

Conclusion and Future Work

In this paper 3 DoF metamorphic structures were optimized considering inertia characteristics. Structures were generated through the utilization of an optimization process of metamorphic structure’s assembly, considering dynamic decoupling and isotropy. A global performance measure contingent on DCI proved sufficient to acquire information of the conditioning of the GIM. The presented scoring function for anatomy evaluation was essential for the deeper study of the anatomy richness of the optimized structures. It was shown that, not only metamorphic structures may be designed to achieve high dynamic performance, but they also achieve enhanced performance with a variety of possible anatomies. For future work, the utilization of more performance measures and the study of

586

N. A. Stravopodis et al.

the effects of mass-inertia optimization to the acceleration characteristics of the end-effector would provide greater clarity on the optimal design of metamorphic structures. Acknowledgment. This research has been financially supported by General Secretariat for Research and Technology (GSRT) and the Hellenic Foundation for Research and Innovation (HFRI) (Code: 1184).

References 1. Yang, D.C.H., Tzeng, S.W.: Simplification and linearization of manipulator dynamics by the design of inertia distribution. Int. J. Robot. Res. 5(3), 120–128 (1986) 2. Youcef-Toumi, K., Asada, H.: The design of open-loop manipulator arms with decoupled and configuration-invariant inertia tensors. In: Proceedings of the 1986 IEEE International Conference on Robotics and Automation, vol. 3. IEEE (1986) 3. Chung, W.K., et al.: On the dynamic characteristics of balanced robotic manipulators. In: Japan-USA Symposium on Flexible Automation, Control and Design of Robotics. Japan-USA Symposium (1986) 4. Park, H.S., Cho, H.S.: An approach to the design of ideal robotic manipulators having simple dynamic characteristics. Proc. Inst. Mech. Eng. Part B: Manag. Eng. Manuf. 201(4), 221–228 (1987) 5. Park, H.S., Cho, H.S.: General design conditions for an ideal robotic manipulator having simple dynamics. Int. J. Robot. Res. 10(1), 21–29 (1991) 6. Multi-Annual Roadmap (MAR) For Robotics in Europe. https://ec.europa.eu/ digital-singlemarket/en/news/multi-annual-roadmap-call-ict-24-robotics-nowavailable 7. Valsamos, C., Moulianitis, V., Aspragathos, N.: Kinematic synthesis of structures for metamorphic serial manipulators. J. Mech. Robot. 6(4), 041005 (2014) 8. Liu, G., Abdul, S., Goldenberg, A.A.: Distributed modular and reconfigurable robot control with torque sensing. In: 2006 International Conference on Mechatronics and Automation. IEEE (2006) 9. Li, Z., Melek, W.W., Clark, C.: Decentralized robust control of robot manipulators with harmonic drive transmission and application to modular and reconfigurable serial arms. Robotica 27(2), 291–302 (2009) 10. Chen, I.-M., Yang, G.: Automatic model generation for modular reconfigurable robot dynamics, pp. 346–352 (1998) 11. Matone, R., Roth, B.: Designing manipulators for both kinematic and dynamic isotropy. In: Proceedings Tenth CISM-IFToMM Symposium on Theory and Practice of Robots and Manipulators (Romansy 11) (1997) 12. Asada, H.: Dynamic analysis and design of robot manipulators using inertia ellipsoids. In: Proceedings of the 1984 IEEE International Conference on Robotics and Automation, vol. 1. IEEE (1984) 13. Yoshikawa, T.: Dynamic manipulability of robot manipulators. Trans. Soc. Instrum. Control Eng. 21(9), 970–975 (1985) 14. Chiacchio, P., et al.: Reformulation of dynamic manipulability ellipsoid for robotic manipulators. In: Proceedings of the 1991 IEEE International Conference on Robotics and Automation. IEEE (1991)

Structure Evaluation of Serial Metamorphic Manipulators

587

15. Bowling, A., Khatib, O.: Analysis of the acceleration of non-redundant manipulators. In: Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots, vol. 2. IEEE (1995) 16. Graettinger, T.J., Krogh, B.H.: The acceleration radius: a global performance measure for robotic manipulators. IEEE J. Robot. Autom. 4(1), 60–69 (1988) 17. Bowling, A., Khatib, O.: The dynamic capability equations: a new tool for analyzing robotic manipulator performance. IEEE Trans. Robot. 21(1), 115–123 (2005) 18. Bowling, A., Kim, C.: Dynamic performance analysis for non-redundant robotic manipulators in contact. In: 2003 IEEE International Conference on Robotics and Automation (Cat. No. 03CH37422), vol. 3. IEEE (2003) 19. Ma, O., Angeles, J.: Optimum design of manipulators under dynamic isotropy conditions. In: 1993 Proceedings IEEE International Conference on Robotics and Automation. IEEE (1993) 20. Valsamos, C., Moulianitis, V.C., Aspragathos, N.: Metamorphic structure representation: designing and evaluating anatomies of metamorphic manipulators. In: Advances in Reconfigurable Mechanisms and Robots I, pp. 3–11. Springer, London (2012) 21. Murray, R.M., Li, Z., Sastry, S.: A Mathematical Introduction to Robotic Manipulation. CRC Press, Boca Raton (1994). Print

Vision-Way Testing in Design of Small Compliant Mechanisms Jaroslav Hricko(&)

and Stefan Havlik

Institute of Informatics, Slovak Academy of Sciences, Banská Bystrica, Slovakia {hricko,havlik}@savbb.sk

Abstract. Compliant structures and mechanisms represent the base parts of the broad class of micro-robotic or sensory devices. The crucial problem in designing compliant mechanisms is that their flexural characteristics can not be exactly derived as in cases of classic structures with revolute joints. For this reason, the experimental approach should be applied in design procedures. This paper describes the optical experimental equipment and vision-based technique that enable to measure and to evaluate real performance characteristics of models, or final compliant structures, especially real motions/deflections of structures/particular segments, force - compliance characteristics and dynamics. Keywords: Compliant mechanisms
(Micro) robotic devices
Vision system
Digital Image Correlation

1 Introduction The design procedure of any compliant mechanisms [1] frequently used in microrobotic positioning or sensory devices should include the careful evaluation of calculated parameters usually verified by experimental approaches. This demand closely relates to their behavior for specific applications where high positioning accuracy is required. Modeling and mathematical calculations are always based on theoretical assumptions and material parameters then any results can be considered as rough approximations. Similarly, the errors due to small deviations of dimensions of the compliant segments/hinges, caused by the non-precise production process, resulting in the unwanted deflections and in the deterioration of the final positioning accuracy. In particular, this effect can be clearly manifested on flexible joints situated at the beginning of the kinematic chain, i.e. situated near the actuators. The difference between results from mathematical models and real measurements is the clamant problem for designing precise compliant mechanisms. There are research works [2, 3] that ascertain that the difference of results between a general (linear) mathematical description of one flexure element and measured values from the real part is within the range 2% to 10%. More complex flexible objects are being made up of dozens of elastic elements. Considering the above problems it is necessary to include the test and measurement procedure in order to evaluate the real characteristics of the proposed mechanisms. As to measuring used for evaluation forces and displacements of compliant structures there © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 S. Zeghloul et al. (Eds.): RAAD 2020, MMS 84, pp. 588–595, 2020. https://doi.org/10.1007/978-3-030-48989-2_62

Vision-Way Testing in Design of Small Compliant Mechanisms

589

most frequently used are sensory devices working on the strain gauges, magnetostriction or piezoelectric principles. Any implementation of such type of sensor into a proposed compliant structure has unwanted effects on the output characteristics. For this reason, wireless and contactless sensory techniques are more suitable. In the case of wireless connection was adopted a method where the measured value is converted to a digital form and transmitted by conventional wireless technology (Wi-Fi, Bluetooth). But such an approach leads to further possible measurement errors (conversion, transmission, etc.). As a more suitable approach, it was developed the wireless method for sensing small displacements/forces based on the electromagnetic principle, where the information about measured value corresponds to the frequency [4–6]. There is only one disadvantage; the principle needs the parallel resonance circuit to be directly made on the deformable structure. Within the spectrum of all possible principles, the most suitable seems to be optical sensing and vision way for evaluation deflections of the compliant structure. This paper describes the vision-based equipment and algorithms that enable to measure and to evaluate real performance characteristics of models or final compliant mechanical structures, especially real motions/deflections of structures (including segments) segments, force/compliance characteristics, and dynamics. The goal of the paper is to compare different approaches for measurement of the small displacements in order to choose which of them can be used on various compliant structures and mechanisms.

2 The Proposed Test-Bed System for Optical Measurements 2.1

Concept and Configuration

The optical way of measurement, in general, can be used for several purposes. There are [13]: – – – –

Measuring and characterization of the measured flexural and kinematics quantities Identification and evaluation of the material parameters Cross-validation, verification/correctness of the design procedure Control, measurement of the current device state and use it as feedback for control system

For our proposed optical test-bed the modular concept was adopted. Proposed configuration enables performing experiments on various types of sensors (deformable parts of sensors) or compliant mechanical structures. The reconfigurable frame consists of the aluminum profiles what enables to realize performance test on the physical samples of small devices with dimensions of some few millimeters up to several tenths centimeters. As actuation devices there are used linear card motor, the group of the motorized micrometer heads and some solenoids. The range of actuators motion is within 4 mm up to 25 mm. The sensory part of the experimental platform consists of the force/torque sensors and laser distance sensors with resolution 1 µm. The CMOS camera with the varifocal and telecentric objectives was chosen. This choice enables to upgrade the sensory system with respect to various objects (Table 1).

590

J. Hricko and S. Havlik Table 1. Technical parameters of the vision system components Camera Sensor Image size Sensor size Resolution Pixel size Interface

Sony IMX264, CMOS 2/3″ 8.4 mm  7.1 mm 2448 px  2048 px 3.45 µm  3.45 µm USB 3.0

Telecentric lens Magnification Working distance Telecentricity Distortion Field depth Object field of view

0.138 181.8 mm