Smart Electromechanical Systems: Behavioral Decision Making [352, 1 ed.] 3030681718, 9783030681715

Table of contents :
Preface
Contents
Methods and Principles of Designing of Decision Making System of the SEMS
Logical and Mathematical Method of Making Behavioral Decisions
1 Introduction
2 Stages of the Formation of Behavioral Decisions
3 Fuzzification Data and Forming Images
4 The Adoption of Reflective Solutions
5 Informed Decision Making
6 Conclusion
References
Patterns in Intelligent Control Systems for Robotic Systems
1 Introduction
2 Requirements for the Autonomy and Intelligence of Combat Cyber-Physical Systems
3 Initial Assumptions and Hypotheses
4 The Model of a Behavioral Pattern Fuzzy Description
5 A Model for Choosing an Agent When Implementing a Pattern
6 Modeling Patterns. Basic Modeling Points
7 Conclusion
References
Using Binary Relationships in Decision Making
1 Introduction
2 The Tasks of Situational Control of a Group of Dynamic Objects
3 Generalized Description of the Task of Situational Control of the SEMS Group
4 Mathematical Methods for Using Binary Relations in Decision
5 Conclusion
References
Decision-Making by the Autonomous Symbiotic Self-Relocating Massage Robot “Triangel” Based on SEMS After the Fall of Patient on Surface
1 Introduction
2 Description of Symbiotic Robot “Triangel” and Main Types of Possible Fallings of Patient During Massage
2.1 Description of Symbiotic Robot “Triangel”
2.2 Description of Main Types of Possible Fallings of Patient to Surface
2.3 Description of Main Types of Possible Falling of Patient to Stairs of Staircases
3 Decision-Making by the Autonomous Symbiotic Self-relocating Massage Robot “Triangel” After the Falling of Patient on Surface
3.1 The Axiomatics of the Decision-Making Model
3.2 Algorithm for Decision-Making by the Symbiotic Massage Robot “Triangel” After the Patient Falls to the Surface
4 Conclusions
References
Decision-Making by the Autonomous Symbiotic Self-Relocating Massage Robot “Octahedral Dodekapod” Based on SEMS During the Upper or Lower Limb Massage
1 Introduction
2 Description of Symbiotic Robot “Octahedral Dodekapod” and Main Types of Possible Fallings of Patient During Massage
2.1 Description of Symbiotic Robot “Octahedral Dodekapod”
2.2 Description of Main Types of Possible Fallings of Patient to Surface
2.3 Description of Main Types of Possible Falling of Patient to Stairs of Staircases
3 Decision-Making by the Autonomous Symbiotic Self-Relocating Massage Robot “Octahedral Dodekapod” After the Falling of Patient on Surface
4 Conclusions
References
Methods and Principles of Designing of Decision Making System of the SEMS Group
Problems with Secure Control of SEMS Group
1 Introduction
2 The Principles of Safe Control
3 Managing the Safe Movement of the Group Through the Intersection, Taking into Account the Rules of Passage
4 Managing the Group’s Safe Movement Based on Priorities
5 Conclusion
References
Synthesis of Optimal Program Control for Synchronizing the Movements of a Group of SEMS Modules
1 Introduction
2 Statement of the Problem of Optimal Control Synthesis for a Mechanism Consisting of SEMS Modules
3 The Problem of Synthesizing the Trajectory of the SEMS Module by the Planner in Layer 3
4 Motion Trajectory Synthesis Using Gradient Optimization Methods
5 Trajectory Synthesis Using the Multidimensional Patchwork Shell Method (MPSM)
6 Synthesis of a Trajectory Using a Method Based on the Construction of the Boundaries of the Region of Feasible Solutions
7 Conclusions
References
Task Scheduling Within Robots’ Collectives of Arbitrary Structures
1 Introduction
2 NCSs: Features, Legends and Our Proposals
3 SA for Groups of Robots
4 Conclusion
References
Meta-Heuristic Algorithm for Decentralized Control of a Robots Group to Search for the Maximum of an Unknown Scalar Physical Field
1 Introduction
2 Problem Statement and Basic Definitions
3 General Scheme and Main RGDC Algorithm Procedures
4 Software Implementation and Computational Experiment
5 Conclusion
References
Position Control of UGV Group for COVID (Virus SARS-CoV-2COVID) Localization and Primary Treatment Within Indoor Environment
1 Introduction
2 Problem Statement
3 Mathematical Model for Robot Moving in Group
4 Software for Robots
5 Adaptation of Existing Service Robots’ Prototypes Towards COVID-Situation Activities
6 Conclusions
References
Mathematical and Computer Modeling of the Decision Making System
Using Diagrams Influence in Group Control SEMS
1 Introduction
2 Statement of the Control Problem
3 Options for Using Influence Diagrams When Making a Decision
4 Features of the Search for the Optimal Decision in Various Variants of Situational Control Structures
5 Conclusion
References
Models for Decision Making Support Systems in Robotics
1 Introduction
2 Forecasting Model
3 Adaptive Control Model
4 Conclusions
References
Coalition Game Model of FANET Grouping Control Based on the Method of Local Threats and Counter-Threats and Swarm-Leader Model
1 Introduction
2 Statement of the Problem of a FANET-System Coordinating Control Optimization
2.1 Coalition Structure
2.2 A Group Dynamics Model
2.3 A Vector Efficiency Indicator
3 General View of Waisbord—Zhukovsky Sufficient LTCT-Optimality Conditions
4 Modified Sufficient Conditions for LTCT
5 Optimization Method
6 Conclusion
References
Robotic Wheelchair Control System for Multimodal Interfaces Based on a Symbolic Model of the World
1 Introduction
2 Basic Architecture of the System
2.1 Representation of Semantic Map
2.2 Low-Level Components
2.3 Sensory Subsystem
2.4 Types of User Interfaces
2.5 Manipulator
3 Testing
4 Results and Discussion
5 Conclusion
References
Methods and Principles of Information Processing
Classification of Images in Decision Making in the Central Nervous System of SEMS
1 Introduction
2 Statement of the Problem of Inductive Formation of Images
3 Algorithms for the Formation of Decision Rules
4 Logical-Probabilistic and Logical-Linguistic Algorithms
5 Testing Algorithms
6 Conclusion
References
Image Classification System in the SEMS Selection Environment
1 Introduction
2 The Block Diagram of the System
3 The Principle of Operation of the System
4 Conclusion
References
Principles of Forming the Language of Sensation for Decision Making in the Central Nervous System of SEMS
1 Introduction
2 Algorithm of Formation of the Language of Sensations of the Robot
2.1 Quantization of the Surrounding Space
2.2 Fuzzification of Sensory Information
2.3 Image Formation in the Display of the Surrounding Space
2.4 Formation of Images by Combining Images from Different Senses
3 Conclusion
References
Generation of Control Commands in the Group SEMS with Multi-Channel Optic-Electronic Sensors
1 Introduction
2 Specific Features of the Commands Generation to Control the System for Shape Adaptation of the Radio Telescope’s Composite Mirror Using a Multi-Channel Optic-Electronic Sensor
2.1 Structure of the Robotic System for Aligning the Position of the Composite Mirror Parts in the Millimetron Radio Telescope
2.2 Diagram of the Multi-Channel Optic-Electronic Sensor for the Head Hexapod
2.3 Sequence of the Control Command Generation Using Optic and Electronic Multi-Channel Sensor
3 Computer Models for the Control Commands Generation
3.1 Coordinate Systems Which Determine the Position of Sensor Components
3.2 Calculating the Image Coordinates of the Radiation Mark on the Receiver
3.3 General Algorithm for Modelling the Optic-Electronic Multi-Channel Sensor
3.4 Calculating the Required Quantity of Referent Radiation Marks for Accurate Determination of the Base Unit’s Shift
3.5 Calculating the Required Quantity of Radiation Marks for Accurate Determination of the Reflecting Lobe’s Position
4 Conclusion
References
Increasing the Reliability of Decision Making by Improving the Characteristics of Optoelectronic Channels Ensuring the Separation of Complex Shape Fruit
1 Introduction
2 Generalized OCSS of Complex-Shape Fruits
3 Specificity of Receiving and Processing Information in OCSS
3.1 Images Pre-processing in OCSS
3.2 Features Extraction
3.3 Fruit Quality Classification
4 Application of Interframe Difference Algorithm in OCSS
5 Conclusion
References

Citation preview

Studies in Systems, Decision and Control 352

Andrey E. Gorodetskiy Irina L. Tarasova   Editors

Smart Electromechanical Systems Behavioral Decision Making

Studies in Systems, Decision and Control Volume 352

Series Editor Janusz Kacprzyk, Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland

The series “Studies in Systems, Decision and Control” (SSDC) covers both new developments and advances, as well as the state of the art, in the various areas of broadly perceived systems, decision making and control–quickly, up to date and with a high quality. The intent is to cover the theory, applications, and perspectives on the state of the art and future developments relevant to systems, decision making, control, complex processes and related areas, as embedded in the fields of engineering, computer science, physics, economics, social and life sciences, as well as the paradigms and methodologies behind them. The series contains monographs, textbooks, lecture notes and edited volumes in systems, decision making and control spanning the areas of Cyber-Physical Systems, Autonomous Systems, Sensor Networks, Control Systems, Energy Systems, Automotive Systems, Biological Systems, Vehicular Networking and Connected Vehicles, Aerospace Systems, Automation, Manufacturing, Smart Grids, Nonlinear Systems, Power Systems, Robotics, Social Systems, Economic Systems and other. Of particular value to both the contributors and the readership are the short publication timeframe and the world-wide distribution and exposure which enable both a wide and rapid dissemination of research output. Indexed by SCOPUS, DBLP, WTI Frankfurt eG, zbMATH, SCImago. All books published in the series are submitted for consideration in Web of Science.

More information about this series at http://www.springer.com/series/13304

Andrey E. Gorodetskiy Irina L. Tarasova •

Editors

Smart Electromechanical Systems Behavioral Decision Making

With 80 Figures and 2 Tables

123

Editors Andrey E. Gorodetskiy Institute for Problems in Mechanical Engineering Russian Academy of Sciences Saint Petersburg, Russia

Irina L. Tarasova Institute for Problems in Mechanical Engineering Russian Academy of Sciences Saint Petersburg, Russia

ISSN 2198-4182 ISSN 2198-4190 (electronic) Studies in Systems, Decision and Control ISBN 978-3-030-68171-5 ISBN 978-3-030-68172-2 (eBook) https://doi.org/10.1007/978-3-030-68172-2 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

Smart electromechanical systems (SEMS) used in cyber physical systems (CPhS). Cyber physical systems the ability to integrate computing, communication and storage of information, monitoring and control of the physical world objects. The main tasks in the field of theory and practice CPhS are to ensure the efficiency, reliability and safety of functioning in real time. It is important to keep in mind that the behavior of the system is based on making decisions based on information received from the sensors of the Central nervous system (CNS) about the environment and its own state. The task of making a decision about the behavior of SEMS in a group interaction of several SEMS is much more complicated, since in this case additional information about the planned behavior of other members of the group is necessary. The purposes of the publication is to introduce the latest achievements of scientists of the Russian Academy of Sciences and leading universities in the theory and practice decision-making in SEMS the CNS, as well as familiarization with the development of methods for their design and modeling based on the principles of bionics, adaptability, intelligence and parallelism in information processing and computing. Topics of primary interest include, but are not limited to the following: Methods and Principles of Designing of Decision Making System of the SEMS; Methods and Principles of Designing of Decision Making System of the SEMS Group; Mathematical and Computer Modeling of the Decision Making System; Methods and Principles of Information Processing. This book is intended for students, scientists and engineers specializing in the field of smart electromechanical systems and robotics, and includes many scientific domains such as receipt, transfer and pre-treatment measurement information, decision making theory, control theory.

v

vi

Preface

We are grateful to many people for their support in writing this book. A list of their names cannot be provided here, but we are deeply grateful to all of them. Saint Petersburg, Russia November 2020

Andrey E. Gorodetskiy Irina L. Tarasova

Contents

Methods and Principles of Designing of Decision Making System of the SEMS Logical and Mathematical Method of Making Behavioral Decisions . . . Andrey E. Gorodetskiy and Irina L. Tarasova

3

Patterns in Intelligent Control Systems for Robotic Systems . . . . . . . . . Gennady P. Vinogradov

15

Using Binary Relationships in Decision Making . . . . . . . . . . . . . . . . . . . Andrey E. Gorodetskiy, Vugar G. Kurbanov, and Irina L. Tarasova

29

Decision-Making by the Autonomous Symbiotic Self-Relocating Massage Robot “Triangel” Based on SEMS After the Fall of Patient on Surface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sergey N. Sayapin Decision-Making by the Autonomous Symbiotic Self-Relocating Massage Robot “Octahedral Dodekapod” Based on SEMS During the Upper or Lower Limb Massage . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sergey N. Sayapin

37

53

Methods and Principles of Designing of Decision Making System of the SEMS Group Problems with Secure Control of SEMS Group . . . . . . . . . . . . . . . . . . . Andrey E. Gorodetskiy, Irina L. Tarasova, and A. Yu. Kuchmin Synthesis of Optimal Program Control for Synchronizing the Movements of a Group of SEMS Modules . . . . . . . . . . . . . . . . . . . . A. Yu. Kuchmin Task Scheduling Within Robots’ Collectives of Arbitrary Structures . . . Alexander Fridman and Boris A. Kulik

73

81 95

vii

viii

Contents

Meta-Heuristic Algorithm for Decentralized Control of a Robots Group to Search for the Maximum of an Unknown Scalar Physical Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Anatoliy P. Karpenko and Inna A. Kuzmina Position Control of UGV Group for COVID (Virus SARS-CoV2COVID) Localization and Primary Treatment Within Indoor Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 I. L. Ermolov, M. M. Knyazkov, S. A. Sobolnikov, A. N. Sukhanov, and V. M. Usov Mathematical and Computer Modeling of the Decision Making System Using Diagrams Influence in Group Control SEMS . . . . . . . . . . . . . . . . 129 Andrey E. Gorodetskiy and Irina L. Tarasova Models for Decision Making Support Systems in Robotics . . . . . . . . . . . 145 I. L. Ermolov, S. S. Graskin, and S. P. Khripunov Coalition Game Model of FANET Grouping Control Based on the Method of Local Threats and Counter-Threats and Swarm-Leader Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Evgeny M. Voronov, Vladimir A. Serov, and Dmitry A. Kozlov Robotic Wheelchair Control System for Multimodal Interfaces Based on a Symbolic Model of the World . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 P. S. Sorokoumov, M. A. Rovbo, A. D. Moscowsky, and A. A. Malyshev Methods and Principles of Information Processing Classification of Images in Decision Making in the Central Nervous System of SEMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Andrey E. Gorodetskiy, Irina L. Tarasova, and Vugar G. Kurbanov Image Classification System in the SEMS Selection Environment . . . . . 197 Andrey E. Gorodetskiy and Irina L. Tarasova Principles of Forming the Language of Sensation for Decision Making in the Central Nervous System of SEMS . . . . . . . . . . . . . . . . . . . . . . . . 201 Andrey E. Gorodetskiy, Irina L. Tarasova, and Vugar G. Kurbanov Generation of Control Commands in the Group SEMS with Multi-Channel Optic-Electronic Sensors . . . . . . . . . . . . . . . . . . . . . 211 Igor A. Konyakhin and Minh Hoa Tong

Contents

ix

Increasing the Reliability of Decision Making by Improving the Characteristics of Optoelectronic Channels Ensuring the Separation of Complex Shape Fruit . . . . . . . . . . . . . . . . . . . . . . . . . 229 Ba Minh Dinh, Aleksandr N. Timofeev, Igor A. Konyakhin, and Valery V. Korotaev

Methods and Principles of Designing of Decision Making System of the SEMS

Logical and Mathematical Method of Making Behavioral Decisions Andrey E. Gorodetskiy and Irina L. Tarasova

Abstract Problem statement: currently, for the creation of robotic systems based on modules of Smart Electro Mechanical Systems SEMS, the development of systems that contribute to the endowment of robots with the ability to independently, without human intervention, formulate tasks and successfully perform them is important. To do this, they must be equipped not only with more advanced sensors of sensations (sensors), but also have the ability to understand the language of sensations, i.e. have feelings such as “friend–stranger”, “dangerous–safe”, “loved–unloved”, “nice– unpleasant”, etc. Purpose of research: search for ways to develop Central Nervous System of the Robot, ensuring that after collecting numerical information from the robot’s sensory system, the robot’s language of sensations is formed and optimal decisions are made about expedient behavior. Results: the stages of forming behavioral decisions based on logical and mathematical processing of sensory information are analyzed. The solutions of optimization problems with constraints are analyzed using mathematical programming methods, as well as using the method of habitability situation, when the desired solution is replaced by an analog, similar to a human. When solving such problems, situations are described by systems of equations in algebra modulo two and decisions about optimality are made based on the concept of sequential preference of one of the compared options to another. Practical significance: The proposed methods of forming the language of sensations and the principles of deductive, inductive and abduction decision-making based on information from the Central Nervous System of the Robot using algebraization and matrix solution of systems of logical equations can be effectively applied in the formation of strategies and tactics for control intelligent robots in conditions of incomplete certainty.

A. E. Gorodetskiy (B) · I. L. Tarasova Institute for Problems in Mechanical Engineering of the Russian Academy of Sciences, (IPME RAS), V.O., Bolshoj pr., 61, St. Petersburg 199178, Russia e-mail: [email protected] I. L. Tarasova e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_1

3

4

A. E. Gorodetskiy and I. L. Tarasova

Keywords Smart Electro Mechanical Systems · SEMS · Sensors · Central nervous system · Robot · The language of sensations · Decision making · Expedient behavior

1 Introduction The functioning of the automatic control system (ACS) of the robot relies on information from sensor systems regarding the environment and the state of the robot itself. However, in order for robots created on the basis of SEMS modules to be able to independently formulate tasks and perform them successfully, without human intervention, they must be equipped not only with more advanced sensory sensors (sensors), but also have the ability to understand the language of sensations. That is, to have feelings such as “friend–alien”, “dangerous–safe”, “loved–unloved”, “pleasant– unpleasant”, etc. In the presence of such abilities in the central nervous system of the robot (CNSR), it becomes possible to independently decision-making regarding expedient behavior [1, 2]. In particular, as a result of solving systems of logical equations formed on the basis of the language of sensations, robots can acquire the ability for reflexive and conscious reasoning.

2 Stages of the Formation of Behavioral Decisions After collecting numerical information from the sensor system of the robot, it becomes possible to proceed to the formation of the language of sensations of the robot. In this case, it is necessary to perform the following steps (operations), which are included in the software of the robot: – Fuzzification of numerical data received from the sensor system, i.e. obtaining high-quality data of a logical type; – Selection of images based on the combination of quality data using of logical inference rules; – Formation of binary evaluations of images of the type “dangerous–not dangerous”, “strong–weak”, “bad–good”, etc. on the basis of solving systems of logical equations that form binary relations; – Formation of reflective reasoning based on logical analysis of binary evaluations of images in the environment of the robot; – Formation of goals for the functioning of the robot based on the choice of reflective reasoning corresponding to the maxima (minima) of the used quality criteria; – Decision-making about expedient behavior to achieve the formed goals based on solving optimization problems with constraints.

Logical and Mathematical Method of Making Behavioral Decisions

5

3 Fuzzification Data and Forming Images Operation fizzifikatsii numerical data is widely used in intelligent control systems, [3] and in the intelligent robot control systems. For example, when forming databases of expert regulators [4]. After performing the fuzzification operation, sets X i are formed for each sensor measuring channel, containing sets of logical variables x ij . For example, for the channel for measuring the brightness of the image, you can obtain the following logical variables: x 11 —“very dark”, x 12 —“dark”, x 13 —“semi-dark”, x 14 — “semi-light”, x 15 —“light”, x 16 —“semi-bright”, x 17 —“bright”, x 18 —“very bright”. The obtained logical variables for various points of the space surrounding the robot can be true (x ijk = 1) or false (x ijn = 0). In this case, often there may be a situation where when fuzzification numerical data about the truth or falsity of the received one or other logical variables can only say with some confidence. In this case, each obtained logical variable x ijk is supplied with a corresponding attribute in the form of probability values P{x ijk = 1} or membership function μ(x ijk ) [5], which are stored in the database along with them. In addition, together with logical variables are stored in the database the coordinates of points of the surrounding space of the robot corresponding to each logical variable. Fuzzification of data coming through the sensory information channels of the CNSR from various sensors is an important operation for further logical constructions when decision-making. After fuzzification data possible imaging to select the environment and their classification. The operation of selecting images in the space surrounding the robot is widely used in the systems of technical vision of intelligent robots [6]. In this setting, this operation, in the simplest case, is reduced to combining into one set M i those points in space that have the same set of logical variables with the same attributes and provided that the distance to the nearest neighboring point with the same parameters does not exceed some predetermined value. In this case, the coordinates of the center of gravity of the obtained images are also determined. After combining points into sets, the latter can obtain additional qualitative parameters in the form of logical variables yij obtained after, for example, analyzing the geometric parameters of these images (areas, volumes, contours, etc.). These additional parameters: y11 —“large volume”, y21 —“smooth contour”, etc. are entered into the database in the section “set of images” together with other logical parameters of sets and coordinates of their centers of gravity. The contents of this section of the database are also updated when changing the environment of the robot. When situations of incomplete certainty arise in the process of combining points in space into a set (image) due to, for example, the probabilistic attributes of logical variables, it is necessary, in addition to geometric measures of proximity of points, to introduce additional measures of proximity, such as the admissible spread of values of the probability of logical variables in neighboring points. Formation of binary estimates of images is carried out by logical analysis of the parameters of images. To do this, you first need to draw up rules, such as “if– then”, assigning a given image of one or another binary assessment. For example,

6

A. E. Gorodetskiy and I. L. Tarasova

if the image is very bright, large and quickly moves towards the robot, then this image (object) is very dangerous. The system of such rules is entered into the CNSR knowledge base at the stage of robot creation. In some cases, it can be corrected during the operation of the robot through training or self-learning [7]. With a large number of such rules, it is advisable to reduce them to a system of algebraic equations modulo two or to the algebra of Zhegalkin’s logic [8]. In this case, we obtain matrix equations, the solution of which is easily parallelized by matrix processors. The latter allows you to dramatically speed up the logical analysis of the parameters of images. However, this approach will usually be obtained matrix system is very large dimension. In a real CNSR, not all components of logical equations (not all combinations of logical variables) are physically realizable and can be discarded. As a result of such a reduction, we obtain a matrix system of equations mod2 of a lower dimension [5]: C ∗ R = G,

(1)

gde: C ⊂ A, R ⊂ F, G ⊂ B, B is a binary vector of dimension n, F is a fundamental vector of a logical system of dimension n, built from combinations of logical variables obtained during fuzzification of sensory data, and supplemented by 1 in place of the last element, A is a rectangular binary matrix of dimension [n, m]. In addition, not every solution obtained from (1) a specific robot is feasible in the current environment (the state of functioning of the environment). This means that the solution obtained by the CNSR from (1) must satisfy the constraints, which can also be expressed in the form of systems of logical equations [8]: C i ∗ R i = H,

(2)

Cj ∗ Rj = D

(3)

where C i and C j are constraint matrices obtained by analogy with the matrix C, H and D are binary vectors obtained by analogy with the vector G, Ri ⊂ Fi and Rj ⊂ Fj . Many solutions obtained by the CNSR when solving Eqs. (1)–(3) will naturally lead to ambiguity in the behavior of the robot. A person in this situation behaves expediently or purposefully intuitively, relying on his own experience, or a genetically inherent behavioral stereotype. Therefore, the procedure for pattern recognition in the case of their representation in algebra modulo 2 requires the setting of rules or algorithms for processing the linguistic attribute part that characterizes logical variables when performing addition and multiplication operations on them mod 2. Linguistic attributes characterizing images form non-metrizable sets Bi . In this case, during recognition, the choice of the best class from the set of alternative ones can be based on the procedure for finding binary relations Bi g Bcj . Where Bcj is a set characterizing an ideal image from the class C jm under consideration, to which we want to get as close as possible, and g is a two-place predicate on the analyzed

Logical and Mathematical Method of Making Behavioral Decisions

7

sets, which can be given. For example, by specifying the formulas of a logicalmathematical language or by specifying a formalized linguistic expression [3]. In this case, the problem of identifying the best approximation is reduced to two tasks. The first is the task of obtaining the sets Bi , Bcj , and the second is the construction of an optimal procedure g that allows one to obtain a quantitative estimate of the proximity of Bi to Bcj . It is advisable to start creating the initial base for constructing g by selecting metrizable subsets in each of the compared sets (for example, subsets of decision probabilities), for whose elements relations and numerical measures of proximity can be specified. The next, most difficult step is to order the elements of non-metrizable subsets. It is very likely that to solve this problem, you will need to build a new system of logical equations, the solution of which will lead either to metrizable sets, or to ordered ones. In the first case, we immediately get numerical measures of proximity. In the second case, these measures will have to be built anew. As possible numerical estimates, the cardinality of sets, the number of matching elements, the number of groups of matching elements, and so on can be used. Any recommendations on the choice of these or other estimates can not be recommended at present due to the lack of knowledge of such models. Therefore, if it is impossible to order non-metrizable sets, the decision about the greatest proximity of any set to the standard should be made by the developer or operator himself, based on their preferences, experience and intuition [5]. The most commonly used and easily constructed binary functional relationships include the following: – estimation by the maximum deviation of power sets: 

xi −



i

yi = ,

(4)

i

where x i = 1 and yi = 1 for non-zero (non-empty) elements of the compared sets and, respectively, x i = 0 and yi = 0 for zero (empty) elements of the compared sets, and  is the numerical proximity estimate. – estimation based on the standard deviation of set capacities:  2  2      xi − yi = δ, i

(5)

i

where δ is the numerical estimate of proximity. – probabilistic estimation based on the maximum deviation of power sets:  i

P(xi = 1)xi −

 i

P(yi = 1)yi =  P ,

(6)

8

A. E. Gorodetskiy and I. L. Tarasova

where P (.) is the probability, and p is the numerical probability estimate of proximity. – probabilistic estimation based on the standard deviation of set capacities:  2  2      P(xi = 1)xi − P(yi = 1)yi = δ P , i

(7)

i

where δ p is a numerical probability estimate of proximity. Using these binary functional relationships makes it easy to rank Bi images by their proximity to Bcj standards, and at the same time allows you to enter a numerical proximity score.

4 The Adoption of Reflective Solutions The robot’s reflexive reasoning can be formed on the basis of logical analysis of binary evaluations of images in its environment. To do this, you need to create rules of the “if–then” type of reaction to a particular binary evaluation of the image, taking into account its location and the state of the robot itself. For example, (1) (2)

if the image is very dangerous and is located nearby, the robot must move away from it; if the image is very dangerous, is nearby and there is a large good image nearby, the robot must hide behind it.

These rules are compiled and entered into the knowledge base at the stage of creating the robot. There can be a lot of them and they can be adjusted during operation. At the same time, it is also advisable to bring them to the system of algebraic equations modulo two or to the algebra of logic Zhegalkin for parallelization of calculations. The program for translating a system of rules into algebraic equations modulo two should be included in the mathematical support of the CNSR. The formation of robot functioning goals based on the choice of reflexive reasoning obtained after analyzing binary evaluations of images surrounding the robot is a complex problem associated with solving poorly formalized multi-criteria optimization problems [9]. In this case, it is often necessary to choose not one specific goal, but a sequence of consecutive goals when the previous goals are successfully completed. At the design stage of the robot, it is impossible to foresee all the situations that the robot may be in when making a decision about choosing the purpose of functioning. Therefore, the robot’s memory is filled with possible situations based on the expected operating conditions and the corresponding possible goals with an index of their effectiveness. Then the CNSR should have such software that could, by evaluating acceptable reflexive reasoning and available suitable most effective goals for functioning in a given situation, create a sequence of goals that would provide the

Logical and Mathematical Method of Making Behavioral Decisions

9

maximum (minimum) quality criteria expressed numerically. The formation of such a quality criterion is a complex and time-consuming task, the solution of which is primarily associated with the formation and solution of a number of logical problems that lead to the formula for calculating the quality criterion [9]. When choosing optimal reflexive reasoning described by systems of logical equations in the Zhegalkin algebra, it is necessary to solve optimization problems with restrictions. In this case, the simplest solutions will be those where it is possible to build a scalar quality criterion, including from attributes of logical variables. In this case, the optimal search can be reduced to mathematical programming (MP) problems [10]. In the MP problem, we need to calculate an n-dimensional vector X that optimizes (converts to a maximum or minimum, depending on the content of the problem) the quality criterion of the solution f 0 (x), subject to the restrictions f j (x) ≤ uj , j = 1, 2, …, r, x ∈ G, where f j —known scalar functional, uj —given numbers, G—a predetermined set of n-dimensional space Rn . Thus, the MP task has the form: f 0 (x) → ext/ f j (x) ≤ u j ,

j = 1, 2, . . . , r, x ∈ G ⊆ R n .

(8)

For the probabilistic attribute part of logical variables in the specified systems of equations [9], the optimization goal can be to search for those identification rows of the matrix of the system of logical equations describing the solution that give the true values of logical functions yi with the maximum values of probabilities P{yi = 1}. Then the quality criterion can be expressed as follows: f 0 (Y ) =

n 

P{yi = 1} → max.

(9)

i=1

The probability values P{yi = 1} can be calculated approximately using the algorithm described in [9]. If the analysis of a complex CNSR reveals that the influence of certain components of yi on its behavior is different, then the quality criterion (9) should be given the form: f 0 (Y ) =

n 

βi P{yi = 1} → max,

(10)

i=1

where β i is the assigned weight coefficients. If the attribute part of logical variables in the specified systems of equations [9] contains membership functions, the optimization goal may be to search for those identification rows of the matrix of the system of logical equations describing the solution that give values of logical functions yi with the maximum values of their membership functions μ(yi ). Then the quality criterion can be expressed as follows:

10

A. E. Gorodetskiy and I. L. Tarasova

f 0 (Y ) =

n 

μ(yi ) → max

(11)

i=1

The values of the membership functions μ(yi ) can be calculated using the algorithms described in [9]. If the analysis of a complex CNSR reveals that the influence of certain components of yi on its behavior is different, then the quality criterion (11) should be given the form: f 0 (Y ) =

n 

βi μ(yi ) → max,

(12)

i=1

where β i is the assigned weight coefficients. If the attribute part of logical variables in the specified systems of equations [9] contains intervals [aji , bji ], then the following scalar functional can be used: J1 =

m  n  j

k ji (b ji − a ji ) → min,

n n    2 J2 = k ji (b ji − a ji ) − c ji → min, j

J3 =

m  n  j

J4 =

(14)

i

 2 k ji (b ji − a ji ) − (boji − a 0ji ) → min,

(15)

i

n m    b k ji (b ji − b0ji )2 + k aji (a ji − a 0ji )2 → min, j

(13)

i

(16)

i

where k ji , k bji , k aji —coefficients of preference of the decision-maker (DM) on optimality, cji —the desired DM interval width, b0ji , a 0ji —the desired DM interval boundaries. However, the recommendations available in various literatures [11–14] for calculating integrals of complex logical functions with known intervals of logical variables are still very contradictory and may give completely unacceptable results. This issue is discussed in more detail in [14]. The most acceptable results in solving this problem can be obtained using multi-step generalized programming [15] and software environments such as A-life [16].

Logical and Mathematical Method of Making Behavioral Decisions

11

5 Informed Decision Making After completing the formation of a sequence of goals for the robot’s functioning, it is necessary to make a decision about the appropriate behavior to achieve the formed goals. The task of providing robots with appropriate behavior skills is still at an early stage. Currently, the most fully studied problems of choosing optimal solutions in conditions of incomplete certainty of interval, probabilistic, or linguistic type [9]. The process of making a decision about appropriate behavior can be significantly accelerated by recognizing the formed M i images, i.e. assigning them to certain classes of C jm images containing the so-called ideal M i * images, for which the previously accepted optimal solutions are known (M i * ∈ C jm ). In this case, you can use the method of the situation of familiarity [17] (analogous to intuition in humans), i.e., replace the desired solution with an analog. A person in the process of thinking and making decisions based on the processing of available information usually adheres to one of two styles of deductive or inductive. There is also a third, poorly studied and rarely encountered type of thinking—the abductive. When using such approaches in the CNSR, it is necessary to solve a number of optimization problems with restrictions. The basis for solving these problems can be various methods of mathematical programming, mathematical programming in ordinal scales, generalized mathematical programming and multi-step generalized mathematical programming [10, 15]. A number of new approaches to solving optimization problems with interval uncertainty are described in [14]. When solving such optimization problems of solutions described by systems of equations in algebra modulo two, decisions about optimality can be made based on the concept of sequential preference of one of the compared options to another. When using this approach in the CNSR, it is advisable to set the acceptable set of alternatives not by inequalities, but by certain conditions of preference for the selected options. To solve such problems, you can generalize the scheme of mathematical programming, moving from quantitative scales to ordinal ones, i.e., moving from models that require the assignment of functions that define the goals and limitations of the problem to models that take into account the preferences of the persons involved in choosing the solution. This extends the range of applications of the theory of extreme problems and can be useful in a number of choice situations [3, 18]. The transition to problems of mathematical programming in ordinal scales, generalized mathematical programming and multi-step generalized mathematical programming is described in more detail in [5]. In this case, when choosing the optimal solution to a system of logical equations, part of the attributes of logical variables can be linguistic expressions that describe preferences in the form of, for example, score ratings formed based on the analysis of the opinions of decision makers. Moreover, there is a fundamental possibility of ordering preferences. In deductive decision-making, the process of thinking in the Central Nervous System begins at the global level and then moves down to the local level [19]. The technical equivalent of this type of thinking can be the optimization process, when first the best possible solution is found based on the available information, and

12

A. E. Gorodetskiy and I. L. Tarasova

then the solution is corrected by checking all the restrictions based on the available information. In this case, after calculating the quality criteria for all possible solutions, all the solutions found are ranked. The solutions are then checked for compliance with the restrictions, starting with the first one that has the highest quality criteria. In this case, the first of the tested solutions that meets the restrictions is considered optimal. In inductive decision-making, the process of thinking in the Central nervous system begins with the analysis of individual decisions and then the search for a common, global conclusion [19]. The technical equivalent of this type of thinking can be the optimization process, when first all solutions are checked for the feasibility of type constraints based on available information, and then the best solution is found out of all possible solutions based on type criteria. In abduction decision-making according to Peirce, cognitive activity in the Central nervous system is an interaction of induction, deduction and abduction [20, 21]. In this case, abduction makes the acceptance of plausible hypotheses by explaining the facts, with the help of induction, testing of the hypotheses put forward is implemented, and by deduction, consequences are deduced from the accepted hypotheses. A technical analogue of this type of thinking can be the process of searching for an optimal solution by analogy, when from all possible solutions, the solutions that are closest to the existing solutions stored in the CNSR database and that gave good results in the past are first selected using image recognition methods [22]. Then you can use deductive and/or inductive decision making methods to select the best quality criteria. Comparing the described decision making methods, we can conclude that the abduction method is the fastest by analogy with intuition, but its reliability depends on the completeness of the database of good decisions from past experience, i.e. it strongly depends on the time of operation of similar robots in similar environments. The deductive method is faster than the inductive method for a large number of constraints, since it does not require checking the constraints for all solutions. With complex quality criteria and a small number of restrictions, the inductive method can give a faster result, since it will reject the search for solutions based on complex quality criteria for solutions that are unacceptable by restrictions.

6 Conclusion The stages of forming decision-making in the CNSR based on the use of systems of equations modulo two or systems of logical equations in the Zhegalkin algebra are considered. Features of pattern recognition described by systems of equations in algebra modulo two are described. The article shows the effectiveness of using the habitual situation method in the CNSR, which is an analog of human intuition and allows replacing the desired solution with an analog. This dramatically increases the speed of formation of reflexive reasoning. When choosing optimal reflexive reasoning described by systems of logical equations in the Zhegalkin algebra, as well as when

Logical and Mathematical Method of Making Behavioral Decisions

13

making optimal decisions about appropriate behavior to achieve the formed goals, it is desirable to reduce the search for the optimum to well-studied problems of mathematical programming. If part of the attributes of logical variables in the system of equations of CNSR are linguistic expressions, the more natural way to choose the optimal solution is to switch to the concept of sequential preference of one of the compared options to another. To solve such problems, you can generalize the scheme of mathematical programming, moving from quantitative scales to ordinal ones, i.e., moving from models that require the assignment of functions that define the goals and limitations of the problem to models that take into account the preferences of the persons involved in choosing the solution. The proposed principles of deductive, inductive and abduction decision-making based on information from the Central nervous system using algebraization and matrix solution of systems of logical equations are effectively applied in the formation of strategies and tactics for managing intelligent robots in conditions of incomplete certainty. In this case, the fastest decision-making will be when using the abduction principle, which includes elements of deductive and inductive thinking. The credibility and reliability of decision-making in this approach can be improved during the operation of the robot, if you include in the control system elements of self-learning, adding to the base of selected good decisions that gave the right decisions in the past. Acknowledgements The present work was supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAA-A19119120290136-9 and is supported by grants RFBR No. 18-01-00076 and No. 19-08-00079.

References 1. Khakhalin, G.K.: Applied ontology in the language of hypergraphs. In: Proceedings of the Second All-Russian Conference with International Participation “Knowledge-OntologyTheory” (UMBRELLA-09), Novosibirsk, pp. 223–231 (2009) (in Russia) 2. Akoff, R., Emeri, F.: O celeustremlennyh sistemah [On Purposeful Systems], 269 pp. Sov. Radio Publ., Moscow (1974) (in Russia) 3. Gorodetskiy, A.: Osnovy teorii intellektual’nyh sistem upravleniya [Fundamentals of the Theory of Intelligent Control Systems], 313 pp. LAP LAMBERT Academic Publishing GmbH@Co. KG (2011) 4. Gorodetskiy, A.E., Erofeev, A.A.: Principy postroeniya intellektual’nyh sistem upravleniya podvizhnymi ob”ektami [Principles of building intelligent control systems for mobile objects]. Avtomat. i Telemekh./Automat. Telemech. 9, 1485–1491 (1997) (in Russian) 5. Gorodetskiy, A.E., Tarasova, I.L.: Nechetkoye matematicheskoye modelirovaniye plokho formalizuyemykh protsessov i system [Fuzzy Mathematical Modeling of Poorly Formalized Processes and Systems], 336 pp. Polytechnic University Publ., St. Petersburg (2010) (in Russian) 6. Moshkin, V.I., Petrov, A.A., Titov, V.S., Yakushenkov, Yu.G.: Tekhnicheskoe zrenie robotov [Technical Vision of Robots], 272 pp. Mashinostroenie Publ, Moscow (1990) (in Russian) 7. Nikolenko, S.I., Tulup’ev, A.L.: Samoobuchaiushchiesia sistemy [Self-Learning Systems], 288 pp. MTsNMO, Izdatel’stvo (Moskovskii tsentr nepreryvnogo matematicheskogo obrazovaniia) (2009) (in Russian)

14

A. E. Gorodetskiy and I. L. Tarasova

8. Gorodetskiy, A.E., Dubarenko, V.V., Erofeev, A.A.: Algebraicheskii podkhod k resheniiu zadach logicheskogo upravleniia [Algebraic approach to the solution of tasks of logical control]. Avtomat. i Telemekh./Automat. Telemech. 61(2), 295–305 (2000) (in Russian) 9. Gorodetskiy, A.E., Kurbanov, V.G., Tarasova, I.L.: Methods of synthesis of optimal intelligent control systems SEMS. In: Gorodetskiy, A.E. (ed.) Smart Electromechanical Systems, 277 pp. Springer International Publishing (2016). https://doi.org/10.1007/978-3-319-27547-5 10. Tabak, D., Kuo, B.: Optimal’noe upravlenie i matematicheskoe programmirovanie [Optimal Control and Mathematical Programming], 280 pp. Nauka Publ., Moscow (1975) (in Russian) 11. Alefel’d, G., Khertsberger, I.: Vvedenie v interval’nye vychisleniia [Introduction to the Interval Calculation], 360 pp. Mir Publ., Moscow (1987) (in Russia) 12. Levin, V.I.: Raschet dinamicheskikh protsessov v diskretnykh avtomatakh s neopredelennymi parametrami s pomoshch’iu nedeterministskoi beskonechnoznachnoi logiki [Calculation of dynamic processes in discrete machines with uncertain parameters using non-deterministic infinite-logic]. Kibern. Sist. Anal./Cybern. Syst. Anal. 3, 15–30 (1992) (in Russian) 13. Levin, V.I.: Nepreryvnaia logika i ee primenenie [Continuous logic and its application]. Inf. Tekhnol./Inf. Technol. 1, 17–21 (1997) (in Russian) 14. Kuchmin, A.Y.: Ob odnom metode nelineinogo programmirovaniia s proizvolnymi ogranicheniiami [A method for nonlinear programming with arbitrary constraints]. Informatsionno-upravliaiushchie sistemy/Inf. Control Syst. (2), 2–9 (2016) (in Russian) 15. Iudin, D.B.: Vychislitel’nye metody teorii priniatiia reshenii [Computational Methods of Decision Theory], 320 pp. Nauka Publ., Moscow (1989) (in Russian) 16. Gorodetskiy, A.E., Dubarenko, V.V., Tarasova, I.L., Shereverov, A.V.: Programmnye sredstva intellektual’nykh sistem [Software Intelligent Systems], 171 pp. Izdatel’stvo SPbGTU, St. Petersburg (2000) (in Russian) 17. Gorodetsky, A.E.: Fuzzy decision making in design on the basis of the habituality situation application. In: Reznik, L., Dimitrov, V., Kacprzyk, J. (eds.) Fuzzy Systems Design. Social and Engineering Applications, pp. 63–73. Physica-Verlag. A Springer-Verlag Company, New York (1998) 18. Gorodetskiy, A.E., Kurbanov, V.G., Tarasova, I.L.: Ergaticheskie metody analiza processov ekspluatacii i prinyatiya reshenij pri povrezhdeniyah i avariyah energoob”ektov [Ergatic methods of analysis of exploitation processes and decision-making in case of damage and accidents of power facilities]. Informatsionno-upravliaiushchie sistemy/Inf. Control Syst. 67(6), 29–36 (2013) (in Russian) 19. Kondakov, N.I.: Logicheskii slovar’-spravochnik [Logical Dictionary-Reference Book], 720 pp. Nauka Publ., Moscow (1975) (in Russian) 20. Vagin, V.N., Golovina, E.Yu., Zagoryansky, A.A., Fomin, M.V.: Dostovernyi i pravdopodobnyi vyvod v intellektual’nykh sistemakh [Reliable and plausible conclusion in intelligent systems]. In: Vagin, V.N., Pospelov, D.A. (eds.) 2nd Edition Revised and Enlarged, 712 pp. FIZMATLIT Publ. (2008) (in Russian) 21. Peirce, Ch.S.: Rassuzhdenie i logika veshchei [Reasoning and logic of things]. In: Lectures for the Cambridge Conference (1898) (in Russian) 22. Gorodetskiy, A.E.: Ob ispol’zovanii situacii privychnosti dlya uskoreniya prinyatiya resheniya v intellektual’nyh informacionno-izmeritel’nyh sistemah [The use of a situation accustomed to accelerate the adoption of intellectual solutions in information and measuring systems]. In: Physical Metrology: Theoretical and Applied Aspects, pp. 141–151. KN Publ., St. Petersburg (1996) (in Russian)

Patterns in Intelligent Control Systems for Robotic Systems Gennady P. Vinogradov

Abstract A robotic system is considered as a cyber-physical object. Purpose: giving a robotic system the property that guarantees the fulfillment of a certain mission in an uncertain and poorly formalized environment. Tasks: to consider how a robotic system affects a person while considering him an element of the robotic system control system (human in the loop control). To identify ways of transferring experience from a human to a machine with adaptive behavior. Method: to investigate a monitoring scheme for a cyber-physical system in order to identify behavioral patterns based on data obtained in real conditions. Results: there is an introduced concept of a typical situation. The proposed formal model of a behavioral pattern for a typical situation is considered as a human experience unit with a certain degree of confidence in obtaining the desired states. Behavior in a typical situation is associated with a choice in a purposeful state situation. There is a fuzzy description for a choice situation model. The author introduces value indicators of the purposeful state by a result and efficiency for a pattern. He also determines how an agent evaluates the purposeful state desirability by a result and effectiveness of its achievement in a situation of choice. It is proposed to model patterns with a limited natural language subset (case based reasoning). To implement the described approach, there is a developed software system that allows modeling the environment (context) and an agent’s behavior pattern from different points. There are four basic perception points for collecting and interpreting information to identify a behavioral pattern model. Experimental studies are carried out for relatively simple behavioral and cognitive pattern models when controlling an autonomous underwater vehicle, assessing the combat readiness of special reaction forces, and others. The implementation of the proposed procedures has resulted in models with synthesized: (a) intuitive understanding of the agent’s abilities, (b) direct observation of the agent’s work, and (c) researcher’s explicit knowledge in the agent’s subject domain. Practical significance is associated with solving the problem of computational complexity when creating systems with human-like behavior. G. P. Vinogradov (B) Tver Research Institute of Centerprogram Systems, 50 let Oktyabrya Avenue 5A, Tver 170024, Russia e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_2

15

16

G. P. Vinogradov

Keywords Decision making · Purposeful systems · Fuzzy judgement · Choice situation · Cyber-physical system

1 Introduction A cyber-physical object is an informationally connected set of physical components, onboard measurement systems, onboard executive systems, an onboard computer system, with implemented control algorithms, and a control station with displays and controls. Such object must have the self-sufficient behavior that guarantees the fulfillment of a certain mission. It is possible to achieve the desired increase in the effectiveness of such complexes in an undetermined and poorly formalized environment mainly by improving the intelligent component of their control system. However, it should be noted that the vast majority of research in this area remains at the theoretical level [1–10]. There is a gap between primitive behavioral models of artificial entities, for example, in swarm robotics, their interaction models and expectations from practice [11–13]. By now, it has become clear that it is possible to achieve the desired sharp increase in the efficiency of robotic systems, mainly by directing designers’ and scientists’ efforts to improve the control system intelligent component: (1) a set of algorithms for onboard control systems; (2) algorithms for the activities of the crew that controls a cyber-physical system. These components form the “cooperative intelligence” of a cyber-physical system, which allows creating a functionally integral object from a set of separate systems of onboard equipment aimed at performing the task of the current session of a functioning robotic system. An autonomous intelligent system (hereinafter referred to as an agent) showing human-like behavior is a system that includes the following components (Fig. 1):

A module for communicating with other systems

Onboard measuring systems

Tasks and obligations

Onboard intelligence (onboard computers, algorithmic support)

External environment (object domain) Fig. 1 An enlarged diagram of an intelligent autonomous system

Onboard execution systems

Patterns in Intelligent Control Systems for Robotic Systems

17

• onboard measuring devices (or a set of onboard measuring devices) that function as sensors which allow obtaining information about the environment state and their own state; • onboard execution units (or a set of onboard execution units) that function as effectors which help the system to affect the external environment and itself; • means of communication with other systems; • “onboard intelligence”, which can include onboard computers, their software, as well as control center operators that are the carrier of a set of algorithms for solving problems of the subject area obtained due to training and experience. Such system exists in time and space, interacts with other agents, with the environment when performing combat tasks and obligations using available mods of action. The agent performs the assigned tasks based on an understanding of his condition and subjective ideas about the state of the environment and the combat situation development, as well as information received through a communication module. The agent is able to predict changes in the environment affected his actions and evaluate their usefulness.

2 Requirements for the Autonomy and Intelligence of Combat Cyber-Physical Systems The role of automated systems when performing combat tasks should be considered from the standpoint of their impact on a human. They should help a commander by making his work easier and more efficient. At the same time, a commander must be an element of a control system (human in the loop control) of the cyber-physical system. Their interaction should ensure the experience transfer both from a human to the system and in the opposite direction, thereby providing the adaptive behavior. For example, the main difficulty for any autonomous system is the recognition of situations in the environment. The complexity and multiplicity of situations that arise during the mission performance make it impossible to identify them based on the results of multiple tests and form a knowledge base on their basis. Consequently, it is necessary to implement an additional monitoring scheme for the cyber-physical system to identify situation classes and successful modes of action in order to form behavioral models (patterns) based on data obtained in real conditions [14, 15]. This scheme guarantees a controlled evolution of self-sufficiency when solving tasks by combat units that include autonomous cyber-physical systems.

18

G. P. Vinogradov

3 Initial Assumptions and Hypotheses Usually, the situations that an autonomous system faces are difficult enough for their constructive formalization by traditional formal methods, but they are described well by natural language means. There also is their resolution experience and description, for example, by fuzzy logic means. The bearer of such experience is called a leader. Leaders share their experience through communication tools in the chosen language. Let us accept the hypothesis that human experience/behavior should be considered as a function of the interaction between a situation and a human. A situation can be interpreted as a component of the cause for its subjective reflection in a person. A person chooses a certain behavior based on a subjective representation of a situation, influences a situation, and changes it. At the same time, the processes occurring in a human mind when performing certain actions lead to expanding his ability structure (knowledge, experience). A cyber-physical system behavior model should also take into account this phenomenon of mutual influence. With this approach, the concept of “typical situation” (TS) turned out to be constructive [16, 17]. This part of a cyber-physical system operation is a functionally closed and has a clearly defined meaningful purpose. It appears as a whole in various (real) sessions, being detailed in them according to the conditions and the available ways of resolving problematic subsituations arising in TS [11]. When a cyber-physical system is fully intellectualized, TS and the modes of action form an individual behavioral pattern as a reaction to it. A person, while mastering his experience, also aims to aggregate it by creating pattern models. Therefore, a pattern model should be considered as a unit of human experience, for which a person has a certain degree of confidence in obtaining the desired states in a situation similar to a typical one (cluster). V. Finn has shown that an ideal intelligent system should have 13 types of abilities. At the present stage, only a part of these abilities can be implemented and only in interaction with a person. For example, “this is a product of the sequence “goal-plan-action”, the ability to reflect, the ability to integrate knowledge, the ability to clarify unclear ideas, the ability to change the knowledge system when receiving new knowledge”. He notes that it is impossible to exclude a person from this mode. Therefore, an intelligent system for military purposes cannot be completely autonomous and must be considered as a partner human-machine system with a pattern as the unit of knowledge. Definition. A pattern is the result of the activity of a natural or artificial entity associated with an action, decision-making, its implementation, etc., which was carried out in the past and is considered as a template (sample) for repeated actions or as a justification for actions according to this pattern.

Patterns in Intelligent Control Systems for Robotic Systems

19

4 The Model of a Behavioral Pattern Fuzzy Description Behavior in TS is associated with a choice that occurs in a purposeful state situation [12]. Let us consider a behavioral model in the form of a fuzzy description of a choice situation model. It is proposed to build a possible variant of such construction using “paradigm grafting” of ideas from other sciences, for example [12, 18]. A purposeful state consists of the following components: • a subject who making the choice (agent), k ∈ K . • choice environment (S), which is a set of elements and their essential properties, a change in any of which can cause or produce a change in a purposeful choice state. Some of these elements may not be system elements and form an external environment for it. The impact of the external environment is described using a set of variables. • Available modes of actions ckj ∈ C k , j = 1, n of the k-th agent that are known to him and can be used to achieve the i-th result (also called alternatives). Each mode of this set has a set of parameters called control actions. • Possible results for environment S that are significant for an agent—oik ∈ O k , i = 1, m. The results are assessed using a set of parameters called the output parameters of a purposeful state situation. • A method for assessing the properties of the results obtained after choosing a mode of action. Obviously, the assessment of the result should reflect the result value for an agent and thus reflect their personality. • Constraints reflecting the requirements imposed by the choice situation on output variables and control actions. • A domain model, which is a set of relationships that describe the dependence of control actions, parameters and disturbances with output variables. • An agent constraint model. It is described in detail in [17]. Regardless of a constraint description type, we will assume that the agent has a certain degree of confidence about the possibility of changing a part of constraints towards expanding a set of possible choice options (alternatives). For the described components, let us introduce measures to assess the purposeful state. 1.

We will assume that the agent factors that are environ is able to distinguish  mental characteristics X k = xik , i = 1, N . The agent evaluates the influence of each factor using a linguistic variable μkx (xik ): xik →[0, 1]. Let us introduce a parameter for the agent to assess his situational awareness in a purposeful state situation  k k N k i=1 μx x i x i k Es =  N (1)  k . k i=1 μx x i We can define the following constraint:

20

G. P. Vinogradov

  σ k Es k ≥ σ0k ,

2.

where σ0k is a certain threshold level of agent’s awareness due to using his own information sources. We will assume that in order to describe the influence of the selected factors on the results oik , i = 1, m, the agent uses an approximation in the form of the following production rules: If x1 is Ark1 and if x2 is Ark2 and … and if x N is Ark N , then oik = f irk (x1 , x2 , . . . , x N ), r = 1, R, i = 1, m

(2)

where R is the number of production rules, r is the current production rule number, oik = f irk (x1 , x2 , . . . , x N ) is an explicit function that reflects the agent’s idea of the causal relationship of input factors with possible results for the r-th  k are fuzzy variables defined on X k = xik , i = 1, N . rule; Ari Mathematical models, a verbal description, graphs, tables, algorithms, etc. might be used as a function f irk (•). Since ckj it is a function of the external environment state parameters taken into account and system properties, a set of assumptions about their possible values forms a scenario of the possible state of the external environment, the system functionality. The implementation of scenarios, for example using the rules (2) allows forming an idea of possible results oik . The ambiguity in choosing a mode of action might be described as the degree of confidence in the need to apply it to obtain a result oik . This estimate can be described by the linguistic variable   ψ kj = ψ kj ckj ∈ C k |si ∈ S → oik ∈ [0, 1].

3.

This measure is agent’s individual characteristic, which can change after training and gaining experience, as well as a result of the communication interaction of agents with each other and with an operator. Therefore, ψ kj =   ψ kj ckj ∈ C k |si ∈ S, I k → oik ∈ [0, 1], where I k is the information available to the agent at the time point tk . Choosing a mode of action ckj when the agent makes a decision in a purposeful state situation to achieve a result oik is associated with building a quantitative assessment of the chosen solution properties, as shown in [12]. The list of properties and parameters is based on experience, knowledge, intelligence and the depth of his understanding of a decision-making situation. A correct description of the properties and parameters of a mode of action is one of the main conditions for the choice ckj will lead to the result oik . The choice of the list of properties and their parameters that characterize them depends on the agent (his personality). Let us represent the possible results

for a given environment for k k agent’s choice in the form oi ∈ oi j , j = 1, J , where oikj is a set of possible

Patterns in Intelligent Control Systems for Robotic Systems

4.

5.

21

results when choosing the j-th mode of action, i ∈ I is a set of results that the k-th agent takes into account. It’s obvious that oikj = oikj (si ), si ∈ S. The value of the oik results. Since oikj = oikj (si ) and si = S(ckj ), the value of the i-th type of result is estimated by the following linguistic variable ϕik (oik (ckj )) ∈ [0, 1]. The function ϕik (oik (ckj )) for the result oik will be a monotonic transformation, since ϕik (•) it translates the range of the function oik (ckj ) into the set of linguistic variable values. Since the base value of the linguistic variable corresponds to fuzzy variables, this transformation transfers the range of the function oik into the range of the base fuzzy variables. Considering the result, the effectiveness of a mode of action is the confidence in obtaining this result using this mode of action at the known (or estimated) costs for its implementation. The confidence degree E ikj that a certain mode of action ckj will lead to a result oik in the environment S if the agent chooses it: E ikj = E ikj (oik |A choses ckj in S) ∈ [0, 1]. It is a linguistic variable that expresses the agent’s individual assessment of the consequences of choice in terms of costs.

5 A Model for Choosing an Agent When Implementing a Pattern The three linguistic variables μik (xik ), ψikj , E ikj introduced above form a model of the agent’s ideas about the purposeful choice situation. Since ckj it can be described in terms X ik and the agent has an idea of the dependence in the form of a rule base that links ckj the value of the possible i-th result oik , it is possible to determine the value of the purposeful state by the i-th result oik for the k-th agent according to the rule [6, 17]:  Eϕik =

  k k k o ϕ (c ) • oikj (s k ) j∈J i j ij j   .  k k k j∈J ϕi j oi j (c j )

In a similar way, we can assess the purposeful state value for the k-th agent by the efficiency for the i-th type of result:  E E ik =

j∈J

  E E ikj oik (ckj ) • ψik (ckj ) .  k k j∈J ψi (c j )

The agent’s assessment of the desirability of a purposeful state by the i-th result and the effectiveness of its achievement in a choice situation is given in the form of a linguistic variable [19] k k = χ1k (Eϕik ) ∈ [0, 1], χi2 = χ2k (E E ik ) ∈ [0, 1] χi1

(3)

22

G. P. Vinogradov

We can define the following restrictions: 

k χi1 (Eϕik ) ≥ χ10 and

i



k χi2 (E E ik ) ≥ χ20

i

where χ10 and χ20 are the agent’s expectations from the mission that reflect the balance between costs and achieved results oik . The model of the agent’s choice situation in TS is the set of structural and functional properties that (in his opinion) the choice situation has and which affect his satisfaction or dissatisfaction with the situation. There is another group of factors that determine the result implementation: will, risk proneness, self-esteem, motivation. These factors make it possible to talk about such indicator as confidence ρik (oik ) in obtaining a result oik in a situation of choice when using one of the possible modes of action ckj ∈ C k . Based on the hypothesis of rational behavior, the agent forms a decision according to the rule ⎛ ⎞  Pik (s ∈ S) = Arg max⎝ Eϕi (oik (ckj )) − E E ik (oik (ckj ))⎠ ckj

σ

i  k

k

(Iti ),

j∈J

∈C ⊆ M, oik ∈ O k   k k χi1 (Eϕik ) ≥ χ10 , χi2 (E E ik ) ≥ χ20

ckj



Iti

i

Es (X ) ≥ k

σ0k

(4)

Since the choice is related to the agent’s ideas about the choice situation, it is necessary to include the knowledge base (2) in (4). The relations (4) describe the agent’s (cyber-physical system) behavioral pattern when striving to achieve the i-th result. The agent considers (4) a pattern as a way of describing a problem, a principle and an algorithm for its solution, which often arises, and its solution might be used many times without reinventing anything. The value indicators of the purposeful state for the result Eϕik and the purposeful of the integral value indicator of the state value for the efficiency E E ik are elements  purposeful state for the k-th individual i Eϕik • E E ik . Given his confidence degree in obtaining a result ζik , an expected specific value indicator will be the following:   E Vk =

i

 Eϕik − E E ik • ζik  k i ζi

(5)

This means that if two subjects are in the same situation of choice, then the difference in their behavior should be manifested in specific value estimates by the result and effectiveness and in the degree of confidence in achieving the goal.

Patterns in Intelligent Control Systems for Robotic Systems

23

Fig. 2 The intelligent agent’s reasoning scheme (TOTE model)

The relationships (4–5) mean that when the agent wants to achieve some result, he has several alternative ways of achieving it with the methods of varying efficiency, and his confidence in obtaining the desired result is significant. Such model of autonomous agent’s individual behavior supposes forming a knowledge base by learning based on experimental experience, which makes it possible to implement the “cooperative intelligence” evolution due to an artificial cognitive process similar to that of natural entities [1, 20]. I should be noted that this capability is absent in knowledge-based systems, since it lacks a computer model of adaptive behavior. Thus, the general principles of agent’s reasoning are quite traditional and include the following three main phases (Fig. 2): • Perception—receiving data and building a scene model in a loaded world; • Cognition—analysis and forming a scenario of the subject’s actions to achieve the set goals; • Execution of the intended scenario with a constant comparison of expected and observed results. Unlike other similar systems, the system under consideration implements these phases through two basic mechanisms closely related to each other: abstraction and concretization.

6 Modeling Patterns. Basic Modeling Points Pattern modeling involves a limited natural language subset including modeling of case based reasoning, which forms a specific part of human experience—metaexperience. To implement the described approach, there is a developed software system that allows modeling the environment (context) and the agent’s behavior pattern from different points. We have selected four basic perception points for collecting and interpreting information in order to identify a behavioral pattern model.

24

G. P. Vinogradov

Fig. 3 A simplified representation of an avatar 3D model

They are: the first point (a person’s own point of view), the second point (situation perception from another person’s point of view), the third point (situation perception from an uninterested observer’s point of view), the fourth perception point implies considering the situation from the point of view of the involved system. Since we assume that each point uses different visions of a situation and of possible modes of action, the integration and coordination of viewpoints allows the agent to expand his understanding of the purposeful state situation and a behavioral pattern. Modeling from the first point assumes that a person with experience in fulfilling a mission implements it in the system independently and examines the pattern(s) used in this case. A testee shows his behavior by performing voice control of an “avatar” (see Fig. 3). Rectangles and the way of their positioning on the avatar are shown in red. The disadvantage of this method is that the accuracy of object recognition decreases, but at the same time, this method saves hardware resources and time for calculating intersections. This scheme for determining intersections will be used similarly to implement a hit in a fire contact situation. A testee performs actions in accordance with the scheme shown in Fig. 4. These functions are for sorting, and therefore for speeding up object processing. The implementation of the agent model visual function represents seeing objects through simple forms, for example, in this case they are cubes and their vertices, as well as ignoring objects that are not of value to the model, for example, walls and others. The eye is implemented as an empty object that is used as an endpoint for constructing a visual ray located at the head level. To be realistic, it will also be animated for the cases of head rotation during character animation.

Patterns in Intelligent Control Systems for Robotic Systems

25

Fig. 4 A reflexive approach scheme for identifying a behavioral pattern from the first position: is information flows

The entire visual part is reduced to 3 main functions: 1. determining whether the object is in sight; 2. determining the distance to the object; 3. constructing vectors from a simplified object model to an object responsible for agent’s eyes. The function of object detection in the eye visibility scope. The function make it possible to see those objects that are in the eye visibility scope, thereby reducing the cost of detailed processing of all objects. A schematic implementation of the scope is shown in Fig. 5. The agent’s location in the world is in blue. The viewing angle is 120°. The function of object detection within eyesight. Another function for sorting objects and saving calculation time is an area divided into priorities (see Fig. 5). The yellow and red priority zones will be selected if there are no objects in the green priority zone. Now these zones are 50 and 100 m, respectively. In the future, it might be improved and in terms of time consumption, the objects that are located farther from the eye may require longer focusing time. The function of object-eye intersection detection. The function works on the principle of finding the intersection between the points of the rough object model and an “eye”. A ray is built between two points; if the ray hits an object, an “eye” does not see this point. If an “eye” sees at least one point of the object, then the entire object is visible. The action pattern analysis is performed from the researcher’s point of view. It is important to emphasize that in order for the agent to describe already performed

26

G. P. Vinogradov

Fig. 5 The eye visibility scope green is high priority; objects in this area will always be selected. Now it is 20 m. Also in this zone, the objects will be named

activities according to his own pattern (Fig. 4), the subject in question must leave his previous activity point and move to a new point that is external in relation to both already performed actions and the future projected activity. This is called the first level reflection: considering agent’s previous position, his new point will be called the reflexive one, and the knowledge generated in it will be reflexive knowledge, since it is taken in relation to the knowledge developed in the first point. The above reflexive output scheme will be the first abstract model characteristic of reflection in general. The second position possibly assumes a full imitation of agent’s behavior, when a researcher tries to think and act as close as possible to the agent’s thoughts and actions using the model obtained in the first point. This approach allows understanding at an intuitive level the essential but unconscious aspects of the modeled agent’s thoughts and actions, thus to refine a model. Modeling from the third point is to observe the modeled agent’s behavior as a disinterested observer. The third point assumes constructing a model of a mode of action from the point of view of a specific scientific discipline related to the agent’s subject domain. The fourth position presupposes an intuitive synthesis of all received ideas in order to obtain a model with maximum values of specific value indicators by a result and efficiency. This approach involves implicit and explicit information. It is possible that the agent knows or understands the essence of some activity but is not able to perform it (conscious incompetence). Conversely, the agent is able to perform some actions well but does not understand the way to do them (unconscious competence). Having a perfect command of a skill implies both the ability “to do what you know” and the ability “to know what you do”. Nevertheless, many behavioral and psychological elements that ensure the success of agents’ actions remain unconscious and only intuitive. As a result, they are unable to describe the mechanisms of any abilities directly. Moreover, some agents deliberately avoid thinking about what they are doing and how they are doing it due to fearing that this knowledge will interfere with intuitive actions. Therefore, one of the modeling goals is to identify unconscious

Patterns in Intelligent Control Systems for Robotic Systems

27

competence and make it to conscious in order to understand it better, improve and transfer a skill. Cognitive and behavioral competence are modeled either “implicitly” or “explicitly”. Implicit modeling involves taking the second point in relation to the subject of modeling in order to achieve intuitive understanding of subjective experiences of a given person. Explicit modeling involves taking the third point in order to describe an explicit structure of the modeled agent’s experience so that it can be transmitted to others. Implicit modeling is primarily an inductive process for accepting and perceiving the structures of the surrounding world. Explicit modeling is a deductive process for describing and implementing this perception. Both processes are necessary for successful modeling. Without an implicit stage, there can be no effective intuitive base for building an explicit model. On the other hand, without an explicit phase, the modeled information cannot be translate into techniques or means and be transmitted to others. Implicit modeling itself helps a person develop personal, unconscious skill in relation to the desired behavior (this is how young children usually learn). However, creating a technique, mechanism or a skill that can be taught or transmitted to others, requires explicit modeling. Experimental studies involved relatively simple behavioral and cognitive patterns models, for example, when controlling an autonomous underwater vehicle, assessing the combat readiness of special reaction forces, and others. The implementation of the proposed procedures has resulted in models with synthesized: (a) intuitive understanding of the agent’s abilities, (b) direct observation of the agent’s work, and (c) researcher’s explicit knowledge in the agent’s subject domain.

7 Conclusion Intelligent technologies that use pattern theory have significant prospects, as they allow you to solve problems of computational complexity. The formal model of the behavior pattern for the Autonomous object management system, presented in the article, describes the mechanisms for forming subjective representations and assessments of the components of the choice situation, a choice model that takes into account motives and obligations. Building a model involves identifying the bearer of the most successful behavior model (leader). Obtaining and analyzing information for model identification is based on four positions of its perception and extraction. Getting information is based on an active experiment. The described approach was used in the design of a control system for a group of Autonomous uninhabited underwater vehicles for performing search and rescue missions. Acknowledgements The work has been financially supported by the Russian Foundation for Basic Research (RFBR), project No. 170100728.

28

G. P. Vinogradov

References 1. Gorodetskii, V.I.: Self-organization and multiagent systems: II. Applications and the development technology. J. Comput. Syst. Sci. Int. 51, 391–409 (2012) 2. ACL—Agent Communication Language. https://fipa.org/specs/fipa00061/SC00061G.pdf. Accessed 19 Jan 2016 3. AgentBuilder—An Integrated Software Toolkit That Allows Software Developers to Quickly Develop Intelligent Software Agents and Agent-Based Applications. https://www.agentbuider. com. Accessed 19 Jan 2016 4. Bernon, C., Gleizes, M.P., Peyruqueou, S., Picard, G.: ADELFE: a methodology for adaptive multi-agent systems engineering. In: Proceedings of 3rd International Workshop on Engineering Societies in the Agents World, pp. 156–169 (2002) 5. Bonomi, F., Milito, R., Zhu, J., Addepalli, S.: Fog computing and its role in the internet of things. In: Proceedings of the First Edition of the MCC Workshop on Mobile Cloud Computing (MCC 2012), pp. 13–16. New York (2012) 6. Burrafato, P., Cossentino, M.: Designing a multi-agent solution for a bookstore with PASSI methodology. In: Giorgini, P., Lesperance, Y., Wagner, G., Yu, E. (eds.) Proceedings of International Conference on Agent-Oriented Information Systems, pp. 119–135 (2002) 7. Caire, G., Coulier, W., Garijo, F.J., Gomez-Sanz, J.J.: Agent oriented analysis using MESSAGE/UML. In: Agent-Oriented Software Engineering II. LNCS, vol. 2222, pp. 119–125. Springer-Verlag (2002) 8. Cohen, P., Levesque, H.: Teamwork. Nous 25, 487–515 (1991) 9. Cougaar Agent Architecture. https://cougaar.org/wp/documentation/tutorials/. Accessed 19 Jan 2016 10. Deloach, S.: Analysis and design using MaSE and agentTool. In: Proceedings of 12th Midwest Artificial Intelligence and Cognitive Science Conference (MAICS). Miami University Press (2001) 11. Fedunov, B.E.: Constructive semantics for developing onboard intelligence algorithms for anthropocentric objects. J. Comput. Syst. Sci. Int. 5 (1998) 12. Vinogradov, G.P.: Modeling decision-making by an intelligent agent. Softw. Syst. 3, 45–51 (2010) 13. Vinogradov, G.P., Kuznetsov, V.N.: Modeling client behavior considering subjective ideas about the situation of choice. Artif. Intell. Decis. Mak. 3, 58–72 (2011) 14. Deloach, S., Garcia-Ojeda, J.: The O-MaSE methodology. In: Cossentino, M., Hilaire, V., Molesini, A., Seidita, V. (eds.) Handbook on Agent-Oriented Design Processes, pp. 253–285. Springer-Verlag, Berlin-Heidelberg (2014) 15. Filatova, N., Bodrina, N., Sidorov, K., Shemaev, P., Vinogradov, G.: Biotechnical system for the study of processes of increasing cognitive activity through emotional stimulation. In: Kovalev, S., Tarasov, V., Snasel, V., Sukhanov, A. (eds.) Proceedings of 4th International Scientific Conference IITI’19. Advances in Intelligent Systems and Computing, vol. 1156, pp. 548–558. Springer Nature Switzerland AG 2020 (2020) 16. Zadeh, L.: The concept of a linguistic variable and its application to approximate reasoning. Inf. Sci. 8, 199–251, 301–357 (1975); 9, 43–80 (1976) 17. Borisov, P.A., Vinogradov, G.P., Semenov, N.A.: Integration of neural network algorithms, nonlinear dynamics models, and fuzzy logic methods in prediction problems. J. Comput. Syst. Sci. Int. 1, 78–84 (2008) 18. Dilts, R.: Modeling with NLP. Meta Publ. (1998) 19. Vinogradov, G.P.: A subjective rational choice. In: JPCS 803 012176 (2017) 20. Gorodetskii, V.I.: Self-organization and multiagent systems: I. Models of multiagent selforganization. J. Comput. Syst. Sci. Int. 51, 256–281 (2012)

Using Binary Relationships in Decision Making Andrey E. Gorodetskiy, Vugar G. Kurbanov, and Irina L. Tarasova

Abstract Problem statement: the construction of adequate models of a complex system is an integral part of the process when searching for optimal or best solutions in situational control of a group of intelligent electromechanical robots (SEMS). In many cases, the behavior of the SEMS group in the environment of choice is not fully defined. One of the effective ways to overcome uncertainties is the use of various fuzzy mathematical models and binary relations in the search for the best solutions in group control systems. Purpose of research: development of methods for using binary relations to find the best solutions in situational management of the SEMS group. Results: mathematical methods for assessing the adequacy of the logicalinterval, logical-probabilistic and logical-linguistic SEMS models are considered; Methods for assessing the proximity of a decision to a reference solution based on the use of binary relations and mathematical programming, mathematical programming in ordinal scales, and generalized mathematical programming are proposed and analyzed. Practical significance: the research results can be used to assess the adequacy of various models for making optimal decisions in situational control of the SEMS group. Keywords Situational control · Smart electromechanical system · SEMS · Logical-probabilistic model · Logical-interval model · Logical-linguistic model · Adequate model · Binary relation · Decision making · Mathematical programming

A. E. Gorodetskiy (B) · V. G. Kurbanov · I. L. Tarasova Institute for Problems in Mechanical Engineering of the Russian Academy of Sciences (IPME RAS), V.O., Bolshoj pr., 61, St. Petersburg 199178, Russia e-mail: [email protected] V. G. Kurbanov e-mail: [email protected] I. L. Tarasova e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_3

29

30

A. E. Gorodetskiy et al.

1 Introduction Decision-making on control the behavior of a group of interacting SEMS as dynamic objects is determined by structural approaches to organizing situational control of a group of robots and the methods of situational control used [1]. In this case, management consists in making management decisions as problems arise when solving a group task in a dynamically changing environment of choice [2]. Such control can be attributed to the optimization problems of situational control [3, 4]. The quality of situational control in solving various problems of group control depends on the structural organization of control and on decision-making methods. Among the control structures, the following can be distinguished: decentralized without the allocation of a robot leader; decentralized with a leader, centralized with an operator, combined with an operator and without a leader and combined with an operator and a leader [5]. The choice of control type is determined by the available technical means and the type of group task to be solved. Moreover, the choice of a decision-making method for situational control of a SEMS group largely depends on the type of group control scheme.

2 The Tasks of Situational Control of a Group of Dynamic Objects A typical task of situational control is the problem of the optimal transition of a group of controlled objects from a certain initial variety of points of space to a finite variety, and the dimensions of these varieties can be arbitrary if the phase spaces of the controlled objects themselves are taken into account. It is completely obvious that in technical systems not only control parameters, but also the coordinates of control objects must obey certain physical restrictions. Traditionally, under optimal control, such objects control problems are considered: problems when each object can be described by a system of ordinary differential equations [6]; problems of optimal hit of objects in moving points of space, tasks when the movement of pursued objects (moving points of space) is not known in advance, and information about them comes only with time; tasks when the pursued objects are controllable and their movement is described by a system of differential equations [7, 8]. Finally, in situational control of intelligent systems with appropriate behavior, which include smart electromechanical systems SEMS [9], control can consist in choosing the best solution from a variety of alternative solutions with a fuzzy and not necessarily probabilistic or statistical description of the dynamics of control objects and the environment, and also with a non-scalar indicator of the quality of the control system. Such problems relate to decision theory [10], i.e. decision-making problems

Using Binary Relationships in Decision Making

31

on the optimality of the system. For cases where it is possible to indicate a scale— the objective function, the value of which determines the solution, the theory and methods of mathematical programming are known and well developed [11], which allow for a qualitative and numerical analysis of the clear-cut solution optimization problems that arise in this case. Taking into account the uncertainties that may arise when solving decision problems with a fuzzy mathematical description of complex systems, including the SEMS group operating in a poorly formalized environment; it is possible to use these mathematical programming methods with more or less success in these cases [12]. In the simplest decision-making situation, the Decision Maker (DM) pursues a single goal and this goal can be formally defined as a scalar function, i.e. quality criterion of choice. In this case, the values of the quality criterion can be obtained for any admissible set of argument values. It is also assumed that the domain of determination of the selection parameters is known, i.e. component of the selected vector, or, in any case, for any given point, it can be established whether it is an acceptable choice, i.e. Does it belong to the domain of determining the quality criterion for a solution. In such a situation, the problem of choosing a solution can be formalized and described by a Mathematical Programming model (MP). In other cases, one should use Mathematical Programming in Ordinal Scales (MPOS), Generalized Mathematical Programming (GMP), or Multi-step Problems of Generalized Mathematical Programming (MsPGMP) [13].

3 Generalized Description of the Task of Situational Control of the SEMS Group In general, the solution to the group problem of situational control of a group of SEMS is the synthesis of a search algorithm, i.e. an ordered set of
⊂  from a set of alternative combinations of controls U(t k ) of the best combinations of control laws of each member of the group from U(t k ) based on estimation of quality Q built with the system of preferences E and the environment of choice O(t k ) [12]:  ⊂ 2U × Q U , where 2U is the set of all U subsets, QU is the set of all quality estimations of (cortages from 2 to |U|), × is the sign of the Descartes multiplication. To make effective decisions in a particular situation in the environment of selection O(t k ) and the corresponding changes in a group of cooperating robots the best way, the people controlling computer programs, decision-makers, must have certain principles or rules that contain the fundamental requirements for effective control, the most important of which are expertise, ability to make decisions in the absence of precedents, the existence of link between situational variables, when all the factors of the situation are integrated, being a system and affecting each other, there is a

32

A. E. Gorodetskiy et al.

dual influence of factors, when situational factors have different and sometimes even contradictory characteristics [12]. To implement these principles, certain decision-making methods have been developed for situational control of dynamic objects [13]. Most often, situational management uses methods of system and situational analysis, factor and cross-factor analysis, genetic analysis, diagnostic method, expert-analytical method, methods of analogies, morphological analysis and decomposition, methods of simulation, game theory, etc. However, the greatest effect and quality of control are achieved when a system of methods is applied in a complex, which allows you to see the control object from all sides and avoid miscalculations. One of the effective methods in such a system can be decision-making based on the use of an equivalence relation, which has the properties of reflexivity, symmetry and transitivity [14]: Oi ∼ O  ,

(1)

where Oi is the evaluated solution and O is the reference solution. To evaluate the quality of decision-making in the ratio (1), you need to enter a measure of proximity δ ij of the decision to the reference one. Then, the binary relation will be used instead of the relation (1): Oi δ ij Oj .

4 Mathematical Methods for Using Binary Relations in Decision In the process of searching for the best decision to be made using binary relations, it is necessary to consistently solve the following basic mathematical problems, namely, determining the adequacy of mathematical models of control objects and the environment, constructing a set of accepted Oi , and reference O decisions, constructing a set of binary relations δ ij and calculating values characterizing these binary relations. When constructing a mathematical model, the researcher usually takes into account only the most significant factors for achieving the set control goals. At the same time, the adequacy of the model depends on the control goals and control quality criteria adopted for optimization. Building a perfectly adequate model is fundamentally impossible due to the practical impossibility of taking into account the infinite number of parameters of the original object. As a rule, the behavior of the SEMS group in the environment of choice is not fully defined. Therefore, when searching for optimal or best solutions for situational management of the SEMS group, fuzzy mathematical models of dynamic objects and functioning environments are usually used, among which one can distinguish: Logical-Interval (LIM), Logical-Probabilistic (LPM) and Logical-Linguistic (LLM)

Using Binary Relationships in Decision Making

33

[15]. Moreover, their logical and mathematical part can be written in the following form [16]: X (t + 1) = A ⊗ x(t) ⊕ B ⊗ u(t) ⊕ r Y (t) = C ⊗ x(t) ⊕ D ⊗ u(t) ⊕ h

 ,

(2)

where X(t) is the extended binary state vector. u(t)—input vector; Y(t)—output vector; r, h—0,1 vectors; A, B, C, D—0.1 matrices, ⊗—multiplication according to mod 2, ⊕—addition according to mod 2. Moreover, each component x i , uj , yk of vectors X, u, Y should be characterized by the corresponding values of their probabilities or membership functions or intervals, the calculations of which are described in detail in [15]. The process of assessing the adequacy of such models of complex systems in general form can be reduced to solving the problem of finding a binary relation gi , which is an element or a subset of the set G (gi ⊆ G) and which corresponds to the relation I i go I  when the constraints I i qi U i and I  qi U i (qi ⊆ Q, i = 1, 2, …, m), where I i and I  are mathematical models or images of the estimated i-th and reference model, G and Q are some fixed compact sets, go is the best binary ratio, and U i are the models or images given a priori restrictions. In this case, we can assume that the plans or strategies and tactics of building a model are admissible by the i-th constraint if the pair (I i , U i ) ∈ qi and the pair (I  , U i ) ∈ qi , and the plan or strategy and tactics of building the model are optimal if the pair (I i , I  ) ∈ g0 , q0 is the preference of the decision maker, the cardinality of the set go is minimal and the elements of the set are ordered according to some feature. The relations g and q can be expressed as a system of logical equations [13]: CG = E

(3)

or DQ = Y.

(4)

Vectors G and Q have dimension N and in the most general case can have N = 2n − 1 component of the form: < g1 , g2 , . . . , gn , g1 g2 , g1 g3 , . . . , gn−1 gn , g1 g2 g3 , . . . , gn−2 gn−1 gn , . . . , g1 g2 . . . gn−1 gn >, < q1 , q2 , . . . , qn , q1 q2 , q1 q3 , . . . , qn−1 qn , q1 q2 q3 , . . . , qn−2 qn−1 qn , . . . , q1 q2 . . . qn−1 qn > . The components gi of the vector G is logical variables that characterize the proximity of the objects and relations of the constructed model I i to the elements and relations of the ideal model I  . The qi components of the Q vector are logical variables that characterize the correspondence of objects and relations of the constructed model I i to the elements and relations of the U i constraint model.

34

A. E. Gorodetskiy et al.

Matrices C and D consist of identification strings ci and d i having the dimension of vectors G and Q and containing elements 0 and 1 in the specified order. For example, c1 = |0 0 1 1 . . . 0 1| Vector E has the dimension of vector G, its ei components can take the value 1 with some probabilities Pi (ei ). The vector Y has the dimension of the vector Q and its yi components can take the value 1 with some probabilities Pi (yi ). The values of these probabilities are calculated using the probabilities of the gi and qi . components. In this case, the probability values can be calculated approximately according to the algorithm described in [17]. Mathematical methods for solving problems of determining go are described in detail in [13]. After choosing an adequate model, it is necessary to start calculating the quantities characterizing the binary relations gij . In the process of solving this problem for SEMS group control systems, in which control objects are described by LPM, LLM or LIM, measures of the quality of decisions made are also set in the form of binary relations describing the preferences of the decision maker, in the form of, for example, point scores formed on the basis of analysis of the opinion of experts in a given area. Then the estimation of the proximity of the decision to the reference (optimal) one is reduced to the problem of mathematical programming in ordinal scales [10]. In contrast to the MPOS problem, optimization by the Generalized Mathematical Programming method corresponds to the choice of the decision to be made based on comparing its characteristics with the characteristics of an ideal solution, and not their parameters [10]. Mathematical methods for solving these problems are also described in detail in [13]. At the same time, the search for the optimal solution can be automated by artificial intelligence software.

5 Conclusion If the SEMS control system contains uncertainties of a Logical-Probabilistic, Logical-Linguistic or Logical-Interval type, the search for optimal control can be carried out using mathematical programming methods, when there is a fundamental possibility of constructing a scalar quality criterion, including from attributes (probabilities, membership functions or intervals) logical variables. In this case, the quality of optimization will mainly be determined by the correctness of constructing a binary relation describing the measure of the proximity of the designed SEMS control system to the ideal one. This can be a time consuming and complex task, often involving solving a number of logical problems. The quality of setting and solving these problems depends on the experience and skill of the developer as a decision maker. To increase objectivity in assessing the optimality, it is advisable to make the decision maker collective with the involvement of the customer in the work on building a binary relationship.

Using Binary Relationships in Decision Making

35

In a situational control system of a SEMS group without an operator, each SEMS must have a reference solution Oj and a measure of the proximity δ ij of the estimated solution to the reference one. Differences in Oj and δ ij for different SEMS will lead to collisions in group interaction. In addition, obtaining adequate Oj and objective δ ij , taking into account the no determinism of models of the environment of choice and/or incompleteness, can be very laborious and not always justified. Therefore, it is advisable to have an operator or a leader in the situational management system of the SEMS group. Acknowledgements The present work was supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAA-A19119120290136-9 and is supported by grants RFBR No. 18-01-00076 and No. 19-08-00079.

References 1. Gorodetskiy, A.E., Tarasova, I.L.: Situational control a group of robots based on SEMS. In: Gorodetskiy, A.E., Tarasova, I.L. (eds.) Smart Electromechanical Systems. Group Interaction, pp. 9–18. Springer International Publishing (2018). https://doi.org/10.1007/978-3-31999759-9_2 2. Gorodetskiy, A.E., Kurbanov, V.G., Tarasova, I.L.: Decision-making in central nervous system of a robot. Inf. Control Syst. 1, 21–30 (2019). https://doi.org/10.15217/issnl684-8853.2018. 1.21 3. Pospelov, D.A.: Situacionnoe upravlenie: Teoriya i praktika [Situation Control: Theory and Practice], 286 pp. Nauka Publ., Moscow (1986) (in Russian) 4. Kunc, G., O’Donnel, S.: Upravlenie: sistemnyj i situacionnyj analiz upravlencheskih funkcij [Control: System and Situation Analysis of Control Functions], 588 pp. Progress Publ., Moscow (2002) (in Russian) 5. Gorodetskiy, A.E., Tarasova, I.L., Kurbanov, V.G.: Situational control of the group interaction of mobile robots. In: Gorodetskiy, A.E., Tarasova, I.L. (eds.) Smart Electromechanical Systems. Situational Control, pp. 91–101. Springer International Publishing (2020). https://doi.org/10. 1007/978-3-030-32710-1_7 6. Gorodetskiy, A.E., Tarasova, I.L.: Upravlenie i nejronnye seti [Control and Neural Networks], 312 pp. Polytechnic University Publ., St. Petersburg (2005) (in Russian) 7. Vajsbord, E.M., Zhukovskij, V.I.: Vvedenie v differencial’nye igry neskol’kih lic i ih prilozhenie [Introduction to the Differential Game of Several Persons and Their Application], 304 pp. Sov. Radio Publ., Moscow (1980) (in Russian) 8. Ajzeks, R.: Differencial’nye igry [Differential Games], 479 pp. Mir Publ., Moscow (1967) (in Russian) 9. Gorodetskiy, A.E.: Smart electromechanical systems modules. In: Gorodetskiy, A.E. (ed.) Smart Electromechanical Systems, pp. 7–15. Springer International Publishing (2016). https:// doi.org/10.1007/978-3-319-27547-5_2 10. Iudin, D.B.: Vychislitel’nye metody teorii priniatiia reshenii [Computational Methods of Decision Theory], 320 pp. Nauka Publ., Moscow (1989) (in Russian) 11. Tabak, D., Kuo, B.: Optimal’noe upravlenie i matematicheskoe programmirovanie [Optimal Control and Mathematical Programming], 280 pp. Nauka Publ., Moscow (1975) (in Russian) 12. Gorodetskiy, A.E.: The principles of situational control SEMS group. In: Gorodetskiy, A.E., Tarasova, I.L. (eds.) Smart Electromechanical Systems. Situational Control, pp. 3–13. Springer International Publishing (2020). https://doi.org/10.1007/978-3-030-32710-1_1

36

A. E. Gorodetskiy et al.

13. Gorodetskiy, A.E., Kurbanov, V.G., Tarasova, I.L.: Methods of synthesis of optimal intelligent control systems SEMS. In: Gorodetskiy, A.E. (ed.) Smart Electromechanical Systems. Studies in Systems, Decision and Control, vol. 49, pp. 25–44. Springer International Publishing (2016). https://doi.org/10.1007/978-3-319-27547-5_4 14. Kostrikin, A.I.: Vvedenie v algebru [Introduction to Algebra], pp. 47–51. Nauka Publ., Moscow (1977) (in Russian) 15. Gorodetskiy, A.E., Tarasova, I.L.: Nechetkoye matematicheskoye modelirovaniye plokho formalizuyemykh protsessov i system [Fuzzy Mathematical Modeling of Poorly Formalized Processes and Systems], 336 pp. Polytechnic University Publ., St. Petersburg (2010) (in Russian) 16. Gorodetskiy, A.E., Tarasova, I.L., Shkodyrev, V.P.: Matematicheskoe modelirovanie intellektual’nyh sistem upravleniya: Modelirovanie determinirovannyh intellektual’nyh sistem upravleniya [Mathematical Modeling of Intelligent Control Systems: Modeling of Deterministic Intelligent Control Systems], 181 pp. Polytechnic University Publ., St. Petersburg (2016) (in Russian) 17. Gorodetsky, A.E., Dubarenko, V.V.: A combinatorial method for the evaluation of probabilities of complex Boolean functions. Zh. Vychisl. Mat. Mat. Fiz. 39:7, 1246 (1999); Comput. Math. Math. Phys. 39:7, 1201–1203 (1999) (in Russian)

Decision-Making by the Autonomous Symbiotic Self-Relocating Massage Robot “Triangel” Based on SEMS After the Fall of Patient on Surface Sergey N. Sayapin

Abstract Problem statement: in this article the essence of the symbiotic robot “Triangel” based on SEMS which comprises in its mechanical union with the patient’s body and with a single coordinate system, is shown. As a result, the symbiotic robot “Triangel” is able to perform massage manipulations of the patient’s back or chest not only in the passive position of the body, but also on the moving body, regardless of its spatial orientation. Also the patient can simultaneously massage its chest and back by two symbiotic robots “Triangel”. The symbiotic robot “Triangel” can function both autonomously and under the remote supervision of a massage therapist via the Internet. However, when moving the patient during the massage with the help of a symbiotic robot “Triangel”, he may fall, for example, tripping, slipping or as a result of loss of consciousness. Purpose of research: development of algorithms for decision-making by the symbiotic robot “Triangel” after a patient falls to the surface. Results: possible types of patient falling to the surface and their classification are shown. The decision-making model axiomatics and the algorithms developed on its basis for decision-making by single or two symbiotic robots “Triangel” installed simultaneously on the patient’s chest and back are presented. The decision-making algorithms are formed depending on the patient’s condition, for example in the following cases: the patient is conscious and does not need help; the patient is conscious and needs help; the patient is unconscious; the patient has died. Practical significance: decisions-making by the symbiotic robot “Triangel” both autonomously and under the remote control of the massage therapist in real time via the Internet will reduce the risk of a threat to the patient’s health or life after falling to the surface during the massage. Keywords SEMS · Autonomous symbiotic self-relocating massage robot · Decision making by single robot and with a leader operator S. N. Sayapin (B) Mechanical Engineering Research Institute of the Russian Academy of Sciences (IMASH RAN), Moscow, Russia e-mail: [email protected] Bauman Moscow State Technical University, Moscow, Russia © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_4

37

38

S. N. Sayapin

1 Introduction It is known that massage is currently considered as one of the most effective non-drug methods of disease prevention, treatment, rehabilitation, and fatigue relief. Therefore, now massage is used in clinics, hospitals, medical centers, as well as at home, etc. All types of massage are performed by the masseur or patient (self-massage) manually or using massage devices. To enhance the effect of traditional manual techniques, the masseur can use a sliding cupping massage (SCM) of the back and chest [1, 2]. To robotization SCM, in IMASH RAN a portable autonomous symbiotic selfrelocating planar parallel massage robot “Triangel” with 3 DOF (degrees of freedom) based on SEMS (Smart Electromechanical System) is developed [3–6]. The “Triangel” is the robot built on the basis of a new symbiotic approach, consisting in its mechanical discrete unification with the patient’s body. As a result of the symbiotic approach, the patient’s body becomes the carrier element of the parallel robot itself, and the massage can be performed regardless of the spatial position of the body, as well as when the patient moves indoors. However, if the massage is performed while walking, the patient may fall, for example, tripping, slipping, or as a result of loss of consciousness. One of the most effective ways to solve this problem is the intellectualization of the symbiotic massage robot “Triangel”. Therefore, it is important to develop axiomatics and algorithms for decision-making by “Triangel” in the case of a patient falling to the surface during SCM. The description of the symbiotic robot “Triangel”, main possible types of falling of the patient to the surface during SCM, and the axiomatics of the decision-making model and algorithms for decision-making by “Triangel” after the patient falls to the surface are presented below.

2 Description of Symbiotic Robot “Triangel” and Main Types of Possible Fallings of Patient During Massage 2.1 Description of Symbiotic Robot “Triangel” The essence of the symbiotic approach to creating the robot “Triangel” for SCM is to combine it with the body patient and single coordinate system and ensure the independent functioning of the robot “Triangel” and the patient during the SCM. Thanks to the symbiotic approach, the patient can walk, perform routine and other work, as well as fulfill physiological needs during SCM. The structural scheme of “Triangel” (a), the scheme of installation “Triangel” on back and massage movements for SCM (b) and example of vertical SCM of back by the full size pneumatic prototype of “Triangel” (c) are shown in Fig. 1. “Triangel” (Fig. 1a) contains a massage device in the form of active triangular module ABC, consisting of three identical bars, whose ends are hinged with the

Decision-Making by the Autonomous Symbiotic Self-Relocating …

39

Fig. 1 Structural scheme of “Triangel” (a), scheme of installation “Triangel” on back and massage movements for SCM (b), and vertical SCM of back by the full size pneumatic prototype of “Triangel” (c) (see text for explanation)

massage vacuum cups (MVC) 1. The active triangular rod parallel structure of “Triangel” with 3DOF provides with geometric form constancy and the work of the rods only for tension/compression, and as a result, high rigidity and low specific weight. Each of the rods of the ABC module is equipped with a linear drive 2 with sensors force 3, relative displacement 4 and relative speed 5. All MVC 1 are hermetically connected by flexible hoses 6 to the corresponding channels of the air dispenser 7. The air dispenser 7 is equipped with pressure sensors 8 and a vacuum pump 9, which are electrically connected to the control system 10 in the form of a neurocomputer 11 with analog-to-digital and digital-to-analog converters (ADC and DAC). The inputs of the control system 10 through the ADC data bus are connected to the ADC outputs 12, 13, 14, 15 of the force sensors 3, relative displacement sensors 4, relative speed sensors 5 and pressure sensors 8, respectively. The outputs of the control system 10 through the output data buses of the DAC 16 are connected to the inputs of the power amplifiers 17 and linear drives 2, the air dispenser 7 and the vacuum pump 9. The control system 10 provides real-time monitoring and control. “Triangel” has the miniature three-axis gyroscopes-accelerometers (not shown in Fig. 1) which are installed at the vertices of the ABC module. Also, the “Triangel” has a two-way

40

S. N. Sayapin

loudspeaker communication with the patient, as well as wireless communication with wrist devices for measuring pulse, blood pressure, arrhythmia, temperature, etc. (not shown in Fig. 1). The patient places the ABC module on the prepared surface of the his body and fixes it by vacuuming MVC 1 (Fig. 1b). After fixing the ABC module on the patient’s body, the patient–robot “Triangel” symbiosis is formed, whose symbionts can continue to function independently of each other. The symbiosis the patient–robot “Triangel” is preserved regardless of the patient’s movements and spatial orientation. The pressure in the MVC 1 is regulated using the air dispenser 7, pressure sensors 8 and the vacuum pump 9 by commands from the control system 10 so that two MVC 1 are hermetically fixed on the back motionless and third MVC 1—movable. Next, individual data of the patient are entered into the control system 10, which include the contours of massaged body parts associated with the basic coordinate system of the symbiosis, the coordinates of body parts that are not allowed for the SCM, and the coordinates of the centers of two non-removable MVC 1. The scheme of installing the robot “Triangel” on the back and manipulation movements of SCM for various diseasesare are shown in Fig. 1b: a—osteochondrosis of the spine and lumbago; b—pneumonia and bronchitis; c—myositis and lumbosacral radiculitis [1, 3–6]. The geometric immutability of the ABC module allows you to determine the coordinates of its vertices and control their movements by measuring lengths of all rods using sensors of the relative movements 4. All MVC 1 in the process of SCM alternately become movable and non-movable. With the help of the robot “Triangel”, you can perform local vacuum massage and massage by shifting and stretching muscle tissues (Fig. 1b) [1, 3–6]. To determine the kinematic and dynamic parameters of the robot “Triangel”, IMASH RAS produced and tested its full-sizes pneumatic prototype in the form of a triangular parallel mechanism with 3 DOF (Fig. 1c), made on the basis of disposable three-component syringes with a volume of 60 ml. Syringes were used as pneumatic linear drives and vacuum pumps, and syringe cases were used as MVC. Elastic tubes with roller clips from standard infusion systems were used as connecting flexible hoses. During the testing of the pneumatic prototype robot “Triangel”, an SCM was performed, including air suction from MVC and vertical movement of the pneumatic prototype of the robot “Triangel” along the massaged back (Fig. 1c). With the help of a pneumatic prototype of the robot “Triangel”, the real possibility of conducting SCM was demonstrated. The linear drives and vacuum pump of the pneumatic prototype of the robot “Triangel” were controlled manually. All images were captured using a Canon digital camera IXUS 990 15 [5, 6].

Decision-Making by the Autonomous Symbiotic Self-Relocating …

41

2.2 Description of Main Types of Possible Fallings of Patient to Surface A falling on the surface is a type of rotational movement of a vertically positioned body around a horizontal or sagittal axis that passes through the foot. The fallings of the patient to the surface during Autonomous back or chest massage by robot “Triangel” can occur as a result of a violation of the balance of the body from a standing position or when walking on an uneven or slippery surface, as well as with diseases of the bones and joints of the lower extremities or with neurovascular disorders accompanied by loss of consciousness. The classification of the main types of fallings to surface is shown in Fig. 2. Depending on how the patient falls (face down or on the side) and the position of the hands at the moment of impact, he may have the following injuries: abrasions, hematomas and wounds at the points of contact, fractures of the clavicle, humerus, ulnar process, forearm bones, leg bones, spinous processes of the vertebrae, ribs, etc. The most serious damage is caused from a impact to the head when falling flat on your back [7–9]. The falling of the patient to the surface may be uncoordinated or coordinated ones. For uncoordinated falling backwards is characterized by the presence of abrasions on the hands and hemorrhages in the soft tissues of the occipital area of the head. Such type of falling is the most traumatic, as body reaches the maximum level of rotational velocity and the effectiveness of the protective action of the exposed hands reduces. The impact of the body with the surface in an uncoordinated falling occurs

Fig. 2 Classification of fallings to the surface

42

S. N. Sayapin

Fig. 3 Computer diagram of uncoordinated backwards falling of the patient’s body with “Triangel” and wrist devices

begin the back of the head, shoulder blades, lower back, buttocks, and then the upper and lower limbs (Fig. 3) [8]. The location of the “Triangel” and the wrist devices on the patient’s chest and upper limb during the uncoordinated his fall to the surface are shown in Fig. 3 also. The coordinated backward falling is performed in a protective position (squat, head bent to the chest, grouped state of body parts, hands pointing back and to the sides), an attempt to turn the torso sideways with a turn in the same direction of the knee area to soften the impact contact (Fig. 4) [8].

2.3 Description of Main Types of Possible Falling of Patient to Stairs of Staircases In the case of a passive uncoordinated falling of a person on a flight of stairs after losing their balance, in most cases, the body turns relative to the fulcrum and simultaneously slips along it. This is followed by the separation of the body from the support and its rollover and flight with rotation until the impact, followed by rolling on the back or rolling over the head. The most traumatic is an uncoordinated falling of the patient’s body from a standing position, since this develops the maximum angular velocity in the head area (Fig. 5). In the passive uncoordinated backwards falling (Fig. 5a), the places of impact are the back of the head, shoulder blades, lower back and buttocks, as well as the lower and upper limbs. In case of an uncoordinated falling face down (Fig. 5b), injuries are localized primarily on the front surface of the upper body. In case the uncoordinated

Decision-Making by the Autonomous Symbiotic Self-Relocating …

43

Fig. 4 Computer diagram of coordinated backwards falling of the patient’s body with “Triangel” and wrist devices

Fig. 5 Computer diagrams of main types of patient’s fallings on the staircase (“Triangel” and wrist devices are not shown): uncoordinated falling backwards (a); uncoordinated falling face down (b); uncoordinated face-down fall from the lower steps (c), middle (d), and upper steps (b)

44

S. N. Sayapin

falling from a half-sitting position which caused by the patient falling to his knees in a semi-conscious state (not shown in Fig. 5), the amount of damage may be minimal, since the impact on the step occurs with less energy. When falling from the lower steps, the impact occurs with a flat surface of the staircase (Fig. 5c). When falling from the middle steps, the torso bends along the contour of the angle formed by the staircase and its flat surface (Fig. 5d). When falling from the upper steps, the body hits the inclined ribbed surface of the staircase and then slides down it (Fig. 5e). In the case of a coordinated falling, a defensive reaction may occur, in which the patient squats sharply while simultaneously bringing his hands palms forward or groups his body (not shown in Fig. 5). In the passive uncoordinated falling from the standing position to the onto his side (Fig. 6), after being unbalanced, the patient’s body leans down with increasing acceleration and hits the steps with the side surface of the trunk, the shoulder, and finally the head. After that, the body bounces and moves down with a turn on the back or on the stomach, followed by a repeated impact and stopping the movement. In the case of the coordinated falling (not shown in Fig. 6), the defensive reaction may occur, in which the following movements of the body parts occur: turning torso, squatting, bending head to the chest and pulling hands back with the palms down.

Fig. 6 Computer diagrams of the passive uncoordinated falling of the patient’s body with “Triangel” and wrist devices from the standing position to the onto his side

Decision-Making by the Autonomous Symbiotic Self-Relocating …

45

The symbiotic massage robot “Triangel” is a discrete mechanical connecting with the massaged area of the patient’s body. Due to this interaction, all the movements of the massaged area of the body coincide with the spatial transport motion of the robot itself. As result, data of the three-axis miniature gyroscopes-accelerometers, which are installed in the vertices of the “Triangel” and the wrist devices on the forearm (not shown in Fig. 1), it is possible to judge the trajectories of movements and accelerations of body parts, which after comparison with the model diagrams in Figs. 4, 5 and 6, can be used to judge the nature of the fallings and the patient’s condition. Such data allows the massage robot “Triangel” to make decisions that are most favorable and safe for the patient after his fall. The algorithms for decisionmaking by the symbiotic robot “Triangel” after a patient falls to the surface are presented below.

3 Decision-Making by the Autonomous Symbiotic Self-relocating Massage Robot “Triangel” After the Falling of Patient on Surface 3.1 The Axiomatics of the Decision-Making Model Decision making in artificial intelligence is called solving a problem by search and deduction (a method of reasoning focused on finding plausible explanatory hypotheses). The resulting solution to the problem is, with a certain degree of probability, the cause of it and is similar, for example, to the solution of a system of linear equations. A single intelligent robot must work on a generalized task that comes to it from a human (operator) or a higher-level control system, for example, in the case of decision-making by a group of robots with a leader. The intelligent capabilities of such a robot should ensure that the task is divided into a number of simple sequential operations and the choice of an algorithm for performing each of these operations. Thus, decision-making refers to the choice of the implemented algorithm for the robot to perform this work or separate operations into which it is divided. Therefore, the level of decision-making should be determined depending on the volume and degree of complexity of the given work [10]. To make decisions in multi-criteria optimization problems, we will use the utility function method according to Brahman [11]. The method represents a set of basic rules that determine the sequence and content of the necessary actions, and assumes the following logic of movement: 1. 2. 3.

formulate notions of “usefulness” and “payment for usefulness”; selection of private criteria of usefulness and of payment for usefulness; creating a rating matrix in the form of a table with rows as private criteria and columns as alternatives (preferences);

46

S. N. Sayapin

4.

converting non-metric criteria (if any) to quantitative indicators of the membership function; representation of usefulness and of payment for usefulness as functions of private criteria; calculating usefulness and of payment for usefulness for all alternatives (preferences) and building a selection field.

5. 6.

Under the usefulness of a technical system, we will understand a certain quantitative characteristic of the degree of fulfillment its functional purpose. The payment for usefulness reflects the costs that must be made in order to achieve a useful effect. At the same time, in some cases, it is advisable to reduce the payment for usefulness by partially reducing certain functional characteristics. In [12], a classification of types of decision-making procedures with many criteria is given, which distinguishes “a priori”, “a posteriori”, “adaptive” and mixed by the type of additional information used in these procedures. Procedures of a priori type are based on the assumption that it is possible to determine the best ratio between the requirements imposed by different criteria a priori (before the decision). It is obvious that such a possibility (as in the problem under consideration) it doesn’t always exist. In addition, the choice of a specific type of global goal function cannot be made in isolation from the task being solved. A posteriori procedures are usually associated with the presence of some system of hypotheses or axioms that must be checked for each specific decision-making situation. These axioms are, as it were, additional and are not included in the formal model of the problem. If checking the axiomatics leads to a positive result, it allows in some cases to build a mechanism for choosing the best one. The main advantage of a posteriori procedures in comparison with a priori ones is a clear definition of the conditions under which they can be used. Thus, to solve a multi-criteria decision-making problem, we choose a posteriori procedures. We construct the axiomatics of a posteriori procedures using the utility theory of Fishburne [13]. In this case, the system of preferences of the decision maker is represented as a binary relation set on a set of alternatives. By the binary relation on a set X, we mean the set R of ordered pairs (x, y), where x ∈ X, y ∈ X, i.e. some subset of the direct product X × X, which is usually called the complete binary relation on X. So, for example, if (x, y) ∈ R, then this can be written as xRy (x is in relation to R with y). We assume that the preferences of the decision maker are arranged as follows: for any two alternatives x and y, the decision maker can determine either x is better than y (y π x), or that y is better than x (x π y), or that x and y are equally good (y ≈ x). Here and further, the “π” symbol means strict preference; “~”—indifference; “≈”—equivalence and “⇔”—“then and only then”. The case of the relationship of indifference “~” when for the decision maker there is no any difference between x and y, or the decision maker is not sure about his preference for x or y or consider x and y are incomparable on the preference to be excluded for the reason that we consider the problem of choosing a single best alternative and you need to make assumptions about the preferences allow to hope for the possibility of solving this

Decision-Making by the Autonomous Symbiotic Self-Relocating …

47

problem. It is easy to see that if the individual alternatives are incomparable, the problem of choosing the best alternative may be unsolvable. In fact, it is assumed that any information necessary for selection can be obtained from the decision maker. However, it must also be assumed that the information received from the decision maker is consistent. So, if the decision maker prefers alternative x to alternative y, then it makes sense to require that this precludes the possibility of preference for y over x (i.e., the symmetry property). Transitivity is also a reasonable consistency criterion for individual preferences. For example, if x is preferable to y and y is preferable to z, then common sense assumes that x is preferable to z. In addition, the presence of a nontransitive cycle xRy, yRz, xRz on a set of three alternatives makes the problem of choosing the best alternative unsolvable. Thus, we conclude that for our multi-criteria problem to be solvable, the “be better” relation must be fairly close to weak ordering, i.e. the binary relation must satisfy the properties of asymmetry and transitivity. Here the property of asymmetry implies the property of non-reflexivity. Similarly, we will represent the relation “~” as an equivalence relation on the set X. It is easy to see that any scalar function u(x) given on X induces on it, in general, two binary relations-weak ordering and equivalence, which are constructed as follows: xπ y ⇔ u(x) < u(y) and x ∼ y ⇔ u(x) = u(y) The converse is also true: for weak ordering “π” and equivalence “~”, under some additional regularity assumptions, there is a function u(x) such that x π y and x ~ y imply u(x)< u(y) and u(x) = u(y), respectively. Thus, in this situation, the preferences of the decision maker are described by some scalar function u(x), which is unknown in advance. Taking into account this axiomatics, an algorithm for making decisions by the symbiotic massage robot “Triangel” in the event of a patient falling during a back or chest massage is developed, which is presented below.

3.2 Algorithm for Decision-Making by the Symbiotic Massage Robot “Triangel” After the Patient Falls to the Surface As noted above, “Triangel” allows the patient to move around any rooms and staircase while performing an Autonomous back or chest massage. At the same time, on the one hand, the patient receives additional comfort, and on the other hand, there is a danger of falling to the surface if walking inside the room or on staircase with simultaneous massage. In General, a solution is understood as a set of actions from the side of the decision maker to the object (system, complex, etc.) of management that allows you to bring this object to the desired state or achieve the goal set for it. At the same time, the decision-making process of the decision maker is characterized by the following features:

48

S. N. Sayapin

• the ability to choose from alternative options (if there is no alternative, then there is no choice and, therefore, there is no solution); • having a goal, since purposeless choice is not considered a solution; • the need for a volitional act from the decision maker. Currently, a large number of algorithms and models for making project and management decisions have been developed in relation to the most diverse areas of human activity [10–20]. However, there is no algorithm that can always be applied, regardless of what kind of problem the SEMS-based robot had to face, what caused it, what factors affect it, whether they are manageable, etc. Let’s consider what the first stage of the decision-making algorithm should be as applied to the symbiotic massage robot “Triangel” in the event of a patient falling to the surface during a back or chest massage. Analysis of publications has shown that most authors put the first stage of the decision-making algorithm to identify and analyze the problem situation. However, other authors believe that the entire process should begin with setting goals and objectives [10–20]. To implement the decision-making ability by the autonomous symbiotic massage robot “Triangel”, programs are installed in its control system 10 (Fig. 1a) that simulate various types and species of human fallings to the surface (Figs. 3, 4, 5 and 6), similar to the programs described in [7–9]. Also, before the SCM procedure, data on the patient’s weight and height are additionally entered into the control system 10 (Fig. 1a). Based on the global goal and taking into account the framing effect [18], the following decision-making algorithm was developed for the robot “Triangel” after the patient falls to the surface (Fig. 7), which includes three blocks similar to the algorithm described in [17]. The first block of the decision-making algorithm includes the following steps: • • • • •

situation analysis; goal-setting. The “situation analysis” stage includes two operations: problem identification; collection and analysis of necessary information. Problem identification includes the following actions of the robot “Triangel”:

1.

2. 3.

Establishing and registering the fact of the patient falling to the surface, characterized by accelerated movement of his body down. Registration of the fact of falling is carried out by the control system 10 “Triangle” (Fig. 1a) according to indications from gyroscopes-accelerometers installed in the vertices of the “Triangel” and wrist devices (not shown in Fig. 1). the SCM is terminated and all MVC 1 (Fig. 1) are made motionless. Determining the type of falling (Figs. 2, 3, 4, 5 and 6) and its species (coordinated or uncoordinated falling). Here, based on the indications of gyroscopesaccelerometers, the trajectory of the patient’s torso and arm is constructed, it is compared with the model body movements when a person falls, described in [7–9], and the type and species of falling is determined.

Decision-Making by the Autonomous Symbiotic Self-Relocating …

49

Fig. 7 Block diagram of the decision-making algorithm for the single robot “Triangel” with the leader operator

4.

Establishing the place of the patient’s fall using a GPS or GLONASS satellite navigation system.

Collection and analyzing the necessary information includes determining the patient’s pulse, pressure, respiratory rate, and body temperature, and based on them, taking into account the type and species of falling that characterize possible injuries and injuries of the patient, and forming an assessment of the patient’s condition after the falling. The specificity of the first block of the decision-making algorithm is that the “goal setting” stage and the “problem identification” operation are carried out almost in parallel and are very closely intertwined. As a result, in most cases it is difficult to determine the priority between goal-setting and problem identification. Therefore, in our case, the “goal-setting” stage is reduced to a global goal when making decisions by the robot “Triangel”, characterized by reducing the negative consequences of the possible falling of the patient to the surface. The second block of the decision-making algorithm includes the following steps: • developing alternatives; • development of criteria for selecting alternatives;

50

S. N. Sayapin

• choose one of the alternatives. To increase the efficiency of decision-making by the “Triangel” robot, alternative solutions to the problem were formed depending on the patient’s condition for the following cases: • • • •

the patient is conscious and does not need help; the patient is conscious and needs help; the patient is unconscious; the patient died. The third block of the decision-making algorithm includes the following steps:

• implementation of the selected alternative; • control and evaluation of resources. At the implementation stage, the selected alternative is implemented. Parallel to the implementation stage of the chosen alternative is the stage of control the evaluation of the results of the decision. Feedback is difficult to distinguish in any particular stage of the decision-making algorithm, since it functions both in separate stages and between them throughout the entire process. The selected alternatives and their corresponding actions are listed below. The choice of the alternative “the patient is conscious and does not need help” is made as a result of the following actions. If the patient is in a medical facility, data on the fact of falling and getting up independently, the type and species of falling with possible traumatic consequences, as well as data on pulse, pressure, temperature and respiratory rate are transmitted to the nurse on duty and the massage therapist. Then, via satellite voice communication with the patient, they are offered help. In case of refusal of assistance, the massage therapist decides to continue or terminate the SCM procedure in accordance with the rules established for him. If a patient requests help, the alternative “the patient is conscious and needs help” is selected, and the following actions are performed. Through GPS or GLONASS, the location coordinates are additionally sent to the nurse on duty of the medical institution and the appropriate specialists are sent to provide the patient with the necessary assistance. If the SCM was performed by the patient at home, the above information is transmitted via satellite to the rescue service and the massage therapist. Also, the patient’s condition is monitored using a loud-speaking satellite connection and data on the patient’s pulse, pressure, temperature, and respiratory rate, and oral recommendations are given to him and others (if they are present) about the necessary actions before the arrival of the doctor or the arrival of emergency help. The choice of the “the patient is unconscious” alternative is made in the case of an uncoordinated species of falling, the absence of subsequent body movements, and the presence of breathing and pulse. If the patient is in a medical facility, data on the fact of the falling, the type and species of falling with possible traumatic consequences, as well as data on pulse, pressure, temperature and respiratory rate are transmitted

Decision-Making by the Autonomous Symbiotic Self-Relocating …

51

to the nurse on duty and the massage therapist. Further, attempts are made to establish voice contact with the patient via satellite voice communication. If the patient regains consciousness, they are offered help and, depending on the response, one of the previous alternatives is selected. If the patient does not regain consciousness, then perform actions according to the alternative “the patient is conscious and needs help”. At the same time, data on the patient’s pulse, pressure, temperature and respiratory rate are continuously monitored. If the patient is at home and there is no one around, the massage therapist can use the Internet for the remote control of the robot “Triangel” to attempt to bring the patient to consciousness by shaking the patient’s back or chest by organizing reciprocating movements of the MBB 1 with the corresponding vibration effect on the skin and muscles of the patient’s back or chest. In case of complete disappearance of pulse, pressure and respiration without their subsequent resumption within the set time, as well as in the absence of any body movements, then actions are performed according to the alternative “the patient died”. In this case, an additional message about the fact of death is sent to the police with the coordinates of the patient’s location. The choice of the alternative “patient died” can be made after any of the types and species of falling in the case of complete disappearance of pulse, pressure and respiration without their subsequent resumption within a set time and in the absence of any body movements. In this case, the actions are carried out similar to the above.

4 Conclusions The possibility of walking of the patient inside the room during the SCM of his back or chest by the autonomous symbiotic self-relocating massage robot “Triangel” was established, which increases the comfort of the patient’s conditions. The possibility of the patient falling to the surface while walking in the room during the SCM, as well as the types and species of falling, was revealed. Possible consequences of a patient’s falling during SCM depending on the type and species of falling was revealed. The ability to make timely decisions, both alone and in interaction with the leader operator, allows the robot “Triangel” to reduce the negative consequences of the patient falling to the surface.

References 1. Dubrovsky, V.I., Dubrovskaya, A.V.: Lechebnyy massazh (Therapeutic massage). GEOTARMedia, Moscow (in Russian) (2004) 2. Beck, M.F.: Theory & practice of therapeutic massage (5th edn.). CENGAGE Learning, New York (2016) 3. Sajapin, S.N., Sajapina, M.S.: Movable massaging apparatus and method for giving massage with use thereof. Russian Federation Patent 2,551,939, 10 June 2015 (2015)

52

S. N. Sayapin

4. Sayapin, S.N., Sayapina, M.S.: A self-driven multifunctional vacuum massager based on an active triangular module of parallel structure. Biomed. Eng. 51(2), 147–151 (2017) 5. Sayapin, S.N.: Intelligence self-propelled planar parallel robot for sliding cupping-glass massage for back and chest. In: Hu, Z., Petoukhov, S., He, M. (eds.) Advances in Artificial Systems for Medicine and Education, pp. 166–175. Springer, Cham, Switzerland (2018) 6. Sayapin, S.N.: Analiz i perspektivy primeneniya avtonomnogo simbioznogo samoperemeshchayushegosya massazhnogo robota «Triangel» na bortu pilotituemoj kosmicheskoy stancii (Analysis and Prospective Use of Autonomous Symbiotic Self-Relocating Body Massage Robot «Triangel» Onboard Piloted Space Stations). Aviakosmicheskaya i Ekologicheskaya Meditsina (Russia) 54(1): 75–78 (in Russian) (2020) 7. Avdeyev, A.I.: Travma na lestnichnom marshe: biomekhanika, diagnostika, morfologiya (ustanovleniye sobytiy i obstoyatelstv proisshestviya) (Trauma on a flight of stairs: biomechanics, diagnostics, morphology (establishing the events and circumstances of the incident)) Publishing house of the regional clinical hospital, Khabarovsk (In Russian) (2001) 8. Zarubina, S.V.: Sudebno-meditsinskaya otsenka povrezhdeniy. voznikayushchikh pri padenii na ploskosti i ego biomekhanicheskiye aspekty (Forensic assessment of injuries caused by falling on a surface, and its biomechanical aspects). Medical Science Candidate’s Dissertation. Altai state medical University, Barnaul (In Russian) (2007) 9. Kryukov, V.N., Buromskiy, I.V.: Rukovodstvo po sudebnoy meditsine (Guide to forensic medicine). Norma, INFRA, Moscow (in Russian) (2015) 10. Nikano, E.: Vvedeniye v robototekhniku (Introduction to robotics). Translated from Japanese. Mir, Moscow (in Russian) (1988) 11. Brahman, T.R.: Mnogokriterial’nost’ i vybor al’ternativy v tekhnike (Multi-criteria and choice of alternatives in technique). Radio i svyaz’, Moscow (in Russian) (1984) 12. Dubov, YuA., Travkin, S.I., Yakimec, V.N.: Mnogokriterial’nye modeli formirovaniya i vybora variantov system (Multi-criteria models of forming and choosing variants of systems). Nauka, Moscow (in Russian) (1986) 13. Fishburn, P.C.: Utility theory for decision making. John Wiley & Sons Inc, NewYork-LondonSydney-Toronto (1970) 14. Keeney, R.L., Raiffa, H.: Decisions with multiple objectives: preferences and value tradeoffs. John Wiley & Sons Inc, NewYork-London-Sydney-Toronto (1976) 15. Sayapin, S.N.: Model’ optimal’nogo proektirovaniya transformiruemyh kosmicheskih radioteleskopov lepestkovogo tipa (Model of optimal designing for transformable petal-type space radio telescopes.). Technical Science Candidate’s Dissertation. Obninsk Institute of atomic energy, Obninsk (in Russia) (1997) 16. Keat, P.G., Young, P.K.Y., Erfle, S.E.: Managerial economics: economic tools for today’s decision makers. Pearson, Boston (2014) 17. Khlynov, S.A.: Algoritm prinyatiya upravlencheskih reshenij v posrednecheskoj organizacii (Algorithm of acceptance administrative decisions in the intermediary organization). Ekonomika, Statistika i Informatika (Russia). 3: 97–100 (in Russian) (2014) 18. Sokol, A.F., Shurupova, R.V.: Effekt frejminga i ego vliyanie na prinyatie reshenij v medicine (po koncepcii D. Kanemana i A. Tverski) (Framing effect and its influence on decision-making in medicine (according to the concept of D. Kahneman and A. Tversky)). Medicinskij sovet (Russia) 6: 166–168 (in Russian) (2017) 19. Felder, S., Mayrhofer, T.: Medical decision making. Springer-Verlag, Berlin, Germany (2017) 20. Rozikhodjaeva, G.A., Rozikhodjaeva, D.A.: Sravnitel’nyj analiz modelej prinyatiya reshenij v voprosah diagnostiki zabolevanij (Comparative analysis of models of decision-making systems in issues of diagnostics of diseases). Problemy sovremennoj nauki i obrazovaniya (Russia) 15: 97–99 (in Russian) (2017)

Decision-Making by the Autonomous Symbiotic Self-Relocating Massage Robot “Octahedral Dodekapod” Based on SEMS During the Upper or Lower Limb Massage Sergey N. Sayapin Abstract Problem statement: in this article the essence of the symbiotic robot “Octahedral dodekapod” based on SEMS which comprises in its mechanical union with the patient’s upper or lower limb and with a single coordinate system, is shown. As a result, the symbiotic robot “Octahedral dodekapod” is able to perform massage manipulations of the patient’s upper or lower limb not only in the passive position of this limb, but also on the moving limb, regardless of its spatial orientation. Also it is possible to simultaneously massage the patient’s upper and lower limbs with two or several symbiotic robots “Octahedral dodekapod”. The symbiotic robot “Octahedral dodekapod” can function both autonomously and under the remote supervision of a massage therapist via the Internet. However, during autonomous massage, a situation may arise that requires immediate decision-making aimed at preventing threats to the patient’s life or health, for example, unacceptable values of blood pressure, pulse, temperature, as well as loss of consciousness. Purpose of research: development of algorithms for decision-making by the symbiotic robot “Octahedral dodekapod” in case threats to the patient’s life or health during upper or lower limb massage, for example, unacceptable values of blood pressure, pulse, temperature, as well as loss of consciousness. Results: abnormal situations requiring immediate decision-making by the symbiotic robot “Octahedral dodekapod” during the massage are shown. Algorithms for decision-making by a symbiotic robot “Octahedral dodekapod” installed on the patient’s limb have been developed. The decision-making algorithms are formed depending on the patient’s condition, for example in the following cases: the patient is conscious and does not need help; the patient is conscious and needs help; the patient is unconscious; the patient has died. Practical significance: decisionsmaking by the symbiotic robot “Octahedral dodekapod” both autonomously and under the remote control of the massage therapist in real time via the Internet will reduce the risk of a threat to the patient’s health or life after falling to the surface during the upper or lower limb massage. S. N. Sayapin (B) Mechanical Engineering Research Institute of the Russian Academy of Sciences (IMASH RAN), Moscow, Russia e-mail: [email protected] Bauman Moscow State Technical University, Moscow, Russia © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_5

53

54

S. N. Sayapin

Keywords SEMS · Autonomous symbiotic self-relocating massage robot · Decision making by single robot and with a leader operator

1 Introduction It is known that massage is currently considered as one of the most effective non-drug methods of disease prevention, treatment, rehabilitation, and fatigue relief. Therefore, now massage is used in clinics, hospitals, medical centers, as well as at home, etc. At the same time, all types of upper and lower limb massage are performed by a masseur or patient (self-massage) manually or using massage devices [1, 2]. To robotization the upper or lower limb massage, in IMASH RAN a portable autonomous symbiotic self-relocating spatial parallel massage robot “Octahedral dodekapod” with 12 DOF (degrees of freedom) based on SEMS (Smart Electromechanical System) is developed [3–6]. The “Octahedral dodekapod” is the robot built on the basis of a new symbiotic approach, consisting in its mechanical discrete unification with the patient’s upper or lower limb. As a result of the symbiotic approach, the patient’s the upper or lower limb becomes the carrier element of the parallel robot itself, and the massage can be performed regardless of the spatial position of the body, as well as when the patient moves indoors. However, if the massage is performed while walking, the patient may fall, for example, tripping, slipping, or as a result of loss of consciousness. One of the most effective ways to solve this problem is the intellectualization of the symbiotic massage robot “Octahedral dodekapod”. Therefore, it is important to develop axiomatics and algorithms for decision-making by “Octahedral dodekapod” in the case of a patient falling to the surface during the upper or lower limb massage. The description of the symbiotic robot “Octahedral dodekapod”, main possible types of falling of the patient to the surface during the upper or lower limb massage, and the axiomatics of the decision-making model and algorithms for decision-making by “Octahedral dodekapod” after the patient fallings to the surface are presented below.

2 Description of Symbiotic Robot “Octahedral Dodekapod” and Main Types of Possible Fallings of Patient During Massage 2.1 Description of Symbiotic Robot “Octahedral Dodekapod” The essence of the symbiotic approach to creating the robot “Octahedral dodekapod” for the upper or lower limb massage is to combine it with the patient’s massaged limb and single coordinate system and ensure the independent functioning of the robot “Octahedral dodekapod” and the patient during the upper or lower limb massage.

Decision-Making by the Autonomous Symbiotic …

55

Fig. 1 Structural (a) and kinematic (b) diagrams of the robot “Octahedral dodekapod”

Thanks to the symbiotic approach, the patient can walk, perform routine and other work, as well as fulfill physiological needs during the upper or lower limb massage. Structural (a) and kinematic (b) diagrams of the robot “Octahedral dodekapod” with 12 DOF are shown in Fig. 1. The “Octahedral dodekapod” is a universal active spatial mechanism of parallel structure with 12 degrees of freedom in the form of active octahedral module (AOM) 1, which has a high degree of specific rigidity to allow a low specific weight [7–9]. The ribs of AOM 1 consist of 12 rods whose ends are joined by spherical or equivalent hinges at the six vertexes 2, four at each (Fig. 3). The rear face ABC is positioned on the side opposite the direction of movement and the front face DEF is on the side of the direction of movement. Each rod has a linear drive (LD) 3 with a relative displacement sensor (RDS) 4 and has the ability to change the length of LD 3 in response to control commands from the control system (CS). The mid parts of the rods on the rear and front faces are fitted with radial supports with force sensors (RSFS) 6 providing the robot manipulator with adaptive sleeves (Figs. 1 and 2). Each RSFS 6 is attached to an LD 3 at the maximum-length position (Fig. 1). Temperature sensors (TS) 7 and insulated electrical contacts (not shown) are attached to the contact surfaces of the RSFS to monitor temperature on the skin surface at the relevant points and to measure the electrical resistance and potential difference between contact points. LD 3 on all bars of the AOM 1 is fitted with axial force sensors (AFS) 8 and relative speed sensors (RSS) 9. The vertexes 2 are fitted with combined spatial position 10 and acceleration 11 sensors in the form of miniature tri-axial blocks of gyroscope accelerometer units which provide operative monitoring of the spatial positions of each vertex 2 and vibroacceleration along each

56

S. N. Sayapin

Fig. 2 The full-size mechanical prototype of face AOM with different types massage grippers: elastic plates (a, b, and c), (face ABC or DEF), combination from elastic plates and rod with massage rollers (d, e, and f), rigid plates (g), combination from elastic and rigid plates and elastic rod with massage rollers (h)

axial rod with LD 3. Each vertex 2 can be fitted with massage devices and ultrasound sensors (not shown) facing the centers of the rear and front faces. CS 5 consists of neurocomputer 12, programmable algorithm unit (PAU) 13, and digital-to-analog converter (DAC) 14. CS inputs 5 are switched through the analog-to-digital converter data bus to the outputs of ADC 15 of RSFS 6 and AFS 8, ADC 16 of RDS 4, and ADC 17 of the combined spatial position and acceleration sensors 10 and 11, ADC 18 of RSS 9, and ADC 19 of TS 7. The outputs of CS 5 are switched through the output data bus to the corresponding inputs of the PAU 13 and sequentially connected DAC 14, power amplifier 20, and LD 3. Also, the “Octahedral dodekapod” has a two-way loudspeaker communication with the patient (not shown in Fig. 1), as well as satellite communication with the Massage therapist and the hospital. (Figure 1a). The grippers of RSFS 6 can be elastic, rigid, or combined and can change their size over a wide range. Therefore, the same grippers are able to compress different parts of both the upper and lower limbs without any additional devices. Rod lengths of AOM 1 were selected using a full-size mechanical prototype its face ABC (DEF) with grippers. Figure 2 the full-size mechanical prototypes of different types massage grippers (face ABC or DEF) and principle of upper limb compression by such grippers is shown. The “Octahedral dodekapod” including AOM 1 operates as follows. Shortening of LD 2 on the front and rear faces allows RSFS 6 to grasp and squeeze the outer

Decision-Making by the Autonomous Symbiotic …

57

Fig. 3 Cyclogram showing movement of the AOM along the forearm (a); kinematic diagram of the rear face of the AOM with grasping elastic plates (b); radial grasping of forearm by face ABC (c)

surface of the limb with the required force (Figs. 1 and 3). However, it is clear from Figs. 2 and 3b, that movement of AOM 1 along the limb requires a significant change in the lengths of the rods of the rear and front faces of AOM 1, leading to significant increases in its size and weight and requiring a separate AOM 1 for each part or segment of limb. To avoid these drawbacks, RSFS 6 are made as tensometric elastic elements shaped as plates or rods with their ends firmly attached to the ends of LD 3. This produces the necessary contact between RSFS 6 and the front and rear faces of AOM 1 and the limb surface with transmission of the necessary radial forces to it using a single AOM1 size (Figs. 2 and 3). Figure 3b shows changes in the diameter of the contact of the limb with the bars of the AOM1 face with (Ø and d) and without (Ø1 and d1) RSFS 6 when the length of the bars decreases by amount . To perform massage, the AOM 1 is placed on the upper or lower limb, for example on the patient’s forearm at the wrist, and makes the necessary movements and massage manipulations (Fig. 3). LD 3 and CS 5 drive concordant changes in the lengths of the AOM ribs. This leads to the specified spatial movements of vertexes 2 relative to the basic coordinates system. Operative control of the applied forces at the contact points of the RSFS 6 with the external surface of the skin is mediated by force sensors 6 and 8. Control commands from CS 5 are generated on the basis of data from

58

S. N. Sayapin

force sensors 6 and 8, RDS 4, RSS 9, and combined spatial position and acceleration sensors 10 and 11. Signals from the sensors are passed to the inputs of ADC 15, 16, 18, and 17 respectively and via the data bus to CS 5 of neurocomputer 12. After real-time processing by PAU 13, control commands are formed which pass via DAC 14 and power amplifier 20 to actuators LD 3, which change the geometry of OM 1 (transformation). Data from RDS 4 operating as linear positioners are used to control the lengths of all the bars of the OM. The geometrical constancy of OM 1 allows the spatial coordinates of vertexes 2 to be specified in terms of the measured lengths of all bars and their movements to be controlled through the spatial displacements of the l-coordinate manipulator [10]. Data from spatial position sensor 10 increase the accuracy of these movements. The main rehabilitation modes performed by the “Octahedral dodekapod” are described below. The first mode is the organization of stroking and rubbing manipulations of the patient’s upper or lower limbs. Stroking movements consist of sliding the hand (hands) across the skin, not producing folds, with different levels of pressure, while rubbing consists of displacing, moving, and stretching the tissues in different directions [1, 2]. Figure 3 shows a cyclogram of the movement of the OM across the forearm and its radial girth by face ABC using RSFS 6. Before AOM 1 is positioned on the forearm, the geometrical parameters of the parts to be massaged and the associated basic coordinates system are entered into CS 5. The coordinates of the body parts not to be massaged are then entered into CS 5. AOM 1 is then placed on the initial massage site (Fig. 30) and LD 3 are switched on to decrease the lengths of bars AB, BC, CA, DE, EF, and FD. RSFS 6 move to the centers of the corresponding faces before attachment of the rear face ABC to the limb and compression of the forearm by the front face DEF with the specified force (Fig. 3I) as determined by data from force sensors 6 and 8. After positioning of LD 3, the coordinates of the vertexes of the faces relative to the bast coordinate system are computed and stroking and rubbing manipulations are performed via changes in the lengths of the bars making up the side faces of AOM 1. At the end of massaging the initial part, the lengths of the bars of the front face DEF are increased to attain detachment from the limb and concordant changes in the lengths of LD 3 of the bars of the side faces turn the front face DEF relative to the rear face ABC through the specified angle (not shown in Fig. 3). The front face ABC then compresses a new part of the limb and the stroking or rubbing manipulations are repeated as previously. Commands from CS 5 to the corresponding LD 3 then drive the autonomous movement of AOM 1 along the limb to the new area. After detachment of the front face (Fig. 3II), increases in the lengths of the bars of the side faces move it along the limb (Fig. 3III) and, via RSFS 6, the limb is compressed by the front face and rear face ABC detaches (Fig. 3IV). Decreases in the lengths of the bars of the lateral faces then move the rear face ABC along the limb (Fig. 3V) and attach it with the specified force (Fig. 3I). The manipulation can be performed by the front or rear face. The algorithm for limb stroking and rubbing manipulations and movement of AOM 1 to the new site is then repeated.

Decision-Making by the Autonomous Symbiotic …

59

The second mode consists of organizing longitudinal and transverse kneading manipulations of the patient’s upper or lower limbs, with continuous or discontinuous grasping, pressing, squeezing, moving, and “milling” of the tissues (muscles). Longitudinal and transverse kneading are discriminated [1, 2]. Organization of these manipulations is carried out in the same way as in the previous mode by concordant changes in the lengths of the bars of AOM 1. The third mode consists of organizing mobilization and manipulations at the elbow or knee joint of the limbs. To do this, you should use one AOM 1 (Fig. 1b) or dual (Fig. 4a). Mobilization consists of passive, rhythmically repeated movements at the joints within their physiological ranges [1, 2]. In organizing mobilization and manipulation procedures at the elbow or knee joints, LD 3 attach the front face of AOM 1 to the shoulder or thigh and the rear face to the forearm or knee respectively, at specified distances from each other at the sites concerned (Fig. 4b). Data from sensors 4 and 10 are then used to compute the coordinates of vertexes 2 relative to the base coordinate system. Concordant changes in the lengths of the bars of the side faces of AOM 1 then perform alternating axial mobilizing longitudinal and rotatory actions. The working course of LD 3 is controlled using data from AFS 8 and combined spatial position 10 and acceleration 11 sensors, and the lengths of bars AD, AF, BD, BE, CE, and CF are controlled using data from RDS 4. Limb position is controlled using spatial position sensor 10 and the rate of movement of the faces is controlled using data from RSS 9 and spatial acceleration sensor 11. The fourth mode consists of organizing vibratory manipulations. In this case, the algorithms described above for alternating movements on stroking, rubbing, and kneading of the limb are organized at specified vibratory frequencies, amplitudes, and

Fig. 4 General view of dual AOM (a) and diagram showing positioning of the AOM on the patient’s upper and lower limbs (b)

60

S. N. Sayapin

strengths, which are determined using data from spatial position 10 and acceleration 11 sensors, force sensors 6 and 8, RDS 4, and RSS 9. After manipulations, AOM 1 moves itself to the new specified area and the algorithm for vibratory manipulations is repeated. The fifth mode consists of organizing manipulations to perform formed movements of the free part of the limb relative to the adjacent part. This involves using dual AOM 1 (Fig. 4a) combined to form a paired module and a common face; the common face is placed in the area of the joint (Fig. 4b). Then, using the massage program specified in CS 5, the specified formed (passive) spatial movements of the free part of the limb relative to the adjacent part are performed. In all modes, the patient has the option to switch the device off, and CS 5 immediately sends the corresponding signal to the masseur. All massage actions are recorded in CS 5 for subsequent analysis and prescription of new procedures. In modes 3 and 5, data from AFS 8 on LD 3 are used to assess the resistive force of the joint on organization of passive movement of the joint for diagnosis of its level of mobility (functionality). Furthermore, TS 7 can be used to provide continuous measurement of limb temperature at contact points and its data can be used to assess overheating during massage procedures, for example rubbing. Measurements of the electrical resistance between the contact sites of the radial supports on the limb during massage with measurements of the distance between them and their temperatures allow skin dryness and individual tolerance of massage to be assessed such that the optimum mode of treatment for the patient can be selected. The grippers can also be equipped with pulse and pressure sensors, which will allow real-time monitoring of them at certain points of the upper limb. The received information about the pulse, pressure and temperature is transmitted to the massage therapist or Hospital in real time. Data transmission can be carried out via satellite (Fig. 1a) and the Internet.

2.2 Description of Main Types of Possible Fallings of Patient to Surface A falling on the surface is a type of rotational movement of a vertically positioned body around a horizontal or sagittal axis that passes through the foot. The fallings of the patient to the surface during autonomous upper or lower limb massage by robot “Octahedral dodekapod” can occur as a result of a violation of the balance of the body from a standing position or when walking on an uneven or slippery surface, as well as with diseases of the bones and joints of the lower extremities or with neurovascular disorders accompanied by loss of consciousness. The classification of the main types of falls on the surface coincides with the classification shown in Fig. 2 in [11]. Depending on how the patient falls (face down or on the side) and the position of the hands at the moment of impact, he may have the following injuries: abrasions,

Decision-Making by the Autonomous Symbiotic …

61

hematomas and wounds at the points of contact, fractures of the clavicle, humerus, ulnar process, forearm bones, leg bones, spinous processes of the vertebrae, ribs, etc. The most serious damage is caused from a impact to the head when falling flat on your back [7, 8, 12]. The falling of the patient to the surface may be uncoordinated or coordinated ones. For uncoordinated falling backwards is characterized by the presence of abrasions on the hands and hemorrhages in the soft tissues of the occipital area of the head. Such type of falling is the most traumatic, as body reaches the maximum level of rotational velocity and the effectiveness of the protective action of the exposed hands reduces. The impact of the body with the surface in an uncoordinated falling occurs begin the back of the head, shoulder blades, lower back, buttocks, and then the upper and lower limbs (Fig. 5) [7]. The location of the “Octahedral dodekapod” on the patient’s upper limb during the uncoordinated his falling to the surface are shown in Fig. 5 also. The coordinated backward falling is performed in a protective position (squat, head bent to the chest, grouped state of body parts, hands pointing back and to the sides), an attempt to turn the torso sideways with a turn in the same direction of the knee area to soften the impact contact (Fig. 6) [7].

Fig. 5 Computer diagram of uncoordinated backwards falling of the patient’s body with “Octahedral dodekapod”

62

S. N. Sayapin

Fig. 6 Computer diagram of coordinated backwards falling of the patient’s body with “Octahedral dodekapod”

2.3 Description of Main Types of Possible Falling of Patient to Stairs of Staircases In the case of a passive uncoordinated falling of a person on a flight of stairs after losing their balance, in most cases, the body turns relative to the fulcrum and simultaneously slips along it. This is followed by the separation of the body from the support and its rollover and flight with rotation until the impact, followed by rolling on the back or rolling over the head. The most traumatic is an uncoordinated falling of the patient’s body from a standing position, since this develops the maximum angular velocity in the head area (Fig. 7). In the passive uncoordinated backwards falling (Fig. 7a), the places of impact are the back of the head, shoulder blades, lower back and buttocks, as well as the lower and upper limbs. In case of an uncoordinated falling face down (Fig. 7b), injuries are localized primarily on the front surface of the upper body. In case the uncoordinated falling from a half-sitting position which caused by the patient falling to his knees in a semi-conscious state (not shown in Fig. 7), the amount of damage may be minimal, since the impact on the step occurs with less energy. When falling from the lower steps, the impact occurs with a flat surface of the staircase (Fig. 7c). When falling from the middle steps, the torso bends along the contour of the angle formed by the staircase and its flat surface (Fig. 7d). When falling from the upper steps, the body hits the inclined ribbed surface of the staircase and then slides down it (Fig. 7e). In the case of a coordinated falling, a defensive reaction may occur, in which the patient squats sharply while simultaneously bringing his hands palms forward or groups his body (not shown in Fig. 7). In the passive uncoordinated falling from the standing position to the onto his side (Fig. 8), after being unbalanced, the patient’s body leans down with increasing

Decision-Making by the Autonomous Symbiotic …

63

Fig. 7 Computer diagrams of main types of patient’s fallings on the staircase with massage robot “Octahedral dodekapod”: uncoordinated falling backwards (a); uncoordinated falling face down (b); uncoordinated face-down fall from the lower steps (c), middle (d), and upper steps (b)

acceleration and hits the steps with the side surface of the trunk, the shoulder, and finally the head. After that, the body bounces and moves down with a turn on the back or on the stomach, followed by a repeated impact and stopping the movement. In the case of the coordinated falling (not shown in Fig. 8), the defensive reaction may occur, in which the following movements of the body parts occur: turning torso, squatting, bending head to the chest and pulling hands back with the palms down. The symbiotic massage robot “Octahedral dodekapod” is a discrete mechanical connecting with the massaged area of the patient’s upper or lower limb. Due to this interaction, all the movements of the massaged area of the body coincide with the spatial transport motion of the robot itself. As result, data of the three-axis miniature gyroscopes-accelerometers, which are installed in the vertices of the “Octahedral dodekapod” and the wrist devices on the forearm, it is possible to judge the trajectories of movements and accelerations of body parts, which after comparison with the model

64

S. N. Sayapin

Fig. 8 Computer diagrams of the passive uncoordinated falling of the patient’s body with “Octahedral dodekapod” from the standing position to the onto his side

diagrams in Figs. 5, 6, 7, and 8, can be used to judge the nature of the fallings and the patient’s condition. Such data allows the massage robot “Octahedral dodekapod” to make decisions that are most favorable and safe for the patient after his fall. The algorithms for decision-making by the symbiotic robot “Octahedral dodekapod” after a patient falls to the surface are presented below.

3 Decision-Making by the Autonomous Symbiotic Self-Relocating Massage Robot “Octahedral Dodekapod” After the Falling of Patient on Surface The axiomatics of the decision-making model of the massage robot “Octahedral dodekapod” is identical to the axiomatics of the “Triangel” massage robot presented in [11]. As noted above, “Octahedral dodekapod” allows the patient to move around any rooms and staircase while performing the autonomous upper or lower limb massage. At the same time, on the one hand, the patient receives additional comfort, and on

Decision-Making by the Autonomous Symbiotic …

65

the other hand, there is a danger of falling to the surface if walking inside the room or on staircase with simultaneous massage. In General, a solution is understood as a set of actions from the side of the decision maker to the object (system, complex, etc.) of management that allows you to bring this object to the desired state or achieve the goal set for it. At the same time, the decision-making process of the decision maker is characterized by the following features: • the ability to choose from alternative options (if there is no alternative, then there is no choice and, therefore, there is no solution); • having a goal, since purposeless choice is not considered a solution; • the need for a volitional act from the decision maker. Currently, a large number of algorithms and models for making project and management decisions have been developed in relation to the most diverse areas of human activity [9, 11, 13–22]. However, there is no algorithm that can always be applied, regardless of what kind of problem the SEMS-based robot had to face, what caused it, what factors affect it, whether they are manageable, etc. Let’s consider what the first stage of the decision-making algorithm should be as applied to the symbiotic massage robot “Octahedral dodekapod” in the event of the patient falling to the surface during the upper or lower limb massage. Analysis of publications has shown that most authors put the first stage of the decision-making algorithm to identify and analyze the problem situation. However, other authors believe that the entire process should begin with setting goals and objectives [9, 13–22]. To implement the decision-making ability by the autonomous symbiotic massage robot “Octahedral dodekapod”, programs are installed in its control system 5 (Fig. 1a) that simulate various types and species of human fallings to the surface (Figs. 5, 6, 7, and 8), similar to the programs described in [7, 8, 12]. Also, before the upper or lower limb massage procedure, data on the patient’s weight and height are additionally entered into the control system 5 (Fig. 1a). Based on the global goal and taking into account the framing effect [20], the following decision-making algorithm was developed for the robot “Octahedral dodekapod” after the patient falls to the surface (Fig. 9), which includes three blocks similar to the algorithm described in [11, 19]. The first block of the decision-making algorithm includes the following steps: • • • • •

situation analysis; goal-setting. The “situation analysis” stage includes two operations: problem identification; collection and analysis of necessary information.

Problem identification includes the following actions of the robot “Octahedral dodekapod”: 1.

Establishing and registering the fact of the patient falling to the surface, characterized by accelerated movement of his body down. Registration of the fact of falling is carried out by the control system 5 “Octahedral dodekapod”

66

S. N. Sayapin

Fig. 9 Block diagram of the decision-making algorithm for the single robot “Octahedral dodekapod” with the leader operator

2. 3.

4.

(Fig. 1a) according to indications from gyroscopes-accelerometers installed in the vertices 2 of the “Octahedral dodekapod”. the upper or lower limb massage is terminated and AOM 1 (Fig. 1) is made motionless. Determining the type of falling (Figs. 5, 6, 7, and 8) and its species (coordinated or uncoordinated falling). Here, based on the indications of gyroscopesaccelerometers, the trajectory of the patient’s upper or lower limb is constructed, it is compared with the model body movements when a person falls, described in [7, 8, 12], and the type and species of falling is determined. Establishing the place of the patient’s fall using a GPS or GLONASS satellite navigation system.

Collection and analyzing the necessary information includes determining the patient’s pulse, pressure, and body temperature, and based on them, taking into account the type and species of falling that characterize possible injuries and injuries of the patient, and forming an assessment of the patient’s condition after the falling. The specificity of the first block of the decision-making algorithm is that the “goal setting” stage and the “problem identification” operation are carried out almost in parallel and are very closely intertwined. As a result, in most cases it is difficult to determine the priority between goal-setting and problem identification.

Decision-Making by the Autonomous Symbiotic …

67

Therefore, in our case, the “goal-setting” stage is reduced to a global goal when making decisions by the robot “Octahedral dodekapod”, characterized by reducing the negative consequences of the possible falling of the patient to the surface. The second block of the decision-making algorithm includes the following steps: • developing alternatives; • development of criteria for selecting alternatives; • choose one of the alternatives. To increase the efficiency of decision-making by the “Octahedral dodekapod” robot, alternative solutions to the problem were formed depending on the patient’s condition for the following cases: • • • •

the patient is conscious and does not need help; the patient is conscious and needs help; the patient is unconscious; the patient died. The third block of the decision-making algorithm includes the following steps:

• implementation of the selected alternative; • control and evaluation of resources. At the implementation stage, the selected alternative is implemented. Parallel to the implementation stage of the chosen alternative is the stage of control the evaluation of the results of the decision. Feedback is difficult to distinguish in any particular stage of the decision-making algorithm, since it functions both in separate stages and between them throughout the entire process. The selected alternatives and their corresponding actions are listed below. The choice of the alternative “the patient is conscious and does not need help” is made as a result of the following actions. If the patient is in a medical facility, data on the fact of falling and getting up independently, the type and species of falling with possible traumatic consequences, as well as data on pulse, pressure, and temperature are transmitted to the nurse on duty and the massage therapist. Then, via satellite voice communication with the patient, they are offered help. In case of refusal of assistance, the massage therapist decides to continue or terminate the upper or lower limb massage procedure in accordance with the rules established for him. If a patient requests help, the alternative “the patient is conscious and needs help” is selected, and the following actions are performed. Through GPS or GLONASS, the location coordinates are additionally sent to the nurse on duty of the medical institution and the appropriate specialists are sent to provide the patient with the necessary assistance. If the upper or lower limb massage was performed by the patient at home, the above information is transmitted via satellite to the rescue service and the massage therapist. Also, the patient’s condition is monitored using a loud-speaking satellite connection and data on the patient’s pulse, pressure, and temperature and

68

S. N. Sayapin

oral recommendations are given to him and others (if they are present) about the necessary actions before the arrival of the doctor or the arrival of emergency help. The choice of the “the patient is unconscious” alternative is made in the case of an uncoordinated species of falling, the absence of subsequent body movements, and the presence of pulse. If the patient is in a medical facility, data on the fact of the falling, the type and species of falling with possible traumatic consequences, as well as data on pulse, pressure and temperature are transmitted to the nurse on duty and the massage therapist. Further, attempts are made to establish voice contact with the patient via satellite voice communication. If the patient regains consciousness, they are offered help and, depending on the response, one of the previous alternatives is selected. If the patient does not regain consciousness, then perform actions according to the alternative “the patient is conscious and needs help”. At the same time, data on the patient’s pulse, pressure and temperature are continuously monitored. If the patient is at home and there is no one around, the massage therapist can use the Internet for the remote control of the robot “Octahedral dodekapod” to attempt to bring the patient to consciousness by shaking the patient’s upper or lower limb by organizing its movements by AOM 1. In case of complete disappearance of pulse and pressure without their subsequent resumption within the set time, as well as in the absence of any body movements, then actions are performed according to the alternative “the patient died”. In this case, an additional message about the fact of death is sent to the police with the coordinates of the patient’s location. The choice of the alternative “patient died” can be made after any of the types and species of falling in the case of complete disappearance of pulse and pressure without their subsequent resumption within a set time and in the absence of any body movements. In this case, the actions are carried out similar to the above.

4 Conclusions The presented Autonomous symbiotic self-shifting massage robot “Octahedral dodecapod” can also be used in everyday conditions for qualified Autonomous programmable self-massage. The connection of CS to the Internet allows an online connection between the “Octahedral dodekapod”, the patient, and the masseur to be set up for operative monitoring of massage and taking decisions when emergencies arise, for example sending commands to CS to stop the device and provide medical care at the patient’s home. Patients unable to visit therapeutic institutions independently can thus have massage under on-line control by a masseur without the physical presence of the masseur at the patient’s location. The possibility of walking of the patient inside the room during the upper or lower limb massage by the autonomous symbiotic self-relocating massage robot “Octahedral dodecapod” was established, which increases the comfort of the patient’s conditions.

Decision-Making by the Autonomous Symbiotic …

69

The possibility of the patient falling to the surface while walking in the room during the upper or lower limb massage, as well as the types and species of falling, was revealed. Possible consequences of a patient’s falling during the upper or lower limb massage depending on the type and species of falling was revealed. The ability to make timely decisions, both alone and in interaction with the leader operator, allows the robot “Octahedral dodecapod” to reduce the negative consequences of the patient falling to the surface.

References 1. Dubrovsky, V.I., Dubrovskaya, A.V.: Lechebnyy massazh (Therapeutic massage). GEOTARMedia, Moscow (in Russian) (2004) 2. Beck, M.F.: Theory & practice of therapeutic massage, 5th edn. CENGAGE Learning, New York (2016) 3. Sayapin, S.N.: An adaptive portable spatial rehabilitation robot manipulator and a means of organizing movements and its use for patient diagnosis. Russian Federation Patent 2,564754, 10 October 2015 (2015) 4. Sayapin, S.N.: Principles of the design of an adaptive mobile spatial rehabilitation manipulator robot based on an octahedral dodecapod. Biomed. Eng. 51(4), 296–299 (2017) 5. Sayapin, S., Sayapina, E.: Novel approach to creation of portable self-propelled autonomous massage robots with triangular and octahedral parallel structures. WSEAS Trans. Comput. 16, 124–132 (2017) 6. Sajapin, S.N.: Smart parallel robots for massage. IOP Conf. Ser.: Mater. Sci. Eng. 468(1), 1–7 (2018). https://doi.org/10.1088/1757-899X/468/1/012028 7. Zarubina, S.V.: Sudebno-meditsinskaya otsenka povrezhdeniy. voznikayushchikh pri padenii na ploskosti i ego biomekhanicheskiye aspekty (Forensic assessment of injuries caused by falling on a surface, and its biomechanical aspects). Medical Science Candidate’s Dissertation. Altai state medical University, Barnaul (In Russian) (2007) 8. Kryukov, V.N., Buromskiy, I.V.: Rukovodstvo po sudebnoy meditsine (Guide to forensic medicine). Norma, INFRA, Moscow (in Russian) (2015) 9. Nikano, E.: Vvedeniye v robototekhniku (Introduction to robotics). Translated from Japanese. Mir, Moscow (in Russian) (1988) 10. Koliskor, ASh.: Razrabotka i issledovanie promyshlennyh robotov na osnove l-koordinat. (Development and study of industrial robots based on l-coordinates). Stanki i Instrument (Russia) 12: 21–24 (in Russian) (1982) 11. Sayapin, S.N.: 1.4. Decision-making by the autonomous symbiotic self-relocating massage robot “triangel” based on SEMS after the fall of patient on surface (in this collection) 12. Avdeyev, A.I.: Travma na lestnichnom marshe: biomekhanika, diagnostika, morfologiya (ustanovleniye sobytiy i obstoyatelstv proisshestviya) (Trauma on a flight of stairs: biomechanics, diagnostics, morphology (establishing the events and circumstances of the incident)) Publishing house of the regional clinical hospital, Khabarovsk (In Russian) (2001) 13. Brahman, T.R.: Mnogokriterial’nost’ i vybor al’ternativy v tekhnike (Multi-criteria and choice of alternatives in technique). Radio i svyaz’, Moscow (in Russian) (1984) 14. Dubov, YuA., Travkin, S.I., Yakimec, V.N.: Mnogokriterial’nye modeli formirovaniya i vybora variantov system (Multi-criteria models of forming and choosing variants of systems). Nauka, Moscow (in Russian) (1986) 15. Fishburn, P.C.: Utility theory for decision making. John Wiley & Sons Inc, NewYork-LondonSydney-Toronto (1970)

70

S. N. Sayapin

16. Keeney, R.L., Raiffa, H.: Decisions with multiple objectives: preferences and value tradeoffs. John Wiley & Sons Inc, NewYork-London-Sydney-Toronto (1976) 17. Sayapin, S.N.: Model’ optimal’nogo proektirovaniya transformiruemyh kosmicheskih radioteleskopov lepestkovogo tipa (Model of optimal designing for transformable petal-type space radio telescopes). Technical Science Candidate’s Dissertation. Obninsk Institute of atomic energy, Obninsk (in Russia) (1997) 18. Keat, P.G., Young, P.K.Y., Erfle, S.E.: Managerial economics: economic tools for today’s decision makers. Pearson, Boston (2014) 19. Khlynov, S.A.: Algoritm prinyatiya upravlencheskih reshenij v posrednecheskoj organizacii (Algorithm of acceptance administrative decisions in the intermediary organization). Ekonomika, Statistika i Informatika (Russia). 3: 97–100 (in Russian) (2014) 20. Sokol, A.F., Shurupova, R.V.: Effekt frejminga i ego vliyanie na prinyatie reshenij v medicine (po koncepcii D. Kanemana i A. Tverski) (Framing effect and its influence on decision-making in medicine (according to the concept of D. Kahneman and A. Tversky)). Medicinskij sovet (Russia) 6: 166–168 (in Russian) (2017) 21. Felder, S., Mayrhofer, T.: Medical decision making. Springer-Verlag, Berlin, Germany (2017) 22. Rozikhodjaeva, G.A., Rozikhodjaeva, D.A.: Sravnitel’nyj analiz modelej prinyatiya reshenij v voprosah diagnostiki zabolevanij (Comparative analysis of models of decision-making systems in issues of diagnostics of diseases). Problemy sovremennoj nauki i obrazovaniya (Russia) 15: 97–99 (in Russian) (2017)

Methods and Principles of Designing of Decision Making System of the SEMS Group

Problems with Secure Control of SEMS Group Andrey E. Gorodetskiy, Irina L. Tarasova, and A. Yu. Kuchmin

Abstract Problem statement: Currently, to create robot motion control systems based on modules of smart electromechanical systems SEMS, it is important to develop methods and algorithms that contribute to endowing robots with abilities independently, without human intervention, to formulate safe control problems and successfully solve them in conditions of incomplete certainty. Purpose of research: Analysis of the ways of development of methods for safe movement control of a group of robots and determination of the principles for making optimal decisions taking into account the elimination or minimization of. Results: Analyzed the decisionmaking steps for the sound management based on logical-mathematical processing of sensory information and solving optimization problems with constraints multistep methods generalized mathematical programming. When solving such problems, it is proposed to describe uncertainties by systems of equations in algebra modulo two. Attributes of logical variables reduced to intervals. Decisions about optimality are made based on the concept of sequential preference of one of the compared options to the other. Practical significance: The proposed principles and methods of decisionmaking can be effectively applied in the formation of algorithms for safe motion control of a group of intelligent robots under conditions of incomplete certainty. Keywords Smart electromechanical systems · SEMS · Decision making · Purposeful behavior · Collisions · Optimum and safe

A. E. Gorodetskiy (B) · I. L. Tarasova · A. Yu. Kuchmin Institute for Problems in Mechanical Engineering of the Russian Academy of Sciences (IPME RAS), St. Petersburg 199178, Russia e-mail: [email protected] I. L. Tarasova e-mail: [email protected] A. Yu. Kuchmin e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_6

73

74

A. E. Gorodetskiy et al.

1 Introduction To create a future of humanity that will be literally filled with robots and all sorts of “smart” systems, it is required that these robots and artificial intelligence systems have “instincts” that allow them to avoid collisions with obstacles and with each other while driving. However, if these instincts are too strong, the robots will be too slow, which will negatively affect the effectiveness of their actions. To solve this problem, we need to develop algorithms that constantly strive to find the optimal balance between speed and safety, which will allow robots to always act with high efficiency. Collision avoidance is the main aspect of the systems of all vehicles and other robotic devices that can move completely independently, in automatic mode. Some of the developers of control systems for robot cars deliberately allow them to commit minor traffic violations in the event of a collision hazard. In addition, the task of managing a group of robots has an additional complexity due to the need to ensure coordination between robots. In complex robotic systems, each robot must satisfy its own kinematic equations, as well as existing phase constraints, including dynamic constraints that ensure that there are no collisions between robots.

2 The Principles of Safe Control Safe control is closely related to survivability control, algorithms that are included in the mathematical support of intelligent robots. In this case, the behavior of SEMS can be adjusted due to the flexible response of the automatic survivability control system included in the automatic control system (ACS) to sudden changes in time of external conditions and the internal state of the SEMS itself. The most studied and frequently encountered tasks of survivability control are adaptation, hot redundancy, compensation, and borrowing [1]. Less studied and less common is the problems of stress and stupor or switching on emergency mode. In the process of development of robot ACS and their intellectualization, new modes of their functioning began to appear. In particular, robots created on the basis of SMS are able to work as part of a group of robots under the control of an operator or a higher-level ACS [2]. In this case, there may be situations when the operator’s instructions and/or higher-level ACS will contradict the internal state of the SEMS itself. Another, no less difficult task is to build algorithms for checking the feasibility of conditions. They should probably rely on simulating the behavior of SEMS when executing the proposed operator instructions and/or top-level control system instructions. At the same time, it is desirable that the created algorithms can take into account the possible rapid degradation of SEMS and include survivability control mechanisms in advance with the output of messages to the upper level of group control about the undesirability or danger of the proposed behavior instructions.

Problems with Secure Control of SEMS Group

75

When forming a set of acceptable controls (SEMS behavior instructions), it is first necessary to identify and write to the SEMS ACS database the acceptable values of parameters of individual group members, as well as their static and dynamic characteristics. Then, based on the purpose of a particular SEMS, you need to make a list of possible U ki (t) instructions. Next, you need to identify a set of acceptable Y d (t) behavior instructions by mathematical and computer modeling of the dynamic configuration space. The task is divided into two stages. At the first stage, for example, using computer simulations of SEMS, invalid U*ki (t) instructions are identified among possible U ki (t) instructions, which lead to the output of certain parameters and characteristics beyond the acceptable limits: Uki∗ (t) ⊆ Uki (t) These instructions should be excluded from the possible: Ukid (t) = Uki (t)/Uki∗ (t) At the second stage, dangerous instructions are identified among U dki (t) instructions, i.e. those U oki (t) whose frequent repetition leads to rapid degradation of SEMS with subsequent failures and breakdowns. In this case, logical-probabilistic and logical-linguistic modeling of SEMS degradation is required [3, 4] with analysis of degradation time. If the degradation time of the t di system during repeated application of any U dki (t) instruction is less than the permissible t dop (t di < t dop ), then these instructions are classified as dangerous U oki (t) and they are excluded from the possible ones. Therefore: Yd (t) = (Ukid (t)/Ukio (t)) Next, among the Y d (t) instructions, we identify those U cki (t) that can lead to collisions. They are also excluded from the possible ones. As a result, the safe control instructions will be: Us (t) = (Yd (t)/Ukic (t)) In some cases, when implementing group control systems for robots, some instructions issued by the top-level ACS (coordinator-planner ACS) may not be clear to the SEMS ACS, although they were considered acceptable by the simulation results. This, for example, may be due to incomplete adequacy of the models used. You can partially remove such instructions from acceptable ones by semantic analysis of instructions for correctness and non-inconsistency, and by organizing a dialogue between interacting SEMS ACS. In order to achieve a specific goal for a group of robots, each robot can perform a pre-defined sequence of actions without collisions in the case of a deterministic environment. In the case of a nondeterministic environment, this sequence must be

76

A. E. Gorodetskiy et al.

found by the control system of a group of robots in the process of achieving the goal. At the same time, you first need to synthesize a single control system to stabilize the robot relative to a certain point in the state space with phase constraints. Then you need to look for optimal robot trajectories in the form of points in the state space for robots to move from different initial conditions to the specified end positions. First, we solve the problem of choosing the optimal route for all members of the group without intersections. If the routes of the group members do not intersect and the time to reach the goal of moving the group does not exceed the required time, then the solution to the task is found, otherwise they move on to the next stage. At this stage, the task of driving group members in the case of possible intersections of traffic routes is solved. If the time to reach the goal of moving the group, taking into account the passage of intersections without collisions, does not exceed the required time, then the solution to the task is found, otherwise they proceed to the next stage. At this stage, the problem of safe traffic with the crossing of the trajectories and dynamics of movements with delays at the intersections to avoid collisions, applying the rules of journey of intersections or prioritization of journey of intersections of members of the group.

3 Managing the Safe Movement of the Group Through the Intersection, Taking into Account the Rules of Passage In this case, in the vicinity of the intersection, the surrounding space L 3 is allocated with the dimension L x , L y , L z along the X, Y, and Z axes. It is divided at the beginning of control t 0 into cells eq (t o ) with constant steps hx , hy , hz on the X, Y, Z axes. Robot cars A = {a1 , a2 ,…, an } are located at points S = {s1 , s2 , …, sn } of the surrounding space L 3 and in their corresponding cells. Each of them is characterized by the speed of movement, acceleration, and target points F = {f 1 , f 2 ,…, f n } of this space. They need to arrive at the time t f in the minimum time T. Moreover, the number of possible collisions of robot cars must satisfy the inequality: 

m i j (tk ) ≤ M

(1)

i, j

where: M—the maximum allowed number of collisions, i, j-robot numbers from numbers from 1 to n(i = j), k—the number of the moment of collision time from the time interval T, the value mij (t k ) is determined from the logical expression: «If at time t k the trajectory r i of the robot ai intersects the trajectory r j of the robot aj , that is, r i ∩ r j = 0, then mij (t k ) = 1, otherwise mij (t k ) = 0». Cell sizes (steps hx , hy , hz ) are selected larger than the dimensions of the largest robot car. Each eq (t k ) cell is characterized by the presence or absence of ai robot cars

Problems with Secure Control of SEMS Group

77

and Bi (t k ) obstacles. In addition, each cell is characterized by the interaction of the robot car with the environment in the form of matrices G(t k ) = {G1 (t k ), G2 (t k ),…, Gn (t k )}, describing the effect of the cell environment (road surface, humidity, temperature, etc.) on the dynamic state of the robot. In the linear formulation, these are the transfer functions of the work-car perturbation. If rules are used, the set of cells is characterized by Rm (t k ) traffic rules through an intersection of the type: if-then. These rules are determined by the type of intersection. For example: When passing through an intersection: If: There are no traffic lights and no additional signs and Each street has 1 lane of traffic then: necessary to turn to the right. If: There is a robot car in front of the intersection and It moves at a speed greater than controlled then: the controlled robot car continues to the intersection (moves to the next cell) without braking. The selection environment O(t) containing cells and robot cars changes over time t, i.e. it is dynamic. It can be split into O(t k ) layers with some constant or variable hk step depending on the dynamic properties of robot cars and the disturbing properties of the cell environment. Then, taking into account restrictions of type (1) and other logical, logical-probabilistic, and logical-linguistic restrictions, such as the type of intersection rules, the optimization problem will be: T = t f −t0 → min, where: t f is the end time of the group’s movement, and t 0 is the start time of the movement. This problem can be solved sequentially for each layer of the O(t) selection environment. However, this approach does not guarantee that the entire group of vehicles A = {a1 , a2 , …, an } will pass through the intersection, since the selection environment at each subsequent step depends on the decisions made in the previous steps and may change over time. Therefore, it is necessary to solve this problem using forecasting and modeling sequences of situations during the transition from one layer to another before reaching the final goal.

4 Managing the Group’s Safe Movement Based on Priorities Using the well-known potential field control method to solve this problem is inefficient, since it is essentially kinematic and may not be acceptable for fast-moving SEMS. An example of another approach to solving the problem of safe management

78

A. E. Gorodetskiy et al.

of a group of SEMS with priority setting can be the problem of interaction in the warehouse of robot forklifts built on the basis of SEMS modules. In this task, you need to move three robots from the specified points (areas) to the end points (a rectangular storage room) without colliding with the set robot priority values: PR1 —for robot R1 , PR2 —for robot R2 , and PR3 —for robot R3 , for example, as numbers 1,2,3. In addition, priorities can be determined during the movement of robots in the event of a dangerous situation (possible collision). For this purpose, robot parameters can be compared. For example, if the size of the i-th robot is larger than the j-th, then its priority is greater (PRi > PRj ). Other robot parameters can also be used (speed, proximity to an intersection, weight, destination, etc.). More fine-grained prioritization will be used when comparing several robot parameters at once, especially taking into account their significance by introducing significance coefficients: PRi =

N 

pin kin ,

n=1

where: pin is the nth parameter of the i-th robot, and k in is the coefficient of significance of this parameter. Algorithms for controlling these robots at each step of the movement, i.e. when moving from one cell eq (t i ) of the configuration space to an adjacent one, eq+1 (t i+1 ) determines the possibility of a collision (r i ∩ r j = 0, then mij (t k ) = 1, otherwise mij (t k ) = 0). If mij (t k ) = 1, the control algorithm determines the robot that has the highest priority and gives it a command to pass the intersection, and gives the other robot a command to delay the passage for the time when the first robot passes the intersection, taking into account the maximum braking time, depending on the speed and road conditions. Building such an optimal algorithm is a difficult task, since some of the parameters of control objects and the environment, taking into account possible obstacles on traffic routes, are not fully defined. They can be described as logical-probabilistic and/or logical-linguistic expressions [5]. To solve such problems, it is necessary to use multi-step generalized mathematical programming [6] and software tools of the A-life type [7]. The solution can be significantly simplified by reducing logicalprobabilistic and logical-linguistic expressions to logical-interval expressions [8]. In this case, the group’s travel time will be slightly longer, but the safety is higher.

5 Conclusion The principles and stages of decision-making for safe movement control of a group of robots constructed on the basis of SEMS modules are considered. It is suggested that when forming a set of acceptable controls (SEMS behavior instructions), it is first advisable to identify and record the acceptable values of

Problems with Secure Control of SEMS Group

79

parameters of individual group members, as well as their static and dynamic characteristics, in the SEMS ACS database. The next step is to identify a set of acceptable behavior instructions for group members by mathematical and computer modeling of the dynamic configuration space. The paper describes a mathematical formulation of the problem of controlling the safe movement of a group through an intersection, taking into account the rules of passage. It is proposed to solve this problem using forecasting and modeling sequences of situations during the transition from one layer to another before reaching the final goal. When solving decision-making tasks safe motion control of a group of robots based on their priorities to build an algorithm of motion control of a group of robots should start with the definition of each step of movement, the possibility of collision and the robot with higher priority to command the passage of the intersection, and another robot to give a command to delay the passage at the time of passage crossing the first robot with the maximum brake time, depending on speed and road conditions. To solve such problems, you can use multi-step generalized mathematical programming and software tools such as A-life. Acknowledgements The present work was supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAA-A19119120290136-9 and is supported by grants RFBR No. 18-01-00076 and No. 19-08-00079.

References 1. Gorodetskiy, A.E., Tarasova, I.L.: Upravlenie i nejronnye seti [Control and neural networks]. St. Petersburg, Polytechnic. University Publication, p. 312 (In Russian) (2005) 2. Gorodetskiy, A.E., Tarasova, I.L.: Smart electromechanical systems: group interaction (Studies in systems, decision and control), vol. 174, p. 337. Springer International Publishing (2018). https://doi.org/10.1007/978-3-319-99759-9 3. Ziniakov, V.Y., Gorodetskiy, A.E., Tarasova, I.L.: Control of vitality and reliability analysis. In: Gorodetskiy, A.E. (ed.) Smart Electromechanical Systems, pp. 193–204. Springer International Publishing (2016). https://doi.org/10.1007/978-3-319-27547-5_18 4. Ziniakov, V.Y., Gorodetskiy, A.E., Tarasova, I.L.: System failure probability modelling. In: Gorodetskiy, A.E. (ed.) Smart Electromechanical Systems, pp. 25–44. Springer International Publishing (2016). https://doi.org/10.1007/978-3-319-27547-5_4 5. Gorodetskiy, A.E., Kurbanov, V.G., Tarasova, I.L.: Methods of synthesis of optimal intelligent control systems SEMS. In: Gorodetskiy, A.E. (ed.) Smart Electromechanical Systems, pp. 205– 216. Springer International Publishing (2016). https://doi.org/10.1007/978-3-319-27547-5_19 6. Iudin, D.B.: Vychislitel’nye metody teorii priniatiia reshenii [Computational methods of decision theory], p. 320. Moscow, Nauka Publishing (In Russian) (1989) 7. Gorodetskiy, A.: Osnovy teorii intellektual’nyh sistem upravleniya [Fundamentals of the theory of intelligent control systems] LAP LAMBERT Academic Publishing GmbH@Co. KG, p. 313 (2011) 8. Gorodetskiy, A.E., Tarasova, I.L. Kurbanov, V.G.: Reduction of logical-probabilistic and logicallinguistic constraints to interval constraints in the synthesis of optimal SEMS. In: Gorodetskiy, A.E., Tarasova, I.L. (eds.) Smart Electromechanical Systems: Group Interaction, pp. 77–90. Springer International Publishing (2018). https://doi.org/10.1007/978-3-319-99759-9_7

Synthesis of Optimal Program Control for Synchronizing the Movements of a Group of SEMS Modules A. Yu. Kuchmin

Abstract Problem statement: At present, in the direction of the development of cyberphysical systems, the main attention is paid to the issues of interaction of groups of self-organizing dynamic SEMS modules with a parallel kinematic scheme in order to develop optimal, fast and reliable methods and algorithms for controlling cooperative movement based on technologies from the field of decision making. The classical formulation of the problem in this situation leads to problems of multidimensional conditional optimization with a huge number of constraints, including nonlinear ones, including linguistic, interval and probabilistic characteristics that are difficult to formalize for numerical methods. Classical universal methods for solving such problems do not give guaranteed results for possible combinations of input parameters from a wide range of values, have a low probability of finding a correct solution in a small fixed time interval allotted for making a decision. Purpose of research: development of technology for creating special fast and reliable methods for synchronizing movements based on mathematical programming methods. Results: the priority directions of development of methods of optimal synchronization in such systems are determined, in particular, the advantages of the organization of control according to the “weak link” are shown. Fast algorithms for conditional optimization have been developed, which guarantee a solution in a number of steps not exceeding a given fixed value. These algorithms are focused on using interval data. Practical significance: In the course of the work, a software package was developed for designing systems for optimal synchronization of elements of SEMS modules during their group interaction and modeling control systems for multi-link robots composed of such modules, the basic component of which is the SEMS block, which is an adaptive platform moved by electromechanical jacks (actuators). Keywords Smart electromechanical systems · SEMS · Decision making · Optimality A. Yu. Kuchmin (B) Institute for Problems in Mechanical Engineering of the Russian Academy of Sciences, (IPME RAS), V.O., Bolshoj pr., 61, St. Petersburg 199178, Russia e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_7

81

82

A. Yu. Kuchmin

1 Introduction One of the poorly formalized problems in the field of control of distributed systems with a parallel kinematic scheme in the framework of the classical optimization approach is the synchronization of SEMS modules and their components during group movement under constraints on phase coordinates in order to ensure the safety of movements and the absence of conflicts or special positions. The key task is to synthesize programmed trajectories of complex systems, consisting of interacting dynamic SEMS modules [1–7] with a parallel kinematic scheme. It is necessary to create, on the one hand, an easy to implement and, on the other hand, an effective algorithm for synthesizing the optimal trajectory of motion in the presence of complex nonlinear constraints. Therefore, the author sees the solution to this problem in the development of highly specialized methods and algorithms for optimal synchronization of the movements of SEMS modules and their elements, based on the idea of organizing control according to the “weak link” based on the provisions of the theory of constraints. The transfer of a complex multi-link system with a large number of degrees of freedom from the initial state to the desired one can be organized by different trajectories using different groups of drives [8]. It is required to find the optimal translation strategy according to the specified criteria and restrictions, which are described in detail in the next section. As will be described below, a similar problem in a general setting is difficult to solve even for simple systems. Practical experience in the synthesis of such systems shows that when planning movements, the most stringent constraints are the characteristics and phase coordinates of critical subsystems, which determine the dynamics of the system and the allowed areas of movements. This management has received the name: management according to the “weak link”, according to the most stringent restrictions that absorb all the rest. Thus, for a certain type of target control criteria, the optimal solution to the control problem will be on the boundary of the region of feasible solutions formed by the constraints of “weak links”. The article describes the general structure of a multi-level control system for such multi-link objects, and the ideas of creating algorithms for the synthesis of program movements using the “weak link” method are explained using the example of the trajectory planner of the SEMS module and its elements under constraints. A fast algorithm for calculating the programmed trajectory, taking into account the speed and torque of electric drives, is presented.

2 Statement of the Problem of Optimal Control Synthesis for a Mechanism Consisting of SEMS Modules Calculation of the laws of optimal control of a group of SEMS modules is a complex multicriteria optimization problem of large dimension with a large number

Synthesis of Optimal Program Control for Synchronizing …

83

of nonlinear and interval constraints describing the kinematic and dynamic relationships in such systems, as well as the conditions for the physical realizability of movements in such systems, taking into account special provisions, such as loss of stability structures and jamming. For this reason, the “head-on” solution of such a problem by using classical methods in practice is impossible. The structure of the problem depends on the topology of the SEMS modules and the way they interact [9–12]. Typical criteria for optimality in such systems are: • The criterion for maximum performance J T , which consists in minimizing the transition time of the system from the initial position to the desired final. • The criterion of minimum displacements J L , which consists in minimizing displacements of individual SEMS modules when the specified time and accuracy characteristics of the movement of the system as a whole are achieved. This criterion is of great practical importance and ultimately leads to the creation of compact systems with minimum stroke lengths of the actuators of individual SEMS modules. • The criterion for maximum accuracy J e , which consists in minimizing the pointing and positioning errors of the SEMS modules. • The criterion of the minimum power of movements J P , which in this case is used as a more convenient analogue of the criterion for the minimum energy consumption (maximization of the energy resource). • The criterion for smooth movement J S , especially important, for example, when creating medical robotic manipulators, where continuous tracking of a given trajectory is required without abrupt rearrangements of modules. • The criterion for smooth movement, especially important, for example, when creating medical robotic manipulators, where continuous tracking of a given trajectory is required without abrupt rearrangements of modules. The constraints in such systems are: 1. 2. 3.

4. 5.

Interval constraints of system state Gx and control constrains Gu . Interval constraints on the moments and forces of the actuators Gf , constrains on the movements Gl and the speed of movements Gv of the actuators. The constrains GS that determine the configuration of the space in which such multi-module systems move, as well as restricted areas in which they cannot enter. The constrains related to kinematics GK and dynamics GD of such systems. The constrains due to interaction with the external environment GL . Such constraints are usually non-stationary, often random, and poorly formalized.

Thus, in the general case, the problem of synthesizing the optimal control of such systems can be formulated as a nonlinear multidimensional multicriteria conditional optimization problem with a large number of poorly formalized constraints, often of an interval nature. The classic approach to solving such problems is the use of nonlinear programming methods.

84

A. Yu. Kuchmin

In a general setting, such problems, even for simple systems (tripod, hexapod), do not have effective methods of solution, many control issues are solved by technical means, such as: processing of friction pairs, devices for selecting backlash, installing limit switches, clutches, etc. All these measures make it possible to achieve the required characteristics within a short time of operation of such systems, without removing the issue of the systems entering modes where special provisions are implemented that lead to accidents. In the general setting, the optimal control problem for a system composed of SEMS modules has the form. The criteria can be expressed as follows: JT = min TK , u



TK

JL = min u

i

TK Je = min u

li2 dt,

0

 T   y − yg M y − yg dt,

0



TK

J P = min u

i

(vi f i )2 dt,

0

  JS (t) = min (y(t) − ys.m. (t))T M(y(t) − ys.m. (t)) , u

(1)

where, u—vector of control, I—actuator number, T K —transient time, l—movement of the actuator, v—actuator travel velocity, f actuator force, y—system output coordinates, for example, the coordinates of the tool being moved by the system, yg — desired system output coordinates (control target), M—diagonal matrix of normalizing coefficients, ys.m. —a model of the trajectory of motion, taking into account the transient process, satisfying the conditions of smoothness. The constrains can be described as follows: G x : x j,min ≤ x j ≤ x j,max , G u : u j,min ≤ u j ≤ u j,max , G l : li,min ≤ li ≤ li,max , G v : vi,min ≤ vi ≤ vi,max , G f : f i,min ≤ f i ≤ f i,max , G S : x j ∈ H j , G K : 1 (x)x + 2 (x)x + ς 1 = 0, G D : x = 3 (x)x + 4 (x)u + 5 (x)ξ + ς 2 , G L : ξ  = 6 (x, t)ξ + ς 3 ,

(2)

where, j—the vector component number, min and max indices denote the minimum and maximum values of the corresponding variables, x—the state vector, H—the areas of phase space in which movement of the mechanism is allowed, 1 , 2 , 3 , 4 , 5 —the matrices in models of kinematics and dynamics of the mechanism, ς1 ,

Synthesis of Optimal Program Control for Synchronizing …

85

ς2 , ς3 —the vectors that determine the non-rigid fulfillment of the corresponding constraints for the real mechanism due to the inaccuracy of the models, usually have an interval or random nature, ξ—the external loading vector, 6 —matrix in the model of external loadings. As already noted, in the direct formulation, the solution of problem (1, 2) causes great difficulties even for the simplest systems of a parallel kinematic scheme; on the basis of the classical approaches of the theory of optimal control, effective algorithms have not been created that allow real-time computations with the required accuracy in the onboard controller. So with different numerical methods of solving, different results can be obtained from each other, with different organization of the multicriteria selection procedure. One of the ways to overcome this problem is to represent (1, 2) in the form of a hierarchical system of interacting subproblems of optimal control. For (1, 2), a wellknown hierarchical division into layers (decision-making levels) can be applied; in this case, simplified subproblems can be solved sequentially-parallel in different time scales. A four-level traffic control system is proposed, the higher layer number corresponds to the higher hierarchy level: Layer 1: Tactical controller of the SEMS module, controls the movements of the actuators, with non-stationary linear constraints on the phase coordinates of the drives and control actions. Layer 2: The group controller of the SEMS module, controls the movements of the SEMS bases, makes corrections in the synchronization of the actuators in order to prevent the development of special positions and compensate for the influence of external loadings, which can be compensated using this SEMS module, with non-stationary linear constraints on the phase coordinates of the SEMS module. Layer number 3. The SEMS module trajectory planner calculates the optimal trajectory of the SEMS module, with a simplified system of constraints obtained from layer 4. Layer number 4. The trajectory planner of the entire system, which, based on the strategic goal of control, calculates the trajectory of the entire system in such a way that the mechanism does not leave the region of the phase space H, in which movement is allowed. The boundaries of the H region are modified according to data obtained from lower levels. Strategic trajectory, recalculated in the form of a control goal and a system of constraints for level 3. In contrast to the well-known control approaches using multilayer systems in robotics, an iterative procedure for changing the characteristics of each layer is proposed, which is a recurrent solution of a hierarchical sequence of mathematical programming problems. Since the volume of the article does not allow covering the entire complex structure of the control system for such objects, the main ideas of the proposed method will be shown on the example of work organization and synthesis of the optimal trajectory of programmed motion in layer 3. Below will be given a method for calculating the programmed trajectory of the SEMS module in the presence of restrictions on positioning error, transient time, displacement velocity, displacement acceleration, displacement geometry and moments of electric drives. The result of the calculations

86

A. Yu. Kuchmin

obtained using this method is the programmed trajectory of the block and control actions (elongation of the rods, the velocity and acceleration of their elongations), which are transferred to layer 2 as input data.

3 The Problem of Synthesizing the Trajectory of the SEMS Module by the Planner in Layer 3 As already noted, in layer 3 the optimal trajectory of the SEMS module is calculated, with a simplified system of restrictions obtained from layer 4. When formulating this problem, you can use the methods for synthesizing the optimal control of the hexapod described in [13] and develop them for an adaptive platform with a parallel kinematic scheme and moved by n actuators (n-pod) [14]. Thus, in the case of an n-hearth, the output data of the trajectory planner of the SEMS module is the development of the laws of extension of the rods of each actuator, taking into account the velocity of the actuator drives, the inertia of the n-hearth and constrains on the phase coordinates and geometry of displacements. The trajectory of movement of the n-pod is specified in the base coordinate system as three linear and three angular movements of the control point. The requirements for the dynamics, accuracy and quality of the transient process for the SEMS module as a closed system are set using a linear reference dynamic model in deviations from the desired state (control goal). The laws of variation of linear ηr and angular ηϕ guidance errors, determined by this model, take into account the dynamics of electric drives and the inertia of the n-pod through the time constant of the SEMS module and have the form: ⎡ ⎡  ⎤  ⎤ − Tt − Tt − t − t a1 a4 h − T e T1 h − T e T4 T T e e h 1 h 4 T −T T −T ⎢ h 1 ⎢ h 4 ⎥ ⎥ ⎢ 2 ⎢ a5 − Tt − Tt − t ⎥ − t ⎥ h − T e T2 h − T e T5 , η T T e = e ηr = ⎢ Tha−T ⎢ ⎥ ⎥, (3) h 2 ϕ h 5 2 ⎣ ⎣ Th −T5  ⎦ ⎦ t t − Tt − − Tt − a3 a 6 Th e h − T3 e 3 Th e Th − T6 e T6 Th −T3 Th −T6 where, ai —the initial deviation along the corresponding coordinate, T h —the time constant of SEMS module, T i —the time constants of a closed system with a controller, t—a time. The values T h and ai are calculated on layer 4, taking into account the displacements of the entire system, based on the kinematic and dynamic models of the multi-module mechanism and the area of permissible displacements. The outputs of the SEMS trajectory planner is the elongations l j and elongation rates l j of the actuators, which depend on the deviations ηr and ηϕ . Thus, it is necessary to find the optimal values of the transient process time T K and time constants T i of the closed-loop system controller at a given pointing accuracy for each coordinate δi , and simplified constraints on the maximum lengths, actuator rates, with constraints on the coordinates, movement speed and acceleration of the SEMS module, converted into relative displacements in layer 4.

Synthesis of Optimal Program Control for Synchronizing …

87

In mathematical form, the search for the optimal trajectory is formulated as the problem of maximum performance in the presence of nonlinear constraints: J = min{x1 },

(4)

k1 Th ≤ x1 ≤ Tmax ,

(5)

x

list of constrains:

0 < xi+1 ≤

k2 , Th

(6)

xi+7 > 0,

(7)

x1 − xi+7 > 0,

(8)

−ai xi+1 ≤ λimax , Th   −x1 sign(ai )δi sign(ai )δi −x1 xi+1 Th x i+1 − e − Th e − = 0, ai ai   1 −xTi+7 ai −x x i+7 i+1 = 0, e h − xi+1 e Th − x 1 Th −λimax ≤

(9) (10) (11)

i+1

−vimax ≤

 −ai  −xTi+7 e h − e−xi+7 xi+1 ≤ vimax , 1 Th − xi+1 l j + l0 j ≤ lmax j ,   l  ≤ l  j

max j ,

(12) (13) (14)

where, x T max x i+7 k1, k2 λimax vimax

the vector of variables, x 1 = T K the maximum transient time, xi+1 = T1i ; the moments in time at which the rates of change of linear and angular deviations have maximum values in absolute value the weighting factors the maximum values of acceleration of change of linear and angular deviations the maximum value of the rates of change of linear and angular deviations; i = 1..6;

88

A. Yu. Kuchmin

lmax j lmax j l0 j

the maximum length of the j-th actuator the maximum elongation rate of the j-th actuator the initial value of the actuator length, j = 1..n.

Constraints (5, 6) determine the minimum permissible value of the transient process time and time constants T i , associated with the fact that the response speed of the closed system as a whole cannot be higher than the response speed of its subsystems. Constraints (7, 8) determine the minimum and maximum permissible values of x i+7 , since the selected laws of change in the length of the actuators have pronounced stages of acceleration and deceleration. Constraints (9) determine the maximum modulus of permissible values of the acceleration of linear and angular deviations. This restriction was obtained by double differentiating of (3) with respect to time and maximizing the found acceleration on the interval [0, T K ]:     1 − Tt −ai −ai 1 − Tt 1 − Tt −xi+1 t i h h = xi+1 e . (15) e − e − e ηi  = Th − Ti Ti Th Th Th − x1 i+1 It is not difficult to show that ηi  have maximum values in absolute value at t = 0 i x . The maximum values of these accelerations λimax can be equal to ηi (t0 ) = −a Th i+1 determined through the maximum starting moments of the actuator electric drives. Constraints (10) determine the maximum allowable pointing errors. They are obtained by simplifying the following equation: ηi (TK ) =

  x 1 −x1 xi+1 ai − T1 h − T = sign(ai )δi . e e h 1 xi+1 Th − xi+1

(16)

Constraints (11) determine the times at which the rates of change of linear and angular deviations have maximum values in absolute value. Obtained from the condition that acceleration (15) is equal to zero at t = x i+7 . Constraints (12) determine the maximum modulus of admissible values of the rates of change of linear and angular deviations at the moments of time x i+7 . Constraints (13, 14) determine the range of linear displacements and linear displacement velocities of the actuators.

4 Motion Trajectory Synthesis Using Gradient Optimization Methods The standard method for solving problem (4–14) is the use of gradient or Newton’s optimization methods [15, 16]. Functional (4) has a linear objective function that depends only on the time of the transient process, which, when using gradient or Newton’s optimization methods, leads to poor conditioning of calculations and poor convergence of numerical methods. To improve convergence, another functional can

Synthesis of Optimal Program Control for Synchronizing …

89

be introduced that will provide the necessary scaling in calculations: 

 6   mi J = min x1 + + m i+6 xi+7 , x xi+1 i=1

(17)

where, mi , mi+6 —the scale factors selected in such a way as to ensure good convergence of optimization methods. The choice of these coefficients is difficult in view of the nonlinear form of the objective function in (16) and constraints (5–14). In practice, it is easy to determine the ranges of variation of the scale factors by reference transients at the design stage. In this case, the scale factors m can be introduced into the problem as additional variables with appropriate constrains on their ranges. Then (17) takes the form: 

 6   mi + m i+6 xi+7 , J = min x1 + x,m xi+1 i=1

(18)

and the system of constraints (5–14) is supplemented with interval constraints: m min,i ≤ m i ≤ m max,i ,

(19)

where, m min,i and m max,i

the minimum and maximum values of the corresponding coefficients.

The resulting problem (5–14, 18, 19) belongs to the class of constrained optimization problems and the use of gradient or Newton’s methods is possible only when using the Lagrange multiplier method. As a result, we get a complex nonlinear multivariate function with a large number of variables. Numerical experiments using similar methods have shown their low efficiency in solving this problem. The methods showed acceptable results in the case when the deviation of the parameters of the sought solutions did not exceed 10–15% of the reference ones synthesized at the design stage. Therefore, algorithms based on such methods cannot be used in controllers for trajectory synthesis in real time, since the probability of obtaining adequate trajectories with their help is very low for wide ranges of changes in the phase coordinates of the SEMS module.

90

A. Yu. Kuchmin

5 Trajectory Synthesis Using the Multidimensional Patchwork Shell Method (MPSM) To solve problem (4–14), one can use the method described in [17], the main idea of which is as follows: if the objective function is monotonic and the system of constraints forms a closed region, the optimal solution should be sought at the boundary of this region. Problem (4–14) corresponds to the conditions of this method; therefore, it is necessary to construct this domain by analyzing constraints (5–14). In [17], the analysis of exponential constraints of the form (10–12) is described in detail. It is proposed to express the variables x i+1 in terms of x 1 , using constraints (10); for this, the replacement wi = x 1 x i+1 is introduced, then (10) takes the form: e−wi = −μi wi + σi , σi = μi =

sign(ai )δi , ai

sign(ai )δi Th −x1 − e Th , μi > 0, σi > 0. ai x1 x1

Then the variables x i+1 are found from the formula:   σ 1 σi 1 −1 − μ ix ( ) 1 i − = gi (x1 ), xi+1 = e + f μi (x1 )x1 x1 μi (x1 ) μi (x1 ) > 0, μi (x1 ) − μi (x1 ) ln μi (x1 ) − σi < 0,

(20)

(21)

where, f −1

the Lambert function.

It is easy to see that the variables x i+7 can be expressed in terms of g(x 1 ) using the constraints (11): xi+7 =

1 Th

  1 1 = h i (x1 ). ln Th gi (x1 ) − gi (x1 )

(22)

As a result of the transformations performed, the original multivariate optimization problem with 13 variables is reduced to a one-dimensional search problem under the constraints: k1 Th ≤ x1 ≤ Tmax ,

(23)

k2 , Th

(24)

ε ≤ gi (x1 ) ≤

h i (x1 ) ≥ ε,

(25)

Synthesis of Optimal Program Control for Synchronizing …

x1 − h i (x1 ) ≥ ε, −λimax ≤ −vimax

−ai gi (x1 ) ≤ λimax , Th

 −h x  i ( 1) −ai −h i (x1 )gi (x1 ) Th e ≤ vimax , ≤ −e Th − gi (x1 1 )

91

(26) (27) (28)

l j (x1 ) + l0 j ≤ lmax j ,

(29)

   −lmax j ≤ l j (x 1 ) ≤ l max j ,

(30)

μi (x1 ) ≥ ε,

(31)

μi (x1 ) − μi (x1 ) ln μi (x1 ) − σi ≤ ε,

(32)

σi (x1 ) ≥ ε,

(33)

ε—the small positive value that determines the accuracy of calculations. Each constraint from (23–33) is an inequality in one variable x 1 . These inequalities can be addressed independently. The solution to each inequality is a system of intervals for x 1 , the boundaries of which are generally determined by complex nonlinear expressions. The solution to the optimization problem is the smallest boundary of one of the intervals, provided that there is an interval or number that satisfies the system of constraints (23–33). The numerical implementation of the search method for the variable x 1 has a high parallelization factor of calculations and can be solved in its original form at the design stage or in a high-performance controller. For quick calculations, the boundaries of the intervals can be approximated on the interval k 1 T h ≤ x 1 ≤ T max by fractions, the exponents of the polynomials of the numerators and denominators of which do not exceed 4. In this case, the constraints are described by adequate systems of linear inequalities in one variable x 1 . This problem was described as an interval problem and boiled down to finding the upper and lower estimates for x 1 . The solution to the maximum speed problem will be the lower bound for x 1 . Unlike the methods described in Sect. 4, the proposed algorithm does not use the differentiation operation, has a high parallelization ratio and finds a solution or reveals its absence in a predicted number of steps in the entire operating range of variable parameters and the state of the SEMS module. In numerical experiments, the algorithm found a solution or proved their absence in 95% of cases.

92

A. Yu. Kuchmin

6 Synthesis of a Trajectory Using a Method Based on the Construction of the Boundaries of the Region of Feasible Solutions Numerical and analytical studies of the problem (4, 23–33) made it possible to create a simpler algorithm for finding a solution, which consists in the fact that it is possible to identify a number of critical points of the boundaries of the region of feasible solutions that can be a solution to the problem. This algorithm can be used when searching for a trajectory near the one obtained using the algorithm from the previous section. Algorithm for finding the settling time T K and time constants T i : 1. 2. 3. 4. 5.

Start. Input of initial data. The lower bounds of the values of the time constants according to time constant of drive are calculated: Ti,1 = Tk2h . The lower bounds of the values of the time constants according to maximums |ai | . of drive accelerations are calculated: Ti,2 = λmax Th i The lower bounds of the values of the timeconstants  according to maximums vimax Th Th    max . of drive velocities are calculated: Ti,3 = ln |ai | v T v max T f −1 ln

6. 7.

8.



10.

|ai |

h

i

|ai |

h

The maximum values  of the corresponding time constants are calculated: Ti,4 =  max Ti,1 , Ti,2 , Ti,3 . Calculation of the maximum time of the transient process using constraints (10) at the values of time constants Ti,4 and constraints (29, 30) determined from the mechanics model of the SEMS module. Calculating the resulting time constants: Ti,5 =

9.

i

σi∗

μi∗ TK

+

   −1 K σ∗ 1 1 −1 δi δi Th −T , μi∗ = − exp − i∗ f , σi∗ = − e Th . ∗ |ai | |ai |TK TK μi μi TK

Checking constraints (23–33). If everything is fulfilled, then the solution is found, the value of the time of the transient process is determined in step 7, and the time constants in step 8. If at least one is not fulfilled, then the problem has no solution. End.

Despite the rejection of pure one-dimensional search and return to multidimensional optimization, the proposed algorithm in steps 3–6, 8 has simple analytical formulas for calculating the desired variables, which increases the speed and accuracy of calculations.

Synthesis of Optimal Program Control for Synchronizing …

93

7 Conclusions The article describes the structure of a multi-level control system for a complex multitier dynamic object, composed of SEMS modules, based on the “weak link” control method. The main problems of optimal control of such systems are analyzed. It is shown that one of the main tasks is the synthesis of optimal programmed trajectories of motion of such systems, taking into account the constraints. The method of synthesizing such trajectories is demonstrated by the example of a motion planner for the SEMS module with a parallel kinematic scheme (n-pod). The method is universal and can be used to synthesize optimal trajectories of both SEMS modules themselves and systems consisting of such modules. In this example, n-pod is used as a control object, the model of which and constraints on characteristics are specified by expressions (29, 30). This formulation can also be used to synthesize systems consisting of SEMS modules when replacing the original n-pod model with a simplified model of the mechanics of the entire system. Effective algorithms for conditional optimization are proposed for solving the problem of synthesizing programmed movements of such systems that do not use the differentiation operation. These algorithms have a high parallelization factor and find a solution for a predictable number of steps in the entire operating range of variable parameters and the state of the SEMS module. Acknowledgements The present work was supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAA-A19119120290136-9 and is supported by grants RFBR No. 18-01-00076 and No. 19-08-00079.

References 1. Gorodetskiy, A.E., Kurbanov, V.G. (eds.): Smart Electromechanical Systems: The Central Nervous System, p. 266. Springer International Publishing AG (2017) 2. Gorodetskiy, A.E. (ed.): Smart Electromechanical Systems, p. 277. Springer International Publishing Switzerland (2016) 3. Volkomorov, S.V., Kaganov, Y.T., Karpenko, A.P.: Modelling and optimization of some parallel mechanisms. New Technologies, Moscow, p. 32 (2010) (In Russian) 4. Glazunov, V.A., Koliskor, A.S., Kraynev, A.F.: Spatial mechanisms of parallel structure. M.: Science, p. 96 (1991) (In Russian) 5. Zenkevich, S.L., Yushchenko, A.S.: Bases of Control of Handling Robots. M.: MSTU.-2004.480 pages (In Russian) 6. Merlet, J.P.: Parallel Robots (Solid Mechanics and Its Applications). Springer, Berlin (2004) 7. Heylo, S.V., Glazunov, V.A., Palochkin, S.V.: Handling mechanisms of parallel structure. The dynamic analysis and management—M.: MGUDT, p. 86 (2014) (In Russian) 8. Gorodetskij, A.E., Kurbanov, V.G., Tarasova, I.L.: Adaptive Gripping device. Patent for the invention of RUS 2624278 C1 12/07/2016 (In Russian) https://www1.fips.ru/wps/PA_Fip sPub/res/Doc/IZPM/RUNWC1/000/000/002/624/278/%D0%98%D0%97-02624278-00001/ DOCUMENT.PDF (04.07.2019)

94

A. Yu. Kuchmin

9. Gorodetskiy, A.E.: Smart electromechanical systems architectures. In: Gorodetskiy, A. E. (ed.), Smart Electromechanical Systems, pp. 17–23. Springer International Publishing (2016). https:// doi.org/10.1007/978-3-319-27547-5_3 10. Kurbanov, V.G., Gorodetskiy, A.E., Tarasova, I.L.: Automatic control systems of SEMS. In: Gorodetskiy, A.E. (ed.), Smart Electromechanical Systems, pp.73–79. Springer International Publishing (2016). https://doi.org/10.1007/978-3-319-27547-5_7 11. Tarasova, I.L., Gorodetskiy, A.E., Kurbanov, V.G.: Mathematical models of the automatic control systems SEMS modules. In Gorodetskiy, A.E. (ed.), Smart Electromechanical Systems, pp. 149–158. Springer International Publishing (2016). https://doi.org/10.1007/978-3-31927547-5_14 12. Gorodetskiy, A.E., Tarasova, I.L., Kurbanov, V.G.: Methods of dealing with jamming in automatic control systems of the modules SEMS. In Gorodetskiy, A.E. (ed.), Smart Electromechanical Systems, pp. 217–224. Springer International Publishing (2016). https://doi.org/10. 1007/978-3-319-27547-5_20 13. Artemenko, Y. N., Agapov, V. A., Dubarenko, V. V., Kuchmin, A. Y.: Co-operative control of subdish actuators of radio telescope. Informatsionno-upravliaiushchie sistemy 4, 2–9 (2012) (In Russian) 14. Kuchmin, A. Y., Dubarenko, V. V.: Linearized model of the mechanism with parallel structure. In Gorodetskiy, A.E., Kurbanov, V.G. (eds.) Smart Electromechanical Systems: The Central Nervous System, p. 266. Springer International Publishing AG (2017). https://doi.org/10.1007/ 978-3-319-53327-8_13 15. Tabak, D., Kuo, B.C.: Optimal Control by Mathematical Programming, p. 280. New Jersey, Prentice-Hall, Englewood Cliffs (1971) 16. Zangwill, W.I.: Nonlinear Programming. A Unified Approach, p. 312. Englewood Cliffs, New Jersey, Prentice-Hall (1969) 17. Kuchmin, A. Y.: A nonlinear programming method with arbitrary restrictions. Informatsionnoupravliaiushchie sistemy [Information and Control Systems] (2), 2–10 (2016) (In Russian). https://doi.org/10.15217/issn1684-8853.2016.2.2

Task Scheduling Within Robots’ Collectives of Arbitrary Structures Alexander Fridman and Boris A. Kulik

Abstract Problem statement: To coordinate interactions in groups of robots that jointly perform a task, it is necessary to correctly distribute subtasks for each robot. In general, robots in a group can have equal ranks or be divided into subgroups, each of which contains one leader (coordinator) responsible for the distribution of tasks and coordination of interactions within the subgroup. To solve this problem, it is necessary to correctly manage such teams. Purpose of research: It is required to develop a method for flexible distribution and redistribution of tasks in teams of robots with an arbitrary group structure, taking into account changes in the current situation. Results: It is proposed to solve the described problem on the basis of the previously proposed by the authors quantitative assessments of situation awareness and its three main aspects (steps), namely perception of the essential elements of the environment, comprehending (estimation of their importance) and forecasting of possible future states. It is assumed that the work is carried out in the absence of a conscious (targeted) counteraction of the environment, that is, the task has a commercial (civilian) character. The presented approach differs by: considering both normal and emergency operation of the team under study; operative formation of areas of responsibility for each decision-making robot. Practical significance: Development allows to objectify decision-making support in teams of robots with an arbitrary organizational structure of subgroups. Keywords Robots’ cooperation · Quantitative assessment of situation awareness · Coordination of robots’ interactions · Normal and emergency operation mode

A. Fridman Institute for Informatics and Mathematical Modelling, Kola Science Centre of RAS, Apatity, Russia e-mail: [email protected] B. A. Kulik (B) Institute for Problems in Mechanical Engineering of the Russian Academy of Sciences, St. Petersburg, Russia e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_8

95

96

A. Fridman and B. A. Kulik

1 Introduction In our previous paper [11], we proposed some numerical estimates for measuring the situational awareness (SA) (see, for instance, [6, 8, 17]) and its three main levels (stages), namely understanding of what is happening in the environment in order to comprehend how information, events, and one’s own actions will affect goals and objectives at the moment and in the near future. These estimates were there used for solving coordination problems in peer-ranked and hierarchical groups of interacting robots. However, we did not consider a possible case when a group of robots (agents) needs to change involved decision-makers (DMs). Such a case can arise if a DM turns to an emergency state and/or its functioning becomes ineffective. To investigate this case, we further analyze a group of robots as a net-centric system (NCS) ([7, 13, etc., etc.) and apply our numerical estimates of SA in order to decide proper DMs and operatively schedule tasks to members of the group. The rest part of this paper is organized as follows. A brief analysis of the NCSs features and applications is given to conclude their essential characteristics with regard to our goal. Then an SA synopsis is provided. Finally, our ideas on choosing DMs and their areas of responsibility (ARs) are given on the basis of the SA degree (SAD) currently achieved by every DM.

2 NCSs: Features, Legends and Our Proposals The net-centric approach was initially developed for military tasks [7], as swift changes of situations during a battle leave no time for solving all emerging problems by means of a centralized control. Net-centric applications of warfare suppose to reach strategic goals via information and communication networks. However, nowadays this kind of networks is present in most complex computerized environments, and there exist a huge number of applications attributed by their authors as net-centric. To begin classifying such environments, we divided them into military [3] and commercial (civilian) applications. Then we excluded the first type of applications from the further analysis since they cannot be subjected to detailed investigation due to secrecy reasons. Respectively, we rejected game methods of creation net-centric environments as the problems and systems analyzed below stipulate absence of conscious counteractions to the decisions to be made. So, we have to just consider uncertainties, which always exist in so-called “moves by nature” from game theory [30]. However, the remaining civilian areas called “net-centric” ones are also quite numerous: business structures and transport networks [13], computer environments themselves [27], socio-economic formations [26], swarm associations [20], etc. [19, 24]. To our minds, the first alarming feature of the aforementioned and other publications is that they lack formalized or standardized definition of the concept of “net-centric”. Some papers state that net-centric structures are alternative and opposite to the hierarchical approach [14], no fewer texts consider these

Task Scheduling Within Robots’ Collectives of Arbitrary Structures

97

approaches as compatible ones, for instance, by admitting hierarchical subsystems within the network [22] or even building an NCS as a three-layer hierarchy [12]. Most publications declare the need and importance of the integrated use of information, but rarely discuss goals, principles, and even more so algorithms for such use. Multi-agent systems [18], network organizations [21], and distributed systems [28], and active systems [23], and other things [15, 25] are also referred to as netcentric ones. Strictly speaking, even the legitimacy of combination of the terms “netcentric” and “system” in the descriptions of most applications raises serious doubts, since according to the generally accepted principle in system analysis, any system should include only subsystems and elements, in the absence of which the goal of the system is not achieved (it is the necessary condition for existence of any system called integrity), but their presence guarantees appearance of new properties that no components of the system had (it is a sufficient condition for existence of the system called emergentness) [2]. Of course, in complex systems, the integrity requirement is weakened due to the need of providing structural redundancy to achieve robustness that is the ability of a system to achieve its goal (possibly with less efficiency) in case of failure of some of its components or connections. However, no one has cancelled the demand for emergentness, and it was not possible to find its justification for NCSs in the analyzed publications. Nevertheless, self-organization [4], survivability [5], and a synergistic effect [29] are declared for NCSs without any explanations together with other pleasant features of existing net-centric structures. In connection with the foregoing, below we try to objectify the concept of a “netcentric system” based on the concept of SA that was originally present in works on NCSs [1], but did not have (as far as we can judge according to open publications) any constructive implementation. To keep the terminology correct, we assume that any net-centric structure is functionally and territorially a network, each node (vertex) in which has its own goal, defined by some quality criterion, and makes decisions to extremise this criterion (for definiteness – to maximise it). All hierarchical relationships are “hidden” in the internal structure of nodes, so the latter can have arbitrary complexity in spatial and functional terms. Depending on the situation, each network node can become the centre of some connected part or the entire system as a whole, then such node gains the right and opportunity to coordinate interactions of nodes that became subordinate to it (they are included in its AR [16]). Initially, the network is a peer-to-peer one, not only and not so much in the purely network sense of this term, but in the equality of nodes in terms of decision making. During the operation of such an NCS, the ranks of the nodes change depending on the number of nodes subordinate to them. To determine the current rank and AR of each NCS node, it is proposed to apply the paradigm of SA [9].

98

A. Fridman and B. A. Kulik

3 SA for Groups of Robots General principles of achieving SA within a group of interacting robots were introduced in [11], here we just need to mention that our previously developed situational modelling system (SMS) (for example, [10]) is used hereafter as the closest prototype of an NCS. The SMS is based on a situational conceptual model (SCM) proposed for studying industrial-natural complexes (INCs). This model incorporates elements of a subject domain and relations among them in order to formalise object structures and cause-effect links, which are essential within frames of the current study of an INC. The model admits three types of elements, namely: objects (organisational parts of the INC), resources (data the objects exchange) and processes, which emulate transformations of resources. While an SCM is being formed, some possible alternative structures of the INC under investigation are entered into the model by means of OR-relations in objects decomposition and resources generation. Eventually, objects compose a hierarchy that reflects organizational connections among them. During the model construction, objects are associated with spatial parameters on an electronic map (GIS) so as to provide a univocal correlation between the conceptual and geographical representation of the INC. NCSs can be assigned to INCs as well, since almost all of them have immanent spatial structure and are subject to changes in characteristics over time. Their main differences are that INCs have the hierarchical structure and constant ARs for every decision-maker (it can change only when the SCM is modified). Conversely, the task of forming ARs in NCS is specific since ARs can change any time, so it is required to quickly determine them for each decision-maker. We assume that the structure (conceptual model) of any NCS can be described by a graph with alternative subgraphs, and at each moment of time, only one definite (non-redundant) structure is realized that corresponds to a sufficient situation in the SMS. Under this assumption, quite extensive analogies are possible between the SCM and the NCS model, the structure of which is similar to the computer network [10] obtained by simulating the dynamics of an INC. Formulas for numerical assessing the degree of SA achievement and its three stages (see Sect. 1) for INCs are described, for example, in [16]. Based on the foregoing, these dependencies, by analogy with cognitive classification in conceptual spaces [10], are proposed to be used in NCSs for operative determination of the AR for each decision-making node (DMN) of a network, i.e. a robot in our case, as follows: • the current value of the SAD is calculated for all nodes of the network; among them, the nodes with local SAD maxima are distinguished; these nodes become DMNs; • the closest nodes with a lower SAD are included in the AR of each DMN. In tasks where information flows are important, distances between nodes are determined by configuration of the computer network; geographic distances are used for material flows. The nodes that fall into ARs of several DMNs are allocated according to additional criteria or randomly.

Task Scheduling Within Robots’ Collectives of Arbitrary Structures

99

After formation of ARs, each DMN solves problems of coordination of interactions (distribution of tasks) among the nodes included in its AR. In the general case, coordination methods depend on the subject area of the NCS; apparently, in most systems, gradient coordination can be applied [10]. DMNs are reassigned if a significant change is detected in SADs for any nodes of the NCS.

4 Conclusion It is shown that usage of numerical estimates of the situation awareness makes it possible to quickly construct areas of responsibility for the robots considered as decision-making nodes in a net-centric system and thus improve coordination of interactions among such robots in cases when coordinating robots lose their SA. Acknowledgements The authors would like to thank the Russian Foundation for Basic Researches (grants 18-29-03022, 18-07-00132, 18-01-00076, and 19-08-0079) for partial funding of this research.

References 1. Abrosimov, V.K.: Forming of situational awareness in large net-centric systems. In: Management of development of large-scale systems. In Materials of the Tenth International Conference. M.: IPU RAS, pp. 300–303 (2017) (in Russian) 2. Albekov, N.N.: Emergence as an object of modern science. Mod. Probl. Sci. Educ. 2, 421 (2015). (in Russian) 3. Arquilla, J., Ronfeldt, D.F.: Networks and Netwars: The Future of Terror, Crime, and Militancy. Rand Corporation, Santa-Monica (2001) 4. Arseniev, D.A., Tolmachev, S.G., Shkodyrev, V.P.: Adaptive-stochastic models of selforganization and control in net-centric systems. In: Control in marine and aerospace systems (UMAS-2016) Materials of the 9th Multiconference on control issues. SPb: Central Research Institute “Electrical Appliance”, pp. 90–95 (2016) (in Russian) 5. Asharina, I.V., Lobanov, A.V., Grishin, VYu., Sirenko, V.G.: Problems of creating survivable net-centric control systems for spacecraft constellations. Innov. Inf. Commun. Technol. 1, 325–332 (2017). (in Russian) 6. Banbury, S., Tremblay, S.: A Cognitive Approach to Situation Awareness: Theory and Application, pp. 317–341. Ashgate Publishing, Aldershot, UK (2004) 7. Cebrowski, A.K., Garstka, J.J.: Network-centric warfare: its origin and future. In: U.S. Naval Institute Proceedings. Annapolis (Maryland) (1998) 8. Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Hum. Factors 37(1), 32–64 (1995) 9. Endsley, M.R.: Final reflections: situation awareness models and measures. J. Cogn. Eng. Decis. Mak. 9(1), 101–111 (2015) 10. Fridman, A.Y.: Situational management of the structure of industrial-natural systems. Methods and models. LAP, Saarbrucken, Germany (2015) (In Russian)

100

A. Fridman and B. A. Kulik

11. Fridman, A.Y., Kulik, B.: Assessment of situational awareness in groups of interacting robots. In: Gorodetskiy, A.E., Tarasova, I.L. (eds.) Smart Electromechanical Systems. Series: Studies in Systems, Decision and Control, vol. 261, pp. 25–33. Situational Control. Springer International Publishing Switzerland (2020) 12. Ivanyuk, V.A., Abdikeev, N.M., Pashchenko, F.F., Grineva, N.V.: Net-centric management methods. Strateg. Manage. 1, 26–34 (2017). (in Russian) 13. Kapustyan, S.G., Dyachenko, A.A.: Distributed information management system for automated multi-robot technical transport and warehouse complex. Mechatron. Autom. Manage. 7, 34–39 (2012). (in Russian) 14. Karpova, A.V.: Net-centric concept—a new reality in the modern economy. Econ. Bus. Theory Pract. 10, 146–149 (2016). (in Russian) 15. Kuj, S.A., Tsvetkov, V.A.: Net-centric management and cyber-physical systems. Educ. Resour. Technol. 2(19), 86–92 (2017). (in Russian) 16. Kulik, B.A., Fridman, A.Y.: Quantitative assessment of situational awareness in the system of conceptual modeling. In: System analysis in design and management: Collection of scientific papers XXIII Intern. Scientific and practical conf. Part 3, pp. 449–460. Publishing House Polytechnic Press, St. Petersburg (2019) (in Russian) 17. Lundberg, J.: Situation awareness systems, states and processes: a holistic framework. In Theoretical Issues in Ergonomics Science (2015) 18. Masloboev, A.V.: Cognitive technology of dynamic formation and configuration of problemoriented multi-agent virtual spaces. Vestnik MSTU 16(4), 748–760 (2013). (in Russian) 19. Masloboev, A.V.: Decision-support technology in net-centric management of regional security. Inf. Technol. Bull. 2, 117–126 (2019). (in Russian) 20. Matrenin, P.: Description and implementation of swarm intelligence algorithms using a systematic approach. Softw. Eng. 27–34 (2015) (in Russian) 21. Net-centric approach. (2020). https://www.kg.ru/technology/network-centric/. (addressed 2020) (in Russian) 22. Novikov, D.A.: Theory of Management of Organizational Systems. M.: MPSI (2005) (in Russian) 23. Novikov, D.A., Smirnov, I.M., Shokhina, T.E.: Mechanisms of Control of Dynamically Active Systems. M.: IPU RAS (2002) (in Russian) 24. Ryazanov, V.A.: Net-centric approach to the management of fire protection forces. Fires Emerg. 3, 61–69 (2010). (in Russian) 25. Savkin, V.I., Proka, N.I., Krygin, A.A.: Network centrism in state support of small forms of managing the agrarian sector of the Russian economy. Econ. Anal. Theory Pract. 47(398), 45–52 (2014). (in Russian) 26. Svoevolin, VYu.: Net-centric principle of management in socio-economic systems. Terra Econ. 11(4–2), 12–15 (2013). (in Russian) 27. Toma, Y.G.: Differentiation of the application levels for net-centric concept on examples of existing information systems and decentralized financial resources. Internet J. Naukovedenie 9(6). https://naukovedenie.ru/PDF/15EVN617.pdf (2017) (in Russian) 28. Trakhtengerts, E.A., Pashchenko, F.F.: Some features of net-centric control in large-scale networks. Prob. Mech. Eng. Autom. 4, 12–21 (2015). (in Russian) 29. Trakhtengerts, E.A., Pashchenko, F.F.: Synergetic effects in net-centric systems. Sens. Syst. 11(219), 3–12 (2017). (in Russian) 30. Watson, J.: Strategy: An Introduction to Game Theory, 3rd edn. W.W. Norton and Co., New York (2013)

Meta-Heuristic Algorithm for Decentralized Control of a Robots Group to Search for the Maximum of an Unknown Scalar Physical Field Anatoliy P. Karpenko and Inna A. Kuzmina

Abstract Problem formulation: An important class of group robotics tasks is the problem of spotting the extreme of an unknown scalar physical field using a robots group. For instance, the task of robotic detection of radioactive, chesmical, biological zones or other contamination of land, air or water can be set in the same mode. The methodological basis of the work is the bionic approach, the essence of which is to build a control system for a robots group through the usage algorithms inspired by nature and human society. Purpose: Development of a decentralized robots group control system, which feature small autonomous shooting satellite robots. Methods: An original meta-heuristic robots group decentralized control (RGDC) algorithm for solving the problem, based on the Intelligent Ice Fishing algorithm, which was proposed by the authors earlier, is offered. The main procedures of the algorithm are: the local optimization in the region of the current position of the robot; near relocation of the robot in the case where there is a stagnation of the local search process or when the advances of one of its closest neighbors “significantly” exceed the advances of the robot; far relocation, if there is a stagnation of the search process and the advances of its closest neighbors “do not significantly” exceed the advances of the robot. Results: A software that simulates a group of robots functioning has been developed. The software implements the proposed method of controlling the considering group of robots. A significant number of computational experiments were performed to estimate the effectiveness of the method on a number of test three-dimensional multiextreme problems. The results of the study show a satisfactory, from a practical point of view, probability of the global extreme localization. Discussion: The results of the research are planned to be used in the development of the control system of the group robotic system.

A. P. Karpenko · I. A. Kuzmina (B) Bauman Moscow State Technical University, Moscow, Russia e-mail: [email protected] A. P. Karpenko e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_9

101

102

A. P. Karpenko and I. A. Kuzmina

Keywords Group robotics · SEMS · Decentralized control of a robots group · Nature-inspired global optimization algorithm · Localization of infected areas of the earth’s surface

1 Introduction In the form of an unknown scalar physical field extremes localization problem by robots group, can be used to determine zones of radioactive, chemical, biological or other contamination of the area, damage to malignant algae, turbulence, sea temperature and salinity, and other similar [1–3]. The methodological basis of the work is the bionic approach—an approach based on technical copying of effective solutions founded by living species in the process of their natural selection [4]. For this class of tasks, the implementation of this approach is reduced to the group of robots control system construction with using swarm algorithms—particle swarm optimization, bee colony, ant colony and so on [5–7]. As a basic bionic algorithm, we use a modification of the Intelligent Ice Fishing Algorithm (IIFA), which takes into account the features of the considered robots hardware implementation. The IIFA belongs to the class of “trace” algorithms, when the coordinates and results of a certain number of previous tests are taken into account during the population agents evolution. The algorithm is intelligent in the sense that each of its agents can build an approximation of the fitness function (a surrogate model) in the considered part of the search area based on information about its own trace and the traces of a certain number of neighboring agents; it also can find the coordinates of local and global maxima of this approximation. The IIFA is distinctly meta-heuristic, that allows to form on its basis a large number of specific heuristic population-based global optimization algorithms. The article is organized as follows. In Sect. 1, we present the problem formal statement, as well as give the main definitions and notations. In Sect. 2, we give the problem statement and basic definitions. Section 3 contain the general RGDC algorithm scheme and its basic procedures. Section 4 is devoted to the RGDC algorithm software implementation and the study of its effectiveness.

2 Problem Statement and Basic Definitions    We use the following notations. = X |X − ≤ X ≤ X + ∈ n is the study ndimensional region (parallelepiped) of the search space, where X − , X + —are the boundaries of the region, and the inequalities are understood component by component; t is the number of the current search iteration; X i (t) = X i is the current position of the robot Mi , i ∈ [1 : m]; X i (t + 1) is the position of this robot at the next iteration; X ∗ is the desired vector of optimal coordinates X which delivers the maximum

Meta-Heuristic Algorithm for Decentralized Control of a Robots …

103

value f ∗ of the fitness function f (X ), which formalized the intensity of the scalar field [8]. We consider, that n = 2 or n = 3. The deterministic maximization problem is considered max 

X∈

⊂n

  f (X ) = f X ∗ = f ∗

(1)

In the initial state the group in floating or flying robots Mi , i ∈ [1:m] is homogeneous. Mi is represent a mother robot (M-robot) that carries a fixed number μ of small ejected (non-returnable) satellite robots Si, j (S-robots); j ∈ [1:μ]. M-robots can measure values of the scalar field f (X ) at its stop points, the total number of which does not exceed tˆ. Each of the satellite robots Si, j can perform a single measure at a distance from the robot Mi , not exceeding δ X . The follow strong assumptions are used: • the M- and S-robots test results are accurate, it means, that measured function f (X ) values in the test points are not distorted by noise and/or measurement errors; • M-robot can get the accurate information about their extended tracks from each of the other M-robots at any time, which also include information about the made by their satellite robots tests. The ending search criterion is achievement by each of the M-robots a given number of iterations tˆ. In this case, all robots transmit information about their traces to the central control system. Based on this information, a decision about the the desired global maximum position is made.

3 General Scheme and Main RGDC Algorithm Procedures The RGDC algorithm general scheme is shown at Fig. 1. The main algorithm procedures are: initialization (initial placement of M-robots); local search series; short-range relocation; long-range relocation; end of the search. Initialization. In the algorithm initializing process, the robots Mi , i ∈ [1:m] are evenly placement in the search area P. Local search series. A local search series for Mi , i ∈ [1:m] robot is a set of its local optimization steps that end in short-range or long-range relocations. Let X i (0) ∈ di (0) ⊂ be the coordinates of the starting robot Mi , i ∈ [1:m] point search in this local search series (obtained as a result of initialization, shortrange or long-range relocations). Here di (0)—is explored by the robot Mi search area, radius r . The area di (0) may include traces of this or other robots. Therefore, in the local search series, a stochastic algorithm based on the search with prohibitions used [9, 10].

104

A. P. Karpenko and I. A. Kuzmina

Fig. 1 The RGDC algorithm general scheme

The scheme of the local search series by t-iteration for the robot Mi , i ∈ [1:m] is as follows (Fig. 2).

Fig. 2 To the local search scheme for the robot Mi : n = 2; •—robots traces; areas

—ε-prohibited

Meta-Heuristic Algorithm for Decentralized Control of a Robots …

(1)

(2)

105

Determine the set of the Mi robot’s closest neighbors {Mi (t)} N B , obtain these  N B and the corresponding values of the fitness robots current traces X il (t)  N B function f il (t) . The radius of the close neighbors zone isR N . The current attractiveness ai (t) of the area di (t) is estimated ai (t) = λe ei (t) + λ p pi (t),

(2)

  N B   where ei (t) = |{X i (t)} ∈ di | +  X il (t) ∈ di  is area development, equal  N B to the total number of those parts of the tracks {X i (t)}, X il (t) , that belong f imax (t) to this area; pi (t) = f min (t) is the area prospects; λe , λ p is given weighting i

(3) (4)

(5)

(6)

factors; f imax (t), f imin (t) is the maximum and minimum values in the set  N B { f i (t)}, f il (t) respectively.  N B Based on the traces {X i (t)}, X il (t) , a set of ε-forbidden subareas  ε dik (t) ∈ di (t) (a set of bans) are formed.  ε Using the Monte Carlo method, satellite robots Si j at the region dik (t) ∈   di (t) points X il , j ∈ [1:μi (t)] are placed and then the value of the objective function at these points are calculated. The number μi (t) of S-robots is assumed to be proportional to the area di (t) attractiveness ai (t) (this number may be zero) and not exceeding the maximum possible value μmax .  N B   Based on the coordinates of those points of the sets {X i (t)}, X il (t) , X il , that belong to the area di (t), as well as the corresponding values of the fitness function, a surrogate (local) model f˜iL (X ) of this function in the area di (t) is build.  The point X˜ i∗ of the function f˜iL (X ) maximum is found. If X˜ i∗ ∈ , the robot Mi moved to a point X i (t + 1). Otherwise,  the robot moved to the point that is the projection of the point X˜ i∗ onto the area boundary [11].

Short-range relocation is carried out by M-robot if at least one of the following conditions is performed: (a) (b)

the optimization process stagnation; the one of the nearest neighbors success “significantly” exceeds own robot’s success.

The scheme of the robot’s short-range relocation procedure looks like this (Fig. 3).  N B  (1) A region Di ∈ of radius R N covering the tracks {X i (t)}, X il (t) , whose center is located at the point X i (t), is constructed. (2) Based on the robot’s traces and the corresponding values of the objective function, we construct a near surrogate model f˜iN (X ) of this function. We find   approximate positions X i∗k and values of the function f˜iN (X ) maxima (local    and global) f˜ X i∗ = f˜i∗ in the area Di . k

k

106

A. P. Karpenko and I. A. Kuzmina

Fig. 3 To the short-range relocation procedure scheme for the robot Mi : n = 2; •—robots traces; —ε-prohibited areas

(3) (4) (5)

  We associate with the each points X i∗k ∈ X i∗k the area dik radius of r with the center at this point.    Similarly to (2), based on the sets X i∗k , f˜i∗k we calculate the attractiveness aik of each dik area.     Using attractions aik , from the areas dik select di area (using the roulette method [12]).

Long-range relocation is carried out by M-robot if at least one of the following conditions is performed: (a) (b)

the optimization process stagnation; the one of the nearest neighbors success “not significantly” exceeds own robot’s success. The scheme of the Mi robot’s short-range relocation procedure looks like this.

(1)

(2)

Determine the current set of this robot far neighbors {Mi (t)} F B and get current  F B positions X il (t) from all these neighbors. The far neighborhood radius is RF. Based on this information, we find a subarea DiF of the maximum square that  F B does not contain the set points X il (t) .

Meta-Heuristic Algorithm for Decentralized Control of a Robots …

(3)

107

The center of the subarea DiF use as the center of the new area di (0).

The end of the search. The completion condition is that each robot reaches the specified number of iterations tˆ.

4 Software Implementation and Computational Experiment The RGDC algorithm efficiency is evaluated with the indicators ξ1 , ξ2 , . . . , ξ10 , which make sense of the estimates for localization probabilities of one (global) maximum, any two local maximums, …, all ten local maximums, consequently [13, 14]. As a criterion for localization of l maximum, we use condition. X˜ l − X l < δ f , l ∈ [1:10], where X˜ l , X l —is found by the algorithm and the exact global or local solution, consequently; δ f —is the required localization accuracy. The peculiarity of the Shekel function is that it allows setting the number and the intensity of maximum [15]. The range of valid values of function is set equal to D = { X | − 10 ≤ xi ≤ 10}. For two-dimensional case (|X | = 2) landscape of the function Shekel is illustrated in Fig. 4, where it is accepted: c = (1, 620, 782, 21, 10, 841, 90, 550, 410.8);

Fig. 4 The landscape of the Shekel function for given parameters, n = 2

108

A. P. Karpenko and I. A. Kuzmina

ai = {(5, 89; 4, 53)(−5, 71; −9, 98)(−8, 23; 4, 57)(3, 9; −7, 05) (−3, 76; −2, 34)(0, 87; 0, 35)(−9, 52; −3, 87)(4, 23; −1, 87) (−5, 35; 6, 85)(3, 49; 4, 75)} The values of the Shekel function maximums and their coordinates are shown in Table 1. Computational experiments were performed for the following values of free parameters: • • • • • • • • •

M-robots number m = 30; localization accuracy δ f = 0, 01; initial number of S-robots μ = 30; maximum number of simultaneously fired S-robots μmax = 2; weight factors λe =, λ p = 1; near and far visibility area radiuses R N = 2, R F = 4; developed area radius r = 1; local search stagnation determination parameter s f = 0, 01; “neighboring” robot success determination parameter d f = 0, 5.

Table 2 and Fig. 5 shows the experimental RGDC algorithm performance indicators values, n = 2; ξg —global maximum localization probability.

5 Conclusion As we noted above, the basic RGDC algorithm is a powerful meta-heuristic. Therefore, we can offer a large number of ways to improve its effectiveness. We can change the ratio of intensification and diversification algorithm properties during its execution by using dynamic (software and/or adaptive) parameters [15, 16]. For example, we can intensify (diversify) the search by programmatically decreasing (increasing) the parameter values r, R as the iteration number increases. The vector of the free RGDC algorithm parameters is great. Therefore, it is possible to formulate the “optimal” values of all or some of these parameters search problem, that is, the problem of (parametric) algorithm metaoptimization. The problem can be solved by parameter setting methods, as well as by adaptive control algorithms [17]. The RGDC algorithm uses surrogate models f˜iL (X ), f˜iN (X ). Generally speaking, as these models can be used the kriging functions, Artificial Neural Networks, Support Vector Machine, Multivariate Nonparametric Regression functions, polynomial Regression functions, radial Basis Functions, Gaussian processes functions, etc. [18–20]. Due to the relatively low surrogate models computational complexity, various population algorithms can be used to find their local and global maximums [21].

0,57

5,89; 4,53

0,86

1,41

3

−8,23; 4,57

2

−5,71; −9,98

1

Table 1 Shekel function maximums 4 0,55

3,9; −7,05

5 1,00

−3,76; −2,34

6 1,39

0,87; 0,35

7 0,60

−9,52; −3,87

8 1,99

4,23; −1,87

9 2,50

−5,35; 6,85

10 1,48

3,49; 4,75

Meta-Heuristic Algorithm for Decentralized Control of a Robots … 109

110

A. P. Karpenko and I. A. Kuzmina

Table 2 The computational experiment results ξ1

ξ2

ξ3

ξ4

ξ5

ξ6

ξ7

ξ8

ξ9

ξ10

ξg

1,00

0,78

0,39

0,23

0,19

0,14

0,05

0,02

0,00

0,00

0,14

Fig. 5 The estimates for localization probabilities

References 1. Shan, S., Wang, G.G.: Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions. Struct Multi. Optim. 41(2), 219–241 (2010) 2. Van Laarhoven, P.J.M., Aarts, E.H.L.: Simulated annealing. In Simulated Annealing: Theory and Applications, pp. 7–15. Springer, Dordrecht (1987) 3. Michalewicz, Z., Schoenauer, M.: Evolutionary algorithms for constrained parameter optimization problems. Evol. Comput. 4(1), 1–32 (1996) 4. Wright, A.H.: Genetic algorithms for real parameter optimization. In Foundations of Genetic Algorithms, vol. 1. Elsevier, pp. 205–218 5. Kennedy, J: Particle swarm optimization. In Encyclopedia of Machine Learning, pp. 760–766 (2010) 6. Karaboga, D., Basturk, B.: Artificial bee colony (ABC) optimization algorithm for solving constrained optimization problems. In International Fuzzy Systems Association World Congress, pp. 789–798. Springer, Berlin (2007) 7. Dorigo, M., Blum, C.: Ant colony optimization theory: a survey. Theor. Comput. Sci. 344(2–3), 243–278 (2005) 8. Karpenko, A.P., Svianadze, Z.O.: Meta-optimization based on self-organizing map and genetic algorithm. Opt. Mem. Neural Netw. 20(4), 279–283 (2011) 9. Forrester, A.I.J., Keane, A.J.: Recent advances in surrogate-based optimization. Prog. Aerosp. Sci. 45(1–3), 50–79 (2009) 10. Kerschke, P., Trautmann, H.: Automated algorithm selection on continuous black-box problems by combining exploratory landscape analysis and machine learning. Evol. Comput. 27(1), 99–127 (2019) 11. José Antonio Martín, H., de Lope, J., Maravall, D.: Adaptation, anticipation and rationality in natural and artificial systems: computational paradigms mimicking nature. Nat. Comput. 8(4), 757–775

Meta-Heuristic Algorithm for Decentralized Control of a Robots …

111

12. Branke, J., Elomari, J.A.: Meta-optimization for parameter tuning with a flexible computing budget. In Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 1245–1252. ACM (2012) 13. Nobile, M.S., et al.: Fuzzy self-tuning PSO: a settings-free algorithm for global optimization. Swarm Evolu. Comput. 39, 70–85 (2018) 14. Neumüller, C., et al.: Parameter meta-optimization of metaheuristic optimization algorithms. In International Conference on Computer Aided Systems Theory. Springer, Berlin, pp. 367–374 (2011) 15. Mersmann, O., et al.: Exploratory landscape analysis. In Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, pp. 829–836. ACM (2011) 16. Beiranvand, V., Hare, W., Lucet, Y.: Best practices for comparing optimization algorithms. Optim. Eng. 18(4), 815–848 (2017) 17. Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002) 18. Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evol. Comput. 9(2), 159–195 (2001) 19. Eiben, Á.E., Hinterding, R., Michalewicz, Z.: Parameter control in evolutionary algorithms. IEEE Trans. Evol. Comput. 3(2), 124–141 (1999) 20. Gong, Y.-J., Li, J.-J., Zhou, Y., Li, Y., Chung, H.S.-H., Shi, Y.-H., Zhang, J.: Genetic learning particle swarm optimization. IEEE Trans. Cybern. 46(10), 2277–2290 (2016) 21. Kavetha, J.: Coevolution evolutionary algorithm: a survey. Int. J. Adv. Res. Comput. Sci. 4(4), 324–328 (2013)

Position Control of UGV Group for COVID (Virus SARS-CoV-2COVID) Localization and Primary Treatment Within Indoor Environment I. L. Ermolov, M. M. Knyazkov, S. A. Sobolnikov, A. N. Sukhanov, and V. M. Usov

Abstract Problem statement: control task for group of robots performing localization of contamination area and its primary disinfection is an important scientific problem. In order to increase efficiency of robots—members of the team one should produce algorithms for decision making, robots distribution within contamination area, minimizing number of robots effectively treating the area. Purpose: to produce algorithms for collaboration of robots in group performing detection and disinfection of indoor spaces from COVID-19 virus (SARS-CoV-2COVID). Methods: algorithms for robots’ motion in case of insufficient information about environment. This includes stationary and moving obstacles, which were not considered during preplanning phase, decision making by separate robots of the group in case of emerging situations. Creating of mathematical model of robots’ motion in group, which implies optimal distance among robots. Selection of most effective express-analysis methods for COVID-19 detection and ways of its contamination area localization. Analysis of space disinfection by UV-radiation or spraying various suspended matters from on-board of robots. Results: a new mathematical model was created and studied basing on suggested algorithm of mobile robots’ behavior and on suggested ways to detect and dispose virus within closed areas. Basing on this model new software I. L. Ermolov (B) · M. M. Knyazkov · A. N. Sukhanov Ishlinsky Institute for Problems in Mechanics of the Russian Academy of Sciences, Prospect Vernadscogo 101-1, 119526 Moscow, Russia e-mail: [email protected] M. M. Knyazkov e-mail: [email protected] A. N. Sukhanov e-mail: [email protected] S. A. Sobolnikov Federal State Budgetary Educational Institution of Higher Education “Moscow State Technological University” STANKIN, Vadkovsky per., 3a, 127055 Moscow, Russia e-mail: [email protected] V. M. Usov Institute for Biomedical Problems RAS, 76a, Khoroshevskoe Shosse, 123007 Moscow, Russia e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_10

113

114

I. L. Ermolov et al.

was developed. This software allows to control group of robots effectively, implementing trajectory planning for robots within indoor spaces, to change dynamic features of robots, formulate obstacles and forbidden areas. Discussions: results of this study have proved effectiveness of using mobile robots (agents) for localization of contamination area and its primary disinfection from COVID-19 virus and other pathogens. Keywords Group of robots · Motion planning · Pandemic · COVID-19 · Software · Control algorithms

1 Introduction Usage of robots may be efficient as in well-known areas, so in new challenging circumstances. One of the latter is a pandemic situation. Pandemic took place a few times throughout mankind’s history. However it’s our challenge now to use robots for the sake of fighting COVID pandemic. We expect robots to be especially efficient as for localization of contaminated areas so for primary disinfection of contaminated spaces. A main robotic challenge in such cases is controlling robots working in groups [1, 2, 3]. Topic of controlling robots in open spaces were thoroughly discussed in [4, 5, 6]. However there are a lot of cases when robots have to act within indoor spaces. These indoor usages have their own specifics [7]. Usually they demand more sophisticated algorithms of robots’ behavior, because their motion is more restricted within indoor environment. Once we explore practical aspects of using robots indoor we expect them to interact with humans, other mobile and stationary objects. For localization of indoor areas contaminated by pathogens one may use a network of stationary sensors. However it’s cheaper and more efficient to use a group of UGVs (equipped with air sensors and disinfection equipment) moving within indoor space. This is because humans’ trajectories (hence distribution within space) is rather chaotic and it’s easier to track them by robots than to distribute sensors all over. But this creates some difficulties in functioning of such systems, including necessity to create algorithms which would deal with lack of information regarding map of environment, which were not considered during preliminary trajectory planning for robot. For the benefit of such algorithms a new appropriate mathematical model of robots’ motion in group should be created. The latter should secure optimal distance between robots during motion as well. Therefore in order to solve a task for COVID-19 (SARS-CoV-2COVID) virus detection and follow-up disinfection in indoor spaces by means of mobile robots following tasks should be solved: – Design of mathematical model for robots‘ group motion; – Design of robot’s behavior algorithms in case of mobile and stationary obstacles. – Selection of methods for pathogens detection and follow-up disinfection of air.

Position Control of UGV Group for COVID (Virus SARS-CoV-2COVID) …

115

Fig. 1 Robot R80 by XAG, China

2 Problem Statement For indoor spaces disinfection the following 2 UGVs platforms are used: on base of UV-radiation disinfection and on liquid antiseptics dispersion. Before SARSCoV-2COVID pandemic these methods were used only within hospitals’ premises. However these days it’s emerging to use them in heavily concourse D areas, e.g. in trade centers, service centers etc. One may use similar robots from agricultural applications for herbicides dispersion Fig. 1. Using these robots enable to rapidly introduce unmanned disinfection, utilizing already existing experience. However most of such devices function in teleoperating mode, hence considerable upgrade is needed for their autonomous functioning. Robots using UV-radiation Fig. 2are rather effective for pathogens elimination. However the radiation may harm human, so the space should be processes without humans’ presence or by low dosage of radiation. In order to quickly process large indoor spaces it is suggested to use a group of robots distributed with this space [8], see Fig. 3. In order to implement complete coverage of the processed area a special mathematical model for robots’ motion is needed. It should also consider and ensure specific distance between robots. Dynamic and static obstacles should be avoided by usage of specific robots’ behavior algorithms, which also ensure safety for humans who are present next to robots.

3 Mathematical Model for Robot Moving in Group Successful solution of navigation task for group of robots should implement each robot’s motion along with target velocity and positioning. Besides requested precision of positioning and velocity there exist restrictions for time for data processing. In that

116

I. L. Ermolov et al.

Fig. 2 Robot AMY-ANGEEI UVC Sterilizing Robot AMY-M2-W with UV-lamp on board

case control system functioning undermine processing of incoming data before new portion of data is given; this is called real-time mode [9]. Therefore it is emerging to unload communication channel, which links robots within their group, and hence to perform calculations on-board of robots. In order to achieve high speed of decision making within control system a special knowledge base should be realized. The latter should incorporate heuristic algorithms to reduce solution of some tasks in some specific situations [10]. Let’s study a model of group of nonholonomic robots, performing some common task. Suppose a global stationary Cartesian coordinate system O X Y Z is given in static opened linked restricted subset W of 3D real space W ∈ R3 . In this subset there exist a group of n + 1 robots Ai , i = 0, 1, . . . , n. Each lapse of time each robot of the group should know its state vector. The latter keeps a set of parameters which completely describe position and orientation of robot in space. This vector will look like:

Position Control of UGV Group for COVID (Virus SARS-CoV-2COVID) …

117

Fig. 3 Computer modeling of trajectory planning of mobile robots’ group

⎞ xi ⎜y ⎟ ⎜ i⎟ ⎜ ⎟ ⎜z ⎟ pi (t) = ⎜ i ⎟, ⎜ αi ⎟ ⎜ ⎟ ⎝ βi ⎠ γi ⎛

(1)

here xi , yi , z i —coordinates of robot in space, αi , βi , γi —orientation angles of robot between directions of linear velocity vectors x˙i , y˙i , z˙ i and axes O X , OY and O Z correspondingly. Besides state vector control system of robot should receive information about current velocity and acceleration. The latter could be obtained either from odometry sensors or from inertial sensors (IMU) [11, 12]. During movement of robot across various floors, its coordinate z i can be obtained from barometer. If within group a hierarchical system for tasks distribution is used, then there may be a master-robot. It will formulate common motion trajectory for a group, as well as control task implementation, regulate common velocity, reconfiguring formation of group depending on specific circumstances (jamming, obstacles, failure, signal loss, contamination detection, command from operator etc.) Fig. 4. For keeping group’s formation robots should keep specific distance between each other. This parameter depends on following factors. Firstly, communication among robots should not be interrupted, hence target distance should be di < r , gde r— communication distance depending on communication equipment on board of robots.

118

I. L. Ermolov et al.

Fig. 4 Distribution of robots in space

Secondly, orientation error should converge to zero. This way the control law for robot in group with current parameters (di (t), ϕi (t)), where ϕi (t)—current orientation vector will be determined as:  |Di (t) − di (t)| < εd , (2) |i (t) − ϕi (t)| < εϕ here Di (t)—target position vector, which is determined by current task,—target orientation of robot, εd → 0, εϕ → 0. In case of some robots of the group have received a task to move a bulky load, then formula 2 will be updated by similar restriction on velocity and acceleration:

⎧

˙ i (t) − d˙i (t) < εd˙ D ⎪

⎪ ⎨

˙ i (t) − ϕ˙i (t) < εϕ˙ , ⎪ εd˙ → 0 ⎪ ⎩ εϕ˙ → 0

(3)

This is necessary in order to implement smoothness of motion and corresponding maneuvers Fig. 5. According to this control law position and orientation of robot in space in time lapse t should coincide with target position and orientation of robot with some known delay (sampling period). In order to detect obstacles and follow-up image recognition and classification onto static and dynamic structures and also humans’ detection stereo vision cameras should be installed Fig. 6. These cameras allow to detect distance to the object [13], as well a determine objects’ motion parameters (direction, velocity and acceleration) [14]. Figure 6 presents stereo vision system functioning. Here C x and C y —focal plane   coordinates of camera 1, and C x i C y —focal plane coordinates of camera 2, f =

Position Control of UGV Group for COVID (Virus SARS-CoV-2COVID) …

119

Fig. 5 Robots moving bulky load

Fig. 6 Stereo vision scheme



f —focal distance of cameras. Tx —a distance among cameras’ centers along OX axis, points s and s’—coordinates of the of the projection of sought point S (part of obstacle) on planes x y and x  y  . In case of known focal distances of images Fx and Fy we get projection matrix P for one of cameras: ⎛

⎞ Fx 0 C x −Fx Tx P = ⎝ 0 Fy C y 0 ⎠. 0 0 1 0 Projection points coordinates in such case could be obtained as:

(4)

120

I. L. Ermolov et al.

⎞ ⎛ ⎛ ⎞ XS x ⎟ ⎜ ⎝ y ⎠ = P ⎜ Y S ⎟. ⎝ ZS ⎠ z W

(5)

Coordinates of point S can be obtained via reprojected matrix Q: ⎛

1 ⎜0 ⎜ ⎜ Q=⎜ ⎜ ⎝0 0 Then sough coordinates

 XS W

, YWS ,

0 0 −C x 1 0 −C y 0 Fx 0  x 0 − T1x C x −C Tx

ZS W



⎞ ⎟ ⎟ ⎟ ⎟. ⎟ ⎠

(6)

of point S will be found as:

⎛

⎞ ⎛ ⎞ XS x ⎜ YS ⎟ ⎜ ⎟ ⎜ ⎟ = Q ⎜ y ⎟.  ⎝ ZS ⎠ ⎝χ − x ⎠ W 1

(7)

Comparing this points’ coordinates data along various time intervals, one may detect whether object is moving. In case of object’s motion we may get motion’s direction and velocity.

4 Software for Robots The software has been developed to simulate and control a group of mobile robots (MR) detecting COVID-19. During the work of the software, the following tasks are solved: The software has been developed for modeling and control mobile robots’ group detecting COVID-19. The software provides: – – – – – –

input virus exploring area; automatic deployment of robots inside the aria for full coverage; automatic robots goal position assignment; automatic optimal path finding for each robot; simulating of robots’ motion and collision avoidance; control input generating for real mobile robots.

The software includes three forms. The main form shows the map of the environment, provides editing the map and simulation visualization. This form calculates coverage of a user-defined area by robots, the distribution of goal positions between

Position Control of UGV Group for COVID (Virus SARS-CoV-2COVID) …

121

robots, trajectories generation and simulation. The second form provides editing environment and robots parameters. The third form contains robots control system and allows the developed algorithms to be applied on real equipment. The software in the course of its work creates and uses files in the.txt format, which store an array of values containing information about a digital map of the area and various parameters of robots and the environment. The main user interface Fig. 7 consists of two parts. The first part is the map. The map takes up most of the application window and has a field that contains 60 × 60 cells. The second part of the interface is the editing panel located on the right side. The editing panel contains a set of tools needed for control and modeling. At the initial stage of working with the software, a map of the working area is created. Various elements are applied to the map using the Map Elements menu. The map may include impassable obstacles. To create them, user should select the “Obstacle” item in the menu and draw the elements on the map using the mouse. These elements are marked red on the map. It is also possible to create a relief (menu item “Relief”). The height values are set either with the mouse—the longer you click on the area, the higher the height value, or manually in the “Height input” field. The maps can be saved and loaded.

Fig. 7 Main user interface

122

I. L. Ermolov et al.

In the menu “Environment parameters” user enters robots and environment parameters, which affect the detection process. The border of the survey area is outlined with the mouse after selecting the “Coverage area” menu item. Then the software automatically selects the area of the detection zone. Each cell of the map that falls into the survey area is highlighted in black. After that the software distributes robots over the zone. The cells corresponding to the locations of the robots are marked on the map in green to ensure full coverage of the selected area. In order to start the simulation, user should press the “Start” button. After finding the optimal paths the simulation starts. At each calculating cycle positions of the robots are determined in accordance with the selected mathematical model. After all robots have taken their places, the area is considered as surveyed. The “Save Map” and “Load Map” buttons located in the lower right corner of the window allow user to save and load custom room maps. Maps are saved in.txt format. The presented software is built as modular. This allows user to debug and test many modules and, first of all, the developed planning algorithms, in a virtual environment without real robot participation. In this case, the transition to a real robot is carried out only by adding modules that are responsible for interacting with specific executive devices. The rest of the modules remain unchanged. The module for robots’ control is called from the main form by pressing the “Transmit” button.

5 Adaptation of Existing Service Robots’ Prototypes Towards COVID-Situation Activities Service robots’ prototypes can be classified according to their assignments as follows: – overall environment monitoring; – environment monitoring and testing for contamination; – prerequisite arrangement and cleaning for preparation of indoor space for followup disinfection; – humans warning about following-up disinfection, its duration, requested security measures including safe zones in case of emergency; – disinfection of spaces; – posterior arrangement of area including airing, labeling etc.; – activities scheduling and problems monitoring; – noninvasive monitoring (next-to-bed) and live monitoring by means of smart gadgets etc. – noninvasive monitoring of humans along the trajectory ad libitum (as a result of dialog) or hidden as a search for potential infection carrier (with fever, speech difficulties, behavior deviation, exalted air contamination).

Position Control of UGV Group for COVID (Virus SARS-CoV-2COVID) …

123

For some of these functions one may find existing prototypes and on-board equipment. Those already exist in trial versions or were announced for future potential usage. (1)

Prototype for indoor spaces disinfection.

Panasonic (Japan) has proposed to use their autonomous robots HOSPI for fighting COVID-19 [15]. Original model of HOSPI was used in healthcare institutions for collection and distribution of medicine, medical documents, laboratory samples. It was designed in order to compensate lack of paramedical personnel. Nowadays it was upgraded to HOSPI-mist version. According to [15] paramedical HOSPI-mist can disinfect contaminated areas. This robot can move autonomously along indoor spaces and disperse disinfectant agent. Through this a risk of contamination of medical personnel is reduced considerably. The robot is equipped with special set of nozzles for dispersion of disinfectant agents and also by a special set of sensors to navigate within premises of medical institutions. An operator may download to robot’s memory a plan of specific building. HOSPImist can move autonomously avoiding collisions with both stationary objects and moving humans. Robot can generate optimal motion trajectory considering schedule of presence of medical personnel and hospital patients. (2)

Prototype for patients.

“Promobot. Thermocontrol” robot (Russia) contains noncontact thermometer, which in installed on transportation module, video camera for faces detection and recognition and a set of HMI. All data can be stored on-board or in cloud. This robot can measure temperature at distance 25 cm. Thermometry’s duration 3–5 s with a precision ± 0,2 °C. [16, 17]. (3)

Prototype for checking status of patients.

Institute of Automation and Electrometry of the Siberian Branch of the Russian Academy of Sciences together with ScientificCoin Inc. (Russian) have developed a HealthMonitor—gas analyzer which can on noninvasive basis make a diagnosis of patient status. This is done by analyzing the contents of expired air. (4)

Prototype for testing sanitary state of spaces.

Shvabe Corp. (Rostec, Russia) has developed and tested a device [URL: https:// www.mk.ru/science/2020/04/03/rossiyskie-vrachi-sozdali-pribor-dlya-opredelen iya-koronavirusa-v-vozdukhe.html] which can detect toxins, viruses (including COVID-19) and bacteria in indoor spaces. These devices are called “Detector-Bio” and “Ether-Bio”. “Detector-Bio” takes air probes, then extracts microparticles and pathogenic biological agents from the probes into aqueous solution for further analysis. “Ether-Bio” has a so-called immuno-chip. This is a special diagnostic test system.

124

(5)

I. L. Ermolov et al.

Prototype on basis of smartphone.

George Washington University (USA) develops a wearable mobile device to detect COVID-19 by a smartphone [18, 19]. The principle of operation of this device is in visual reaction of special protein-based agent to the virus pathogen.

6 Conclusions Robots can play essential role in fighting COVID-19 challenge. An effective solution is seen through upgrading existing ground robots by installing relevant sensors and disinfection equipment components. Existing technologies allow to collect a puzzle for efficient usage of robots in this area. Acknowledgements The present work was supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAAA-A20120011690138-6.

References 1. Brumitt, B.L., Stentz, A.: GRAMMAPS: a generalized mission planner for multiple robots in unstructured environments. IEEE International Conference on Robotics and Automation, vol. 2, pp. 1564–1571. Leuven, Belgium (16–20 May, 1998) 2. Kalyaev, A.V.: Decentralized system for planning and managing a team of transport robots text. In: Kalyaev, A.V., Kalyaev, I.A. (eds.) Problems of Neurocybernetics: Collection of Articles. Works of Scientific. conference—Rostov-on-Don: Publishing House of the Russian State University, pp. 128–129 (1983) 3. Groß, R., Marco D.: Group transport of an object to a target that only some group members may sense. 8th International Conference, Birmingham, UK, September 18–22, 2004. Proceedings, Parallel Problem Solving from Nature—PPSN VIII volume 3242 of the series Lecture Notes in Computer Science pp. 852–861 4. Ermolov, I.L.: Emerging issues of robots to be used in groups. Smart Electromechanical Systems. Studies in Systems, Decision and Control vol. 174, pp. 3–7. Springer International Publishing (2019) 5. Gradetsky, V.G., Knyazkov, M.M.: Unmanned vehicle system in unstructured environments: challenge and current status. 21th International Symposium On Measurement And Control In Robotics ISMCR’2018 26–28 September 2018—Catalogue and Proceedings, International CBRNE Institute, Les Bons Villers, Belgium Rue de Sart-Dames-Avelines, 8A 86210 Les Bons Villers (Frasnes-lez-Gosselies) vol. 3, pp. 18–39. Belgium 6. Yamaguchi, H.: A cooperative hunting behavior by mobile robot troops. IEEE International Conference on Robotics and Automation vol. 4, pp. 3204–3209. Leuven, Belgium (16–20 May, 1998) 7. Tsiamis, A., Bechlioulis, C.P., Karras, G.C., Kyriakopoulos, K.J.: Decentralized object transportation by two nonholonomic mobile robots exploiting only implicit communication. 2015 IEEE International Conference on Robotics and Automation (ICRA) pp. 171–176 (26–30 May, 2015)

Position Control of UGV Group for COVID (Virus SARS-CoV-2COVID) …

125

8. Gradetsky, V.G., Ermolov, I.L., Knyazkov, M.M., Sobolnikov, S.A.: Construction of mobile communication networks based on ground-based autonomous mobile robots. Mechatron. Autom. Control 11, 126–129 (2011) 9. Dyachenko, A.A.: The problem of formation of the system in the UAV group. Proceedings of the Southern Federal University. Technical Science, vol. 128. N 3 (2012) In Russian 10. Das, A.K., et al.: A vision-based formation control framework. IEEE Trans. Rob. Autom. 18(5), 813–825 (2002) 11. Bunyakov, V.A., Yurevich, E.I.: Technical Vision in Robotics, p. 67. Saint Petersburg, Asterion (2008). In Russian 12. Burguera, A., Gonzalez, Y., Oliver, G.: Sonar sensor models and their application to mobile robot localization. Sensors (Basel). 9(12), 10217–10243 (2009) 13. Ahmed, M.F.: Development of a stereo vision system for outdoor mobile Robots. Abstract of Thesis Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Master of Science, p. 78 (2006) 14. Panasonic robots will cleanse hospitals from COVID-19. https://www.robogeek.ru/roboty-vmeditsine/roboty-panasonic-ochistyat-bolnitsy-ot-covid-19. Accessed 27 July 2020 15. Russian developments will help prevent COVID-19 in Latin America. https://www.iksmedia. ru/news/5682525-Rossijskie-razrabotki-pomogut-profi.html. Accessed 27 July 2020 16. Siberian scientists have found a new way of testing for coronavirus. https://coronavirus-monitor. ru/ru/novosti/sibirskie-uchenye-nashli-novyi-sposob-testirovaniya-na-koronavirus. Accessed 27 July 2020 17. Rostech tested devices for detecting coronavirus in the air. https://rostec.ru/news/rostekh-isp ytal-ustroystva-dlya-vyyavleniya-koronavirusa-v-vozdukhe/. Accessed 27 July 2020 18. Scientists in the USA have created a device for the rapid detection of COVID-19. https://www. kommersant.ru/doc/4432672. Accessed 27 July 2020 19. Miniature device could instantly diagnose COVID-19. https://medicalxpress.com/news/202007-miniature-device-instantly-covid-.html. Accessed 27 July 2020

Mathematical and Computer Modeling of the Decision Making System

Using Diagrams Influence in Group Control SEMS Andrey E. Gorodetskiy and Irina L. Tarasova

Abstract Problem statement: modern robotic systems can be built on the basis of Smart electromechanical systems (SEMS), which, due to the presence of the central nervous system (CNS), exhibit appropriate behavior. When controlling the behavior of such groups, it is necessary, first of all, to solve the problem of choosing the optimal group control of SEMS for each specific group task. One way to solve the problem is to build and analyze all combinations of control types and decision making methods. Purpose of research: the search for the optimal solution for various variants of situational control structures and the research of the decision-making process using influence diagrams. Methods: Influence diagrams are considered as Bayesian networks of trust, enhanced by the concepts of options, solutions, benefits, security and effectiveness. The tops of the decision determine the temporal seniority of the network and effectiveness. Results: the process of finding the optimal solution for situational control of a group of robots is proposed to be performed at each control step for each robot of the group with modeling the behavior of the entire group at the final choice of the optimal plan for situational control. Practical significance: the research results can be used in situational control of a group of interacting SEMS among themselves without human involvement in various tasks, for example, in driving vehicles that perform coordinated movements, a group of robotic collectors performing joint operations, etc. Keywords Smart electromechanical systems · Central nervous system of the robot · Situational control · Group control · Environment of choice · Decision making process

A. E. Gorodetskiy · I. L. Tarasova (B) Institute for Problems in Mechanical Engineering of the Russian Academy of Sciences (IPME RAS), V.O., Bolshoj pr., 61, St. Petersburg 199178, Russia e-mail: [email protected] A. E. Gorodetskiy e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_11

129

130

A. E. Gorodetskiy and I. L. Tarasova

1 Introduction Modern robotic systems can be built on the basis of intelligent electromechanical systems (SEMS) [1, 2], which possess due to the presence of the central nervous system (CNS) [3–5] reasonable behavior. When controlling the behavior of a group of interacting SEMS is necessary, first of all, to assess the ability of making the right decisions in an uncertain environment, taking into account the static and dynamic characteristics of each SEMS and select the environment and taking into account the characteristics of the decision in the CNS of individual SEMS. In addition, the task of control a group of SEMS has additional complexity because of the need to ensure coordination of interactions between them. Quality control in solving joint group task will depend on the type of control group behavior SEMS, and the method of decisionmaking. Therefore, when solving the problem of choosing the optimal SEMS group control system for each specific group task, an analysis of all combinations of control types and decision-making methods is necessary. Recent tasks relate to the tasks of situational control [6–8]. With a structural approach to the organization of situational control of the SEMS group, it is necessary for all SEMS to collect together information on environmental parameters, on the current status of individual SEMS groups, on planned actions by members of the SEMS group, etc. After collecting information, in the general case, a model O(t k ) of the choice environment is created at time t k (control step). Then the planning of situational control of the SEMS group will be [7]: – in dividing a group task into subtasks: O(t0 ) ⇒ O(t1 ) . . . O(t0 ) ⇒ O(t f ), C(t1 )

C(t f )

(1)

  where: C(tk ) = ca1 (tk ), ca2 (tk ), . . . , can (tk ) cai (tk )—control action given to the robot ai at time t k , k = 0,1,…f., – in the distribution between the SEMS of a group of solutions to subtasks so that the solution to the group problem is optimal, for example, is carried out in the shortest time, taking into account the existing restrictions, including on information interaction.

2 Statement of the Control Problem In the general case, the environment of choice O(t k ) can be characterized at some point in time t k by the following tuple:    m i j (tk ), M, Bi (tk ), G(tk ), Dm (tk ) >, O(tk ) =< A(tk ), N (t0 ), F t f , i, j

(2)

Using Diagrams Influence in Group Control SEMS

131

where: A(t k )—SEMS group, N(t 0 )—the initial position of the SEMS group in the surrounding area, F(t f )—end position of the SEMS group in the surrounding area,  i, j m i j (tk )—number of collisions, M—maximum number of collisions, Bi (t k )— characteristic of movement restrictions, G(t k )—SEMS medium interaction matrix, Dm (t k )—established traffic rules. A typical task of situational control of a group may be the task of transferring a group of robots A = {a1 , a2 , …, an }, placed at time t 0 at points N = {s1 , s2 ,…, sn } of a bounded space L 3 ⊂ E 3 , (E 3 − is a three-dimensional Euclidean space), to the target points v celevye toqki F = {f 1 , f 2 ,…, f n }} of this space at time t f in the minimum time [9]: T = t f −t0 → min,

(3)

with a minimum probability of collision of robots PA → min.

(4)

Usually, using various mathematical optimization methods, condition (4) is replaced by an inequality of the form: 

m i j (tk ) ≤ M,

(5)

i, j

where: M—maximum number of collisions, i, j—numbers of robots from numbers from 1 to n (i = j), k—time point number from time interval T, the value of mij (t k ) is determined from the logical expression. When searching for a solution to the problem, the environment of choice can be divided into layers O(t k ) with some constant or variable step hk :    O(t) = O(t0 ), . . . , O(tk ), . . . , O t f .

(6)

   Each layer O(tk ) k = 0, 1, . . . t f / h k contains the surrounding space L 3 , divided into cells eq (t k ) with constant or variable steps hx, hy , hz along the axes X, Y, Z. Here q is the cell number, q = 1,2,…,Q.   O(tk ) = e1 (tk ), e2 (tk ), . . . , e Q (tk ) .

(7)

Each cell eq (t k ) is characterized by the presence or absence of the robot ai , and obstacles Bi (t k ), for example, in the form of: prohibiting traffic signs vi (t k ), traffic signs wi (t k ), road markings dopono pazmetki γ(t k ), etc. Bi (tk ) = {vi (tk ), wi (tk ), γ (tk ), . . . .}.

(8)

132

A. E. Gorodetskiy and I. L. Tarasova

In addition, each cell is characterized by the interaction matrixes of the robot with the medium G(tk ) = {G 1 (tk ), G 2 (tk ), ..., G v (tk )}, which describe the influence of the cell medium on the dynamic state of the robot. A set of cells represents the rules of motion (if–then). The complexity of the task, which requires the use of situational control methods is that the parameters and functions that characterize a cell eq (t k ), may be deterministic, stochastic or not completely certain. When solving such problems, it is possible to use Influence Diagrams (ID).

3 Options for Using Influence Diagrams When Making a Decision In fact, Influence Diagrams are Bayesian Belief Networks (BBN), extended by the concepts of variants V, decisions D and utility (usefulness) U. The vertices of the solution, or rather, the instructions contained in them, determine the temporary seniority of the network [10]. Network fragments for the influence diagram are shown in Fig. 1. In order to search, the best in some sense decisions in the diagrams of the “vertices of utility” influence are associated with the state of the network. Each vertices of utility (usefulness) contains a usefulness function that links each configuration of her parents’ state with usefulness. When making a decision, proceed from the likelihood of network configuration. Therefore, you can calculate the expected usefulness of each alternative and choose the alternative with the highest expected usefulness. This is the principle of maximum expected usefulness. ID may contain several usefulness vertices. Moreover, the general usefulness function is the sum of all local usefulness functions. The decision-making process using such ID will be carried out in the following order: • After observing the values of the variables that are the parents of the first vertices of the decision, it is necessary to find out the maximum usefulness for the alternatives; • The expert calculates these usefulness under the assumption that all future decisions are optimal, using all available information at the time of each decision.

V

D

Fig. 1 Network fragments

D

V

D

U

Using Diagrams Influence in Group Control SEMS

133

The complexity of constructing and investigating ID is largely determined not by the number of vertices of chances, but by the complexity of their relationships, both among themselves and, especially, with the vertices of decisions and usefulness. Currently, there are a number of software implementations of the shells of expert systems based on BBN, which allow you to operate not only discrete, but also continuous random variables. An example of such software is «Hugin» [11]. However, when using BBN containing both continuous and discrete variables, there are a number of limitations: • The discrete variables cannot have continuous parents; • The continuous variables must have a normal distribution law; • The distribution of a continuous variable with discrete parents and continuous parents is a normal distribution. The logical conclusion in a BBN with continuous and discrete states is the distribution of the probabilities and parameters of the Gaussian distribution laws throughout the network, depending on the decisions (evidence) obtained. The process of logical conclusion is based on fairly complex mathematical algorithms. In the decision-making process, it is important not only to find a decision, but to find the best decision in terms of control security, i.e. minimize collisions in group control, and at the same time the best in the sense of utility (usefulness) containing a usefulness function that connects each configuration of her parents’ state with usefulness. Therefore, it is advisable to combine the safety functions S and usefulness U with various assigned weighting factors and the formation of the efficiency function E. Then, in ID, safety vertices S and usefulness U will have common heirs of efficiency E. Each vertices E contains an efficiency function, which connects each configuration of the state of its parents S and U with efficiency E (Fig. 2). As before, the decision should be based on the probability of network configuration. Therefore, you can calculate the expected efficiency of each alternative and select the alternative with the highest expected efficiency. This is the principle of maximum expected performance. The effectiveness E i of the i-th decision Di can be calculated using the formula:   E i = K iU Ui + K iS Si ,

(9)

Fig. 2 Fragments of the influence diagram

D

U

S

E

134

A. E. Gorodetskiy and I. L. Tarasova

where: U i is the utility of the i-th decision, U i = 1/T (see formula 3), K i U is the coefficient of significance of the utility, S i is the safety of the i—the decision, S i = 1/M (see formula 5) and K i S is the safety significance coefficient. ID may contain several performance vertices. Moreover, the overall efficiency function is the sum of all local efficiency functions. The decision-making process using such influence diagrams will be carried out in the following order: • After observing the values of the variables that are the parents of the first vertices of the decision, it is necessary to find out the maximum efficiency for the alternatives; • The expert calculates these efficiencies under the assumption that all future decisions will be optimal, using all available information at the time of each decision. The complexity of constructing and investigating such ID is determined by the complexity of the security and usefulness relationships between themselves and with efficiency vertices, as well as the relationship of usefulness and security vertices with decisions. Therefore, the process of finding the optimal decision for situational control of a group of robots is performed at each control step for each robot in the group. At the same time, the following variants of situational control structures are possible: – – – – – – – –

Without a coordinator or leader; With a coordinator that determines the effectiveness for each robot; With a coordinator who determines the utility for each robot; With a coordinator who determines security for each robot; With a leader that determines the performance of each robot; With a leader that determines the utility for each robot; With a leader who determines security for each robot With a coordinator who determines the utility for each robot, and a leader who determines the safety for each robot; – With a coordinator who determines the safety of each robot, and a leader who determines the utility for each robot.

4 Features of the Search for the Optimal Decision in Various Variants of Situational Control Structures When choosing a structure without a coordinator and leader at each control step for the every robot of the group, the influence diagram is analyzed (Fig. 3). Therefore, to calculate the efficiencies E ij of the decisions made Dij , each robot must be aware of the behavior of all members of the group. For this, robots must have a developed Information-Measuring System (IMS) with the ability to quickly exchange information with other robots. After identifying the intentions of other robots due to the exchange of information, situations may arise when a number of robots initially made

Using Diagrams Influence in Group Control SEMS

135

Vi(t0

Ui1

Dij

Dim

Sim

Ui

Ei

Uij Ei1

Di1

Si1

Sij Eij

Vi(t1)

Fig. 3 Fragment of ID in the structure without a coordinator and leader

the decision may not be optimal, and in some cases not safe. Then these robots are looking for a new optimal solution. In this regard, a “stupor” situation may arise when robots begin to infinitely recalculate the effectiveness of decisions and, accordingly, the process of completing the required task by the group stops. To get out of this state, the members of the group make not the best decision in the sense of usefulness, but the safest one, which ensures at least the elimination of collisions between members of the group. You can slightly improve the controllability of such a group by introducing robot priorities and using rules for acceptable movements in each member of the robot group. When choosing a structure with a coordinator that determines the efficiency for the every robot (Fig. 4) at each control step, the coordinator calculates the usefulness and safety of decision options for all robots and can additionally inform group members about decisions made. To do this, the coordinator and robots must have a developed IMS with the ability to quickly exchange information. In this case, the coordinator tightly controls the behavior of each robot. In this case, considerable time is required to calculate the effectiveness of all decisions and flexibility in control is lost during a sharp change in the situation, for example, due to breakdowns and failures, or an unexpected change in the environment of choice. When choosing a structure with a coordinator determining the utility for the every robot (Fig. 5) at each control step, the coordinator calculates the usefulness of the decision options for all robots and transfers the calculated values to the group members. The members of the group calculate the security of the decisions and determine the optimal efficiency of the decision using the values of the usefulness of the decisions received from the coordinator. At the same time, the coordinator and robots should have a developed IMS with the ability to quickly exchange information,

136

A. E. Gorodetskiy and I. L. Tarasova

K

Vi(t0

Ei1

Eim

Eij Di1

Ui1

Dim

Sij

Ui

Dij

Uij

Sim

Si1

Vi(t1)

Fig. 4 Fragment of ID in the structure with a coordinator determining the effectiveness for each robot

K

Vi(t0

Ui1

Ui

Uij Dij Di1

Si1

Eij Ei1

Sij

Dim

Sim

Eim

Vi(t1)

Fig. 5 Fragment of ID in the structure with a coordinator determining the usefulness for each robot

Using Diagrams Influence in Group Control SEMS

137

K

Vi(t0

Si1

Sim

Sij Dij Di1

Ei1 Ui1

Uij

Eij

Dim

Ui Eim

Vi(t1)

Fig. 6 Fragment of ID in a structure with a coordinator determining security for each

and the control system of a group of robots should be more flexible and secure Sudden changes in the situation, for example, due to breakdowns and failures, or an unexpected change in the environment of choice can be taken into account by members of the group when calculating the effectiveness of decisions. When choosing a structure with a coordinator determining security for the every robot (Fig. 6) at each control step, the coordinator calculates the security of decision options for all robots and transfers the calculated values to group members. Members of the group are calculated usefulness decisions and determine the optimum efficiency decisions with decisions obtained from the coordinator of security values. At the same time, the coordinator and robots must have a developed IMS with the ability to quickly exchange information. However, the group control system becomes safer compared to the previous structure, since the coordinator calculates the safety of decisions made by robots taking into account the analysis of the behavior of all members of the group. In this case, sudden changes in the situation, for example, due to breakdowns and failures, or an unexpected change in the environment of choice can be taken into account by members of the group when calculating the effectiveness of decisions. This control structure provides a safer control, but not the most efficient. When selecting the structure of the leader of determining the efficiency of the every robot (Fig. 7), the robot—each control step, the leader evaluates the usefulness and safety of possible decisions for all robots. To do this, robots must have a developed IMS with the ability to quickly exchange information. At the same time, the leader coordinates the behavior of each robot, setting priorities and rules for unacceptable behavior. In this case, considerable time is required to calculate the effectiveness of all decisions and flexibility in control is lost during a sharp change in the situation, for

138

A. E. Gorodetskiy and I. L. Tarasova

Ul1

Ui1 REim l1

Ei1

Si1

Sl1 Dl1

Di1 Vi(t0)

Vi(t1)

Vl(t0)

Vl(t1)

DRii1

Dl Eim

Sim

Slm

Elm Uim

Ulm

Fig. 7 Fragment of ID in a structure with a leader determining the effectiveness for each robot

example, due to breakdowns and failures, or an unexpected change in the environment of choice. The control structure does not require an additional coordinator. However, the reliability and overall performance of a group is highly dependent on the status and performance of the leader. In a structure with a leader determining the utility of decisions for the every robot (Fig. 8), the leader robot at each control step calculates the usefulness of decision options for all robots, and robots determine the security of decisions and, accordingly, their effectiveness. To do this, robots must have a developed IMS with the ability to quickly exchange information. At the same time, the leader coordinates the implementation of the subtasks of each robot, setting priorities, and each robot evaluates the safety of movements, taking into account the rules of unacceptable behavior. At the same time, the security of the group increases, but the effectiveness greatly depends on the status and information and computing capabilities of the leader. When choosing a structure with a leader that determines the security of decisions for the every robot (Fig. 9), the leader robot at each control step calculates the security of decision options for all robots, and the robots determine the usefulness of the decisions and, accordingly, their effectiveness. To do this, robots must have a developed IMS with the ability to quickly exchange information. The leader coordinates the safety of movements of group members, and each robot evaluates the usefulness of

Using Diagrams Influence in Group Control SEMS

139

El1

Ei1 RUiml1

Ui1

Si1

Si1 Di1

Di1 Vi(t0)

Vi(t1)

Vl(t0)

Vl(t1)

DRimi1

Dlm

Uim

Sim

Slm

Ulm Eim

Elm

Fig. 8 Fragment of ID in the structure with a leader determining the usefulness for each robot

the decisions made and chooses the most effective ones, taking into account safety. The safety of the group increases if the robot leader has a more developed IMS compared to other members of the group, the requirements for which in assessing the environment of choice are reduced. When choosing a structure with a coordinator determining the utility for the every robot and a leader determining safety for each robot (Fig. 10) at each control step, the coordinator distributes the task into subtasks for each robot and calculates the utility of the decisions of each robot, and the robot leader coordinates safe movement of group members with setting priorities. Robots calculate the effectiveness of decisions. In this case, the responsibilities for determining the effectiveness are distributed between the coordinator and the robot leader, which can increase the reliability of the functioning of the group. When choosing a structure with a coordinator determining security for the every robot and a leader determining usefulness for the same robot (Fig. 11), at each control step, the coordinator calculates the security of robot decisions, and the robot leader calculates the usefulness of robot decisions based on their behavior in a group. Robots calculate the effectiveness of decisions. In this case, the responsibilities for determining the effectiveness are distributed between the coordinator and the leader robot. This will increase security by taking into account a larger number of influencing factors by the coordinator.

140

A. E. Gorodetskiy and I. L. Tarasova

El1

Ei1

m

Ul1

RSl1im

Si1

Ui1

Dl1 Di1

Vi(t0)

Vi(t1)

Vl(t0)

DRimi1

Vl(t1)

Dlm Sim

Uim

Ulm

Slm Eim

Elm

Fig. 9 Fragment of ID in a structure with a leader determining security for each robot

5 Conclusion A comparison of the features of using the considered Influence Diagrams for making decisions in the given control structures of a group of robots shows that all of them can be used in situational group control systems. The feasibility of using specific structures depends on the tasks being solved by the group, the properties of the environment for the functioning of the group, the characteristics of the group members and available resources for the realization of the control system. The complexity of constructing and investigating influence diagrams depends on the type of group control structure and is determined by the complexity of the security and utility relationships between themselves and with efficiency vertices, as well as the relationships of usefulness and security vertices with decisions. Therefore, the process of finding the optimal solution for situational control of a group of robots is advisable to perform at each control step for each robot in the group with modeling the behavior of the entire group when finally choosing the optimal plan for situational control. In a group control structure without a coordinator and a leader, their informationmeasuring and control systems have higher requirements than in a structure with a leader and a coordinator.

Using Diagrams Influence in Group Control SEMS

141

El1 Ui1 RUiml1

Si1

Dl1

Ei1 Sl1

Di1

Vi(t0)

Vi(t1)

K

Vl(t0)

Vl(t1)

URim i1

Slm Sim

Eim

Dl

Ulm Dim

Elm

Fig. 10 Fragment of ID in the structure with a coordinator determining the usefulness and a leader determining safety for each robot

The research results can be used in situational control of a group of interacting SEMS among themselves without human participation in a variety of tasks, for example, in driving vehicles that perform coordinated movements, a group of collector robots performing joint operations, etc.

142

A. E. Gorodetskiy and I. L. Tarasova

El1 Si1 RSim l1

Ui1

Dl1

Ei1 Ul1

Di1

Vi(t0)

Vi(t1)

K

Vl(t0)

Vl(t1)

D Rii1

Ulm Sim

Eim

Dl

Slm Uim

Elm

Fig. 11 Fragment of ID in a structure with a coordinator determining security and a leader determining utility for each robot

Acknowledgements The present work was supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAA-A19119120290136-9 and is supported by grants RFBR No. 18-01-00076 and No. 19-08-00079.

References 1. Gorodetskiy, A.E.: Smart electromechanical systems modules. In: Gorodetskiy, A.E. (ed.) Smart Electromechanical Systems. Studies in Systems, Decision and Control. vol. 49, pp. 7–15. Springer International Publishing. (2016). https://doi.org/10.1007/978-3-319-27547-5_2 2. Gorodetskiy, A.E.: Smart electromechanical systems architectures. In: Gorodetskiy, A.E. (ed.) Smart Electromechanical Systems. Studies in Systems, Decision and Control. vol. 49, pp. 17– 23. Springer International Publishing. (2016). https://doi.org/10.1007/978-3-319-27547-5_3 3. Gorodetskiy, A.E., Tarasova, I.L., Kurbanov, V.G.: Challenges related to development of central nervous system of a robot on the bases of SEMS modules. In: Gorodetskiy, A.E., Kurbanov, V.G. (eds.) Smart Electromechanical Systems: The Central Nervous System. Studies in Systems, Decision and Control. vol. 95, pp. 3–16. Springer International Publishing. (2017). https://doi. org/10.1007/978-3-319-53327-8_1 4. Gorodetskiy, A.E., Tarasova, I.L., Kurbanov, V.G.: Logical-mathematical model of decision making in central nervous system SEMS. In: Gorodetskiy, A.E., Kurbanov, V.G. (eds.) Smart Electromechanical Systems: The Central Nervous System. Studies in Systems, Decision and

Using Diagrams Influence in Group Control SEMS

5.

6. 7.

8.

9.

10.

11.

143

Control. vol. 95, pp. 51–60. Springer International Publishing. (2017). https://doi.org/10.1007/ 978-3-319-53327-8_4 Gorodetskiy, A.E., Kurbanov, V.G., Tarasova, I.L.: Decision-making in central nervous system of a robot. Inf. Control Syst. (1):21–29 (2018). (In Russia). https://doi.org/10.15217/issnl6848853.2018.1.21 Pospelov, D.A.: Situacionnoe upravlenie: Teoriya i praktika [Situational Control: Theory and Practice], p. 286. Moscow, Nauka Publication (1986). (In Russia) Gorodetskiy, A.E.: The principles of situational control SEMS group. In: Gorodetskiy, A.E., Tarasova, I.L. (eds.) Smart Electromechanical Systems: Group Interaction. Studies in Systems, Decision and Control, vol. 174, pp. 3–13. Springer International Publishing. (2020) https://doi. org/10.1007/978-3-030-32710-1_1 Gorodetskiy, A.E., Tarasova, I.L.: Situational control a group of robots based on SEMS. In: Gorodetskiy, A.E., Tarasova, I.L. (eds.) Smart Electromechanical Systems: Group Interaction. Studies in Systems, Decision and Control. vol. 174, p. 337. Springer International Publishing. (2018). https://doi.org/10.1007/978-3-319-99759-9_2 Gorodetskiy, A.E., Tarasova, I.L., Kurbanov, V.G.: Situational control of the group interaction of mobile robots. In: Gorodetskiy, A.E., Tarasova, I.L. (eds.) Smart Electromechanical Systems: Group Interaction. Studies in Systems, Decision and Control. vol. 261, pp. 91–101. Springer International Publishing. (2020). https://doi.org/10.1007/978-3-030-32710-7 Romanov, V.P.: Intellektual’nye informacionnye sistemy v ekonomike: Uchebnoe posobie [Intelligent Information Systems in The Economy: Textbook]. In: Tihomirova, N.P. (ed.) p. 496. Moscow, Ekzamen Publication. (2003). (In Russia) Available at: https://hugin.sourceforge.net/

Models for Decision Making Support Systems in Robotics I. L. Ermolov, S. S. Graskin, and S. P. Khripunov

Abstract Problem statement: Most of modern robotics systems are still lowautonomous, i.e. they can not perform in extreme environments having preloaded scenarios. The reason of this is a complicity to predict possible situations and how those can further evolve. Because of this human-operator keeps being an essential part controlling robot in emerging situations. However this approach has a weakness that in sophisticated situations human-operator may not act correctly. Purpose: to study models supporting decision making by human-operator. These models should simulate intellectual activities of high-skilled operators in various emerging situations. Methods: within these studies we plan to investigate 2 main steps in creating models for supporting decision making. These are the following: 1. Integrated usage of logical–linguistic (based on natural language) and mathematical (calculation) formal means for qualitative and quantitative information processing. This would allow both to formalize logics of high-skilled operators and to consider dynamics of control processes. 2. Creation of procedures which would implement sefl-learning (adaptation in wide sense) in case of a priori uncertainty. Those should be based on logical structure of likely-based (fuzzy) conclusions, using inductive inference and analogy inference, which are inherent to mind of humans. Results: It is expected that this approach to build model for decision support in robotics systems will increase quality of decision making by human-operator. This statement is supported by a fact of intensive usage of human-operators’ experience especially in emerging situations.

I. L. Ermolov (B) Ishlinsky Institute for Problems in Mechanics of the Russian Academy of Sciences, Moscow, Russia e-mail: [email protected] S. S. Graskin Bauman Moscow State Technical University, Moscow, Russia e-mail: [email protected] S. P. Khripunov Scientific Council for Robotics and Mechatronics of the Russian Academy of Sciences, Moscow, Russia e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_12

145

146

I. L. Ermolov et al.

Keywords Decision support systems · Robotic systems · Decision modeling

1 Introduction One of main paradigms of modern robotics is robots’ assistance to humans with, where appropriate, complete removal of human from dangerous and emerging environment and especially in contingency. Under extreme conditions we understand such environment, which contains high risks, expected uncertainty, high dynamics and consequently a shortage of time for decision making. Usually this also causes a inability to implement preliminary designed solutions and algorithms. Contingency is especially typical for new conditions or events, which were not studied before, such as outer space, underwater, natural or anthropogenic disasters, civil conflicts and other authentic events of that kind. In order to use robots in such occasions robots should have a relevant control architecture basing on mission goal, robots’ functionality and specifics of environment. Special challenge is controlling robot in highly dynamic nonstationary environment with incomplete or contradictory information. In such conditions a key-role is played by adaptive decision supporting models which secure stable functioning and production of effective decisions in-time, as it is done by high-skilled operators in their professional area. One should mention that these models should either become a part of remote control device [1] or become a part of on-board component implementing decision support in autonomous mode. It looks to be efficient to use AI methodology as a basis for decision support models in robotics. This methodology allows to model human’s logic of reasoning while taking decisions in contingency [2]. As key-phases of control decisions production we outline following: events forecasting and decision generation based on forecasting [3].

2 Forecasting Model In order to efficiently control a robot in highly dynamic environment we should be able to forecast possible situation development. This is also done via constant tracking objects and predictioning their behavior within robot’s mission. E.g. this task may seen while securing flights safety in airports area or by protecting nuclear power plant by intercepting UAVs. While implementing this task we might constantly track location and motion of trespassing UAVs in order to control intercepting UAVs. Continuity of observation can be implemented basing on forecasting trespassing UAVs’ trajectory and forecasting its forthcoming locations. For that we might know

Models for Decision Making Support Systems in Robotics

147

logics of their behavior and their motion law. This way such forecasting includes in fact two tasks: modeling of logic of behavior and motion modeling. Modeling of behavior includes disclosure of objects’ mission and prediction of its possible actions basing on stored knowledge and experience. This information as a rule are presented in qualitative mode and correspond heuristic logical expressions. The later can be formalized basing on logical linguistic models [4]. One of popular methods to model vehicles is mathematical extrapolation model. Within it basing on modeling results of objects’ behavior some hypothesis of its motion is selected. Consequently object’s position it time will be determined using this hypothesis. Therefore for forecasting of activities of some object under observation it is necessary to create such generalized approach that it would allow to model (simulate) in frame of one model borth logics of behavior and motion parameters within forecasting time. E.g. suppose that at some time t0 some trespassing UAV was detected. Its coordinates and motion parameters have been calculated. Now we should determine its position at some instant of time t. Once we consider logic of behavior of trespassing UAV in relation to monitored area, one may formulate corresponding heuristic rules of its behavior and as a consequence area of its probable locations. In that case this area will be a part of whole are of possible locations of UAV if we consider its transport properties only. During optimization of aerial objects’ motion optimization we may produce its trajectory in various instants of time t 1 , t 2 , …, t k of forecasting period [t 0 , t]. As soon as actual poison of object will be renfined forecasting estimation will be done and in case of substantial deviation will be corrected. This way at first stage of forecasting model functioning a space of possible decisions will be reduced down to more probable by using heuristics and at second stage it will be optimized by traditional control theory and operations analysis methods. A logical outcome of the forecasting task (in relation to this example) we may determine a rational behavior of intercepting UAV which counteracts trespassing UAV. In general forecasting model is in fact a complex logic-mathematical model. It integrates linguistic heuristic component together with calculated component. This creates in one model a synergy of quantitative and qualitative information, increasing forecasting model adequacy and as an outcome a more effective control of a robot. In case of contingency such model creates decision support to the questions “What to be done?” and “How to act?”. Technologies of integrated modeling (models fusion) by means of fusing and overlapping of heterogeneous components within one model mutually enrich and expand their functionality. This creates new synergetic effect [5] to such models.

148

I. L. Ermolov et al.

3 Adaptive Control Model Design of control system which is able to implement flexible functioning [6] as well as capable to create correct controls-recommendations in new, unpredictable situations [7] can be done through implementation of adaptation procedures, or even self-learning (self-organization). Controlling robot in such situation may be characterized as high phenomenological task (as extraordinary, unrepeated or unique situation). This doesn’t allow to consider specifics of environment in full scope as a part of robot’s model, and hence to obtain ready solutions. These circumstances demand strong necessity to create procedures for estimation of contingency and as an outcome generation of mostly preferred control solutions. As a basis for such procedures we propose to use logical rules of plausible reasoning, including inductive inference and analogical inference [8]. This approach allows to organize adaptive control procedure using generalization of particular knowledge up to regularity. This can be done by using existing knowledge for solution of similar tasks, as it’s done by human. In general case derivation scheme will look as: Sp, Sp → R p , Rp where Sp S p → Rp Rp

description of some pattern situation with corresponding control solution; conclusion scheme like “If …, then …”. Its left part contains pattern situation, and its right part states corresponding control solution. control solution for situation S p .

Let’s bring an example here. Suppose during robot’s mission some contingence has occurred, which didn’t happen before. In that case control system doesn’t have ready solution to tackle this situation. In that case a search will be initiated to find some previously-known situation with similar description and existing control solution. Phase of recognizing similarities and differentiation will be search of similarities among description of current situation St and pattern situations S p . Situations descriptions contain sets of facts, which characterize situation. Set of facts is a set of parameters with their values and their weight, which reflect their influence on control decision. Once some degree of similarity will be achieved (this should be assigned ahead) a corresponding control solution is obtained, estimated (upon its efficiency) and is given as a recommended solution. In case of success simultaneously control model will be updated basing on data related to contingence. This update contains update of pattern parameters’ weights (e.g. depending on frequency of these parameters addressing).

Models for Decision Making Support Systems in Robotics

149

In case of lack of similarities differentiation method is used. Then a system search for a degree of differentiation among current situation and pattern situation. Once system meets pattern with too high degree of differentiation, that relevant pattern is blocked and remaining patterns are studied for possible application. Finally the less different pattern is selected as recommended for taking decision. Proposed approach for decision search in contingence situations provides adaptation of decision support system to contingence. Modification procedure allows model’s self-learing once practical experience is collected.

4 Conclusions It’s proved here that realization of proposed approach to decision support models for robots’ control in complex environments will facilitate increase of efficiency of teleoperated robots to the best operators’ experiences and solutions. Such approach will be able to forecast situation development, adapt to contingency and accumulate new knowledge along with working experience. Simultaneously this will increase reliability and quality of control within reasonable control tasks distribution among human and robot. Acknowledgements This work was partially supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAAA-A20120011690138-6.

References 1. Ermolov, I.L., Khripunov, S.P.: Formulating Generalized Structure of Robotic Systems № 1, “Robotics and Technical Cybernetics”, (2017) 2. Pospelov, D.A.: Reasoning modeling. Reasoning Acts Analysis. Moscow, “Radio i Svayz”, (1989) 3. Khripunov, S.P., Vasiliev, S.N., Blagodaryashev, I.V.: Approach to Synthesis of Intelligent Control Systems for Group of Robots №. 2, vol. 15, Moscow, “Radiotechnika”, InformantionManagement Systems, (2017) 4. Pospelov, D.A.: Logical-linguistic Models in Control Systems. Moscow, “Energoatomizdat”, (1981) 5. Ermolov, I.: Hierarchical data fusion architecture for autonomous systems. Acta IMEKO. 8(4) (2019) 6. Diane, S.A.K., Manko, S.V., Margolin, I.D., Novoselskiy, A.K.: Hierarchical scenarios for behavior planning in autonomous robots, Proceedings of the 2019 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering, ElConRus (2019) 7. Ermolov, I., Gradetsky, V., Knyazkov, M., et al.: Parameters identification in UGV group for virtual simulation of joint task. Proceedings of 14th International Conference on Electromechanics and Robotics “Zavalishin’s Readings” vol. 154 of Smart Innovation, Systems and Technologies. Springer Nature Singapore Pte Ltd Singapore, (2020) 8. Pospelov, D.A.: Situational Control: Theory and Practice. Moscow, “Nauka”, (1986)

Coalition Game Model of FANET Grouping Control Based on the Method of Local Threats and Counter-Threats and Swarm-Leader Model Evgeny M. Voronov, Vladimir A. Serov, and Dmitry A. Kozlov

Abstract Problem statement: Modern remote monitoring systems usually contain several FANET (Flying Ad Hoc Network)-subsystems that share limited resources and operate in a rapidly changing environment. FANET is a mobile network that includes a set of unmanned aerial vehicles, some of which carries out ground monitoring, and the others are used as nodes of a special communication network. Thus, the remote monitoring system is a structurally complex multi-object system that functions under conflict interactions and uncertainty. At the same time, FANET is subject to strict requirements in terms of network communication quality, survivability, monitoring quality. Satisfy these requirements can be achieved due to the high mobility of nodes and by the FANET-infrastructure components optimal control strategy forming. One of the promising areas of multi-object systems research is the technology of unmanned aerial vehicles swarm with a leader, using game methods of decision-making and control, based on the aggregation of different principles of conflict equilibrium, and allows to take into account the changing conditions of conflict interaction of FANET components. At the same time, a swarm intelligence technology, methodological and algorithmic basis for aggregation of the conflict equilibrium principles is not sufficiently developed, which prevents the practical implementation of effective group decision making and group control strategies under conflict and uncertainty. Purpose of research: development of a methodology for group decision-making and control based on combining the coalition game approach, “threats and counter-threats” principle and the swarm-leader model. Results: a development of a coalition game model of group decision making and group control of FANET components based on the “threats and counter-threats” principle and swarmleader model. A development of the group control laws synthesis method on the E. M. Voronov (B) Bauman Moscow State Technical University, 2nd Baumanskaya St., 5, Moscow 105005, Russia e-mail: [email protected] V. A. Serov · D. A. Kozlov MIREA—Russian Technological University, Vernadskogo Av., 78, Moscow 119454, Russia e-mail: [email protected] D. A. Kozlov e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_13

151

152

E. M. Voronov et al.

basis of “threats and counter-threats” principle and modified Waisbord-Zhukovsky sufficient conditions of local “threats and counter-threats” equilibrium. Practical significance: the proposed approach to group control allows to form scenarios of FANET components collective behavior aimed on changing its configuration. A group actions modeling based on the coalition game approach allows to take into account influence of different types of uncertain factors on the FANET performance. Algorithmic support of the group decision making and group control laws synthesis, allows on-board implementation. Keywords FANET (Flying Ad Hoc Network) · Swarm-leader model · Remote monitoring systems · Coalition game model of group decision making and group control · Local threats and counter-threats equilibrium

1 Introduction Modern remote monitoring systems usually contain several FANET (Flying Ad Hoc Network)-subsystems that share limited resources and operate in a rapidly changing environment. FANET is a mobile network that includes a set of unmanned aerial vehicles, some of which carries out ground monitoring, and the others are used as nodes of a special communication network. Thus, the remote monitoring system is a structurally complex multi-object system that functions under conflict interactions and uncertainty. At the same time, FANET is subject to strict requirements in terms of network communication quality, survivability, monitoring quality. To satisfy all the requirements, it is necessary to provide the possibility of structural and target adaptation of FANET when changing the operating conditions of the system. This can be achieved, on the one hand, due to the high mobility of nodes and dynamically changing structure, on the other hand, through the use of intelligent situational analysis systems that ensure the coordination of information interaction between FANET components and the formation of an optimal group control strategy. One of the promising areas of multi-object systems research is the technology of unmanned aerial vehicles swarm with a leader, using game methods of decision-making and control, based on the aggregation of different principles of conflict equilibrium, and allows to take into account the changing conditions of conflict interaction of FANET components [1–12]. However, a swarm intelligence technology, methodological and algorithmic basis for aggregation of the conflict equilibrium principles is not sufficiently developed, which prevents the practical implementation of effective group decision making and group control strategies under conflict and uncertainty. In article we propose a two-level scheme for creating a scenario of collective behavior of FANET-system components aimed at changing its configuration. At the high level, the problem of optimizing the coordinating control of a swarm of FANET subsystems leaders is solved, which ensures that each leader is brought to a given point in the terminal area. At the lower level, for each FANET-subsystem, the problem of stabilizing the position of individual UAVS relative to the leader and relative to

Coalition Game Model of FANET Grouping Control …

153

each other is solved based on the use of information about the mutual position and methodology [13]. The article deals with a high-level problem, which is solved by developing a group control laws synthesis method based on the “threats and counterthreats” principle and modified sufficient equilibrium conditions for local “threats and counter-threats” by Waisbord-Zhukovsky [14].

2 Statement of the Problem of a FANET-System Coordinating Control Optimization The mathematical model a FANET-system coordinating control includes the following components: a coalition structure; a group dynamics model; a vector performance indicator.

2.1 Coalition Structure The FANET-system coalition structure characterizes conditions of conflict interaction components within a separate FANET, as well as between different FANETsubsystems included in the remote monitoring system.  The coalition structure is formalized as a decomposition P of the set L = 1, l of participants in the conflict (unmanned aerial vehicle—UAVS). ⎫ ⎧    n ⎬ ⎨  K j = ∅; Kj = L . (1) P = K1 , . . . , Kn Ki ⎭ ⎩  i= j j=1   Members of the coalition Ki (FANETi ), i ∈ N = 1, n , are united by common interests and act in concert as a single player. We assume that the movement of each coalition is based on the “swarm with a leader” model. Coalitions Ki and K j , i = j, i, j ∈ N have different goals, different leaders, and act uncoordinated, i.e. there is a non-coalition interaction between them.

2.2 A Group Dynamics Model The group dynamics model of the FANET-system describes the joint movement of coalition leaders and is given as a system of ordinary differential equations x˙ = f(x, u1 , . . . , un ), x(t0 ) = x0 , ui ∈ Ui .

(2)

154

E. M. Voronov et al.



In (2) xT = x1T , . . . , xnT , where xi —coordinates vector of the coalition Ki (FANETi ) UAV-leader; u = {ui , i ∈ N}; ui —control vector of the UAV-leader of the coalition icii Ki ; f—nonlinear vector-function. We assume that the control vector functions ui ∈ Ui , i ∈ N, allow parametrization, namely, for the player i piecewise continuous control u i (t):    u i (t) = q1i · (1[t − t0 ] − 1[t − t1 ]) + . . . + qki · 1 t − tk−1 − 1[t − tk ] , i ∈ N, (3) where qi min ≤ q ji ≤ qi max , t ∈ [t0 , T ].

2.3 A Vector Efficiency Indicator A vector efficiency indicator of coordinating control is a vector target functional J(u) = [J1 (u), . . . , Jn (u)]T , where component Ji (u)—efficiency indicator of the coalition leader’s control Ki , that makes sense as a loss indicator. The structure of the functional Ji (u) takes into account the restrictions imposed by the requirements: – getting the leaders of coalitions Ki ∈ P at a finite time in the center of the terminal area, which is expressed by a system of restrictions-equalities: i (T, x(T )) = 0, i ∈ N;

(4)

– mutual distance of leaders Ki , K j ∈ P, i = j, with the goal of non-intersection coalitions, which is formalized as a system of restrictions-inequalities: i (x(t))≤0, i ∈ N.

(5)

With this in mind, the functional Ji (u) is set as: T Ji (u) = i (T, xi (T )) +

G i (t, x, u, i (x)) · dt t0

T = i (T, xi (T )) +

Fi (t, x, u) · dt, i ∈ N.

(6)

t0

Minimization of the functional (6) formalizes simultaneous satisfaction of the requirements for the coalition Ki leader: getting to the center of the terminal area; ensuring distance from the leaders of other coalitions K j ∈ P, i = j; minimization energy costs.

Coalition Game Model of FANET Grouping Control …

155

Thus, the optimization problem of the coordinating control of a FANETsubsystems leaders swarm can be formulated in the a coalition game form:    = L, P, {Ui }i∈N , {Ji (u)}i∈N , ℘ .

(7)

With a fixed coalition P, the problem (7) is interpreted as a non—coalition   structure game on a set N = 1, n of players-leaders of coalitions Ki , i ∈ N, for which the Nash equilibrium principle is used as the optimality principle ℘. However, in [15, 16] it is shown that the concept of threats and counter-threats is more general and most acceptable from the point of view of practical applications. Definition 1 [15] In problem (7), the threat of player i is the possibility of changing the control ui to vi ∈ Ui such that the inequality is satisfied: Ji (vi , uN\i ) < Ji (ui , uN\i ),

(8)

where N\i is a counter-coalition made up of all players except the i-th. In order for player i to have no desire to change the situation (ui , uN\i ) when the inequality (8) is satisfied, the counter-coalition N\i must be able to replace its controls uN\i with controls yppavleni vN\i ∈ UN\i so that for the set v = (vi , vN\i ) the counter-coalition N\i counter-threat conditions are satisfied: Ji (vi , vN\i )≥Ji (ui , uN\i ); JN\i (vi , vN\i ) < JN\i (vi , uN\i ).

(9)

Definition 2 A set (ui , uN\i ) is a threat and counter-threat (TCT) for player i if counter-coalition N\i has a counter-threat for any of its threats.   Definition 3 A set ui , uN\i is called a TCT-optimal solution of the game if for any threat of any player i, the counter-coalition N\i has a counter-threat. Comment. Nash equilibrium is a special case of the TCT-optimal solution of the game (7). In [15], the concepts of local threats and counter-threats (LTCT) are formulated, and sufficient conditions for LCTC-optimality of solutions to the differential coalition game are formed.

156

E. M. Voronov et al.

3 General View of Waisbord—Zhukovsky Sufficient LTCT-Optimality Conditions The indicator of loss of the coalition Ki leader: T Ji (u) = i (T, xi (T )) +

Fi (t, x, u1 , . . . , un )dt, i ∈ N.

(10)

t0

The local threat of player i is the possibility of player i replacing the control ui (t) T with ui (t) vi (t) ∈ Ui , ui (t) − vi (t)2 dt < ε, so that inequality (8) is satisfied. t0

A local counter-threat of the counter-coalition N\i is the possibility of replacing counter-coalition N\i control uN\i (t) with vN\i (t) ∈ UN\i ,  T  uN\i (t) − vN\i (t)2 dt < ε, to satisfy the inequalities system (9). t0

The local pattern of threats and counter-threats is taken into consideration for correction of the TCT-optimal solutions obtained using evolutionary algorithms [17, 18]. Definition 4 [15] A local threat and counter-threat for player i is a set of controls u(t) = {ui (t), u N \i } ∈ U for which there is a constant ε > 0 such that for any local threat of player i, the counter-coalition N\i has a local counter-threat. Definition 5 [15] If the same set of controls is a local threat and counter-threat for any player i, then u(t) is called the LTCT-optimal solution of the game (7). The stable properties of LTCT-optimal solutions generalize the known properties of Nash equilibrium. To obtain sufficient conditions, the class of acceptable variations ui (t) i uN\i (t) is limited to acceptable controls of the form vi (t) = ui (t) + γi · u¯ i (t), vN\i (t) = uN\i (t) + γN\i · u¯ N\i (t), where u¯ i ∈ Ui , u¯ N\i ∈ UN\i , and γi , γN\i are constants. The constants may be chosen to be so small in absolute value, that with limited values u¯ i , u¯ N\i there is: T   2  ui (t) − vi (t)2 + uN\i (t) − vi (t) · dt < ε. t0

Systems of the following type are introduced

(11)

Coalition Game Model of FANET Grouping Control …

ξ˙ j (t) = A(t) · ξ j (t) + B j (t) · u¯ j (t), ξ(t0 ) = 0, ∂f ; ∂x

157

(12)

∂f ∂u j

( j = i, N\i) are Jacobi matrices. Further, for   0 . convenience, we will denote the LTCT-solution as u0 = ui0 , uN\i

where A(t) =

B(t) =

  0 ∈ U to be a local threat and Theorem 1 [15] In order u0 (t) = ui0 (t), uN\i counter-threat for the player i, it is sufficient that: (a) (b)

vectors g1 (i) = gi,N\i (t) i g2 (i) = gN\i,N\i (t) was linearly independent (equality α1 g1 (t) + α2 g2 (t) = 0 is possible only if α1 = α2 = 0); for any acceptable u¯ i = u¯ i (t) there was an inequality

T

  gi,i (t), u¯ i (t) dt = 0,

t0

where (·, ·) is the scalar product;

T gk, j (t) = BTj (t) · Y−1 (t) ⎫ ⎧   T ⎬ ⎨ 0 0 0 T, x ∂ (T ) (τ, x (τ ), u (τ )) ∂ F k k + Y T (τ) · · dτ × YT (T ) · ⎭ ⎩ ∂x ∂x t

+

∂ Fk (t, x (t), u (t)) , k, j = i, N\i. ∂u j 0

0

As in the general case, these sufficient conditions for local TCT are also difficult for practical applications. Really, for satisfaction condition (a), there is, in turn, a necessary condition: if g1 and g2 are linearly independent, then the Gramm determinant [15]     g ,g g ,g   =  1 1  1 2   > 0, g2 , g1 g2 , g2

 T   where gi , g j = gi , g j dt, (·, ·)—scalar product. For the first, if this condition is t0

satisfied, the functions may be linearly dependent, and for the second, this condition is difficult to verify. To check condition (b), you need to solve the integral equation. T t0

  g K ,K (t), u¯ K (t) dt = 0

158

E. M. Voronov et al.

and make sure, that in except for trivial solution u¯ i (t) ≡ 0, which should not be included in Ui , for all u¯ i (t) ∈ Ui there is no solution for all. Herewith the cernel of the equation gi,i (t) has a complex form. Finally, in all cases it is necessary to have a precise description of the transition function X(t, τ) = (Y(t), YT (τ)) of the system (12). The conditions considered can be used for linear-quadratic game models [15, 16]. To solve more general application problems, they need to be modified.

4 Modified Sufficient Conditions for LTCT   0 to be a local threat and counterTheorem 2 [14] For a set u0 (t) = ui0 (t), uN/i   ¯ threat to the player i, it is sufficient that for any allowed u(t) = u¯ i (t), u¯ N \i ∈ U a system of inequalities is satisfied: 

∂ f i (0, 0) = 0 ; ∂γi

∂ f i (0, 0) > 0; ∂γN\i

∂ f N\i (0, 0) < 0, ∂γN\i

(13)

where 

   ∂k T, x0 (T ) , ξ j (T ) ∂x      T  ∂ Fk t, x0 (t), u0 ∂ Fk , ξ j (t) + + , u¯ j dt, ∂x ∂u0j

∂ fk = ∂γ j

(14)

t0

k, j = [i, N \i], ξ˙ j (t) = A(t) · ξ j (t) + B j (t) · u¯ j (t),  ξ j (t0 ) = 0, A =

   ∂f ∂f , , Bj = ∂x ∂u j

0 x˙ 0 (t) = f(x, ui0 , uN\i ), x(t0 ) = x0 ; v j ≡ u0j + γ j · u¯ j ,

(15) (16) (17)

v j , u0j , u¯ j ∈ U j , the vector of small values γ j o is chosen from the condition T t0

where ε > 0—small value.

  0 u − v j 2 · dt < ε, j

(18)

Coalition Game Model of FANET Grouping Control …

159

If the indicators have the meaning of efficiency indicators, then the signs of the second and the third inequalities in (13) are inversed.

5 Optimization Method The conditions of theorem 2 assume, that inequalities (13) satisfied on the set of admissible coalition controls u¯ j , j ∈ (i, N/i). Therefore, each of the three scalar products in expression (14) is a product of a vector that depends on the optimal control and trajectory, on any vector from the allowed set. If in the first term the set ξ j (T ) has the meaning of the reachability set for t = T , then in the integral term (14) there is an ensemble of trajectories ξ j (t) and a set of controls u¯ j (t). One of the possible variants for methodological simplification of the algorithm structure is to reduce the original problem to such form, when for to get u0 it is sufficient to use only the reachability areas (RA). To do this, enter additional coordinates: x˙ 0i = Fi (t, x, u), x0i (t0 ) = 0; x˙ 0N/i = FN/i (t, x, u), x0N/i (t0 ) = 0,

(19)

and the original problem is reduced to a problem with a terminal indicator ¯ i (¯x(T ), T ); JN/i =  ¯ N/i (¯x(T ), T ), Ji = i (x(T ), T ) + x0i (T ) =    where x¯ (T ) = x0i (T ), x0N/i (T ), x(T ) —extended vector. Then,     ¯ k x¯ 0 (T ), T ∂ ∂ fi , ξ¯ j (T ) , k, j = (i, N/i) = ∂γ j ∂ x¯

(20)

(21)

¯ ¯ j (t) · u¯ j (t), ξ¯ j (t0 ) = 0, u¯ j ∈ U j · ξ¯ j (t) + B ξ˙¯ j (t) = A(t)

(22)

0 0 x˙¯ (t) = f¯(x0 , ui0 , uN/i ), x¯ (t0 ) = x¯ 0 ,

(23)

where the last system has the form ⎧ 0 0 0 0 ⎪ ⎨ x˙ 0i (t) = Fi (t, x , ui , uN/i ), x0i (t0 ) = 0; 0 0 0 x˙ 0N/i (t) = FN/i (t, x , ui0 , uN/i ), x0N/i (t0 ) = 0; ⎪ ⎩ x˙ 0 (t) = f(x0 , u0 , u0 ), x(t ) = x . 0 0 i N/i In this interpretation, sufficient conditions take the form of a system

(24)

160

E. M. Voronov et al.

⎧  ¯  ∂ Φi (T,¯x0 (T )) ∂ fi ⎪ ¯i (T ) = 0 = , ξ ⎪ ⎪ ∂γi ⎨   ¯ ∂ x¯ 0 ∂ Φi (T,¯x (T )) ∂ fi ¯N\i (T ) > 0 , = , ξ ∂γN\i ∂ x¯ ⎪   ⎪ 0 ⎪ ⎩ ∂ fN\i = ∂ Φ¯ N\i (T,¯x (T )) , ξ¯ (T ) < 0 ∂ x¯

∂γN\i

(25)

N\i

if there are links (22)–(24). Here and further we consider piecewise continuous scalar or vector controls u j (t) or program-corrected control laws u j (t) = u j (t, x(tr −1 )), tr −1 ≤t≤tn = T, r = 1, n,

(26)

0 ), that on the sets of admissible controls Thus, it is necessary to find a pair (ui0 , uN\i u¯ i ∈ Ui , u¯ N\i ∈ UN\i , and, as a consequence, on the sets ξ¯i (t), ξ¯N\i (t), ensures the satisfaction of the inequalities system (25). The algorithm is implemented using the method of moments by N. Krasovsky [19], since it operates with (RA) and allows us to find the normals of hyperplanes tangent to RA.

Theorem 3 [19] Optimal control that drives a trajectory ξ¯ j (t) of the system ξ˙¯ j (t) = A · ξ¯ j (t) + B¯ j · u¯ j , ξ¯ j (t0 ) = 0

(27)

to the tangent point of RA and the hyperplane, as well as the normal vector at the tangent point, are determined when solving the problem T min max

 =1

u¯ j

T · X(T, τ) · B¯ j · u¯ j (τ) · dτ = 0,

(28)

t0

where X(T, τ) is the matrix of fundamental solutions of the system: ⎡

⎤ x11 (T, t) · · · x1n (T, t) ⎢ ⎥ .. .. X(T, t) = ⎣ ⎦. . .

(29)

xn1 (T, t) · · · xnn (T, t)

In accordance with the results obtained, the general structure of the control optimization method based on the combination of modified sufficient LUCU conditions and the method of moments by N. Krasovsky can be represented as an iterative process including the following steps. Step 1. Step 2.

Reduction of the original statement to the form (20)–(24). Formation of the inequalities system (25) (modified sufficient conditions for LTCT.

Coalition Game Model of FANET Grouping Control …

Step 3.

Step 4.

Step 5.

161

The formation of the initial approximations u0 and the surrounding area δU valid values znaqeni u ∈ U. For this purpose, we use a library of evolutionary algorithms for multi-criteria conflict optimization [17, 18], which provide search for optimal solutions with any given accuracy ε. Formation of the system (27) (A, B, X(T, t)) based on approximations u0 ∈ U; solution of the problem (28) for determining the boundaries of the normal cones Con , satisfying the modified sufficient LTCT conditions, as well as the corresponding cones Con ξ , formed by the vectors ξ¯i (T ), ξ¯N\i (T ), tangent to RA. Solving the problem Pareto-optimization u ∈ U for a set of coalitions i i N\i and for initial approximations u0 ∈ U and additional constraints, formed at step 4 in one of two types:u0 ∈ Con ; u0 ∈ Con ξ¯ , i.e. satisfying the system of inequalities (25) for ξ¯i (T ) and ξ¯N\i (T ).

6 Conclusion A coalition game model of group control of FANET components based on the threats and counter-threats principle and swarm technology with a leader was formed. A method for group control laws synthesis based on the threats and counter-threats principle and modified sufficient conditions for the local threats and counter-threats equilibrium by Weisbord-Zhukovsky is developed. The proposed approach to group control allows you to create scenarios for collective behavior of FANET components aimed at changing its configuration. Modeling group actions based on a coalition game approach allows you to take into account the impact of various types of uncertain factors on FANET performance indicators.

References 1. Akkarajitsakul, K., Hossain, E., Niyato, D.: Coalition-based cooperative packet delivery under uncertainty: a dynamic bayesian coalitional game. IEEE Trans. Mobile. Comput. 12(2), 371– 385 (2013) 2. Altman, E., Kumar, A., Singh, C., Sundaresan, R.: Spatial SINR games of base station placement and mobile association. IEEE/ACM Trans. Network. (TON) 20(6), 1856–1869 (2012) 3. Chuyko, J., Polishchuk, T., Mazalov, V., Gurtov, A.: Wardrop equilibria and price of anarchy in multipath routing games with elastic traffic. Game Theory Appl. pp. 9–19 (2012) 4. Eidenbenz, S., Kumar, A., Zust, S.: Equilibria in topology control games for ad hoc networks and generalizations. Mobile Network Appl. 11(2), 143–159 (2006) 5. Han, Z., Niyato, D., Saad, W., Basar, T., Hjerungnes, A.: Game theory in wireless and communication networks: theory, models, and applications. Cambridge University Press. p. 536 (2012) 6. Jaramillo, J.J., Srikant, R.: A game theory based reputation mechanism to incentivizc cooperation in wireless ad hoc networks. Ad Hoc Network. 8(4), 416–429 (2011)

162

E. M. Voronov et al.

7. Long, C., Zhang, Q., Li, B., Yang, H., Guan, X.: Non-cooperative power control for wireless ad hoc networks with repeated games. IEEE J. Select. Area. Commun. 25(6), 1101–1112 (2007) 8. Ren, H., Meng, M.Q.-H.: Game-theoretic modeling of joint topology control and power scheduling for wireless heterogeneous sensor networks. IEEE Trans. Automat. Sci. Eng. 6(4), 610–625 (2009) 9. Saad, W., Han, Z., Ba§ar, T., Debbah, M., Hjarungnes, A.: Network formation games among relay stations in next generation wireless networks. IEEE Trans. Commun. 49(9), 2528–2542 (2011) 10. Shi, H.-Y., Wang, W.-L., Kwok, N.-M., Chen, S.-Y.: Game theory for wireless sensor networks: a survey. Sensors 12(7), 9055–9097 (2012) 11. Voronov, E. M.: Methods of optimization of multiobject multicriteria systems control on the basis of stable-effective gaming solutions. BMSTU, Moscow. 576c, (2001) 12. Serov, V.A., Voronov, E.M., Kozlov, D.A.: Hierarchical neuro-game model of the FANET based remote monitoring system resources balancing. In: Gorodetskiy, A., Tarasova, I. (eds.) Studies in Systems, Decision and Control. Smart Electromechanical Systems. Situational Control, vol. 261, pp. 117–130. Springer International Publishing. https://doi.org/10.1007/978-3-030-327 10-1 13. Group control of moving objects in uncertain environments. In: Kh. Pshikhopov, V. M.: FIZMATLIT, p. 305. ISBN 978-5-9221-1674-9. (2015) 14. Obnosov, B. V., Voronov, E. M., Mikrin, E. A., Serov, V. A. et al.: Stabilization, guidance, group control, and system modeling of unmanned aerial vehicles. Modern approaches and methods. vol. 2. Publishing House of Bauman Moscow state technical University, Moscow, p. 520 (2018). ISBN 978-5-7038-5058-9 15. Weisbord, E. M., Zhukovsky, V. N.: Introduction to differential games of several persons and their applications. M.: Sov. Radio, p. 304 (1980) 16. Zhukovsky, V. I., Chikriy, A. A.: Linear-quadratic differential games. Kiev: Naukova Dumka. p. 320 (1994) 17. Serov, V. A., Voronov, E. M.: Evolutionary algorithms of stable-effective compromises search in multi-object control problems. In: Gorodetskiy, A., Tarasova, I. (eds.) Studies in systems, decision and control. Smart Electromechanical Systems. vol. 174, pp. 17–29. Springer International Publishing, (2019). https://link.springer.com/book/10.1007/978-3-319-99759-9. https:// doi.org/10.1007/978-3-319-99759-9 18. Serov, V. A.: Adaptive fitness functions in evolutionary game models of control optimization in structurally complex systems. Bull Bauman Moscow State Tech Univ. Ser. Instrumentation. 113, pp. 111–122 (2017) 19. Krasovsky, N. N.: Control of the dynamic system. The problem of the minimum guaranteed result. Nauka, Moscow. p. 520 (1985)

Robotic Wheelchair Control System for Multimodal Interfaces Based on a Symbolic Model of the World P. S. Sorokoumov, M. A. Rovbo, A. D. Moscowsky, and A. A. Malyshev

Abstract Problem statement: Wheelchairs have a widespread usage both as a rehabilitation tool and as an assistive device in the daily lives of people with disabilities. However, some disabilities make it difficult to use traditional wheelchairs that are operated manually or with a joystick. This paper describes an intelligent control system for wheelchair automatization, which allows the user to give commands by different means—voice, eye movements, muscle tension. The object recognition system and logical processing of commands supports a wide variety of interfaces and commands. To achieve these goals, a semiotic model of the world is used. Purpose of research: development of a control system for a robotic wheelchair that supports multimodal interfaces and has a high level of automation that enables efficient operation for users with various disabilities. Results: The paper describes the developed architecture of the control system based on the semiotic model of the world, modules for the speech interface, gaze control, and the interface based on myosensors. The navigation system and the processing module for the semiotic world model ensure the safe execution of user commands, including movement, object recognition and processing of commands that contain references to known objects. The system supports interfacing with a manipulator, which is controlled using a linguistic model: a language for describing actions representing the admissible movements of the manipulator in the form of a formal grammar. The study tested the proposed system on a developed model of the robot and a detailed model of the room in a Gazebo environment, as well as on the corresponding software and hardware implementation of the robotic wheelchair. Practical significance: Automation and the use P. S. Sorokoumov (B) · M. A. Rovbo · A. D. Moscowsky · A. A. Malyshev National Research Center “Kurchatov Institute”, Moscow, Russia e-mail: [email protected] M. A. Rovbo e-mail: [email protected] A. D. Moscowsky e-mail: [email protected] A. A. Malyshev e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_14

163

164

P. S. Sorokoumov et al.

of models and methods of artificial intelligence in the development of wheelchairs allows us to make them more versatile, which increases the number of users for whom a particular model is suitable, reduces the strain on the operator of the wheelchair and expands the capabilities of the wheelchair. The proposed usage of the sign models achieves these goals by combining logical processing of commands as a means of handling multimodal interfaces and executing complex commands. Keywords Semiotic models · Wheelchair · Mobile robot · Manipulator control · Multimodal interface

1 Introduction In the modern world a huge number of people use wheelchairs [1]. First models of wheelchairs were manually driven, but electric-driven systems are becoming more common. They are usually controlled by a joystick, and, despite the fact that this greatly facilitated the use of a wheelchair, many categories of disabled people are not able to apply them due to the nature of disorders, for example, paralysis or tremor. It is not possible to simply replace the joystick with another input device that reads, for example, speech commands, eye movements, or other signals that the operator is capable of giving. In practice these types of input devices are prone to numerous disadvantages such as accidental command issuance, command issuance by an unauthorized person and absence of quick cancel commands. These drawbacks dramatically reduce safety of the wheelchair. The only solution seems to be the intellectualization of the wheelchair, that is, development of a control system that can automatically correct user errors. Safety issues should be given special attention in this case. The easiest way to monitor and estimate the safety of an automated system is to use interpreted approaches, i.e. representation of the solution to any given problem as a sequence of stages with unambiguously clear meaning. If there is sufficient information about changes in the state of the system, it is possible to understand the cause of the problems encountered and to overcome them. The interpretability requirements are well met by logical inference. One of the modern ways of their practical application are semiotic systems, in which signs are used to describe the state of a robot. Here signs are data structures that combine data on objects known to an intelligent agent and methods of interaction with these objects. Different authors formalized semiotic systems in different ways, but here a model described in [2] used. It describes strictly formally both the structure and the functioning of the world representation, including the sign generation procedure. Earlier, the convenience of this approach for solving problems of multimodal wheelchair control was already noted in [3], but there the semiotic model was used mainly as a common interface for different types of input devices. The possibility of deriving a new sign from the perceptions of agents, which is essential for the

Robotic Wheelchair Control System for Multimodal Interfaces …

165

normal operation of the sign system, was not then applied. Instead, the set of signs was assumed to be known and fixed. Further in this paper, first the general structure of a semiotic control system for a robotic wheelchair is described, then proposals for solving important applied problems facing it are discussed in more detail: the execution of user commands of various types, the operation of the manipulator and the generalization of sensory data to update the world model. The developed components were tested on both software model and real robot. The selected approach was shown to be applicable.

2 Basic Architecture of the System The wheelchair control system is organized hierarchically to ensure that the results of each component are interpretable. Its basic structure is described in detail in [3]. The following functional levels can be distinguished: • direct control of executive devices (motors, sensors, etc.) by hardware-dependent commands; • control of robots by hardware-independent instructions using low-level abstractions as “mobile platform with differential control”, “range finder”, etc.; • solving primitive problems, that can be modeled by finite state machines (FSM); • solving of complex problems, that is, those that are described using “metaFSM”—FSMs the states of which correspond to primitive problems. That process, once started, determines the behavior of the robot; • achieving the goals stated by a user and represented in symbolic semiotic model. The range of tasks that are usually assigned to a wheelchair is rather narrow and mostly come down to spatial movements. In addition, the current hardware limitations (primarily related to sensors and navigation devices) do not allow automated wheelchairs to move safely outdoors, so the variety of working environments is small. In the simplest case, the problems of wheelchair navigation in a room have already been solved, including by systems capable of learning [4]. However, adding to the used semiotic system the ability to form new abstractions can significantly expand the capabilities of the system. For compatibility with modern frequently used software packages, the described software was developed as services of a common robot programming tool—ROS framework. It supports both real and simulated robots. Since the differences between these two situations are mainly focused on low levels of abstraction, the results of running a high-level control system on a model and on a real robot are usually quite similar.

166

P. S. Sorokoumov et al.

2.1 Representation of Semantic Map According to [2], the sign is a tuple with four components : name n; image p, represented by a set of predicates—true statements about the signified object; meaning m, represented by a set of rules; and the personal meaning a, also represented by a set of rules. In the developed system, a rule is understood as an operator of the STRIPS model, that is, the triple , where cond is the set of predicates—conditions for the rule’s applicability, add is the set of predicates that become true after the rule is executed, del is the set of predicates that become false after the rule is executed. A high-level representation of the world of a wheelchair consists of signs of the described type. An example of a symbolic view of the world is shown in Fig. 1. Two signs (subjective “me” of the robot and “door1”) are demonstrated in details, three another (“robot”, “door” and “object”) represent object hierarchy. Possible action of the robot “move to” also include references to other signs. In the simplest variant of semiotic model all components, except for the image p, are fixed; the image representing the current knowledge of the model about the signified object is changed either by rules from m and a, or by external services that implement the replenishment of sensory data. Thus, the structure of connections of the semiotic subsystem with other components can be represented as a diagram in Fig. 2. The images are replenished with sensory data as they arrive, by rules as they are triggered, or by a special service for the signs generation; data from images is analyzed to launch complex tasks. The processing results can be interpreted due to logical inference in semiotic representation. Updating of images, in accordance with [2], should use the history of changes, that is, the precedents of interaction between the agent and the signified entity. The general

N:

N: me M

door1

near

P

type

A

X

object

A

type M

robot door

type

subclass

subclass

move subject to

Fig. 1 Part of semiotic model

P

near

object Y

type

Robotic Wheelchair Control System for Multimodal Interfaces … Fig. 2 Interaction of the semiotic subsystem with other components

167

Image reneval Sensory services

Images of signs

Sign-generation services

Rule-based system Semiotic subsystem

Analysis of images

Launcher for complex tasks

solution to this problem in too difficult, because the agent will need a purposeful study of the surrounding objects, which can distract him from fulfilling the operator’s instructions. However, due to the limitations of the subject area, in some cases this task can be partially solved using ordinary sensory data. The robot’s computer vision system can recognize objects and describe them in semiotics terms. But it is difficult to use this data for inference, because the types of these objects greatly depend on the recognizer used: for example, a human figure can be designated as “Man”, “Human”, “Woman”, “Person”, etc. Often a recognizer in different visibility conditions denotes an object differently. The robot should perceive recognized objects as obstacles and mark them on the map accordingly. However, if the class of the object indicated by the recognizer is initially unknown, it is undesirable to consider the object as an obstacle in the sign system, because in this case moving objects and parts of stationary objects will also be considered obstacles, cluttering the model and slowing down its operation. Based on this, it is useful to create a separate procedure that can update the set of signs by identifying the recognized objects with obstacles. The proposed algorithm of this procedure is as follows: 1. 2.

3.

Obtain sensory data and data about the existing obstacles from the images of the signs; Considering each element of sensory data as a candidate for obstacles, identify it with a certain spatial model (i.e. distribution of the obstacle in space) based on its recognized size and, possibly, assumptions about the shape; Compare the obtained data with the information about the existing obstacles; exclude candidates for which the calculated reliability of coincidence with the existing obstacle is greater than the threshold specified. This step cancels processing of known obstacles and their parts;

168

4.

5.

P. S. Sorokoumov et al.

Compare the data with obstacle candidates obtained from the previous step. Specially selected comparison operation produces a joint assessment of matching. If the calculated certainty that the object is an obstacle is greater than a threshold, then it can be stated that the object is indeed a new obstacle, and a sign for it should be generated. The image of this sign includes information about the object as an obstacle and features from the recognizer; If the object could not be matched with the obstacle, then it is necessary to reduce its reliability and save it until the next iteration; if the reliability decreases below the threshold value, the candidate is excluded from further processing.

A schematic example of the algorithm’s operation is shown in Fig. 3. New obstacles and intermediate data for the next iteration are generated by information about known obstacles from a previous step and sensory data. The algorithm is not universal because threshold values and reliability processing method have to be selected empirically. In addition, the removed obstacles do not disappear automatically. Nevertheless the outlined approach is still useful for some specific tasks. Fig. 3 An example of a heuristic obstacle recognition algorithm for the semiotic subsystem

Raw data from sensors

Known obstacles

Candidates

New obstacles

New candidates

Robotic Wheelchair Control System for Multimodal Interfaces …

169

2.2 Low-Level Components The robotic intelligent wheelchair is a modified chair Ortonica Pulse 330. The standard motor driver has been replaced with a Roboteq MDC2460 with highprecision encoders. The control system, running under Kubuntu Linux 20.04 OS and ROS Noetic framework, is deployed on a single-board computer Intel BOX NUC7. Another single-board computer interacts with the hardware and software running exclusively under Windows OS. Robot uses RPLidar A1 laser range scanner, Microsoft Kinect 360 infrared camera, and five HC-SR04 ultrasonic rangefinders installed in the rear arch (Fig. 4). The software control system provides two levels of wheelchair control. The first is operational control. It allows a user to control the wheelchair directly using gaze or joystick. The second one accepts goal points from the operator, that can be given via the gaze interface by looking at a specific point on the floor. The navigation is then handled by an automatic planner. In both modes the motor driver device provides speed control to achieve a given velocity quickly and smoothly, while maintaining acceleration limits to prevent jerks while driving. When a direct control command is given, a subsystem analyzes the user’s input taking into account the recognized obstacles to block commands that can lead to

м 12

5м 5м

Fig. 4 Sensor coverage fields of the wheelchair: green area for the laser range scanner, red for the Kinect sensor, blue for ultrasonic sensors

170

P. S. Sorokoumov et al.

emergency situations: collision, fall, overturn. The data from obstacle sensors is aggregated in the ROS navigation obstacle maps for this purpose. The reflex module uses this data to prevent the execution of speed commands leading to collisions with obstacles based on a predictive model, like the one used in the DWA local planner [5] to obtain feasible states. It calculates a zone that will be touched by the wheelchair while moving at the current speed for a short (several seconds) period of time. If an intersection of the zone and obstacle is detected, the command is blocked and the wheelchair stops. In the future, we plan to add a fullfledged control with a predictive model to the reflex module. This should allow not only to interrupt the commands leading to a collision, but to correct them in an acceptable interval so that the collision is avoided while the wheelchair continues moving in roughly the same direction that it was commanded to. This mode of operation will reduce the load on the operator in the direct control mode by processing the input commands while allowing for some error. For the second type of control (autonomous movement towards a given goal) the standard ROS navigation stack software is used with the TEB planner. TEB (Timed Elastic Band) planner is a local motion planner [6]. It provides optimal movement along a given trajectory and reacts to dynamic obstacles, ensuring efficient and safe movement along the route to the requested point. The route is generated by the global trajectory planner A* that uses the known terrain map and is regularly called to replan the route to account for emerging obstacles and new information about the surrounding area. Goal designation by using an interface just with a monitor and a mouse may be difficult for an operator with disabilities. One of the possible solutions implemented in this robotic wheelchair is an eye tracking interface. When it is well calibrated, this system allows to calculate the goal point rather accurately by projecting a ray from the user’s eyes towards the surface on which the wheelchair stands (the surface is assumed to be perfectly flat and horizontal). Thus, in order to set the point to which you want to drive, it is enough to look at it. However, despite the simplicity of this interface, it has a number of limitations, one of which is a small sector in which the user’s gaze can be detected. Therefore, this system can be used only in combination with direct control to be able to rotate towards the goal. Also, in difficult environments, where there is no eye contact with the final goal, a user has to send a sequence of commands, which is also not very convenient. Therefore, the development of more sophisticated interfaces for defining the high-level user command is required. Despite the presence of an autonomous navigation system and manual computer-assisted control, these tools alone are not enough for comfortable usage of the robotic wheelchair. To solve this problem, the interface has a high-level semiotics-based control module that supports commands given by speech, for example “move to the object”. This command performs the movement of the wheelchair to a specified object or point in space, bringing it to an obstacle-free area next to the target. But the command processing system in semiotic form must correctly match the object indicated in speech and the corresponding representation in the model of the world. In this case, ambiguities may arise: for example, persons often omit the features that distinguish a specific object unambiguously, if they are talking about the nearest object directly in

Robotic Wheelchair Control System for Multimodal Interfaces …

171

front of it. To make the system more intuitive to operate, ambiguity resolution heuristics are used, that can, for example, choose the closest known object. However, it is important to note that such a decision can lead to errors in the interpretation of the operator’s instructions. After converting the oral instruction to a semiotic form, the generated movement command is sent to the navigation stack using the special ROS action service which moves a robot in a random point of the specified zone allowing interrupting the movement and tracking its completion. Thus, the operator can either directly control the wheelchair in the assisted manual (direct) control or give a more abstract command, which will be executed by the wheelchair using the information about the objects available in its model of the world and the recognition system’s ability to detect them. The oral command is converted to a semantic network by a pattern recognition module that uses language-depended standard syntax templates. The generated instructions for semiotic system are then matched with semiotic network that represents the robot’s world model. The absence of matches is interpreted as an a nonexecutable command, while multiple matches — as an ambiguous command. For example, the command “move to the door” looks like these RDF triplets: Action type action. Action subject me. Action action_type move_to. Action object Goal. Goal type door. Matching of such a network with the robot’s model of the world leads to the search for such values of the linguistic variable Goal that there is an element “Goal type door” in the view of the world. The valid Goal values are the names of all doors known to the system. If there is more than one such name, ambiguity arises, which can be corrected by a heuristic choice. At the moment, either the nearest target or the target located directly in front of the robot will be selected. Another option is to clarify the goal by specifying additional features of Goal in a new command (for example, its location relative to other objects). This form of command representation can be used with any type of input device to specify both tactical and strategic instructions.

2.3 Sensory Subsystem In order to implement the proposed system, powerful perception subsystem is required. Since high-level voice commands contain the names of objects, it is necessary to recognize and localize them. A network with the SSD MobileNet architecture trained on the Open Image dataset is now used. The choice of this architecture is dictated by the fact that the on-board computer does not have a graphics accelerator to work with more accurate networks like Mask-RCNN and Faster-RCNN. Localization of objects provided by Microsoft Kinect infrared camera. Distance to an object is calculated by median averaging of distances to each pixel of the recognized object.

172

P. S. Sorokoumov et al.

Due to the fact that parts of the background fall into the bounding rectangle, this approach obviously carries a certain measurement error, which mainly depends on the shape of the object. Since the goal of the developed semiotic control interface is partly to eliminate the shortcomings of the eyetracking interface, it is required not only to localize the objects recognized at the moment, but also to remember all the objects that were encountered. Then user will then be able to move to any known object giving only one command. Objects with their coordinates form the semantic layer of the map. While any robot localization method can be used as a localization layer, now the system uses RTAB-MAP [7] to build obstacle map. All localized objects are saved in a special semantic map layer, their coordinates for the map provided by recognition subsystem. Positions of these objects are distributed in space due to inevitable localization errors. Hierarchical clustering algorithm determines which of newly obtained data points refer to known objects or new ones. The position of the object in this case is defined as the center of the cluster. However, the system was designed for real-time mapping, and the cost of solving the clustering problem increases dramatically with the number of recorded measurements. To avoid this, a buffer of free points is created, where all new observations are first written. If, during clustering of this buffer, the cluster size exceeds the specified threshold, then the cluster is saved, and the measurements that fell into it are removed from the buffer. When there are saved clusters in the system, the new data points are first checked for cluster membership. A point belongs to a cluster if it has lesser Mahalonobis distance to its center than a given threshold. To calculate the Mahalonobis distance, a covariance matrix is used, that built from the members of the given cluster. When a new center point is added, the covariance matrix is updated using the moving average method. In addition to information about the location of the object and its spatial distribution, the system is able to determine the parameters of the circumscribed cylinder of the object. These parameters are the width and height of the bounding rectangle (issued by the recognition system) converted into real coordinates. Size of the cylinder is used to approximate size of object that can be used by the semiotic system and to validate the stored clusters. Due to the uneven movement of the robot, this method does sometimes create copies of single object. Therefore, if two clusters after update have an intersection of the framing cylinders, then these clusters are merged. The general scheme of the process is shown in Fig. 5.

2.4 Types of User Interfaces Generation of robot commands from voice instructions given in a language close to natural is performed in several stages. The command text is recognized first. The analysis of the syntactic structure of the received sentence is carried out by an interpretation system of third-party developers, which forms a description of the sentence structure indicating the roles of its components. Finally, at the last stage, the characteristics essential for the instruction are isolated from this description and

Robotic Wheelchair Control System for Multimodal Interfaces …

173

Localized object

Comparison with known clusters

Buffer of free points

Clustering

Cluster size check

Known clusters

Validation

Fig. 5 General scheme of the object mapping process

then grouped to obtain a complete command. The selection of features is carried out by a separate semiotic subsystem, the inference rules of which determine the correspondence of the syntactic structures of a sentence in natural language and parts of a command. In the current version, these rules are set manually, which is a significant drawback of the system. If a natural language sentence allows more than one interpretation, command extraction is performed iteratively, until the first success. Despite the obvious disadvantages of this approach, it worked quite successfully for the chosen subclass of natural language sentences. To control the wheelchair with a eyes, the Tobii 4C eye tracker, fixed in front of the user, is used. It should be noted that in this case the user observes the environment directly, and not through the monitor, as has been done in some proposed gaze control interfaces. Currently, there are three control modes available. In the first mode, the area in front of the user is divided into several subareas, each of which is responsible for a specific command. There are 7 commands in total: forward, backward, left, right, right, left and stop. When the user’s eye hits one of the areas (Fig. 6), the corresponding command is executed. This mode is most conveniently used with some command confirmation device, which can be either tactile, voice, or EEG, for severely injured patients. Although this mode is quick to learn and intuitive, it lacks the ability to fine tune the movement velocity. The second mode allows user to adjust the speed of movement by gaze position; accordingly, the higher the gaze, the higher the linear velocity—and vice versa, the stronger the gaze is taken to the side, the higher the angular velocity in this direction

174

P. S. Sorokoumov et al. 1

2

3

4

5

6

7

Fig. 6 The area in front of the user

(Fig. 7). Despite the precise control of the wheelchair movement, this mode requires constant concentration on the control task and is not so easy to learn. The third mode is high-level control, which allows operators to select a point in space with their gaze; then the wheelchair autonomously reaches the chosen point. However, in this mode, users can move only to the point that is in theirs field of view and there is no possibility to choose the orientation of the wheelchair at the final point. Moreover, this mode also requires a confirmation action. Therefore, a system was developed when all three modes are available to the user at once, and switching between them and confirmation of commands occurs using voice.

Fig. 7 Wheelchair speed depending on gaze point in second mode

Robotic Wheelchair Control System for Multimodal Interfaces …

175

One of the components of the wheelchair control system is the voice subsystem. It consists of hardware (acoustic system, microphone and wireless headset) and a set of ROS nodes. A variety of on-board input and output devices allows to customize the system for the operator. The microphone of the web camera installed above the eyetracker is used as an onboard microphone; acoustic system is stereo speakers; the headset with advanced bone conduction technology is used. The voice subsystem includes two main components—a text generation unit and a text recognition unit. The generated text messages are used, for example, to report the current control mode. Audio generation is provided by the open source synthesizer rhvoice via the speech dispatcher interface. The OCR node is used as one of the sources of commands or their validation. A part of the open-source CMUSphinx speech recognition toolkit, pocketsphinx, is used for recognition. This toolkit works without connecting to the global network, which is a serious advantage over most of the solutions used. At the moment, a user dictionary is used for recognition—a limited set of words with phonetics that can be recognized by the subsystem. The dictionary is easily expandable, and the limitation of its size is set to increase the reaction speed of the subsystem: the larger the dictionary, the more time it takes to recognize. The grammar of recognizing phrases is described in JSGF (Java Speech Grammar Format). The grammar rules are loaded in ROS recognition node, afterwards are used to recognize phrases in input audio stream.

2.5 Manipulator It is planned to install an automated robotic manipulator on the wheelchair, suitable for opening doors without human intervention. At the moment, it is being tested on the software model of wheelchair only. The task facing this manipulator is complicated by several factors at once compared to other common types of manipulations: • the shape of the manipulation objects (door handles) and their surface differ in a wide range; • the type of grip required for a stable movements and the opening patterns differ significantly (Fig. 8); in particular, even the gripping point can be located further or closer to the pivot axis of the door handle and the door itself, which complicates handling; • handles can be located both on the left and on the right side of the door; the door can be opened both by pushing and pulling. In some cases even people run into difficulties with the direction of opening the door; • the door might be closed, so that an attempt to open it will be unsuccessful; in this case, it is necessary to protect the robot from damage due to excessive efforts; • parameters of some types of knobs (in particular, the moment of force and angle of rotation required for rotation) cannot be identified by visual signs; to determine them, it is necessary to perform some additional procedure for each new knob (for example, attempting to turn the knob with a fixed effort).

176

P. S. Sorokoumov et al.

Fig. 8 Three common types of door handles, differing in the type of manipulations required when opening: a gripping an oblong object, not sensitive to the place of touch; b the capture of an elongated object, sensitive to the place of touch, and rotation; c capture of the body and rotation

To solve this problem, several different approaches have been developed in the theory of robotic manipulators. When solving the manipulation problem, several hierarchically ordered solution stages can be distinguished [8]: 1. 2.

3.

Formulation of the goal of manipulation, for example, “open the door”; Highlighting the stages of manipulation necessary to achieve the goal: “move the grip of the manipulator from the current position to the door handle”, then “grip the handle”, then “turn the grip”, etc.; For each stage: a. b.

c.

Determination of the trajectory of the manipulator, for example, in the form of a sequence of traversed points or line segments; Adding it to a state that allows precise formulation of commands for the manipulator, for example, building smooth transitions from segment to segment; Sending created commands to the manipulator, sometimes with correction of its behavior according to the data of sensor devices.

It can be seen that in order to achieve the goal, it is necessary to solve the tasks of stages from 2 to 3b. Tasks of a higher hierarchical level are solved by a semiotic subsystem, a lower one—by ready-made tools for solving inverse kinematics problems, for example MoveIt! or RoBoy. At the same time, it is highly desirable to obtain interpretable solutions, because otherwise it is not possible to generalize them with an extremely limited test sample. Linguistic methods meet the requirements for solving this problem. One of their classes is the creation of a language for defining movements [7, 9, 10] or (as development of the previous one) using the technology of modular movements [11]. The language for describing movements is a context-free grammar, the symbols of which are sets of actions on a controlled system at a given time; In this case, the language interpreter plays the role of an intermediary between the discrete language description of the system’s actions and continuous signals to executable devices. Let the controlled robot be modeled by a system of differential equations [12]:

Robotic Wheelchair Control System for Multimodal Interfaces …



177

x˙ = f(x) + G(x)u , y = h(x)

where x is a state vector, u is a vector of control actions, y is a vector of measured parameters, and f, G, h are functions connecting them. In this case, the control system can generate signals that depend on the measured values only: u = {ui (t, y(t))} Then the terminal symbols of the language for describing movements are triples of the form: σ = {U, ξ, T}, where U is the function that sets the control actions of the given symbol, T is the start time of symbol processing, ξ is the interrupt function, i.e. a y-dependent predicate that determines the end of execution of the given symbol. In other words, the control process is described as the execution of a sequence of characters: t < T1 : x˙ = f(x) + G(x)u1 , σ1 = {u1 , ξ1 , T0 }, ξ1 (t < T1 ) = 0, ξ1 (T1 ) = 1 t < T2 : x˙ = f(x) + G(x)u2 , σ2 = {u2 , ξ2 , T1 }, ξ2 (t < T2 ) = 0, ξ1 (T2 ) = 1 ... In this case, only one symbol is executed at a time, and the duration of its processing depends on the corresponding function ξ. Obviously, it is possible to combine two sequentially processed symbols into one, for which U is equal to U1 on the initial segment of execution and U2 on the final one, and ξ is calculated on the initial segment according to the rules for ξ1 , and after ξ1 becomes true for the first time—according to the rules for ξ2 . In other words, such characters can form strings; accordingly, we can talk about the description of the language they form, called the motion definition language (MDL). Its alphabet in the simplest case is finite and consists of a set of symbols σ, but after the introduction of the scaling operation: (α, β)σ = {αU, ξ, βT} Infinite alphabets (extended MDL, EMDL) also can be used. The grammars that describe these languages are context-free. The original publications and subsequent works describe numerous examples of applications of this approach both to manipulations and to other control problems. The advantages of this approach include the versatility of presentation and ease of implementation on a specifically selected system. Separate parts of the trajectories in the state space are represented by separate small complexes of changes in several robot mates at once, which can be combined according to certain laws, resulting

178

P. S. Sorokoumov et al.

in integral movements. The character sets can be quite limited if they describe the sequential stages of the processes characteristic of the manipulator, independently of each other. The disadvantages of this approach include the need for manual assignment of grammar in relation to a specific task, because it seems to be difficult to automatically generate adequate inference rules for symbols described in a similar form; translating characters from representing one manipulator to another is also non-trivial. Another approach, also using linguistic abstraction, but less rigorous, is formulated in [13]. Manipulations with objects are represented by a set of diverse actions, consisting of functions f i , each of which, according to the initial x 0 and final x 1 states of the robot, determines its trajectory leading from the initial state to the final one: DAi = fi (x0 , x1 , t0 ) = {q(t)|t0 < t < t1 , q(t0 ) = x0 , q(t1 ) = x1 } where q is the vector of the robot’s state, t 0 is the start time of the trajectory, t 1 is the end time. In this case, the domain of definition of each of the heterogeneous actions is not all possible states of the robot, but only those satisfying some a priori constraints, described formally. The author uses the following actions as a set of heterogeneous actions suitable for describing typical manipulation tasks (not only for a stationary manipulator, but also for one installed on a mobile platform): • • • • •

transit—movement of the robot without load; transfer—moving with the held object; pushing an object with a closed manipulator; approach/retreat of the manipulator for gripping or releasing an object; pick up an object lying on the floor with a manipulator.

Further, if we consider the trajectory as a function that transfers the robot from the initial state to the final state, then we can pose the problem of finding a path from the initial state to the final state, and each stage of the path will correspond to one of the actions. For each of the primitives of this method (DAMA—diverse action manipulation algorithm) with a known robot, the domains and the method for trajectory tracing can be calculated, that is, in principle, they are determined. However, in these calculations, in addition to the parameters of robots, it is inevitably necessary to take into account the properties of manipulated objects. The limitation of this concept is indicated, for example, by the fact that a plate lying on the floor will be captured by the pickup action, and the one standing on the table—by approach; perhaps the pickup action was introduced due to the difficulties in solving this particular problem by the approach action. Although this concept is of great interest in general, its applicability is therefore limited. In the task at hand, it turned out to be expedient to use a combination of the two indicated approaches. Opening the three main types of door handles shown in Fig. 8 has been described as combinations of dissimilar actions. For example, for the second of the grips shown, first the action “grip the elongated object” is performed, then the

Robotic Wheelchair Control System for Multimodal Interfaces …

179

action “rotate the object”. Moreover, each of the heterogeneous actions is described as an MDL symbol, i.e. in the form of a set of actions on the robot’s motors that provide the desired movement. The advantages of such a solution are the system’s learnability and the preservation of the interpretability of actions; disadvantages—the need for an extensive system of heterogeneous actions. However, it can be expected that due to the nature of the problem solved by the manipulator, the growth of the system required for normal operation under real conditions will be insignificant.

3 Testing A large room with an area circa 250 m2 is used as a test environment (Fig. 9). The room contains various objects—doors, tables, plants, columns, posters. Several voice commands are defined as test tasks, some of them contain ambiguities: – “go to the door”; – “go to the door on the left”; – “go to the door behind the table near the poster”, The system was tested both on a real robotic wheelchair (Fig. 10) and on its software model in the Gazebo environment. The launch was carried out in a hall, some objects of which the robot can recognize. To reduce the impact of recognition errors on objects, ArUco markers with the designation of their types were attached. The representation of the map of the resulting hall with the marks of the detected objects is shown in Fig. 11.

Fig. 9 Test environment (model in Gazebo)

180

P. S. Sorokoumov et al.

Fig. 10 Test wheelchair

Fig. 11 Semantic map of obstacles in the real test room visualized using rviz; cylinders represent recognized objects

Robotic Wheelchair Control System for Multimodal Interfaces …

181

To check the operability of the control system, high-level commands were used to force the wheelchair to move either in a given direction or to stated objects.

4 Results and Discussion The proposed solutions were implemented as ROS services and integrated into the existing robot control system. Model experiments were carried out to study the performance of the heuristic algorithm for obtaining obstacles. A fixed number of obstacles were modeled by two-dimensional Gaussian distributions, normalized to a maximum value of 1. The standard deviation was chosen uniformly from the range 0.5–1 m, obstacles were evenly distributed over a square area of 10 × 10 m within 20 work cycles. The graph of the dependence of the average execution time of one cycle on the number of obstacles on each of them is shown in Fig. 12. It can be seen that the growth of the processing time lags behind the expected quadratic, apparently due to the overlap of some of the obstacles on top of each other. The Gaussians were used to model the obstacle confidence distributions. The module for working with the manipulator when opening the door was implemented as a service of the ROS system. The compiled grammars turned out to be able to correctly generate test sets of heterogeneous actions (a capture action and two turning actions), which indicates the applicability of the proposed control principle. The correctness of the work of the semiotic control system was checked by issuing commands in the RDF format with subsequent control of their implementation.

Fig. 12 Dependence of the processing time of a set of obstacles on its size

182

P. S. Sorokoumov et al.

Fig. 13 Reaction of the robot model to the command “Drive to poster behind the table”, on the left initial state is shown, on the right the chosen target and path to it

Fig. 14 Reaction of the robot model to the command “Drive to the door to your left”, on the left initial state is shown, on the right the chosen target and path to it

The system executed all commands of the test sample correctly; in the absence of interpretation options, no action was taken. Figures 13 and 14 show the correct reactions of the robot to the issued commands—the wheelchair paved the way to the desired object and began to move.

5 Conclusion This paper proposes extensions and additions for the previously described semiotic wheelchair control system, which made it possible to ensure the movement of the wheelchair in indoor environments, as well as extension of the set of signs of the semiotic system. The developed control system is based on the semiotic model of the world and provides an acceptable level of interpretability of operations. It is enhanced with an object recognition and mapping module, which allow for easy control of the wheelchair by an operator using speech and gaze interfaces. The common problems of these slow or inaccurate high-level interfaces are mitigated by a high degree of autonomy of the wheelchair that has a navigation and safety systems (reflexes module) to increase safety of the operator. The semiotic model can also support

Robotic Wheelchair Control System for Multimodal Interfaces …

183

extensions to the wheelchair system, like a manipulator. The developed system was tested on a Gazebo environment and on the corresponding hardware of a real robotic wheelchair. Some test results are shown in the video www.youtube.com/watch?v= fXkSfqILkaM. Acknowledgements The work was supported by NRC “Kurchatov institute”, order № 1361 from 25.06.2019.

References 1. World Health Organization, World Bank. World report on disability (2011) 2. Osipov, G.S., et al.: Semiotic world representation for subject of behaviour, p. 264. Fizmatlit, Moscow (2018). in Russian 3. Karpov, V.E., et al.: Architecture of a wheelchair control system for disabled people: towards multifunctional robotic solution with neurobiological interfaces. Sovrem Tehnol. v Med. 11(1), 90 (2019) 4. Rovbo, M., Moscowsky, A., Sorokoumov, P.: Hierarchical control architecture for a learning robot based on heterogenic behaviors. Commun. Comput. Inf. Sci. 1093 (2019) 5. Fox, D., Burgard, W., Thrun, S.: The dynamic window approach to collision avoidance. IEEE Robot. Autom. Mag. 4(1), 23–33 (1997) 6. Rösmann, C., Hoffmann, F., Bertram, T.: Integrated online trajectory planning and optimization in distinctive topologies. Rob. Auton. Syst. 88, 142–153 (2017) 7. Liu, Z., et al.: Motion description language for trajectory generation of a robot manipulator (2017) 8. Zhao, R.: Trajectory planning and control for robot manipulations. Université Paul Sabatier, Toulouse III, (2015) 9. Brockett, R.W.: Hybrid models for motion control systems BT—essays on control: perspectives in the theory and its applications. In: Trentelman, H.L., Willems, J.C. (eds.), pp. 29–53. Birkhäuser Boston, Boston, MA (1993) 10. Brockett, R.: Language driven hybrid systems. Proceedings of 1994 33rd IEEE Conference on Decision and Control. 4, 4210–4214 (1994) 11. Liu, Z., et al.: A new type of industrial robot trajectory generation component based on motion modularity technology. J. Robot, 2020 (2020) 12. Manikonda, V., Krishnaprasad, P., Hendler, J.: Languages, pp. 199–226. Hybrid architectures and motion control, Math. Control Theory Behav. (1998) 13. Barry, J.L.: Manipulation with diverse actions. Massachusetts Institute of Technology. Massachusetts Institute of Technology, p. 201 (2013)

Methods and Principles of Information Processing

Classification of Images in Decision Making in the Central Nervous System of SEMS Andrey E. Gorodetskiy, Irina L. Tarasova, and Vugar G. Kurbanov

Abstract Problem statement: one of the important problems in modern intelligent robots, equipped with the Central nervous system (CNS), is the problem of forming images based on the analysis of sensory data. To solve this problem, it is necessary to build a classification model, on the basis of which, by logical analysis of the found regularities, the objects considered in the CNS can be attributed to a class. Purpose of research: mathematical formulation of the problem of forming images in the Central nervous system robot, analysis of the decisive rules for assigning images to a particular class of images and the construction of simple logical-probabilistic and logical-linguistic classification algorithms. Results: the mathematical formulation of the problem of forming images in the Central nervous system robot is formulated. In this problem statement, a set of images in the form of ordered sets with elements in the form of logical variables that take the value 0 or 1 is formed based on logical and mathematical analysis of sensory data about the robot’s selection environment. Each element of such sets is characterized by a set of features (attributes), whose values can be numeric, logical, or symbolic. Then these sets are compared in pairs with similar sets of reference images stored in the CNS database, and using the decision rules, the presented images are assigned to a particular class of images. Algorithms for constructing decision rules are analyzed and logical-probabilistic and logical-linguistic algorithms for implementing decision rules for image classification are obtained. Practical significance: The proposed principles of image formation in the CNS robot as a result of the analysis of sensory data characterized by a set of attributes with a certain degree of confidence, which can be set in the form of probability values or membership functions, and the resulting logical-probabilistic A. E. Gorodetskiy (B) · I. L. Tarasova · V. G. Kurbanov Institute for Problems in Mechanical Engineering of the Russian Academy of Sciences (IPME RAS), V.O. Bolshoj Pr., 61, 199178 St. Petersburg, Russia e-mail: [email protected] I. L. Tarasova e-mail: [email protected] V. G. Kurbanov e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_15

187

188

A. E. Gorodetskiy et al.

and logical-linguistic algorithms for their classification can be used in intelligent robots when processing incomplete and not completely reliable information. Keywords Smart electromechanical systems · SEMS · Selection environment · Sets · Images · Logical decision rules · Decision making · Classification · Sensory data

1 Introduction One of the important problems in modern intelligent robots equipped with a Central Nervous System (CNS) [1] is the problem of inductive formation of images, concepts, or generalizations based on the analysis of sensory data. It is associated with the search for logical patterns based on the construction of rules that can explain the facts and predict new or missing inherent in the desired or formed image. Therefore, the purpose of the analysis is to build a set of logical connections inherent in the image, i.e. building a classification model. When the classification model is built, on the basis of the found patterns, we can attribute the objects considered in the central nervous system to some class [2].

2 Statement of the Problem of Inductive Formation of Images In this case, the task of forming images from images can be as follows. Based on the analysis of sensory data on the environment of the choice of the robot, a set of images S = {S 1 , S 2 , …, S n } is formed. The robot database contains many reference images (objects) O = {O1 , O2 , …, Om }. The indicated sets are ordered sets, and the elements of these sets are Logical Variables (LV) that take the value 0 or 1. Each image S i and each object Oj are characterized by sets of signs (attributes). The number of attributes is fixed, attribute values can be numeric, logical, symbolic. Among the set O of all objects represented in the database, let us single out the subset V 1 ⊂ O - the set of objects related to concept 1 (class 1). Then the remaining set of objects W 1 = O\V 1 will not be related to the concept (class). Moreover, O = V1 ∪ W1 , V1 ∩ W1 = ∅ At the first step, among the set of images S, we select the set of images SV1 ⊂ S, belonging to the class V 1 and assign them the name of this class. Then, among the set O, we select V 2 ⊂ W 1 - the set of objects related to concept 2 (class 2) and among the set of remaining images S1 = S\SV1 we select the images SV2 ⊂ S1 belonging to the class V 2 . Give them the name of this class. Further we will continue the process of such selection until all database objects have been exhausted, i.e. ((((O\V1 )\V2 ) . . .)\Vk ) = Wk = ∅. If it is so that all database objects are exhausted,

Classification of Images in Decision Making …

189

but the remaining set of images is ((((S\SV1 )\SV2 ) . . .)\SVVk = Sk = ∅, then these images are assigned the name of the new class (k + 1) and they are temporarily stored in a database with this name. There can be many such images. In this case, the task of dividing to class (k + 1) into new reference images and, accordingly, assigning these images to newly introduced standards can be posed. By analogy with [2], to extract the q-th class from images, we can construct a training set K q = K q + ∪ K q − , where K q + ⊂ Vq and K q − ⊂ Wq . Based on the training sample K q , a rule is constructed that separates the positive and negative objects of the training sample. The decisive rule is correct if it subsequently successfully recognizes objects that were not originally included in the training set.

3 Algorithms for the Formation of Decision Rules A number of algorithms are known that form decision rules in the form of a decision tree, or a set of production rules. First of all, these are the ID3 and C4.5 algorithms [3, 4], the CN2 algorithm [5] and a number of others. The result of these algorithms are generalized concepts presented in the form of decision trees or a set of production rules. A Decision tree is a tree that associates an output value with each input example, while constructing a path from the root vertex to one of the final vertices. At each of the intermediate vertices (nodes), conditions are checked, and the final vertices (leaves) are indicated by the names of the solution (usually this is the name of the class to which the example belongs). It is necessary that at each intermediate vertex the results of the verification of conditions be mutually exclusive and exhaustive. The choice of a sequence of checks of conditions when moving from the root to the leaves of the tree in [3, 4] is determined by criteria that are associated with the concept of entropy. One of the problems for the above algorithms is the complexity that occurs when processing incomplete and conflicting information. In [6], the effect of noise in the initial data on the efficiency of classification models obtained using generalization algorithms was studied. The most difficult version of the noise in the database tables was considered, related to the presence of contradictions in the training set, and it was shown that the most significant influence on the decrease in classification accuracy is provided by the presence in the training set, on the basis of which the decision tree is constructed, contradictory examples, i.e., examples, attributed to different classes, with the complete coincidence of informative attributes. The UD3 (Uncertain Decision Tree) algorithm developed by S. Fakhrahmad and S. Jafari [7] is also analyzed there. This algorithm is based on the ideas of the ID3 algorithm. However, the construction of the decision tree and the subsequent classification of test cases here are associated with the use of an additional technique based on the comparison of bit strings associated with the attributes of the training sample. The main goal here is to provide the right solution when classifying examples in case the data set contains conflicting examples.

190

A. E. Gorodetskiy et al.

The presence of conflicting examples in the training set K q can lead to the fact that during the construction of the decision tree by the UD3 algorithm, all informative attributes are already used for checks, but it is impossible to assign the name of a certain class to the sheet. Such a decision tree may contain rules that give an ambiguous classification, that is, it will be fuzzy [8]. Therefore, an additional method for resolving contradictions at the classification stage, described in [6], is needed. The results of studies and experiments presented in [6] showed the advantages of the UD3 algorithm for reducing the influence of noise of the considered type in comparison with the ID3 and C4.5 algorithms. In particular, at a noise level of up to 25%, the classification accuracy and for various training samples, the classification accuracy according to the UD3 algorithm ranged from 78% to 85%, and according to the C4.5 algorithm, it ranged from 75% to 80%.

4 Logical-Probabilistic and Logical-Linguistic Algorithms Typically, images S i formed in the central nervous system of a robot as a result of analysis of sensory data are characterized by a set of k-th attributes with a certain amount of confidence, which can be specified in the form of probability PSik , or membership function μ Sik [9, 10]. In this case, the following algorithms can be used to assign the presented image S i to any class, the reference image Oi from the database. The logical-probabilistic algorithm LP1 consists in the following sequence of actions: All attributes of the reference images of the database are numbered and written as a string, consisting of N attributes. For each j-th reference image, we write a string of attribute probabilities in such a way that the string has dimension N and 1 is put in the place of the attribute that is in this standard, otherwise 0. For example, POik = (000111 · · · 110). For each i-th presented (classified) image, we write a string of attribute probabilities in such a way that the string has dimension N and in place of the attribute that is in the image, its probability value PSik is set, otherwise 0. For example, PSik = (0PSi2 0PSi4 PSi5 PSi6 · · · PSik · · · 00). For each presented (classified) image and each reference image, we calculate the differences  in the values of the elements of their rows and sum these differences Sij = (POij − PSik )2 . k

1. 2.

We calculate for each i-th (i = 1,2, …, n) images min Sij according to j-th standards (j = 1,2,…, m). Providing a minimum Sij j-th standard is the desired standard corresponding to the i-th (classified) image.

Classification of Images in Decision Making …

191

The improved logical and probabilistic algorithm LP2 differs from the previous LP1 in that it takes into account the importance of an attribute by multiplying the probabilities PSik of the image attributes by significance coefficients g Sik > 0, for example: PSi = (0g Si2 PSi2 0g Si4 PSi4 g Si5 PSi5 g Si6 PSi6 · · · g Sik PSik · · · 00). Accordingly, in the probability line of the attributes of the standards, it is necessary to place the same significance factors g Sik in place 1, for example: PO j = (000g Si4 g Si5 g Si6 · · · g Sik · · · 00). The logical-linguistic algorithm LL1 differs from the LP1 algorithm in that instead of the probabilities of image attributes, it is necessary to set the membership functions μ Sik in it, for example: PSi = (0μ Si2 0μ Si4 μ Si5 μ Si6 · · · μ Sik · · · 00). The improved logical-linguistic algorithm LL2 differs from the LL1 algorithm in that it takes into account the importance of an attribute by multiplying the membership functions of µsik image attributes by significance coefficients g Sik > 0, for example: PSi = (0g Si2 μ Si2 0g Si4 μ Si4 g Si5 μ Si5 g Si6 μ Si6 · · · g Sik μ Sik · · · 00). Accordingly, in the probability line of the attributes of the standards, it is necessary to place the same significance factors gsik in place 1, for example: PO j = (000g Si4 g Si5 g Si6 · · · g Sik · · · 00).

5 Testing Algorithms A study of the ability of the LP1, LP2, LL1, LL2 algorithms to work with conflicting information was carried out by computer modeling of image classification using the software package MatLab system. Machine experiments were carried out to assess the ability of the developed algorithms to classify images in the presence of contradictions and noise in the data on their attributes. The following types of images were analyzed: cars (x 1 ), people (x 2 ), large animals (x 3 ). Images of cars x 1 were characterized by the presence of the following attributes: 1. 2. 3. 4. 5.

The smell of gasoline (x 11 ), which was divided into three gradations: weak (x 111 ), moderate (x 112 ), strong (x 113 ). Temperature of the motor (x 12 ), which was divided into three gradations: low (x 121 ), average (x 122 ), big (x 123 ). Clearance (x 13 ), which was divided into three gradations: small (x 131 ), average (x 132 ), big (x 133 ). The ratio of the lengths of the trunk-interior-motor (x 14 ), which was divided into four gradations: 1: 2: 1 is x 141 , 1: 3: 1 is x 142 , 0.5: 3: 1 is x 143 , 2: 1: 1 is x 144. Sound level x 15 , which was divided into three gradations: quiet x 151 , average x 152 , strong x 153 .

The classification of car images was carried out according to the following types: jeep X 1d , crossover X 1k , sedan X 1s , hatchback X 1x , pickup X 1p . Moreover, in the

192

A. E. Gorodetskiy et al.

reference attribute sets (strings) of these types of vehicles, the presence of an attribute was designated 1, and the absence of 0. Therefore, the following reference strings were used in the classification: – – – – –

for the jeep: X 1d ⇒ {0 1 0 0 1 0 0 0 1 0 1 0 0 0 1 0}; for the crossover: X 1k ⇒ {1 0 0 1 0 0 0 1 0 0 1 0 0 0 1 0}; for a sedan: X 1s ⇒ {1 0 0 1 0 0 1 0 0 1 0 0 0 1 0 0}; for the hatchback:X 1x ⇒ {1 0 0 1 0 0 1 0 0 0 0 1 0 1 0 0}; for pickup: X 1 p ⇒ {0 0 1 0 0 1 0 1 0 0 0 0 1 0 0 1}. Images of people x 2 were characterized by the presence of the following attributes:

1. 2. 3. 4. 5.

Height x 21 , which was divided into three gradations: small x 211 , average x 212 , large x 213 . Wrinkles x 22 , which are divided into three gradations: no wrinkles x 221 , few wrinkles x 222 , a lot of wrinkles x 223 . The sound of steps x 23 , which are divided into three gradations: quiet x 231 , average x 232 , strong x 233 . Walking speed x 24 , which was divided into three gradations: low x 241 , average x 242 , big x 243 . The ratio of the width of the shoulders to the width of the hips x 24 , which was divided into five gradations: – the width of the shoulders is much greater than the width of the hips x 251 , – shoulder width greater than hips x 252 , – the width of the shoulders is approximately equal to the width of the hips x 253 , – shoulder width less than hips x 254 , – shoulder width much less than hips x 255 .

6.

Temperature x 26 , which was divided into three gradations: low x 261 , normal x 262 , big x 263 .

The classification of images of people was carried out according to the following types: man X 2m , old man X 2om , child X 2c , old woman X 2ow and woman X 2w. As in the previous case, in the reference attribute sets (strings) of these types of people, the presence of one attribute or another was designated 1, and the absence 0. Therefore, when the classification used the following reference strings: – – – – –

for man: X 2m ⇒ {001 010 001 001 10000 010}; for the old man: X 2om ⇒ {010 001 100 100 01000 010}; for a child: X 2c ⇒ {100 100 010 010 00100 010}; for the old woman: X 2ow ⇒ {010 001 100 100 00010 010}; for woman: X 2w ⇒ {010 010 010 001 00001 010}.

Images of large animals x 3 were characterized by the presence of the following attributes:

Classification of Images in Decision Making …

1. 2. 3. 4. 5. 6.

193

The smell of x 31 , which was divided into four gradations: cow smell x 311 , the smell of a horse x 312 , smell of deer x 313 , smell of ram x 314 . Height x 32 , which was divided into three gradations: small x 321 , average x 322 , large x 323 . Horns x 33 , which were divided into four gradations: no horns x 331 , small x 332 , average x 333 , large x 334. Wool x 34 , which was divided into two gradations: short x 341 , long x 342 . Tail x 35 , which was divided into three gradations: small x 351 , average x 352 , big x 353 . Temperature x 36 , which was divided into three gradations: low x 361 , normal x 362 , big x 363 .

The classification of images of large animals was carried out according to the following types: horse X 3h , cow X 3c , bull X 3b , deer X 3d , elk X 3e , ram X 3r , sheep X 3s . As in the previous case, in the reference attribute sets (strings) of these types of animals, the presence of one attribute or another was designated 1, and the absence of 0. Therefore, the following reference strings were used in the classification: – – – – – – –

a horse: X 3h ⇒ {010 000 110 001 000 1010}; the cow: X 3c ⇒ {100 001 000 101 000 1010}; the bull: X 3b ⇒ {1000 010 0001 10 001 010}; deer: X 3d ⇒ {0010 010 0001 10 010 010}; elk: X 3e ⇒ {0010 001 0001 10 001 010}; a ram: X 3r ⇒ {0001 100 0010 01 100 010}; sheep: X 3s ⇒ {0001 100 0100 01 100 010}.

To study the effect of noise on the classification of noisy images in the software package, we used a block for introducing noise into the data on probabilities or membership functions of attributes by introducing the probabilities or membership functions of the attributes of the presented images in the form of random variables with a uniform distribution principle. In this case, the following option was introduced for introducing noise into the data. For image attributes, to which random numbers in the range of 0.75–1 were generated in the attribute string of the reference, random numbers in the range of 0–0.25 were generated in image attributes to which random numbers in the range of attribute attributes of 0 were generated. An example of a generated random string of attributes of the image “ram”: (0.1 0.15 0.2 0.8 0.95 0.04 0.01 0.1 0.85 0.04 0.01 0.2 0.8 0.75 0.15 0.1 0.1 0.8 0.1) The main stages of the experiment were as follows: – – – –

selection and loading of standards with a set of attribute lines stored in the database, the choice of the type of noise introduced into the test set of images, the construction of test classification programs based on the studied algorithms, generating random strings of probabilities and membership functions of the checked image types,

194

A. E. Gorodetskiy et al.

– the use of test programs for the classification of checked samples of images according to the standards of the database. The results of test experiments showed approximately the same accuracy of the LP1 and LL1 algorithms above the accuracy given in [6] and obtained for the UD3 algorithm, which forms decision rules in the form of a decision tree or a set of production rules. The LP2 and LL2 algorithms, by introducing significance factors, better classify close images, such as cows and bulls or ram and sheep.

6 Conclusion The construction of a set of logical connections inherent in the image, i.e. the construction of the classification model allows, based on the found patterns (image attributes), to relate the objects considered in the central nervous system to any class. If the images formed in the central nervous system of the robot as a result of the analysis of sensory data are characterized by a set of attributes with a certain amount of confidence that can be specified in the form of probability or membership function, then the following algorithms can be used for their classification: logicalprobabilistic LP1 and LP2 or logical - linguistic LL1 and LL2. Computer experiments have shown the advantages of the introduced algorithms LP1, LL1, LP2 and LL2 in speed and classification accuracy compared with Quinlan algorithms ID3 and C4.5. If there are similar rows of attributes in the classified images, it is advisable to introduce attribute significance coefficients (LP2 and LL2 algorithms). Acknowledgements The present work was supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAA-A19119120290136-9 and is supported by grants RFBR No. 18-01-00076 and No. 19-08-00079.

References 1. Gorodetskiy, A. E., Kurbanov, V.G.: Smart Electromechanical Systems: the Central Nervous Systems. Springer International Publishing, 2017, p. 270. https://doi.org/10.1007/978-3-31953327-8 2. Vagin, V. N., Golovina, E. Y., Zagoryansky, A. A., Fomin, M. V.: Dostovernyi i pravdopodobnyi vyvod v intellektual’nykh sistemakh [Reliable and plausible conclusion in intelligent systems]. In: Vagin, V.N., Pospelov, D.A. (eds.) 2nd edn Revised and Enlarged, FIZMATLIT Publ., p. (2008). (in Russian) 3. Quinlan, J.R.: Induction of decision trees. Mach. Learn. 1(1), 81–106 (1986). https://doi.org/ 10.1007/BF00116251 4. Quinlan, J.R.: Improved use of continuous attributes in C 4.5. J. Artif. Intell. Res. 4(1), 77–90 (1996). https://doi.org/10.1613/jair.279

Classification of Images in Decision Making …

195

5. Clark, P., Niblett, T.: The CN2 induction algorithm. Mach. Learn. 3(4), 261–283 (1989). https:// doi.org/10.1007/BF00116835 6. Vagin, V. N., Krupetskov, A.V., Fomina, M. V.: An algorithm for constructing decision trees in the presence of discrepancies in the data. Seventeenth national conference on artificial intelligence with international participation KII-2019, vol. 2, pp. 182–191. Collection of scientific works, Ulyanovsk, Russia, October 21–25, 2019 7. Fakhrahmad, S. M., Jafari, S.: Uncertain decision tree inductive inference. Int. J. Electron. 98(10) (2008). https://doi.org/10.1080/00207217.2011.593138 8. Gorodetskiy, A. E., Tarasova, I. L.: Nechetkoye matematicheskoye modelirovaniye plokho formalizuyemykh protsessov i system. [Fuzzy Mathematical Modeling of Poorly Formalized Processes and Systems]. St. Petersburg, Polytechnic. University Publ., p. 336 (2010). (in Russian) 9. Gorodetsky, A. E., Tarasova, I. L.: Algebraic methods for obtaining and converting images in the technical diagnosis of complex systems under conditions of incomplete certainty. (Part 1). Informatsionno-upravlyayushchiye sistemy (Information and Control Systems). 5, 10–14 (2010). (in Russian) 10. Gorodetsky, A. E., Tarasova I. L.: Algebraic methods for obtaining and converting images in the technical diagnosis of complex systems under conditions of incomplete certainty. (Part 2). Informatsionno-upravlyayushchiye sistemy(Information and Control Systems). 6, 22–25 (2008). (in Russian)

Image Classification System in the SEMS Selection Environment Andrey E. Gorodetskiy and Irina L. Tarasova

Abstract Problem statement: currently, it is important to create image classification systems in the robot selection environment that allow using simple algorithms to quickly classify images in real time and make decisions in conditions of incomplete and not completely reliable information. Purpose of research: development of an image classification system in the SEMS selection environment with incomplete and not fully reliable information received from the technical vision system. Results: the structure and flowchart of the image classification system in the SIMS selection environment using simple logical-probabilistic and logical-linguistic decisionmaking algorithms in conditions of incomplete certainty is synthesized. The modes of operation of such a system are considered. Practical significance: The proposed classification system allows using simple logical-probabilistic and logical-linguistic algorithms for classifying images obtained in the sensory system of intelligent robots, when processing incomplete and not completely reliable information and making decisions about appropriate behavior. Keywords Smart electromechanical systems · SEMS · Selection environment · Images · Logical decision rules · Decision making · Classification · Images classification system

1 Introduction When creating the Central Nervous System of Robots (CNSR) based on SEMS modules [1], one of the most important problems is the problem of creating systems for making decisions about appropriate behavior [2] based on the analysis of the choice environment. The environment of choice in the CNSR is mainly characterized A. E. Gorodetskiy (B) · I. L. Tarasova Institute for Problems in Mechanical Engineering of the Russian Academy of Sciences, (IPME RAS), V.O, Bolshoj pr., 61, St. Petersburg 199178, Russia e-mail: [email protected] I. L. Tarasova e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_16

197

198

A. E. Gorodetskiy and I. L. Tarasova

by images formed on the basis of sensory data [3]. Therefore, it is currently important to create image classification systems in the robot selection environment that allow using simple algorithms to quickly classify images in real time and make decisions in conditions of incomplete and not completely reliable information. One of the variants of such a system can be a system that implements the simple logical-probabilistic LP1, LP2 and logical-linguistic LL1, LL2 image classification algorithms described in [4].

2 The Block Diagram of the System The classification system in question has a flowchart shown in Fig. 1. The system contains a control unit (1), database standards (2), sensory system CNSR (3), the units attribute string references (4) and classified images (5), line of cases (6) and membership functions (7) attribute references, the rows of probabilities (8) and membership functions (9) attributes of the classified images, the blocks of the row select references (10) and classified images (11), a database of coefficients of importance (12), blocks multiplication on the coefficients of importance of attributes references (13) and classified images (14), the difference calculation blocks (15) and difference sum blocks (16), the sum database (17), the minimum sum calculation block (18), and the image class designation block (19) corresponding to the minimum sum. Fig. 1 Classification system

6 2

13

10

4

15 7 11 16

8 3

14 17

5 9

1

12

18 19

Image Classification System in the SEMS Selection Environment

199

3 The Principle of Operation of the System Control unit 1 sends commands to unit 2 to select the first class of the standard of the classified image and to unit 3 to select the attributes of the image formed in the technical vision system [5]. Block 2 passes the attributes of the first reference to block 4, which forms a string of reference attributes, and then the probabilities of these attributes are passed to block 6, which forms a string of probabilities corresponding to the attribute string. The values of the membership functions for these attributes are sent to block 7, which generates a string of membership function values corresponding to the row of selected attributes. Block 3 passes the attributes of the classified image to block 5, which generates a string of attributes for this image of the same dimension as the reference string. Then the probabilities of these attributes are sent to block 8, which generates a probability string corresponding to the attribute string, or passes the membership functions of these attributes to block 9, which generates a string of membership functions corresponding to the string of selected attributes. Block 10 at the command of control unit 1 selects a probability string from block 6 and passes it to block 13, or a string of membership functions from block 7 and passes it to block 13. Block 11 at the command of control unit 1 selects a probability string from block 8 and passes it to block 14, or a string of membership functions from block 9 and passes it to block 14. Block 12, at the command of control unit 1, passes the attribute significance coefficients to blocks 13 and 14, where they are multiplied by the corresponding probabilities or membership functions. Blocks 13 and 14 pass lines of probabilities or membership functions to block 15, where the differences between the elements of the lines of the reference and the classified image are calculated. These differences are passed from block 15 to block 16, where the sum of differences for the first class of the standard is calculated. This amount is transferred from block 16 to block 17 for storage. Then control unit 1 sends a command to unit 2 to select the second class of the standard of the classified image, and then, similarly to the previous one, the sum of differences for the second class of the standard is calculated, which is also transmitted from unit 16 to unit 17 for storage. Then control unit 1 sends a command to unit 2 to select the next reference class of the classified image, and then, similarly to the previous one, the sum of differences for this reference class is calculated, which is also transmitted from unit 16 to unit 17 for storage. The calculation process continues until all the reference classes are exhausted. In this case, block 17 will contain the sum of differences for all classes of the standard. After that, control unit 1 sends a command to block 18 to determine the minimum of all stored amounts, and after the end of determining this amount, block 18 sends it to block 19, which determines the number of the reference class to which the classified image belongs. This is the end of the classification process for this image, and unit 19 sends a signal to control unit 1 that the system is ready to classify the next image.

200

A. E. Gorodetskiy and I. L. Tarasova

4 Conclusion The described classification system makes it possible to use simple algorithms LP1, LL1, LP2 and LL2 in the CNSR to classify images in the selection environment obtained in the CNSR sensor system. Using such a system makes it easier and faster for the robot to make an independent decision about appropriate behavior when processing incomplete and not completely reliable information. Acknowledgements The present work was supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAA-A19119120290136-9 and is supported by grants RFBR No. 18-01-00076 and No. 19-08-00079.

References 1. Gorodetskiy, A.E.: Smart electromechanical systems modules. In: Smart Electromechanical Systems./Studies in Systems, Decision and Control 49, pp. 7–15. Springer International Publishing, Berlin (2016). https://doi.org/10.1007/978-3-319-27547-5_2 2. Gorodetskiy, A.E.: The principles of situational control SEMS group. In: Smart Electromechanical Systems. Situational Control./Studies in Systems, Decision and Control 261, (pp. 3–13). Springer International Publishing, Berlin (2020). https://doi.org/10.1007/978-3-030-32710-1_1 3. Gorodetskiy, A.E., Tarasova I.L.: Decision making an autonomous robot based on matrix solution of systems of logical equations that describe the environment of choice for situational control. In: Smart Electromechanical Systems: Situational Control./Studies in Systems, Decision and Control 261, pp. 259–274. Springer International Publishing (2020). https://doi.org/10.1007/ 978-3-030-32710-1_20 4. Gorodetskiy, A.E., Tarasova, I.L., Kurbanov, V.G.: Classification of images in decision making in the Central nervous system of SEMS (in this volume) 5. Alpatov, B.A.: Babayan. Tekhnologii obrabotki i raspoznovaniya izobrazhenij v bortovyh sistemah tekhnicheskogo zreniya [Image processing and recognition technologies in on-board technical vision systems], Vestnik Rjazanskogo gosudarstvennogo radiotehnicheskogo universiteta/ Ryazan State Radio Engineering University Bulletin, 2 (60), 34–44 (2017). https://doi. org/10.21667/1995-4565-2017-60-2-34-44. (in Russian)

Principles of Forming the Language of Sensation for Decision Making in the Central Nervous System of SEMS Andrey E. Gorodetskiy, Irina L. Tarasova, and Vugar G. Kurbanov

Abstract Problem statement: at present, the development of modern robots is closely related to the creation of their sensory languages, on the basis of which a figurative representation of the environment and intellectual interaction of robots with each other is possible. The latter requires not only the solution of individual specific problems of processing and comprehending the sensory information of robots, but the creation of an integral algorithm that takes into account all the senses of the robot: organs of vision, hearing, smell, taste, touch, etc. Purpose of research: development of an algorithm for the formation of the language of sensations of the robot, containing the stages of quantizing the surrounding space in the visibility zone of the sensor system of the robot with the assignment of the resulting pixels names in the form of a pixel number; fuzzification of sensor information of the robot for each pixel of the surrounding space; forming in the memory the Central Nervous System of Robots (CNSR) the display of the surrounding space in the form of pixels with their coordinates and fuzzified data; forming images in the display of the surrounding space for each sense organ of the robot; forming images by combining images from different senses; assigning names to images in the form of words in the English language and recording them in the CNSR database. Results: an algorithm for the formation of the language of sensations of the robot is developed, using the logical and mathematical processing of the sensor data of the robot with their transformation into logical expressions mod 2 with the attributes of variables in the form of probabilities, membership functions and linguistic expressions. When using this algorithm, semantic data about the space surrounding the robot is generated in the CNSR database, on the basis of which the robot makes behavioral decisions using A. E. Gorodetskiy (B) · I. L. Tarasova · V. G. Kurbanov Institute for Problems in Mechanical Engineering of the Russian Academy of Sciences, (IPME RAS), V.O, Bolshoj pr., 61, St. Petersburg 199178, Russia e-mail: [email protected] I. L. Tarasova e-mail: [email protected] V. G. Kurbanov e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_17

201

202

A. E. Gorodetskiy et al.

typical behavioral algorithms stored in the robot’s knowledge base. Practical significance: The considered principles and algorithms for the formation of the language of sensations in the CNSR based on the use of systems of equations mod 2 and the results of the analysis of the features of pattern recognition described by systems of equations in algebra mod 2 can be effectively used in intelligent robots to form reflexive reasoning and make decisions about appropriate behavior. Keywords Smart electromechanical systems · SEMS · Environment of choice · Quantization · Fuzzification · Images · Algorithm · Language of sensations · Logical decision rules · Decision making · Central nervous system of a robot

1 Introduction The development of modern robots is closely connected with the creation of their languages of sensations, on the basis of which a figurative representation of the environment and the intellectual interaction of robots between themselves and with the human operator is possible. In this area, many developments are devoted to the control of robots in different conditions. For example, in [1, 2], a spoken language is proposed as a convenient interface (ELI—Extensible Language Interface) for controlling a mobile robot. It is designed to interpret speech commands to perform the tasks of extracting and transmitting information for use in specific, narrow tasks, such as caring for the elderly. In order to use it effectively, a number of basic terms must be associated with perception and motor skills. Therefore, at present there is a wide range of tasks for which the robot using the ELI cannot be pre-programmed. For example, such as the nature of specific tasks in the household that he may be asked to perform. In [3], an algorithm is proposed for teaching the robot to see various objects. Developed robotic vision systems are based on what animals are supposed to see as developers. That is, they use the concept of layers of neurons, as in the brain of animals. Engineers program the structure of the system, but do not develop an algorithm that works in this system. Since the 1970s, robotics engineers have been thinking about reducing information for displaying images in a computer’s memory, using the features of images. These can be lines or points of interest, such as angles or certain textures. Algorithms are created for finding these functions and tracking them from the image frame to the image frame in the video stream. This significantly reduces the amount of data from millions of pixels in an image to several hundred or thousands of objects. Then the engineers think about how the robot can realize what they saw and what it will need to do. They write software that recognizes patterns in images to help the robot understand what’s around it. It should be noted that certain specific tasks have been solved for the processing and comprehension of the sensory information of robots, but there is no integral algorithm that takes into account all the organs of the robot’s sense: organs of sight, hearing, smell, taste, touch, etc. no. Therefore, in order that intelligent robots could

Principles of Forming the Language of Sensation for Decision …

203

independently, go without human intervention, formulate tasks and successfully accomplish them, they must not only be equipped with sensation sensors (sensors), but also have the ability to understand the language of sensations, i.e., have sensations such as “yours is alien”, “dangerous - safe”, “beloved - unloved”, “pleasantly - unpleasantly”, etc., formed as a result of solving systems of logical equations describing the environment in the language of feelings. For this, it is possible to use logical inference systems, which in intellectual systems are associated with solving systems of logical equations [4]. They may have a higher dimension. The number of variables usually exceeds the number of equations, which leads to non-uniqueness of the solution. Using the Zhegalkin algebra [5] allows one to perform algebraization of the problem, so that the Euclidean norm can serve as a scalar measure of the quality of the solution. At the same time, to solve it, you can use a method similar to the Gauss elimination method when solving linear systems of algebraic equations with real numbers. This technique can be the basis for providing the robot with the ability to form a sensation language in the database of the “Central Nervous System of the Robot” (CNSR). In this case, the robot has the opportunity to make independent decisions regarding expedient behavior [6].

2 Algorithm of Formation of the Language of Sensations of the Robot The central nervous system of a robot is built by analogy with the central nervous system of a person who has sensory organs that perceive information about the environment and their own state. Therefore, the solution to the problem of creating a central nervous system of a robot is reduced, first of all, to research and development of circuits of the type of the following circuit (consisting of approximately seven blocks): 1—(robot sensors) → 2 (signal receiving channel), 3 —(primary processing of measuring signals) → 4 —(combining signals, fuzzification, recognition, classification, decision making) → 5 —(transmission channel of control signals), 6-— (transformation and formation of a control action) → 7 —(moving, stretching and other actions of the working parts of the robot). The description of all CNSR blocks is described in detail in [7]. It should be noted that one of the most promising options for the mathematical implementation of the fuzzification block is a logical-mathematical model for the formation of behavioral processes based on the analysis of sensations in the form of signals from the robot’s sensory system. To do this, the robot’s sensor system collects environmental information from various sensors and transmits it to the CNSR. Next, the measurement signal preprocessing unit and the fuzzification, recognition, and decision-making unit [8] of the CNSR processor form the robot’s sensory language using the following algorithm:

204

1.

2.

3. 4. 5. 6. 7.

8. 9.

A. E. Gorodetskiy et al.

Quantization of the surrounding space in the visibility zone of the robot’s sensor system with the assignment of the resulting pixels a value in the form of a pixel number. Fuzzification of the robot’s sensory information for each pixel of the surrounding space and the formation in the memory of the CNSR of the display of the surrounding space in the form of pixels with their coordinates and fused data. The formation of images in the display of the surrounding space for each sense organ of the robot. Formation of images by combining images from different senses. Assigning names to images in the form of words of the English language. Write words in the form of a combination of letters of the English language. If for any images there was no suitable word of the English language, then such images can be combined with others in various combinations until all possible combinations have been used. If any combination finds the appropriate words of the English language, then these names are assigned to these combinations of images. If after the completion of operation (8) any images cannot be found suitable words, such images are given a name in the form of a new word from a combination of English letters and the corresponding message is transmitted to the robot community to legitimize a new reference word and the corresponding image. English language and the corresponding message is transmitted to the community of robots to legitimize the new reference word and the corresponding image. Let us consider in more detail the basic operations of this algorithm.

2.1 Quantization of the Surrounding Space The center of gravity of the robot is placed in the center of the Euclidean space E 3 . The boundaries of the sensitivity zones (intervals) of the sensor system are determined: [−X, + X], [−Y, + Y ], [−Z, + Z]. The result is a three-dimensional subspace C ⊂ E3. This subspace is divided into quanta along the X axis with a step hx , along the Y axis with a step hy , and along the Z axis with a step hz . The quanta in [−X, + X], [−Y, + Y ], [−Z, + Z]) are assigned numbers i, j, k, respectively. As a result, the entire subspace C will be divided into many pixels pijk . Each pijk pixel will correspond to information measured by the CNSR sensor system on sensations of the organs of vision, hearing, smell, taste, touch, etc.

Principles of Forming the Language of Sensation for Decision …

205

2.2 Fuzzification of Sensory Information A very important operation for forming the language of sensations is fuzzification of sensory data assigned to pixels and recorded in the CNSR database. To do this, first of all, it is necessary to combine the sensory information of each pixel pijk into groups that form, like a person, the following senses of the robot: vision in the form of the set E; hearing in the form of the set R; sense of smell in the form of the set S; taste in the form of a set U; the sense of V. In each of the introduced sets, it is possible to distinguish the subsets forming them that characterize the properties of the observed pixel (object): E i ⊂ E, Ri ⊂ R, Si ⊂ S, Ui ⊂ U, Vi ⊂ V The set of such subsets depends on the set of sensors that form the sensory organs of a particular robot. For example, for vision, the following subsets can be introduced: E 1 —image brightness; E 2 —image color; E 3 —flashing frequency; E 4 —rate of change of brightness; E 5 —speed of color change, etc. For hearing, the following subsets can be introduced: R1 —sound power; R2 is the key; R3 is the interval; R4 —rate of change of volume; R5 —rate of change of tonality; R6 —interval change rate, etc. For the sense of smell, the following subsets can be introduced: S 1 —type of smell; S 2 —odor intensity; S 3 is the rate of rise or fall of the odor; S 4 is the rate of change of the type of smell; S 5 —odor interval, etc. For taste, the following subsets can be introduced: U 1 —type of taste; U 2 is the power of taste; U 3 —rate of change in taste, etc. For touch, the following subsets can be entered: V 1 —flatness of the surface; V 2 —dry surface; V 3 —surface temperature, etc. The data forming these subsets are extracted from signals from sensors of the senses of robots by their fuzzification [7]. This data can be of logical, logical-probabilistic or logical-linguistic types. Data of a logical type is formed from data or signals from sensors of the sense organs of robots by quantizing the entire range of a specific sensor and assigning n , (where n = 1,2,… N is the number of a quantum), the names of logical variables that take values: true (1) or false (0). For example, logical variables are formed by quantizing the entire range of the acoustic sensor and assigning the obtained n quanta to the names of logical variables that take the value true (1) or false (0). Then, if the range of the sound intensity sensor lies in the range from 0 dB to 80 dB, then by entering a quantum of 20 dB, you can divide the entire range of the change in sound intensity into four quanta 1 = [0, 20], 2 = [20, 40], 3 = [40, 60], 4 = [60, 80]. Then, the quantum 1 can be given the name Rr1 {very weak sound}, the quantum 2 should be given the name Rr2 {weak sound}, the quantum 3 should be given the name Rr3 {strong sound}, and the quantum 4 should be given the name Rr4 {very strong sound}. In particular, if, for example, the sensor shows the sound intensity r = 50 dB, then after fuzzification, the following values of the logical variables Rr1 =

206

A. E. Gorodetskiy et al.

0, Rr2 = 0, Rr3 = 1, Rr4 = 0 and the corresponding intervals described above will be entered into the CNSR database as attributes of these logical variables. When receiving data of a logical-probabilistic type, the probabilities P(r i ) are additionally added to the attributes, which can be determined under the normal law of the distribution of sound strength as follows: P(rn ) = 2((3) −((rn − m/σ )) where: a is the lower boundary of the quantum, b is the upper boundary of the quantum, m = (b-a)/2 is the expected value and σ = (b + a)/6 is the standard deviation,  (.)–Gaussian standard distribution function, which corresponds to the simplest normal law with parameters m = 0, σ = 1 and whose values are known. Naturally, for logical variables corresponding to quanta, which do not include sensor readings, the probabilities will be zero. In particular, if the sensor shows the sound strength r = 50 dB, then after fuzzification, the following values of the logical variables Rr1 = 0, Rr2 = 0, Rr3 = 1, Rr4 = 0 and the following attributes corresponding to them will be entered into the CNSR database: for Rr1 —interval [0; 20] and the probability P (r 1 ) = 0; for Rr 2 , the interval [20; 40] and the probability P (r 2 ) = 0; for Rr3 , the interval [40; 60] and the probability P (r 3 ) = 1; for Rr4 , the interval [60; 80] and P (r 4 ) = 0. It should be noted that in the formation of logical-probabilistic variables, the quantization of the sensor range can be carried out with overlap. For example, if the range of the sound intensity sensor lies in the range from 0 dB to 75 dB, then by entering a quantum value of 30 dB, you can break the entire range of sound strength into the following four quanta: [0; 30]; [15; 45]; [30; 60]; [45; 75]. Then, if the sound intensity sensor shows r = 50 dB, then after fuzzification, the following values of the logical variables Rr1 = 0, Rr2 = 0, Rr3 = 1, Rr4 = 1 and the following attributes corresponding to them will be entered into the CNSR database: for Rr1 —the interval [0; 30] and the probability P(r 1 ) = 0; for Rr2 , the interval [15; 45] and the probability P(r 2 ) = 0; for Rr3 , the interval [30; 60] and the probability P(r 3 ) = 0.12; for Rr4 , the interval [45; 75] and the probability P(r 4 ) = 0.12. When receiving data of a logical-probabilistic type and with a uniform law of the distribution of sound strength, the probabilities P(r n ) are additionally added to the attributes, which can be determined as follows:   2(b−rn ) , i f rn ≥ m b−a P(rn ) = 2(rn −a) , i f rn < m b−a In this case, for the above example, after fuzzification, the following values of the logical variables Rr1 = 0, Rr2 = 0, Rr3 = 1, Rr 4 = 1 and the following attributes corresponding to them will be entered into the CNSR database: for Rr1 , the interval [0; 30] and the probability P(r 1 ) = 0; for Rr2 , the interval [15; 45] and the probability P(r 2 ) = 0; for Rr3 , the interval [30; 60] and the probability P(r 3 ) = 0.25; for Rr4 , the interval [45; 75] and P(r4 ) = 0.25.

Principles of Forming the Language of Sensation for Decision …

207

When receiving data of a logical-linguistic type, the membership functions are additionally added to the attributes, which can be determined in the triangular form of the function as follows:   2(b−rn ) , i f rn ≥ a+b b−a 2 m(rn ) = 2(rn −a) , i f rn < a+b b−a 2 In this case, for the above example, after fuzzification, the following values of the logical variables Rr1 = 0, Rr2 = 0, Rr3 = 1, Rr4 = 1 and the following attributes corresponding to them will be entered into the CNSR database: for Rr1 , the interval [0; 30] and the value of the membership function μ(r 1 ) = 0; for Rr2 , the interval [15; 45] and the value of the membership function μ(r 2 ) = 0; for Rr3 , the interval [30; 60] and value of the membership function μ(r 3 ) = 0.25; for Rr4 , the interval [45; 75] and μ(r 4 ) = 0.25. Thus, after fuzzification of sensory data in the database for each pixel there will be a set of logical, logical-probabilistic and logical-linguistic variables. The next step in creating a robot sensation language will be the task of forming images in the surrounding space for each sensory organ.

2.3 Image Formation in the Display of the Surrounding Space The formation of images is performed for the display of the surrounding space for each sensory organ individually. In particular, for the view there will be a display of the surrounding space C E ⊂ C, for hearing C R ⊂ C, for the sense of smell C S ⊂ C; for the taste of C U ⊂ C and for the sense of C V ⊂ C. In each of these mappings, adjacent pixels with equal values of logical variables and close values of their attributes can be combined. Then in the spaces of the sense organs C E , C R , C S , C U and C V we get sets of images ImE , ImR , ImS , ImU , ImV with certain contours. Since attributes of logical variables can be intervals, probabilities, membership functions, etc., for each type of attribute it is necessary, accordingly, to introduce a measure of proximity δ  , δP , δμ. After the operation of combining pixels into image sets ImE (i), ImR (i), ImS (i), ImU (i), ImV (i), in each space of the sensory organs C E , C R , C S , C U and C V , one can depict image contours and give each circuit a name. As a result, there will be five cards K E , K R , K S , K U and K V with sets of image contours ImE (i), ImR (i), ImS (i), ImU (i), ImV (i), where i = 1,2,…. It should be noted that, when processing two images, a preliminary analysis is first performed (spectral and correlation analysis), which includes the selection and application of the most suitable filter (linear filtering), on the basis of which contour representations (polygonal contours) are formed.

208

A. E. Gorodetskiy et al.

2.4 Formation of Images by Combining Images from Different Senses Typically, the formation of images from images consists in operations on images such as intersection, union, or symmetric difference of ordered sets and assigning the result to one or another reference image stored in the database. If there is not a single suitable image in the database, then such a combination of images is given the name of the new image, which is placed in the database for temporary storage. When this new image is repeated many times during the operation of the robot, this image becomes a reference image and is given a permanent name. To assign a combination of images to a particular standard, it is necessary to introduce a measure of proximity of ordered sets. Among the most well-known proximity measures (criterion), the following can be distinguished [9]: estimation by the maximum deviation of cardinalities of sets; estimate of the standard deviation of the cardinality of sets; probabilistic assessment of the maximum deviation of cardinalities of sets; probabilistic estimate of the standard deviation of cardinalities of sets. Using these criteria allows you to rank combinations of images according to their proximity to the reference image and at the same time allows you to enter a numerical estimate of proximity. The process of forming images based on information from the senses of the robot is carried out in the following sequence. First, we look for the presence of images that are close to the database standards in each of the K E , K R , K S , K U , and K V cards. The found images are assigned the names of the standards. They are recorded in the observable data base of the central nervous system together with their coordinates and are excluded from the corresponding maps. Then a sequential pairwise overlay of cards is performed on each other with the operation of intersecting the sets K E , K R , K S , K U , and K V . Similarly, three (four, five) cards are superimposed on top of each other. At each intersection of images, the presence of images close to the database standards is searched for. The found intersections of the images are assigned the names of the standards. They are recorded in the observable data base of the central nervous system together with their coordinates and excluded from the corresponding map intersections. Therefore, in each subsequent intersection, the maps corrected by the results of deletions are involved. If any images appear in the corrected intersection, then new names are assigned to them. They are also recorded with their coordinates in the observational database of the central nervous system. At the last stage, in the symmetric differences of the sets from the maps K E , K R , K S , K U , and K V , the presence of images close to the standards in the databases is searched. Moreover, it is similar at first by the operation of the symmetric difference of two sets, then three, four and five. The found symmetrical differences of the images are assigned the names of the standards. They are recorded in the database of observable data of the Central nervous system together with their coordinates and

Principles of Forming the Language of Sensation for Decision …

209

are excluded from the respective map associations. Therefore, in each subsequent association, cards corrected by the results of deletions are involved. Thus, semantic data about the space surrounding the robot are generated in the central nervous system database, based on which the robot makes behavioral decisions [8] using standard behavioral algorithms stored in the robot knowledge base. These algorithms are recorded in the knowledge base of the robot at the stage of its creation, based on its purpose. Therefore, such algorithms will be called genetic. However, after the formation of the semantic database of the space surrounding the robot, it may turn out that two or more images are partially or completely present in the same place in space. Therefore, it is necessary to adjust the semantic database in order to exclude detected collisions. Such adjustment is closely related to the formation of semantic data—pragmatic, corresponding to the problem being solved by the robot at the moment.

3 Conclusion An important task for intelligent robots is to provide independent decision-making regarding appropriate behavior. The paper proposes an algorithm for the formation of the robot sensation language, which allows robots to be provided with the possibility of reflective and informed reasoning. For this purpose, the following procedures are proposed: quantization of the surrounding space, fuzzification of sensory information, image formation in the display of the surrounding space, image formation by combining images from different senses and assigning words to the generated language. Acknowledgements The present work was supported by the Ministry of Science and Higher Education within the framework of the Russian State Assignment under contract No. AAA-A19119120290136-9 and is supported by grants RFBR No. 18-01-00076 and No. 19-08-00079.

References 1. Connell, J.: In robots that talk and listen, In: Markowitz, J. (ed.), De Gruyter, 2014) 2. Connell J., E. Marcheret, S.M. Kudoh, Nishiyama. In: Proceeding. Artificial General Intelligence Conference (AGI-12), LNAI 7716, 21–30, December 2012 3. Theconversation.com/how-do-robots-see-the-world-51205 4. Kurbanov VG, Burakov MV (2018) Solving of logic functions systems using genetic algorithm. In: Proceedings of the II International Scientific and Practical Conference “Fuzzy Technologies in the Industry—FTI 2018”, 410–417. (2018). http://ceur-ws.org/Vol-2258/paper49.pdf 5. Zhegalk I.I.: Arifmetizatsiya simvolicheskoy logiki. [Arithmetization Symbolic Logic]. Matematicheskii sbornik [Mathematical Collection], 35:3–4, 335, (1928). (In Russian) 6. Dobryn D.A.: Intellectual robots yesterday, today, tomorrow. In: Proceedings of the X National Conference on Artificial Intelligence with International Participation KII-2006 Moscow, Fizmatlit Publish, vol. 2, pp. 20–32. (in Russian) (2006)

210

A. E. Gorodetskiy et al.

7. Gorodetskiy, A.E., Tarasova, I.L., Kurbanov, V.G.: Challenges related to development of central nervous system of a robot on the bases of SEMS modules. In: Gorodetskiy, A.E., Kurbanov, V.G. (eds.), Smart Electromechanical Systems: The Central Nervous System, Studies in Systems, Decision and Control vol 95, pp. 3–17. (2017). https://doi.org/10.1007/978-3-319-53327-8_1 8. Gorodetskiy, A.E., Kurbanov, V.G., Tarasova, I.L. Decision-making in central nervous system of a robot. Inf. Control Syst. 2018:(1):21–30. https://doi.org/10.15217/issnl684-8853.2018.1.21. (In Russian) 9. Gorodetskiy, A.E., Tarasova, I.L., Kurbanov, V.G.: Behavioral decisions of a robot based on solving of systems. In: Gorodetskiy, A.E., Kurbanov, V.G. (eds.), Smart Electromechanical Systems: The Central Nervous System, Studies in Systems. Decis. Control 95, pp. 61–71, 2017. https:// doi.org/10.1007/978-3-319-53327-8_5

Generation of Control Commands in the Group SEMS with Multi-Channel Optic-Electronic Sensors Igor A. Konyakhin and Minh Hoa Tong

Abstract Problem statement: nowadays the using of Smart electromechanical systems (SEMS) is effective for creating robotic systems, which have the ability to quickly generate the necessary control commands due to the presence of the Central nervous system (CNS). When managing the behavior of such groups, it is necessary, first of all, to monitor the environment and evaluate the group’s ability to make correct decisions according to the parameters of each SEMS. Using one SEMS in a group of optic-electronic sensors increases the reliability of the situation assessment and speeds up the generation of control commands. Such a SEMS becomes a coordinator whose commands have a high priority for execution by other SEMS. One of the most effective ways to generate correct decisions is the using a computer and physical simulation by the SEMS coordinator, whose parameters are determined by the environment influenced signals from the optic-electronic sensors. Purpose of research: the search for the optimal structure of the models for the coordinator which it is depended by various external situations and researching the decision-making process based on signals from the SEMS coordinator’s optic-electronic sensors. Methods: computer and physical models that are concluded by the coordinator are analyzed, on the one hand, as receivers of signals about changes in the external environment from multi-channel optic-electronic sensors, and on the other hand as data generators for controlling other SEMS in the group. Solutions are determined by modeling the changes in the external situation. Results: the SEMS group for aligning the relative positions of large objects is considered. The SEMS group is formed by line and parallel structures, and one of the SEMS is the group coordinator and it generates priority control commands for the other SEMS. To provide the SEMS coordinator features, physical and computer models were synthesized to simulate various types of noise and deviations of environmental parameters and determine the estimation of the position error of objects. It is shown that the use of a multi-channel opticelectronic sensors allows the SEMS coordinator to obtain reliable data for modeling I. A. Konyakhin (B) · M. H. Tong ITMO University, St. Petersburg, Russia e-mail: [email protected] M. H. Tong e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_18

211

212

I. A. Konyakhin and M. H. Tong

and generating control commands for the another SEMS in the group. A sequence of results which are generated by models and a strategy for their further using in the process of finding the optimal solution for situational management of a SEMS group with a coordinator is proposed. Practical significance: the research results can be used in situational control of a group of interacting SEMS, where a coordinator with a multi-channel optoelectronic sensor generates the priority control commands, for example, in the control of the system for adapting the surface shape of telescopes and radio telescopes with sectional reflecting elements, a group of robotic collectors with coordinator performing joint operations, etc. Keywords Smart electromechanical systems · Optic-electronic sensors · Situation modeling · Decision making process · Situational control · Group control

1 Introduction The principles of generating commands to control a group of robots are determined, in particular, by program of actions performed by the group, its structure, special features of the group’s robots, their artificial intelligence level. To carry out complex practical tasks which require performing coordinated actions by robots under changing environmental conditions, it is effectively to use robotic systems in the form of a group of Smart Electromechanical Systems (SEMS). Structure of the system which comprises tiers made up of serial and parallel circuits of SEMS devices, faces the task of optimizing the hierarchy of control commands by structural levels with commands generated by Head SEMS on upper levels while commands of SEMS controlled by them being on lower levels. However, the “ordinary” Slave SEMS at lower tiers control their built-in actuator components using a combination composed of the command from the “upper” Head SEMS and their own commands generated by built-in micro-processor [1–3]. If the robotic systems are designed to set a group of objects in a pre-defined position in space or to align their mutual location in the performance of operations for multi-point processing, assembly or monitoring of the complex structure status, in this case, the Head SEMS of the group contains an optic-electronic sensor [4, 5]. Optic-electronic sensor generates an image of the external space, analysis of which considerably increases adequacy of control commands to be established for Slave SEMS. To improve accuracy of control, the Head SEMS uses physical and computer models of optic and electronic components of the optic-electronic sensor. The simulation data ensure compensation for the influence of noises and interference in optic channels of the sensor. This article reviews the special features inherent in the functioning of the Head SEMS the structure of which includes multi-channel optic-electronic sensor to measure the position parameters of objects in space. As an example, control of the robotic system with the heterogeneous structure “Head SEMS—Slave SEMS” is analyzed, which is designed to align the position of the composite mirror parts in the Millimetron Space Radio Telescope.

Generation of Control Commands in the Group SEMS …

213

2 Specific Features of the Commands Generation to Control the System for Shape Adaptation of the Radio Telescope’s Composite Mirror Using a Multi-Channel Optic-Electronic Sensor 2.1 Structure of the Robotic System for Aligning the Position of the Composite Mirror Parts in the Millimetron Radio Telescope Millimetron Space Radio Telescope is being created in accordance with the program for space research of the Russian Federation [6]. Main mirror of the Radio Telescope, 12 m in diameter and with a focal distance of 2.8 m, is of parabolic shape; the secondary mirror, 0.6 m in diameter, is of hyperboloid shape with an eccentricity of 1.0711105 (Fig. 1). The main mirror consists of the central part in the form of a single parabolic mirror, 3 m in diameter, with a hole in the center, 0.6 m in diameter and the peripheral part containing 24 Reflecting Lobes. For final orbital insertion, peripheral part of the main mirror is folded into a cylinder, 3 m in diameter. Upon reaching the final orbital, the Reflecting Lobes unfold and, using the adaptation system, their position is aligned to achieve a parabolic shape. Relocation of the 24 Reflecting Lobes for them to be placed in the required position is ensured by 12 robotic systems. The robotic system includes Head SEMS and a group of two Slave SEMS. 12 Head SEMS are installed along the circle on the Base Ring, inside diameter of the Ring equals 220 mm, outside diameter is 600 mm (Fig. 1). Fig. 1 Main mirror system structure of millimetron radio telescope

214

I. A. Konyakhin and M. H. Tong

Main Mirror Axis Optic-electronic Base Unit Head Hexapod Mobile Platform

Radiation Marks

Reflecting Lobe Slave Hexapod Mobile Platform Slave Hexapod Base Platform

Base Ring Head Hexapod Base Platform

Central mirror Referent Radiation Mark

Fig. 2 Structure of robotic system

Each of the 24 Slave SEMS moves one Reflecting Lobe during the adaptation process. Coordinates deviation of the Reflecting Lobe points from the theoretical paraboloid after its relocation should not exceed the value of σ = 0.02 mm. The structure of one robotic system is shown in Fig. 2. In the proposed system, SEMS are configured based on Hexapod [5]. The Head Hexapod Base Platform is secured to the Base Ring while on its Head Hexapod Mobile Platform the receiver Base Unit of the optic-electronic sensor is installed. The Slave Hexapod Base Platform is secured to the frame, whereas the Reflecting Lobe of the main mirror is installed on the Slave Hexapod Mobile Platform. One Head Hexapod generates control commands for two Slave Hexapods.

2.2 Diagram of the Multi-Channel Optic-Electronic Sensor for the Head Hexapod In the suggested system, the Slave Hexapods, following the commands from the Head Hexapods, move the Reflecting Lobes to the position which corresponds to the theoretical paraboloid of the main mirror. Closure of the feedback loop in the following sequence: “Head SEMS—move command—Slave SEMS— performing the command—lobe move monitoring” is made using the multi-channel optic-electronic sensor which is part of the Head SEMS structure. The multi-channel optic-electronic sensor comprises 3 main parts [7]. The first part constitutes 17 Radiation Marks in the form of point sources of optic radiation which are located on the mirror system components, the second part is the Base Unit receiver installed on the Head Hexapod Base Platform—Figs. 2, 3. The third

Generation of Control Commands in the Group SEMS …

215

Fig. 3 Structure of the multi-channel optic-electronic sensor

part of the sensor is the Engineering Model of one receiver channel “Radiation Mark—objective lens—matrix”. One Head SEMS controls the position of two Reflecting Lobes. On each Reflecting Lobe there are 6 Radiation Marks, in addition, 5 Referent Radiation Marks are located on the edge of the Central Mirror—Fig. 3 [8]. An infrared emission diode (IED) with power of 20 mW is used as a Radiation Mark. Receiver Base Unit contains an objective lens with a focal distance of 350 mm and 11 matrix receivers. We used CMOS matrix receivers OV05620 Color CMOS QSXGA with 2592 × 1944 pixels and a pixel size of 2.2 × 2.2 μm produced by Omni Vision as image analyzers [9, 10]. Images on the Radiation Marks are formed on matrix receivers. As per Fig. 3, images of two proximately located Radiation Marks are generated on the first and fifth matrixes in the first row and in the central matrixes in the second and third row; images of three proximately located Radiation Marks are generated on the central matrix in the first row; image of one Radiation Mark is generated on the remaining 6 matrixes. When the system is functioning, video frames from photo receiver matrixes are processed by Head SEMS’ own processor and coordinates of the images on the matrixes are calculated which are later used to calculate the coordinates of Radiation and Referent Marks themselves in the coordinate system of the Radio Telescope. Image coordinates of the Radiation Marks are calculated as mean value of the 10 frames. Based on the Radiation Marks coordinates, spatial position parameters of the Reflecting Lobes is determined, their deviation is calculated from the theoretical paraboloid and commands for Slave Hexapods are generated which move

216

I. A. Konyakhin and M. H. Tong

their Mobile Platform and shift the Reflecting Lobe to the required position. The measured coordinates of the Referent Radiation Marks are used to determine offsets and turns of the Base Unit in relation to the nominal position. Based on the measured mismatch, Head SEMS generates auto compensation commands in line with which Head Hexapod moves its Mobile Platform and restores the initial position of the Base Unit. The Engineering Model of the measuring channel located in the Base Unit is used for updating the parameters of the computer models used in Head SEMS to generate control commands.

2.3 Sequence of the Control Command Generation Using Optic and Electronic Multi-Channel Sensor In order to achieve high accuracy for position of Reflecting Lobe points in space, Head SEMS, apart from control commands for Slave SEMS, also generate special commands to calibrate and change the structure of the optic and electronic multichannel sensor. These special commands are generated using computer models which are loaded into the operating memory of the Head SEMS processor. The computer model parameters are determined during operation of the robotic system using the Engineering Model of the measuring channel. Diagram of the Engineering Model is shown in Fig. 4. The Engineering Model simulates the measuring channel “IED—objective lens— matrix receiver” in the Base Unit. The Engineering Model consists of infrared emission diode (IED) 1 and matrix receiver 2, which are similar to those used in the measuring channel of the Base Unit. Installed upstream of IED 1 there is a diaphragm 3 with hole diameter of 0.2 mm; objective lens 4 with a focal length of 50 mm that generates image 5 of diaphragm 3 on matrix receiver 2. The Engineering Model determines the actual error in measurement of image coordinates on the matrix receiver caused by noise [9]. The following algorithm is used: 1.

10 frames are read from matrix receiver 2. For each online frame, x, y coordinates of image 5 are determined at the receiver platform [9, 11].

Fig. 4 Engineering model diagram: 1—IED, 2—matrix receiver, 3—diaphragm, 4—objective lens, 5—image of diaphragm 3

2

5

2f

4

3

2f

1

Generation of Control Commands in the Group SEMS …

2.

3.

217

2 sequences of measured coordinate random values x i , yi (i = 1…10) have been received. R.m.s. deviation of σx and σy in mm is determined for these two sequences of numbers. Actual error in measurement of image coordinates is determined using the following formula:

σ=

max{ σ x, σ y} , d

(1)

here σ—r.m.s. of measured image coordinates in parts of receiver’s pixel dimension, d—pixel dimension of matrix receiver in millimeters. The received value σ is sent to the computer models located in the operating memory of Head SEMS. The cycle for generation of Head SEMS control commands consists of 5 steps (Fig. 5). Step 1. The σ value of error in the measurement image coordinates is loaded from the Engineering Model into the computer models. Step 2. Using the computer model of measurement, calculation is made of the quantity of referent marks which is necessary to measure the position of the optic and electronic Base Unit with required accuracy. At the same time, another computer model is used for calculation of the quantity of Radiation Marks on the Reflecting Lobe in the case of which the measurement error of the optic-electronic sensor is going to be small enough for aligning the Reflecting Lobe with the theoretical paraboloid with required accuracy. When the optic-electronic sensor is working, it is required to use the minimum number of Radiation Marks to reduce the amount of dissipated heat and decrease the load on the cryogenic system for cooling the Radio Telescope mirror. Step 3. Generation of command to actuate the determined necessary number of Referent Radiation Marks and Radiation Marks on the Reflecting Lobe. Step 4. Coordinates of Referent Radiation Marks are measured in the coordinate system of the Base Unit. As a result, the difference is determined between the actual position of the Base Unit and the required theoretically accurate position. Step 5. A command for Head Hexapod is generated. With this command, the Head Hexapod Mobile Platform moves the optic and electronic Base Unit into the theoretically accurate position. Step 6. After aligning the Base Unit into the theoretically accurate position, an authorizing command is generated for the optic-electronic sensor based on which measurement of the Reflecting Lobe position is made. Step 7. Difference is determined between the actual position of the Reflecting Lobe and the theoretical position corresponding to paraboloid. A command is established based on which the Slave Hexapod Mobile Platform moves the Reflecting Lobe into the position corresponding to the theoretical paraboloid. End of cycle.

218

I. A. Konyakhin and M. H. Tong

Technique Model Error σ calculation

σ Computer Model of measuring coordinates of the Referent Radiation Marks Command

Computer Model of measuring coordinates of the Radiation Marks

Command

Switching the Referent Radiation Marks

Number of the switched Referent Radiation Marks

Command Moving the Base Unit by Head Hexapod Mobile Platform

σ

Number of the switched Radiation Marks

Switching the Radiation Marks

Command

Measuring the shifts of the Base Unit

Measuring the shifts of the Reflecting Lobe

Permitting

Moving the Reflecting Lobe by Slave Hexapod Mobile Platform

Fig. 5 Cycle for generation of the robotic system control commands

3 Computer Models for the Control Commands Generation 3.1 Coordinate Systems Which Determine the Position of Sensor Components Suppose T (X0 Y0 Z0 ) is the main coordinate system of the Radio Telescope’s measuring system. The center of the coordinate system coincides with the geometric center of the main mirror paraboloid. Orientation of the axes is as follows: X0 — directed parallel to the elevation axis of the Radio Telescope, Y0 —perpendicular to

Generation of Control Commands in the Group SEMS …

219

it, Z0 —directed to the primary radiator and coincides with the axis of the mirror system—Fig. 6. Each of the multi-channel Base Units of the sensor is located on the central mirror next to the apex of the paraboloid, while the node point and optic axis of its objective lens is located within plane xOz of intermediate coordinate system A (xyz) of this Base Unit. Origin of the intermediate coordinate system coincides with origin of the main coordinate system, while plane xOz forms the φ angle with plane X0 OZ0 of the main coordinate system T (X0 Y0 Z0 ), “z”—system axes are co-directional. Nodal point of the Base Unit’s objective lens is located within the xOz plane of the intermediate coordinate system xyz and has the coordinates (Ux , Uy , Uz ) in it. Connected with the nodal point of the Base Unit’s objective lens is the origin OH of its system of coordinates H (XH YH ZH ), whose OH XH axis forms the θ angle with the Ox axis of the coordinate system, whereas the XH OH ZH plane coincides with the xOz plane. Axis OH XH is aimed at controlled point P0 in the middle part of the Lobe, in which Radiation Mark is located (shown in red color in Fig. 4) and in the event of accurate spatial orientation of the Base Unit, it coincides with the optic axis of the objective lens. In order to define position of the Lobe in space, it is required to pre-define the distance between the Radiation Marks on the Lobe surface. Therefore, an additional coordinate system L (XL YL ZL ), connected with the Lobe is introduced. In case the Lobe is not offset, axes of this system are co-directional to the axes of the A (xyz) system, while in the event when the Radio Telescope adaptation system is under operation, it moves together with the Lobe.

Z0

P (XS,YS,ZS)

z

ZS

XH

ZL

ZH

XS

P0

Y0 y

YL YH

XL OL

x

OH YS

O

X0

Fig. 6 Coordinate systems which determine location of the radiation marks and the base unit

220

I. A. Konyakhin and M. H. Tong

Suppose the coordinates of controlled point P on surface of the main mirror paraboloid in which the Radiation Mark is located are defined in coordinate system T (X0 Y0 Z0 ), namely: PT (XT , YT , ZT ). It is preferable to define the nominal position of the controlled point in the coordinate system S (XS YS ZS ), with axis OZS coinciding with axis OZ0 of coordinate system T (X0 Y0 Z0 ), while plane XS OZS forms a certain angle ϕ with plane X0 OZ0 . In this case, the ϕ angle is determined by the arrangement of the reflecting surface Lobes. Then point P will have coordinates PS (XS , YS , ZS ) in system S (XS YS ZS ).

3.2 Calculating the Image Coordinates of the Radiation Mark on the Receiver Suppose the control point P, in which the Radiation Mark is placed in local coordinate system S is determined by the expression: ⎤ XS ⎢ YS ⎥ ⎥ PS = ⎢ ⎣ ZS ⎦ 1 ⎡

(2)

Let us determine the coordinates of this point in coordinate system L: Position of the local coordinate system S (XS YS ZS ) in the global coordinate system T (X0 Y0 Z0 ) is defined by turning at angle φ around axis OZ0 . Axes of the coordinate system T (X0 Y0 Z0 ) in the local coordinate system A (xyz) are defined by turning at angle (−ϕ) around axis Oz. Position of center of the L (XL YL ZL ) coordinate system in the A (xyz) coordinate system has the OL (Lx , Ly , Lz ) coordinates without an additional turn. Further, position of the local coordinate system center of Base Unit H(XH YH ZH ) in the local coordinate system A (xyz) has coordinates OH (Ux , Uy , Uz ). Transfer from system H to system A is determined by rotations at the (−ψ) angle around the OH XH axis, at the (−θ ) angle around the OH YH axis and at the (−ξ ) angle around the OH ZH axis with subsequent linear transfer of its origin OH (Ux , Uy , Uz ) to point OA (0, 0, 0). As a result, the matrix for transfer from coordinate system S to coordinate system H of the Base Unit is determined by the product of the Radiation Mark coordinates in control point P and by the following expression MSH = MAH · MAL · MTA · MST

(3)

Here, MAH, MAL , MTA , MST are the matrices describing transfer between coordinate systems [12].

Generation of Control Commands in the Group SEMS …

221

Coordinates of point P in the coordinate system of Base Unit H (XH YH ZH ) are determined by the following expression: ⎤ XH ⎢ YH ⎥ ⎥ · PS = ⎢ ⎣ ZH ⎦ 1 ⎡

PH = M S H

(4)

In accordance with the rule of geometric optics and “virtual chamber” mathematical apparatus, coordinate points of the object and coordinate points of the image are interrelated by the following expression: ⎤ ⎡ Xi H ⎢ Y iH ⎥ ⎢ ⎢ ⎥ ⎢ ⎣ Zi H ⎦ = ⎢ ⎣ 1 ⎡

f XH+ f 

0

0

0

f XH+ f 

0

0 0

0 0

⎤ ⎡ ⎤ XH 0⎥ YH ⎥ ⎥ ⎢ ⎥, ⎥·⎢ ⎣ ZH ⎦ 0⎦ 1 1 0

f XH+ f 

0

(5)

where f —focal length of the objective lens. Let us define the coordinate systems which determine the position of the receiver matrixes in the Base Unit. Each matrix Mi (i—No. of the Radiation Mark to be registered in the controlled point) is connected to the coordinates system C (XC YC ZC )— Fig. 8, axis OC XC of which coincides with the sight line, i.e. the direction from the matrix center through the nodal point of objective lens OH onto controlled point P (Fig. 7). Coordinates of center of each photo-integrated matrix in its coordinate system are defined as MC (0, 0, 0). Origin of the coordinate system of each receiver matrix OC does not coincide with the origin OH of coordinate system H (XH YH ZH ) of the Base Unit and the center of the receiver matrix has a coordinate in the coordinate system of the Base Unit MH (PHx , PHy , PHz ) where PHx is the distance from the nodal point of the objective lens to the surface of the receiver matrix. Reciprocal position of the axes of these coordinate systems is defined by the following three angles α1 , α2 , α3 . Angle α3 defines rotation of plane XC OC YC in relation to axis OH Z of the coordinate system of the Base Unit, angle α2 —defines rotation of plane XC OZC in relation to axis OC YC . Angle α1 defines possible rotation of the matrix coordinate system in relation to axis OXC , since axis OXC is always perpendicular to the matrix, one can assume α1 to be equal to 0. Then, the coordinates of the image in coordinate system C (XC , YC , ZC ) are defined by the following expression: ⎤ 1 ⎢ YC ⎥ ⎥ ⎢ ⎣ Z C ⎦ = MC H ⎡

1

⎤ Xi H ⎢ Y iH ⎥ ⎥ ·⎢ ⎣ Zi H ⎦ ⎡

1

(6)

222

I. A. Konyakhin and M. H. Tong

Z0

P (XS,YS,ZS)

X Z

P0 Y OH

Y0

XC M0

α3 ZC

YC

α1

X0

α2 M

OC

Fig. 7 Coordinate systems which determine location of CMOS matrixes

where MCH is a matrix for transformation of coordinates, corresponding to angles α1 , α2 , α3 and linear offset PHx , PHy , PHz . Matrices for transfer between the coordinate systems and “object—image” transformation system are determined by the geometric and optic parameters of the sensor.

3.3 General Algorithm for Modelling the Optic-Electronic Multi-Channel Sensor Model for measuring the position of the object is based on expressions (2),…,(6), which define the linkage between the coordinates of the Radiation Mark on the Lobe and coordinates of its image on the receiver matrix in the receiver channel “radiation mark—objective lens—receiver matrix”. Assessment of influence exercised by different factors on the measuring accuracy of the optic-electronic sensor is made in accordance with the following algorithm. 1. 2.

Coordinates PSi (XSi , YSi , ZSi ), i = 1,…,6 of the Radiation Marks on the Reflecting Lobe are defined with its position being on the theoretical paraboloid. Based on (2),…,(6) coordinates PCi (XCi , YCi , ZCi ), i = 1,…,6 of Radiation Marks’ images on the matrix receivers are defined.

Generation of Control Commands in the Group SEMS …

3.

4.

5.

223

Using random number generator, minimum 100 values of the factor (parameter of matrices (3), (4) or image coordinates (6)) under study are created. At the same time, random deviation is simulated. Equation system (2),…,(6) is solved in “reverse” direction relative to coordinates of the Radiation Marks on the Lobe and an array of random values is found PSi,j (XSi,j , YSi,j , ZSi,j ), i = 1,…,6; j = 1,…,100 of the coordinates for the marks on the Lobe which have been “measured” by the sensor with a certain error. As a result of processing the array of values, r.m.s. values σi , i = 1,…,6 are determined of the error in measuring Radiation Marks coordinates on the Lobe with further calculating the deviation of the Lobe from the theoretical position on the paraboloid.

For example, when determining the influence exercised by noises, random values which simulate the error of measuring the image coordinates on the receiver matrix are added directly to coordinates PCi (XCi , YCi , ZCi ). Dependence of the error of measuring linear and angle parameters of the Lobe position which has been determined is given in Fig. 8. If coordinates PSi (XSi , YSi , ZSi ) of the Referent Radiation Marks are to be assumed as unchanging parameters, one can then simulate the influence exercised by shifts and rotations of the Base Unit relative to the theoretical position in terms of accuracy in measuring the coordinates of the Radiation Mark on Reflecting Lobe position. The dependence thus obtained is given in Fig. 9. From the data given it follows that there is a strong dependence of the opticelectronic sensor’s accuracy on the shifts of the Base Unit relative to the theoretical position.

3.4 Calculating the Required Quantity of Referent Radiation Marks for Accurate Determination of the Base Unit’s Shift To determine the quantity of Referent Radiation Marks which need to be switched on, calculation is made of the error value in measuring coordinates and turns of the Base Unit at pre-determined r.m.s. error of measuring the Referent Mark’s image coordinates. Figure 10 shows the dependence of the error in measuring the position of the Base Unit on the quantity of switched-on Referent Marks for the error of measuring image coordinates σ = 0.05 of pixel. As it appears from the Figures, the required accuracy of measuring the Base Unit’s shift is attained with 5 Referent Marks switched on. In the event that from the Engineering Model another actual value σ of the error in measuring the image coordinates is loaded, the simulation cycle is repeated over again.

224

I. A. Konyakhin and M. H. Tong

Fig. 8 Dependence of the error in measuring linear a and angle b parameters of the Lobe position on error σ in measuring image coordinates (in parts of pixel)

3.5 Calculating the Required Quantity of Radiation Marks for Accurate Determination of the Reflecting Lobe’s Position To determine the quantity of Radiation Marks which need to be switched on when measuring the position of the Lobe, calculation is made of the error in shaping a complete paraboloid from the Lobes, the position of which has been measured with this accuracy at pre-determined r.m.s. error of measuring the Radiation Mark image coordinates. Figure 11 shows the dependence of the paraboloid shape deviation on the quantity of switched-on Radiation Marks for the error of measuring image coordinates σ = 0.05 of pixel.

Generation of Control Commands in the Group SEMS …

225

Fig. 9 Dependence of the error in measuring the coordinates of the Radiation Mark of the Lobe on the value of the Base Unit’s shift (a) and rotation (b)

As it appears from the Figure, the required accuracy of shaping the paraboloid is attained with 6 switched-on Radiation Marks on each Lobe. In the event that from the Engineering Model another actual value σ of the error in measuring the image coordinates is loaded, the simulation cycle is repeated over again.

226

I. A. Konyakhin and M. H. Tong

Fig. 10 Dependence of error in measuring the linear (a) and angular (b) position parameters of the Base Unit on the quantity of Referent Marks

4 Conclusion We have reviewed the peculiarities of generating control commands for the SEMS group with the following structure “Head SEMS and Slave SEMS”, if the Head SEMS uses a multi-channel optic-electronic sensor. In this case, in order to generate commands, the simulation results obtained from the models located in the operating memory of Head SEMS are used. The following features are analyzed:

Generation of Control Commands in the Group SEMS …

227

Fig. 11 Dependence of the error in shaping complete paraboloid from Lobes on the quantity of switched-on Radiation Marks

1. 2. 3.

Use of Engineering Models to obtain a number of parameters loaded into computer models; Use of computer simulation results to generate the auto calibration commands of the initial position of the Head Hexapod’s Mobile Platform; Use of computer simulation results to generate commands aimed at optimizing and changing the structure of the optic-electronic multi-channel sensor.

The above mentioned problems have been considered using the example of the robotic system for adaptation of the composite mirror shape of the Millimetron Radio Telescope. Principles of building such a robotic system and the algorithms of generating the commands can be used to create systems to monitor objects by position of their points or in the devices for automatic assembly of structures to ensure a required reciprocal position of separate parts. Acknowledgements This work was financially supported by Ministry of Science and Higher Education of Russian Federation, Grant 08-08.

References 1. Nogin, A.A., Konyakhin, I.A.: Optoelectronic SEMS for preventing object destruction. Studies Syst. Decis. Control 261, 199–204 (2020) 2. Nogin, A.A., Konyakhin, I.A.: The position monitoring robotic platforms of the radiotelescope elements on base of autocollimation sensors. Stud. Syst. Decis. Control - 174, 297–304 (2019)

228

I. A. Konyakhin and M. H. Tong

3. Petrochenko, A.V., Konyakhin, I.A.: Method of constructing a system of optical sensors for mutual orientation of industrial robots for monitoring of the technosphere objects. Stud. Syst. Decis. Control - 95, 105–116 (2017) 4. Konyakhin, I.A., Sakhariyanova, A.M.: Optic-electronic system for measuring angular position of SEMS units on base of autoreflection sensors. Stud. Syst. Decis. Control - 174, 305–312 (2019) 5. Konyakhin, I.A., Vasilev, A.S., Petrochenko, A.V.: Electrooptic converter for measuring linear shifts of the section boards at the main dish of the radiotelescope. Stud. Syst. Decis. Control 49, 269–277 (2016) 6. “MILLIMETRON Space Observatory,” https://millimetron.ru (24 Aug 2020) 7. Konyakhin, I.A., Tong, M.: Multi-matrix optic-electronic systems for measuring the line shifts of the points on the radio-telescope main mirror. Proceedings of SPIE - 11053, 1105307 (2019) 8. Phuong, H., Gorbachev, A., Konyakhin, I., Minh Hoa, T.: Optical-electronic system for measuring spatial coordinates of an object by reference marks. Stud. Syst. Decis Control 261, 217–227 (2020) 9. Vasileva, A.V., Vasilev, A.S., Konyakhin, I.A.: Vision-based system for long-term remote monitoring of large civil engineering structures: design, testing, evaluation. Measurement Sci. Techn.- 29, No. 11, pp. 115003, (2018) 10. Colour and monochrome megapixel USB 2.0 TV camerAS. JSC EVS, St.Petersburg, Russia. http://www.evs.ru/eng/tv_kam7.php 11. Zhukov, D.V., Konyakhin, I.A., Usik, A.A.: Iterative algorithm for determining the coordinates of the images of point radiators. Journal of Optical Technology 76(1), 36–38 (2009) 12. Korn, G.A., Korn, T.M.: Mathematical handbook for scientists and engineers: definitions, theorems, and formulas for reference and review, vol. 1152. Dover Publications, New York (2000)

Increasing the Reliability of Decision Making by Improving the Characteristics of Optoelectronic Channels Ensuring the Separation of Complex Shape Fruit Ba Minh Dinh, Aleksandr N. Timofeev, Igor A. Konyakhin, and Valery V. Korotaev Abstract Introduction. To develop reliable solutions, robotic systems use several channels for contactless control of moving objects. It is especially active in contactless robotic systems for evaluating the quality of agricultural products. Determination of informative parameters of fruits (shape, size, surface texture and color) is a characteristic feature of the complexes, which are used to make final estimation fruit’s quality. Purpose. The research of the enhancement robotic optoelectronic channels, that allow increasing of the reliability of decision-making during separation in the flow of complex shape fruits (pitaya, pineapple and custard apple) by combining information about their shape, size, surface texture and color from several images, which are obtained for various fields of observation with illumination of the optical pulsed sources. Methods. To solve this problem, the authors propose research on digital multi-angle color images processing of fruits from several video cameras with their synchronous multidirectional pulse sources of illumination. The number, spatial mutual location of video cameras and sources, as well as the time procedures for their interaction allow improving the accuracy of determining the parameters of shape, size, surface texture and color, which ultimately will increase the reliability of decision-making about the quality of the fruit. Practical relevance. Computer simulation of the spatial placement of video cameras (up to 5 pieces) and their calibration methods with impulse sources of illumination can increase the accuracy of determining geometric shape parameters (curvature, center of gravity, convexity, roundness coefficient and eccentricity) by 8%. Experimental results on an installation with video cameras and eight impulse sources of illumination confirmed the increase image contrast by 2 times.

B. M. Dinh (B) · A. N. Timofeev · I. A. Konyakhin · V. V. Korotaev ITMO University, Faculty of Applied Optics, St. Petersburg, Russia e-mail: [email protected] A. N. Timofeev e-mail: [email protected] I. A. Konyakhin e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 A. E. Gorodetskiy and I. L. Tarasova (eds.), Smart Electromechanical Systems, Studies in Systems, Decision and Control 352, https://doi.org/10.1007/978-3-030-68172-2_19

229

230

B. M. Dinh et al.

Keywords Robotic complex · Optoelectronic channel · Fruit flow · Pulse illumination · Image processing · Sorting criteria

1 Introduction Quality control during the collection and separation of agricultural products, as a rule, is carried out in a fairly short time. The need to increase the productivity of these processes has led to active research and development of intelligent robotic systems [1, 2]. Obviously, robotic sorting systems are much faster than any manual quality control procedures and allow for effective inspection of all products using objective and stable criteria. The emergence of both technical [3–5] and software tools for machine vision also contributes to active development of automatic sorting system [6, 7]. Analysis of various optoelectronic channels and the sorting system (OCSS) shows that, during their operation, many problems are arisen, such as the solution of which lies not only in the optimal arrangement of elements, but also in the procedures for their interaction [8–10]. The currently developing OCSS should ensure reliable recognition of the fruits surrounding them in three-dimensional space. At the same time, OCSS solves a wide range of problems associated with contactless measurements of the dimensions of objects and the determination of their relative position in space [11, 12]. For subsequent processing and accurate determination of the fruit’s informative parameters, it is necessary to obtain images with specific parameters (contrast, color rendering, brightness, etc.). In the case of determining morphological features (size and shape), the surface area, perimeter, values of the main and secondary axes of image are used to quantify the object’s size [13, 14]. Fruits grown on the territory of Vietnam, pitaya (Fig. 1a), sugar apple (Fig. 1b), and pineapple (Fig. 1c) have a complex shape, specific surface texture and different colors. Therefore, it is important to study the ways that allow increasing the reliability of sorting fruits by combining information from several obtained images of observation fields, which illuminated by pulsed sources of optical radiation with required spectral composition.

Fig. 1 Photos complex-shape fruits a—pitaya, b— custard apple, c—pineapple

Increasing the Reliability of Decision Making …

231

Fig. 2 The structure of the OCSS for monitoring parameters of complex-shape fruits

2 Generalized OCSS of Complex-Shape Fruits Based on the analysis of fruit and vegetable sorting systems [15–17], a generalized structure of multi-cameras OCSS was formed, consisting of video cameras VC (Fig. 2), pulsed illumination sources IS, control block BC and computer. The proposed structure makes it easy to configure for the class of the set requirements for automatic sorting. In this case, system provides simultaneously operation of several channels (5 channels in Fig. 2), which operates in different modes. The required number of VC and IS in the proposed structure, their location and analysis of information processing procedures to find informative parameters of fruit for their sorting are the subject of research.

3 Specificity of Receiving and Processing Information in OCSS According to the specified procedures, transformation of information in multicameras OCSS can be represented by the following diagram (Fig. 3). In this diagram, optical signals of the illumination sources IS1.1 –ISM.3 reflected from the controlled objects CO1 –CON by the optical systems OS1 –OSM of the video cameras VC1 –VCM

232

B. M. Dinh et al.

Fig. 3 Generalized scheme for converting information in OCSS; S—source; MCIS—module control of illumination source; IS—illumination source; MPP—module preliminary processing; VC—video camera

is formulated a distribution of irradiance in their photosensors Sensor1 –SensorM . In the preliminary preprocessing modules, this distribution is converted into digital images, which are then processed in the processing module of control block to extract informative features for the final decision and classification by the classifier K. During preprocessing, the control module synchronizes the signals of IS1.1 –ISM.3 with the operation of video cameras VC1 –VCM and adjusts their parameters to obtain maximum contrast. The generated database contains static images and dynamic images of fruits in the sequence of frames. Static images reflect only the peak intensity level, while dynamic images capture the shape of the fruit, changing over time. Main stages of information transformation in the OCSS: – Pre-processing of images; – Extraction of informative features; – Decision and classification.

3.1 Images Pre-processing in OCSS During preliminary preprocessing, as a rule, the areas in which the controlled fruits are located have been found, then cropping and scaling of images have been carried out, simultaneously the image contrast have been changed due to adjustable parameters of video cameras and the directions of illumination. Localization of the control area makes it possible to determine the size and location of the fruits on the images, which obtained from the video cameras VC1 –VCM . The most commonly used localization methods are:

Increasing the Reliability of Decision Making …

– – – –

233

Method Viola-Jones (Viola–Jones object detection, VJ); Single shot multi-box detector (SSD) [18]; Histogram of oriented gradients (HOG); Max margin object detection (MMOD) [19].

Cropping and scaling of the control area is carried out according to the coordinates obtained by the methods of localization of the control area. Since the control areas have different sizes, it is necessary to scale the image, i.e. bring all images to the same resolution. For these tasks, the following methods have been used: – Bessel’s correction [20]; – Gaussian distribution. Alignment’s object permitted reduces their differences. So, for example, for each object a reference image is selected, which is divided according to color components or the most informative control areas (for example, the main body of the fruits), remaining images are aligned with respect to the reference images. For this task, it is possible to use: – Scale-invariant feature transform (SIFT) [21]; – Region of interest (ROI) [22]. Adjustment of the contrast allows to smooth images, reduce noise, increase the contrast of the test image, and improve the saturation. Moreover, adjustment of the contrast also permitted deal with problem of effective illumination of the test object. In robotic systems, the following contrast adjustment methods are used: – Histogram equalization (HE) [23]; – Linear contrast stretching [24]. To adjust the contrast in OCSS, the above methods should be supplemented by using method of frame-to-frame difference of imaging fields, which are illuminated from different directions [25]. Various image manipulations, such as rotation or displacement, can increase the versatility of images and expand databases. Extending the database with variable images is useful for deep learning methods. Whereas, for traditional methods, it is more useful to use models based on contrast equalization, which, on the contrary, reduce the variability associated with changes in the position of the object, that reduces differences in shape and increase similarity. For this case, for each type of fruits, its own standard is selected, according to which the images of each type are aligned (Fig. 1). Selection of the correct pre-processing methods takes a lot of time, since the speed and accuracy of determining the informative parameters of the fruits depends on it.

234

B. M. Dinh et al.

3.2 Features Extraction An important stage of information processing in OCSS is the extraction of informative features. At this stage, findings of the elements, which are the most informative for further processing, are realized. To determine the geometric parameters of the fruits, extraction of informative features could be based on several basic methods: – LEM (line edge map) descriptor measures the similarity of the parameters of the controlled objects based on their boundaries; – ASM (Active shape model) [26] detects the edges of objects using landmarks, which are series of points of informative features; – AAM (Active appearance model,) [27]—an extended version of ASM, which also forms the texture features of the controlled objects; – HOG allows to compare images of objects in the direction of gradients [3]; – Fuzzy membership functions; – SIFT (scale invariant features transformation) determines potential points of interest in the image by defining maximum and minimum for Gaussian filter function, filters are applied at different scales and rotations [28, 29]; – Curvelet Transform [30] transmits information about location of the object and spatial frequency.

3.3 Fruit Quality Classification Fruit quality classification is the last stage in converting information into OCSS and at this stage develops a decision on assigning the fruit to a particular quality category [31]. The traditional machine classification methods used in OCSS: – Support vector machine (SVM) [32] used to highlight informative parameters of defects and is characterized by high accuracy and well-proven efficiency in many areas of image recognition [33, 34]; – Softmax method is one of the most commonly used logistic regressions for multiclass separation has been tested as a fast training classifier [35]; – K-Nearest neighbors method (KNN) [34] is an instance-based learning classifier in which hypotheses are built directly from training instances.

4 Application of Interframe Difference Algorithm in OCSS As a rule, intraframe and interframe image filtering is used to increase the detection probability and optimize methods for searching of motion vectors in video monitoring [36–38]. Algorithms for intraframe processing can be quite effective when the size of the selected objects is less than individual background details with constant or slowly

Increasing the Reliability of Decision Making …

235

changing brightness. If the sizes of the selected objects are commensurate with the “background” objects, interframe processing algorithms may be more effective, which allow using the difference in the dynamics of the relative motion of the selected and “background” objects as an identification feature. In general, the process of automatic information extraction based on the interframe processing algorithm proposes analysis of several images (at least two) obtained at different points in time. As a result of such analysis, not only the detection of objects can be carried out, but also the determination of their informative parameter: dimensions, coordinates, speed, and direction of movement in space. The main difficulty arising in this case is usually associated with the fact that moving objects, in the general case, are less contrasting than stationary objects in the field of view of the video cameras, and, therefore, it is not possible, by means of simple amplitude selection, to separate the useful signal from the moving object against interfering background signals. And so the process of extracting information about moving objects is split into two stages: – Formation of an interframe difference signal (IDS), in which all information about changes in the image, and at the same time, there are no (or significantly suppressed) interfering level drops corresponding to stationary objects in the frame; – optimal processing of IDS to extract necessary information with maximum reliability. In the considered OCSS, to increase the contrast in resulting image of a moving fruits, we can use the information integration procedure by calculating the interframe difference brightness of the frames, registered with multidirectional illumination sources. In the proposed procedure, the initial image of the fruits is first captured with pulsed illumination from one direction (base frame Fig. 4a). After that, a new image of the fruits is registered with pulsed illumination from a different direction (Fig. 4b) and it is compared with the base frame and the segmented image is finally formed (Fig. 4c). As a result of this operation, contrasting areas are well distinguished in relation to the previously present background. This differential approach based on finding image points using a difference scheme. The method used in the experiments is based on calculating the first-order

Fig. 4 Interframe difference calculation explanation: base frame (a), current frame (b), resulting image (c)

236

B. M. Dinh et al.

derivatives. Assumed that, the brightness of a point remains constant for a short period of time, i.e. ∂ I (x, y, t)/∂t = 0 For a two-dimensional machine  vision space with the u and v axes, the optical flow  equation can be written as E x , E y ·u, v = −E t [27]. Optical flow component in the  T  direction of the luminance gradient E x , E y described by the relation E t / E x2 + E y2 Hence the equation is formed: ∇ I (x, y, t) · u, vT + It (x, y, t) = 0

(1)

Under the assumption of second-order derivatives: ∂∇ I (x, y, t)/∂t = 0 And revealing the differential and the gradient over the spatial coordinate variables x, y, it turns out: 

Ix x (x, y, t) Ix y (x, y, t) Ix y (x, y, t) I yy (x, y, t)

       u 0 I (x, y, t) · = + tx It y (x, y, t) v 0

(2)

The algorithm is based on minimization of Eq. (3), composed of a smoothing part and a part based on the Eq. (1): ¨ (∇ I · v + It )2 + λ2 (∇u2 + ∇v2 )d xd y

(3)

D

where x = (x, y)T , v = (u, v)T , It = I (x, y, t), D—region, in which optical flow is sought, λ—coefficient determines the level of significance of smoothing part of the functional (3). The experiments were carried out on a setup containing an optical-electronic channel in the form of a VC video camera (Fig. 5a) (VEC 535 with an OV5620 CMOS QSXGA 5.17 MPixel matrix), IS1 and IS2 optical illumination sources (LED lamps with an optical flow 1000 lm) and a personal computer (laptop under Windows 7). The video camera VC was positioned frontally at 1000 mm, and the pulsed illumination IS1 and IS2 was carried out alternately to the object of research at different angles ϕ from 10 to 50 angular degrees. Processing images was carried out in LabVIEW, a program based on interframe difference algorithm was implemented. The quality of the images, which obtained from the OCSS was assessed by the magnitude of the absolute contrast η obtained with a stationary background.

Increasing the Reliability of Decision Making …

237

, relative unit

, angular degree.

a)

b)

Fig. 5 Experiment scheme (a) and dependence of the contrast η of two different elements of the fruit’s border on the angles ϕ of synchronous illumination by sources IS1 and IS2

The experimental results showed that the contrast η in the final image increased by almost 2 times from the backlight angles (Fig. 5b). At the same time, processing the geometric parameters of segmented images of pitaya (Fig. 1a) before and after applying the interframe difference algorithm showed a decrease in the RMS estimates of the average fruit size by 8%. The studies confirm the effectiveness of using interframe difference algorithm in the preliminary processing of images.

5 Conclusion An increase in the reliability of decision-making in intelligent systems with opticalelectronic channels ensured the separation of complex-shape fruits is possible by improving the preliminary processing of images using interframe difference algorithm. The procedure of synchronous pulsed illumination sources from different directions allows to equalize the contrast in the image, which is the informative parameter of image segmentation in determining the size and shape of fruits. The proposed algorithm formation and processing of the interframe difference signal allows, with an intense level of interference and synchronous multidirectional illumination, to achieve the required contrast, which is a few percent relative to the elements of the surrounding background in original images. The results of the experiment showed that the contrast η in the final image can increase by 2 times from the backlight angles. It was found that when processing the geometric parameters of segmented images of pitaya before applying the interframe difference algorithm and after, it showed a decrease in the estimates of the RMS of the size of the fruit by 8%. For further research, the authors plan to consider the influence of calibration in the multi-cameras OCSS scheme on the error in determining the geometric parameters of complex-shape fruits.

238

B. M. Dinh et al.

Acknowledgements This work has been supported by the Ministry of Science and Higher Education of the Russian Federation, Grant 08-08.

References 1. Blasco, J., Aleixos, N., Molto, E.: Machine vision system for automatic quality grading of fruit. Biosyst. Eng. 85(4), 415–423 (2003) 2. Edan, Y., Han, S., Kondo, N.: Automation in agriculture. In: Nof, S.Y. (ed.) Handbook of Automation, pp. 1095–1128. Springer, West Lafalette, IN, USA (2009) 3. Idris, M., Arof, H., Tamil, E., Noor, N., Razak, Z.: Review of feature detection techniques for simultaneous localization and mapping and system on chip approach. Inf. Technol. J. 8, 250–262 (2009) 4. Tao, Y., Zhou, J.: Automatic apple recognition based on the fusion of color and 3D feature for robotic fruit picking. Comput. Electron. Agric. 142, 388–396 (2017) 5. McCarthy, C.L., Hancock, N.H., Raine, S.R.: Applied machine vision of plants: A review with implications for field deployment in automated farming operations. Intell. Serv. Robot. 3, 209–217 (2010) 6. Ramli, R., Idris, M.Y.I., Hasikin, K., Karim, A., Khairiah, N., AbdulWahab, A.W., Ahmedy, I., Ahmedy, F., Kadri, N.A., Arof, H.: Feature-based retinal image registration using D-saddle feature. J. Healthc. Eng. 2017, 1–15 (2017) 7. Rachmawati, E., Supriana, I., Khodra, M.L.: Toward a new approach in fruit recognition using hybrid RGBD features and fruit hierarchy property. In: 2017 4th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI), pp. 1–6. http://ieeexp lore.ieee.org/document/8239110/. (September). https://doi.org/10.1109/EECSI.2017.8239110 (2017) 8. Rosten, E., Porter, R., Drummond, T.: Faster and better: A machine learning approach to corner detection. IEEE Trans. Pattern Anal. Mach. Intell. 32, 105–119 (2010) 9. Li, L., Zhang, Q., Huang, D.: A review of imaging techniques for plant phenotyping. Sensors 14, 20078–20111 (2014) 10. Rosell, J.R., Sanz, R.: A review of methods and applications of the geometric characterization of tree crops in agricultural activities. Comput. Electron. Agric. 81, 124–141 (2012) 11. Andújar, D., Rueda-Ayala, V., Moreno, H., Rosell-Polo, J.R., Valero, C., Gerhards, R., Fernández-Quintanilla, C., Dorado, J., Griepentrog, H.-W.: Discriminating crop, weeds and soil surface with a terrestrial LIDAR sensor. Sensors 13(11), 14662–14675 (2013) 12. Khojastehnazhand, M., Omid, M., Tabatabaeefar, A.: Determination of tangerine volume using image processing methods. Int. J. Food Prop. 13(4), 760–770 (2010) 13. Riyadi, S., Rahni, A.A., Mustafa, M.M., Hussain, A.: Shape characteristics analysis for papaya size classification. In: Research and development, pp. 1–5 (2007) 14. Khojastehnazhand, M., Omid, M., Tabatabaeefar, A.: Development of lemon sorting system based on color and size. Afr. J. Plant Sci. 4(4), 122–127 (2010) 15. Gongal, A., Amatya, S., Karkee, M., Zhang, Q., Lewis, K.: Sensors and systems for fruit detection and localization: A review. Comput. Electron. Agric. 116, 8–19 (2015) 16. Sachdeva, V.D., Fida, E., Baber, J., Bakhtyar, M., Dad, I., Atif, M.: Better object recognition using bag of visual word model with compact vocabulary. In: Proceedings of the 13th International Conference on Emerging Technologies (ICET), Islamabad, Pakistan, pp. 1–4, 27–28 Dec 2017 17. Noordam, J., Hemming, J., Van Heerde, C., Golbach, F., Van Soest, R., Wekking, E.: Automated rose cutting in greenhouses with 3D vision and robotics: Analysis of 3D vision techniques for stem detection. In: van Straten, G., Bot, G.P., van Meurs, W.T.M., Marcelis, L.F. (eds.) Acta Horticulturae 691, pp. 885–889. Leuven, Belgium, ISHS (2005)

Increasing the Reliability of Decision Making …

239

18. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A.C.: SSD: single shot multibox detector. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9905, pp. 21–37. https://doi.org/10.1007/978-3-319-46448-0_2 (2016) 19. King, D.E.: Max-margin object detection [lektponny pecypc]. URL: https://arxiv.org/pdf/ 1502.00046.pdf, cvobodny. z. angl. (data obpaweni: 13.12.2019) 20. Mohan, P.G., Prakash, C., Gangashetty, S.V.: Bessel transform for image resizing. In: Proceedings 18th International Conference on Systems, Signals and Image Processing (IWSSIP 2011), pp. 75–78. Sarajevo, Bosnia-Herzegovina (2011) 21. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004). https://doi.org/10.1023/B:VISI.0000029664.99615.9 22. Naz, S., Ziauddin, S., Shahid, A.R.: Driver fatigue detection using mean intensity, SVM, and SIFT. Int. J. Interact. Multimed. Artif. Intell. 5(4), 86–93 (2019). https://doi.org/10.9781/iji mai.2017.10.002 23. Wang, X., Chen, L.: Contrast enhancement using feature-preserving bi-histogram equalization. Signal Image Video Process. 12(4), 685–692 (2018). https://doi.org/10.1007/s11760-0171208-2 24. Mustapha, A., Oulefki, A., Bengherabi, M., Boutellaa, E., Algaet, M.A.: Towards nonuniform illumination face enhancement via adaptive contrast stretching. Multimed. Tools Appl. 76(21), 21961–21999 (2017). https://doi.org/10.1007/s11042-017-4665-2 25. Cuevas, E., Zaldívar, D., Pérez-Cisneros, M., Oliva, D.: Block-matching algorithm based on differential evolution for motion estimation. Eng. Appl. Artif. Intell. 26(1), 488–498 (2013) 26. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Underst. 61(1), 38–59 (1995). https://doi.org/10.1006/cviu. 1995.1004 27. Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001). https://doi.org/10.1109/34.927467 28. Warif, N.B.A., Wahab, A.W.A., Idris, M.Y.I., Salleh, R., Othman, F.: SIFT-symmetry: A robust detection method for copy-move forgery with reflection attack. J. Vis. Commun. Image Represent. 46, 219–232 (2017) 29. Panchal, P., Panchal, S., Shah, S.: A comparison of SIFT and SURF. Int. J. Innov. Res. Comput. Commun. Eng. 1, 323–327 (2013) 30. Fu, X., Fu, K., Zhang, Y., Zhou, Q., Fu, X.: Facial expression recognition based on Curvelet transform and sparse representation. In: Proceedings 14th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNN-FSKD 2018), pp. 257–263. Huangshan, China (2018). https://doi.org/10.1109/fskd.2018.868698 31. Dubey, S.R.; Jalal, A.S.: Apple disease classification using color, texture and shape features from images. Signal Image Video Process. 10, 819–826 (2016) 32. Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Berlin/Heidelberg, Germany (2013) 33. Danades, A., Pratama, D., Anggraini, D., Anggriani, D.: Comparison of accuracy level Knearest neighbor algorithm and support vector machine algorithm in classification water quality status. In: Proceedings of the International Conference on Frontiers of Information Technology (FIT), Islamabad, Pakistan, 19–21 Dec 2016, pp. 137–141 34. Syaliman, K., Nababan, E., Sitompul, O.: Improving the accuracy of k-nearest neighbor using local mean based and distance weight. J. Phys. Conf. Ser. 978, 012047 (2018) 35. Pellegrini, T.: Comparing SVM, Softmax, and shallow neural networks for eating condition classification. In: Proceedings of the Sixteenth Annual Conference of the International Speech Communication Association, Dresden, Germany, 6–10 Sept 2015 36. Yuan, X., Shen, X.: Block matching algorithm based on particle swarm optimization for motion estimation. In: IEEE ICESS’08 International Conference on Embedded Software and Systems, 2008, pp. 191–195, (2008)

240

B. M. Dinh et al.

37. Xiaoping, L., Songze, L., Feng, X., Boxing, Z., Yanhong, W.: Fast aerial UAV detection using improved inter-frame difference and SVM. J. Phys. Conf. Ser. 1187(3), 032082 (2019). https:// doi.org/10.1088/1742-6596/1187/3/032082 38. Davies, D., Palmer, P., Mirmehdi, M.: Detection and Tracking of Very Small Low Contrast Objects. http://homepages.inf.ed.ac.uk/rbf/CVonline/motion.htm (2003)