Intelligent Computing: Proceedings of the 2021 Computing Conference, Volume 1 (Lecture Notes in Networks and Systems, 283) [1st ed. 2022] 3030801187, 9783030801182

Table of contents :
Editor’s Preface
Contents
Analytical View on Non-invasive Measurement of Moving Charge by Various Topologies of Wannier Qubit
1 Motivation for Weak Measurement of Moving Charged Particles
2 Single Electron Devices as Detectors of Moving Charged Particles
3 Dynamic of Two Qubit Electrostatic Entanglement Under Influence of Weak Measurement
4 Concept of Position Dependent Electrostatic Roton Qubit
4.1 Case of Symmetric Semiconductor Roton
4.2 Case of Asymmetric Semiconductor Quantum Roton
4.3 Action of Phase Rotating Gate Described Analytically
5 Analogy Between Single-Electron Charged Based in 2 Coupled Quantum Dots Qubit and 1/2 Spin Qubit
6 Particle in Rotating Magnetic Field and Relation to the Charge Qubit
7 Spin 1/2 Dynamics of Single Electron in Magnetic Field in Angle Coordinates and Its Mapping to Single-Electron Charge Qubit Dynamics in System of 2 Coupled Quantum Dots
8 Conclusions
References
Multi-population Genetic Algorithm with the Actor Model Approach to Determine Optimal Braking Torques of the Articulated Vehicle
1 Introduction
2 Mathematical Formulation of the Model of Investigation
3 Definition of Optimization Problem
4 Genetic Algorithm with Actor Model Approach
5 Numerical Simulations
6 Conclusions
References
A Novel Three-Way Merge Algorithm for HTML/XML Documents Using a Hidden Markov Model
1 Introduction
2 Hidden Markov Models (HMMs)
3 The ``Three-Way Merge'' Approach as a Hidden Markov Model (HMM)
4 Implementation and Experimental Results
5 Conclusion
References
Scaling Out Transformer Models for Retrosynthesis on Supercomputers
1 Introduction
2 Previous Work
2.1 Computer Aided Retrosynthesis
2.2 Distributed Training
3 Approach
3.1 Dataset
3.2 Transformer Model
3.3 Distributed Training
3.4 System Specifications
4 Experiments
5 Results
5.1 Accuracy on Test Set
5.2 Scaling Performance
6 Discussion
7 Conclusion
References
Naming Processes in Multichannels with Beeps in the Weak Model
1 Introduction
2 Related Work
3 Technical Preliminaries
4 Naming in the Known Number of Processes Model
4.1 Proof of Correctness and Analysis
5 Naming in the Unknown Number of Processes Model
5.1 Proof of Correctness and Analysis
6 Conclusions
References
Distributed Evolution of Deep Autoencoders
1 Introduction
2 Related Work
3 Autoencoders
3.1 Image Denoising
3.2 Manifold Learning
4 Evolving Deep Autoencoders
4.1 Network Construction
4.2 Network Evolution
4.3 System Architecture
5 Experiments
6 Discussion
7 Conclusion
References
Computational Power of a Hybrid Algorithm for Solving the Multiple Knapsack Problem with Setup
1 Introduction
2 Related Works
3 Hybrid Methods for PMKPS
3.1 The Model
3.2 A Hybrid Method
3.3 Enhancing the Hybrid Method
4 Experimental Part
4.1 Effect of the Valid Constraints
4.2 Behavior of EHM Versus Other Available Methods
5 Conclusion
References
Accelerated Quantum Computation based on Quantum Coherent Dynamics of Evanescent Photons in Liquid Water
1 Introduction
2 A Brief Overview of QED Coherence in Liquid Water
3 Quantum Tunneling-Coupling Interaction Between Water Coherent Domains
3.1 “Supercoherence” and the Interaction Between Coherent Domains
3.2 Quantum Coupling Between Water CDs
4 Quantum (Hyper) Computation by Means of Water Coherent Domains
4.1 The Concept of “Universal” Quantum Computation
4.2 The Realization of Two-Qubits Gates by Means of Water Coherent Domains
4.3 The Realization of One-Qubit Gates by Means of Water Coherent Domains
5 A Novel Computer Architecture for Quantum Hypercomputation
6 Summary and Outlook
References
Computer Scientist’s and Programmer’s View on Quantum Algorithms: Mapping Functions’ APIs and Inputs to Oracles
1 Introduction
2 History and Related Work
2.1 Quantum Computing
2.2 Quantum Algorithms
2.3 Quantum Oracles
3 Problem Statement
4 Deutsch-Jozsa Algorithm
5 Grover’s Algorithm
6 Quantum Oracles
6.1 The Role of Quantum Oracles
6.2 Quantum Oracles as Input Representation
7 Case Studies
8 Conclusions and Final Thoughts
References
Orca: A Software Library for Parallel Computation of Symbolic Expressions via Remote Evaluation on MPI Systems
1 Introduction
2 Background and Motivation
2.1 What is Wrong with MPI?
2.2 Related Works
2.3 Symbolic Expression
3 Programming Model
3.1 Module Structure
3.2 Overview of APIs
3.3 Marshalling
4 Experiments and Discussion
4.1 A Case Study: Distributed Reduction Operation
4.2 Performance Evaluation
5 Conclusions and Future Work
References
Solving High-Dimensional Nonlinear Equations with Infinite Solutions by the SVM Visualization Method
1 Introduction
2 Methodology
3 Examples
4 Conclusion
References
A Novel GPU Implementation for Image Stripe Noise Removal
1 Introduction
2 Stripe Denoising Strategy and Algorithm
2.1 Parallel Implementation via GPU-CUDA
3 Results
3.1 Quality Measures
3.2 Performance Analysis
4 Conclusions
References
DevOps for Open Source Multimedia Frameworks
1 Introduction
2 DevOps Architecture in the Open Source
3 Methodology
3.1 Example of FFmpeg/GStreamer DevOps Deployment
3.2 VAAPI-Fits, Pre-checkin and Cartwheel
4 Conclusion
References
Designing a Cost-Efficient Network for a Small Enterprise
1 Introduction
2 Related Works
2.1 Reduce the Cost of Labour
2.2 Adjusting the Network Layers
2.3 Use Wireless More
2.4 Cheaper and Scalable Hardware
2.5 Cheaper and Scalable Hardware
3 Problem Statement
3.1 Building Layout
3.2 Services
3.3 Functional and Non-functional Requirements
4 Proposed Approach
4.1 Topology
4.2 Wireless
4.3 Energy
5 Simulation Results
5.1 Simulations and Ping Result of BTC Old Network
5.2 Simulations and Ping Result of BTC New Network
6 Evaluation
6.1 Throughput
6.2 Energy Consumption
6.3 Cleaner Environment
6.4 Topology Adjustment
6.5 Running Cost/Saving
7 Discussion
8 Conclusion
References
Staff Rostering Optimization: Ideal Recommendations vs. Real-World Computing Challenges
1 Staff Rostering
2 Ideal Recommendations
3 Computational Intelligence Framework
4 Recommendations vs. Practice
5 Academic and Practical Findings
6 Conclusions and Future Work
References
Resolution of the Frobenius Problem with an Adiabatic Quantum Computer
1 Numerical Semigroups and the Frobenius Problem
2 The Ising Spin Problem in Adiabatic Quantum Computing
3 Resolution of the Frobenius Problem with an Adiabatic Quantum Computer
4 A Word on Conclusions
References
The Design of Customizable Distributed Algorithms for InDiGO Framework
1 Introduction
2 Problem Motivation and Related Work
2.1 Problem Motivation
2.2 Related Work
3 Development of Customizable Algorithms
3.1 Interaction Sets
3.2 Membership Criteria for Interaction Sets
3.3 Rules for Dynamic Updates to the Interaction Sets
4 Mutual Exclusion Algorithm Example
4.1 The Mutual Exclusion Problem
4.2 Description of Lamport's Mutual Exclusion Algorithm
4.3 Customized Version
5 Conclusion and Future Work
5.1 Conclusion
5.2 Future Work
References
Computational Optimization of Voids on 3D Woven Composites Truss Structures During Infusion
1 Introduction
2 Isotropic Flow Simulations
3 Results and Discussion
4 Results and Discussion
5 Results
6 Discussion
7 Conclusions
References
Towards AGI: Cognitive Architecture Based on Hybrid and Bionic Principles
1 Introduction
2 Principles of the Hybrid Paradigm of Artificial Intelligence
3 Bionic Principles Suitable for Cognitive Architecture
4 The Architecture of Artificial Intelligent Agent
5 Conclusion
References
An Array Abstraction to Amortize Reasoning About Parallel Client Code
1 Introduction
1.1 Clean Semantics and Shared Data in the Presence of Parallelism
1.2 The Interference Contract and Partitions
2 A Motivating Example
2.1 Verification Challenges with This Approach
3 The Splittable_Array Abstraction
3.1 Divide-and-Conquer Client Code Using SplittableArray
3.2 Reusability and Modifiability
3.3 Efficiently Realizing SplittableArray
4 A Hierarchy of Array Abstractions
4.1 Three Distinct Array Abstractions
4.2 Layered Implementations
5 Related Work
6 Conclusions
References
Parallel Accurate and Reproducible Summation
1 Introduction
2 Notations and Background
2.1 Floating-Point Arithmetic
2.2 Related Work
2.3 Demmel and Hida Algorithms
3 Accurate Summation
3.1 Accurate Summation in Sequential
3.2 Accurate Summation in Parallel
4 Numerical Experiments
4.1 Numerical Accuracy of the Algorithms
4.2 Reproducibility
4.3 Our Sequential Algorithm Vs Demmel and Hida Algorithm
5 Conclusion
References
Cell Assembly-based Task Analysis (CAbTA)
1 Introduction
1.1 Cell Assemblies and Brains
1.2 Task Analysis and Symbolic Artificial Intelligence
1.3 Cell Assembly-based Task Analysis
2 The Cell Assembly-based Task Analysis Models
2.1 The Simplified Cell Assembly Model (SCAM)
2.2 The Cell Assembly Architecture Relationship (CAAR) Diagram
2.3 The Main Analysis Document (MAD)
3 Method
3.1 The Task
3.2 Data Collection
3.3 CAbTA Method
4 Results
4.1 Time Results
4.2 SCAM Results
4.3 CAAR Results
5 Discussion
5.1 Cell Assembly-based Task Analysis
5.2 Artificial Intelligence, Neuroscience and Psychology
6 Conclusions
References
PDSA Computing
1 Changing World
2 Discover How It Behaves
2.1 Try Models One After Another
2.2 Sensing and Actuation
3 Human Movement
4 Euclidean Space and Non-euclidean Space
4.1 Mahalanobis Distance (MD)
4.2 Mahalanobis-Taguchi System (MTS)
5 Static to Dynamic
6 Perception Motivation Decision Making Action Emotion (PMDAE) Cycle
7 Performance Indicator
8 Mahalanobis Distance-Pattern (MDP) Approach
9 Natural Computing
9.1 Mind-Body-Brain
9.2 The Octopus
References
An Iterative Randomized Rounding Algorithm for the k-Clustering Minimum Completion Problem with an Application in Telecommunication Field
1 Introduction
2 Related Work
3 A Randomized Rounding Algorithm
3.1 A Basic Rounding Procedure
3.2 A Modified Rounding Strategy-Based Method
3.3 Local Constructive Search
3.4 An Iterative Randomized Rounding Algorithm
4 Computational Results
4.1 The First Phase of the Randomized Rounding Procedure
4.2 Effect of the Hill-Climbing Strategy
4.3 Behavior of IRRA Vs Other Methods
5 Conclusion
References
Non-dominated Ranking Biogeography Based Optimization Algorithm for Virtual Machine Placement in Cloud Computing
1 Introduction
2 Background
3 Problem Formulation
3.1 Power Consumption
3.2 Resource Wastage
4 The Non-dominated Ranking Biogeography Based-Optimization
4.1 Biogeography Based Optimization
4.2 The Adapted BBO for the VMP Problem: NRBBO
5 Tests and Results
5.1 Comparison of NRBBO with VMPACS and VMPMBBO
5.2 Comparison of NRBBO with NSGA II
6 Conclusion
References
Multi-core Aware Virtual Machine Placement for Cloud Data Centers with Constraint Programming
1 Introduction
1.1 Virtualization
1.2 Virtual Machine
1.3 Data Center
1.4 VM Placement
1.5 Motivation
1.6 Problem Statement
2 Related Work
3 Multi-core Aware VMP Model using Constraint Programming (MCA-VMP)
3.1 Resource Allocation Manager
3.2 MCA-VMP Problem Formulation
3.3 MCA-VMP with Constraint Programming
3.4 Performance Evaluation
3.5 MCA-VMP Algorithm
3.6 Simulation Setup
3.7 Simulation Results
3.8 Simulation Analysis
4 Conclusion and Future Work
References
Finding Critical Packet-Drop Levels of Streaming at Cloud Edge Networks and the Proposed Solution
1 Introduction
2 Testbed Network Architecture
2.1 Application Configuration
2.2 Profile Configuration
2.3 Background Traffic
3 Performance Results and Proposed Solution
3.1 Effect of Background Traffic
3.2 QoS Voice Versus Video Traffic
3.3 Dropped Traffic
3.4 Throughput
3.5 Network Utilization
4 Conclusion
References
Cyber Security in Cloud: Risk Assessment Models
1 Introduction
2 Review of the Literature of Cloud Risk Assessment Models
2.1 Models of the Literature
2.2 Conclusions of the Literature Review
2.3 Research Problem
3 Characteristics of the Theoretical Cloud Risk Assessment Reference Model
4 Classification of Cloud Risk Assessment Models
5 Related Work
6 Conclusions and Future Work
References
In-Place VM Migration for Fast Update of Hypervisor Stack
1 Introduction
2 Virtualization Appliance
3 VM In-Place Live Migration
4 Update Methods
4.1 Partial Update
4.2 Kexec Update
4.3 Reboot Through Firmware Update
5 Future Works
5.1 Pre-booted Update
5.2 Multi-OS Update
6 Conclusion
References
Robustness Comparison of Scheduling Algorithms in MapReduce Framework
1 Introduction and Related Work
2 System Model
3 Load Balancing Algorithms
3.1 Priority Algorithm
3.2 Balanced-PANDAS Algorithm
3.3 MaxWeight Based Algorithm
4 Robustness Comparison of Scheduling Algorithms
5 Conclusion and Future Work
References
Seamless Update of a Hypervisor Stack
1 Introduction
2 Background
2.1 Faster Reboot
2.2 Partial HV Stack Update
2.3 Seamless Upgrades
3 Upgrade Methods
3.1 HVPlexing
3.2 HVInception
3.3 HVPartitioning
4 Device Migration
5 Conclusion
References
Developing a Framework of Critical Factors Affecting the Adoption of Cloud Computing in Government Systems (ACCE-GOV)
1 Introduction
2 Theoretical Background
2.1 Overview of Cloud Computing
2.2 Adoption of the Cloud
3 Research Model and Hypotheses
3.1 Theoretical Models
4 Research Model
4.1 Technological Context
4.2 Organizational Context
4.3 Environmental Context
4.4 Social Context
5 Research Methodology
5.1 Measurement
5.2 Data
6 Results
6.1 Confirmatory Factor Analysis (CFA)
6.2 Validity and Reliability Assessment
7 Data Analysis
7.1 Structural Equation Model (SEM)
7.2 Descriptive Statistics
8 Discussion
8.1 Technology Dimension
8.2 Organizational Dimension
8.3 Environmental Dimension
8.4 Social Dimension
8.5 Implications for Practice
8.6 Implications for Theory
9 Research Limitations
10 Conclusions
References
Approximate Analysis of Deviant Behavior on Social Media
1 Introduction
2 State of the Art
3 Solution Methodology
4 Intelligent Technology Application
5 Implementation Results
6 Conclusion
References
Investigating the Characteristics of the Response Waiting Time in a Chat Room
1 Introduction
2 Related Work
3 Waiting Time: Analysis Methodology
3.1 What is the True Nature of RWT?
3.2 Comparing Results from the Fitted Power Law with Other Distributions
4 Factors Affecting RWT in Pair Conversation
4.1 The Impact of Number of Message Exchange on Response Waiting Time
4.2 Dynamism of a Single User's RWT with Other Users at Different Chat Sessions
4.3 Time of Day, Day of Week and RWT Interaction
4.4 Results Compared with Previous Models
5 Discussion
6 Conclusion
References
Some Socio-demographic Usage Patterns in Ephemeral Content Sharing Social Networks: The Case of Snapchat
1 Introduction
2 Related Work
3 Methodology
3.1 Procedure
3.2 Survey and Participants
4 Obtained Results
4.1 Age Groups
4.2 Level of Education
4.3 Gender
4.4 Marital Status
5 Discussion
6 Conclusion and Future Work
References
Automatic Extraction of Emojiset
1 Introduction
2 Related Work
3 Methodology of Extracting the Emojisets
4 Dataset
5 Emojiset Analysis
5.1 Control Emojisets
5.2 NBA Emojisets
5.3 Politics Emojisets
5.4 Epidemic Emojisets
5.5 Education Emojisets
5.6 Cars Emojisets
6 Analysis of Sequences of Emojisets
7 Conclusion and Future Work
Appendix
References
How Twitter Interactions Leak Political Trends
1 Introduction
2 Background of Twitter Political Tendency
3 Political Interaction Modeling
4 Political Interaction Analysis
4.1 User Interactions Dataset
4.2 Inspection Processes
5 Interaction Weights
5.1 Interaction Weight Algorithm (IWA)
5.2 IWA Evaluation
6 Users Interaction Analysis Strategy (UIAS)
6.1 ETL Engine
6.2 Data Warehouse
6.3 Tendency Processing
7 UIAS Evaluation
8 Conclusions
References
Public Reaction on Social Media During COVID-19: A Comparison Between Twitter and Weibo
1 Introduction
2 Background
2.1 Power of Social Media Analysis
2.2 Importance of Social Media After Crisis
2.3 COVID-19 Related Works
3 Method
3.1 Data Collection
3.2 Data Analysis
4 Results
4.1 Trends on Volume
4.2 Sentiment
4.3 Most Posted Topics
4.4 Top Mentions
4.5 Content of Posts
5 Discussion
6 Limitations
7 Conclusion
References
Word Frequency Analysis of Community Reaction to Religious Violence on Social Media
1 Introduction
2 Background
3 Method
4 Results
5 Conclusion
References
Humans Digital Avatar Reconstruction for Tactical Situations Animation
1 Introduction
2 Related Work
3 Formal Task Description
4 Method Description
4.1 Video Preprocessing
4.2 Extracting the Background Image
4.3 Human Poses Detection and Estimation
4.4 Calculation of the ``Color'' Parameter of Body Parts
4.5 Constructing the Graphical Representation of the Scene
5 Results
References
Action Recognition in Handball Scenes
1 Introduction
2 Experiment Workflow for Action Recognition in Handball
2.1 The Dataset
2.2 The Metrics
2.3 The Action Recognition Models
3 Experimental Results and Observations
3.1 Comparison of the Performances of Action Recognition Models
3.2 Analysis of Model Performance with Respect to the Frame Selection Strategy
4 Conclusion
References
Application of Robotics to Domestic and Environmental Cleanup Tasks
1 Introduction
2 Inside the Home: Case Study - Laundry
2.1 Clothes to the Washer
2.2 Clothes from the Washer
3 Inside the Home: Case Study – Cleaning Up Toys
4 Outside the Home: Case Study – Gutter Cleaning
5 In the Wider World: Case Study – Environmental Cleanup
6 Conclusions
References
RF Mapping for Sensor Nodes Connectivity and Communication Signal Recovery
1 Introduction
2 RF Mapping Versus Physical Mapping
2.1 Modeling of RF Mapping Recognition Method
2.2 RF Signal Strength Measurements
3 Physical Obstacles Experiments
3.1 Copper Wall Obstacle
3.2 Copper Cage Obstacle
4 Simulation and Physical Results Validation
4.1 Wall Results Validation
4.2 Cage Results Validation
5 Gradient Descent Method
6 Gradient Computation of the RF Signal
7 Gradient Descent Formulation
8 Various Gradient Scenarios
9 Gradient Algorithm Experimental Results
10 Conclusion
References
DBA-Filter: A Dynamic Background Activity Noise Filtering Algorithm for Event Cameras
1 Introduction
2 Dynamic Vision Sensors
2.1 Principle of Operation
3 Filtering
3.1 TS-Filter
3.2 BA-Filter
4 Results
4.1 Experimental Setup
4.2 Qualitative Evaluation
4.3 Quantitative Evaluation
5 Conclusion
References
Gamification in a Learning Resource for the Study of Human Computer Interaction
1 Introduction
1.1 HCI Pedagogic Challenges
1.2 HCI Educational Aspirations
2 Pedagogy for HCI
2.1 Blended Learning
2.2 Dialogic Feedback
2.3 Simulations
2.4 Gamification
2.5 Summary
3 Designing a Gamified Learning Experience
4 Methodology
5 Results
5.1 Attendance and Attainment
5.2 Student Interviews
6 Discussion
6.1 Blended Learning
6.2 HCI Requirements
6.3 Simulation and Dialogic Feedback
6.4 Gamified Features
7 Conclusions
Appendix 1
References
Machine Learning Control for Mobile Robot by Approximation Extremals by Symbolic Regression
1 Introduction
2 Statement of the Control System General Synthesis Problem
3 Solving the Optimal Control Problem
4 Variation Cartesian Genetic Programming
5 Computation Experiment
References
Towards Synchronous Model of Non-emotional Conversational Gesture Generation in Humanoids
1 Introduction
2 Background
2.1 Fuzzy Semantic Algebra for Gestures
2.2 Synchronized Motions in Gestures
2.3 Modeling Head-Based Conversational Gestures
2.4 Haptics
2.5 Conceptual Graphs and Dependencies
3 Related Works
3.1 Cognitive Hand-Gesture Classification
3.2 Recognizing Simple Hand-Gestures
3.3 Gesture Generation in Humanoids
4 An Extended Gesture Classification
4.1 Integrated Gestures
5 Linguistic Gestures Modeling
6 Future Work
References
MQTT Based Appliance Control and Automation with Room Occupancy Monitoring Using YOLO
1 Introduction
2 Methodology
2.1 Hardware Components and Software Protocols
2.2 Room Occupancy Accounting Using Video Processing
2.3 Implementation of MQTT-Based Sensors
2.4 Appliance Control and Automation System Architecture
3 Results and Discussion
3.1 Room Occupancy Accounting
3.2 Appliance Control and Automation System
4 Conclusion
References
Multi-scale Defect Detection Network for Tire Visual Inspection
1 Introduction
2 Multi-scale Defect Detection Network
2.1 Semantic-Aware Network
2.2 Texture-Aware Network
2.3 Multi-scale Network Integration
2.4 Preprocessing and Training
3 Experiments
3.1 Implementation Details
3.2 Ablation Experiments
3.3 Comparison with State-of-the-Art Methods
4 Conclusion
References
Guided Filter Bank
1 Introduction
2 Related Works and Motivations
3 Guided Filter Bank
3.1 Case 1 : h=2
3.2 Case 2 : h=1
3.3 Case 3 : h=12
4 Analysis of the Filters
4.1 Properties of GF and L1 GF
4.2 Property of L0.5 GF
5 Applications
5.1 Image Smoothing
5.2 Image Denoising
5.3 Edge Detection
5.4 Image Detail Enhancement
5.5 X-Ray Image Enhancement
6 Conclusion
References
High-Speed Multi-person Tracking Method Using Bipartite Matching
1 Introduction
2 Related Work
3 Our Approach
3.1 Single Frame Pose Estimation
3.2 Tracking Task Description and Definitions
3.3 Cost Function
3.4 Visual Feature Matching
3.5 Track Matching and Update
4 Experiments
4.1 The PoseTrack Dataset
4.2 Experiment Details
4.3 Results
5 Discussion
5.1 Dataset Challenges
5.2 Study Limitations and Future Research
6 Conclusion
References
Modular Image Synthesizer for Annotated Test Sets on Incremental Parameter Fields
1 Introduction, Related Work
2 Ground Truth Synthesizer
2.1 Overview
2.2 System Structure
2.3 Rendering and Saving of Metadata
2.4 Parameter
2.5 Metadata
3 Dataset Generation First Run
4 Dataset Testing: Running Object Detector on the Generated Dataset
5 Summary and Outlook
References
Learning Action Correlation and Temporal Aggregation for Group Representation
1 Introduction
2 Related Work
3 Method
3.1 Visual Feature Extraction
3.2 Action Correlation
3.3 Temporal Aggeration
3.4 Inference
4 Experiments
4.1 Datasets
4.2 Implementation Details
4.3 Results and Analysis
5 Conclusions
References
Self-adaptive Perception Model for Action Segment Detection
1 Introduction
2 Related Work
2.1 Action Analysis
2.2 Action Detection
3 Method
3.1 Problem Formulation
3.2 Our Proposed Model (Fig.1)
3.3 Visual Encoding (Fig.2)
3.4 Boundary Prediction (Fig.3)
3.5 Discriminator
3.6 Learning
4 Experiments
4.1 Dataset and Setup
4.2 Details
4.3 Performance
4.4 Ablation Study
4.5 Comparison with Published Methods
5 Conclusion
References
Hierarchical Feature Generating Network for Zero-Shot Learning by Knowledge Graph
1 Introduction
2 Related Work
2.1 Zero-Shot Learning
2.2 Generative Adversarial Network
3 Approach
3.1 Problem Definition
3.2 An Overview of Framework
3.3 Aggregation Model
3.4 Hierarchical-GAN for Feature Generation
3.5 Zero-Shot Classification and Iteratively Learning
3.6 Network Details
4 Experiments
4.1 Datasets
4.2 Evaluations
4.3 Effectiveness of HGAN
4.4 GZSL Results
5 Discussion and Conclusion
References
Comparing Partitions: Metric Characterizations, Mean Partition, and Distortion
1 Introduction
2 The Space of Partitions
3 Characterizing Measures for Comparing Partitions
3.1 Characterizing Metrics Associated to Lower Valuations
3.2 Characterizing Metrics Associated to Upper Valuations
4 Consensus of Partitions
4.1 Mean Partition and q-Quota Rules
4.2 VI, dK and d in the Ambit of Consensus
5 Reduction of the Search Space
6 How Does the Space of Partitions Distort When a Metric d is Replaced by Another d'?
7 Conclusions
References
A Preliminary Comparison Between Compressive Sampling and Anisotropic Mesh-Based Image Representation
1 Introduction
2 Compressive Sampling and Mesh-Based Image Representation
2.1 Compressive Sampling
2.2 AMA Representation
3 Comparison Results
3.1 Comparison Metrics
3.2 Comparison Results
4 Conclusions
References
Self-exciting Point Processes with Image Features as Covariates for Robbery Modeling
1 Introduction
2 Related Work
3 Methodology
4 Implementation
5 Results
6 Deployment for the Allocation of Law Enforcement Resources
6.1 Hotspots Prediction
6.2 Surveillance Cameras System
6.3 Location-Allocation of Police Stations
7 Conclusions
References
Development of an Automatic Document to Digital Record Association Feature for a Cloud-Based Accounting Information System
1 Introduction
2 Related Literature
2.1 Image Enhancing Methods
2.2 Use of Optical Character Recognition Technology
2.3 Information Retrieval
2.4 Automated Association of Digitized Documents to Digital Records
3 Methodology
3.1 Information System Development
3.2 Performance Evaluation Method
3.3 Image Cropping
3.4 Image Color Pre-processing
3.5 Information Retrieval
3.6 Database Record Association
4 Results and Discussion
4.1 Results
5 Conclusion
References
Improving Accuracy and Latency in Image Re-identification by Gallery Database Cleansing
1 Introduction
1.1 Nomenclature
2 Background
3 Problem Definition
4 Related Work
5 Our Approach
6 Results
7 Future Work/Work in Progress
8 Conclusion
References
Real-Time Crack Detection Using ROV
1 Introduction
2 Method
2.1 ROV Design
2.2 Classical Detection Algorithm
3 Results and Discussions
4 Conclusion
References
Web-Based Learning for Cultural Adaptation: Constructing a Digital Portal for Persian Speaking Immigrants in Finland
1 Introduction
2 Context of the Study
3 Background of the Study
4 The Designed Digital Portal for Cultural Adaptation
4.1 Theory to Practice
4.2 Designing the Portal
4.3 Driving a Practical Test Website
5 Principles for Creating the Website
5.1 Sensible Classification
5.2 Simplicity
5.3 Meaningfulness
5.4 Minimality
5.5 Clear Instruction
5.6 Feedback
6 Results
7 Limitations
8 Conclusion
References
Enhanced Reinforcement Learning Model for Extraction of Objects in Complex Imaging
1 Introduction
2 Related Work
2.1 Region-Based Segmentation
2.2 Edge-Based Segmentation
2.3 Clustering-Based Segmentation
2.4 Other Method
2.5 Region Growing
3 Results of Our Algorithm
3.1 Our Algorithm and Results
4 Conclusions
References
Causal BERT: Language Models for Causality Detection Between Events Expressed in Text
1 Introduction
2 Event-Structure and Causality in Natural Language Text
3 Dataset
4 Problem Definition and Proposed Methodology
4.1 Methodology
4.2 Problem Definition
4.3 Models
5 Experiments and Results
5.1 Experiments
5.2 Learning Setup: Experiment Settings
5.3 Results and Analysis
6 Related Work
7 Conclusion
References
Social Media Mining for Business Intelligence Analytics: An Application for Movie Box Office Forecasting
1 Introduction
2 Theoretical Background
3 Methodology
3.1 Sample and Information Sources
3.2 Mining Methods
3.3 Variables
3.4 Methods
4 Results
5 Discussion
6 Conclusions
References
Solving the Story Cloze Test Using Graph Feature Extraction
1 Introduction
2 Related Works
3 Data
3.1 SCTv1.0
3.2 SCTv1.5
4 Models and Methods
4.1 NLP Information
4.2 Word Embeddings
4.3 Sentiment Analysis
4.4 Sequential Graphs
4.5 Graph Analysis
4.6 Graph Featurization
4.7 ML Models
4.8 Metrics
5 Experiments
5.1 Word Embedding Results
5.2 Sentiment Results
5.3 NLP Integration Results
5.4 Cluster Analysis
5.5 ML Results
6 Analysis
6.1 Word Embedding
6.2 Sentiment Analysis
6.3 NLP Integration
6.4 Cluster Analysis
6.5 ML Models
6.6 Error Analysis
7 Conclusion
References
MegaLite-2: An Extended Bilingual Comparative Literary Corpus
1 Introduction
2 Related Works
3 The MegaLite-2 Corpora
3.1 Pre-processing
3.2 The Spanish Corpus: Megalite-Es
3.3 The French Corpus MegaLite-Fr
4 Word2vec Training: Embeddings
5 Study on the Parallel Corpora
5.1 POS Version
5.2 Lemmas Version
6 N-grams Language Model
7 Study Using Divergence of Probability Distribution
7.1 Jensen-Shannon Divergence
7.2 DJS Results
8 Conclusion and Final Remarks
References
VBSRL: A Semantic Frame-Based Approach for Data Extraction from Unstructured Business Documents
1 Introduction and Motivations
1.1 Motivations
2 State of the Art
3 Theory of VBSRL
3.1 VBSRL by Examples
4 Conclusions
References
Generating Correction Candidates for OCR Errors using BERT Language Model and FastText SubWord Embeddings
1 Introduction
2 Literature Review
3 Methodology
3.1 BERT
3.2 FastText
4 Results
5 Conclusion and Future Work
References
Measuring Meaning
1 Introduction
2 Semantic Representation
3 Participants and Cognate Semantic Roles
4 Related Work
5 The Raison d’être of the Pairings
6 Question Answering
7 Methodology
8 Meaning Measured
9 Concluding Remarks
References
The Impact of Agile Methods and “DevOps” on Day 2+ Operations for Large Enterprises
1 Introduction
2 Background
3 Understanding Day 2+ Operations
3.1 Challenge: Sustainment in a Continuous Engineering Model
3.2 Challenge: Balancing Information Technology (IT) Operations
4 Case Study
5 Enabling Operations to Take Advantage of Agile and DevOps
5.1 Enhance Day2 + Operations and Continuous Sustainment
5.2 Encourage Community Operations Collaboration
5.3 Empower Operators as “Citizen Coders”
6 Conclusions
References
Mapping Risk Assessment Strategy for COVID-19 Mobile Apps’ Vulnerabilities
1 Introduction
2 Background
3 Method
4 Result and Possible Solutions
4.1 Credential Leaks
4.2 Insecure Communication
4.3 HTTP Request Libraries
5 App’s Privacy Risk Assessment Strategy
6 Discussion
7 Conclusion
References
Explainable Graph-Based Search for Lessons-Learned Documents in the Semiconductor Industry
1 Introduction and Research Purpose
2 Background and Related Work
3 Search Methods
3.1 Multidimensional Domain-Specific Knowledge Graph
3.2 Explainable Graph Search Functionality
3.3 Search Interface and Feedback Collection
4 Evaluation and Results
5 Conclusion and Future Perspectives
References
Improving the Efficiency of Industrial Enterprise Management Based on the Forge Software-analytical Platform
1 Introduction
2 Scientific-Technical Solution
2.1 Increasing the Productivity of the Technological Processes
2.2 Increasing the Productivity of Process Control
2.3 Increasing Asset Productivity
2.4 Increasing Staff Productivity
2.5 Increasing the Productivity of the Enterprise as a Whole
2.6 Cyber-Security Aspect in Forge Platform
3 Conclusion
References
Consumer-Oriented Web of Things Solution for Competitive Swimmers
1 Introduction
2 Related Work
3 Solution Approach
4 System Overview
5 Limitations
6 Implementation Details
7 Validation and Evaluation
8 Future Work
9 Conclusion
References
Factors Influencing the Adoption of Online Banking in South Africa: An UTAUT Study
1 Introduction
2 Online Banking
3 Theoretical Framework
3.1 Unified Theory of Acceptance and Use of Technology
3.2 Development of Hypotheses
3.3 Moderating Factors
4 Research Model
5 Research Methodology
6 Data Analysis
7 Future Work
8 Conclusion
References
Machine Learning Application in LAPIS Agile Software Development Process
1 Introduction
2 Related Studies
3 LAPIS (Logo Agile Process Improvement System)
4 Retrospective Study
5 Measurement and Improvement Method
6 Machine Learning Sentiment Analysis
7 Sentiment Analysis of Retrospective Data
7.1 Data Preprocessing
7.2 Modeling
7.3 Evaluation
8 Results and Future Studies
References
Transforming HR and Improving Talent Profiling with Qualitative Analysis Digitalization on Candidates for Career and Team Development Efforts
1 Introduction
2 Creating the Questionnaire Set
3 Text Analysis
4 Data Analysis and Visualizations
5 Population in Group
6 Similarity in a Group
7 Discussion
8 Conclusions and Future Research Recommendations
References
Author Index

Citation preview

---

Lecture Notes in Networks and Systems 283

Kohei Arai   Editor

Intelligent Computing Proceedings of the 2021 Computing Conference, Volume 1

Lecture Notes in Networks and Systems Volume 283

Series Editor Janusz Kacprzyk, Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland Advisory Editors Fernando Gomide, Department of Computer Engineering and Automation—DCA, School of Electrical and Computer Engineering—FEEC, University of Campinas— UNICAMP, São Paulo, Brazil Okyay Kaynak, Department of Electrical and Electronic Engineering, Bogazici University, Istanbul, Turkey Derong Liu, Department of Electrical and Computer Engineering, University of Illinois at Chicago, Chicago, USA; Institute of Automation, Chinese Academy of Sciences, Beijing, China Witold Pedrycz, Department of Electrical and Computer Engineering, University of Alberta, Alberta, Canada; Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland Marios M. Polycarpou, Department of Electrical and Computer Engineering, KIOS Research Center for Intelligent Systems and Networks, University of Cyprus, Nicosia, Cyprus Imre J. Rudas, Óbuda University, Budapest, Hungary Jun Wang, Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong

The series “Lecture Notes in Networks and Systems” publishes the latest developments in Networks and Systems—quickly, informally and with high quality. Original research reported in proceedings and post-proceedings represents the core of LNNS. Volumes published in LNNS embrace all aspects and subfields of, as well as new challenges in, Networks and Systems. The series contains proceedings and edited volumes in systems and networks, spanning the areas of Cyber-Physical Systems, Autonomous Systems, Sensor Networks, Control Systems, Energy Systems, Automotive Systems, Biological Systems, Vehicular Networking and Connected Vehicles, Aerospace Systems, Automation, Manufacturing, Smart Grids, Nonlinear Systems, Power Systems, Robotics, Social Systems, Economic Systems and other. Of particular value to both the contributors and the readership are the short publication timeframe and the world-wide distribution and exposure which enable both a wide and rapid dissemination of research output. The series covers the theory, applications, and perspectives on the state of the art and future developments relevant to systems and networks, decision making, control, complex processes and related areas, as embedded in the fields of interdisciplinary and applied sciences, engineering, computer science, physics, economics, social, and life sciences, as well as the paradigms and methodologies behind them. Indexed by SCOPUS, INSPEC, WTI Frankfurt eG, zbMATH, SCImago. All books published in the series are submitted for consideration in Web of Science.

More information about this series at http://www.springer.com/series/15179

Kohei Arai Editor

Intelligent Computing Proceedings of the 2021 Computing Conference, Volume 1

123

Editor Kohei Arai Faculty of Science and Engineering Saga University Saga, Japan

ISSN 2367-3370 ISSN 2367-3389 (electronic) Lecture Notes in Networks and Systems ISBN 978-3-030-80118-2 ISBN 978-3-030-80119-9 (eBook) https://doi.org/10.1007/978-3-030-80119-9 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Editor’s Preface

It is a great privilege for us to present the proceedings of the Computing Conference 2021, held virtually on July 15 and 16, 2021. The conference is held every year to make it an ideal platform for researchers to share views, experiences and information with their peers working all around the world. This is done by offering plenty of networking opportunities to meet and interact with the world-leading scientists, engineers and researchers as well as industrial partners in all aspects of computer science and its applications. The main conference brings a strong program of papers, posters, videos, all in single-track sessions and invited talks to stimulate significant contemplation and discussions. These talks were also anticipated to pique the interest of the entire computing audience by their thought-provoking claims which were streamed live during the conferences. Moreover, all authors had very professionally presented their research papers which were viewed by a large international audience online. The proceedings for this edition consist of 235 chapters selected out of a total of 638 submissions from 50+ countries. All submissions underwent a double-blind peer-review process. The published proceedings has been divided into three volumes covering a wide range of conference topics, such as technology trends, computing, intelligent systems, machine vision, security, communication, electronics and e-learning to name a few. Deep appreciation goes to the keynote speakers for sharing their knowledge and expertise with us and to all the authors who have spent the time and effort to contribute significantly to this conference. We are also indebted to the organizing committee for their great efforts in ensuring the successful implementation of the conference. In particular, we would like to thank the technical committee for their constructive and enlightening reviews on the manuscripts in the limited timescale. We hope that all the participants and the interested readers benefit scientifically from this book and find it stimulating in the process.

v

vi

Editor’s Preface

Hope to see you in 2022, in our next Computing Conference, with the same amplitude, focus and determination. Kohei Arai

Contents

Analytical View on Non-invasive Measurement of Moving Charge by Various Topologies of Wannier Qubit . . . . . . . . . . . . . . . . . . . . . . . . Krzysztof Pomorski

1

Multi-population Genetic Algorithm with the Actor Model Approach to Determine Optimal Braking Torques of the Articulated Vehicle . . . . Kornel Warwas and Szymon Tengler

56

A Novel Three-Way Merge Algorithm for HTML/XML Documents Using a Hidden Markov Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nikolaos G. Bakaoukas and Anastasios G. Bakaoukas

75

Scaling Out Transformer Models for Retrosynthesis on Supercomputers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Joris Mollinga and Valeriu Codreanu Naming Processes in Multichannels with Beeps in the Weak Model . . . 118 Layla S. Aldawsari and Tom Altman Distributed Evolution of Deep Autoencoders . . . . . . . . . . . . . . . . . . . . . 133 Jeff Hajewski, Suely Oliveira, and Xiaoyu Xing Computational Power of a Hybrid Algorithm for Solving the Multiple Knapsack Problem with Setup . . . . . . . . . . . . . . . . . . . . . . . . . 154 Samah Boukhari, Isma Dahmani, and Mhand Hifi Accelerated Quantum Computation based on Quantum Coherent Dynamics of Evanescent Photons in Liquid Water . . . . . . . . . . . . . . . . . 169 Luigi Maxmilian Caligiuri Computer Scientist’s and Programmer’s View on Quantum Algorithms: Mapping Functions’ APIs and Inputs to Oracles . . . . . . . . 188 Ilie-Daniel Gheorghe-Pop, Nikolay Tcholtchev, Tom Ritter, and Manfred Hauswirth

vii

viii

Contents

Orca: A Software Library for Parallel Computation of Symbolic Expressions via Remote Evaluation on MPI Systems . . . . . . . . . . . . . . . 204 Ahmet Artu Yıldırım Solving High-Dimensional Nonlinear Equations with Infinite Solutions by the SVM Visualization Method . . . . . . . . . . . . . . . . . . . . . 218 Yu-Yuan Lin and Jeng-Nan Tzeng A Novel GPU Implementation for Image Stripe Noise Removal . . . . . . . 232 Pasquale De Luca, Ardelio Galletti, and Livia Marcellino DevOps for Open Source Multimedia Frameworks . . . . . . . . . . . . . . . . 244 Juan Zhao, Ullysses Eoff, Focus Luo, Guangxin Xu, Haihao Xiang, and Hongcheng Zhong Designing a Cost-Efficient Network for a Small Enterprise . . . . . . . . . . 255 Lucky Osemwengie, Fahimeh Jafari, and Amin Karami Staff Rostering Optimization: Ideal Recommendations vs. Real-World Computing Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 Kimmo Nurmi, Jari Kyngäs, and Nico Kyngäs Resolution of the Frobenius Problem with an Adiabatic Quantum Computer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 J. Ossorio-Castillo and José M. Tornero The Design of Customizable Distributed Algorithms for InDiGO Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 Valeriy Kolesnikov Computational Optimization of Voids on 3D Woven Composites Truss Structures During Infusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326 Spiridon Koutsonas and Hasan Haroglu Towards AGI: Cognitive Architecture Based on Hybrid and Bionic Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 R. V. Dushkin An Array Abstraction to Amortize Reasoning About Parallel Client Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 Alan Weide, Paolo A. G. Sivilotti, and Murali Sitaraman Parallel Accurate and Reproducible Summation . . . . . . . . . . . . . . . . . . 363 Farah Benmouhoub, Pierre-Loic Garoche, and Matthieu Martel Cell Assembly-based Task Analysis (CAbTA) . . . . . . . . . . . . . . . . . . . . 383 Dan Diaper and Chris Huyck PDSA Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 Shuichi Fukuda

Contents

ix

An Iterative Randomized Rounding Algorithm for the k-Clustering Minimum Completion Problem with an Application in Telecommunication Field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 Mhand Hifi and Shohre Sadeghsa Non-dominated Ranking Biogeography Based Optimization Algorithm for Virtual Machine Placement in Cloud Computing . . . . . . 423 Asma Bouhank and Mourad Daoudi Multi-core Aware Virtual Machine Placement for Cloud Data Centers with Constraint Programming . . . . . . . . . . . . . . . . . . . . . . . . . . 439 Nagadevi and Kasmir Raja Finding Critical Packet-Drop Levels of Streaming at Cloud Edge Networks and the Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . 458 Nader F. Mir, Chandramouli Santhanam, and Kriti Sharma Cyber Security in Cloud: Risk Assessment Models . . . . . . . . . . . . . . . . 471 Carlos Bendicho In-Place VM Migration for Fast Update of Hypervisor Stack . . . . . . . . 483 Pavel Tatashin and William Moloney Robustness Comparison of Scheduling Algorithms in MapReduce Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494 Amirali Daghighi and Jim Q. Chen Seamless Update of a Hypervisor Stack . . . . . . . . . . . . . . . . . . . . . . . . . 509 Pavel Tatashin and William Moloney Developing a Framework of Critical Factors Affecting the Adoption of Cloud Computing in Government Systems (ACCE-GOV) . . . . . . . . . 520 Naif Al Mudawi, Natalia Beloff, and Martian White Approximate Analysis of Deviant Behavior on Social Media . . . . . . . . . 539 Anton Ivaschenko, Arkadiy Krivosheev, Anastasia Stolbova, and Pavel Sitnikov Investigating the Characteristics of the Response Waiting Time in a Chat Room . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 548 Gibson O. Ikoro, Raul J. Mondragon, and Graham White Some Socio-demographic Usage Patterns in Ephemeral Content Sharing Social Networks: The Case of Snapchat . . . . . . . . . . . . . . . . . 563 Ons Al-Shamaileh, Anas Aloudat, Amal Alrayes, and Hakim Hacid Automatic Extraction of Emojiset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 580 Phil Roesch, Caleb Franco, William Bernier, and Salem Othman How Twitter Interactions Leak Political Trends . . . . . . . . . . . . . . . . . . 591 M. Solé, F. Giné, and M. Valls

x

Contents

Public Reaction on Social Media During COVID-19: A Comparison Between Twitter and Weibo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 612 Tian Wang, Ian Brooks, and Masooda Bashir Word Frequency Analysis of Community Reaction to Religious Violence on Social Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 626 Samah Senbel, Carly Seigel, and Emily Bryan Humans Digital Avatar Reconstruction for Tactical Situations Animation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634 Ilya Zuev, Alexander Gilya-Zetinov, Alexander Khelvas, Egor Konyagin, and Julien Segre Action Recognition in Handball Scenes . . . . . . . . . . . . . . . . . . . . . . . . . 645 Kristina Host, Marina Ivasic-Kos, and Miran Pobar Application of Robotics to Domestic and Environmental Cleanup Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657 Amritha Ananthanarayanan, Chase G. Frazelle, Sowmya Kethireddy, Chen-Ho Ko, Rohan Kumar, Vignesh Prabhu, Bhargav Srinivas Vasudevan, and Ian D. Walker RF Mapping for Sensor Nodes Connectivity and Communication Signal Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 666 Mustafa Ayad and Richard Voyles DBA-Filter: A Dynamic Background Activity Noise Filtering Algorithm for Event Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685 Sherif A. S. Mohamed, Jawad N. Yasin, Mohammad-Hashem Haghbayan, Jukka Heikkonen, Hannu Tenhunen, and Juha Plosila Gamification in a Learning Resource for the Study of Human Computer Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 697 Dawn Carmichael, Claire MacEachen, and Jacqueline Archibald Machine Learning Control for Mobile Robot by Approximation Extremals by Symbolic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 718 Askhat Diveev Towards Synchronous Model of Non-emotional Conversational Gesture Generation in Humanoids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737 Aditi Singh and Arvind K. Bansal MQTT Based Appliance Control and Automation with Room Occupancy Monitoring Using YOLO . . . . . . . . . . . . . . . . . . . . . . . . . . . 757 Janine Grace B. Abad, Diana G. Romero, Jerome M. Dolalas, Raymark C. Parocha, and Erees Queen B. Macabebe Multi-scale Defect Detection Network for Tire Visual Inspection . . . . . . 771 Minghua Wei, Ren Wang, and Qiang Guo

Contents

xi

Guided Filter Bank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 783 Hui Yin, Yuanhao Gong, and Guoping Qiu High-Speed Multi-person Tracking Method Using Bipartite Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 793 Alexander Gilya-Zetinov, Alexander Bugaev, Alexander Khelvas, Egor Konyagin, and Julien Segre Modular Image Synthesizer for Annotated Test Sets on Incremental Parameter Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 810 Benny Platte, Rico Thomanek, Christian Roschke, Tony Rolletschke, Frank Zimmer, and Marc Ritter Learning Action Correlation and Temporal Aggregation for Group Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 823 Haoting Wang, Kan Li, and Xin Niu Self-adaptive Perception Model for Action Segment Detection . . . . . . . . 834 Jiahe Li, Kan Li, and Xin Niu Hierarchical Feature Generating Network for Zero-Shot Learning by Knowledge Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 846 Yi Zhang, Kan Li, and Xin Niu Comparing Partitions: Metric Characterizations, Mean Partition, and Distortion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 857 Jyrko Correa-Morris, Abel Urra-Yglesias, Estefano Reyes, Juan Martínez, and Belarmino Gonzalez A Preliminary Comparison Between Compressive Sampling and Anisotropic Mesh-Based Image Representation . . . . . . . . . . . . . . . . 876 Xianping Li and Teresa Wu Self-exciting Point Processes with Image Features as Covariates for Robbery Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 886 Mateo Dulce Rubio, Paula Rodríguez Díaz, Juan S. Moreno Pabón, Álvaro J. Riascos, and Jorge E. Camargo Development of an Automatic Document to Digital Record Association Feature for a Cloud-Based Accounting Information System . . . . . . . . . . 899 Daniel S. Jabonete and Marlene M. De Leon Improving Accuracy and Latency in Image Re-identification by Gallery Database Cleansing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 911 Niall O’. Mahony, Sean Campbell, Anderson Carvalho, Lenka Krpalkova, Daniel Riordan, and Joseph Walsh Real-Time Crack Detection Using ROV . . . . . . . . . . . . . . . . . . . . . . . . . 922 Haythem El-Messiry, Hany Khaled, Ahmed Maher, Amin Ahmed, and Faris Hussian

xii

Contents

Web-Based Learning for Cultural Adaptation: Constructing a Digital Portal for Persian Speaking Immigrants in Finland . . . . . . . . 930 Zahra Hosseini and Jussi Okkonen Enhanced Reinforcement Learning Model for Extraction of Objects in Complex Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 946 Usman Ahmad Usmani, Arunava Roy, Junzo Watada, Jafreezal Jaafar, and Izzatdin Abdul Aziz Causal BERT: Language Models for Causality Detection Between Events Expressed in Text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 965 Vivek Khetan, Roshni Ramnani, Mayuresh Anand, Subhashis Sengupta, and Andrew E. Fano Social Media Mining for Business Intelligence Analytics: An Application for Movie Box Office Forecasting . . . . . . . . . . . . . . . . . 981 Belén Usero, Virginia Hernández, and Cynthia Quintana Solving the Story Cloze Test Using Graph Feature Extraction . . . . . . . . 1000 Grabaskas Nathaniel MegaLite-2: An Extended Bilingual Comparative Literary Corpus . . . . 1014 Luis-Gil Moreno-Jiménez and Juan-Manuel Torres-Moreno VBSRL: A Semantic Frame-Based Approach for Data Extraction from Unstructured Business Documents . . . . . . . . . . . . . . . . . . . . . . . . . 1030 Simone Scannapieco, Andrea Ponza, and Claudio Tomazzoli Generating Correction Candidates for OCR Errors using BERT Language Model and FastText SubWord Embeddings . . . . . . . . . . . . . 1045 Mahdi Hajiali, Jorge Ramón Fonseca Cacho, and Kazem Taghva Measuring Meaning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1054 Ignazio Mauro Mirto The Impact of Agile Methods and “DevOps” on Day 2+ Operations for Large Enterprises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1068 R. Cherinka, S. Foote, J. Burgo, and J. Prezzama Mapping Risk Assessment Strategy for COVID-19 Mobile Apps’ Vulnerabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1082 Tanusree Sharma, Hunter A. Dyer, Roy H. Campbell, and Masooda Bashir Explainable Graph-Based Search for Lessons-Learned Documents in the Semiconductor Industry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1097 Hasan Abu-Rasheed, Christian Weber, Johannes Zenkert, Roland Krumm, and Madjid Fathi

Contents

xiii

Improving the Efficiency of Industrial Enterprise Management Based on the Forge Software-analytical Platform . . . . . . . . . . . . . . . . . . 1107 Nodirbek Yusupbekov, Fakhritdin Abdurasulov, Farukh Adilov, and Arsen Ivanyan Consumer-Oriented Web of Things Solution for Competitive Swimmers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1114 Rusho Yonit, Resh Amit, Benaroch Mickael, Bronshtein Nikita, Shahar Lior, and Beni Valotker Factors Influencing the Adoption of Online Banking in South Africa: An UTAUT Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1128 Mathias Mujinga Machine Learning Application in LAPIS Agile Software Development Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1136 Tuğrul Tekbulut, Nurdan Canbaz, and Tuğba Öztürk Kaya Transforming HR and Improving Talent Profiling with Qualitative Analysis Digitalization on Candidates for Career and Team Development Efforts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1149 Antonis Vatousios and Ari Happonen Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1167

Analytical View on Non-invasive Measurement of Moving Charge by Various Topologies of Wannier Qubit Krzysztof Pomorski1,2(B) 1 Cracow University of Technology, Faculty of Computer Science and Telecommunications, Department of Computer Science, 31-155 Krak´ ow, Poland 2 Quantum Hardware Systems, 94-056 L o ´d´z, Poland http://www.quantumhardwaresystems.com

Abstract. Detection of moving charge in free space is presented in the framework of single electron CMOS devices. It opens the perspective for construction of new type detectors for beam diagnostic in accelerators. General phenomenological model of noise acting on position based qubit implemented in semiconductor quantum dots is given in the framework of simplistic tight-binding model. At first linear position-based qubit also known as Wannier qubit is considered with the situation of being excited by external movement of charged particle in its proximity. Analytical formulas describing the change of qubit state are derived analytically. In the next steps the roton semiconductor qubit representing closed loop of coupled quantum dots is given and the effect of external moving charge on the change of qubit state is determined by analytical formulas. Roton qubit physical state can be controlled by external magnetic and electric field what is the extension of controlling mechanism in comparison with linear Wannier qubit, when we have N quantum dots aligned. Keywords: Weak measurement Tight-binding model

1

· Linear and circular Wannier qubits ·

Motivation for Weak Measurement of Moving Charged Particles

In nature matter has the attribute of having electric charge. Interaction between charged particles is the foundation base for atoms and molecules. Currently various experiments are conducted with charged particles as present in CERN and DESY accelerators. The controlled movement of charged particles as protons, electrons, light and heavy ions or other elementary particles takes place under static and time-dependent electric field and magnetic field generated in well organized pattern that is consequence of Maxwell equations. In particular one uses the magnetic focusing to keep the accelerator beam confined to c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1–55, 2022. https://doi.org/10.1007/978-3-030-80119-9_1

2

K. Pomorski

certain finite geometrical space. Moving charges generate electric and magnetic field what is reflected in time-dependent electric field and time-dependent vector potential field. Such time-dependent electric and magnetic field can be sensed by various types of detectors. If movement of the charged particle is traced only by time-dependent fields that are generated by observed particle one can deal with weak non-invasive measurement. Moving particle will always bring some action on the detector. On another hand the detector will respond to the action of external time-dependent magnetic and electric field. This detector response will generate counter electric and magnetic field that will try to compensate the effect of field trying to change the state of detector. Therefore one has mutual interaction between moving charged particles and detector. However if the speed of moving particles is very high this interaction will take very short time and will only slightly deflect the trajectory of moving charged particle that is under observation. Therefore one deals with weak measurement that is changing the physical state of object under observation in perturbative way [1]. Now we will concentrate on the description of the detector of moving charged particles. One can choice various types of detectors as measuring device for example: superconducting SQUIDs, Josephson junctions, NV-diamond sensors or single electron semiconductor devices. Because of rapid developments in cryogenic CMOS technology and scalability of those detectors we will concentrate on single electron semiconductor devices as most promising detectors for massive use described by [1, 3,4,8].

2

Single Electron Devices as Detectors of Moving Charged Particles

Quite recently it has been proposed by Fujisawa and Petta to use single electron devices for classical and quantum information processing. This technology relies on the chain of coupled quantum dots that can be implemented in various semiconductors. In particular one can use CMOS transistor with source and drain with channel in-between that is controlled by external polarizing gate as it is depicted in Fig. 1. Recent CMOS technologies allow for fabrication of transistor with channel length from 22 nm to 3 nm. If one can place one electron in sourcechannel-drain system (S-CH-D) than one can approximate the physical system by two coupled quantum dots. It is convenient to use tight-binding formalism to describe electron dynamics with time. In such case instead of wavefunction of electron it is useful to use maximum localized wavefunctions (Wannier functions) of that electron on the left and right quantum dot that are denoted by |1, 0 and |0, 1. One obtains the following simplistic Hamiltonian of position-based qubit given as H = Ep (1) |1, 0 1, 0| + Ep (2) |0, 1 0, 1| + |t|1→2 |1, 0 0, 1| + |t|2→1 |0, 1 1, 0| . (1) Here Ep (1) or Ep (2) has the meaning of minima of confining potential on the left or right quantum dot. It can be recognized as localized energy on the left

Analytical View on Non-invasive Measurement of Moving Charge

3

or right quantum dot. The tunneling process between left and right quantum dot or classical movement electron between left and right quantum dot can be accounted by the term |t|1→2 that has the meaning of delocalized energy (energy participating in particle transfer between quantum dots). If electron kinetic energy is much beyond the potential barrier separating left and right quantum dot that one can assign the meaning of kinetic energy to the term |t|1→2 or |t|2→1 . The quantum state of position based qubit is given as superposition of presence on the left and right quantum dot and is expressed by the formula |ψ = αt |1, 0 + βt |0, 1 ,

(2)

where |1, 0 = wL (x), |0, 1 = wR (x) are maximum localized functions on the left and right side of position based qubit. In case of position dependent qubit we have   d Ep1 ts12 |ψ = E(t) |ψ . (3) i |ψ = ∗ ts12 Ep2 dt For simplicity we consider Ep1 = Ep2 = Ep , ts12 = ts . We have two eigenenergies E1 = Ep − ts and E2 = Ep + ts and eigenstates are as follows   1 +1 1 , |E1  = √ (|1, 0 − |0, 1) = √ 2 2 −1   1 +1 1 . (4) |E2  = √ (|1, 0 + |0, 1) = √ 2 2 +1 In general case we have superposition of energy levels E1 and E2 as |ψ = cE1 eiφE1 |E1t + cE2 eiφE2 |E2t and in details we have ⎞ ⎛ E1 t ⎜ i ⎟ ⎟ iφE1 ⎜+cE1 e iφ |ψ = e + e E2 ⎜ E1 ⎟ ⎝ t⎠ −cE1 e i

⎞ ⎞ ⎛ ⎛ ⎞ E2 E1 E2 t⎟ t t⎟ iφ ⎜+eiφE1 c ⎜+c ⎜α(t)⎟ E1 e i + e E2 cE2 e i ⎟ E2 e i ⎟ ⎜ ⎜ ⎜ ⎟ = = ⎟ ⎟ ⎜ ⎜ ⎝ ⎠ E2 ⎠ E2 ⎠ E1 ⎝ ⎝ t t t β(t) +cE2 e i −eiφE1 cE1 e i + eiφE2 cE2 e i ⎛

(5) where |cE1 |2 + |cE2 |2 = 1 (sum of occupancy probability of energy level 1 and 2) and |α(t)|2 + |β(t)|2 = 1 (sum of probability of occupancy of left and right side by electron). Under influence of very quickly moving charge we have   d Ep1 + f1 (t) ts12 + f3 (t) |ψ = E(t) |ψ . (6) i |ψ = ∗ ts12 + f3 (t)∗ Ep2 + f2 (t) dt More exactly we have i

d dt



α(t) β(t)

 =

Epef f 1 (t) tef f −s12 (t) tef f −s12 (t)∗ Epef f 2 (t)



α(t) β(t)

 =

Ep1 + f1 (t) ts12 + f3 (t) ∗ t∗ s12 + f3 (t) Ep2 + f2 (t)



α(t) β(t)

 = E(t)

α(t) β(t)

(7) Single proton movement in proximity of position based qubit f1 (t) =

N protons

V1 (k)δ(t − t1 (k)), f2 (t) =

k=1

N protons

V2 (k)δ(t − t1 (k)),

k=1

f3 (t) =

N protons

k=1

(V3 (k) + iV4 (k))δ(t − t1 (k)).

(8)

4

K. Pomorski

In general case one shall have effective values of Epef f 1 (t), Epef f 2 (t), tef f −s12 (t) and tef f −s21 (t) given by formulas  +∞ 2 d2 ∗ Epef f 1 (t) = dxwL (x)(− + Vpol (x) + Vp (t))wL (x), 2me dx2 −∞  +∞ 2 d2 ∗ Epef f 1 (t) = dxwR (x)(− + Vpol (x) + Vp (t))wR (x), 2me dx2 −∞  +∞ 2 d2 ∗ tef f −s12 (t) = dxwR (x)(− + Vpol (x) + Vp (t))wL (x), 2me dx2 −∞  +∞ 2 d2 ∗ tef f −s21 (t) = dxwL (x)(− + Vpol (x) + Vp (t))wR (x), (9) 2me dx2 −∞ where wL (x) and wR (x) are maximum localized states (Wannier functions) in the left and right quantum dots and where Vpol (x) is the qubit polarizing electrostatic potential with Vp (t) as electrostatic potential coming from proton moving in the accelerator beam. For simplicity let us consider 3 terms perturbing single electron qubit Hamiltonian f1 (t) = V1 δ(t − t1 ), f2 (t) = V2 δ(t − t2 ), f3 (t) = (V3 + iV4 )δ(t − t2 )

(10)

and we obtain the modified Hamiltonian of qubit as i



d

α(t) β(t)

dt



 =

Ep1 + V1 δ(t − t1 ) ts12 + (V3 + iV4 )δ(t − t1 ) Ep2 + V2 δ(t − t1 ) t∗ s12 + (V3 − iV4 )δ(t − t1 )



α(t) β(t)

= E(t)

 α(t) β(t)

(11)

and is the system of two coupled differential equations d α(t) = (Ep1 + V1 δ(t − t1 ))α(t) + (ts12 + (V3 + iV4 )δ(t − t1 ))β(t), dt d i β(t) = (Ep2 + V2 δ(t − t1 ))β(t) + (t∗s12 + (V3 − iV4 )δ(t − t1 ))α(t), dt

i

(12) that can be rewritten in discrete form as 1 (α(t + δt) − α(t − δt)) = (Ep1 + V1 δ(t − t1 ))α(t) + (ts12 + (V3 + iV4 )δ(t − t1 ))β(t), 2δt 1 ∗ i (β(t + δt) − β(t − δt)) = (Ep2 + V2 δ(t − t1 ))β(t) + (ts12 + (V3 − iV4 )δ(t − t1 ))α(t), 2δt

i

(13)

 t −δt Applying operator t11−δt dt to both sides of previous equations with very small δt → 0 we obtain two algebraic relations as − + + i(α(t+ 1 ) − α(t1 )) = V1 α(t1 ) + (V3 + iV4 )β(t1 ), − + + i(β(t+ 1 ) − β(t1 )) = V2 β(t1 ) + (V3 − iV4 )α(t1 ).

(14)

Linear combination of quantum states of qubit before the measurement is expressed by quantum states of qubit after weak measurement that was due to the interaction of qubit with external passing charged particle so we obtain + − (i − V1 )α(t+ 1 ) − (V3 + iV4 )β(t1 ) = iα(t1 ),

Analytical View on Non-invasive Measurement of Moving Charge + − (i − V2 )β(t+ 1 ) − (V3 − iV4 )α(t1 ) = iβ(t1 ).

Last equations can be written in the compact form as  −   +   α(t1 ) α(t1 ) (i − V1 ) −(V3 + iV4 ) = i −(V3 − iV4 ) (i − V2 ) β(t+ β(t− ) 1 1) or equivalently  +   −1  −  α(t1 ) (i − V1 ) −(V3 + iV4 ) α(t1 ) = i − iV ) (i − V ) −(V β(t+ β(t− ) 3 4 2 1 1)

5

(15)

(16)

(17)

and it implies that quantum state after weak measurement is obtained as the linear transformation of the quantum state before the measurement so ⎞ ⎞ ⎛  ⎛ + −  α(t1 ) ( + iV2 ) (−iV3 + V4 ) ⎝α(t1 )⎠ ⎠ = ⎝ + − 2 2 (−iV − V ) ( + iV ) β(t1 ) β(t ) ( + iV1 )( + iV2 ) + V3 + V4 3 4 1 1

(18)

and hence 

α(t+ 1 ) β(t+ 1 )

=



ˆ =M



 −   +   α(t1 ) α(t− α(t1 ) M1,1 M1,2 1 ) = − + M2,1 M2,2 β(t1 ) β(t1 ) β(t− 1 )

1 (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))   Mr(1,1) + iMi(1,1) Mr(1,2) + iMi(1,2) × . Mr(2,1) + iMi(2,1) Mr(2,2) + iMi(2,2)

(19)

ˆ as M1,1 Now we identify diagonal parts of matrix M M1,1 =

(2 (2 + V22 + V32 + V42 )) + (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

((−2 V1 + V2 (−V1 V2 + V32 + V42 ))) )= + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 ))) 1 × = (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

+ i(

(4

× [(2 (2 + V22 + V32 + V42 )) + i((−2 V1 + V2 (−V1 V2 + V32 + V42 )))] = 1 = [Mr(1,1) + iMi(1,1) ] (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

(20)

and M2,2 M2,2 =

(2 (2 + V12 + V32 + V42 )) + (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

((−2 V2 + V1 (−V1 V2 + V32 + V42 ))) )= (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 ))) 1 × = (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

+ i(

× [(2 (2 + V12 + V32 + V42 )) + i((−2 V2 + V1 (−V1 V2 + V32 + V42 )))] = 1 [Mr(2,2) + iMi(2,2) ]. = (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

(21)

6

K. Pomorski

Non-diagonal parts of matrix are given as M1,2 =

((−(V1 + V2 )V3 + (2 − V1 V2 + V32 )V4 + V43 )) × + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

(4

((2 V3 + (V1 + V2 )V4 + V3 (−V1 V2 + V32 + V42 ))) )= (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 ))) 1 × = (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

× (−i)(

× [((−(V1 + V2 )V3 + (2 − V1 V2 + V32 )V4 + V43 )) +

+ (−i)(((2 V3 + (V1 + V2 )V4 + V3 (−V1 V2 + V32 + V42 ))))] = 1 [Mr(1,2) + iMi(1,2) ] = (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

(22)

and M2,1 = (−(

(4

(((V1 + V2 )V3 + (2 − V1 V2 + V32 )V4 + V43 )) )) + + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

((−2 V3 + (V1 + V2 )V4 − V3 (−V1 V2 + V32 + V42 ))) )= (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 ))) 1 × = (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

+ i(

× [−(((V1 + V2 )V3 + (2 − V1 V2 + V32 )V4 + V43 )) +

+ i((−2 V3 + (V1 + V2 )V4 − V3 (−V1 V2 + V32 + V42 )))] = 1 [Mr(2,1) + iMi(2,1) ]. = (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

(23)

We recognize energy transfer during proton movement in close proximity to position dependent qubit so the quantum state after weak measurement is given as ⎛ =

iφE1m c

|ψm = e

E1 t E1m e i

E1 iφE1m cE1m e i t ⎝+e E1 −eiφE1m cE1m e i t

+ +

iφE2m c

|E1  + e

E2 t E2m e i

⎞

E2 eiφE2m cE2m e i t ⎠ E2 eiφE2m cE2m e i t

⎛

E1 t i E1 iφE1 −e cE1 e i t

iφE1 cE1 e ⎝+e

|E2  =

 =

M1,1 M1,2 M2,1 M2,2

⎞

E2 t i ⎠ E2 iφE2 e cE2 e i t

+ eiφE2 cE2 e +

(24)

We obtain the quantum state after weak measurement in the form as e

iφE1m c

E1 t

E1m

=

e− i √ 2





M1,1 M1,2 M2,1 M2,2

1 −1



and iφ e E2m c

E2 t

E2m

e− i = √ 2

11





M1,1 M1,2 M2,1 M2,2



⎞ E1 E2 iφE1 cE1 e i t + eiφE2 cE2 e i t ⎠ ⎝+e E1 E2 −eiφE1 cE1 e i t + eiφE2 cE2 e i t ⎛

⎞

⎛

E1 t i E1 iφE1 −e cE1 e i t

iφE1 cE1 e ⎝+e

(25)

E2 t i ⎠ E2 iφE2 e cE2 e i t

+ eiφE2 cE2 e +

(26)

We have e

iφE1m c

E1m

1 (i(φE2 +(E1 −E2 )t)) = √ (cE2 e (M1,1 + M1,2 − M2,1 − M2,2 ) + 2 + cE1 e

(iφE1 ) (M

1,1

− M1,2 − M2,1 + M2,2 ))

(27)

Analytical View on Non-invasive Measurement of Moving Charge

7

and 1 eiφE2m cE2m = √ (cE2 e(i(φE1 +(−E1 +E2 )t)) (M1,1 − M1,2 + M2,1 − M2,2 ) + 2 + cE2 e(iφE2 ) (M1,1 + M1,2 + M2,1 + M2,2 ))(28) Last expression are given with parameters of weak measurement as iφE1m cE1m

1

=

e

×

2(4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

  2 2 2 2 × (cE1 |(2 + V1 + V2 + 2V1 V3 + 2V3 (V2 + V3 ) + 2V4 )Cos[φE1 ] + 2

2

2

2

3

+ cE2 ((−V1 + V2 ) + 2( − V1 V2 + V3 )V4 + 2V4 )Cos[φE2 + (E1 − E2 )t] + 2

2

2

2

2

+ cE1 ( (V1 + V2 − 2V3 ) + (V1 + V2 + 2V3 )(V1 V2 − V3 − V4 ))Sin[φE1 ] +  2 2 2 + cE2 ((V1 − V2 )( − V1 V2 + V3 ) + 2(V1 + V2 )V4 + (V1 − V2 )V4 )Sin[φE2 + (E1 − E2 )t])) + 2

+ i[((cE1 (− (V1 + V2 − 2V3 ) − (V1 + V2 + 2V3 )(V1 V2 − V3 − V4 ))Cos[φE1 ] − 2

− cE2 ((V1 − V2 )( − V1 V2 +

2 V3 )

+ 2(V1 + V2 )V4 + (V1 − 2

2

2 V2 )V4 )Cos[φE2

+ (E1 − E2 )t] +

2

2

cE1 (2 + V1 + V2 + 2V1 V3 + 2V3 (V2 + V3 ) + 2V4 )Sin[φE1 ] +   2 2 2 2 3 + cE2 ((−V1 + V2 ) + 2( − V1 V2 + V3 )V4 + 2V4 )Sin[φE2 + (E1 − E2 )t]))

(29)

and consequently we have iφE2m cE2m

e

×

=

 

1 2(4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 ))) 2

2

2

2

×

2

((cE2 (2 + V1 + V2 − 2(V1 + V2 )V3 + 2V3 + 2V4 )Cos[φE2 ] + 2

2

3

−cE1 ((V1 − V2 )(V1 + V2 ) + 2( − V1 V2 + V3 )V4 + 2V4 )Cos[φE1 + (−E1 + E2 )t] + 2

2

2

+cE2 ( (V1 + V2 + 2V3 ) + (V1 + V2 − 2V3 )(V1 V2 − V3 − V4 ))Sin[φE2 ] + 2

+cE1 ((V1 − V2 )( − V1 V2 +



2 V3 )

− 2(V1 + V2 )V4 + (V1 −

2 V2 )V4 )Sin[φE1

2



+ (−E1 + E2 )t])) + 2

2

+i (−cE2 ( (V1 + V2 + 2V3 ) + (V1 + V2 − 2V3 )(V1 V2 − V3 − V4 ))Cos[φE2 ] + 2

2

2

+cE1 (−(V1 − V2 )( − V1 V2 + V3 ) + 2(V1 + V2 )V4 + (−V1 + V2 )V4 )Cos[φE1 − (E1 − E2 )t] + 2

+cE2 (2 + 2

2 V1

+

2 V2

2

2

− 2(V1 + V2 )V3 + 2V3 + 2V4 )Sin[φE2 ] 2

3

−cE1 ((V1 − V2 )(V1 + V2 ) + 2( − V1 V2 + V3 )V4 + 2V4 )Sin[φE1 − (E1 − E2 )t])

 

(30)

It is quite straightforward to obtain probability of occupancy of energy E1 by electron in position based qubit after weak measurement (one interaction with passing charge particle) and it is given as (cE1m )2 =

2 × (4(4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))) × [(c2E1 (42 + (V1 + V2 + 2V3 )2 ) + c2E2 ((V1 − V2 )2 + 4V42 ) +

+ 2cE1 cE2 (−(V1 − V2 )(V1 + V2 + 2V3 ) + 4V4 )Cos[φE1 − φE2 − (E1 − E2 )t1 ] − − 4cE1 cE2 [(V1 − V2 ) + (V1 + V2 + 2V3 )V4 ]Sin[φE1 − φE2 − (E1 − E2 )t1 ])]

(31)

8

K. Pomorski (cE2m )2 =

4(4

+ (−V1 V2 + 

V32

+

V42 )2

2 × + 2 (V12 + V22 + 2(V32 + V42 )))

× c2E2 (42 + (V1 + V2 − 2V3 )2 ) + c2E1 ((V1 − V2 )2 + 4V42 ) − − 2cE1 cE2 ((V1 − V2 )(V1 + V2 − 2V3 ) + 4V4 )Cos[φE1 − φE2 − (E1 − E2 )t] +  + 4cE1 cE2 ((V1 − V2 ) − (V1 + V2 − 2V3 )V4 )Sin[φE1 − φE2 − (E1 − E2 )t]

(32)

Consequently we obtain phase imprint on energy eigenstate E1 given by the relation  iφE1m

=

e

1 2(4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

×

  2 2 2 2 × (cE1 |(2 + V1 + V2 + 2V1 V3 + 2V3 (V2 + V3 ) + 2V4 )Cos[φE1 ] + 2

2

2

2

3

+ cE2 ((−V1 + V2 ) + 2( − V1 V2 + V3 )V4 + 2V4 )Cos[φE2 + (E1 − E2 )t] + 2

2

2

+ cE1 ( (V1 + V2 − 2V3 ) + (V1 + V2 + 2V3 )(V1 V2 − V3 − V4 ))Sin[φE1 ] +  2 2 + cE2 ((V1 − V2 )( − V1 V2 + V3 ) + 2(V1 + V2 )V4 + (V1 − V2 )V4 )Sin[φE2 + (E1 − E2 )t])) + 2

2

2

2

+ i[((cE1 (− (V1 + V2 − 2V3 ) − (V1 + V2 + 2V3 )(V1 V2 − V3 − V4 ))Cos[φE1 ] 2(V1 + V2 )V4 + (V1 − 2

2

2 V2 )V4 )Cos[φE2

+ (E1 − E2 )t] +

2

2

+ cE1 (2 + V1 + V2 + 2V1 V3 + 2V3 (V2 + V3 ) + 2V4 )Sin[φE1 ] +   2 2 2 2 3 × + cE2 ((−V1 + V2 ) + 2( − V1 V2 + V3 )V4 + 2V4 )Sin[φE2 + (E1 − E2 )t]))  ×

2 (4(4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))) 2

2

2

2

2

×

2

× [(cE1 (4 + (V1 + V2 + 2V3 ) ) + cE2 ((V1 − V2 ) + 4V4 ) + + 2cE1 cE2 (−(V1 − V2 )(V1 + V2 + 2V3 ) + 4V4 )Cos[φE1 − φE2 − (E1 − E2 )t] − − 1 2

− 4cE1 cE2 [(V1 − V2 ) + (V1 + V2 + 2V3 )V4 ]Sin[φE1 − φE2 − (E1 − E2 )t])]

(33)

and phase imprint on energy eigenstate E2 given by the relation iφE2m

e

=

1 ×   (4 + (−V1 V2 + V32 + V42 )2 + 2 (V12 + V22 + 2(V32 + V42 )))

  2 2 2 × i (−cE2 ( (V1 + V2 + 2V3 ) + (V1 + V2 − 2V3 )(V1 V2 − V3 − V4 ))Cos[φE2 ] + 2

2

2

+ cE1 (−(V1 − V2 )( − V1 V2 + V3 ) + 2(V1 + V2 )V4 + (−V1 + V2 )V4 )Cos[φE1 − (E1 − E2 )t] + 2

+ cE2 (2 +

2 V1

2

2

− 2(V1 + V2 )V3 + 2V3 + 2V4 )Sin[φE2 ] −  2 2 3 − cE1 ((V1 − V2 )(V1 + V2 ) + 2( − V1 V2 + V3 )V4 + 2V4 )Sin[φE1 − (E1 − E2 )t]) +  2 2 2 2 2 + ((cE2 (2 + V1 + V2 − 2(V1 + V2 )V3 + 2V3 + 2V4 )Cos[φE2 ] − 2

+

2 V2

2

3

− cE1 ((V1 − V2 )(V1 + V2 ) + 2( − V1 V2 + V3 )V4 + 2V4 )Cos[φE1 + (−E1 + E2 )t] + 2

2

2

+ cE2 ( (V1 + V2 + 2V3 ) + (V1 + V2 − 2V3 )(V1 V2 − V3 − V4 ))Sin[φE2 ] +   2 2 2 × + cE1 ((V1 − V2 )( − V1 V2 + V3 ) − 2(V1 + V2 )V4 + (V1 − V2 )V4 )Sin[φE1 + (−E1 + E2 )t]))  2

2

2

2

2

2

× ((cE2 (4 + (V1 + V2 − 2V3 ) ) + cE1 ((V1 − V2 ) + 4V4 ) −

Analytical View on Non-invasive Measurement of Moving Charge

9

− 2cE1 cE2 ((V1 − V2 )(V1 + V2 − 2V3 ) + 4V4 )Cos[φE1 − φE2 − (E1 − E2 )t] + − 1 + 4cE1 cE2 ((V1 − V2 ) − (V1 + V2 − 2V3 )V4 )Sin[φE1 − φE2 − (E1 − E2 )t]))

2

(34) .

In general case of the position based qubit with complex values of hopping coefficients ts12 = tsr + itsi brings eigenenergies    1 − (Ep1 − Ep2 )2 + 4 (t2si + t2sr ) + (Ep1 + Ep2 ) , E1 = 2   1 (Ep1 − Ep2 )2 + 4 (t2si + t2sr ) + (Ep1 + Ep2 ) (35) E2 = 2 that are referred to the eigevectors  ⎛

⎞ i(tsi −itsr ) (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep1 +Ep2 ⎜−   ,⎟ 2  ⎟ ⎜ √2 ⎜ 2 tsi +t2sr 14 (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep1 +Ep2 +t2si +t2sr ⎟

|E1  = ⎜ ⎜ ⎜ ⎝ and

√

t2si +t2sr

  2  1 (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep1 +Ep2 +t2si +t2sr 4

⎟ ⎟ ⎟ ⎠

 ⎞ (Ep1 −Ep2 )2 +4(t2si +t2sr )+Ep1 −Ep2 i(tsi −itsr ) ⎜   ,⎟ 2  ⎟ ⎜ √2 ⎜ 2 tsi +t2sr 14 (Ep1 −Ep2 )2 +4(t2si +t2sr )+Ep1 −Ep2 +t2si +t2sr ⎟

⎛ |E2  = ⎜ ⎜ ⎜ ⎝

√

t2si +t2sr

  2  1 +(Ep1 −Ep2 )2 +4(t2si +t2sr )+Ep1 −Ep2 +t2si +t2sr 4

⎟. ⎟ ⎟ ⎠

The quantum state before weak measurement is given in the form as   E1 t E2 t α(t), |ψt  = cE1 eiφE1 e i |E1  + cE2 eiφE2 e i |E2  = β(t), x and the quantum state after measurement is given in the form as   E1 t E2 t αm (t), iφE1m i iφE2m i e |E1  + cE2m e e |E2  = . |ψt  = cE1m e βm (t), x and for the weak measurement taking part at time instant t1 we have    −   αm (t+ α(t1 ), M1,1 M1,2 1 ), = . M M βm (t+ β(t− ), 2,1 2,2 1 1 ), x x or equivalently  ∗  ∗

  M1,1

− ∗ M2,1 + ∗ + ∗ − ∗ αm (t1 ) , βm (t1 ) , x = α(t1 ) , β(t1 ) , x ∗ ∗ M1,2 M2,2

(36)

10

K. Pomorski

Last two relations implies 1 =





+ ∗ + ) , βm (t )∗ , 1 1 x

αm (t

that simply means







+   αm (t ), 1 = α(t− )∗ , β(t− )∗ , + 1 1 x βm (t ) 1 x



∗ ∗ M1,1 M2,1 ∗ ∗ M1,2 M2,2



∗ ∗ M2,1 M1,1 ∗ ∗ M1,2 M2,2

M1,1 M1,2 M2,1 M2,2

 = ˆ 1 =



 M1,1 M1,2 M2,1 M2,2



− ), 1 − β(t ) 1

α(t

= 1



1 0 0 1

(37)

.

Such reasoning can be conducted also for many particle states. Basing on the conducted analysis we obtain the parameters of the quantum state in eigenenergy representation after weak measurement at time t1 in the form as E1



√

2

√

eiφE1m cE1m = e− i t1  , × 2

√



−i(tsi −itsr ) (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep1 +Ep2 t2si +t2sr    √ 2 √ 1 t2si +t2sr + 14 (−Ep1 +Ep2 + (Ep1 −Ep2 )2 +4(tsi +t2sr ))2 (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep1 +Ep2 +t2si +t2sr 4

t2si +t2sr



M1,1 M1,2 M2,1 M2,2

× 

√

E1

= e− i t1

2

√

t2si +t2sr

1 4

 × 



2

α(t− 1 ), = − β(t1 ),  , ×

 √2 2 (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep1 +Ep2 tsi +tsr  √ 2 t2si +t2sr + 14 (−Ep1 +Ep2 + (Ep1 −Ep2 )2 +4(t2si +t2sr ))2 (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep1 +Ep2 +t2si +t2sr

−i(tsi −itsr )  

√





− M1,1 α(t− 1 ) + M1,2 β(t1 ) − M2,1 α(t− 1 ) + M2,2 β(t1 )

= e−

 1 − 2

√

2 (Ep1 −Ep2 )2 +4(t2 si +tsr )+(Ep1 +Ep2 ) i

t1

×



(Ep1 − Ep2 )2 + 4 (t2si + t2sr ) − Ep1 + Ep2 −i(tsi − itsr ) −   (M1,1 α(t− 1 ) + M1,2 β(t1 )) + 2  t2si + t2sr 14 (Ep1 − Ep2 )2 + 4 (t2si + t2sr ) − Ep1 + Ep2 + t2si + t2sr  t2si + t2sr − (M2,1 α(t− + 1 ) + M2,2 β(t1 )) =  t2si + t2sr + 14 (−Ep1 + Ep2 + (Ep1 − Ep2 )2 + 4(t2si + t2sr ))2 = eiφE1m cE1m .

(38) and E



2 eiφE2m cE2m = e− i t1 √   √2 2 +i(tsi −itsr ) (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep2 +Ep1 tsi +tsr    , × √ 2 √ √ t2si +t2sr + 14 (−Ep2 +Ep1 + (Ep1 −Ep2 )2 +4(t2si +t2sr ))2 2 t2si +t2sr 14 (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep2 +Ep1 +t2si +t2sr



×  E2

e− i t

2

√

M1,1 M1,2 M2,1 M2,2

 − α(t1 ), = β(t− 1)  , × 2

 √ √2 2 +i(tsi −itsr ) (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep2 +Ep1 tsi +tsr    √ 2 √ t2si +t2sr + 14 (−Ep2 +Ep1 + (Ep1 −Ep2 )2 +4(tsi +t2sr ))2 t2si +t2sr 14 (Ep1 −Ep2 )2 +4(t2si +t2sr )−Ep2 +Ep1 +t2si +t2sr



× √ 1 2

− M1,1 α(t− 1 ) + M1,2 β(t1 ), = − − M2,1 α(t1 ) + M2,2 β(t1 )

2 (Ep1 −Ep2 )2 +4(t2 si +tsr )+(Ep1 +Ep2 )

t i × = e−  

2 2 2 +i(tsi − itsr ) (Ep1 − Ep2 ) + 4 (tsi + tsr ) − Ep2 + Ep1 −   (M1,1 α(t− 1 ) + M1,2 β(t1 ))) + 2   2 t2si + t2sr 14 (Ep1 − Ep2 )2 + 4 (t2si + t2sr ) − Ep2 + Ep1 + t2si + t2sr  t2si + t2sr − iφE2m (M2,1 α(t− cE2m . + 1 ) + M2,2 β(t1 ))) = e t2si + t2sr + 14 ( Ep2 + Ep1 + (Ep1 Ep2 )2 + 4(t2si + t2sr ))2

Analytical View on Non-invasive Measurement of Moving Charge

11

Fig. 1. (Top left): Scheme of position based qubit as given by [6] and act of weak measurement by external charged probe [3]; (Top right): Act of passage of charged particle in the proximity of position based qubit and renormalization of qubit confining potential due to the external perturbation; (Middle Left): Scheme of various energy levels present in qubit [4]; (Middle Right): Different effective potential of single electron obtained by various voltages applied to 3 electrodes; (Bottom): Different qubit eigenenergy levels for different confining potential cases from (Middle Right). It is worth mentioning that passing electric charge can induce quantum system transitions between many energetic levels.

12

3

K. Pomorski

Dynamic of Two Qubit Electrostatic Entanglement Under Influence of Weak Measurement

We have the following two interacting qubits (1 and 2) Hamiltonian with nodes (1,2) and (1 , 2 ) for isolated quantum system [3] given in the form ⎛ 2 + Ep1 + d q ts2 ts1 0 E ⎜ p1 1,1 ⎜ ⎜ q2 t∗ E + E + 0 t ⎜  p1 s1 s2 p2 d ⎜ 1,2 H =⎜ 2 ⎜ t∗ 0 Ep2 + Ep1 + d q ts2 ⎜ s1 ⎜ 2,1 ⎝ q2 ∗ 0 t∗ t E + E p2 s1 s2 p2 + d 2,2

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎟. ⎟ ⎟ ⎟ ⎠

(39)

Placement of external probing particle affecting only qubit 1 modifies this Hamiltonian Ep1 + f1 (t) + Ep1 + H=

t∗s2

q2 d1,1

ts2 Ep1 + f1 (t) + Ep2 +

t∗s1 + f3 (t) − if4 (t) 0

0

0

ts1 + f3 (t) + if4 (t)

d1,2

0 t∗s1

ts1 + f3 (t) + if4 (t) q2

Ep2 + f2 (t) + Ep1 + t∗s2

+ f3 (t) − if4 (t)

q2 d2,1

ts2 Ep2 + f2 (t) + Ep2 +

q2 d2,2

Let us investigate the equations of motion for 2-qubit system under influence of external charge particle. Let us assume that the quantum system is given as ⎛ ⎞ γ1 ⎜γ2 ⎟ 2 2 2 2 ⎟ |ψ(t) = ⎜ ⎝γ3 ⎠ , |γ1 (t)| + |γ2 (t)| + |γ3 (t)| + |γ4 (t)| = 1 γ4

(40)

We end up with four equation system given as [Ep1 + f1 (t)) + Ep1 +

q2 d1,1

]γ1 (t) + ts2 γ2 (t) + ts1 γ3 (t) + [f3 (t) + if4 (t)]γ3 (t) = i

∗

ts2 γ2 (t) + [Ep1 + f1 (t) + Ep2 +

q2 d1,2

]γ2 (t) + [ts1 + f3 (t) + if4 (t)]γ4 (t) = i

∗

(ts1 + f3 (t) − if4 (t))γ1 (t) + [Ep2 + f2 (t) + Ep1 + ∗

q2 d2,1

]γ3 (t) + ts2 γ4 (t) = i

∗

(ts1 + f3 (t) − if4 (t))γ2 + ts2 γ3 (t) + (Ep2 + f2 (t) + Ep2 +

q2 d2,2

d dt d dt d dt

)γ4 (t) = i

γ1 (t),

(41)

γ2 (t),

(42)

γ3 (t),

(43)

d dt

γ4 (t)

(44)

with single proton movement in proximity of position based qubit generating f1 (t) =

N protons 

V1 (k)δ(t − t1 (k)), f2 (t) =

k=1

N protons 

V2 (k)δ(t − t1 (k)),

k=1 N protons 

(V3 (k) + iV4 (k))δ(t − t1 (k)),

(45)

f1 (t) = V1 δ(t − t1 ), f2 (t) = V2 δ(t − t1 ), f3 (t) = (V3 ) + iV4 )δ(t − t1 ).

(46)

f3 (t) =

k=1

that in most simple version has the form

Analytical View on Non-invasive Measurement of Moving Charge

13

 t −δt Applying operator t11−δt dt to both sides of 41 equation with very small δt → 0 we obtain four algebraic relations as − + + i(γ1 (t+ 1 ) − γ1 (t1 )) = V1 γ1 (t1 ) + [V3 + iV4 ]γ3 (t1 ), + − + i(γ2 (t1 ) − γ2 (t1 )) = V2 γ2 (t1 ) + [V3 + iV4 ]γ4 (t+ 1 ),

− + + i(γ3 (t+ 1 ) − γ3 (t1 )) = [V3 − iV4 ]γ1 (t1 ) + V2 γ3 (t1 ), − + + i(γ4 (t+ 1 ) − γ4 (t1 )) = [V3 − iV4 ]γ2 (t1 ) + V2 γ4 (t1 ),

and is equivalent to the relation ⎛

i − V1 1 ⎜ 0 ⎜ ⎝ i −[V3 − iV4 ] 0

⎞⎛

⎞

⎛

(47) ⎞

γ1 (t+ γ1 (t− 0 −[V3 + iV4 ] 0 1 ) 1 ) ⎟ ⎜ ⎜ − ⎟ γ2 (t+ 0 −[V3 + iV4 ]⎟ )⎟ i − V1 ⎜γ2 (t1 )⎟ 1 ⎟⎜ ⎟ ⎜ + ⎟ = ⎜ ⎠ 0 i − V2 0 ⎝γ3 (t− ⎝γ3 (t1 )⎠ 1 )⎠ − 0 i − V2 −[V3 − iV4 ] ) (t ) γ4 (t+ γ 4 1 1

(48)

what brings the system of coupled quantum states of position based qubit after the passage of charged particle at time t+ 1 in dependence on the quantum state + − with condition t = t + Δt for Δt → 0 is given in the algebraic form at time t− 1 1 1 as ⎛

+

γ1 (t

⎞

)

1 ⎟ ⎜ +     ⎜γ2 (t+ )⎟  + 1  t1 dt H(t ˆ  ) ψ(t− ) 1 ⎟ ψ(t1 ) = ⎜ ⎜ + ⎟ = i 1 − t1 ⎝γ3 (t1 )⎠

+ γ4 (t ) 1 ⎞−1 i − V1 0 −[V3 + iV4 ] 0 ⎜ ⎟ 0 −[V3 + iV4 ]⎟ 0 i − V1 ⎜ = i ⎜ ⎟ ⎝−[V3 − iV4 ] ⎠ 0 0 i − V2 0 −[V3 − iV4 ] 0 i − V2

⎛

⎛

⎞

− γ1 (t )

⎛

− γ1 (t )

(49)

⎞

1 ⎟ 1 ⎟ ⎜ ⎜   ⎜γ2 (t− )⎟ ⎜γ (t− )⎟ − ˆ ⎜ 2 1 ⎟ ˆ  1 ⎟ ⎜ ⎜γ (t− )⎟ = M ⎜γ (t− )⎟ = M ψ(t1 ) = ⎝ 3 1 ⎠ ⎝ 3 1 ⎠ − γ4 (t ) 1

− γ4 (t ) 1

⎞ ⎞⎛ − γ1 (t ) M1,1 M1,2 M1,3 M1,4 1 ⎟ ⎜ − ⎟ ⎜M ⎟ ⎜ M2,2 M2,3 M2,4 ⎟ ⎜γ2 (t1 )⎟ ⎜ = ⎜ 2,1 ⎟ − ⎟ = ⎝M3,1 M3,2 M3,3 M3,4 ⎠ ⎜ ⎝γ3 (t1 )⎠ ⎛

M4,1 M4,2 M4,3 M4,4

⎛

M1,1 M1,2 M1,3 M1,4

⎞

− γ4 (t ) 1

⎜M ⎟ ⎜ 2,1 M2,2 M2,3 M2,4 ⎟ − − − − ⎟ [γ (t ) |x1  + γ2 (t1 ) |x2  + γ3 (t1 ) |x3  + γ4 (t1 ) |x4 ] = ⎝M3,1 M3,2 M3,3 M3,4 ⎠ 1 1

= ⎜

M4,1 M4,2 M4,3 M4,4 ˆ (|E  E | + |E  E | + |E  E | + |E  E |)[γ (t− ) |x  + γ (t− ) |x  + γ (t− ) |x  + γ (t− ) |x ] = = M 1 1 2 2 3 3 4 4 1 1 1 2 1 2 3 1 3 4 1 4 ˆ (E | |x  γ (t− ) + E | |x  γ (t− ) + E | |x  γ (t− ) + E | |x  γ (t− )) |E  + = M 1 1 1 1 1 2 2 1 1 3 3 1 1 4 4 1 1 ˆ (E | |x  γ (t− ) + E | |x  γ (t− ) + E | |x  γ (t− ) + E | |x  γ (t− )) |E  + +M 2 1 1 1 2 2 2 1 2 3 3 1 2 4 4 1 2 ˆ (E | |x  γ (t− ) + E | |x  γ (t− ) + E | |x  γ (t− ) + E | |x  γ (t− )) |E  + +M 3 1 1 1 3 2 2 1 3 3 3 1 3 4 4 1 3 ˆ (E | |x  γ (t− ) + E | |x  γ (t− ) + E | |x  γ (t− ) + E | |x  γ (t− )) |E  = +M 4 1 1 1 4 2 2 1 4 3 3 1 4 4 4 1 4 ⎞ ⎛ ⎞⎛ − γ1 (t ) |E1  E1 | |x1  |E1  E1 | |x2  |E1  E1 | |x3  |E1  E1 | |x4  1 − ⎟ ⎜|E  E | |x  |E  E | |x  |E  E | |x  |E  E | |x ⎟ ⎜ ⎟ ⎜ γ (t ) ⎜ ⎟ 2 1 2 2 2 2 2 3 2 2 4 ⎟⎜ 2 1 ⎟ = ˆ ⎜ 2 = M − ⎟ ⎝|E3  E3 | |x1  |E3  E3 | |x2  |E3  E3 | |x3  |E3  E3 | |x4 ⎠ ⎜ ⎝γ3 (t1 )⎠ − |E4  E4 | |x1  |E4  E4 | |x2  |E4  E4 | |x3  |E4  E4 | |x4  γ4 (t ) 1 ⎞ ⎛ − γ1 (t ) 1   ⎟  ⎜ ⎜γ (t− )⎟ ˆ |E  [ E | |x  , E | |x  , E | |x  , E | |x  ⎜ 2 1 ⎟] + = M 1 1 1 1 2 1 3 1 4 ⎜γ (t− )⎟ ⎝ 3 1 ⎠ − γ4 (t ) 1 ⎛ ⎞ − γ (t )   ⎜ 1 1 ⎟  ⎜γ2 (t− )⎟ ˆ |E  [ E | |x  , E | |x  , E | |x  , E | |x  ⎜ 1 ⎟ +M 2 1 2 2 2 3 2 4 2 ⎜ − ⎟] + ⎝γ3 (t1 )⎠ − γ4 (t ) 1

14

K. Pomorski ⎛ ⎞ − γ1 (t ) 1 ⎟  ⎜ ⎜γ2 (t− )⎟ ˆ |E  [ E | |x  , E | |x  , E | |x  , E | |x  ⎜ 1 ⎟ +M 3 3 1 3 2 3 3 3 4 ⎜γ (t− )⎟] + ⎝ 3 1 ⎠ 



 ˆ +M

− γ4 (t ) 1

⎞ − γ1 (t ) 1 ⎟  ⎜ ⎜γ2 (t− )⎟ 1 ⎟ |E4  [ E4 | |x1  , E4 | |x2  , E4 | |x3  , E4 | |x4  ⎜ ⎜γ (t− )⎟] = ⎝ 3 1 ⎠ ⎛



− γ4 (t ) 1

⎞ ⎞⎛ − − − − E1 | |x1  γ1 (t ) + E1 | |x2  γ2 (t ) + E1 | |x3  γ3 (t ) + E1 | |x4  γ4 (t ) M1,1 M1,2 M1,3 M1,4 1 1 1 1 − − − − ⎟ ⎜M ⎟⎜ ⎜ E | |x  γ (t ) + E | |x  γ (t ) + E | |x  γ (t ) + E | |x  γ (t ) M M M ⎜ 2 1 1 1 2 2 2 1 2 3 3 1 2 4 4 1 ⎟ 2,2 2,3 2,4 ⎟ ⎜ ⎟ = ⎜ 2,1 ⎟ − − − − ⎟ ⎝M3,1 M3,2 M3,3 M3,4 ⎠ ⎜ ⎝E3 | |x1  γ1 (t1 ) + E3 | |x2  γ2 (t1 ) + E3 | |x3  γ3 (t1 ) + E3 | |x4  γ4 (t1 )⎠ ⎛

M4,1 M4,2 M4,3 M4,4

− − − − E4 | |x1  γ1 (t ) + E4 | |x2  γ2 (t ) + E4 | |x3  γ3 (t ) + E4 | |x4  γ4 (t ) 1 1 1 1

ˆ can be rewritten as and matrix M (V2 − i) 0 −(iV4 + V3 ) 0 ⎜ i 0 −(iV4 + V3 ) ⎟ 0 (V2 − i) .⎟ ×⎜ ⎝ ⎠= 0 (V1 − i) 0 −(−iV4 + V3 ) + V4 + ( + iV1 )( + iV2 ) 0 (V1 − i) 0 −(−iV4 + V3 ) 1 2 2 2 2 = 2 × [( (V1 + V2 )) + i(( − V1 V2 + V3 + V4 ))] × 4 + 2 (V12 + V22 + 2 (V32 + V42 )) + (−V1 V2 + V32 + V42 ) ⎞ ⎛ (V2 − i) 0 −(iV4 + V3 ) 0 ⎟ ⎜ − i) 0 −(iV + V ) 0 (V 2 4 3 ⎟= ×⎜ ⎠ ⎝−(−iV4 + V3 ) 0 (V1 − i) 0 0 (V1 − i) 0 −(−iV4 + V3 ) 1 = 2 ב 4 + 2 (V12 + V22 + 2 (V32 + V42 )) + (−V1 V2 + V32 + V42 ) ⎛ ⎞ 1000

  ⎜0 1 0 0⎟ ⎟ × ((2 V2 (V1 + V2 )) + (2 (2 − V1 V2 + V32 + V42 ))) + i(−3 (V1 + V2 ) + V2 ((2 − V1 V2 + V32 + V42 ))) ⎜ ⎝0 0 0 0⎠ + 0000 ⎛ ⎞ 0000  ⎜0 0 0 0⎟  ⎟ + ((2 V1 (V1 + V2 )) + (2 (2 − V1 V2 + V32 + V42 ))) + i(−3 (V1 + V2 ) + V1 ((2 − V1 V2 + V32 + V42 ))) ⎜ ⎝0 0 1 0⎠ + 0001 ⎛ ⎞ 0010 ⎜0 0 0 1⎟ 2 2 2 2 ⎟ −[( (V1 + V2 )) + i(( − V1 V2 + V3 + V4 ))]V3 ⎜ ⎝1 0 0 0⎠ + 0100 ⎛ ⎞ 0 0 −1 0 ⎜ 0 0 0 −1⎟ 2 2 2 2 . +[i( (V1 + V2 )) − (( − V1 V2 + V3 + V4 ))]V4 +1 0 0 0 0 +1 0 0 ˆ = M

(V32

)2

One ends up with algebraic condition for the quantum state just after t1 = t+ 1 − so we have the relation between quantum state at t+ 1 and t1 expressed in the algebraic way as

Analytical View on Non-invasive Measurement of Moving Charge

E1 t1 

E2 t1

E3 t1

15

E4 t1

|E1  + cE2m eiφE2m e  |E2  + cE3m e  eiφE3m |E3  + cE4m eiφE4 e  |E4  = ⎛ ⎞ γ1 (t+ 1) ⎜γ2 (t+ )⎟ 1 1 ⎟= =⎜ ⎝γ3 (t+ ⎠ 4 + 2 (V 2 + V 2 + 2 (V 2 + V 2 )) + (−V V + V 2 + V 2 )2 × 1) 1 2 1 2 3 4 3 4 + γ4 (t1 ) ⎛ ⎞ 1000

 ⎜ 0 1 0 0⎟ ((2 V2 (V1 + V2 )) + (2 (2 − V1 V2 + V32 + V42 ))) + i(−3 (V1 + V2 ) + V2 ((2 − V1 V2 + V32 + V42 ))) + 0000 0000 ⎛ ⎞ 0000  ⎜ ⎟ 0 0 0 0 2 2 2 2 2 3 2 2 2 ⎟ + (( V1 (V1 + V2 )) + ( ( − V1 V2 + V3 + V4 ))) + i(− (V1 + V2 ) + V1 (( − V1 V2 + V3 + V4 ))) ⎜ ⎝0 0 1 0⎠ + 0001 ⎛ ⎞ 0010 ⎜ ⎟ 0 0 0 1 ⎟ −[(2 (V1 + V2 )) + i((2 − V1 V2 + V32 + V42 ))]V3 ⎜ ⎝1 0 0 0⎠ + 0100 ⎛ ⎞ ⎛ ⎞ 0 0 −1 0 γ1 (t− 1) ⎜ 0 0 0 −1⎟ ⎜γ2 (t− )⎟ 2 2 2 2 1 ⎟ ⎜ ⎟= +[i( (V1 + V2 )) − (( − V1 V2 + V3 + V4 ))]V4 ⎜ ⎝+1 0 0 0 ⎠ ⎝γ3 (t− ⎠ 1) 0 +1 0 0 γ4 (t− 1) 1 = 2 × 4 + 2 (V12 + V22 + 2 (V32 + V42 )) + (−V1 V2 + V32 + V42 ) ⎞ ⎛   + ((2 V2 (V1 + V2 )) + (2 (2 − V1 V2 + V32 + V42 ))) + i(−3 (V1 + V2 ) + V2 ((2 − V1 V2 + V32 + V42 ))) γ1 (t− 1) ⎟  

⎜ ⎟ ⎜ ⎜+ ((2 V2 (V1 + V2 )) + (2 (2 − V1 V2 + V32 + V42 ))) + i(−3 (V1 + V2 ) + V2 ((2 − V1 V2 + V32 + V42 ))) γ2 (t− 1 )⎟ ⎟+ ⎜   ⎜ − ⎟ 2 2 2 2 2 3 2 2 2 ⎜+ (( V1 (V1 + V2 )) + ( ( − V1 V2 + V3 + V4 ))) + i(− (V1 + V2 ) + V1 (( − V1 V2 + V3 + V4 ))) γ3 (t1 )⎟  ⎠ ⎝  + ((2 V1 (V1 + V2 )) + (2 (2 − V1 V2 + V32 + V42 ))) + i(−3 (V1 + V2 ) + V1 ((2 − V1 V2 + V32 + V42 ))) γ4 (t− 1) ⎞ ⎛  − [(2 (V1 + V2 ))V3 + ((2 − V1 V2 + V32 + V42 ))V4 ] − i[(2 (V1 + V2 ))V4 + ((2 − V1 V2 + V32 + V42 ))V3 ] γ3 (t− 1) ⎟ ⎜  ⎟ ⎜ ⎜ − [2 (V1 + V2 ))V3 + ((2 − V1 V2 + V32 + V42 ))V4 ] − i[(2 (V1 + V2 ))V4 + ((2 − V1 V2 + V32 + V42 ))V3 ] γ4 (t− 1 )⎟ ⎟ = ⎜  ⎜ − ⎟ 2 2 2 2 2 2 2 2 ⎜ − [(( − V1 V2 + V3 + V4 ))V4 + ( (V1 + V2 ))V3 ] + i[( (V1 + V2 ))V4 − (( − V1 V2 + V3 + V4 ))V3 ] γ1 (t1 )⎟  ⎠ ⎝ − [((2 − V1 V2 + V32 + V42 ))V4 + (2 (V1 + V2 ))V3 ] + i[(2 (V1 + V2 ))V4 − ((2 − V1 V2 + V32 + V42 ))V3 ] γ2 (t− 1) cE1m eiφE1m e

ˆ [cE1 eiφE1 e =M

− E1 t1 

|E1  + cE2 eiφE2 e

− E2 t1 

|E2  + cE3 e

− E3 t1 

eiφE3 |E3  + cE4 eiφE4 e

− E4 t1 

|E4 ].

Last equation implies four relations cE1m eiφE1m e

E1 t1 

ˆ [cE1 eiφE1 e = E1 | M + cE3 e

− E3 t1 

− E1 t1 

|E1  + cE2 eiφE2 e

eiφE3 |E3  + cE4 eiφE4 e

− E2 t1 

|E2  +

− E4 t1 

|E4 ] = ⎞ γ1 (t− 1) − ⎜ ⎟ ˆ ⎜γ2 (t1− )⎟ , = E1 | M ⎝γ3 (t1 )⎠ γ4 (t− 1) ⎛

(50) cE2m eiφE2m e

E2 t1 

ˆ [cE1 eiφE1 e = E2 | M

− E1 t1 

|E1  + cE2 eiφE2 e

− E2 t1 

|E2  + cE3 e

− E3 t1 

eiφE3 |E3  + cE4 eiφE4 e

− E4 t1 

|E4 ] = ⎛ ⎞ γ1 (t− 1) ⎜γ2 (t− )⎟ 1 ⎟ ˆ⎜ = E2 | M ⎝γ3 (t− ⎠ 1) γ4 (t− 1)

(51)

16

K. Pomorski

cE3m eiφE3m e

E3 t1 

ˆ [cE1 eiφE1 e = E3 | M

− E1 t1 

|E1  + cE2 eiφE2 e

− E2 t1 

|E2  + cE3 e

− E3 t1 

eiφE3 |E3  + cE4 eiφE4 e

− E4 t1 

|E4 ] = ⎛ ⎞ γ1 (t− 1) ⎜γ2 (t− )⎟ 1 ⎟, ˆ⎜ = E3 | M ⎝γ3 (t− ⎠ 1) γ4 (t− 1)

(52)

cE4m eiφE4m e

E4 t1 

ˆ [cE1 eiφE1 e = E4 | M

− E1 t1 

|E1  + cE2 eiφE2 e

− E2 t1 

|E2  + cE3 e

− E3 t1 

eiφE3 |E3  + cE4 eiφE4 e

− E4 t1 

|E4 ] = ⎞ γ1 (t− 1) ⎜γ2 (t− )⎟ 1 ⎟ ˆ⎜ = E4 | M ⎝γ3 (t− ⎠ 1) γ4 (t− 1) ⎛

(53) The probability of occupancy of eigenenergy E1 , E2 , E3 and E4 for interacting qubit system after measurement of charged particle passage is given by |cE1m |2 , |cE2m |2 ,|cE3m |2 ,|cE4m |2 and phase imprint of given eigenenergy state is given by factors eiφE1m eiφ2m ,eiφE3m ,eiφE4m . Let us consider the case of two symmetric qubits whose system is depicted at Fig. 2.

Fig. 2. Concept of two electrostatically interacting qubits and charged particle [2]

We have the following Hamiltonian ˆ = H

Ep1 + Ep1 + Ec(1,1 ) it2si + t2sr it1si + t1sr 0 Ep1 + Ep2 + Ec(1,2 ) 0 it1si + t1sr t2sr − it2si t1sr − it1si 0 Ep2 + Ep1 + Ec(2,1 ) it2si + t2sr 0 t1sr it1si t2sr it2si Ep2 + Ep2 + Ec(2,2 )

(54)

Analytical View on Non-invasive Measurement of Moving Charge

17

that can be simplified by placement of two qubit system in the geometrical configuration giving the following electrostatic energies as Ec(1,1 ) = Ec(2,2 ) = 2 2 . We set Ep2 = Ep2 = Ec1 = qd and Ec(2,1 ) = Ec(1,2 ) = Ec2 = √ 2 q 2 d +(a+b) 2

Ep1 = Ep1 = Ep and we introduce Epp = 2Ep + qd and Epp1 = 2Ep + √

q2

d2 +(a+b)2

First simplfied Hamiltonian of two qubit system has the structure ⎛ ⎞ ts 0 Epp ts ⎜ ⎟ ˆ = ⎜ ts Epp1 0 ts ⎟ H ⎝ ts 0 Epp1 ts ⎠ 0 ts ts Epp

.

(55)

and we assume that ts ∈ R. The eigenstates of simplified Hamiltonian are ⎛

⎛ ⎞ ⎞ 0 − √12 1 ⎜ 0 ⎟ ⎜− √ ⎟ 2⎟ ⎜ ⎟ |E1  = ⎜ ⎝ 0 ⎠ , |E2  = ⎝ √1 ⎠ 2 + √12 0

(56)

and ⎛

|E3  =

√

⎞

(Epp −Epp1 )2 +16t2s −Epp +Epp1  , ⎜ 2√(Epp −Epp1 )2 +16t2 −Epp +Epp1 2 +32t2 ⎟ s s ⎟ ⎜ ⎟ ⎜  4ts ,⎟ ⎜− √ 2 ⎜ 2 2 2 2 (Epp −Epp1 ) +16ts −Epp +Epp1 +32ts ⎟ ⎟ ⎜ ⎟ ⎜  4ts ,⎟ ⎜ − √ 2 ⎜ 2 2 2 2 (Epp −Epp1 ) +16ts −Epp +Epp1 +32ts ⎟ ⎟ ⎜

⎜ ⎝

√

⎟ ⎠

√

⎞

(Epp −Epp1 )2 +16t2s −Epp +Epp1   2 √ 2 (Epp −Epp1 )2 +16t2s −Epp +Epp1 +32t2s

(57)

and ⎛

|E4  =

(Epp −Epp1 )2 +16t2s +Epp −Epp1   , √ ⎜ 2 (Epp −Epp1 )2 +16t2 +Epp −Epp1 2 +32ts2 ⎟ s ⎟ ⎜ ⎟ ⎜ 4ts ,⎟ ⎜ √ 2 ⎜ 2 (Epp −Epp1 )2 +162 +Epp −Epp1 +32t2 ⎟ s ⎟ ts ⎜ ⎟ ⎜  4ts ⎟ ⎜ √ 2 ⎜ 2 (Epp −Epp1 )2 +16t2s +Epp −Epp1 +32t2s ⎟ ⎟ ⎜

⎜ ⎝

√

(Epp −Epp1 )2 +16t2s +Epp −Epp1   2 √ 2 (Epp −Epp1 )2 +16t2s +Epp −Epp1 +32t2s

⎟ ⎠

(58)

18

K. Pomorski

We obtain 1 eiΦE1m cE1m = (ieiEpp t1 (cE1 ei(φE1 −Epp t1 ) (−2i + V1 + V2 ) √ 2 1√ + [ 2cE2 ei(φE2 −Epp1 t1 ) (2V3 + i( − 1)V4 ) 2    1 i t1 (Epp − Epp1 )2 + 16t2s − t1 (Epp + Epp1 ) + 2φE3 −[cE3 exp 2   2 [Epp (V2 − V1 ) + 4ts 4ts (V2 − V1 ) + i( + 1)V4 (Epp − Epp1 )2 + 16t2s +   2 (Epp − Epp1 )2 + 16t2s + 2Epp1 + Epp1 (V2 − V1 ) (Epp − Epp1 )2 + 16t2s + Epp1 (V2 − V1 )]]× +Epp (V1 − V2 )



1

 (Epp − Epp1 )2 + 16t2s

 + (Epp − Epp1 )2 + 16t2s − Epp + Epp1 √  iφ − 1 it (Epp −Epp1 )2 +16ts2 +Epp +Epp1 +[cE4 e E4 2 1 ×

16t2s − (Epp − Epp1 )

  2 [Epp (V1 − V2 ) + 4ts 4ts (V1 − V2 ) + i( + 1)V4 (Epp − Epp1 )2 + 16t2s +  (Epp − Epp1 )2 + 16t2s − 2Epp1 + Epp (V1 − V2 )  2 (V1 − V2 )]] × +Epp1 (V2 − V1 ) (Epp − Epp1 )2 + 16t2s + Epp1  (Epp − Epp1 )2 + 16t2s

1  ]])   (Epp − Epp1 ) (Epp − Epp1 )2 + 16t2s + Epp − Epp1 + 16t2s ×√

1 2 (2 + i(V1 + V2 + V4 (V3 − iV4 )) − V1 V2 + V3 (V3 − iV4 ))

(59) The probability of occurrence of quantum state in energy E1 after weak measurement is therefore equal to |eiΦE1m cE1m |2 . and 1 √ eiΦE2m cE2m = [ieiEpp1 t1 [ [ 2cE1 ei(φE1 −Epp t1 ) (2V3 + i( − 1)V4 ) + 2   √  1i t (Epp −Epp1 )2 +16t2 s −t1 (Epp +Epp1 )+2φE3 +[cE3 e 2 1   (−i( + 1)V4 (Epp − Epp1 ) (Epp − Epp1 )2 + 16t2s − Epp + Epp1 +  +4ts (V2 − V1 ) (Epp − Epp1 )2 + 16t2s + 16i( + 1)t2s V4 )] 1  × +   (Epp − Epp1 )2 + 16t2s 16t2s − (Epp − Epp1 ) (Epp − Epp1 )2 + 16t2s − Epp + Epp1 √  iφ − 1 it (Epp −Epp1 )2 +16t2 s +Epp +Epp1 +(cE4 e E4 2 1 ×   2 ×[i( + 1)V4 (Epp − Epp1 )( (Epp − Epp1 )2 + 16t2s + Epp − Epp1 ) + 4ts (V1 − V2 ) (Epp − Epp1 )2 + 16t2s + 16i( + 1)ts V4 ]) 1  ]+ ×    2 2 (Epp − Epp1 ) + 16ts (Epp − Epp1 ) (Epp − Epp1 )2 + 16t2s + Epp − Epp1 + 16t2s cE2 ei(φE2 −Epp1 t1 ) (−2i + V1 + V2 ) √ ]] × 2 1 ×√ 2 (2 + i(V1 + V2 + V4 (V3 − iV4 )) − V1 V2 + V3 (V3 − iV4 )) +

and

Analytical View on Non-invasive Measurement of Moving Charge

1

it

−

(E

−E

)2 +16t2 +E

19

+E

pp pp1 pp pp1 s eiΦE3m cE3m = −[ie 2 1  1 √ −it1 (Epp +Epp1 ) ( ( 2e (cE2 ei(Epp t1 +φE2 ) 4ts (V1 − V2 ) + i( + 1)V4 (Epp − Epp1 )2 + 16t2s − Epp + Epp1 2  −cE1 ei(Epp1 t1 +φE1 ) ((V1 − V2 )( (Epp − Epp1 )2 + 16t2s − Epp + Epp1 ) + 4i( + 1)ts V4 )) +   (Epp − Epp1 )2 + 16t2s − Epp + Epp1 +[2icE3 16t2s − (Epp − Epp1 )



  1 i t1 (Epp − Epp1 )2 + 16t2s − t1 (Epp + Epp1 ) + 2φE3 2    (Epp − Epp1 )2 + 16t2s − 2ts V4 + i(V1 + V2 ) (Epp − Epp1 )2 + 16t2s + 4ts (V4 + 2iV3 ) ] × 2 

exp

1 ) × (Epp − Epp1 )2 + 16t2s     (Epp − Epp1 )2 + 16t2s + Epp + Epp1 4cE4 ts (Epp − Epp1 )(2V3 + i( − 1)V4 ) exp 12 i 2φE4 − t1  )] × −   (Epp − Epp1 ) (Epp − Epp1 )2 + 16t2s + Epp − Epp1 + 16t2s 1 ×  ×  2 16t2s − (Epp − Epp1 ) (Epp − Epp1 )2 + 16t2s − Epp + Epp1 ×

1 (2 + i(V1 + V2 + V4 (V3 − iV4 )) − V1 V2 + V3 (V3 − iV4 ))

(60) and

1

+cE4 e 2

1

+cE4 e 2

   16t2s + (Epp − Epp1 ) Epp − Epp1 + (Epp − Epp1 )2 + 16t2s )]

2cE4 e

− V3 )) +



×

2 (Epp

1 cE1 ei(φE1 −Epp t1 ) √ − ) 2 Epp1 )2 + 16t2s

  16t2s + (Epp − Epp1 ) Epp − Epp1 + (Epp − Epp1 )2 + 16t2s ))



ts   16t2s + (Epp − Epp1 ) Epp − Epp1 + (Epp − Epp1 )2 + 16t2s   √ 1 i 2φE3 −(Epp +Epp1 )t1 +t1 (Epp −Epp1 )2 +16ts2 ts 2cE3 e 2 −   )(V1 − i) +  2 2 16ts − (Epp − Epp1 ) −Epp + Epp1 + (Epp − Epp1 ) + 16t2s    √ 1 i 2φE3 +t1 Epp +Epp1 + (Epp −Epp1 )2 +16t2s +((e−i(Epp +Epp1 )t1 (e 2    16t2s − (Epp − Epp1 ) −Epp + Epp1 + (Epp − Epp1 )2 + 16t2s cE3 +

ei(φE2 −Epp1 t1 ) cE2 √ + 2

   √ i 2φE4 +t1 Epp +Epp1 − (Epp −Epp1 )2 +16t2s

+4ts ((

(iV4   √ iφE4 − 21 it1 Epp +Epp1 + (Epp −Epp1 )2 +16t2s

1 ×  )(V1 − i) 2 (Epp − Epp1 )2 + 16t2s   √ iφ − 1 it E +E + (Epp −Epp1 )2 +16t2s 2cE4 e E4 2 1 pp pp1 ts ei(φE2 −Epp1 t1 ) cE2 √ +(− +    2 16t2s + (Epp − Epp1 ) Epp − Epp1 + (Epp − Epp1 )2 + 16t2s   √ 1 i 2φE3 −(Epp +Epp1 )t1 +t1 (Epp −Epp1 )2 +16t2s ts 2cE3 e 2 −  )  2 2 16ts − (Epp − Epp1 ) −Epp + Epp1 + (Epp − Epp1 ) + 16t2s

   √ i 2φE4 +t1 Epp +Epp1 − (Epp −Epp1 )2 +16t2s

  √ it1 Epp +Epp1 + (Epp −Epp1 )2 +16t2s

  ei(φE1 −Epp t1 ) cE1 √ ( Epp − Epp1 + (Epp − Epp1 )2 + 16t2s × (( + 2      √  1 i 2φE3 +t1 Epp +Epp1 + (Epp −Epp1 )2 +16t2s 16t2s − (Epp − Epp1 ) −Epp + Epp1 + (Epp − Epp1 )2 + 16t2s cE3 + +[e−i(Epp +Epp1 )t1 (e 2 1

×eiΦE4m cE4m = [ie 2

(61)

20 K. Pomorski

   √ i 2φE4 +t1 Epp +Epp1 − (Epp −Epp1 )2 +16ts2

   16t2s + (Epp − Epp1 ) Epp − Epp1 + (Epp − Epp1 )2 + 16t2s )] ×

   √ 1 i 2φE4 +t1 Epp +Epp1 − (Epp −Epp1 )2 +16t2s +cE4 e 2    1 16t2s + (Epp − Epp1 ) Epp − Epp1 + (Epp − Epp1 )2 + 16t2s )]  ](V3 + iV4 )))] 2 (Epp − Epp1 )2 + 16t2s 1 ×  × 2 2 2 16ts + (Epp − Epp1 ) Epp − Epp1 + (Epp − Epp1 ) + 16t2s (2 + i(V1 + V2 + (V3 − iV4 )V4 ) − V1 V2 + V3 (V3 − iV4 ))

1 cE1ei(φE1 −Epp t1 )  √ − )(V2 − i) 2 2 (Epp − Epp1 )2 + 16t2s   √ + (Epp −Epp1 )2 +16t2s iφ − 1 it E +E 2cE4 e E4 2 1 pp pp1 ts ei(φE2 −Epp1 t1 ) cE2 √ −( +    2 16t2s + (Epp − Epp1 ) Epp − Epp1 + (Epp − Epp1 )2 + 16t2s   √ 1 i 2φE3 −(Epp +Epp1 )t1 +t1 (Epp −Epp1 )2 +16ts2 ts 2cE3e 2 −  )(V3 + iV4 )) +   2 2 16ts − (Epp − Epp1 ) −Epp + Epp1 + (Epp − Epp1 ) + 16t2s   √ iφ − 1 it E +E + (Epp −Epp1 )2 +16t2s ts ei(φE2 −Epp1 t1 ) cE2 2cE4 e E4 2 1 pp pp1 √ +4ts ((− +    2 2 16ts + (Epp − Epp1 ) Epp − Epp1 + (Epp − Epp1 )2 + 16t2s   √ 1 i 2φE3 −(Epp +Epp1 )t1 +t1 (Epp −Epp1 )2 +16t2s ts 2cE3 e 2 ei(φE1 −Epp t1 ) cE1 √ − +   )(V2 − i) − [  2 16t2s − (Epp − Epp1 ) −Epp + Epp1 + (Epp − Epp1 )2 + 16t2s      √  1 i 2φE3 +t1 Epp +Epp1 + (Epp −Epp1 )2 +16ts2 16t2s − (Epp − Epp1 ) −Epp + Epp1 + (Epp − Epp1 )2 + 16t2s cE3 + +[e−i(Epp +Epp1 )t1 (e 2

+cE4 e 2

1

   √  1 i 2φE3 +t1 (Epp +Epp1 + (Epp −Epp1 )2 +16t2s ) (iV4 − V3 )) + Epp − Epp1 + (Epp − Epp1 )2 + 16t2s (([e−i(Epp +Epp1 )t1 (e 2 ×    16t2s − (Epp − Epp1 ) −Epp + Epp1 + (Epp − Epp1 )2 + 16t2s cE3 +

Analytical View on Non-invasive Measurement of Moving Charge 21

22

K. Pomorski

Another step towards more general Hamiltonian for the system of symmetric coupling electrostatic qubits is when we assume the existence of complex values hopping constants it2si + t2sr and it1si + t1sr and in such case the Hamiltonian has the following form ⎞ ⎛ Epp it2si + t2sr it1si + t1sr 0 ⎜ Epp1 0 it1si + t1sr ⎟ ˆ = ⎜ t2sr − it2si ⎟ (62) H ⎝ t1sr − it1si 0 Epp1 it2si + t2sr ⎠ 0 t1sr − it1si t2sr − it2si Epp Later it will be shown that particularly interesting case is Berry phase that can be acquired by single-electron charge qubits. This will be based on the analogies of tight-binding Hamiltonian of single qubit to spin 1/2 particle. Such analogies was firstly expressed by work [3]. We refer to the physical situation two symmetric qubits as by [3] and utilize the correlation function C to capture as to what extent the two electrons are in a correlated state being both either on the left or on the right side that is corresponding to terms N(-,-), N(+,+), or in an anticorrelated state (expressed by terms N(+,-) and N(-,+). Such function is commonly used in spin systems and is a measure of non-classical correlations. Using a tight-binding model describing two electrostatically coupled SELs (Single Electron Lines [3] as it is the case of 2 paralelly placed double quantum dots) and using the same correlation function applicable in the test of Bell theory of entangled spins, we obtain the correlation function C given by formula N (+, +) + N (−, −) − N (−, +) − N (+, −) = N (+, +) + N (−, −) + N (−, +) + N (+, −)   √p √p (t − t )cos(−t (E − E )2 + 4(t − t )2 + φ E1 E2 s1 s2 c1 c2 s1 s2 E10 − φE20 )  =4 (Ec1 − Ec2 )2 + 4(ts1 − ts2 )2  √ √ pE3 pE4 (ts1 + ts2 )cos(−t (Ec1 − Ec2 )2 + 4(ts1 + ts2 )2 + φE30 − φE40 )   + (Ec1 − Ec2 )2 + 4(ts1 + ts2 )2   pE1 − pE2 pE3 − pE4 + , −(Ec1 − Ec2 ) 2 2 2 2 (Ec1 − Ec2 ) + 4(ts1 − ts2 ) (Ec1 − Ec2 ) + 4(ts1 + ts2 ) C=

(63)

where Ec1 and Ec2 are two Coulomb energies corresponding to interactions between nodes ((1, 1 ), (2, 2 )) and ((1, 2 ), (2, 1 )) in the system of 2 coupled Wannier qubits and we have evolution of quantum state expressed as

|ψ >=

and

√

pE1 eiφE10 e

√

E1 t i

pE1 , ..,

E2 t E3 t E4 t √ √ √ |E1 > + pE2 eiφE20 e i |E2 > + pE3 eiφE30 e i |E3 > + pE4 eiφE40 e i |E4

√

(64)

pE4 are probability of occupancy of eigenergies E1 , .., E4 .

Analytical View on Non-invasive Measurement of Moving Charge

and the Hamiltonian of the system is given as ⎛ ⎞ 2Ep + Ec1 ts2 eiβ ts1 eiα 0 −iβ ⎜ 0 ts1 eiα ⎟ ˆ = ⎜ ts2 e−iα 2Ep + Ec2 ⎟ H ⎝ ts1 e 0 2Ep + Ec2 ts2 e+iβ ⎠ 0 ts1 e−iα ts2 e−iβ 2Ep + Ec1

23

(65)

The passage of charged particle close to proximity of two SELs system is inducing new φE10 → φEn10 , φE20 → φEn20 ,φE30 → φEn30 ,φE40 → φEn40 as well as √ √ √ √ √ √ √ √ pE1 → pE1n , pE2 → pE2n , pE3 → pE3n , pE4 → pE4n . Also we encounter α → αn and β → βn as the result of weak measurement conducted on 2SELs by external charged particle. Therefore the correlation function C has changed from C → Cn . Now we consider the interesting case of two electrostatically interacting qubits placed in parallel given as

ˆt = H



Ec1    ⎜ 0 10 10 −Ep (t) t∗s (t) Ep (t) t∗s (t) ⎜ × + × +⎝ 01 01 ts (t) −Ep (t) ts (t) +Ep (t) 0 0

0 Ec2 0 0

0 0 Ec2 0

0 0 ⎟ ⎟= 0 ⎠ Ec1

= E1 |E1 >< E1 | + E2 |E2 >< E2 | + E3 |E3 >< E3 | + E4 |E4 >< E4 | + f12 (t)|E1 >< E2 | + f21 (t)|E2 >< E1 | + +f13 (t)|E1 >< E3 | + f31 (t)|E3 >< E1 | + f14 (t)|E1 >< E4 | + f41 (t)|E4 >< E1 | + +f23 (t)|E2 >< E3 | + f32 (t)|E3 >< E2 | + f24 (t)|E1 >< E4 | + f42 (t)|E4 >< E2 | + ˆ SEL +f34 (t)|E3 >< E4 | + f43 (t)|E4 >< E3 | = H

(66) As it will be shown such system coupled to quantum electromagnetic resonant cavity can be immune from external noise due to the induction of Berry phase. In such case we have the tight-binding Jaymess Cumming model introduced for the first time by [6,7] in the form as

24

K. Pomorski



ˆ = H ˆ ˆ ˆ H QEC + HSEL + HQEC−SEL =  ECav1 |ECav1 >< ECav1 | + ECav2 |ECav2 >< ECav2 | + ECav3 |ECav3 >< ECav3 | + ECav4 |ECav4 >< ECav4 | ×   |E1 >< E1 | + |E2 >< E2 | + |E3 >< E3 | + |E4 >< E4 | +   |ECav1 >< ECav1 | + |ECav2 >< ECav2 | + |ECav3 >< ECav3 | + |ECav4 >< ECav4 | ×  + E1 |E1 >< E1 | + E2 |E2 >< E2 | + E3 |E3 >< E3 | + E4 |E4 >< E4 | + f12 (t)|E1 >< E2 | + f21 (t)|E2 >< E1 | + f43 (t)|E4 >< E3 | + f34 (t)|E3 >< E4 | + f13 (t)|E1 >< E3 | + f31 (t)|E3 >< E1 | + f14 (t)|E1 >< E4 | +  f41 (t)|E4 >< E1 | + f23 (t)|E2 >< E3 | + f32 (t)|E3 >< E2 | + f24 (t)|E1 >< E4 | + f42 (t)|E4 >< E2 | +

+g(ECav1 , E1 , ECav2 , E2 )|ECav2 > |E2 >< ECav1 | < E1 | + g(ECav2 , E2 , ECav1 , E1 )|ECav1 > |E1 >< ECav2 | < E2 | + +g(ECav1 , E1 , ECav2 , E3 )|ECav2 > |E3 >< ECav1 | < E1 | + g(ECav2 , E3 , ECav1 , E1 )|ECav1 > |E1 >< ECav2 | < E3 | + +g(ECav1 , E1 , ECav2 , E4 )|ECav2 > |E4 >< ECav1 | < E1 | + g(ECav2 , E4 , ECav1 , E1 )|ECav1 > |E1 >< ECav2 | +g(ECav1 , E2 , ECav2 , E1 )|ECav2 > |E1 >< ECav1 | < E2 | + g(ECav2 , E1 , ECav1 , E2 )|ECav1 > |E2 >< ECav2 | +g(ECav1 , E2 , ECav2 , E3 )|ECav2 > |E3 >< ECav1 | < E2 | + g(ECav2 , E3 , ECav1 , E2 )|ECav1 > |E2 >< ECav2 | +g(ECav1 , E2 , ECav2 , E4 )|ECav2 > |E4 >< ECav1 | < E2 | + g(ECav2 , E4 , ECav1 , E2 )|ECav1 > |E2 >< ECav2 |

< E4 | + < E1 | + < E3 | + < E4 | +

+g(ECav1 , E3 , ECav2 , E1 )|ECav2 > |E1 >< ECav1 | < E3 | + g(ECav2 , E1 , ECav1 , E3 )|ECav1 +g(ECav1 , E3 , ECav2 , E2 )|ECav2 > |E2 >< ECav1 | < E3 | + g(ECav2 , E2 , ECav1 , E3 )|ECav1 +g(ECav1 , E3 , ECav2 , E4 )|ECav2 > |E4 >< ECav1 | < E3 | + g(ECav2 , E4 , ECav1 , E3 )|ECav1 +g(ECav1 , E4 , ECav2 , E1 )|ECav2 > |E1 >< ECav1 | < E4 | + g(ECav2 , E1 , ECav1 , E4 )|ECav1

> |E3 >< ECav2 | < E1 | +

+g(ECav2 , E1 , ECav3 , E2 )|ECav3 > |E2 >< ECav2 | < E1 | + g(ECav3 , E2 , ECav2 , E1 )|ECav2 +g(ECav2 , E1 , ECav3 , E3 )|ECav3 > |E3 >< ECav2 | < E1 | + g(ECav3 , E3 , ECav2 , E1 )|ECav2 +g(ECav2 , E1 , ECav3 , E4 )|ECav3 > |E4 >< ECav2 | < E1 | + g(ECav3 , E4 , ECav2 , E1 )|ECav2 +g(ECav2 , E2 , ECav3 , E1 )|ECav3 > |E1 >< ECav2 | < E2 | + g(ECav3 , E1 , ECav2 , E2 )|ECav2

> |E1 >< ECav3 | < E2 | + > |E1 >< ECav3 | < E3 | + > |E1 >< ECav3 | < E4 | +

> |E3 >< ECav2 | < E2 | + > |E3 >< ECav2 | < E4 | +

> |E4 >< ECav2 | < E1 | + +g(ECav1 , E4 , ECav2 , E2 )|ECav2 > |E2 >< ECav1 | < E4 | + g(ECav2 , E2 , ECav1 , E4 )|ECav1 > |E4 >< ECav2 | < E2 | + +g(ECav1 , E4 , ECav2 , E3 )|ECav2 > |E3 >< ECav1 | < E4 | + g(ECav2 , E3 , ECav1 , E4 )|ECav1 > |E4 >< ECav2 | < E3 | + +.. +

+g(ECav2 , E2 , ECav3 , E3 )|ECav3 > |E3 >< ECav2 | +g(ECav2 , E2 , ECav3 , E4 )|ECav3 > |E4 >< ECav2 | +g(ECav2 , E3 , ECav3 , E1 )|ECav3 > |E1 >< ECav2 | +g(ECav2 , E3 , ECav3 , E2 )|ECav3 > |E2 >< ECav2 |

> |E2 >< ECav3 | < E1 | +

< E2 | + g(ECav3 , E3 , ECav2 , E2 )|ECav2 > |E2 < E2 | + g(ECav3 , E4 , ECav2 , E2 )|ECav2 > |E2 < E3 | + g(ECav3 , E1 , ECav2 , E3 )|ECav2 > |E3 < E3 | + g(ECav3 , E2 , ECav2 , E3 )|ECav2 > |E3

>< ECav3 | < E3 | >< ECav3 | < E4 | >< ECav3 | < E1 | >< ECav3 | < E2 |

+ + + +

+g(ECav2 , E3 , ECav3 , E4 )|ECav3 > |E4 >< ECav2 | < E3 | + g(ECav3 , E4 , ECav2 , E3 )|ECav2 > |E3 >< ECav3 | < E4 | + +g(ECav2 , E4 , ECav3 , E1 )|ECav3 > |E1 >< ECav2 | < E4 | + g(ECav3 , E1 , ECav2 , E4 )|ECav2 > |E4 >< ECav3 | < E1 | + +g(ECav2 , E4 , ECav3 , E2 )|ECav3 > |E2 >< ECav2 | < E4 | + g(ECav3 , E2 , ECav2 , E4 )|ECav2 > |E4 >< ECav3 | < E2 | + +g(ECav2 , E4 , ECav3 , E3 )|ECav3 > |E3 >< ECav2 | < E4 | + g(ECav3 , E3 , ECav2 , E4 )|ECav2 > |E4 >< ECav3 | < E3 | + +.. + +g(ECav4 , E1 , ECav3 , E2 )|ECav3 > |E2 >< ECav4 | < E1 | + g(ECav3 , E2 , ECav4 , E1 )|ECav4 > |E1 >< ECav3 | < E2 | + +g(ECav4 , E1 , ECav3 , E3 )|ECav3 > |E3 >< ECav4 | < E1 | + g(ECav3 , E3 , ECav4 , E1 )|ECav4 > |E1 >< ECav3 | < E3 | + +g(ECav4 , E1 , ECav3 , E4 )|ECav3 > |E4 >< ECav4 | < E1 | + g(ECav3 , E4 , ECav4 , E1 )|ECav4 > |E1 >< ECav3 | < E4 | + +g(ECav4 , E2 , ECav3 , E1 )|ECav3 > |E1 >< ECav4 | < E2 | + g(ECav3 , E1 , ECav4 , E2 )|ECav4 > |E2 >< ECav3 | < E1 | + +g(ECav4 , E2 , ECav3 , E3 )|ECav3 > |E3 >< ECav4 | +g(ECav4 , E2 , ECav3 , E4 )|ECav3 > |E4 >< ECav4 | +g(ECav4 , E3 , ECav3 , E1 )|ECav3 > |E1 >< ECav4 | +g(ECav4 , E3 , ECav3 , E2 )|ECav3 > |E2 >< ECav4 |

< E2 | + g(ECav3 , E3 , ECav4 , E2 )|ECav4 < E2 | + g(ECav3 , E4 , ECav4 , E2 )|ECav4 < E3 | + g(ECav3 , E1 , ECav4 , E3 )|ECav4 < E3 | + g(ECav3 , E2 , ECav4 , E3 )|ECav4

> |E2 >< ECav3 | < E3 | + > |E2 >< ECav3 | < E4 | + > |E3 >< ECav3 | < E1 | +

> |E3 +g(ECav4 , E3 , ECav3 , E4 )|ECav3 > |E4 >< ECav4 | < E3 | + g(ECav3 , E4 , ECav4 , E3 )|ECav4 > |E3 +g(ECav4 , E4 , ECav3 , E1 )|ECav3 > |E1 >< ECav4 | < E4 | + g(ECav3 , E1 , ECav4 , E4 )|ECav4 > |E4 +g(ECav4 , E4 , ECav3 , E2 )|ECav3 > |E2 >< ECav4 | < E4 | + g(ECav3 , E2 , ECav4 , E4 )|ECav4 > |E4

>< ECav3 | < E2 | + >< ECav3 | < E4 | +

>< ECav3 | < E1 | + >< ECav3 | < E2 | + +g(ECav4 , E4 , ECav3 , E3 )|ECav3 > |E3 >< ECav4 | < E4 | + g(ECav3 , E3 , ECav4 , E4 )|ECav4 > |E4 >< ECav3 | < E3 | = K,L,S,U  =4 (1 − K,L )(1 − K,S )(1 − K,U )(1 − L,S )(1 − L,U )(1 − U,S ) × K,L,S,U =1 ×(g(ECavK , EL , ECavS , EU )|ECavS > |EU >< ECavK | < EL | + g(ECavS , EU , ECavK , EL )|ECavK > |EL >< ECavS | < EU |) + 4 4   (2K − 1)  +[ ω] × IˆSEL + IˆQEC × [ EK |EK >< EK |] + 2 K=1 K=1 +

K,L=4  (1 − K,L )fK,L (t)|EL >< EK |. K,L=1

(67)

We identify that it is possible to construct holonomic SELs system interacting with Quantum Electromagnetic Cavity. It is possible that three ground states are selectively coupled to the excited states by external fields as it was shown by work L.M.Duan, J.I.Cirac and P.Zoller entitled ‘Geometric Manipulation of Trapped Ions for Quantum Computation’ (ArXiv : 0111086v1, 2001). In our case we have three ground states of quantum electromagnetic cavity, ground state of position-based qubit in upper line and ground state of position-based qubit in lower line. The detailed considerations on Quantum Electromagnetic Cavity interacting with single-electron devices is not the main subject of this work. Now we come back to Hamiltonian of the form 62. We obtain four orthogonal eigenstates of the system

    ⎞   (t1si−it1sr)(t2si−it2sr) (Epp−Epp1)2 +4 −2 t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 −Epp +Epp1 ⎟ ⎜ ⎟ ⎜  , ⎜   ⎟    ⎟ ⎜          ⎟ ⎜ 4 −2 2 +t2 2 +t2 2 +t2 2 +t2 +t2 +t2 2 +t2 2 +t2 2 +t2 2 +t2 +t2 +t2 2 +4 −2 ⎟ ⎜2 (E −E ) t t t +t t t t +t −(E −E ) −E +E pp pp pp pp1 pp1 pp1 ⎟ ⎜ 1sr 1sr 2sr 1sr 1sr 2sr 2sr 2sr 1sr 2sr 1si 1si 2si 1si 1si 2si 2si 2si 1si 2si ⎟ ⎜  ⎟ ⎜   ⎟ ⎜ 2 +t1sr2 t2si2 +t2sr2 −t1si2 −t1sr2 ⎟ ⎜ t1si (t1sr+it1si) ⎟ ⎜ ⎟ ⎜  ,   ⎟ ⎜          ⎟ ⎜   ⎟ ⎜ t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 −Epp+Epp1 t1si2 +t1sr2 4 −2 (Epp−Epp1)2 +4 −2 t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 −(Epp−Epp1) ⎟ ⎜ ⎟ ⎜ ⎟  |E1 = ⎜   ⎟ ⎜ ⎟ ⎜ t1si2 +t1sr2 t2si2 +t2sr2 −t2si2 −t2sr2 (t2sr+it2si) ⎟ ⎜ ⎟ ⎜  , ⎟ ⎜        ⎟ ⎜       ⎟ ⎜ ⎟ ⎜ t2si2 +t2sr2 4 −2 (Epp−Epp1)2 +4 −2 t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 −(Epp−Epp1) t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 −Epp+Epp1 ⎟ ⎜ ⎟ ⎜  ⎟ ⎜      ⎟ ⎜ ⎟ ⎜ 2 2 2 2 2 2 2 2 2 (Epp−Epp1) +4 −2 t1si +t1sr t2si +t2sr +t1si +t1sr +t2si +t2sr −Epp+Epp1 ⎟ ⎜ ⎟ ⎜  ⎟ ⎜       ⎟ ⎜         ⎠ ⎝ 2 +t2 +t2 +t2 t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 −(Epp−Epp1) t1si2 +t1sr2 t2si2 +t2sr2 +t1si 24 −2 (Epp −Epp1 )2 +4 −2 2sr −Epp +Epp1 1sr 2si

⎛

    ⎞   (t1si−it1sr)(t2si−it2sr) t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 +Epp−Epp1 (Epp−Epp1)2 +4 −2 ⎟ ⎜ ⎟ ⎜  ,⎟ ⎜       ⎟ ⎜         ⎟ ⎜ 2 +4 −2 2 +t1sr2 t2si2 +t2sr2 (Epp−Epp1) 2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 +Epp−Epp1 +4 −2 2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 ⎟ ⎜2 t1si t1si t1si (Epp−Epp1) ⎟ ⎜ ⎟ ⎜   ⎟ ⎜   ⎟ ⎜ 2 2 ⎟ ⎜ (t1sr+it1si) − t1si2 +t1sr2 t2si2 +t2sr2 +t1si +t1sr ⎟ ⎜ ⎟ ⎜  ,   ⎟ ⎜            ⎟ ⎜   ⎟ ⎜ t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 +Epp−Epp1 +4 −2 t1si2 +t1sr2 (Epp−Epp1) t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 (Epp−Epp1)2 +4 −2 ⎟ ⎜ ⎟ ⎜ ⎟   |E2 = ⎜   ⎟ ⎜ ⎟ ⎜ (t2sr+it2si) − t1si2 +t1sr2 t2si2 +t2sr2 +t2si2 +t2sr2 ⎟ ⎜ ⎟ ⎜  , ⎟ ⎜        ⎟ ⎜         ⎟ ⎜  2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 ⎟ ⎜ +Epp−Epp1 +4 −2 t1si +t1sr t2si +t2sr +t1si +t1sr +t2si +t2sr t2si +t2sr (Epp−Epp1) t1si +t1sr t2si +t2sr +t1si +t1sr +t2si +t2sr (Epp−Epp1) +4 −2 ⎟ ⎜ ⎟ ⎜  ⎟ ⎜      ⎟ ⎜ ⎟ ⎜ 2 2 2 2 2 2 2 2 2 t1si +t1sr t2si +t2sr +t1si +t1sr +t2si +t2sr +Epp−Epp1 (Epp−Epp1) +4 −2 ⎟ ⎜ ⎟ ⎜  ⎟ ⎜      ⎟ ⎜          ⎠ ⎝ 2 (Epp−Epp1) t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 +Epp−Epp1 +4 −2 t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 (Epp−Epp1)2 +4 −2

⎛

Analytical View on Non-invasive Measurement of Moving Charge 25

⎛

    ⎞   2 2 2 2 2 2 2 (t1si −it1sr )(t2si −it2sr ) t2 (Epp −Epp1 )2 +4 2 ⎜ ⎟ 1si +t1sr t2si +t2sr +t1si +t1sr +t2si +t2sr +Epp −Epp1 ⎜ ⎟  ⎜− ⎟      ⎟   ⎜        ⎜ ⎟ (E 2 +t2 2 +t2 2 2 +t2 2 +t2 +t2 +t2 2 +t2 2 +t2 2 +t2 +t2 +t2 2 +4 2 ⎜ 2 ⎟ +E −E +4 2 (E −E ) −E ) t2 t t t +t t t +t +t pp pp pp pp1 pp1 pp1 ⎜ ⎟ 2sr 1sr 2sr 1sr 2sr 1sr 1sr 2sr 1sr 2sr 2si 2si 2si 1si 2si 1si 1si 2si 1si 1si ⎜ ⎟  ⎜ ⎟   ⎜ ⎟ 2 2 ⎜ ⎟ (t1sr+it1si) t1si2 +t1sr2 t2si2 +t2sr2 +t1si +t1sr ⎜ ⎟ ⎜ ⎟  , ,   ⎜ ⎟            ⎜ ⎟   ⎜ ⎟  t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 +Epp−Epp1 +4 2 t1si2 +t1sr2 t2si2 +t2sr2 +t1si2 +t1sr2 +t2si2 +t2sr2 t1si2 +t1sr2 (Epp−Epp1) (Epp−Epp1)2 +4 2 ⎜ ⎟ ⎜ ⎟ ⎟  |E4 = ⎜    ⎜ ⎟ 2 2 ⎜ ⎟ 2 2 2 2 t2si +t2sr +t2si +t2sr (t2sr+it2si) t1si +t1sr ⎜ ⎟ ⎜ ⎟  ⎜ ⎟        ⎜ ⎟         ⎜ ⎟  2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 ⎜ ⎟ t2si +t2sr +t1si +t1sr +t2si +t2sr +Epp−Epp1 +4 2 t2si +t2sr +t1si +t1sr +t2si +t2sr t2si +t2sr t1si +t1sr t1si +t1sr (Epp−Epp1) (Epp−Epp1) +4 2 ⎜ ⎟ ⎜ ⎟  ⎜ ⎟      ⎜ ⎟ ⎜ ⎟ 2 2 2 2 2 2 2 2 2 t1si +t1sr t2si +t2sr +t1si +t1sr +t2si +t2sr +Epp −Epp1 (Epp −Epp1 ) +4 2 ⎜ ⎟ ⎜ ⎟  ⎜ ⎟ ,      ⎜ ⎟          ⎝ ⎠  2 +t2 2 2 2 2 +t2 2 2 2 2 2 2 2 2 2 2 (Epp −Epp1 ) (Epp −Epp1 )2 +4 2 t1si t1si 1sr t2si +t2sr +t1si +t1sr +t2si +t2sr +Epp −Epp1 +4 2 1sr t2si +t2sr +t1si +t1sr +t2si +t2sr

     ⎞   (t1si−it1sr)(t2si−it2sr) −Epp+Epp1+ (Epp−Epp1)2 +4 t1si2 +t1sr2 +t2si2 +t2sr2 +2 t1si2 +t1sr2 t2si2 +t2sr2 ⎟ ⎜ ⎟ ⎜  ⎜−  ⎟        ⎟ ⎜          ⎟ ⎜ ⎟ ⎜ 2 t1si2 +t1sr2 t2si2 +t2sr2 t1si2 +t1sr2 t2si2 +t2sr2 4 t1si2 +t1sr2 +t2si2 +t2sr2 +2 t1si2 +t1sr2 t2si2 +t2sr2 −(Epp−Epp1) −Epp+Epp1+ (Epp−Epp1)2 +4 t1si2 +t1sr2 +t2si2 +t2sr2 +2 ⎟ ⎜ ⎟ ⎜   ⎟ ⎜   ⎟ ⎜ 2 +t1sr2 + 2 +t1sr2 t2si2 +t2sr2 ⎟ ⎜ i(t1si−it1sr) t1si t1si ⎟ ⎜ ⎟ ⎜  −    ⎟ ⎜          ⎟ ⎜   ⎟ ⎜ t1si2 +t1sr2 4 t1si2 +t1sr2 +t2si2 +t2sr2 +2 t1si2 +t1sr2 t2si2 +t2sr2 −(Epp−Epp1) −Epp+Epp1+ (Epp−Epp1)2 +4 t1si2 +t1sr2 +t2si2 +t2sr2 +2 t1si2 +t1sr2 t2si2 +t2sr2 ⎟ ⎜ ⎟ ⎜ ⎟ ⎜   |E3 = ⎜   ⎟ ⎟ ⎜ t1si2 +t1sr2 t2si2 +t2sr2 i(t2si−it2sr) t2si2 +t2sr2 + ⎟ ⎜ ⎟ ⎜  − ⎟ ⎜        ⎟ ⎜        ⎟ ⎜ ⎟ ⎜ t1si2 +t1sr2 t2si2 +t2sr2 −(Epp−Epp1) −Epp+Epp1+ (Epp−Epp1)2 +4 t1si2 +t1sr2 +t2si2 +t2sr2 +2 t1si2 +t1sr2 t2si2 +t2sr2 t2si2 +t2sr2 4 t1si2 +t1sr2 +t2si2 +t2sr2 +2 ⎟ ⎜ ⎟ ⎜  ⎟ ⎜     ⎟ ⎜ ⎟ ⎜ −Epp+Epp1+ (Epp−Epp1)2 +4 t1si2 +t1sr2 +t2si2 +t2sr2 +2 t1si2 +t1sr2 t2si2 +t2sr2 ⎟ ⎜ ⎟ ⎜  ⎟ ⎜       ⎟ ⎜        ⎠ ⎝ 24 t1si2 +t1sr2 +t2si2 +t2sr2 +2 t1si2 +t1sr2 t2si2 +t2sr2 −(Epp−Epp1) −Epp+Epp1+ (Epp−Epp1)2 +4 t1si2 +t1sr2 +t2si2 +t2sr2 +2 t1si2 +t1sr2 t2si2 +t2sr2

⎛

(69)

(68)

26 K. Pomorski

Analytical View on Non-invasive Measurement of Moving Charge

27

with 4 energy eigenvalues

E1 =

   − (Epp − Epp1 )2 + 4 −2 (t21si + t21sr ) (t22si + t22sr ) + t21si + t21sr + t22si + t22sr + Epp + Epp1

1 2

(70)

E2 =

1 2



  (Epp − Epp1 )2 + 4 −2 (t21si + t21sr ) (t22si + t22sr ) + t21si + t21sr + t22si + t22sr + Epp + Epp1

(71)

E3 =

1 2

 − (Epp − Epp1 )2 + 4 2 (t21si + t21sr ) (t22si + t22sr ) + t21si + t21sr + t22si + t22sr

+ Epp + Epp1

(72)

E4 =

1 2

  (Epp − Epp1 )2 + 4 2 (t21si + t21sr ) (t22si + t22sr ) + t21si + t21sr + t22si + t22sr + Epp + Epp1

(73)

4

Concept of Position Dependent Electrostatic Roton Qubit

The idea is depicted in the Fig. 4. Electron placed in the structure has two minima and two maxima in the effective confining potential. After one circulation the wavefunction of such electron is changed by 2Πn phase and is coming back to its original value so we are dealing with periodic boundary conditions. We can express electron dynamics by effective Hamiltonian that is of the form ⎞ ⎛ Ep1 t21 0 t41 ⎜ t12 Ep2 t32 0 ⎟ ⎟ (74) H=⎜ ⎝ 0 t23 Ep3 t43 ⎠ t14 0 t34 Ep4 Hermicity of Hamiltonian implies the condition t∗kl = tlk . If four minima and four maxima of effective confining potential are the same the Hamiltonian reduced to the form ⎞ ⎛ Ep1 ts 0 ts ⎜ ts Ep2 ts 0 ⎟ ⎟ H=⎜ (75) ⎝ 0 ts Ep1 ts ⎠ ts 0 ts Ep2

28

K. Pomorski

After renormalization Hamiltonian can be simplified to be of the form ⎞ ⎛ 0 1 Epq1 1 ⎜ 1 Epq2 1 0 ⎟ ⎟ H=⎜ ⎝ 0 1 Epq1 1 ⎠ 1 0 1 Epq2

(76)

The wavefunction of electron can be expressed in Wannier representation in the following form ⎞ ⎛ γ1 (t) ⎜γ2 (t)⎟ ⎟ (77) |ψ = ⎜ ⎝γ3 (t)⎠ γ4 (t) where |γ1 (t)|2 + .. + |γ4 (t)|2 = 1 and ⎛ ⎞ ⎛ ⎞ 1 0 ⎜0⎟ ⎜0⎟ ⎜ ⎟ , .., ⎜ ⎟ ⎝0⎠ ⎝0⎠ 0 1

(78)

are denoting four orthonormal maximized localized wavefunctions (Wannier functions). The system eigenstates and eigenenergies are given as The passing proton can affect the Hamiltonian in the following way given as H= Ep1 + V1 (t − t1 ) t21 + V3 (t − t1 ) + iV4 (t − t1 ) ⎜t12 + V3 (t − t1 ) − iV4 (t − t1 ) Ep2 ⎜ 0 t23 t14 + V5 (t t1 ) iV6 (t t1 ) 0

0 t41 + V5 (t − t1 ) + iV6 (t − t1 ) ⎟ t32 0 ⎟ Ep3 t43 t34 Ep4

(79)

Such Hamiltonian shall be referred to the situation given in Fig. 3 and in Fig. 4. Let us assume symmetric position-based roton qubit having two minima (denoted by A) and two maxima (denoted by B) of electron confining potential that has periodic boundary condition. Then in first approximation we have semiconductor transmon Hamiltonian of the form

EA + V1 (t − t1 ) ts + V3 (t − t1 ) + iV4 (t − t1 ) ts + V3 (t − t1 ) − iV4 (t − t1 ) EB 0 ts ts + V5 (t − t1 ) − iV6 (t − t1 ) 0

H= 0 ts + V5 (t − t1 ) + iV6 (t − t1 ) ts 0 EA ts ts EB

Such Hamiltonian has eigenvalues as E1 = Ep1 = EA , E2 = Ep2 = EB ,    1 2 2 − (Ep1 − Ep2 ) + 16ts + Ep1 + Ep2 , E3 = 2

(80)

Analytical View on Non-invasive Measurement of Moving Charge

1 E4 = 2

   2 2 + (Ep1 − Ep2 ) + 16ts + Ep1 + Ep2 .

29

(81)

We have eigenvectors 1 |E1  = √ 2

−1 0 +1 0

2

(Ep1 − Ep2 )2 + 16t2s + Ep1 − Ep2

2

+ 32t2s

+ 32t2s

4ts  (Ep1 − Ep2 )2 + 16t2s − Ep1 + Ep2 4ts (Ep1 − Ep2 )2 + 16t2s − Ep1 + Ep2

1

|E4  =  2

(Ep1 − Ep2 )2 + 16t2s − Ep1 + Ep2

2

(82)

−4ts ,  (Ep1 − Ep2 )2 + 16t2s + Ep1 − Ep2 −4ts , (Ep1 Ep2 )2 + 16t2s + Ep1 Ep2

1

|E3  =

0 −1 0 +1

1 , |E2  = √ 2

(83)

(84)

It brings the following effective boundary condition for equation of motion at nearest presence of moving charged particle affecting position based qubit in the form + + − V1 γ1 (t+ 1 ) + (V3 + iV4 )γ2 (t) + (V5 + iV6 )γ4 (t1 ) = i(γ1 (t1 ) − γ1 (t1 )) + − (V3 − iV4 ))γ1 (t+ 1 ) = i(γ2 (t1 ) − γ2 (t1 )) + − (V5 − iV6 )γ1 (t+ 1 ) = i(γ4 (t1 ) − γ4 (t1 ))

(85)

We can express this boundary condition in the matrix form as ⎞ ⎞ ⎞⎛ ⎛ ⎛ γ1 (t− γ1 (t+ (V1 − i) (V3 + iV4 ) 0 (V5 + iV6 ) 1) 1) − ⎟ ⎟ ⎜γ2 (t+ )⎟ ⎜(V3 − iV4 ) ⎜ −i 0 0 1 ⎟ = −i ⎜γ2 (t1 )⎟ (86) ⎟⎜ ⎜ + ⎠ ⎠ ⎠ ⎝ ⎝ ⎝ 0 0 −i 0 γ3 (t1 ) γ3 (t− 1) + − 0 0 −i (V5 − iV6 ) γ4 (t1 ) γ4 (t1 ) Equivalently last matrix equation can be represented as γ1 (t+ 1 ) γ2 (t+ 1 ) γ3 (t+ 1 ) γ4 (t+ 1 )

(V1 − i) (V3 + iV4 ) (V3 − iV4 ) −i 0 0 (V5 − iV6 ) 0

= −i

−1

0 (V5 + iV6 ) 0 0 −i 0 0 −i

γ1 (t− 1 ) γ2 (t− 1 ) γ3 (t− 1 ) γ4 (t− 1 )

ˆ =M

γ1 (t− 1 ) γ2 (t− 1 ) γ3 (t− 1 ) γ4 (t− 1 )

(87)

and we obtain matrix M of the following structure

M=

2 2 +iV1 +V32 +V42 +V52 +V62 (iV3 +V4 ) − 2 +iV +V 2 +V 2 +V 2 +V 2 1 3 4 5 6

0

(iV5 +V6 ) 2 2 2 2 1 +V3 +V4 +V5 +V6

− 2 +iV

1

(V4 −iV3 ) 2 +iV1 +V32 +V42 +V52 +V62 V 2 +V 2 − 2 +iV +V3 2 +V4 2 +V 2 +V 2 1 3 4 5 6

0

0 0 1

(V3 +iV4 )(V5 −iV6 ) − 2 +iV 0 +V 2 +V 2 +V 2 +V 2 1

3

4

5

6

(V6 −iV5 ) 2 +iV1 +V32 +V42 +V52 +V62 (V3 −iV4 )(V5 +iV6 ) − 2 +iV +V 2 +V 2 +V 2 +V 2 1 3 4 5 6

0

2 +iV1 +V32 +V42 2 +iV1 +V32 +V42 +V52 +V62

(88)

30

4.1

K. Pomorski

Case of Symmetric Semiconductor Roton

We can also consider the simplest version of semiconductor roton qubit. Such system can be approximated by four symmetric quantum dots on the square or by semiconductor nanoring of radius R. In both cases the Hamiltonian can be given as ⎞ ⎛ 0 ts eiα Ep ts eiα ⎜ts e−iα Ep ts eiα 0 ⎟ ⎟ (89) H=⎜ ⎝ 0 ts e−iα Ep ts eiα ⎠ −iα ts 0 ts e Ep that has four energy eigenvalues E1 = Ep − 2ts | sin

α α α α |, E2 = Ep + 2ts | sin |, E3 = Ep − 2ts | cos( )|, E4 = Ep + 2ts | cos( )| 2 2 2 2

(90)

and has 4 eigenstates ⎞ ⎞ ⎛ 3 3 −iei 2 α +iei 2 α iα ⎟ ⎜ 1 ⎜ −eiα ⎟ ⎟ , |E2  = 1 ⎜ −e α ⎟ |E1  = ⎜ α i 2 ⎝ ie 2 ⎠ 2 ⎝ −iei 2 ⎠ 1 1 ⎛ i3α ⎞ ⎛ i3α ⎞ −e 2 e 2 iα ⎟ ⎜ 1⎜ 1 +e ⎟ eiα ⎟ ⎟. , |E4  = ⎜ |E3  = ⎜ α iα ⎠ ⎝ ⎝ 2 −e ei 2 ⎠ 2 2 1 1 ⎛

(91)

The presence of electron at nodes 1, 2, 3, 4 is expressed by vectors ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ 1 0 0 0 ⎜0⎟ ⎜1⎟ ⎜0⎟ ⎜0⎟ ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎟ |x1  = ⎜ ⎝ 0 ⎠ , |x2  = ⎝ 0 ⎠ , |x3  = ⎝ 1 ⎠ , |x4  = ⎝ 0 ⎠ , 0 0 0 1 |ψ(t) = γ1 (t) |x1  + .. + γ1 (t) |x4  , |γ1 (t)|2 + .. + |γ4 (t)|2 = 1.

(92)

In all energy eigenstates we see the systematic phase imprint across ring of semiconductor quantum dots what might correspond to the placement of semiconductor ring to the external magnetic field. It is due to magnetic Aharonov-Bohm effect. Otherwise with case of placement of semiconductoring of quantum dots in environment with zero magnetic field we have α = 0. Kinetic energy and momentum of quantum roton (semiconductor “transmon” qubit) is due to non-zero term ts . Otherwise we are dealing with the case when electron position is localized among one between four nodes. If we refer to description of semiconductor roton qubit as ring of quantum dots of radius R we have the condition for single-value wavefunction after one circulation across ring/square expressed mathematically n . This also implies that ts = nt0 by condition 2πkn R = 2πn that implies kn = R

Analytical View on Non-invasive Measurement of Moving Charge

31

and n is integer number. In the conducted considerations we have set n=1 so inside semiconductor ring we have one fluxon (one quantized flux of magnetic field is passing via interior of quantum ring so we have approximate mathemat2 ical condition for flux quantization Ba2 = e n in case of 4 symmetric quantum 2 dots on the square and with condition 2πR2 B = e n in case of nanoring, where B is value of magnetic field in the center of square/nanoring that is presumed to be constant in square/nanoring). We also recognize that 4α = 2Πn what√ brings √ √ √ √ 2 2 2 2 πn α α = 2 and consequently Sin( 2 ) = + 2 , 1, + 2 , 0, − 2 , −1, − 2 , 0, + 22 , .. as √ √ √ √ √ well as. Cos( α2 ) = 0, − 22 , −1, − 22 , 0, + 22 , 1, + 22 , 0, − 22 , .. and therefore the spectrum of eigenenergies E1 , .., E2 can be tunned with magnetic field. This tunning however implies discrete eigenenergies. We can also regulate the spectrum of eigenenergies by electric field as by tunning Ep what we can do in the continuous manner (as chemical potential can be tunned in the continuous manner). Furthermore the Hamiltonian 89 can also refer to the square of four symmetric quantum dots with confining potential having four minima and four maxima. In such case we can tune both ts and Ep in electrostatic way. It is also noticeable that the system of four symmetric quantum dots can be the source of magnetic field as when we are dealing with wavepacket of one electron moving in clockwise or anticlockwise direction and generating one flux (or many) of magnetic field 2 (fluxon given by e ). In such situation we have physical system that is similar to superconducting SQUID that can generate or detect one flux of magnetic field with occurrence of circulating non-dissipative electric current. In both cases we are assuming that quantum wavepacket is coherence along semiconductor or semiconductor ring. However it shall be underlined that in the superconductor the coherence time of wavepacket can be significantly bigger than in the case of semiconductor where decoherence time is in the range of ns. This is because superconductivity is the manifestation of macroscopic quantum effect that is thermodynamically supported so in first approximation non-dissipative circulating current in mesoscopic size SQUID (usually having bigger size that 300 nm 2 and with value of quantized magnetic flux set to 2e , where factor two comes from the fact that Cooper pair participates in electrical transport) can last infinitely long time. This is not the case of semiconductor nanorings that has size usually below 300 nm. We can presume that before weak measurement of probing charge particle we have the quantum system given as |ψ(t1 ) = cE1 e

E1 t1 i

eiφE1 |E1  + ... + cE4 e

E4 t1 i

eiφE4 |E4  ,

(93)

where |cE1 |2 , ... |cE4 |2 are the probabilities of occupancy of energy level E1 , ... E4 by single electron in roton qubit and normalization condition holds |cE1 |2 + ... + |cE4 |2 = 1. After measurement at time t1 by probing particle we have − − E t E t  +  ψ(t ) = cE1m e 1i1 eiφE1m |E1  + .. + cE4m e 4i1 eiφE4m |E4  =

1

ˆ (cE1 e M

− E1 t1 i

eiφE1 |E1  + .. + cE4 e

− E4 t1 i

eiφE4 |E4 ),

(94)

32

K. Pomorski

Again we obtain cE1m e cE4m e

+ E1 t1 i

+ E4 t1 i

e

ˆ (cE1 e eiφE1m = E1 | M

iφE4m

ˆ (cE1 e = E4 | M

− E1 t1 i

− E1 t1 i

eiφE1 |E1  + .. + cE4 e

e

iφE1

|E1  + .. + cE4 e

− E4 t1 i

− E4 t1 i

eiφE4 |E4 ), ..

e

iφE4

|E4 ).

(95)

Consequently we obtain    2 α it1 Ep −2 t2 s sin ( 2 )           √ √ α 2 2 i φ +2t t sin ( 2 ) i φ −2t t2 cos2 ( α i φE3 + 2t1 t2 i φ −2t t2 sin2 ( α s (cos(α)+1) 2 ) +c 2) ) (e−iEp t1 (e E1 1 s cE1 + cE4 e E4 1 s + cE2 e E2 1 s E3 e      √ √  2 α i φ +2t t2 sin2 ( α i φE4 −2t1 t2 i φ + 2t1 t2 s cos ( 2 ) s (cos(α)+1) 2) c 2 + iV1  + V32 + V42 −ei(α−Ep t1 ) (−e E1 1 s +cE3 e E3 E1 +cE4 e       √ 2 α i φ −2t t2 sin2 ( α i α+φE1 −Ep t1 +2t1 t2 s sin ( 2 ) 2 ) )(V + iV )(V − iV )+ +(e −cE2 e E2 1 s −eiα cE1 + 3 4 5 6

cE1m eiφE1m = [e

  √ √ √ −it1 Ep + 2 t2 s (cos(α)+1) eiα



+eiα (e

   2 α i φE3 +4t1 t2 s cos ( 2 )

cE3 e

− cE4 eiφE4

    √ 2 α i φ −t E +2 t2 s sin ( 2 ) −cE2 e E2 1 p −eiα ))(iV5 + V6 )  √ 2 sin2 ( α ) √ i φ −E t +2t t p 1 1 E1 s 2 − −e−iα (e −eiα cE1 +       √ √ √ 2 −it E + 2 ts (cos(α)+1) i φ +4t t2 cos2 ( α 2) eiα cE4 eiφE4 − cE3 e E3 1 s +e 1 p     √ 2 α i φE2 −t1 Ep +2 t2 s sin ( 2 )

−cE2 e −e

−iα

−eiα )    √ 2 α i α+φE1 −Ep t1 +2t1 t2 s sin ( 2 )

2 + iV1  + V32 + V42 + V52 + V62



((e         √ √ √ √ 2 α −it E + 2 t2 i φ +4t t2 cos2 ( α i φ −t E +2 t2 iφE4 s (cos(α)+1) s sin ( 2 ) 2) − c − cE2 e E2 1 p +eiα e 1 p eiα cE3 e E3 1 s −eiα )(iV3 + E4 e             √ √ 2 α i φ +2t t2 sin2 ( α i φE4 −2t1 t2 i φ + 2t1 t2 i φ −2t t2 sin2 ( α s cos ( 2 ) s (cos(α)+1) 2) c 2) V4 )−e−iEp t1 e E1 1 s + cE3 e E3 + cE2 e E2 1 s E1 + cE4 e         √2 √ 2 cos2 ( α ) i φ +2t t2 sin2 ( α i φE4 −2t1 ts i φ + 2t1 ts (cos(α)+1) p t1 ) 2) c 2 )(V5 +iV6 ) + ei(α−E (−e E1 1 s + cE3 e E3 (V3 − iV E1 + cE4 e 4   α 2 2 i φ −2t t sin ( 2 ) −cE2 e E2 1 s ) 2 + iV1  + V52 + V62 )+     √   2 α i φ −t E +2 t2 s sin ( 2 ) +[e−2iα (cE2 e E2 1 p eiα − −eiα  − iV3 + V4 + iV5 − V6 +      √ √ √   2 sin2 ( α ) 2 (cos(α)+1) i φE1 −Ep t1 +2t1 ts −it1 2Ep +3 2 ts iα iα 2 e −e  − iV3 + V4 + iV5 − V6 + e +cE1 e    √ √ 3   2 i φ +t E +4 2 ts (cos(α)+1)  eiα 2 + i eiα (V3 + iV4 ) + V5 + iV6 + (cE3 e E3 1 p    √ √ 3 i φE4 +t1 Ep +2 2 t2 s (cos(α)+1) −cE4 e  eiα 2 + eiα (V4 − iV3 ) − iV5 + V6 ))] √ 1−iα )] × 2 +iV +V 21+V 2 +V 2 +V 2 iα  −e cE1 +

−e

4.2

1

3

4

5

6

Case of Asymmetric Semiconductor Quantum Roton

Symmetric quantum roton can be implemented with usage of concept of positionbased qubits [4] and can be controlled both electrostatically or magnetically. Let us consider slightly more general physical system of two symmetric quantum dots with localized energy set to Ep1 and two the same energy barriers with localized energy set to Ep2 such that Ep1 >> Ep2 or at least Ep1 > Ep2 (Condition Ep1 = Ep2 defines asymmetric quantum roton). Similarly as before we are dealing with magnetic flux quantization in the interior of considered ring. Therefore we have that α = 0 is equivalent to the fact of absence of external magnetic field. Assuming possible hopping given by term ts between nearest nodes via or over potential barrier we arrive to the Hamiltonian given as ⎛ ⎞ 0 eiα ts Ep1 eiα ts ⎜ e−iα ts Ep2 eiα ts 0 ⎟ ⎟ H=⎜ (96) ⎝ 0 e−iα ts Ep1 eiα ts ⎠ −iα −iα e ts 0 e ts Ep2

Analytical View on Non-invasive Measurement of Moving Charge

that has four energy eigenvalues    α 1 − 16t2s sin2 + (Ep1 − Ep2 )2 + Ep1 + Ep2 E1 = 2 2      1 2 α 2 2 + 16ts sin + (Ep1 − Ep2 ) + Ep1 + Ep2 E2 = 2 2    α 1 − 16t2s cos2 + (Ep1 − Ep2 )2 + Ep1 + Ep2 E3 = 2 2     1 α + 16t2s cos2 + (Ep1 − Ep2 )2 + Ep1 + Ep2 E4 = 2 2

33

(97) (98) (99) (100)

and we have four orthonormal eigenstates of the system 1 × |E1  =   2  2 2 2 + −8ts cos(α) + (Ep1 − Ep2 ) + 8t2s + (Ep1 − Ep2 ) − 16t2s (+ cos(α) − 1)  2eiα −1 + eiα ts ,   −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) , −eiα  iα −2 −1 + e ts , 8t2s cos(α) + (Ep1 Ep2 )2 + 8t2s + Ep1 Ep2

(101)

1 × |E2  =   2  2 2 2 − −8ts cos(α) + (Ep1 − Ep2 ) + 8t2s + (Ep1 − Ep2 ) + 16t2s (− cos(α) + 1)  2eiα −1 + eiα ts    iα 2 − −8ts cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 , −e −2 −1 + eiα ts , 8t2s cos(α) + (Ep1 Ep2 )2 + 8t2s + Ep1

Ep2

(102)

1 × |E3  =   2  2 + 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)  −2eiα 1 + eiα ts ,   iα 2 2 e 8ts cos(α) + (Ep1 − Ep2 ) + 8t2s + Ep1 − Ep2 , −2 1 + eiα ts , 8t2s cos(α) + (Ep1 Ep2 )2 + 8t2s + Ep1 Ep2

(103)

1 × |E4  =   2  2 − 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)  ⎛ ⎞ −2eiα 1 + eiα ts ,    ⎜ iα ⎟ 2 2 2 − 8ts cos(α) + (Ep1 − Ep2 ) + 8ts + Ep1 − Ep2 ,⎟ ⎜e ⎜ ⎟=  iα ⎝ ⎠ 1 + e , t −2 s  − 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 1 =  × 2  Ep1 −Ep2 2 (E −E ) 2 − 8 cos(α) + ( ts ) + 8 + p1ts p2 + 16(+ cos(α) + 1)  −2eiα 1 + eiα ,   E −E eiα − 8 cos(α) + ( p1ts p2 )2 + 8 +  −2 1 + eiα , 8 cos(α) + (

Ep1 −Ep2 2 ) ts

+8+

Ep1 −Ep2 ts

Ep1 −Ep2 ts

,

(104)

34

K. Pomorski

It is remarkable to recognize that the occupancy at nodes 1, 2, 3 and 4 E −E is function both of electric (expressed by factor p1ts p2 ) and magnetic field (expressed by non-zero α coefficient). Referring to the state |E4  we have the same probabilities of occupancy nodes 1 and 3 set to 4((cos(α) + 1)2 + (sin(α))2 )  P1 = P3 =   2 Ep1 −Ep2 2 E −E 2 − 8 cos(α) + ( ts ) + 8 + ( p1ts p2 ) + 16(+ cos(α) + 1)

(105)

and probability of occupancy at nodes 2 and 4 nodes set to

P2 = P4 = 

 E −E (− 8 cos(α) + ( p1ts p2 )2 + 8 + 2 − 8 cos(α) + (

Ep1 −Ep2 2 ) ts

+8+(

Ep1 −Ep2 2 ) ts



2

Ep1 −Ep2 ) ts

+ 16(+ cos(α) + 1)

(106)

We use the property (+ cos(α) + 1) = (+ cos( α2 )2 − sin( α2 )2 + 1) = +2 cos( α2 )2 and the property (− cos(α) + 1) = (− cos( α2 )2 + sin( α2 )2 + 1) = +2 sin( α2 )2 and in such case of eigenstate |E4  we obtain the ratio of probabilities

=

  4 2(1 + cos(α)) 4 (cos(α) + 1)2 + (sin(α))2   = E −E E −E E −E (− 8 cos(α) + ( p1ts p2 )2 + 8 + p1ts p2 )2 (− 8(1 + cos(α)) + ( p1ts p2 )2 + =

P1 P3 = = P2 P4 =

Ep1 −Ep2 2 ) ts

8| cos( α2 )| (−

16 cos( α2 )2

+(

Ep1 −Ep2 2 ) ts

+

Ep1 −Ep2 2 ) ts

(107)

In case of state |E3  we have the ratio of probability occupancies given as

=

 4 (cos(α) + 1)2 + (sin(α))2

 E −E (+ 8 cos(α) + ( p1ts p2 )2 + 8 +

Ep1 −Ep2 2 ) ts

P1 P3 = = P2 P4



4 2(1 + cos(α)) =  E −E ( 8(1 + cos(α)) + ( p1ts p2 )2 + =

Ep1 −Ep2 2 ) ts

=

8| cos( α2 )| (+

16 cos( α2 )2

+(

Ep1 −Ep2 2 ) ts

+

Ep1 −Ep2 2 ) ts

(108)

In case of state |E2  we have the ratio of probability occupancies given as

=

P1 P3 = = P2 P4





4 2(1 − cos(α)) 4 (cos(α) − 1)2 + (sin(α))2   = E −E E −E E −E (− −8 cos(α) + ( p1ts p2 )2 + 8 + p1ts p2 )2 (− 8(1 − cos(α)) + ( p1ts p2 )2 + =

Ep1 −Ep2 2 ) ts

=

8| sin( α2 )| (−

16 sin( α2 )2

+(

Ep1 −Ep2 2 ) ts

+

Ep1 −Ep2 2 ) ts

(109)

Analytical View on Non-invasive Measurement of Moving Charge

35

In case of state |E1  we have the ratio of probability occupancies given as

=

P1 P3 = = P2 P4





4 2(1 − cos(α)) 4 (cos(α) − 1)2 + (sin(α))2   = E −E E −E E −E (+ −8 cos(α) + ( p1ts p2 )2 + 8 + p1ts p2 )2 (+ 8(1 − cos(α)) + ( p1ts p2 )2 + =

Ep1 −Ep2 2 ) ts

=

8| sin( α2 )| (+ 16 sin( α2 )2 + (

Ep1 −Ep2 2 ) ts

+

Ep1 −Ep2 2 ) ts

(110)

In general case when the quantum state is superposition of four energetic levels expressed by the equation |ψ(t) = cE1 eiφE1 e

E1 t i

|E1  + cE2 eiφE2 e

E2 t i

|E2  + .. + cE4 eiφE4 e

E4 t i

|E4  . (111)

and probability of finding particle at node 1 is given by P1 (t) = | x1 | |ψ(t) |2 = |cE1 eiφE1 e

E1 t i

E2 t i

E4 t

x1 | |E2  + .. + cE4 eiφE4 e i x1 | |E4  |2 =  cE1 eiφE1 e 2eiα −1 + eiα ts =   + 2  2 + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) − 16t2s (+ cos(α) − 1)  E2 t cE2 eiφE2 e i 2ts eiα −1 + eiα +  + 2  2 − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (− cos(α) + 1)  E3 t −cE3 eiφE3 e i 2ts eiα 1 + eiα +  + 2  2 + 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)  E4 t −cE4 eiφE4 e i 2ts eiα 1 + eiα +   2  2 − 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)

 E1 t cE1 e−iφE1 e− i 2e−iα −1 + e−iα ts   + 2  2 + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) − 16t2s (+ cos(α) − 1)  E2 t cE2 e−iφE2 e− i 2ts e−iα −1 + e−iα +  +  2  2 − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (− cos(α) + 1)  E3 t −cE3 e−iφE3 e− i 2ts e−iα 1 + e−iα +  +  2  2 + 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)  E4 t −cE4 e−iφE4 e− i 2ts e−iα 1 + e−iα + x1 | |E1  + cE2 eiφE2 e E1 t i

2 − 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 )

2

+ 16t2s (+ cos(α) + 1)

(112)

36

K. Pomorski

and E1 t

E2 t

E4 t

P2 (t) = | x2 | |ψ(t) |2 = |cE1 eiφE1 e i x2 | |E1  + cE2 eiφE2 e i x2 | |E2  + .. + cE4 eiφE4 e i x2 | |E4  |2 =    E1 t

cE1 eiφE1 e i (−1)eiα + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 + =   2  2 + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) − 16t2s (+ cos(α) − 1)    E2 t cE2 eiφE2 e i (−1)eiα − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2  + +   2 2 − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (− cos(α) + 1)    E3 t cE3 eiφE3 e i e+iα + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 + +  2  2 + 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)    E4 t cE4 eiφE4 e i e+iα − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 +  2  2 − 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)    E1 t

cE1 e−iφE1 e− i (−1)e−iα + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2   + 2  2 + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) − 16t2s (+ cos(α) − 1)    E2 t cE2 e−iφE2 e− i (−1)e−iα − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 + +  2  2 − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (− cos(α) + 1)    E3 t cE3 e−iφE3 e− i e−iα + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 + +  2  2 + 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)    E4 t cE4 e−iφE4 e− i e−iα − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 +

2 − 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 )

2

+ 16t2s (+ cos(α) + 1)

Analytical View on Non-invasive Measurement of Moving Charge

37

and P3 (t) = | x3 | |ψ(t) |2 = |cE1 eiφE1 e

E1 t i

E2 t i

E4 t

x3 | |E2  + .. + cE4 eiφE4 e i x3 | |E4  |2 =  E1 t iφE1 i e (−2)ts −1 + eiα ts cE1 e + =   2  2 + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) − 16t2s (+ cos(α) − 1)  E2 t cE2 eiφE2 e i (−2)ts −1 + eiα +  + 2  2 − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (− cos(α) + 1)  E3 t cE3 eiφE3 e i (−2)ts +1 + eiα +  + 2  2 + 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)  E4 t cE4 eiφE4 e i (−2)ts 1 + eiα +  2  2 − 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)

 E1 t cE1 e−iφE1 e− i (−2) −1 + e−iα ts   + 2  2 + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) − 16t2s (+ cos(α) − 1)  E2 t cE2 e−iφE2 e− i (−2)ts −1 + e−iα +  + 2  2 − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (− cos(α) + 1)  E3 t cE3 e−iφE3 e− i (−2)ts 1 + e−iα + +  2  2 + 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)  E4 t cE4 e−iφE4 e− i (−2ts ) 1 + e−iα + x3 | |E1  + cE2 eiφE2 e

2 − 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 )

and

2

+ 16t2s (+ cos(α) + 1)

38

K. Pomorski

E1 t

E2 t

E4 t

P4 (t) = | x4 | |ψ(t) |2 = |cE1 eiφE1 e i x4 | |E1  + cE2 eiφE2 e i x4 | |E2  + .. + cE4 eiφE4 e i x4 | |E4  |2 =    E1 t

cE1 eiφE1 e i + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 + =   2  2 + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) − 16t2s (+ cos(α) − 1)    E2 t cE2 eiφE2 e i − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 + +  2  2 − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (− cos(α) + 1)    E3 t cE3 eiφE3 e i + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 + +  2  2 + 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)    E4 t cE4 eiφE4 e i − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 +  2  2 − 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)    E1 t

cE1 e−iφE1 e− i + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2   + 2  2 + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) − 16t2s (+ cos(α) − 1)    E2 t cE2 e−iφE2 e− i − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 + +  2  2 − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (− cos(α) + 1)    E3 t cE3 e−iφE3 e− i + −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2 + +  2  2 + 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 ) + 16t2s (+ cos(α) + 1)    E4 t cE4 e−iφE4 e− i − −8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + Ep1 − Ep2  + 2 − 8t2s cos(α) + (Ep1 − Ep2 )2 + 8t2s + (Ep1 − Ep2 )

2

+ 16t2s (+ cos(α) + 1)

(113) 4.3

Action of Phase Rotating Gate Described Analytically

We inspect the derivation of formula for ration between ψ2 (t) and ψ1 (t) in case of situation Ep1 = Ep2 = Ep = constant and with ts = constant ∈ R. The given assumptions that two system eigenenergies are time independent. However we will assume that due to magnetic Aharonov-Bohm effect it is possible to have phase imprint on position based qubit between nodes 2 and 1. This phase imprint does not change the system energy but changes the system quantum state what has its importance in quantum information processing. In the conducted considerations the probabilities of occupancy of eigenenergies E1 and E2 are constant and determined by initial conditions and are |αE1 |2 and |αE2 |2 . We have |αE2 (t)| = constant, |αE2 (t)| = constant, φE2 (t) = constant and φE1 (t) = constant. The action of external source of vector potential is represented by phase imprints δ1 (t) and δ2 (t) at qubit nodes 1 and 2 and can be implemented by external conductor of electric current or charge moving in free space in the proximity to position-based qubit. We obtain

Analytical View on Non-invasive Measurement of Moving Charge

39

ψ2 (t) = ψ1 (t) =

(−e (+e

−i E1 t 

eiφE1 (0) |αE1 | + e

−i E2 t 

−i E1 t 

eiφE1 (0) |αE1 | + e

−i E2 t 

= =

(−|αE1 | + e (+|αE1 | + e

(−|αE1 | + e

−i (E2 −E1 )t 

(−|αE1 | + e |βE2 |eiφE2 (0) ) eiδ2 (t) |βE2 |ei(φE2 (0)−φE2 (0)) ) i(δ2 (t)−δ1 (t)) = = e −i iδ (t) |βE2 |eiφE2 (0) ) e 1 (+|αE1 | + e  (E2 −E1 ))t |βE2 |ei(φE2 (0)−φE1 (0)) )

−i (E2 −E1 )t 

|βE2 |ei(φE2 (0)−φE2 (0)) )(+|αE1 | + e

+i (E2 −E1 ))t 

|βE2 |e−i(φE2 (0)−φE1 (0)) )

−i (E2 −E1 )t 

|βE2 |ei(φE2 (0)−φE2 (0)) )(+|αE1 | + e

+i (E2 −E1 ))t 

|βE2 |e−i(φE2 (0)−φE1 (0)) )

−i (E2 −E1 )t 

|βE2 |ei(φE2 (0)−φE2 (0)) )(+|αE1 | + e

+i (E2 −E1 ))t 

|βE2 |e−i(φE2 (0)−φE1 (0)) )

1 )t 1 )t ([+|αE1 | + |βE2 |cos( (E2 −E − (φE2 (0) − φE1 (0)))]2 + [|βE2 |sin( (E2 −E − (φE2 (0) − φE1 (0)))]2  

= =

(−|αE1 | + e

−i (E2 −E1 )t 

|βE2 |ei(φE2 (0)−φE2 (0)) )(+|αE1 | + e

1+

1 )t 2|αE1 ||βE2 |cos( (E2 −E 

(|βE2 |2 − |αE1 |2 + |αE2 ||βE2 |(e

−i (E2 −E1 )t 

+i (E2 −E1 ))t 

|βE2 |e−i(φE2 (0)−φE1 (0)) )

− (φE2 (0) − φE1 (0)))

ei(φE2 (0)−φE2 (0)) − e

+i (E2 −E1 )t 

e−i(φE2 (0)−φE2 (0)) ))

1 )t 1 + 2|αE1 ||βE2 |cos( (E2 −E − (φE2 (0) − φE1 (0))) 

(|βE2 |2 − |αE1 |2 ) − 2i|αE2 ||βE2 sin[(E2 − E1 )t − (φE2 (0) − φE2 (0))]

= = +i

1 )t 1 + 2|αE1 ||βE2 |cos( (E2 −E − (φE2 (0) − φE1 (0))) 

ei(δ2 (t)−δ1 (t)) = ei(δ2 (t)−δ1 (t)) = ei(δ2 (t)−δ1 (t)) = ei(δ2 (t)−δ1 (t)) = ei(δ2 (t)−δ1 (t)) =

(|βE2 |2 − |αE1 |2 ) cos(δ21 (t)) + 2 sin(δ21 (t)|αE2 ||βE2 sin[(E2 − E1 )t − (φE2 (0) − φE2 (0))] 1 )t 1 + 2|αE1 ||βE2 |cos( (E2 −E − (φE2 (0) − φE1 (0))) 

2

2

+

(|βE2 | − |αE1 | ) sin(δ21 (t)) − 2 cos(δ21 (t)|αE2 ||βE2 sin[(E2 − E1 )t − (φE2 (0) − φE2 (0))]

= 1 )t 1 + 2|αE1 ||βE2 |cos( (E2 −E − (φE2 (0) − φE1 (0)))  1 × = 1 )t 1 + 2|αE1 ||βE2 |cos( (E2 −E − (φE2 (0) − φE1 (0))) 

(|βE2 |2 −|αE1 |2 ) cos(δ21 (t))+2 sin(δ21 (t)|αE2 ||βE2 sin[(E2 −E1 )t−(φE2 (0)−φE2 (0))] (|βE2 |2 −|αE1 |2 ) sin(δ21 (t))−2 cos(δ21 (t)|αE2 ||βE2 sin[(E2 −E1 )t−(φE2 (0)−φE2 (0))]

+iArcT an

×e =

|ψ2 (t)| e |ψ1 (t)|

iArcT an

(|βE2 |2 −|αE1 |2 ) cos(δ21 (t))+2 sin(δ21 (t)|αE2 ||βE2 sin[(E2 −E1 )t−(φE2 (0)−φE2 (0))] (|βE2 |2 −|αE1 |2 ) sin(δ21 (t))−2 cos(δ21 (t)|αE2 ||βE2 sin[(E2 −E1 )t−(φE2 (0)−φE2 (0))]

=

1 1 )t 1 + 2|αE1 ||βE2 |cos( (E2 −E − (φE2 (0) − φE1 (0))) 

= =

eiφ(t)

(114) and finally we obtain 1 |ψ2 (t)| = , (E −E 2 1 )t |ψ1 (t)| 1 + 2|αE1 ||βE2 |cos( − (φE2 (0) − φE1 (0))) 

(115)

Fig. 3. Sensing of moving charge by the semiconductor transmon qubit (roton qubit). In first approximation only effect of externally moving charge at node 1 is accounted and hopping terms to the nearest neighbours are considered to be changed.

40

K. Pomorski

Fig. 4. Family of semiconductor position-based transmon qubits that can be single or many electron qubits as well as energy eigenvalue qubits or Wannier qubits or spin qubits.

and (|βE2 |2 − |αE1 |2 ) sin( 21 (t)) − 2 cos( 21 (t)|αE1 ||βE2 | sin[(E2 − E1 )t − (φE2 (0) − φE1 (0))] = (|βE2 |2 − |αE1 |2 ) cos( 21 (t)) + 2 sin( 21 (t)|αE1 ||βE2 sin[(E2 − E1 )t − (φE2 (0) − φE1 (0))]

(|βE2 |2 − |αE1 |2 )tan( 21 (t)) − 2|αE1 ||βE2 | sin[(E2 − E1 )t − (φE2 (0) − φE1 (0))] = = ArcT an (|βE2 |2 − |αE1 |2 ) + 2tan( 21 (t)|αE1 ||βE2 sin[(E2 − E1 )t − (φE2 (0) − φE1 (0))]

|αE1 ||βE2 | tan( 21 (t)) − 2 (|βE2 |2 −|αE1 |2 ) sin[(E2 − E1 )t − (φE2 (0) − φE1 (0))] = ArcT an |αE1 ||βE2 1 + 2tan( 21 (t) |βE2 |2 −|αE1 |2 sin[(E2 − E1 )t − (φE2 (0) − φE1 (0))]

φ(t) = ArcT an

(116) |αE1 ||βE2 | Let us set p = tan(φ(t)) and s1 = 2 (|βE2 |2 −|αE1 |2 ) and we obtain

tan(δ21 (t)) − s1 sin(φs (t) = p + s1 sin(φs (t))tan(δ21 (t))p

(117)

and therefore we have arrive to the expression |αE1 ||βE2 | tan(φ(t)) + (2 (|βE2 tan(φ(t)) + s1 sin(φs (t)) p + s1 sin(φs (t)) |2 −|αE1 |2 ) ) sin(φs (t)) = = = |αE1 ||βE2 | 1 − ps1 sin(φs (t)) 1 − tan(φ(t))s1 sin(φs (t)) 1 − tan(φ(t))(2 (|βE2 |2 −|αE1 |2 ) ) sin(φs (t))



 e μ0 Ip (t)  2 ( 4d + ((a + b) − L)2 − 4d2 + ((a + b) + L)2 ) + tan  4π    

 (a + b) − 4d2 + ((a + b) + L)2 − L −(a + b) − L + 4d2 + ((a + b) + L)2 (a + b)    +  Log +   2 (a + b) + 4d2 + (−(a + b) + L)2 − L −(a + b) + L + 4d2 + (−(a + b) + L)2 )    

 (a + b) + 4d2 + (+(a + b) + L)2 − L +(a + b) − L + 4d2 + (−(a + b) + L)2 L   = + × Log    2 −(a + b) + 4d2 + (−(a + b) + L)2 + L −(a + b) − L − 4d2 + (+(a + b) + L)2 √ |α (0)|2 −|αE1 (0)|4 1 sin(t E2 −E − (φE2 (0) − φE1 (0))) tan(φ(t)) + 2 √E1  (1−2αE1 (0)2 )2 √ = |αE1 (0)|2 −|αE1 (0)|4 1 1 − tan(φ(t))2 √ sin(t E2 −E ) − (φE2 (0) − φE1 (0))  2 2

tan(

21 (t))

=

(1−2αE1 (0) )

(118)

Analytical View on Non-invasive Measurement of Moving Charge

41

Finally we obtain the prescription for current function that generates any desired dynamics of φ(t) that is both continuous or discontinuous so we have   4π  2 ( 4d + ((a + b) − L)2 − 4d2 + ((a + b) + L)2 ) + e μ0    

 (a + b) − 4d2 + ((a + b) + L)2 − L −(a + b) − L + 4d2 + ((a + b) + L)2 (a + b)   + Log  +   2 (a + b) + 4d2 + (−(a + b) + L)2 − L −(a + b) + L + 4d2 + (−(a + b) + L)2 )       −1

(a + b) + 4d2 + (+(a + b) + L)2 − L +(a + b) − L + 4d2 + (−(a + b) + L)2 L    × + × Log    2 −(a + b) + 4d2 + (−(a + b) + L)2 + L −(a + b) − L − 4d2 + (+(a + b) + L)2 √ 2 4

tan(φ(t)) + 2 |α 1 √E1 (0)| −|αE12 (0)| sin(t E2 −E − (φE2 (0) − φE1 (0)))  (1−2αE1 (0) )2 √ ×ArcT an |α (0)|2 −|αE1 (0)|4 1 1 − tan(φ(t))2 √E1 sin(t E2 −E ) − (φE2 (0) − φE1 (0))  2 2 Ip (t) =

(1−2αE1 (0) )

(119) On another hand knowing dynamics of external electric current imprinting phase drop on qubit we can determine qubit effective phase dynamics 

T an − 2



 |αE1 (0)|2 − |αE1 (0)|4 E2 − E1 e μ0 Ip (t)   sin(t − (φE2 (0) − φE1 (0))) + tan ( 4d2 + ((a + b) − L)2 − 4d2 + ((a + b) + L)2 ) +   4π (1 − 2αE1 (0)2 )2    

−(a + b) − L + 4d2 + ((a + b) + L)2 (a + b) − 4d2 + ((a + b) + L)2 − L (a + b)  + + Log    2 (a + b) + 4d2 + (−(a + b) + L)2 − L −(a + b) + L + 4d2 + (−(a + b) + L)2 )

+

 1+ 2





L

    +(a + b) − L + 4d2 + (−(a + b) + L)2 (a + b) + 4d2 + (+(a + b) + L)2 − L  ×   −(a + b) + 4d2 + (−(a + b) + L)2 + L −(a + b) − L − 4d2 + (+(a + b) + L)2

× Log 

2



  |αE1 (0)|2 − |αE1 (0)|4 E2 − E1 e μ0 Ip (t)   sin(t − (φE2 (0) − φE1 (0))) × tan ( 4d2 + ((a + b) − L)2 − 4d2 + ((a + b) + L)2 ) +   4π (1 − 2αE1 (0)2 )2    

−(a + b) − L + 4d2 + ((a + b) + L)2 (a + b) − 4d2 + ((a + b) + L)2 − L (a + b)  + + Log    2 2 2 2 2 (a + b) + 4d + (−(a + b) + L) − L −(a + b) + L + 4d + (−(a + b) + L) )

+

L 2



    −1 +(a + b) − L + 4d2 + (−(a + b) + L)2 (a + b) + 4d2 + (+(a + b) + L)2 − L  = φ(t)   2 2 2 2 −(a + b) + 4d + (−(a + b) + L) + L −(a + b) − L − 4d + (+(a + b) + L)

× Log 

(120)

Evolution of dynamics of phase imprint on position based qubit can be both continuous or discontinuous as current flow can be continuous as in case of circuits or discontinues [movement of charge in free space]. Illustration of phase difference evolution with time between 2 and 1 node in symmetric position dependent |ψ2 (t) qubit with no external magnetic field is depicted in Fig. 5. Dependence of |ψ 1 (t)| o time in symmetric position dependent qubit with no external magnetic field is depicted in Fig. 6. Evolution of phase difference in symmetric qubit under influence of time dependent magnetic field is given in Fig. 7. Linear effective phase imprint on time is reported as well as sinusoidal and step-like function. There is infinite class of phase imprinting functions that can be implemented.

42

K. Pomorski

Fig. 5. Dependence of ratio external magnetic field.

|ψ2 (t)| |ψ1 (t)|

in symmetric position dependent qubit with no

Fig. 6. Dependence of phase difference with time between nodes 2 and 1 in symmetric position dependent qubit with no external magnetic field.

5

Analogy Between Single-Electron Charged Based in 2 Coupled Quantum Dots Qubit and 1/2 Spin Qubit

In case of system of two coupled quantum dots one has characteristic oscillations between presence of electron in left or right quantum dots. The condition for such oscillations is the occupancy of at least two eigenenergies and in very real way those oscillations are oscillations between two logical states as in Wannier representation of qubit. As it was indicated by [8] such oscillations are bit similar to Rabi spin oscillations when the spin particle is placed to external time-dependent field so it can flip from top to down of from down to top spin orientation. One shall exploit the fact that Wannier position based qubit is described by Hamiltonian

Analytical View on Non-invasive Measurement of Moving Charge

43

Fig. 7. Dependence of phase difference with time between nodes 2 and 1 in symmetric position dependent qubit with time dependent external magnetic field for the case of linear and sinusoidal dependence of external magnetic field affecting qubit on time.

44

K. Pomorski

 ˆt = H

Ep1 (t) ts12 (t) t∗s21 (t) Ep2 (t)



 Ep1 (t) |ts12 (t)|e−iα(t) . |ts12 (t)|e+iα(t) Ep2 (t)

 =

(121)

If Ep1 = Ep2 and after adding the additional constant potential to previous Hamiltonian one can obtain effective tight-binding Hamiltonian in the form     Ep1 (t) +Ep (t) ts12 (t) |ts12 (t)|e−iα(t) ˆ . (122) = Ht = t∗s21 (t) −Ep (t) |ts12 (t)|e+iα(t) Ep2 (t) Last Hamiltonian is very same in mathematical structure to the Hamiltonian of spin 1/2 in external magnetic field that is given as Bz (t) Bx (t) − iBy (t) = −Bz (t) Bx (t) + iBy (t) ⎞ By (t) −iArcSin( √ )  Bx (t)2 +By (t)2 Bx (t)2 + By (t)2 e ˆ t = μ0 H

⎛ = μ0 Bx

(t)2

+ By

Bz (t) +By (t) iArcSin( √ ) Bx (t)2 +By (t)2 2 (t) e

Bz (t)

(123)

We can say that single-electron charge qubit is always resonating between two logical states in case of static external electric field that is equivalent to Rabi oscillations of spin in time-dependent magnetic field, where the oscilla√ (B 2 +B 2 +B 2 )

x y z . Let us consider two spin tion frequency is equal to ωr = μ0  interaction with two spins given by the state of spin qubit A specified as |ψ(t) >A = αA (t)| ↑>A +βA (t)| ↓>A and spin qubit B described by state |ψ(t) >B = αB (t)| ↑>B +βB (t)| ↓>B . The state of 2 interacting qubits is given as |ψ >= γ1 (t)| ↑>A | ↑>B +γ2 (t)| ↑>A | ↓>B +γ3 (t)| ↓>A | ↑>B +γ4 (t)| ↓>A | ↓>B with |γ1 (t)|2 + |γ2 (t)|2 + |γ3 (t)|2 + |γ4 (t)|2 = 1. We have the Hamiltonian for 2 interacting spins as

Analytical View on Non-invasive Measurement of Moving Charge

45

ˆB + H ˆ A−B = μ0 (BzA σ ˆ =H ˆA + H ˆz + BxA σ ˆx + ByA σ ˆy ) × IˆB + IˆA × μ0 (BzB σ ˆz + BxB σ ˆx + ByB σ ˆy ) + H  + J(k, l)A−B σ ˆk × σ ˆl = k,l

= μ0 (BzA σ ˆz + BxA σ ˆx + ByA σ ˆy ) × IˆB + IˆA × μ0 (BzB σ ˆz + BxB σ ˆx + ByB σ ˆy ) + ˆx × σ ˆx + J(x, y)A−B σ ˆx × σ ˆy + J(x, z)A−B σ ˆx × σ ˆz + +J(x, x)A−B σ +J(y, x)A−B σ ˆy × σ ˆx + J(y, y)A−B σ ˆy × σ ˆy + J(y, z)A−B σ ˆy × σ ˆz + ˆz × σ ˆx + J(z, y)A−B σ ˆz × σ ˆy + J(z, z)A−B σ ˆz × σ ˆz = +J(z, x)A−B σ     BxA (t) − iByA (t) BxB (t) − iByB (t) BzA (t) BzB (t) 10 10 = μ0 × + × μ0 + BxA (t) + iByA (t) BxB (t) + iByB (t) −BzA (t) 01 01 −BzB (t) ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ 0001 0 0 0 −i 0 0 +1 0 ⎜0 0 1 0⎟ ⎜ ⎟ ⎜ ⎟ ⎟ + J(x, y)A−B ⎜ 0 0 +i 0 ⎟ + J(x, z)A−B ⎜ 0 0 0 −1⎟ + +J(x, x)A−B ⎜ ⎝0 1 0 0⎠ ⎝ 0 −i 0 0 ⎠ ⎝+1 0 0 0 ⎠ 1000 +i 0 0 0 0 −1 0 0 ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ 0 0 0 −i 0 0 0 −1 0 0 −i 0 ⎜ 0 0 −i 0 ⎟ ⎜ 0 0 +1 0 ⎟ ⎜ 0 0 0 +i⎟ ⎟ ⎜ ⎟ ⎜ ⎟ +J(y, x)A−B ⎜ ⎝ 0 +i 0 0 ⎠ + J(y, y)A−B ⎝ 0 +1 0 0 ⎠ + J(y, z)A−B ⎝+i 0 0 0 ⎠ + +i 0 0 0 −1 0 0 0 0 −i 0 0 ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ 0 +1 0 0 0 −i 0 0 +1 0 0 0 ⎜+1 0 0 0 ⎟ ⎜+i 0 0 0 ⎟ ⎜ 0 −1 0 0 ⎟ ⎟ ⎜ ⎟ ⎜ ⎟ +J(z, x)A−B ⎜ ⎝ 0 0 0 −1⎠ + J(z, y)A−B ⎝ 0 0 0 +i⎠ + J(z, z)A−B ⎝ 0 0 −1 0 ⎠ = 0 0 −1 0 0 0 −i 0 0 0 0 +1

⎞ μ0 BzA (t) 0 μ0 BxA (t) − iμ0 ByA (t) 0 ⎜ 0 μ0 BzA (t) 0 μ0 BxA (t) − iμ0 ByA (t)⎟ ⎟+ =⎜ ⎠ ⎝μ0 BxA (t) + iμ0 ByA (t) 0 −μ0 BzA (t) 0 0 −μ0 BzA (t) 0 μ0 BxA (t) + iμ0 ByA (t) ⎞ ⎛ μ0 BxB (t) − iμ0 ByB (t) 0 0 μ0 BzB (t) ⎟ ⎜μ0 BxB (t) + iμ0 ByB (t) −μ B (t) 0 0 0 zB ⎟+ +⎜ ⎝ μ0 BxB (t) − iμ0 ByB (t)⎠ 0 0 μ0 BzB (t) 0 0 μ0 BxB (t) + iμ0 ByB (t) −μ0 BzB (t) ⎞ ⎛ J(x, x)A−B − iJ(x, y)A−B 0 0 +J(x, z)A−B ⎟ ⎜ 0 0 J(x, x)A−B + iJ(x, y)A−B −J(x, z)A−B ⎟+ +⎜ ⎠ ⎝ +J(x, z)A−B J(x, x)A−B − iJ(x, y)A−B 0 0 −J(x, z)A−B 0 0 J(x, x)A−B + iJ(x, y)A−B ⎞ ⎛ 0 0 −iJ(y, z)A−B −J(y, y)A−B − iJ(y, x)A−B ⎟ ⎜ 0 0 J(y, y) − iJ(y, x) +iJ(y, z) A−B A−B A−B ⎟+ +⎜ ⎠ ⎝ +iJ(y, z)A−B J(y, y)A−B + iJ(y, x)A−B 0 0 −J(y, y)A−B + iJ(y, x)A−B −iJ(y, z)A−B 0 0 ⎛ ⎞ J(z, x)A−B − iJ(z, y)A−B 0 0 J(z, z)A−B ⎜J(z, x)A−B + iJ(z, y)A−B ⎟ −J(z, z) 0 0 A−B ⎟= +⎜ ⎝ 0 0 −J(z, z)A−B −J(z, x)A−B + iJ(z, y)A−B ⎠ 0 0 −J(z, x)A−B − iJ(z, y)A−B J(z, z)A−B   J J −i −iJ +J J + −J − i(J +J ) ⎛

⎛

(z,z) (z,x) J(z,y) (y,z) (x,z) J(z,x) + iJ(z,y) −J(z,z) J(x,x) + J(y,y) + i(−J(y,x) + J(x,y) ) −J(z,z) +iJ(y,z) + J(x,z) J(x,x) + J(y,y) + i(J(y,x) − J(x,y) ) J(x,x) − J(y,y) + i(J(y,x) + J(x,y) ) −iJ(y,z) − J(x,z) −J(z,x) − iJ(z,y)

(x,x)

(y,y) (y,x) +iJ(y,z) − J(x,z) −J(z,x) + iJ(z,y) J(z,z)

(x,y)

+ ⎞

μ0 (BzA (t) + BzB (t)) μ0 (BxB (t) − iByB (t)) μ0 (BxA (t) + BxB (t)) − iμ0 ByA (t) μ0 (BxB (t) − iByB (t)) ⎜ μ0 (BxB (t) + iByB (t)) μ0 (BzA (t) − BzB (t)) μ0 (BxB (t) + iByB (t)) μ0 (BxA (t) − BxB (t)) − iμ0 ByA (t)⎟ μ0 (−BzA (t) + BzB (t)) μ0 (BxB (t) − iByB (t)) μ0 BxA (t) + iμ0 ByA (t) μ0 (BxB (t) − iByB (t)) μ0 (BxB (t) + iByB (t)) μ0 BxA (t) + iμ0 ByA (t) μ0 (BxB (t) + iByB (t)) μ0 (BzA (t) + BzB (t))

(124) The Hamiltonian obtained for two coupling half spin particle is analogical to the Hamiltonian for two electrostatically interacting Wannier qubits (each having two coupled quantum dots) that is expressed as EpA1 + EpB1 + Ec(1A−1B) t1A→1A,2B→1B t2A→1A t2A→1A,2B→1B ⎜ ⎟ t1A→1A,1B→2B EpA1 + EpB2 + Ec(1A−2B) t2A→1A,1B→2B t2A→1A ˆ =⎜ ⎟ H t1A→2A t1A→2A,2B→1B EpA2 + EpB1 + Ec(2A−1B) t2B→1B t1A→2A,1B→2B t1A→2A t1B→2B EpA2 + EpB2 + Ec(2A−2B)

(125)

46

K. Pomorski

It shall be underlined that process when two particles A and B are moving in synchronous way is less probable than probability when two particles are moving in opposite directions since first process is generating higher magnetic field in outside environment. Those processes can be properly expressed by tight-binding model validation procedure that is sketched in work [15].

6

Particle in Rotating Magnetic Field and Relation to the Charge Qubit

We consider particle with spin 1/2 in magnetic field having constant length B (Fig. 8), but changing its orientation as in accordance to n = (nx , ny , nz ) = (cos(Θ)sin(φ), sin(Θ)sin(φ), cos(φ)) We have given Hamiltonian in the form as   cos(φ) sin(φ)e−iΘ H = μ0 B sin(φ)e+iΘ −cos(φ) We consider the case of constant φ =  H = μ0 B

cos(φ) sin(φ)e+ivt

(126)

(127)

π 4

and time dependent Θ(t) = vt.    √1 √1 e−ivt sin(φ)e−ivt 2 2 = μ0 B √1 +ivt −cos(φ) e − √12 2   μ0 B 1 e−ivt = √ +ivt −1 2 e

(128)

Fig. 8. Parametrization of particle with spin orientation. We identify unit vector n = (nx , ny , nz ) = (cos(Θ)sin(φ), sin(Θ)sin(φ), cos(φ)).

We set B =

√ 2 μ0

and we obtain 

1 e+ivt

e−ivt −1



a(t) b(t)



d = i dt

  a(t) . b(t)

(129)

Analytical View on Non-invasive Measurement of Moving Charge

47

Finally we have d a(t), dt d a(t)e+ivt − b(t) = i b(t), dt

a(t) + b(t)e−ivt = i

(130)

d d From fist equation we have b(t) = eivt [i dt a(t) − a(t)] = eivt [i dt − 1]a(t) and we obtain second equation in the form

a(t)e+ivt = (i

d2 d d d + 1)b(t) = eivt ((1 − v)(i a(t) − a(t)) + (− 2 a(t) − i a(t))), dt dt dt dt d d2 d a(t) = ((1 − v)(i a(t) − a(t)) + (− 2 a(t) − i a(t))), dt dt dt d d d2 d a(t) = (i a(t) − a(t) − vi a(t) + va(t) − 2 a(t) − i a(t)), dt dt dt dt d d d2 d a(t)(2 − v) + vi a(t) = (i a(t) − 2 a(t) − i a(t)), dt dt dt dt d d2 a(t)(2 − v(t)) + ( a(t))(iv(t)) + 2 a(t) = 0, dt dt

(131)

We obtain the analytic solution for v(t) = v = constans given as We obtain the analytic solution for v(t) = v = constans given as √ √ a(t) = c1 et (−iv+(−2+v)) + c2 e−t (−iv+(−2+v)) = c1 − c2 c1 + c2 Cosh(t (−iv + (−2 + v))) + Sinh(t (−iv + (−2 + v)))) =( 2 2

(132)

We have oscillation of spinor component with complex frequency. We extract real and complex value frequency features, so we obtain a(t) = c1 et (−iv+(−2+v)) + c2 e−t (−iv+(−2+v)) =   c1 − c2 c1 + c2 Cosh(t (−iv + (−2 + v))) + Sinh(t (−iv + (−2 + v)))) = =( 2 2  √ v v √ √ t cos( −1 ))+isin( −1 )) (v 2 +(−2+v)2 ) 2 Arcsin ( 2 Arcsin ( (v 2 +(−2+v)2 ) (v 2 +(−2+v)2 ) = c1 e + √ −1 v v 2 +(−2+v)2 ) √ √ −t cos( −1 Arc ( ))+isin( Arc ( )) (v sin sin 2 2 (v 2 +(−2+v)2 ) (v 2 +(−2+v)2 ) +c2 e

(133)

d − 1]a(t), so we have Now we are using fact that b(t) = eivt [i dt

= c1 [i ( iv + ( 2 + v))

d b(t) = eivt [i − 1]a(t) = dt √ √ d eivt [i − 1][c1 et (−iv+(−2+v)) + c2 e−t (−iv+(−2+v)) ] = dt √ √ 1]et (−iv+(−2+v)) + c2 [ i ( iv + ( 2 + v)) 1]e−t (−iv+(−2+v)) ]

(134)

√ √ a(t) = c1 et (−iv+(−2+v)) + c2 e−t (−iv+(−2+v)) = c1 − c2 Sinh(t ( iv + ( 2 + v)))) ( iv + ( 2 + v))) + 2

(135)

We thus have given

=(

c1 + c2 Cosh(t 2

48

K. Pomorski

We have oscillation of spinor component with complex frequency. We extract real and complex value frequency features, so we obtain √ √ 2π 2π 2π ) = c1 e v (−iv+(−2+v)) + c2 e− v (−iv+(−2+v)) = a(t = 0) = c1 + c2 , a(t = v  √ = c1 e

2π v

√ cos( −1 2 Arcsin (

v v √ ))+isin( −1 )) 2 Arcsin ( (v 2 +(−2+v)2 ) (v 2 +(−2+v)2 )

(v 2 +(−2+v)2 )

√ − 2π cos( −1 v 2 Arcsin (

v v √ ))+isin( −1 )) 2 Arcsin ( (v 2 +(−2+v)2 ) (v 2 +(−2+v)2 )

+c2 e

√

+

(v 2 +(−2+v)2 )

(136)

b(t = 0) = c1 [i (−iv + (−2 + v)) − 1] + c2 [−i (−iv + (−2 + v)) − 1]], √  2π 2π ) = c1 [i (−iv + (−2 + v)) − 1]e v (−iv+(−2+v)) + b(t = v √ 1]e−

+c2 [ i ( iv + ( 2 + v))

2π v

(−iv+(−2+v))

],

(137)

Let us now obtain c1 and c2 coefficients from initial state. We have Hamiltonian matrix given as √ √ 1 1 , E1 = − 2, E2 = 2 2 1 1

ˆ = H

(138)

and corresponding eigenvectors |E1 >=

2 √ , 4−2 2 1 √ √ 4−2 2

2  1+ √ , 2(2+ 2) 1 √ 2(2+ 2)

√1−

.|E2 >=

(139)

Spin projection is ±1 and at all steps of motion we have effective angle with z direction as π4 , so it is effective value √12 . and therefore the following condition is fulfilled 1 1− 2 2 1 2 √ = (1 − pE2 )[(  √ ) − ( √ ) ]+ 2 4−2 2 4−2 2 √ 1+ 2 1 1 2 2 +pE2 [( √ ) −( √ ) ] = √2 2 2+ 2 2 2+ 2

(140)

and it gives √ 2 √ 4−2 2

√1− pE2 = 

√ 2 √ 4−2 2

√1−

−√

1

−√ 1 √ 4−2 2  2

√ 4−2 2

−

2

−

√ 1+ 2 √ 2(2+ 2)

√1 2

−

2 1

√ 2(2+ 2)

(141)

This leads to conditions c1 + c2 =

1+ 2 1− 2 √ (1 − pE2 )  √ + pE2  √ , 4−2 2 2 2+ 2

  (c1 − c2 )i (−iv + (−2 + v)) − (c1 + c2 ) = (1 − pE2 )

1 √ √ + pE2 4−2 2

1 2 2+

√

2

(142)

and it implies (c1 − c2 ) =

1 i ( iv + ( 2 + v))

(1 − pE2 )

2− 4

2 √ √ + pE2 2 2

2+ 2 2+

2 √

2

(143)

Analytical View on Non-invasive Measurement of Moving Charge

49

and after adding to c1 + c2 =

we obtain

 1− 2 √ (1 − pE2 ) √ + pE2 4−2 2

1+ 2 2+

2 √

, 2

(144)

2− 2 1 1  (1 − pE2 )  + (1 − 2) √ + 2 i (−iv + (−2 + v)) 4−2 2

√ √ √ 1 2+ 2 1  + (1 + 2) pE2  + √ , 2 i (−iv + (−2 + v)) 2 2+ 2

√ √  2− 2 1 1  c2 = − − (1 − 2) (1 − pE2 )  √ + 2 i (−iv + (−2 + v)) 4−2 2

√ √ √ 2+ 2 1 1 − (1 + 2) pE2 − √ , 2 i (−iv + (−2 + v)) 2 2+ 2 c1 =

(145)

We can compute the square modulus of Berry phase |γ|2 2 2π 2π 2π 2π 2 −|γ|2 = Ln(e−|γ| ) = −Ln[< ψ(x, t = )|ψ(x, t = ) >] = −Ln[|a(t = )| + |b(t = )|] = T T T  T  √ 2π cos( −1 Arc v v ))+isin( −1 Arcsin ( √ )) (v 2 +(−2+v)2 ) sin ( √ v 2 2 (v 2 +(−2+v)2 ) (v 2 +(−2+v)2 ) + = −Ln[|c1 e  √ v v − 2π ))+isin( −1 )) cos( −1 Arcsin ( √ Arcsin ( √ (v 2 +(−2+v)2 ) v 2 2 2 2 2 2 2 (v +(−2+v) ) (v +(−2+v) ) | + +c2 e  √ −1 2π cos( −1 Arc v v 2 +(−2+v)2 ) √ √ ( ))+isin( Arc ( )) (v  sin sin v 2 2 (v 2 +(−2+v)2 ) (v 2 +(−2+v)2 ) + +|c1 [i (−iv + (−2 + v)) − 1]e  √ −2π −1 −1 v v cos( 2 Arcsin ( √ ))+isin( 2 Arcsin ( √ )) (v 2 +(−2+v)2 )  v 2 2 2 2 (v +(−2+v) ) (v +(−2+v) ) |2 ] −c2 [i (−iv + (−2 + v)) + 1]e 2π

2π

2π

2π

= −Ln[(c∗1 e v (wr −iw1 ) + c∗2 e− v (wr −iw1 ) )(c1 e v (wr +iw1 ) + c2 e− v (wr +iw1 ) ) +   2π 2π +iv + (−2 + v)) − 1]e v (wr −iw1 ) − c∗2 [−i (+iv + (−2 + v)) + 1]e− v (wr −iw1 ) ) ×   2π (w +iw ) 2π (w +iw ) − r 1 1 (c1 [i −iv + (−2 + v)) − 1]e v + c2 [i (−iv + (−2 + v)) + 1]e v r )] =

+(c∗1 [−i

= −Ln[(|c1 |2 e

2π 2(w ) r v

2π

2π

2π

+ |c2 |2 e−2 v (wr ) + c∗2 c1 e v (+2iw1 ) + c∗1 c2 e v (−2iw1 ) ) +   2π +(|c1 |2 [−i +iv + (−2 + v)) − 1][+i −iv + (−2 + v)) − 1]e v 2(wr ) +   2π +c∗2 c1 [−i (+iv + (−2 + v)) + 1][+i (−iv + (−2 + v)) + 1]e−2 v (wr ) + +c∗1 c2 [+i (−iv + (−2 + v)) + 1][−i

(+iv + (−2 + v)) + 1]e+2

2π (w ) r v

)]

(146)

7

Spin 1/2 Dynamics of Single Electron in Magnetic Field in Angle Coordinates and Its Mapping to Single-Electron Charge Qubit Dynamics in System of 2 Coupled Quantum Dots

We consider charge qubit in the system of two coupled quantum dots in the form Ep (t) ts (t)∗ ts (t) −Ep (t) −iΘ

Ep cos(f (t)) e e+iΘ ts sin(f (t))

sin(φ)|ts |sin(f (t)) Ep cos(f (t))

a(f (t)) b(f (t))

= i

df d dt df

a(t) b(t) a(f (t)) b(f (t))

= i

d dt

= i

a(t) b(t) d dt

a(t) b(t)

=

(147)

50

K. Pomorski

We can introduce the quantity tan(f (t)) = Arctan ( |tEsp(t)| (t) ).

|ts (t)| Ep (t)

and hence we identify the

angle Θ(t) = f (t) = The total value of virtual magnetic field can be  associated to charge qubit with absence of magnetic field as Bvirtual (t) = 1 (Ep (t))2 + |ts (t)|2 ). The already introduced angle f (t) = Θ(t) can be idenμ0 tified as analogical to the angle of spin 1/2 particle that is subjected to the constant intensity magnetic field that is rotating with time-dependent velocity d dt f (t). If this rotation speed is quite small the spin will follow the magnetic field d direction. We will consider the constant speed of rotation v = dt f (t) = constant and in such case we can formulate the equations of motion in angle coordinate Θ = vt so( dΘ v = dt) that are of the form      vd a(Θ) cos(φ) e−iΘ sin(φ) a(Θ) μB +iΘ . (148) = i b(Θ) sin(φ) −cos(φ) e dΘ b(Θ) We introduce 1 = 

v Bcos(φ)

and we obtain     d a(Θ) a(Θ) 1 e−iΘ tan(φ) . = i 1 b(Θ) −1 e+iΘ tan(φ) dΘ b(Θ)

(149)

We have the set of equation d a dΘ d e+iΘ tan(φ)a − b = i1 b. dΘ a + e−iΘ tan(φ)b = i1

(151)

− a(Θ)] − 1]a(Θ) and we obtain second equation in the form

From fist equation we have b(Θ) d 1 iΘ tan(φ) e [i1 dΘ

(150)

a(t)tan(φ)e+iΘ = (1 i =

=

d 1 iΘ tan(φ) e [i1 dΘ a(Θ)

=

 1  d d d + 1)b(Θ) = (1 i + 1) eiΘ [i1 − 1]a(Θ) = dΘ dΘ tan(φ) dΘ

d d d2 eiΘ ((1 − 1 )(1 i a(Θ) − a(Θ)) + (−21 2 a(Θ) − 1 i a(Θ))), tan(φ) dΘ dt dΘ

d d2 d a(Θ) − i1 a(Θ) − a(Θ)) + (−21 a(Θ))), dΘ dΘ2 dΘ 2 d d d 2 2 d a(Θ) − i1 a(Θ) − a(Θ) − 1 i a(Θ) + 1 a(Θ) − 1 a(Θ)), (tan(φ)) a(Θ) = (i1 dΘ dΘ dΘ2 dΘ 2 d d d 2 2 d a(Θ) − i1 a(Θ) = (i1 a(Θ) − 1 a(Θ)), a(Θ)((1 + (tan(φ)) ) − 1 ) + 1 i dΘ dΘ dΘ2 dΘ 2 d 2 2 d a(Θ) = 0, a(Θ))(i1 ) + 1 a(Θ)((1 + (tan(φ)) ) − 1 ) + ( dΘ dΘ2 ((1 + (tan(φ))2 ) − 1 ) d d2 i a(Θ) +( a(Θ) = 0, a(Θ))( ) + 21 dΘ 1 dΘ2 (tan(φ))2 a(Θ) = ((1 − 1 )(i1

a(Θ)

 1 a(Θ) = c1 e 2 a(Θ) = c1 e

1 2

((1 + (tan(φ))2 )B 2 (cos(φ))2 − vBcos(φ)) d iBcos(φ) d2 +( a(Θ) = 0, a(Θ))( )+ v 2 2 dΘ v dΘ2 2 d 2 2 2 d a(Θ) = 0, a(Θ)(B − vBcos(φ)) + ( a(Θ))(iBcos(φ)v) + v  dΘ dΘ2 d d2 α1 a(Θ) + iα2 a(Θ) = 0, a(Θ) + α3 dΘ dΘ2  √   √  2 2 −4α1 α3 −α2 −4α1 α3 −α2 1 1  − Θ + Θ iα 2 α3 2 α3 − 2Θ e 2α3 , + c2 e a(Θ) = c1 e



 B )2 − Bcos(φ) )+( Bcos(φ) )2 Θ −i 4(( v v v

  Bcos(φ) 1 v )2 − Bcos(φ) )+1 ( )Θ −i 4(( cos(φ) v



  B )2 − Bcos(φ) )+( Bcos(φ) )2 Θ  iBcos(φ) +1 +i 4(( v − Θ 2 v v 2v

+ c2 e

   Bcos(φ) 1 v +1 )2 − Bcos(φ) )+1 ( )Θ +i 4(( cos(φ) 2 v

+ c2 e

e

−

e

,

iBcos(φ) Θ v

(152)

Analytical View on Non-invasive Measurement of Moving Charge

51

1 v 1 The condition 4(( cos(φ) )2 − Bcos(φ) ) + 1 < 0 is equivalent to cos(φ) + 1 v 2 ( 4 cos(φ)) < B and we obtain imaginary frequencies. The coefficient a(Θ) determines b(Θ) as in accordance to formula b(Θ) =

 v d 1 eiΘ [i − 1] c1 e tan(φ) Bcos(φ) dΘ

=



1 d eiΘ [i1 − 1]a(Θ) = tan(φ) dΘ

 1 −i 4(( B )2 − Bcos(φ) )+( Bcos(φ) )2 Θ 2 v v v

+

 B )2 − Bcos(φ) )+( Bcos(φ) )2 Θ  iBcos(φ) +1 +i 4(( v − Θ 2 v v 2v 

+c2 e

=

1

c1 e 2

+

=

e

    Bcos(φ) B Bcos(φ) 1 Bcos(φ) 2 eiΘ eiΘ v 4(( )2 − )+( ) + − × Bsin(φ) 2 v v v 2v tan(φ)

 

eiΘ v Bsin(φ)

 −

1 2



B )2 − −i 4(( v

B 2 Bcos(φ) Bcos(φ) 2 ) − )+( ) v v v

 

4((

Bcos(φ) Bcos(φ) 2 )+( ) v v

 +

B )2 − +1 +i 4(( v 2

c2 e

Bcos(φ) 2v



Θ

+

eiΘ × tan(φ)

−

Bcos(φ) Bcos(φ) 2 )+( ) v v

e

Θ

×

iBcos(φ) − Θ 2v

The Berry phase acquired by spin or charged based qubit after rotation by angle Θ from 0 to 2Π (so we set Θ = 2Π) is



= −Ln

γ = −Ln[b∗ (Θ)b(Θ) + a∗ (Θ)a(Θ)] =     Bcos(φ) B Bcos(φ) 1 Bcos(φ) 2 eiΘ e v 4(( )2 − )+( ) + − × Bsin(φ) 2 v v v 2v tan(φ)

  iΘ

1

× c1 e 2

+

eiΘ v Bsin(φ)

 −

1 2

 4((

B )2 − −i 4(( v

B 2 Bcos(φ) Bcos(φ) 2 ) − )+( ) v v v

  +1 +i 2

1

+

eiΘ v Bsin(φ)

 −

1 2

 4((

B 2 Bcos(φ) Bcos(φ) 2 ) − )+( ) v v v

 

 +

B )2 − +1 +i 4(( v 2



+ c1 e

 1 −i 4(( B )2 − Bcos(φ) )+( Bcos(φ) )2 Θ 2 v v v

c1 e

8

+

−

eiΘ × tan(φ) ∗

Bcos(φ) Bcos(φ) 2 )+( ) v v

B )2 − −i 4(( v

× c2 e

Bcos(φ) 2v



Θ

Θ

×

    eiΘ v B Bcos(φ) Bcos(φ) 1 Bcos(φ) 2 eiΘ 4(( )2 − )+( ) + − × Bsin(φ) 2 v v v 2v tan(φ)

  × c1 e 2



+

B )2 − 4(( v

× c2 e





Bcos(φ) Bcos(φ) 2 )+( ) v v



Bcos(φ) Bcos(φ) 2 )+( ) v v

Bcos(φ) 2v



Θ

+

−

eiΘ × tan(φ)

Bcos(φ) Bcos(φ) 2 )+( ) v v

Θ

∗  B )2 − Bcos(φ) )+( Bcos(φ) )2 Θ +1 +i 4(( v 2 v v

+



+ c2 e

 1 −i 4(( B )2 − Bcos(φ) )+( Bcos(φ) )2 Θ 2 v v v

×

 B )2 − Bcos(φ) )+( Bcos(φ) )2 Θ +1 +i 4(( v 2 v v 

+ c2 e

Conclusions

I have shown the effect of transport of charged particle as proton in accelerator beam acting of single electron in position based qubit placed in the proximity

52

K. Pomorski

of the accelerator. One can conclude that the beam of moving charged particles brings the change of occupancy of energetic levels in position electrostatic qubit and is inducing phase imprint across qubit. In most general case one can expect that two level system represented by qubit will change its initial occupancy (as for example from 2) into N energy levels with phase imprint made on each eigenenergy level. However under assumption that the perturbing factor expressed by moving charge in accelerator beam is weak the conducted considerations are valid. Conducted considerations are also valid for the case of floating potential that is potential polarizing the qubit state. Therefore presented picture can be considered as phenomenological model of noise for electrostatic qubit that provides the description for qualitative and quantitative assessment of noise on two kinds of decoherence times commonly known as T1 and T2 . The presented results were presented at the seminar [5]. In such way one can account for very complicated electromagnetic environment in which position electrostatic semiconductor qubit is placed. In particular one can trace the decay of quantum information encoded in the qubit. One also expects that in the situation of 2 electrostatically qubits the passage of external charged particles is changing the quantum entanglement between qubits and anticorrelation function characterizing two interacting qubits. Part of this work was presented in [3] and in [6, 14, 15]. The results can be extended quite straightforward to the more complicated structures by the mathematical framework given in [6, 7] and [8–15]. Particular attention shall be paid to the structures depicted in Fig. 3. It is worth noticing that similar scheme of detection of moving charge can be applied to interface between Josephson junction and Wannier qubits as indicated by [15].

Fig. 9. Zoo of basic position dependent qubit topologies that could be used for beam diagnostics.

Analytical View on Non-invasive Measurement of Moving Charge

53

Furthermore all conducted reasoning with interaction of external charge on Wannier qubit can be described in the framework of Schroedinger formalism what will give higher accuracy of predictions of observed phenomena [16]. The most interesting of future research direction is analysis of robustness of quantum geometric phase against passage of external charged particle or voltage drift in polarizing electrodes. The general scheme of such approach can be recognized from work [17]. In such case all structures from Fig. 4, Fig. 5, Fig. 6, Fig. 7 and Fig. 8 when placed with the interaction with quantum electrodynamical cavity in the framework of holonomic quantum computation shall show certain

Fig. 10. Additional basic position dependent qubit topologies that could be used for beam diagnostics in quantum sensing and in quantum processing of obtained information. Central structure is matrix of semiconductor quantum dots implementing quantum neural network.

54

K. Pomorski

robustness against voltage drifts in polarizing electrodes as well as against presence of external moving charged particle. The exact answer on possible resistance against external noise or moving charge can rely on the usage of Lindblad formalism describing decoherence effects as well on the usage of Jammes-Cumming tight binding model describing interaction of Wannier qubit with quantum electromagnetic cavity firstly described in [6] and developed further as in [7]. The decoherence of Wannier qubit can be obtained by moving from real values of Ep1 and Ep2 into values Ep1q,r + iEp1q,im and Ep2q,r + iEp2q,im as given by [16]. Furthermore one shall move from complex values of t12 and t12 = t∗21 into complex values of t12q and t12q = t∗21q and such that |t12q | = |t12 | as well as |Ep1 | = |Ep1q,r + iEp1q,im | and |Ep2 | = |Ep2q,r + iEp2q,im |. The prize to be paid for the occurrence of holonomic quantum computation is that every two charge qubits participating in quantum electrostatic swap gate [4] shall be subjected to the external quantum cavity as given in [7,17]. In such case one shall consider the three energy level configuration of the resonant situation for the single-qubit gates, with two microwave fields resonantly coupled to the three levels (see Fig. 9 and 10). Acknowledgment. I would like to thank to M.Sc. Marcin Piontek (University of Lodz) and to PhD Marcin Kowalik (Rzeszow University of Technology) for their assistance in schemes graphical upgrade.

References 1. Fujisawa, T., Hayashi, T., Hirayama, Y.: Electron counting of single-electron tunneling current. Appl. Phys. Lett. 84, 2343 (2004). https://doi.org/10.1063/ 1.1691491 2. Bednorz, A., Franke, K., Belzig, W.: Noninvasiveness and time symmetry of weak measurements. New J. Phys. 15 (2013). https://iopscience.iop.org/article/ 10.1088/1367-2630/15/2/023043 3. Pomorski, K., Giounanlis, P., Blokhina, E., Leipold, D., Staszewski, R.: Analytic view on coupled single electron lines. Semicond. Sci. Technol. 34(12) (2019). https://iopscience.iop.org/article/10.1088/1361-6641/ab4f40/meta 4. Pomorski, K., Giounanlis, P., Blokhina, E., Leipold, D., Peczkowski, P., Staszewski, R.B.: From two types of electrostatic position-dependent semiconductor qubits to quantum universal gates and hybrid semiconductor-superconducting quantum computer. In: Spie, vol. 11054, Superconductivity and Particle Accelerators (2019). https://doi.org/10.1117/12.2525217 5. Pomorski, K.: Detection of moving charge by position dependent qubits. In: CERN 2020-Dublin UCD Webinar, 23 January 2020. https://indico.cern. ch/event/876476/contributions/3693568/. https://vidyoreplay.cern.ch/replay/ showRecordingExternal.html?key=6lP8xJZuDcPToTD 6. Pomorski, K., Staszewski, R.: Analytical solutions for N-electron interacting system confined in graph of coupled electrostatic semiconductor and superconducting quantum dots in tight-binding model with focus on quantum information processing (2019). ArxiV:1907.02094. https://arxiv.org/abs/1907.03180

Analytical View on Non-invasive Measurement of Moving Charge

55

7. Pomorski, K., Staszewski, R.B.: Towards quantum internet and non-local communication in position based qubits. In: AIP Conference Proceedings, vol. 2241, p. 020030 (2020). https://arxiv.org/abs/1911.02094 8. Bashir, I., et al.: Mixed-signal control core for a fully integrated semiconductor quantum computer system-on-chip. In: ESSCIRC (2019). https://ieeexplore.ieee. org/abstract/document/8902885/ 9. Giounanlis, P., Blokhina, E., Pomorski, K., Leipold, D., Staszewski, R.: Modeling of semiconductor electrostatic qubits realized through coupled quantum dots. IEEE Open Access (2019). https://doi.org/10.1109/ACCESS.2019.2909489. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8681511 10. Pomorski, K.: Quantum information processing, quantum communication and quantum Artificial Intelligence in semiconductor quantum computer. In: FQIS 2020-International Workshop for Young Researchers on the Future of Quantum Science and Technology, 3–6 February 2020. https://fqst2020.com/) + Pomorski, K., Staszewski, R., et al.: Programmable quantum matter in semiconductor. In: Seminar in Dublin Institute of Advanced Studies, 24 September 2019. https://www.dias.ie/2019/09/24/tuesday-1st-october-programmablequantum-matter-in-semiconductor-electronics/ + Pomorski, K., Giounanlist, P., Blokhina, E., Staszewski, R.B.: ISIF 2019 Conference on Integrated Functionalities, Dublin, 12 August 2019 11. Pomorski, K., Giounanlist, P., Blokhina, E., Staszewski, R.B.: Modeling quantum universal gates in semiconductor CMOS. In: Scalable Hardware Platforms for Quantum Computing Proceedings of Scalable Hardware Platforms for Quantum Computing, Physikzentrum, Bad Honnef, Germany, 17 January 2019 12. Pomorski, K.: Analytic view on N body interaction in electrostatic quantum gates and decoherence effects in tight-binding model. Int. J. Quant. Inf., 2141001 (2021). https://doi.org/10.1142/S021974992141001X 13. Pomorski, K.: Analytical view on tunnable electrostatic quantum swap gate in tight-binding model (2020). ArXiv: 2001.02513. https://arxiv.org/pdf/2001.02513. pdf 14. Pomorski, K., Staszewski, R.: Description of interface between semiconductor electrostatic qubit and Josephson junction in tight binding model. Acta Physica Polonica A 138(5) (2020) 15. Pomorski, K.: Fundamental description of Wannier qubits in semiconductor (2021). Arxiv:2103.05869 16. Pomorski, K.: Equivalence between classical epidemic model and non-dissipative and dissipative quantum tight-binding model (2020). ArXiv:2012.09923 17. Li, S., Chen, T., Xue, Z.: Fast holonomic quantum computation on superconducting circuits with optimal control. Adv. Quantum Technol. 3, 20000012020 (2020)

Multi-population Genetic Algorithm with the Actor Model Approach to Determine Optimal Braking Torques of the Articulated Vehicle Kornel Warwas and Szymon Tengler(B) University of Bielsko-Biala, Willowa 2, Bielsko-Biala, Poland [email protected]

Abstract. The paper presents an application of message driven optimization to control braking torques on wheels of an articulated vehicle to restore stability during an untripped rollover maneuver. The numerical model of the articulated vehicle and dynamic optimization have been used to calculate appropriate braking torques for each wheel in order to restore stability. The optimization problem requires the equations of motion to be integrated at each optimization step and it is a time-consuming task. Therefore, parallel computing with the use of the Actor Model system has been proposed. The Actor Model has been implemented in the Multi-Population Genetic Algorithm. This paper presents a formulation of Multi-Population Genetic Algorithm with the actor system and results obtained from dynamic optimization. Keywords: Multibody system · Articulated vehicle · Optimization · Actor model · Multi-population Genetic Algorithm · Parallel computing

1 Introduction The problem of dynamic optimization of complex physical systems (e.g. multibody system or articulated vehicle) is a time-consuming process. There are many works focusing on reducing calculation time [1–3]. To improve the efficiency of computing optimization problem parallel and distributed systems are frequently used [1, 2, 4, 5]. There exist algorithms to allow splitting calculations on separate threads, processes, or cluster’s nodes in a arbitrary way. The approach used in order to formulate a system model and implementation of the optimization process is not without significance. Currently, development allows to use the monolithic architecture or the micro-service approach. The advantage of the micro-service is the business logic can be separately processed, and those components can communicate with each other [6]. According to Reactive Manifesto [7, 8], the ability to implement a software message driven in a responsive, resilient and elastic way is very important, and it is the subject of many papers [9–11]. Also, these modifications allow a significant reduction in the calculation time by applying parallel processing © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 56–74, 2022. https://doi.org/10.1007/978-3-030-80119-9_2

Multi-population Genetic Algorithm with the Actor Model Approach

57

and introducing a high scalability of the system. One manner of implementing the system and computations is called the Actor Model approach (AM) [4, 12, 13]. The AM for concurrent and parallel programming is becoming popularity because of its abstraction in development process and languages. Also, it is not insignificant ability to make efficient use of multicore and multiprocessor computers. The AM introduces the actor as a smallest computational unit. Actors can cooperate with each other through a transfer of messages that are asynchronous. The actor can handle and analyze a message according to the assumed logic, create more actors, transfer the message to other actor or expect the new message. There are many existing AM implementations such as Akka, CAF, Orleans, Theron, Pykka [8, 14–16]. Some of the AM systems can be run in a remote environment such as computational grid or cloud systems. There are many studies focusing on the use of parallel computing in relation to evolutionary methods. The most common example is the use of calculations in parallel genetic algorithms, especially the one based on the approach with multiple populations exchanging information with each other. The paper [17] presents the use of a genetic algorithm with multiple populations to find the shortest path routing problem. The authors introduce a vivid concept of “immigrants scheme” to indicate the process of information exchange between populations, and to explain why the algorithm was named iMPGA. In the paper [18], a multi-population genetic algorithm has been used to solve the nonlinear subway vehicle model. According to the authors, the proposed strategy reveals a new method of subway maintenance. Further continuation of issues related to the subway can be found in the paper [19], in which the multipopulation genetic algorithm was used to solve the problem of reducing the energy consumption of subway systems and time tripping. The Dual Population Genetic Algorithm (DPGA) method was implemented [20], using multi-core calculations (MPDPGA). Also, one can observe there attempts at the use of parallel computing in the agent approach, the ancestor of the Actor Model systems. In [21], a multi-population agent co-genetic algorithm with a chain-like agent structure (MPAGA) has been suggested. The algorithm adopted a multi-population parallel searching mode, closed and cycled chain-like agent structure, dynamic neighborhood competition, and orthogonal crossover strategy to realize parallel optimization. The main aim of this study has increased optimization precision and has decreased optimization time. In order to verify the algorithm benchmark test functions have been used. In this paper, the multi-population genetic algorithm with the AM approach was presented, taking into account the process of optimization cornering maneuvering of the vehicle. The optimization goal is to maintain the stability of the articulated vehicle and to prevent its rollover during maneuvering. Rollover of vehicles usually causes serious accidents. The low lateral stability of articulated vehicles leads to this type of accidents [22]. Anti-lock Braking Systems (ABS), Electronic Braking Systems (EBS) and Electronic Stability Programs (ESP), increase vehicle stability and they can steer braking torques for each wheel, which can provide a driver with greater control [3, 23–25]. Many automotive companies work on solutions that increase the vehicle stability. The appropriate example Wabco implements Electronic Stability Control (ESC) and Roll Stability Control (RSC) systems. These systems support the driver in case of road surface slippery, during cornering as well as overtaking and avoiding obstacles (ECS), while the RSC can additionally control, among other things, braking torques independently of the driver’s activity in order to improve the stability of the vehicle. There are also many

58

K. Warwas and S. Tengler

other systems that increase driving safety and comfort, for example: active front steering (AFS), active braking (AB) and active suspension (AS). They were developed based on real vehicle parameters and tested in road conditions [26]. In order to minimize the risk of accidents and costs, virtual models can be implemented to use in the first phase of experiments [3, 25]. The method presented in this paper is based on a virtual model of the articulated vehicle and allows controlling torques of breaking to prevent losing stability. Values of these toques are applied to a wheel and resulting from solving an optimization problem. A formulation of Multi-Population Genetic Algorithm with the actor model and results obtained from dynamic optimization are presented. The results presented in the paper allow to obtain conclusions how actor model can improve optimization process in details.

2 Mathematical Formulation of the Model of Investigation The model of the vehicle is composed as a system of rigid bodies such as: tractor, a semi-trailer and a fifth wheel. The tractor has six generalized coordinates that describing its motion (Fig. 1).

Fig. 1. Model of the tractor

It can be presented in the following form: T  (1) q˜ T = x(1) y(1) z (1) ψ (1) θ (1) ϕ (1)

(1)

where: x(1) , y(1) , z (1) - translational coordinates of the tractor, ψ (1) , θ (1) , ϕ (1) - rotational coordinates of the tractor. Wheels relate to the tractor and each wheel has one rotational coordinate θ (1,i) (i = 1, . . . , 4). The vector of the generalized coordinates can be described as: T  (1) q˜ TW = θ (1,1) θ (1,2) θ (1,3) θ (1,4)

(2)

Additionally, front wheels have steering angles δ (1,1) and δ (1,2) : T  (1) q˜ TS = δ (1,1) δ (1,2)

(3)

Multi-population Genetic Algorithm with the Actor Model Approach

59

Fig. 2. Model of the fifth wheel

The motion of the fifth wheel has been modelled using one generalized coordinate (pitch angle θ (2) ) in relation to the tractor (Fig. 2). The vector of generalized coordinates of the fifth wheel has following form:   (4) q˜ F(2) = θ (2) The model of a semi-trailer (Fig. 3) is modeled as a body with one generalized coordinate, yaw angle ψ (3) .

Fig. 3. Model of the semi-trailer

Its vector of generalized coordinates is written by:   q˜ S(3) = ψ (3)

(5)

The semi-trailer contains six wheels (Fig. 3) and each has one generalized coordinate θ (3,i) (i = 1, . . . , 6). The following vector describes their motion in relation to semitrailer: T  (3) (6) q˜ SW = θ (3,1) θ (3,2) θ (3,3) θ (3,4) θ (3,5) θ (3,6) The Lagrange equations formalism of the second kind and homogenous transformations for formulating equations of vehicle motion was used [3, 26]: ∂E ∂V d ∂E − + = Qj dt ∂ q˙ j ∂qj ∂qj where: E=

N  i=1

E (i) - total kinetic energy of the articulated vehicle,

(7)

60

K. Warwas and S. Tengler

E (i) - kinetic energy of the i-th body, n  V (i) - total potential energy of the articulated vehicle, V = i=1

V (i) - potential energy of the i-th body,   q˙ = ddtq = q˙ j j=1, ..., n - vector of generalized velocities of the system,   Q = Qj j=1, ..., n - vector of generalized forces of the system, N - number of bodies of the articulated vehicle, n - number of generalized coordinates of the articulated vehicle, j = 1, . . . , n. After transformation dynamic equations of motion of the i-th subsystem of the articulated vehicle can be presented in the general form [3]: A(i) q¨ (i) = f (i)

(8)

where: A(i) - mass matrix, f (i) - vector of external, Coriolis, centrifugal and gravity forces. The equations of motion of the particular subsystems of the articulated vehicle can be presented as follows: – subsystem 1 - the tractor ⎡ (1) ⎡ ⎡ ⎤ ⎤ ⎤ (1) (1) AT ,T A(1) fT(1) q˜ T T ,TS AT ,TW ⎢ ⎢ (1) ⎥ ⎢ (3) ⎥ ⎥ (1) (1) (1) (1) = ⎣ q˜ TS A(1) = ⎣ A(1) ⎦, f = ⎣ fTS ⎦ TS,T ATS,TS ATS,TW ⎦, q (1) (3) (1) (1) (1) fTW q˜ TW ATW ,T ATW ,TS ATW ,TW – subsystem 2 – the fifth wheel       (2) (2) (1) (2) A A ˜ q f (2) T ,T T ,F , q(2) = T A(2) = = T(2) , (2) (2) (2) , f AF,T AF,F fF q˜ F – subsystem 3 - the semi-trailer ⎡ (3) (3) (3) AT ,T AT ,F AT ,S ⎢ A(3) A(3) A(3) ⎢ F,F F,S A = ⎢ F,T (3) (3) ⎣ A(3) A A S,T S,F S,S (3) (3) A(3) A A SW ,T SW ,F SW ,S

⎤ ⎡ (1) ⎤ ⎡ (3) ⎤ (3) AT ,SW q˜ T fT (3) ⎥ ⎢ ⎢ (2) ⎥ (3) ⎥ AF,SW ⎥ ⎢ q˜ F ⎥ ⎢ fF ⎥ (3) (3) , q , f = = ⎥ ⎥ ⎥. ⎢ ⎢ (3) ⎣ q˜ S(3) ⎦ ⎣ fS(3) ⎦ AS,SW ⎦ (3) (3) fSW q˜ SW A(3) SW ,SW

The dynamic equations of motion of the articulated vehicle have to take into account constraint equations, which ensue from a kinematic input, describing the course of the steering angle of the front wheels. As a result, the dynamic equations of motion of the articulated vehicle take the following form [3, 26, 27]: Aq¨ + q r = f Tq q¨ = w

(9)

Multi-population Genetic Algorithm with the Actor Model Approach

61

where: ⎡

(1)

(2)

(3)

(2)

(3)

AT ,T + AT ,T + AT ,T AT ,F + AT ,F (2) (3) (2) (3) ⎢ AF,T + AF,T AF,F + AF,F ⎢ ⎢ (3) (3) ⎢ AS,T AS,F A=⎢ (3) ⎢ ATS,T 0 ⎢ ⎢ ⎣ 0 A(3) TW ,T (3) (3) ASW ,F ASW ,T

(3)

(1)

(1)

AT ,S AT ,TS AT ,TW (3) AF,S 0 0 (3) AS,S 0 0 (1) (1) 0 ATS,TS ATS,TW (1) 0 A(1) TW ,TS ATW ,TW (3) ASW ,S 0 0

⎤ (3) AT ,TW (3) AF,SW ⎥ ⎥ ⎥ (3) AS,SW ⎥ ⎥ ⎥ 0 ⎥ ⎥ ⎦ 0 (3) ASW ,SW

- mass matrix ⎡

⎤ 00 ⎢0 0⎥ ⎢ ⎥ ⎢0 0⎥ ⎢ ⎥ ⎢ ⎥ q = ⎢ 1 0 ⎥ - constraintsmatrix, ⎢ ⎥ ⎢0 1⎥ ⎢ ⎥ ⎣0 0⎦ 00 ⎤ ⎡ (1) (2) (3) f T + fT + fT ⎥ ⎢ f (2) + f (3) ⎥ ⎢ F F ⎥ ⎢ (3) fS ⎥ ⎢ f =⎢ ⎥ - vector of Coriolis and external forces, (1) ⎥ ⎢ f TS ⎥ ⎢ (1) ⎦ ⎣ f TW (1)

fSW ⎡

⎤

⎢ r1 ⎥ r = ⎣ ⎦ - vector of constraint reactions, r1  (1,1)  δ¨ w = ¨(1,2) - vector of constraint equations, δ r1 , r2 - torques on the suspension connected to the wheels. Reaction forces of the road on the wheels have been defined according to the DugoffUffelman model described in [3, 26]. The details of the procedure which leads to the formation of the Eq. (9) with a description of elements in matrix A and the vector f are presented in [3]. The articulated vehicle model has been verified by comparing the results obtained from the model with those obtained from the road tests. During the experiment the J-turn maneuver, described in the international standard ISO 7401:2003, has been investigated. A response of the articulated vehicle to a sudden step steering angle has been analyzed (Fig. 4). The tractor of Mercedes Actros 1840 LS Megaspace F015 together with a semitrailer of Kogel SN 24 P 100/1.060 was tested. Correvit sensors and gyroscopes have

62

K. Warwas and S. Tengler

Fig. 4. Course of the front wheels steering angle used during road tests and simulations.

a)

b)

Fig. 5. The Yaw velocity comparison of a) the tractor, b) the semi-trailer resulting from: 1) road tests, 2) presented model.

been used in order to measure yaw velocity of the vehicle and a steering angle of the front wheels. The comparison of the tractor and semi-trailer yaw velocity courses resulting from the model and experiment was shown in Fig. 5. In the presented computer simulation for a calculating error the following integral formula was used: t     e  (i)  te   ψ˙  dt − ψ˙ (i)  dt  M C    0 0 (10) ε(i) =   · 100%   te  (i)    ˙ ψM  dt   0

where: (i) ψ˙ M - measurement yaw velocity, ψ˙ C(i) - simulation yaw velocity. The calculated errors according to (10) have been presented in Table 1.

Multi-population Genetic Algorithm with the Actor Model Approach

63

Table 1. Errors results i Yaw velocity ψ˙ (i)

Error (i) %

1 Tractor

1.2

2 Semi-trailer

4.2

An acceptable correspondence of the results has been achieved. A relative integral error calculated for both courses does not exceed 5%. Therefore, the assumptions of the suspension stiffness do not have a significant impact on the accuracy of the results presented in the paper.

3 Definition of Optimization Problem As was mentioned the vehicles rollover is dangerous road maneuver. This situation takes place usually during an unexpected lane-change maneuver [21] while the vehicle travels through a corner. Such a maneuver will be performed when the vehicle trajectory collides with an obstacle. Then to avoid collisions, the vehicle trajectory is performed to another traffic lane, as shown in Fig. 6.

Fig. 6. Lane changing maneuver during cornering

Stability of the articulated vehicle can be restored by an appropriate control of braking torques applied to each wheel of the vehicle. Considering a vector of braking torque discrete values M(i) of the i-th wheel, one notices a continuous function M (i) (t) which will be obtained, using spline functions of the 3rd order. The vector of the decisive variables contains discrete values of the braking torques of wheels and can be presented in the following form: T    T T T = Mj j=1,...m (11) M = M(1) . . . M(i) . . . M(nw )

64

K. Warwas and S. Tengler

Where: m - number  of decisive variables, M(i) = Mj j=1,...,m(i) ,

m(i) - number of discrete values of the braking torque acting on the i-th wheel, nw - number of wheels. The general form of a dynamic optimization problem can be written as follows [3, 28]:

(X1 , . . . , Xi . . . , Xn ) → min

(12)

where:

- objective function, Xi - objective function dependency, n - number of objective function dependencies. The objective function solving (12) requires integration of equations of dynamic (9). In the problem presented above, the braking torques and the front wheels steering angle must satisfy following requirements: – the vehicle cannot lose a stability during the maneuver, – loss of the vehicle velocity must be as small as possible. These requirements are considered in the objective function and also in the optimization constraints. The stability conditions can be assured by minimizing the functional: ⎞ ⎛ te  te  2 1 ˆ (13) ϕ (1) dt + C2 (v0 − ve )2 dt ⎠ → min ˙ = ⎝C1

(M, q, q) te 0

0

where: C1 , C2 - empirical coefficients, te - time of simulation, v0 - initial  velocity, 2  2 x˙ (1) (te ) + y˙ (1) (te ) - velocity of the vehicle corresponding to te . ve = In the considered optimization problem, the inequality constraints can be presented as follows:   (14) Mmin − M1 . . . Mmin − Mi . . . Mmin − Mm ≤ 0 

 M1 − Mmax . . . Mi − Mmax . . . Mm − Mmax ≤ 0

(15)

where Mmin , Mmax - acceptable minimal and maximal braking torque values. These conditions can be included through additional inequality constraints [28, 29], which can be presented in the following form: gi (M) ≤ 0 where:

(16)

Multi-population Genetic Algorithm with the Actor Model Approach

65

i = 1, . . . , ng , ng - number of the inequality constraints. The constraints can be included in the objective function by the external penalty function [29]:  0  for gi (M) ≤ 0 (17) γi (M) = C1,i exp C2,i gi (M) for gi (M) > 0 where C1,i , C2,i are empirical weights. As a result, the objective function has a following form: ⎞ ⎛ ng te  te   2 1 ˙ = ⎝C1 γi (M) → min (18) ϕ (1) dt + C2 (v0 − ve )2 dt ⎠ +

(M, q, q) te 0

0

i=1

The classical genetic algorithm (CGA) and multi-population genetic algorithm with the AM approach and the immigrants scheme method (iMPAMGA) was used to solve the formulated minimization problem [28, 30, 31]. Further in the article, the genetic algorithm with the AM approach method will be presented in detail.

4 Genetic Algorithm with Actor Model Approach Actors have state and behavior, and they can exchanging messages (Fig. 7).

Fig. 7. Message exchanging in the AM system.

The actor can delegate some part of its tasks until they become small enough to be processed in one piece by one actor to build a fault-tolerant and hierarchical system. An actor can be identified as a separate thread on one host machine, or it can be a process on a cluster node of distributed systems. In this paper, the CAF framework [15, 16] was used to implement the MPGA with the AM approach. CAF allows transparent connecting of actors executing on different cluster node and exchanging messaging using a network. It can use multiple computing elements e.g.: multi-core CPUs, GPGPUs or hardware that is embedded. In the approach presented, a population is created, which will split into several small populations. These small populations continue the search by either exploiting or exploring the solution space. The

66

K. Warwas and S. Tengler

root actor creates coworkers (populations) which implement task for creating generations (Fig. 8) in a parallel way. Those actors produce many children (crossover operators) which also perform further operations in asynchronous way. The child number depends on the individual number in the population. After crossover operation each actor produces a new actor which is responsible for the operation of mutation. Individuals resulting from the mutation are transferred to the actors which calculate the objective function value. This operation usually takes a long time due to integrate a dynamic equation of the whole system. The primary advantage of this solution is that the processing objective function can be performed in a parallel way. The processing data are collected by the next actor which waits for results of the objective functions calculation. Then next population is created according to a rule of natural selection. At the end of each generation, to enhance the diversity and ensure transfer of genetic materials, a small number of the best individuals (immigrants) is sent to other populations. This algorithm is processed until the defined stop criteria is satisfied.

Fig. 8. Actors’ hierarchy in MPAMGA.

In Fig. 9 are presented messages which are sent during one iteration of algorithm in question.

Multi-population Genetic Algorithm with the Actor Model Approach

67

Fig. 9. Message flow between the actors in iMPAMGA.

The real-number representation of genes in chromosomes and the following genetic operators were used [30, 32, 33]: – natural selection, – arithmetical one-point crossover, – non-uniform mutation. The two topologies of migration individuals have been tested: ring migration topology (Fig. 10a) and unrestricted migration topology (Fig. 10b).

Fig. 10. Migration topologies: a) ring, b) unrestricted.

It can be concluded that in the present case parallel processing is performed for the first time at the level of a population of individuals, and again when the genetic operators for each subject in the population are used. Additionally, the AM framework offers a lot of features such as: multithreading calculations, queuing messages, synchronization

68

K. Warwas and S. Tengler

mechanisms and location transparency. The latter is in this case very important because it allows the division of a population into multiple computational units (cluster). This is a configuration rather than programming and, thus, the change of the target environment can be made at any time; once implemented and if necessary, the program can be run in a variety of computing environments.

5 Numerical Simulations The optimal braking torques have been calculated using the Classical Genetic Algorithm (CGA), modification using the Multi-Population AM approach with the immigrants scheme (iMPAMGA). The paper has examined the lane change maneuver during the articulated vehicle cornering. The rollover of the vehicle occurs with no suitable braking torques are applied. The maneuver of lane change starts at t = 3 [s] of the simulation and at the same time braking torques are applied. The torques act during whole simulation (te = 6 [s]). The initial vehicle velocity was v0 = 45 km/h. The physical parameters of the analyzed vehicle were taken from [3]. The braking torques was determined individually for each tractor wheel (wheels 1–4) of the vehicle and the same values of the braking torques at any time were applied on the semi-trailer wheels (group of wheels 5 and 9) Fig. 11.

Fig. 11. Numbering of the articulated vehicle wheels.

Implementation of the mathematical model and optimization algorithms were made in the proprietary program in C++ language. Calculations were performed on the computer Intel Core i7 CPU 2,80 GHz and 16 GB RAM with the OS X system (macOS Sierra 10.12.1 and Apple LLVM version 8.0.0). For integrating motion equations, the BulrishStoer-Deuflhard [23] method with an adaptive step was used. In the problem considered, it has been assumed that the crossover probability was 0.7 while the mutation probability was 0.2. The values obtained for various numbers of individuals in the population using the classical genetic algorithm and the Multi-Population Genetic Algorithm with the AM approach were presented in Fig. 12, 13 and 14. For each variant of calculation assuming a certain number of individuals in a population, 10 calculations (cycle) have been made, in which the appropriate braking torques wheels of the vehicle, starting from a random point, have been selected. One cycle of calculation ranges from of a few minutes to over 6 h for CGA. After calculations, the average values of the objective function and the optimization time have been calculated.

Multi-population Genetic Algorithm with the Actor Model Approach

a)

69

b)

Fig. 12. Values of the objective function a) and optimization calculation time b) obtained for a various number of individuals for: 1) CGA, 2) 2 populations iMPAMGA with the ring topology, 3) 2 populations iMPAMGA with the unrestricted topology.

a)

b)

Fig. 13. Values of the objective function a) and optimization calculation time b) obtained for a various number of individuals for: 1) CGA, 2) 3 populations iMPAMGA with the ring topology, 3) 3 populations iMPAMGA with the unrestricted topology.

In almost all cases, especially for a population containing a larger number of individuals, the average value of the objective function is lower in calculations using the multi-population than the classic algorithm. The differences in the average value of the objective function obtained using the considered topology is insignificant and it cannot be determined unequivocally which of them is better in this particular case. Regardless of the topology, however, the results shown in Fig. 12, 13 and 14(b) prove the enormous saving of computing optimization time when using the Actor Model approach with respect to the classical method. The best results obtained in each multi-population have been presented in Fig. 15, 16 and 17. The presented optimal braking torques acting on the wheels w1 , . . . , w4 are shown in Fig. 15–17a. Other figures (b–d) represent courses of the tractor trajectory, its roll angle, and velocity. The figures also present the results of simulations for lane following (cornering) maneuver 1) and those with the lane change maneuver before optimization 2) and after optimization 3).

70

K. Warwas and S. Tengler

a)

b)

Fig. 14. Values of the objective function a) and optimization calculation time b) obtained for a various number of individuals for: 1) CGA, 2) 4 populations iMPAMGA with the ring topology, 3) 4 populations iMPAMGA with the unrestricted topology.

a)

b)

c)

d)

Fig. 15. Courses for 2 populations iMPAMGA of: a) optimal braking torques, b) the tractor trajectory, c) the tractor roll angle, d) the tractor total velocity.

Large differences in the courses of braking torques can be noticed. However, in each case their acting causes the restoration of the stability of the articulated vehicle. The smallest possible decrease in the velocity of the tractor has been identified for the calculations with 2 populations iMPAMGA (about 42%).

Multi-population Genetic Algorithm with the Actor Model Approach

a)

b)

c)

d)

71

Fig. 16. Courses for 3 populations iMPAMGA of: a) optimal braking torques, b) the tractor trajectory, c) the tractor roll angle, d) the tractor total velocity.

a)

b)

c)

d)

Fig. 17. Courses for 4 populations iMPAMGA of: a) optimal braking torques, b) the tractor trajectory, c) the tractor roll angle, d) the tractor total velocity.

72

K. Warwas and S. Tengler

6 Conclusions The article presents the problem of selecting appropriate braking torques acting on individual wheels of the vehicle during cornering. To determine these torques, dynamic optimization with parallel genetic algorithms and AM approach was used. The obtained results indicate the correctness of the formulation of the optimization task and the applied computational methodology. After applying the results from the optimization, vehicle stability was restored, and the roll angle is significantly less than before the optimization. For medium or high complexity models optimization calculations usually take a long time due to the need to integrate the equations of motion into each optimization step. An attempt was made to shorten the computing time by introducing parallel computing with the use of multi-core CPU. The applied approach from AM allows to increase the programming abstraction and division of the optimization process into small subtasks. Additionally, it does not require thorough knowledge of thread synchronization and other elements of concurrent programming. The computation time has been improved by about 70% compared to the computation with the classical (synchronous) methods. It should be noted that the calculations were performed on a moderate personal computer with 8 CPU cores. The authors predict even greater efficiency gains when using a dedicated computing cluster. Although the results concerning only one road maneuver and the vehicle model, the presented algorithm can be treated as universal, and the use does not have to be focused only on the multibody system. Although the idea of AM has been known in the literature for many years, nowadays interest in this approach can be noticed not only in engineering calculations and scientific works, but also in the commercial sector. This is due to the transparent source code, logical division into computing units (actors) and high scalability of the solution. The authors intend to use the AM approach in conjunction with Particle Swarm Optimization in future plans.

References 1. Eberhard, P., Dignath, F., Kübler, L.: Parallel evolutionary optimization of multibody systems with application to railway dynamics. Multibody Sys.Dyn. 9(2), 143–164 (2003) 2. Augustynek, K., Warwas, K., Pola´nski, A.: Application of the genetic algorithms and distributed computing in task of the reduction of vibrations of a satellite. In: 7th Conference Computer Methods and Systems, pp. 237–242 (2009) 3. Warwas, K.: Analysis and control of motion of articulated vehicles with flexible elements. Ph.D. thesis, University of Bielsko-Biała, Bielsko-Biała (2008) 4. Plotnikova, N.P., Fedosin, S.A., Teslya, V.V.: Gravitation search training algorithm for asynchronous distributed multilayer perceptron model. In: Elleithy, K., Sobh, T. (eds.) New Trends in Networking, Computing, E-learning, Systems Sciences, and Engineering, Lecture Notes in Electrical Engineering, vol. 312, pp. 417–423. Springer, Cham (2015). https://doi.org/10. 1007/978-3-319-06764-3_52 5. Umbarkar, A.J., Joshi, M.S., Hong, W.-C.: Multithreaded Parallel Dual Population Genetic Algorithm (MPDPGA) for unconstrained function optimizations on multi-core system. Appl. Math. Comput. 243, 936–949 (2014) 6. Amaral, M., Polo, J., Carrera D., Mohomed, I., Unuvar, M., Steinder, M.: Performance evaluation of microservices architectures using containers. In: 2015 IEEE 14th International Symposium on Network Computing and Applications (NCA), pp. 27–34 (2015)

Multi-population Genetic Algorithm with the Actor Model Approach

73

7. The Reactive Manifesto (2017). http://www.reactivemanifesto.org 8. Goodwin, J.: Learning Akka. Packt Publishing, Birmingham (2015) 9. Madeyski, L., Kawalerowicz, M.: Software engineering needs agile experimentation: a new ´ practice and supporting tool. In: Madeyski, L., Smiałek, M., Hnatkowska, B., Huzar, Z. (eds.) Software Engineering: Challenges and Solutions. AISC, vol. 504, pp. 149–162. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-43606-7_11 10. Ivanovi´c, D., Carro, M.: Transforming service compositions into cloud-friendly actor networks. In: Franch, X., Ghose, A.K., Lewis, G.A., Bhiri, S. (eds.) ICSOC 2014. LNCS, vol. 8831, pp. 291–305. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-453919_20 11. Vaughn, V.: Reactive Messaging Patterns with the Actor Model Applications and Integration in Scala and Akka. Addison-Wesley, Boston (2016) 12. Hewitt, C.: Actor model of computation for scalable robust information systems. Inconsistency Robustness (2015) 13. Lim, Y.H., Tana, J., Abramson, D.: Solving optimization problems in Nimrod/OK using a genetic algorithm. Procedia Comput. Sci. 9, 1647–1656 (2012) 14. Bernstein, P., Bykov, S.: Developing cloud services using the orleans virtual actor model. IEEE Internet Comput. 20(5), 71–75 (2016) 15. Charousset, D., Schmidt, T.C., Hiesgen, R.: CAF - the C++ actor framework for scalable and resource-efficient applications. In: Proceedings of the 5th ACM SIGPLAN Conference on Systems Programming and Applications (SPLASH 2014) Workshop AGERE! (2013) 16. Charousset, D., Hiesgen, R., Schmidt, T.: Revisiting actor programming in C++. Comput. Lang. Syst. Struct. 56, 105–131 (2016) 17. Cheng, H., Yang, S.: Multi-population genetic algorithms with immigrants scheme for dynamic shortest path routing problems in mobile ad hoc networks. In: Di Chio, C., et al. (eds.) EvoApplications 2010. LNCS, vol. 6024, pp. 562–571. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-12239-2_58 18. Zhang, D., Hu, H.: An optimization on subway vehicle maintenance using a multi-population genetic algorithm. In: Sustainable Development of Critical Infrastructure, pp. 316–323 (2014) 19. Huang, Y., Ma, X., Su, S., Tang, T.: Optimization of train operation in multiple interstations with multi-population genetic algorithm. Energies 2015(8), 14311–14329 (2015) 20. Li, Y., Zeng, X.: Multi-population co-genetic algorithm with double chain-like agents structure for parallel global numerical optimization. Appl. Intell. 32, 292–310 (2010) 21. Yedavalli R.K.: Robust stability and control of multi-body ground vehicles with uncertain dynamics and failures. Technical Report, Ohio State University Research Foundation, Ohio (2010) 22. Yao, Z., et al.: Dynamic simulation for the rollover stability performances of articulated vehicles. J. Automob. Eng. 228, 771–783 (2014) 23. Huang, H.H.: Controller design for stability and rollover prevention of multi-body ground vehicles with uncertain dynamics and faults. Ph.D. thesis, Graduate School of The Ohio State University, Ohio (2009) 24. Warwas, K., Augustynek, K.: Dynamic optimization of articulated vehicle motion for control of stability in critical situation. In: IDAACS 2015: 8th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, vol. 1, pp. 232–237 (2015) 25. Lu, S.-B., Li, Y.-N., Choi, S.-B.: Contribution of chassis key subsystems to rollover stability control. Proc. Inst. Mech. Eng. Part D J. Automob. Eng. 226(4), 479–493 (2012) 26. Adamiec-Wójcik, I.: Modelling dynamics of multibody systems using homogenous transformations. Wydawnictwo Akademii Techniczno-Humanistycznej (2003) 27. Bauchau, O.A.: Flexible Multibody Dynamics, Solid Mechanics and Its Applications. Springer, Heidelberg (2011). https://doi.org/10.1007/978-94-007-0335-3

74

K. Warwas and S. Tengler

˙ 28. Chong, E., Zak, S.: An Introduction to Optimization. Wiley, Hoboken (2013) 29. Press, W., Teukolsky, S., Vetterling, W., Flannery, B.: Numerical Recipes 3rd Edition: The Art of Scientific Computing. Cambridge University Press, Cambridge (2007) 30. Affenzeller, M., Wagner, S., Winkler, S., Beham, A.: Genetic Algorithms and Genetic Programming: Modern Concepts and Practical Applications. CRC Press, Boca Raton (2009) 31. Cantú-Paz E.: Efficient and Accurate Parallel Genetic Algorithms. Springer, Boston (2001). https://doi.org/10.1007/978-1-4615-4369-5 32. Pappa, G., Freitas A.: Automating the Design of Data Mining Algorithms, An Evolutionary Computation Approach. Springer, Berlin (2010).https://doi.org/10.1007/978-3-642-02541-9 33. Bandyopadhyay S., Pal S.: Classification and Learning Using Genetic Algorithms. Springer, Berlin (2007).https://doi.org/10.1007/3-540-49607-6

A Novel Three-Way Merge Algorithm for HTML/XML Documents Using a Hidden Markov Model Nikolaos G. Bakaoukas1 and Anastasios G. Bakaoukas2(B) 1 School of Engineering and Sciences, Computer Science and Programming Department, Foundation College, Mitropoleos Street and Mnisikleous Campus, 105 56 Athens, Greece 2 University of Northampton, Faculty of Arts, Science and Technology, Waterside Campus, University Drive, Northampton NN1 5PH, UK [email protected]

Abstract. Ever since the advent of modern World Wide Web (WWW), collaborative editing of Web Pages at a programming level (via HTML/XML) has been identified as a major programming challenge. Yet surprisingly, relatively little effort has been put into the direction of developing sound algorithms and methodologies for meeting this challenge by automated means. In this paper a novel algorithmic approach to merging HTML/XML code documents is presented that is based on the “Three-way Merge” approach using Hidden Markov Models, the “line-ofcode-per-line-of-code” comparison between the documents involved and the “Nested Parenthesis” principle. The algorithm can be easily extended to any level higher than the “Three-way Merge”, with, of course, its computational complexity increasing accordingly. Keywords: Algorithms · Hidden Markov models Document management · HTML/XML diffing

1

· Version control ·

Introduction

At its foundation every HTML/XML document contains code instructions that are hierarchically structured and aligned in blocks so every document section corresponds to one block in the code. It is exactly this strictly hierarchical structure of the code in a HTML/XML document that primarily allows for the programmers to expand such documents to any length, possessing any level of complexity, without losing control over the larger picture. Secondarily, this fundamental characteristic allows for the implementation of algorithms that, by going step-by-step through the hierarchical structure, can realise in an automated way a number of operations on the target document. In this second category fall those algorithms that if provided with three documents, the original and two edited versions of it, will merge the edited versions to one updated according to amendments document that in the grand majority of cases the editors will need to revisit in order c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 75–101, 2022. https://doi.org/10.1007/978-3-030-80119-9_3

76

N. G. Bakaoukas and A. G. Bakaoukas

to take a final decision on the merging actions proposed by the algorithm and finalise the form and content of it. With this in mind, novel approaches to the problem, corresponding to robust algorithms, have been proposed that realise the above mentioned automation but with the focus being at the same time on the parametrisation aspect of the algorithm so that the need for manual adjustments per case by the editors is highly minimised and algorithm deficiencies are highly (or even completely in some case) eliminated at the same time (such as the well-known “Idempotent” problem) [1]. This is particularly important because the most fundamental of the problems associated with the task of merging two (or more) edited versions of an original HTML/XML document, preventing the provision of a good quality result after the merging process has taken place, is at its core primarily related with the algorithmic structure of the solution applied for integrating the edited copies into a single document containing the changes introduced [2]. To the present, the problem of the automated editing of a number of parallel copies originating from a single original HTML/XML code document with the additional requirement to subsequently merge the copies into a single updated document, is largely addressed by two categories of algorithms. The first category includes algorithms that have been created during the pre-WWW era and their scope was originally of an entirely different nature to that of merging structured documents like HTML/XML documents (e.g. Diff3). The fundamental issue with such algorithms is that addressing changes in hierarchical data is very different from addressing changes in flat data. The second category of algorithms includes those that by taking into account the state of the original document during merging fail to appreciate the statistical value of the merging action they propose per line of code and produce at the output a merged document that is susceptible to the well-known “Idempotent” problem in merging two HTML/XML documents. The result in this case being that after the provision of the merged document it is necessary, in the vast majority of cases, for the editors to view the document in a HTML/XML editor so they can extensively process it manually afterwards (e.g. Diff3, XmlDiff, DeltaXML & 3DM). The “Three-way Merge” approach itself, as realised by every algorithm in the field, requires the existence of an “Original Document” O (or “Last Common Version”) and two “Current Versions” T1 & T2 that have been edited to some extent in respect to O. With the “Diff3” algorithm, currently considered as the most popular for carrying out such kind of merging operations, being exposed as not producing reliable results under all circumstances and not fully possessing the “Local Replicas” (distinct and well separated edited regions are merged without conflicts), the “Idempotence” (the results of merging are fully stable except where edits conflict), the “Stability” (similar inputs lead to similar outputs), and the “Near-complete Success” (when the inputs have been changed in similar ways) properties; new approaches and their algorithmic implementation are needed in order to address the problem with more robustness and effectiveness [2].

A Novel Merge Algorithm For HTML/XML Documents

77

Merging and differentiating XML tools, which compare two XML files and display their differences have been proposed, with the most characteristic example of such tools being IBM’s “XmlDiff”. The principal operational sequence behind such algorithms is that after an initial stage at which the automated merging of the edited versions has been completed, the users are then able to view all the proposed by the algorithm merging actions and manually readjust them in accordance to decisions taken per-merging-action. This select-what-remains procedure ensures that the resulted merged document will fulfil all editors’ requirements [3]. The type of tools that, instead, attempt to directly identify differences between HTML/XML documents by working through their inner code structure, are considered to be the alternatives to the “XmlDiff” approach. Tools like “DeltaXML” can minimise considerably in some cases the need for manual adjustment on the merged document thanks to precisely this strategy [4]. Tools that operate strictly by employing the “Three-way Merge” approach in merging HTML/XML documents, have been proved very popular in addressing a variety of merging situations. More specifically, tools like “3DM” handle every merging situation they are called to address within the boundaries of a built-in “Version Control Engine”. The difference here is that the algorithm underlining the procedure does not take advantage of the HTML/XML code Tags in the documents to be merged and judgment on which merging action is suitable only can be based upon the general structure of the documents [5]. This methodology offers the advantage that the algorithm is then able to provide the merged document at the output relatively quicker than algorithms that do not use a “Version Control Engine”. Still, the way conflicts are handled within the boundaries of such a methodology is rather subjective. The result is that when a conflict (or conflicts) occur during comparing the two edited versions of an original HTML/XML document the decision on which merging action is the more appropriate is taken by considering the “Locally Identified Priority Level” only. Under this condition, those operations that can most correctly recognised and categorised are the “Insert” and “Copy” operations, something that in most of the cases is subsequently leading to the provision of non-optimal differential results [6,7]. Finally, algorithms like “Zhang-Shasha”, “XyDiff”, “DaisyDiff”, “Phoenix”, “X-Diff”, “XyDiff” and “Lindholm” complete the picture of what is available at the present, tools-wise, in the area of merging HTML/XML documents. The tools employ the “Three-way Merge” approach in merging two edited versions of an original document and are capable of constructing and analysing two DOM trees to find their minimal edit distance in order to incorporate a broader definition of the edit operations. Are also capable of presenting a solution for a number of different types of merging by application of their algorithm directly on the target documents without the need to use any kind of an explicit edit script. The majority of these algorithms were capable of achieving efficacy in 26 out of 56 unique testing scenarios and failed in providing any precise matching results in 30 out of 56 unique testing scenarios [8], something that most graph-

78

N. G. Bakaoukas and A. G. Bakaoukas

ically illustrate the fact that the area of HTML/XML documents merging by automated means is a very much open area for computational innovations. The algorithm presented in this paper follows the “Three-way Merge” approach with the capability to be expanded at a level where any number of edited versions of the original HTML/XML document can be simultaneously considered for merging. Of course, as is very often the case with algorithms that their computational complexity is fundamentally depended on the length of the input, the more the edited documents that have to be simultaneously considered for merging, the higher becomes the amount of time required for the output to become available. But still, with the implementation of the algorithm we present here this is only depended on the amount of comparisons required per-line-of-code, while the “State Estimation Cost” of the algorithm itself remains constant to O(|S 2 |) regardless of the number of documents at the input, with S the number of Hidden Markov Model states that for the HTML/XML documents merging task is equal to six (6). To the best of the authors’ knowledge this is the first time that any attempt has been made to apply the well-known and very highly esteemed Hidden Markov Models (HMMs) in the area of HTML/XML documents merging. The introduction of a Hidden Markov Model in the algorithmic procedure of the “Three-way Merge” approach brings a number of advantages that will be discussed in more details in the following sections, spanning from the full parametrisation of all the crucial components of the algorithm, to the adaptability of it through statistical means by one level of past input, to the higher accuracy at a suitability level of the proposed merging action at the output. In general, is a straight forward and easy in its implementation algorithm that still manages to achieve efficient results. In many cases the results produced by the algorithm presented here are directly comparable or even far better than those produced by other far more sophisticated and complex algorithms, something that has to be attributed to the powerfulness of the Hidden Markov Model lying at the heart of the algorithm as a statistical tool and the immense space for parametrisation at the disposal of the user, elements that make it particularly attractive as an algorithm especially in cases where accuracy of results is of fundamental importance. The material in this paper is presented in a total of five sections. In Sect. 1, an overview of the operational procedure of some of the best known algorithms in the area is provided for the benefit of the reader, along with introductory concepts in the field of “Three-way Merge” algorithms. Section 2 presents, and analytically discusses, the Hidden Markov Model (HMM) characteristics and operational structure as used in the algorithm proposed in this paper. The modified “Three-way Merge” approach to include at its core the Hidden Markov Model that calculates the estimation for the final editing decision at each iteration is explained in Sect. 3, while Sect. 4 contains a detailed explanation of the implementation requirements and a detailed discussion on the operation of the algorithm, based on experimental results. Finally, Sect. 5 includes the conclusions of the paper.

A Novel Merge Algorithm For HTML/XML Documents

2

79

Hidden Markov Models (HMMs)

Although Hidden Markov Models is a mathematical tool developed during the modern era of mathematics, the fundamental idea behind them it can be traced back to the times of the ancient Greek mathematicians, in at least two occasions: “From the visible to conclude the invisible” [Seven Sages, Solon, Stobaeus Anthology, ] and “For the unknown may be soonest discerned by reference to the [Isocrates, known” “To Demonicus”, 34]. Hidden Markov Models are used for analysing a generative Observable Sequence that is characterised by some underlying Unobservable Sequence and since the 1960s have been widely used in a number of areas in science, such as Speech Recognition, Activity Recognition, Gene Finding, Gesture Tracking, etc. A Hidden Markov Model (HMM) is, at its foundation, a “Finite State Machine” (FSM) with probabilistic edges the state of which is not directly observable. This makes it a statistical model, a chain, in which the system being modelled is assumed to be a Markov Process possessing hidden states, meaning, a type of Random Process made of a set of Random Variables which change over time. The Hidden Process is assumed to satisfy the Markov property, where the current state St at time t depends only on the previous state, St−1 at time t − 1. This is commonly referred to in literature as the Memoryless Property. A Markov Model with these parameters is called the First-order Markov Model, with the nth -order Markov Model depending on the n previous states. Individual states are matched by a probability function that accepts as input some observable evidence and is effectively the transition function that represents the probability of the process being in some specific state in accordance to a history of previous states [14]. Considering a system that can be fully described by a set of N distinct states, S1 , S2 , S3 , . . . , SN the expectation is for it to undergo a change of state, at regular time intervals, according to a set of probabilities attached to each state (Fig. 1).

Fig. 1. A system consisting of a set of two states with their, probability controlled, state transitions represented by transition arrows.

The system in Fig. 1 consists of a total of two states. Each state is controlled by a set of two probabilistic state transitions with the following characteristics: a & b are equal to some probability value in the interval a, b ∈ [0, 1], c = 1 − a and d = 1 − b. These characteristics serve to satisfy completely the fundamental condition:

80

N. G. Bakaoukas and A. G. Bakaoukas

∀St−1



p(St |St−1 ) = 1

(1)

St ∈S

which simply asserts that the probabilities associated with all edges add up to one. This is a simple example of a Markov Chain. In the case when it is impossible to inspect directly the state of a process, this state need to be estimated out of an observation or series of observations. Under this condition the Markov Chain is transformed into a Hidden Markov Model with the addition of a series of observations and the states of the system becoming hidden states (Fig. 2). A Hidden Markov Model can theoretically possess an N number of hidden states and an M number of observation states. Now, even if we know all the individual states in the State Domain and still the Current State of the system can rarely be known for certain, every time (t) we can try and estimate the system state from whatever evidence is available. Hidden Markov Models, in particular, excel in achieving this target because Stochastic Processes in general have been build with the purpose of being able to estimate and track activity based on limited information. As a result, Hidden Markov Models draw from the best of two worlds, Stochastic Processes and Statistical Decision-making.

Fig. 2. The same system as in Fig. 1 with a set of actions that lead to observations and a probabilistic mapping between observations and states.

In Fig. 2, the system draws from a total number of two observations associated with the system’s states through the probabilistic values e, f , g & h. For the estimation of the current state of a system process, that is, if the system is at some state St after an observation or series of observations, we need to calculate the probability of the system being at this state. By doing this for every state St we obtain eventually a Probability Distribution, for the calculation of which there are five popular algorithms in the HMM sector. These are the “Forward

A Novel Merge Algorithm For HTML/XML Documents

81

Algorithm”, the “Backward Algorithm”, the “Viterbi Algorithm”, the “Segmental K-Means Algorithm” and the “Baum-Welch Re-estimation Algorithm”. For what we present in the following sections the Probability Distribution is estimated using the “Forward Algorithm” [15] but any of the other algorithms could have serve equally well the purpose of estimating the Probability Distribution for merging two edited documents in respect to an original one. By selecting the “Forward Algorithm”, however, we are able to compute directly the possible sequence of “Hidden States” given the “Observable States”. The convenience of this will become apparent in what we will be discussing in the following sections.

3

The “Three-Way Merge” Approach as a Hidden Markov Model (HMM)

The Three-way Merging System is the most fundamental, and well proven, of the merging approaches that have been employed in order to algorithmically address the merging documents problem. Has been used by the grand majority of merging algorithms implemented for the purpose, primarily for its simplicity and straightforward approach in dealing with the about-to-be-merged documents without at the same time loosing in reference to the original document, as well as because of its flexibility and easiness in expanding it to include more than two edited versions of the original document. The system requires the provision of three documents, the “Original Document” O (the last common version) and two “Current Versions” T1 & T2 , which have been modified in some way in respect to O (Fig. 3). In this context there are three possible situations every merging algorithm must be capable of dealing with that arise from a direct comparison between O, T1 and T2 , under the condition that this is taking place in a line-by-line fashion: a) No modification has been introduced in respect to O, for the particular line inspected, on neither T1 nor T2 . b) A modification has been introduced in either T1 or T2 , for the particular line inspected, in respect to O. c) A modification has been introduced in both the T1 and T2 , for the particular line inspected, in respect to O. In the first and the second case no action is essentially required from the part of the merging algorithm. Only the passing of the line under consideration directly to the output for inclusion in the merged document in the first, and the same for the line with modifications in the second. This is so because modifications have a higher priority than no-modifications when dealing with documents, simply because we cannot afford loosing completely modifications introduced in an edited version of the original when passed through a merging procedure. As a result both these cases can be considered as easy to deal with algorithmically because the decision from the part of the merging algorithm is only based on one clearly defined decision parameter. The third case is the most complicated

82

N. G. Bakaoukas and A. G. Bakaoukas

Fig. 3. The “Three-Way Merge” approach requires the provision of three documents at the input: (a) the “Original Document” O, (b) the “Current Version” T1 and, (c) the “Current Version” T2 . The two later documents have been modified in some way in respect to O. For completion, the correct output from the merging procedure is shown in (d). Is the expectation of every well structured merging algorithm to be able to provide this result after accepting at the input the three documents.

of the three, in the sense that a decision is required from the part of the merging algorithm based in more than one decision parameters. The actual number of them depends on the nature of the situation faced. A Hidden Markov Model to be capable of reflecting on all the situations that may arise in the edited versions of the original document, needs to have incorporated in its internal structure decision parameters for addressing all three cases. How this is achieved for the algorithm we present in this paper will be the purpose mainly of this and the following section. In respect to what was presented in [1] and the algorithmic approach discussed there, in this algorithm three key features are maintained. These being, the “Operations’ Priorities” and the “Nested Parenthesis”, which through its implemented mechanism introduces the application of the “Tags’ Priorities”, this time not as an autonomous feature. The first of the three features is implemented directly within the Hidden Markov Model and operates through its structure and algorithmic realisation, while the second (leading to the application of the third as well) is maintained as a separate method in the algorithm that acts in parallel with the HMM. According to the definition of Hidden Markov Models, for the purpose of transforming the “Three-way Merge” approach into such a model we need to construct out of it a HMM system λ = (A, B, π). The parameter A signifies the “State Transition Probabilities”, meaning that the value aij (1 ≤ i, j ≤ N ) in the set of probability values represents the probability of the system being in state j at time t+1, if at time t the system was in state i (that is, the probability of the system making a transition from current state i to state j). The parame-

A Novel Merge Algorithm For HTML/XML Documents

83

ter is evaluated against a Transition Probability Function p, with p(St |St−1 ) the actual probability of the system making a transition from state St−1 to state St . The parameter B signifies the “Action-Observation Probabilities”, that is, bi (Oij ) (1 ≤ i ≤ N , 1 ≤ j ≤ M ) in the set of probability values represents the probability of the observed action Oij being one from the actions sequence {O1 , O2 , O3 , . . . , OM } given that the system is at time t in state i. Finally, the parameter π signifies the “State Initialisation Probabilities”, meaning that the value πi (1 ≤ i ≤ N ) in the set of probability values π = {π1 , π2 , π3 , . . . , πN } represents the probability of the system being in state i at the first iteration of the algorithm. The set π of probability values satisfies the fundamental condition: N 

πi = 1

(2)

i=1

Also, we need to define the additional parameters N , M and T . N is the number of unique states in the HMM, M is the number of unique Action-Observations in the HMM, and T is the length of the Action-Observation sequence (the number of Actions observed). For our implementation T is of a constant length of 1, since strictly one Action-Observation is by definition provided by the “Threeway Marge” approach per algorithmic iteration. With all the above we have achieved the mathematical transformation. In order to achieve the algorithmic transformation out of the mathematical one now, we need before anything else to represent HTML/XML code editing as a Stochastic Process, that is, to model the layout of the editing procedure as a set of states (“System States”) and to explicate a set of actions (“Observations”) that will serve as evidence for the states. In this context, we will define the following six operations that can be performed on the original document in order to collectively (if applied in any combination and to whatever extend) lead to the creation of an edited version of the original document: a) Operation Delete: A line of code in the original document is not present any more in the edited version. b) Operation Move: A line of code in the original document has been moved to another code line in the edited version. c) Operation Copy: A line of code in the original document is now present at the original code line, as well as at least one more code line in the edited version. d) Operation Update: A line of code in the original document has been modified in its content in the edited version. e) Operation Insert: A line of code has been inserted at a particular code line in the edited version. f) Operation No Change: A line of code in the original document can be found identical and in the same code line in the edited version. All the above operations can be though of as operating under clear-cut conditions but the Update operation requires some further clarification because of

84

N. G. Bakaoukas and A. G. Bakaoukas

the slightly complicated way in which it is defined in general. Algorithmically, and within the context defined by the “Three-way Merge” approach, we have two options in how to treat the operation: a) to merely detect if there is a string difference between a line in the original document and the same line in the edited version of it; in which case the implementation into the general algorithm of a simple method to detect and report back the corresponding string difference will suffice (an implementation of an algorithm like the “Wagner–Fischer Algorithm”, the “Levenshtein Distance Algorithm” or any other similar to them would be more than adequate for the purpose), or b) to define in a similar fashion to the above the individual sub-operations possible for the update operation. In this second case the sub-operations can be defined as follows, considering for this illustration of the basic concept that the sub-operations are operating on the input string: “In The Summer”: a) Sub-operation Delete is defined as: any variation of the example result: “In ... Summer”. b) Sub-operation Move is defined as: any variation of the example result: “In ... Summer The”. c) Sub-operation Copy is defined as: any variation of the example result: “In The Summer The”. d) Sub-operation Update is defined as: any variation of the example result: “In This Summer”. e) Sub-operation Insert is defined as: any variation of the example result: “In The Early Summer”. With the sub-operation No Change not requiring mention in this case as a sub-operation, because its result would have been identical to that of the main operation. The six operations we defined are now the states of the HMM into which the “Three-way Merge” approach will be converted. The state transitions are established by application of a rule that can be summarised as: “a current state is considered as connected with another state (can lead to a change-ofstate to that state), if the operation represented by the current state can be achieved on a line of code by application of the operation that is represented by the connected state”. Following this simple rule we can construct the following state connections between system states: ⎧ ⎫ U pdate ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ Delete ⎪ ⎬ M ove a) Operation Update: is connected with the states: ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ Copy ⎪ ⎪ ⎩ ⎭

Insert  b) Operation Delete: is connected with the states:⎧ Delete⎫ ⎨ Delete ⎬ c) Operation Move: is connected with the states: Insert ⎩ ⎭ M ove Copy d) Operation Copy: is connected with the states: Insert

A Novel Merge Algorithm For HTML/XML Documents

85

 Insert ⎧ ⎫ ⎪ ⎪ N o Change ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ M ove ⎪ ⎪ ⎨ ⎬ Copy f) Operation No Change: is connected with the states: Delete ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ U pdate ⎪ ⎪ ⎪ ⎪ ⎪ ⎩ ⎭ Insert

e) Operation Insert: is connected with the states:



With the states and their corresponding state transitions defined (Fig. 4), we can now proceed to identify the state transition probabilities for the system.

Fig. 4. The system states and the system state transitions.

The corresponding state transition probabilities are defined as shown in (Fig. 5), with the indications “From”/“To” (column/row) to signify the transition probability value for the system making a transition from the current state to the related connected state in accordance with the mathematical notation we introduced earlier in this section. All the values in this table (and those of the other tables presented later) have been chosen as ideal for the purpose of illustrating the workings, and discussing the individual parameters involved in the algorithmic approach; are not suggested as optimum parameter values for all possible HTML/XML document merging cases. Through the software implementation of the algorithm the by default set values for these parameters must become available for amendment by the user (parametrisation) in order to achieve the desirable line of behaviour by the algorithm and the desired result at the output. Since in our case the next state of the system cannot be inspected directly and/or predicted (because there can never possibly be enough information for attempting even an estimate of what the final decision can be among editors for a particular line of code that has been modified by both) it must necessarily be estimated from some observation or sequence of observations as required by HMM theory. These observations are no other than the actual individual editing operations applied by the editors on the particular line of code under investigation by the algorithm, as evidenced by what is detected these operations to be in the two edited versions of the original document. As a result of having two edited versions of the original document involved in the “Three-way Merge” approach every observation is necessarily made out of a combination of two operations, the one detected on the line of code from the first edited version and

86

N. G. Bakaoukas and A. G. Bakaoukas

Fig. 5. The system state transition probabilities corresponding to the system states and state transitions.

the one detected on the line of code from the second edited version. The list of all possible observations for the system is as follows: /, /, / /, /, /, /, /, / /, /, /, /, /, / /, /, /, /, /, / These twenty-one system observations fully reflect all possible combinations of actions between the two editors and are linked with the system’s states through “Action-Observation Probabilities”, which are defined individually per actionobservation (Fig. 6).

Fig. 6. The action-observation probabilities corresponding to the system’s twenty-one observations.

One of the novelties in the algorithm we are presenting in this paper is the way these action-observation probabilities are calculated. In Fig. 6 we are presenting

A Novel Merge Algorithm For HTML/XML Documents

87

just one of the possible combinations of values for this matrix since the way it is implemented in the code allows for its full parametrisation. The parametrisation is achieved through another matrix which contains the operations’ priority values (lowest priority 1, highest priority 10). The user(s) can manually set the operations’ priorities as parameters in the algorithm so action-observation probabilities can be calculated accordingly. This way, the two matrices remain bounded in the code and the second fully determines the values in the first maintaining at all times the fundamental condition set in Eq. 2 (Fig. 7).

Fig. 7. The set of priority values between operations that determine the actionobservation probabilities for the system.

To complete the HMM that realises the “Three-way Merge” approach we need next to define the “State Initialisation Probabilities” (Fig. 8). This is the initial system distribution when the algorithm is at its first iteration and with every other iteration, following the first, represents the latest estimation of the state of the system that is used every time for the calculation of the next system’s belief distribution.

Fig. 8. The “State Initialisation Probabilities” for the HMM that realises the “ThreeWay Merge” approach.

At this point we need to emphasise that all values presented in the matrices above have been chosen for the purpose of illustrating the operation of the algorithm we present here through an extended evaluation example that will be discussed in the next section. All these values are defined in the algorithm as parameters (values to variables) and are completely customisable. This means that the actual behaviour of the system can be controlled directly by those parameters as it is the case with every HMM system, and that the user has full powers to parametrise the algorithm by providing the values that simulate a particular behaviour. The full structure of the HMM we constructed is as shown in Fig. 9.

88

N. G. Bakaoukas and A. G. Bakaoukas

Fig. 9. The full structure of the HMM system that realises the “Three-Way Merge” approach for merging two edited versions of an original HTML/XML document.

A Novel Merge Algorithm For HTML/XML Documents

89

Finally, the introduction of the “Nested Parenthesisc” principle in the algorithm completes its operational picture. The principle, as evaluated within the algorithm resembles that used in mathematics to introduce order of execution between calculations in an unambiguous way. The only difference is that the role of “Opening” and “Closing” parentheses is taken over by the “Opening” and “Closing” Tags. In Fig. 10 an illustration of how the “Nested Parenthesis” principle is used in the logical structure of the algorithm to maintain a strictly hierarchical ordering between Tags is presented.

Fig. 10. An illustration of the operation of the “Nested Parenthesis” principle.

The employment of the “Nested Parenthesis” principle, safe guards the algorithm against permitting for unorthodox matching between Tags, something that has been criticized as being one of the major drawbacks for many of the currently existing algorithms (“Diff3”,“XyDiff”, “DaisyDiff”, etc.). Since all priorities set in the algorithm are fully customizable and extendible, the user can introduce similar to the action-observation probabilities priorities for the full extend of HTML/XML Tags, something that covers completely the case of dealing with dynamic Tags very commonly found in XML code. This design for the HMM that fully realises the “Three-way Merge” approach targets one thing, to try and predict what will be the final decision of the editors on the current line of code under investigation by the algorithm, in case there are different individual actions taken in the edited versions of the original document. The outcome for the collective decision is reflected in the HMM system as the current state of it. In order to estimate this current state of the system, and as a consequence the collective decision of the editors, we calculate by using the HMM the probability of the system being at some state St after observing a sequence of two actions O1 & O2 , which we can write as b(St ) = p(St |O1 , O2 ). Doing this for every possible state St gives us a Probability Distribution over the entire state domain. Considering the fact that we are using the Forward Algorithm, the following equation applies per algorithm iteration, which returns the probability of the system being in some state St as the product:  p(St |St−1 )π(St−1 )η (3) b(St ) = B(Ot |St ) St−1 ∈S

With, B the probability of observing action Ot in the state St , p(St |St−1 )π(St−1 ) the probabilities of the system having been at other states in the last algorithm iteration, times the probability of having transitioned over to St , all added up, and η a Normalisation Constant, which ensures that:

90

N. G. Bakaoukas and A. G. Bakaoukas



π(St ) = 1

(4)

St ∈S

As an example let us consider the situation of having to calculate the probability distribution for the “Move” state, after the “Move-Move” action was observed. Drawing values from the matrices in Figs. 5, 6, 7 and 8, the new belief probability for the “Move” state is: b(M ove) = B(M ove − M ove|M ove) ∗ [p(M ove|Delete) ∗ π(Delete) +p(M ove|N o Change) ∗ π(N o Change) + p(M ove|M ove) ∗ π(M ove) +p(M ove|Copy) ∗ π(Copy) + p(M ove|U pdate) ∗ π(U pdate) +p(M ove|Insert) ∗ π(Insert)] ⇒ b(M ove) = (0.5) ∗ {[(0) ∗ (0.3)] + [(0.2) ∗ (0.3)] + [(0.6) ∗ (0.1)] +[(0) ∗ (0.1)] + [(0.2) ∗ (0.1)] + [(0) ∗ (0.2)]} ∗ η ⇒ b(M ove) = 0.07 ∗ η

(5)

With the normalisation constant η defined as: η=

1

s∈S (πs )

(6)

By repeating this calculation for every state in the system’s state domain we take the probability distribution over that domain, which corresponds to an estimate of how likely the system is to be in each state given the current observation.

4

Implementation and Experimental Results

The discussion in the previous section has primarily dealt with the theory of how HMMs can be applied to the problem of merging HTML/XML documents by following the “Three-way Merge” approach, and the defining of the fundamental parameters for the practical realisation of the algorithmic approach we are proposing here. In this section we deal with the practical implementation issues and provide a fully fledged example of how the overall algorithm is operating including observation sequences, initial parameter estimates and decision making. This will serve to fully illustrate how HMMs can be procedurally applied in HTML/XML documents merging situations. Also, as becomes apparent from the discussion in the previous section, in general, the major issues in implementing HMMs is the choice of the underline algorithm (Forward Algorithm, Backward Algorithm, Viterbi Algorithm, etc.), the choice of model size (the number of states), and the choice of observation symbols (discrete or continuous). All three elements are of fundamental importance to the merging problem because the basic philosophy of HMMs is that a single observation or an observation sequence can be well modelled if the parameters of the HMM are carefully and correctly chosen. By definition these choices must be made depending on the system modelled. The

A Novel Merge Algorithm For HTML/XML Documents

91

exact procedure by which we propose to tackle these issues we will analytically present as we proceed with the discussion of the practical example. For any HMM model, containing hidden states, the task of determining which states sequence is the underlying source of some specific sequence of observations is called the “Decoding Task”. In the HTML/XML domain this can be realised as: “given a sequence of editing observations (those from the edited versions, T1 and T2 of the original document) and a HMM, the task of the decoder is defined as to find the best hidden state which corresponds to the collective editors’ decision. More formally, given as input a HMM λ = (A, B, π) and a pair of observations O = {O1 /O2 }, we seek to find the most probable system state St . Now, for each possible hidden state, we can run the “Forward Algorithm” and compute the likelihood of the observation given that hidden state. Then we can choose the hidden state with the maximum observation likelihood. So, our first problem is minimised to computing the likelihood of a particular observation. In our example, we consider the original HTML/XML document and the two edited versions of that document presented in Fig. 3(a)–(c). The target is through the algorithmic procedure we propose to achieve in the end the arrangement presented in Fig. 3(d) for the output document. Starting the procedure we observe that the overall structure of the HMM model is conveniently leading to a series of matrix operations. The representation of the model parameters as matrices can be achieved as follows (from Fig. 5, 6, 7 and 8): Number of States: N = 6, Number of Observations: M = 21 π = [ 0.3 0.3 0.1 0.1 0.1 0.1] ⎡

1 ⎢0 ⎢ ⎢0 A=⎢ ⎢0 ⎢ ⎣0 0 ⎡

0.5 ⎢...0 ⎢ ⎢0 ⎢ ⎢ . . . 0.1 ⎢ ⎢0 ⎢ ⎢...0 B=⎢ ⎢0 ⎢ ⎢...0 ⎢ ⎢0 ⎢ ⎢...0 ⎢ ⎣0 . . . 0.1

0.1 0 0.1 0 0 0.5 0 0 0 0 0 0

0.1 0 0 0 0.1 0.1 0 0.1 0 0 0 0

0.1 0.1 0.2 0.2 0.2 0.2

0.1 0 0 0 0 0.1 0.1 0 0 0.1 0 0

0.2 0 0.6 0 0 0.2

0.1 0 0 0 0 0.1 0 0 0.1 0 0 0.1

0 0 0 0.8 0 0.2

0.1 0 0 0 0 0 0 0.5 0 0 0.1 0

0.2 0 0.2 0.2 0.2 0.2

0 0 0.5 0 0 0 0 0.1 0 0.1 0 0

⎤ 0 0⎥ ⎥ 0⎥ ⎥ 0⎥ ⎥ 0⎦ 1

0 0 0.1 0 0.1 0 0 0.1 0 0 0 0.1

0 0 0.1 0 0 0 0.1 0 0 0.5 0 0

0... 0 0.1 . . . 0 0... 0 0... 0 0.1 . . . 0.1 0... 0.1

⎤ 0 ⎥ ⎥ ⎥ ⎥ 0 ⎥ ⎥ ⎥ ⎥ 0 ⎥ ⎥ ⎥ ⎥ 0 ⎥ ⎥ ⎥ ⎥ 0 ⎥ ⎥ ⎦ 0.5

92

N. G. Bakaoukas and A. G. Bakaoukas

 Q=

0 0 0 0 0 0 1 0 0 0... ...0 0 0 0 0 0 0 0 0 0 0



The “Forward Algorithm” is operating under the following four mathematical conditions, and as an example of its inner structure we present the schematic diagram of its operation, over one iteration, on a far less complicated system that involves only three states and four observations (Fig. 11). Forward Probability: at (j) = A(x1 x2 . . . xt , qt = Sj |λ)  = A(x1 x2 . . . xt , Qt = q1 . . . qt |λ) Qt

=

N 

at−1 (i)aij bj (xt )

(7)

1≤i≤N

(8)

i=1

Initialisation: a1 (i) = πi bi (x1 ), Induction: at (j) =

N 

at−1 (i)aij bj (xt )

1 ≤ j ≤ N, t = 2, 3, . . . , T

(9)

i=1

Termination: A(X|λ) =

N 

aT (i)

(10)

i=1

With the assumption that the last known belief distribution π is defined over the system’s N states, the mathematical calculation for the estimation of the next system state is the following: πt = (A ∗ π). ∗ (B ∗ Q);

(11)

We must normalise this distribution to ensure that the components of the πt vector add up to one. This is achieved by dividing every belief value by the sum of all belief values: (12) πt = η ∗ πtT

A Novel Merge Algorithm For HTML/XML Documents

93

Fig. 11. The schematic diagram of operation for the “Forward Algorithm”, over one iteration, on a far less complicated system that involves only three states and four observations.

For our example (Fig. 3) the algorithmic steps to create the merged HTML/XML document out of the “Original Document” O and the “Current Versions” T1 & T2 are as follows: 1st Iteration: Original

T1 T2

(N o Change) (N o Change)

π = 0.3 0.3 0.1 0.1 0.1 0.1  Q=

0 0 0 0 0 0 1 0 0 0... ...0 0 0 0 0 0 0 0 0 0 0



With the observation being /, the “Forward Algorithm” calculation returns: πt = 0 1.0 0 0 0 0 Which result in the vector π corresponds to the internal state for the system. As a result, no action is taken by the algorithm at the output on the input lines and the one from the original document is passed on to the output intact. So the output document is now the following:

[T ext 1] [T ext 2] [T ext 3]



O =

94

N. G. Bakaoukas and A. G. Bakaoukas

From the mathematical point of view, the new distribution Bt being the latest estimation of the state of the system in the next iteration will be copied into B for the belief distribution computation to begin anew. Following this, the remaining of the steps that complete the algorithmic procedure are as follows: 2nd Iteration: Original T1 T2 [T ext 1] [T ext 1] (Insert) (N o Change)

π = 0 1.0 0 0 0 0 

0 0 0 0 0 0 0 0 0 0... Q= ...1 0 0 0 0 0 0 0 0 0 0



With the observation being /, the “Forward Algorithm” calculation returns: πt = 0 0.3333 0 0 0 0.6667 In the vector π the result corresponds to the internal state for the system. The output document is now the following:

[Blank Line] [T ext 1] [T ext 2] [T ext 3]



O =

3rd Iteration: Original [T ext 1]

T2 T1 [T ext 1] [T ext 1] (N o Change) (N o Change)

π = 0 0.3333 0 0 0 0.6667 

0 0 0 0 0 0 1 0 0 0... Q= ...0 0 0 0 0 0 0 0 0 0 0



A Novel Merge Algorithm For HTML/XML Documents

95

With the observation being /, the “Forward Algorithm” calculation returns: πt = 0 1.0 0 0 0 0 In the vector π the result corresponds to the internal state for the system. The output document is now the following:

[Blank Line] [T ext 1] [T ext 2] [T ext 3]



O =

4th Iteration:

Original [T ext 2]

T2 T1 [T ext 2]

(N o Change) (Insert)

π = 0 1.0 0 0 0 0 

0 0 0 0 0 0 0 0 0 0... Q= ...1 0 0 0 0 0 0 0 0 0 0



With the observation being /, the “Forward Algorithm” calculation returns: πt = 0 0.3333 0 0 0 0.6667 In the vector π the result corresponds to the internal state for the system. The output document is now the following:

[Blank Line] [T ext 1] [Blank Line] [T ext 2] [T ext 3]



O =

96

N. G. Bakaoukas and A. G. Bakaoukas

5th Iteration:

Original [T ext 2]

T2 T1 [T ext 2] [T ext 2] (N o Change) (N o Change)

π = 0 0.3333 0 0 0 0.6667 

0 0 0 0 0 0 1 0 0 0... Q= ...0 0 0 0 0 0 0 0 0 0 0



With the observation being /, the “Forward Algorithm” calculation returns: πt = 0 1 0 0 0 0 In the vector π the result corresponds to the internal state for the system. The output document is now the following:

[Blank Line] [T ext 1] [Blank Line] [T ext 2] [T ext 3]



O =

6th Iteration:

T2 Original T1 [T ext 3] [T ext 3] (Insert) (N o Change) π= 010000  Q=

0 0 0 0 0 0 0 0 0 0... ...1 0 0 0 0 0 0 0 0 0 0



With the observation being /, the “Forward Algorithm” calculation returns: πt = 0 0.3333 0 0 0 0.6667

A Novel Merge Algorithm For HTML/XML Documents

97

In the vector π the result corresponds to the internal state for the system. At this point, because the line to be inserted into the output document is a closing Tag for and before that there is an Tag with a higher priority that has not being closed first, the “Nested Parenthesis” part of the algorithm will introduce automatically an (N ested P arenthesis) line, so the output document is now the following:

[Blank Line] [T ext 1] [Blank Line] [T ext 2] (N ested P arenthesis) [Blank Line] [T ext 3]



O =

7th Iteration:

Original [T ext 3]

T1 T2 [T ext 3] [T ext 3] (N o Change) (N o Change)

π = 0 0.3333 0 0 0 0.6667  Q=

0 0 0 0 0 0 1 0 0 0... ...0 0 0 0 0 0 0 0 0 0 0



With the observation being /, the “Forward Algorithm” calculation returns: πt = 0 1 0 0 0 0 In the vector π the result corresponds to the internal state for the system. The output document is now the following:

[Blank Line] [T ext 1] [Blank Line] [T ext 2] (N ested P arenthesis) [Blank Line] [T ext 3]



O =

98

N. G. Bakaoukas and A. G. Bakaoukas

8th Iteration:

Original

T1 T2

(N o Change) (Insert)

π= 010000 

0 0 0 0 0 0 0 0 0 0... Q= ...1 0 0 0 0 0 0 0 0 0 0



With the observation being /, the “Forward Algorithm” calculation returns: πt = 0 0.3333 0 0 0 0.6667 In the vector π the result corresponds to the internal state for the system. Because the line to be inserted into the output document is a closing Tag for and before it there is no corresponding opening Tag, the “Nested Parenthesis” part of the algorithm will introduce automatically an (N ested P arenthesis) line right after the line reserved for the insertion of the Tag, so the output document is now the following:

[Blank Line] [T ext 1] [Blank Line] [T ext 2] (N ested P arenthesis) [Blank Line] (N ested P arenthesis) [T ext 3] [Blank Line]



O =

9th Iteration:

Original

T1 T2

(N o Change) (N o Change)

π = 0 0.3333 0 0 0 0.6667 

0 0 0 0 0 0 1 0 0 0... Q= ...0 0 0 0 0 0 0 0 0 0 0



A Novel Merge Algorithm For HTML/XML Documents

99



Fig. 12. The final output of the “Forward Algorithm”: (a) the O document, (b) the T1 document and, (c) the T2 document.

With the observation being /, the “Forward Algorithm” calculation returns: πt = 0 1.0 0 0 0 0 In the vector π the result corresponds to the internal state for the system, so the output document is now the following:

[Blank Line] [T ext 1] [Blank Line] [T ext 2] (N ested P arenthesis) [Blank Line] (N ested P arenthesis) [T ext 3] [Blank Line]



O =

With all the outputs per iteration effectively taken together in this last one,  this is the final output of the “Forward Algorithm” for the O document. With the corresponding action to the output of the “Forward Algorithm” per iteration taken on the T1 and T2 documents as well, at the next step the algorithm will bring together the three documents to perform a line-by-line merge between them. The arrangement of documents in the algorithm is now as shown in Fig. 12. After the line-by-line merge of the three documents the final merged version of the original document produced by the algorithm is the one shown in Fig. 13.

100

N. G. Bakaoukas and A. G. Bakaoukas

Fig. 13. The final merged version of the original HTML/XML document as produced by the algorithm proposed in this paper.

A direct comparison between the output as shown in Fig. 13 and the expected correct output for every properly operating merging algorithm as this is presented in Fig. 3d, shows that the algorithm proposed in this paper is capable of producing accurate merging results for HTML/XML documents, under the “Three-way Merge” approach.

5

Conclusion

One of the most fundamental problems restricting the optimum operation of many of the currently available “Three-way Merge” algorithms for HTML/XML documents is their strong dependency, when it comes to decision making, on the original document O. In this paper, a new algorithm proposed for the “Threeway Merge” approach, which uses at its core a Hidden Markov Model (HMM) to estimate the next system state, that is the collective editing decision of the editors on every particular line of the three input documents. The proposed algorithm generates a merged document that is encapsulating a well presented at the browser document within a properly structured HTML/XML programming code. The algorithm proceeds by gathering all important information for each of the documents’ lines directly from them. This fundamental to the algorithm characteristic, along with the reliable results produced by the Hidden Markov Model, makes it capable of achieving far better results than those produced by the existing algorithms in its category, also making it suitable to be used primarily (but not restrictively to) as a programming assisting tool. A key aspect of the algorithm presented in this paper is also its level of flexibility, which is provided by allowing for configurable Hidden Markov Model parameters, along with the “Nested Parentheses” principle. As a result, the algorithm provides the user with the ability to focus on either the HTML element or the XML element features (or even both), whatever is in every case more suitable according to the nature of the documents about to be merged. Acknowledgment. The authors would like to thank editors and anonymous reviewers for their valuable and constructive suggestions on this paper.

A Novel Merge Algorithm For HTML/XML Documents

101

References 1. Bakaoukas, A.G., Bakaoukas, N.G.: A Top-down three-way merge algorithm for HTML/XML documents. In: Arai, K., Kapoor, S., Bhatia, R. (eds.) Computing Conference ’2020, London, UK, 16–17 July, 2020, Intelligent Computing. SAI 2020. Advances in Intelligent Systems and Computing, vol. 1228. Springer, Cham. https://doi.org/10.1007/978-3-030-52249-0-6. Print ISBN: 9783-030-52248-3, Online ISBN: 978-3-030-52249-0 2. Khanna, S., Kunal, K., Pierce, B.C.: A formal investigation of Diff3. In: Arvind, V., Prasad, S. (eds.) Foundations of Software Technology and Theoretical Computer Science (FSTTCS), December 2007 3. IBM Alphaworks: XML diff and merge tool home page. http://www.alphaworks. ibm.com/tech/xmldiffmerge 4. The “DeltaXML” Project. http://www.deltaxml.com. Accessed 29 Mar 2019 5. Lindholm, T.: A three-way merge for XML documents. In: Proceedings of The 2004 ACM Symposium on Document Engineering, pp. 1–10 (2004). https://doi. org/10.1145/1030397.1030399 6. Dinh, H.: A new approach to merging structured XML files. Int. J. Adv. Res. Comput. Eng. Technol. (IJARCET) 4(5) (2015) 7. Ba, M.L., Abdessalem, T., Senellart, P.: Merging uncertain multi-version XML documents, January 2013 8. Oliveira, A., et al.: An efficient similarity-based approach for comparing XML documents. Inf. Syst. 78, 40–57 (2018) 9. Matthijs, N.: HTML, the foundation of the web. http://www.wpdfd.com/issues/ 86/htmlthefoundationoftheweb/ 10. Rozinajov´ a, V., Hluch´ y, O.: One approach to HTML wrappers creation: using document object model tree. In: Proceedings of CompSysTech, pp. 1–6 (2009) 11. Barnard, D.: Tree-to-tree correction for document trees. http://citeseer.ist.psu. edu/47676.html 12. Cobena, G.: A comparative study for XML change detection. http://citeseer.ist. psu.edu/696350.html Chawathe, S.S., Rajaraman, A., Garcia-Molina, H., Widom J.: Change detection in hierarchically structured information. In: Proceedings of The 1996 ACM SIGMOD International Conference on Management of Data, Montreal, Canada, pp. 493–504 (1996) 13. Cobena G., Abiteboul, S., Marian, A.: Detecting changes in XML documents. In: Proceedings of the 18th International Conference on Data Engineering, San Jose, CA, pp. 41–52 (2002) 14. Ghahramani, Z.: An introduction to hidden Markov models and Bayesian networks. Int. J. Pattern Recogn. Artif. Intell. 15(1), 9–42 (2001) 15. Jelinek, F.: Statistical Methods For Speech Recognition (Language, Speech and Communication), 4th edn. Bradford Book, Cambridge (1998). ISBN-13: 9780262100663

Scaling Out Transformer Models for Retrosynthesis on Supercomputers Joris Mollinga(B) and Valeriu Codreanu SURF Corporative, Amsterdam, The Netherlands [email protected]

Abstract. Retrosynthesis is the task of building a molecule from smaller precursor molecules. As shown in previous work, good results can be achieved on this task with the help of deep learning techniques, for example with the help of Transformer networks. Here the retrosynthesis task is treated as a machine translation problem where the Transformer network predicts the precursor molecules given a string representation of the target molecule. Previous research has focused on performing the training procedure on a single machine but in this article we investigate the effect of scaling the training of the Transformer networks for the retrosynthesis task on supercomputers. We investigate the issues that arise when scaling Transformers to multiple machines such as learning rate scheduling and choice of optimizer, and present strategies that improve results compared to previous research. By training on multiple machines we are able to increase the top-1 accuracy by 2.5% to 43.6%. In an attempt to improve results further, we experiment with increasing the number of parameters in the Transformer network but find that models are prone to overfitting, which can be attributed to the small dataset used for training the models. On these runs we manage to achieve a scaling efficiency of nearly 70%. Keywords: Computer aided retrosynthesis · Deep learning · Transformer · High performance computing · Deep learning on supercomputers

1

Introduction

Retrosynthesis is a technique from organic chemistry which involves itself with the planning of organic syntheses. In retrosynthesis a molecule is transformed into smaller and simpler precursor molecules until a set of precursor molecules is found which are commercially available. The precursor molecules can then be used to synthesize the molecule of interest. From the perspective of an organic chemist, retrosynthesis is a very challenging task which is often guided by experience and intuition with often multiple viable paths to the molecule of interest. The first computer aided retrosynthesis code was developed in 1985 by Corey [7]. This program, called LHASA used a template-based approach. It included c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 102–117, 2022. https://doi.org/10.1007/978-3-030-80119-9_4

Scaling Out Transformer Models for Retrosynthesis on Supercomputers

103

a database of templates which were recursively applied to a target molecule. A specific strategy then selects the set of reagents. This process is repeated until a set of commercially available reagents has been found that can be used to synthesize the target molecule. Usually there are multiple valid sets of reagents that can be used and thus the algorithm has to reason about the most effective route. Most early attempts at computer aided retrosynthesis were not very successful, mainly because of the difficulty of maintaining the database with templates. Recent developments in the fields of machine learning and deep learning have boosted research on computer aided retrosynthesis. Although the retrosynthesis problem can be tackled with different machine learning techniques, a popular choice is to treat it as a machine translation task. The product of the chemical reaction is the source language, while the smaller precursors molecules are the target language. The source and target molecules share a common alphabet and grammar. Both recurrent neural networks [17] and Transformer architectures [15, 25] have been applied to this task, where the transformer models showed superior results. Almost all prior research is conducted on single GPU machines. In this article we investigate the effect of scaling the training of the retrosynthesis task on GPU-enabled and CPU-enabled supercomputers. Previous research has shown that there are great benefits in performing this training task on more than one machine. This is known to speed up training and can provide superior results. We first replicate the results from [15] and then improve upon those results by scaling out training to multiple machines. Lastly we also experiment with increasing the parameter count in the models and using high memory systems. We perform scaling experiments on up to 128 CPU nodes and discuss the challenges that come with scaling to a distributed system like learning rate scheduling and choice of optimizer. This paper is structured as follows. In Sect. 2 we discuss previous efforts on computer aided retrosynthesis and distributed training. In Sect. 3 we discuss our approach, describing the dataset, the model, and the distributed training procedure. In Sect. 4 we present the experiments. The results of these experiments are given in Sect. 5. Finally, we discuss our results and present a conclusion in Sects. 6 and 7, respectively.

2 2.1

Previous Work Computer Aided Retrosynthesis

Computer aided retrosynthesis has been an active field since 1985, when the first software package for retrosynthesis was published [7]. During the past decade the advancement in the field of machine learning has also been applied to the task of retrosynthesis. Modern machine-learning based approached can be divided into two categories: (1) template-based approaches and (2) template-free approaches. In this context templates are rules of product describing product and reactant molecules. Templates can be extracted with the help of algorithms or can be hand-crafted. Because the number of templates is very large these have to be

104

J. Mollinga and V. Codreanu

prioritized. The system proposed in [22] learns a conditional distribution over the template set. In [6] templates are sorted using a ranking mechanism. An alternative approach is used in [8], where the joint distribution of templates and reactions is modeled directly with logic variables. Template-free methods learn a direct mapping from products to reactants. Template-free methods often use machine translation techniques like sequence to sequence (seq2seq) models as in [17] or Transformers models as in [15]. The methods use the SMILES [28] representation of molecules which is a string based representation. By treating the problems as a machine translation problem the reactants are generated token by token, or in this context, atom by atom. Different improvements have been suggested to improve template-free methods, like data augmentation [2, 25] and a syntax corrector [30]. Another template-free approach is the work presented in [21] which uses Monte Carlo Tree Search (MCTS) combined with neural networks for retrosynthesis planning. 2.2

Distributed Training

In the quest for developing increasingly complex and expressive neural network architectures demanding more and more computational resources, distributed training has become commonplace and the only way forward to sustain innovation in the field. This is most clearly outlined by the GPT-3 [3] language model featuring 175 billion parameters and requiring 3.14E + 23 floating point operations to train. These state of the art deep learning models are currently impossible to train on a single GPU or CPU. However, scaling out from a single machine to multiple machines is not a straightforward task and may yield inferior results if not performed correctly. When training a model in a distributed fashion, several training strategies can be adopted, with the most prominent being: data parallel, model parallel, and pipeline parallel training. In data parallel training, each worker holds a complete copy of the model, and each training batch gets split across the workers. After the forward pass is completed independently on each worker, the gradients are averaged across all workers during the backward pass, allowing for the neural network parameters to be updated. The advantage of data parallel training is that the workload is uniform, leading to low degrees of load imbalance, particularly if gradient communication is overlapped with computation. However, the disadvantage of this training scheme is that the global batch size (i.e. batch size aggregated across all workers) grows linearly with the number of workers, and this is known to lead to the generalization gap [12] phenomenon, with the proposed solution to increase the number of updates performed in the optimization process. Still, this would cancel most of the benefits of distributed data-parallel training. Nonetheless, there are alternative ways to address the generalization gap, mostly focused on more evolved learning rate schedules and improved optimization techniques [5,10,29]. In model parallel training one divides the model over different workers. This is especially useful when dealing with models that have very high memory requirements and do not fit in the memory of a single device. This comes at the expense

Scaling Out Transformer Models for Retrosynthesis on Supercomputers

105

of high training times, as one worker has to wait for the output of the previous worker, leading to an inherently sequential process. More modern pipelineparallel approaches such as GPipe [13] and PipeDream [11] aim to split a batch into a collection of “micro-batches” to address the computational inefficiencies of model parallel training. This allows the time per training iteration to be reduced, although some overhead related to the “bubbles” in the pipeline remains. In this work we use a data parallel training strategy for the proposed Transformer model, since we consider relatively small models which fit on a single GPU and therefore do not need model parallel training strategies. However, due to the fact that the dataset used is relatively small, the number of updates per epoch decreases heavily as we increase the number of parallel workers. This leads to the generalization gap phenomenon, and ways to overcome it will be discussed in remainder sections of the current work. We use a number of guidelines from [20] and [19] for training models on multiple machines and continue to meet or exceed the accuracy targets.

3 3.1

Approach Dataset

For this research the same dataset as in [15] is used. The dataset contains approximately 50,000 chemical reactions from the USPTO database [18], split in ten reaction classes using the SMILES notation. The SMILES notation is a string notation of molecules. Figure 1 shows an example of a molecule and its corresponding SMILES notation. Each character in the SMILES notation is tokenized according to the same procedure as in [15]. This gives a vocabulary size of 66 tokens, including a start-of-sequence, end-of-sequence and padding token. The dataset contains 40,029, 5,004, and 5,004 examples for training, validation and testing, respectively. Just like in [15] we combine the training and validation set to generate a larger training set of 45,033 samples. In this work we discard the reaction class.

Fig. 1. A molecule and its SMILES representation.

106

J. Mollinga and V. Codreanu

Fig. 2. Multi-head attention architecture.

3.2

Transformer Model

A Transformer model [26] is a type of encoder-decoder neural network. It has shown promising performance on machine translation tasks and other sequence to sequence problems. The transformer architecture does not use any convolutional or recurrent architectures but uses the principle of multihead self-attention: a technique to learn what part of the input is important. The input to the multi-head attention layer is a set of token embeddings h = {h1 , h2 , ..., hN }, hi ∈ Rdh with the output being a separate set of token embeddings h of the same size: h = {h1 , h2 , ..., hN }, hi ∈ Rdh , where N is the sequence length. The first step in the attention mechanism is to get the query, key and value vector by projecting the token embedding: qi = WQ hi , ki = WK hi , vi = WV hi

(1)

Here WQ , WK and WV are different weight matrices. WQ and WK are of shape dk × dh and WV is of shape dv × dh . Here dk and dv are the key and value dimensionality. We then compute the compatibility of the key and query pair. q T kj uij = √i dk

(2)

The next step is to calculate the attention scores using a soft-max function: aij = 

euij uij  j e

(3)

Lastly, the output vector hi is a weighted average of the attention score and value vector:  aij vj (4) hi = j

In [27], Veliˇckovi´c et al. found that using multiple attention heads gives better performance. This allows each node to send different kinds of messages to each

Scaling Out Transformer Models for Retrosynthesis on Supercomputers

107

neighbor. To do this, Eq. 1 to 4 are calculated M times with dk = dv = dMh . This produces M different outputs called him , m ∈ 1, ..., M which are combined O is a matrix of shape dh × dv . using Eq. 5. Here Wm hi =

M 

O  Wm him

(5)

m=0

Lastly the output of the multi-head self-attention layer is added to the original data, layer-wise normalized and fed through a couple of dense layers with again a skip-connection. We refer to the output dimension of the first feedforward layer and the input dimension of the second feedforward layer as hidden size. A schematic overview of the multi-head attention layer is shown in Fig. 2. Transformer networks look at the entire input sequence at once, instead of analyzing it per token like recurrent models. This alleviates problems that can be caused by long range dependencies between tokens. To give the model more understanding of the importance of the position of tokens in the sequence we also use positional embeddings [26] that are added to the token embeddings. The positional embeddings are implemented using the same procedure as in [26]. Unless stated otherwise we use 3 layers in the encoder and decoder, 8 attention heads and an embedding, key size and value size of 64. For inference, we can feed the source sequence to the encoder. This embedded source sequence is then fed to the decoder which generates a target sequence step by step by taking the previous characters it has predicted into account. Upon reporting our results on the test set, we provide results from both a greedy decoding strategy as well as from a beam search decoding strategy [9]. A greedy decoding strategy selects the next token by choosing the token with the highest probability while beam search explores multiple options and chooses the sequence of tokens that eventually leads to the highest probability. This allows for better inference and exploration of the probability landscape. 3.3

Distributed Training

We scale the training procedure to multiple machines using Horovod [23]. Horovod is a library for distributed data parallel training, working together with various machine learning frameworks. With Horovod, each machine (or worker) holds a local replica of the model. After the forward pass on each worker the gradients of all workers are communicated to all other workers and averaged after which each worker performs the backward pass. This synchronization point ensures that all model parameters on all workers are identical after each update step. Of course, one has to broadcast the weights of the model at the beginning of training. One critical step we take in order to allow for efficient scaling of Transformer models is converting assumed-sparse tensors to dense tensors, as described in [4]. This is easily achieved by enabling the sparse as dense flag from Horovod’s DistributedOptimizer class. Apart from syncing the gradient updates, training using multiple workers is not straightforward and one has to take a number of considerations into account for efficient distributed training. These are discussed in the next sections.

108

J. Mollinga and V. Codreanu

Fig. 3. Learning rate schedules for various workers

Dataset Replication. Each worker loads a (local) copy of the dataset. To generate batches each worker samples B samples from the dataset, where B is the batch size on a specific worker. By approximation this ensures that each data point is used for training equally often. In order to allow for each worker to draw a random sequence of B samples, we use different per-worker random seeds. Learning Rate Scheduling. Transformer models are known to benefit from gradually warming up the learning rate. The original Transformer paper [26] uses 4, 000 warmup steps and a decay inversely proportional to the number of optimizer steps taken after the warmup phase. Our baseline method from [15] reports 16, 000 warmup steps as an empirically found number of warmup steps for this specific retrosynthesis task. When training Transformers in a distributed fashion, [20] advises to scale the number of warmup steps inversely proportionally to the number of workers, but less than linearly. After the warmup phase, [26] lets the learning rate decrease proportional to the inverse square root of the number of steps taken. The work from [15] uses a slightly more aggressive decay proportional to the inverse of the number of steps taken. According to [14], applying a cyclic learning rate schedule where the learning rate resets to the maximum value (that is the value reached at the end of the warmup phase) improves results. We also adopt this strategy. An alternative learning rate scheduling strategy is the exponential rate decay strategy from [24]. In this strategy the learning rate is also cyclical but the warmup and decay are linear. Furthermore the maximum learning rate decreases according to an exponential decay. We refer the reader to Fig. 3 for a graphical comparison of the different learning rate schedules used in this work. Choice of Optimizer. During training we found that the performance of the model is very sensitive to the choice of  for the Adam optimizer especially with more than 8 workers. A higher value of  yields less numerical instabilities but reduces the performance. Numerical under or overflow results in NaN values (Not a Number) during training upon which training terminates. In order to

Scaling Out Transformer Models for Retrosynthesis on Supercomputers

109

mitigate this we train with the Adamax optimizer from [16] which does not suffer from this problem. 3.4

System Specifications

All computations are done on worker nodes of the LISA Compute Cluster at the Dutch National Supercomputing Centre SURFSara1 , consisting of dual-socket Intel Xeon Gold 5118 CPUs and four NVIDIA Titan RTX GPUs with 24 GB of GDDR6 SDRAM. We also performed experiments for larger number of workers on the Endeavour Supercomputer2 , as this also enabled us to experiment with larger Transformer models. The Endeavour system consists of dual-socket Intel Xeon 8260L nodes with 192 GB of memory per node. For runs on Endeavour we use two processes per node which allows us to double the number of workers on the same number of nodes. Each process is pinned to a separate socket, maximizing the throughput of the system. For all our experiments we use Python 3.6.10 with Tensorflow(-gpu) 1.15.3. We use Horovod 0.19.5 together with NCCL 2.4.7 (in the GPU case) and Intel MPI (in the CPU case) for the distributed communication backend.

4

Experiments

In this work, our first goal was to replicate the model and results from [15]. We then gradually increase the number of workers for training and optimize the hyper parameters for that specific number of workers. For an increasing number of workers we found that changes to the hyper parameters were required for numerical stability and to improve the validation accuracy of the models. These changes are discussed below. Lastly we experiment with wider models by increasing the number of heads and the hidden size on when using large memory machines. Runs with 1, 4 and 8 workers are performed on GPUs on the LISA Compute Cluster. Runs with 64 and 128 workers are performed on the Endeavour Supercomputer. We summarize the hyper-parameters and experimental settings used in Table 1. 1 Worker as a baseline the exact same training setup and model architecture reported in [15] are used. The learning rate schedule is given by (6) and (7). α=λ·

min(1, u(step), warmup) max(u(step), warmup)

(6)

 u(step) =

1 2

warmup + (step % cycle) if step ≥ cycle step otherwise

https://userinfo.surfsara.nl/. https://www.top500.org/system/176908.

(7)

110

J. Mollinga and V. Codreanu Table 1. Hyper parameters for various experiments

Hyper parameters # workers 1 [15]

4

8

64

128

128-L1

128-L2

Batch size

64

64

64

32

32

32

32

# heads

8

8

8

8

8

8

24

Key size

64

64

64

64

64

128

64

Embedding size

64

64

64

64

64

128

64

Hidden size

512

512

512

512

512

1024

2048

# parameters

1,587,264 1,587,264 1,587,264 1,587,264 1,587,264 6,320,256 5,135,424

# epochs

1,000

Optmizer LR scheduler

Cyclic w/o warm restarts

2,000

2,000

2,000

Adam

Adam

Adamax Adamax Adamax Adamax

Cyclic w Cyclic w Exp warm warm range restarts restarts

2,000 Exp range

1,600 Exp range

2,000 Exp range

Here warmup is the number of warmup steps, λ is a constant, cycle is the number of steps per cycle and step is the number of optimizer steps taken. We use 16, 000 warmup steps and set λ to 20. Training is conducted for 1, 000 epochs, similarly to [15], with a batch size of 64. We show the learning rate in Fig. 3. Lastly we also average the checkpoints of the last 5 cycles after training. 4 and 8 Workers for training on 4 and 8 workers we found that dividing the √ number of warmup steps by 2 nworkers gives stable training. We train for 2, 000 epochs with a batch size of 64 and reset the learning rate every 200 epochs. We found that a sudden increase in the learning rate at the start of a new cycle often results in numerical instabilities and therefore we added a warmup phase to the learning rate in every cycle. During this warmup the learning rate is increased to the maximum learning rate with the same slope as during the initial warmup. The value for λ is left at 20. The model architecture is the same as in [15]. Similarly to [15] we average the checkpoints of the last 5 cycles. 64 Workers and 128 Workers for training on 64 and 128 workers we found that training with the exponential-range cyclic learning rate from [24] results in stabler training. This learning rate schedule is parametrized by Eq. 8 and 9. α = αmin + γ cycle · (αmax − αmin ) · max (0, 1 − x) x=|

step − 2 · lcycle + 1| warmup

(8) (9)

Here γ is a parameter for controlling the exponential decay, cycle is the index of the current cycle, αmax and αmin are the maximum and minimum learning rate respectively, step is the number of steps taken, warmup is the number of

Scaling Out Transformer Models for Retrosynthesis on Supercomputers

111

warmup steps and lcycle is the number of steps per cycle. γ is set to 0.99994, αmin is set to 0.0002 and αmax is set to 0.0075 and 0.0175 for 64 and 128 workers, respectively. We set warmup to 0.5 · lcycle which in turn can be calculated with the batch size, epochs per cycle (200) and number of workers. We refer the reader to Fig. 3 to graphically compare learning rate schedules. We lower the batch size to 32 per worker, and apply the Adamax optimizer. Again we train for 2, 000 epochs and average the last 5 cycles. 128 Workers - Large Models we experiment with larger models by increasing the number of heads, the key size, the embedding size and the hidden size. We report two different experimental settings which we call “L1” and “L2”. For L1 we increase the key size and embedding size to 128 and increase the hidden size to 1, 024. For L1 train for 1, 600 epochs and report results on the last checkpoint while for L2 we increase the number of heads to 24 and increase the hidden size to 2, 048. Changing the key size and the hidden size increases the number of parameters in the model, allowing for more learning capacity. By increasing the number of attention heads we hope the model is able to focus on more parts of the input sequence leading to better results. The learning rate scheduling options are the same as the 128 worker scenario from above. We train for 2, 000 epochs and average weights over the last 5 cycles.

5 5.1

Results Accuracy on Test Set

For the training runs with varying number of workers we report the accuracy of correctly predicting the source molecules on the test with a greedy decoding strategy and with a beam search decoding strategy. We use a beam size of 5 and report the top − 1 accuracy. We compare our results to the results from [15]. These results are shown in Table 2. The results presented in this table is percentage of source molecules predicted correctly from the test set. In order to predict an entire molecule correctly, the model will need to predict all the atoms correctly. Because a molecule can have up to 160 atoms, the model can easily make a mistake. This explains the difference in performance between the character (or atom) based accuracy (reported later on) and the greedy and top−1 accuracy scores. The results indicate that there is a benefit to training on distributed systems. Following the greedy decoding strategy, the performance on the test set is improved by 1.3%, 2.5% and 0.5% for 4, 8 and 64 workers, respectively. Following the beam search decoding strategy this yields an improvement of 2.1%, 1.6% and −0.2%, respectively. Up to 64 workers either the accuracy using a greedy decoding strategy or the accuracy using a beam search decoding strategy improves with respect to the baseline. After that it no longer improves. We assume this is because when training using so many workers, the optimizer does not perform enough gradient updates to improve with respect to the baseline (11 gradient

112

J. Mollinga and V. Codreanu

Table 2. Results for various experiments. Results in bold indicate the best results in that column. # workers Results Greedy Top-1 1 - [15]

40.6%

42.7%

4

41.1%

43.6%

8

41.6% 43.4%

64

40.8%

42.6%

128

37.6%

40.1%

128 - L1

38.9%

40.6%

128 - L2

40.4%

41.6%

updates per training epoch), a conclusion similar to the one from [12]. Training larger models (the L1 and L2 run) does not improve results with respect to the baseline, however these models that were also trained on 128 workers improve upon the base model trained on similar number of workers. Because of the high number of parameters these models tend to overfit on the relatively small training set used, resulting in poorer performance on the test set. Evaluation on the training set shows a training accuracy of 99.6% proving the points on overfitting. 5.2

Scaling Performance

For evaluating the scaling performance we report the time per epoch which is the amount of time it takes to evaluate the entire training dataset once. From this metric we calculate the computational speedup factor . We also report the time till score which is the time it takes to reach a validation accuracy of 97.0%. From these metrics we calculate the convergence speedup factor . This 97.0% is based on character (or atom) prediction performance, not the capability of predicting the entire molecule correctly. This was also discussed in Sect. 5.1. We split the analysis in training performed on GPUs and training performed on CPUs in order to have a fair comparison. We present the scaling results using GPUs in Table 3 and the scaling results using CPUs in Fig. 4. Table 3. Scaling performance on GPU. # of workers Results Time per epoch

Time till score

Computational speedup factor

Convergence speedup factor

1 - [15]

125.6 s

13 h 57 m

1×

1×

4

36.9 s

3 h 35 m

3.4×

3.9×

8

21.0 s

2 h 11 m

6.0×

6.4×

Scaling Out Transformer Models for Retrosynthesis on Supercomputers

113

GPU the time needed to train is drastically reduced by training on an increasing number of machines. If scaling were to be ideal we would expect a speedup of 4× when training on four workers for both the computational speedup factor and convergence speedup factor. The scaling performance in terms of convergence speedup is near optimal for 4 workers but this performance reduced when running with 8 workers. The scaling performance in term of the computational speedup factor are sub-optimal. We attribute this sub-optimal scaling to the relatively low computational complexity of the models. The computational footprint of the models is not very high, therefore the overhead introduced by communicating gradients across workers is relatively large compared to the time spent on computation. Furthermore it can be seen that scaling performance gets worse with increasing number of workers which is to be expected because the overhead introduced by communication to other workers is larger with more workers. CPU in order to make a fair comparison we also performed a number of training runs with up to 256 CPU workers in order to measure the time per epoch metric. All CPU training runs use two workers per node, as this makes optimal use of a dual-socket system. The computational speedup factors for these runs tend to exceed the ones on GPUs, even though the per-worker batch size in the CPU case is only 32. We do not report the results of evaluating these training runs on the test set since these are identical to the runs on GPUs. We show the scaling performance in Fig. 4. We obtain a scaling efficiency close to 70%, leading to dramatic reductions in training times. Furthermore, we perform scaling runs for the L2 large model described previously. These results are also presented in Fig. 4. The speed-up factors achieved on the L2 model are similar to the ones for the original model, and the limitations are mostly due to the relatively small batch size that we use per node (32) in order to achieve reasonable validation accuracy for the trained models. Besides measuring the computational speed-up factors in terms of training throughput, we also present 4 runs going from 8 workers to 64 workers, in order to assess the time it takes to reach certain model accuracy. Figure 5 outlines the benefits of the proposed distributed training system, showing the amount of time saved for reaching a given accuracy target when increasing the node count. All models reach at least 97.0% accuracy on the test set, as also discussed in the previous section. The sudden jumps in the accuracy are caused by the learning rate entering a new cycle and not all runs reported in Fig. 5 have been completed, as a limited training budget of 12 h per run was used for this comparison.

6

Discussion

After extensive experimentation with scaling out Transformer models for this retrosynthesis problem, we realize that the bottleneck is the size of the training dataset. This dataset only contains 45, 033 samples and with 128 workers in just 11 iterations with a per-worker batch size of 32 examples, the entire data sat has been fed to the model once. This limits the scale at which we can train,

114

J. Mollinga and V. Codreanu

Fig. 4. Scaling performance on CPU.

Fig. 5. Validation accuracy for increasing number of workers.

and also the scaling efficiency due to the fact that small per-worker batch sizes can be employed. We think this problem would benefit from a larger dataset, both in terms of solution quality and scaling performance. Even so, we show it is possible to speed up training by a factor of 24.7× using 32 compute nodes (64 workers) and reach reasonable accuracy. We have tried to train our models

Scaling Out Transformer Models for Retrosynthesis on Supercomputers

115

with augmented data from [25] but found this yield inferior results. The small USPTO-50k dataset used in this article warrants overfitting. This is confirmed by the high training accuracy for the wider models which tend to overfit even sooner. This is a known problem when training on small datasets, and in the context of retrosynthesis, [1] mitigates this by pretraining their models on the USPTO-380K dataset after which the model is fine tuned on the USPTO-50k dataset. It is left for further research to investigate how this approach can be applied when training at scale.

7

Conclusion

In this work we have used the Transformer model from [15] and trained it in a distributed fashion using both GPU-enabled nodes and a large-scale CPUbased supercomputer. We have shown that the top-1 accuracy increases by 2.5% when training on a distributed system with 8 workers. Furthermore we shown an improvement in accuracy on a distributed system with up to 64 workers. This trend does not continue when training on a system with 128 workers. We also experimented with heavier models composed of more parameters in an attempt to increase performance. However, these experiments were not successful and led to overfitting. We attribute this to the small size of the training dataset used in this specific task (45, 033 examples) and speculate that pretraining on a larger dataset would boost performance. However, we leave this investigation for future work. Lastly we discussed issues inherent to scaling transformer models to multiple machines, such as learning rate scheduling and optimizer choices. We show that using the exponential range learning rate scheduling scheme does not result in under or overflow issues during training, and mitigates the effects of large-batch training. Furthermore we found the Adamax optimizer to be more stable compared to Adam. On this specific problem we manage to achieve a scaling efficiency of nearly 70% on 128 nodes (256 workers). Acknowledgment. We thank the anonymous referees for their constructive comments, which helped to improve the paper. This project has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 814416. We would also like to acknowledge Intel for providing us the resources to run on the Endeavour Supercomputer.

References 1. Bai, R., Zhang, C., Wang, L., Yao, C., Ge, J., Duan, H.: Transfer learning: making retrosynthetic predictions based on a small chemical reaction dataset scale to a new level. Molecules 25(10), 2357 (2020) 2. Bjerrum, E.J.: Smiles enumeration as data augmentation for neural network modeling of molecules. arXiv preprint arXiv:1703.07076 (2017) 3. Brown, T.B., et al.: Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)

116

J. Mollinga and V. Codreanu

4. Cavdar, D., et al.: Densifying assumed-sparse tensors. In: Weiland, M., Juckeland, G., Trinitis, C., Sadayappan, P. (eds.) ISC High Performance 2019. LNCS, vol. 11501, pp. 23–39. Springer, Cham (2019). https://doi.org/10.1007/978-3-03020656-7 2 5. Codreanu, V., Podareanu, D., Saletore, V.: Scale out for large minibatch SGD: residual network training on imagenet-1k with improved accuracy and reduced time to train. arXiv preprint arXiv:1711.04291 (2017) 6. Coley, C.W., Rogers, L., Green, W.H., Jensen, K.F.: Computer-assisted retrosynthesis based on molecular similarity. ACS Central Sci. 3(12), 1237–1245 (2017) 7. Corey, E.J., Long, A.K., Rubenstein, S.D.: Computer-assisted analysis in organic synthesis. Science 228(4698), 408–418 (1985) 8. Dai, H., Li, C., Coley, C.W., Dai, B., Song, L.: Retrosynthesis prediction with conditional graph logic network. In: Advances in Neural Information Processing Systems, pp. 8872–8882 (2019) 9. Goodman, L., Reddy, R.: Effects of branching factor and vocabulary size on performance. Speech understanding systems: summary of results of the five-year research effort at Carnegie-Mellon University, p. 39 10. Goyal, P., et al.: Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017) 11. Harlap, A., et al.: Pipedream: fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377 (2018) 12. Hoffer, E., Hubara, I., Soudry, D.: Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In: Advances in Neural Information Processing Systems, pp. 1731–1741 (2017) 13. Huang, Y., et al.: Gpipe: efficient training of giant neural networks using pipeline parallelism. In: Advances in Neural Information Processing Systems, pp. 103–112 (2019) 14. Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G.: Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 (2018) 15. Karpov, P., Godin, G., Tetko, I.V.: A transformer model for retrosynthesis. In: Tetko, I.V., K˚ urkov´ a, V., Karpov, P., Theis, F. (eds.) ICANN 2019. LNCS, vol. 11731, pp. 817–830. Springer, Cham (2019). https://doi.org/10.1007/978-3-03030493-5 78 16. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 17. Liu, B., et al.: Retrosynthetic reaction prediction using neural sequence-to-sequence models. ACS Central Sci. 3(10), 1103–1113 (2017) 18. Lowe, D.M.: Extraction of chemical structures and reactions from the literature (2012) 19. Ott, M., Edunov, S., Grangier, D., Auli, M.: Scaling neural machine translation. arXiv preprint arXiv:1806.00187 (2018) 20. Popel, M., Bojar, O.: Training tips for the transformer model. Prague Bull. Math. Linguist. 110(1), 43–70 (2018) 21. Segler, M.H., Preuss, M., Waller, M.P.: Planning chemical syntheses with deep neural networks and symbolic AI. Nature 555(7698), 604–610 (2018) 22. Segler, M.H.S., Waller, M.P.: Neural-symbolic machine learning for retrosynthesis and reaction prediction. Chem.-A Eur. J. 23(25), 5966–5971 (2017) 23. Sergeev, A., Balso, M.D.: Horovod: fast and easy distributed deep learning in tensorflow. arXiv preprint arXiv:1802.05799 (2018)

Scaling Out Transformer Models for Retrosynthesis on Supercomputers

117

24. Smith, L.N.: Cyclical learning rates for training neural networks. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464–472. IEEE (2017) 25. Tetko, I.V., Karpov, P., Van Deursen, R., Godin, G.: Augmented transformer achieves 97% and 85% for top5 prediction of direct and classical retro-synthesis. arXiv preprint arXiv:2003.02804 (2020) 26. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017) 27. Veliˇckovi´c, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. In: International Conference on Learning Representations (2017) 28. Weininger, D.: Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. J. Chem. Inf. Comput. Sci. 28(1), 31–36 (1988) 29. You, Y., et al.: Large batch optimization for deep learning: training bert in 76 minutes. arXiv preprint arXiv:1904.00962 (2019) 30. Zheng, S., Rao, J., Zhang, Z., Xu, J., Yang, Y.: Predicting retrosynthetic reactions using self-corrected transformer neural networks. J. Chem. Inf. Model. 60(1), 47–55 (2019)

Naming Processes in Multichannels with Beeps in the Weak Model Layla S. Aldawsari1,2(B) and Tom Altman1 1

2

Department of Computer Science and Engineering, University of Colorado Denver, Denver, CO, USA {layla.aldawsari,tom.altman}@ucdenver.edu College of Computer and Information Sciences, Princess Nourah Bint Abdulrahman University, Riyadh, Saudi Arabia [email protected] Abstract. A system of processes is examined that communicate with beeps in multiple channels. In the beeping model, the processes have limited communication where they can detect the presence of a signal (beep), or its absence (silence). In the weak model, the processes can choose to transmit a beep on one or more channels and can detect beeps and/or silence on all channels. The processes are anonymous and they begin without names to identify themselves. The objective is to develop distributed naming algorithms for two models when n is either known or unknown, where n is the number of processes. The Multichannel Many Beeps Naming algorithm is a Las Vegas algorithm developed for the model when n is known and that has an optimal time complexity of O(log n) rounds. When n is unknown, a Monte Carlo algorithm was developed, called the Unknown Multichannel Many Beeps Naming. It has an optimal time complexity of O(log n) rounds and a probability of success that is at least 1 − (n log n)−1 . These algorithms show an asymptotic improvement when compared to the existing naming algorithms for this model. Keywords: Naming algorithm · Distributed naming channels · Anonymous processes

1

· Beeping

Introduction

Naming anonymous processes is one of the fundamental problems in distributed computing. In this problem, the processes begin without names to identify themselves, and the goal is to provide unique names to all the processes through a distributed algorithm. In recent years, this problem has been studied in new models where the processes communicate through beeps [12, 25]. In this type of model, known as a beeping model, the processes have limited communication capability, and the channel is known as a beeping channel. The processes communicate in synchronous rounds where a process can choose to either beep or remain silent. A beep is received by all processes, but its source cannot be identified. In this work, we focus on beeping channels that have no collision detection, c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 118–132, 2022. https://doi.org/10.1007/978-3-030-80119-9_5

Naming Processes in Multichannels with Beeps in the Weak Model

119

where the processes are only able to identify the presence of a beep or silence [1,17,25]. The beeping model represents the simplest form of communication, which makes solutions developed for such models applicable in more sophisticated forms of communications. This simplistic nature makes it suitable for analyzing different forms of wireless communications. Thus, it has received much attention in solutions designed for various problems, such as leader election, counting, and broadcasting [7,9,13–15]. Applications for the beeping model exist in several fields, such as wireless sensor networks and biological systems [2,23]. Significant work has been done on designing solutions for naming anonymous processes that communicate with beeps using a single channel [11]. Further work has been done to improve the energy efficiency of naming for a similar model [4]. Recent research has been conducted on naming anonymous processes communicating on multiple channels [3]. This work considers a strong model where a process can only listen or beep on a single specific channel. Therefore, little information can be shared between processes in a single round. In this work, we develop a distributed naming algorithm for assigning unique names to n anonymous processes which communicate with beeps using multiple channels. Specifically, we consider the weak model in which a process can beep and receive feedback on multiple channels in a single round. Thus, much information can be shared between the processes in a single round which is dependent on the number of channels available. A process can choose to transmit one or more beeps on one or more channels simultaneously in a single round. Until now, no naming algorithms were designed specifically for this model, in which the processes are able to share more information as compared to the strong model. The new naming algorithms are explicitly designed for the weak model and will improve its performance. The goal is to develop distributed algorithms for assigning unique names to all anonymous processes when the number of processes n is either known or not known. One of the challenges of developing a naming algorithm in a beeping model is that the source of any beep is unidentifiable. Thus, a process cannot determine the process who transmitted the beep. In addition, there are no collisions which makes the design of the algorithm more complicated. We developed two novel algorithms for naming anonymous processes; one is a Las Vegas algorithm, and the other is a Monte Carlo algorithm. The Multichannel Many Beeps Naming (MMBN) algorithm was developed for the model with a known number of processes which has an optimal time complexity of O(log n) rounds. For the model with an unknown number of processes, the Unknown Multichannel Many Beeps Naming (UMMBN) algorithm has an optimal time complexity of O(log n) rounds and a probability of success that is at least 1 − (n log n)−1 . These naming algorithms outperform existing algorithms at the cost of requiring a high number of channels. They can be used to accelerate the initialization stage in a network of processes without an infrastructure. Furthermore, they could contribute to different areas, e.g., distributed cloud computing and wireless sensor networks.

120

L. S. Aldawsari and T. Altman

This paper is structured as follows. In Sect. 2, we present an overview of related work. Technical preliminaries of the model are provided in Sect. 3. Then, in Sect. 4, the MMBN algorithm is described and its proof is presented. The UMMBN algorithm is described in Sect. 5. Finally, in Sect. 6, we provide the conclusions of our work.

2

Related Work

The naming problem is strongly related to the problem of symmetry because the processes begin with identical states. Angluin showed that randomization is necessary to break the symmetry for a network of nodes that are indistinguishable [6]. First, naming algorithms designed for processes communicating through shared memory are presented. Lipton and Park [19] developed a naming algorithm that uses randomization and can terminate with a high probability of gecio˘glu and success. Their algorithm requires a space size of O(L ∗ n2 ) bits. E˘ Singh [16] designed a Las Vegas naming algorithm for an asynchronous communication model with a known number of processes that have polynomial running time. Teng [26] proposed naming algorithms that use less shared memory for a similar model. Kutten et al. [18] presented an in-depth study of naming for the asynchronous model where processes communicate with shared read-write registers. They proposed a Las Vegas naming algorithm for the model where n is known that has a run time of O(log n) while using O(n) shared registers. A randomized wait-free naming algorithm was developed by Panconesi et al. [24] for processes communicating through single-writer atomic registers. The algorithm can work even with the possibility of crashes. Buhrman et al. [10] demonstrated that naming is more difficult than consensus for an asynchronous model where processes are prone to crashes. Next, some work related to naming with processes communicating through the beeping model are discussed. Chlebus et al. [11] developed two algorithms for naming anonymous processes communicating using a single channel. Las Vegas and Monte Carlo algorithms were proposed when n is either known or unknown and both have a run time of O(n log n) rounds. In [3], two algorithms were constructed for naming anonymous processes communicating via multiple channels with beeps. They considered a strong model where processes can choose to beep or listen on a particular channel based on the channel ID. The naming algorithms have a run time of O(n log n), but have the potential to accelerate the naming procedure when n is dependent on the largest group size. The naming problem is also called an initialization protocol, which is closely related and typically used in ad-hoc networks [27]. Nakano and Olariu [21,22] presented initialization protocols that use randomization for processes communicating with messages for single and multiple channels. Bordim et al. [8] constructed a better performing initialization protocol that requires less energy. Myoupo [20] proposed protocols for initializing nodes communicating through multiple channels when the group sizes are either matching or not.

Naming Processes in Multichannels with Beeps in the Weak Model

121

Next, related works other than naming studies are reviewed. Dufoulon et al. [14] developed a leader election algorithm for multi-hop networks where processes communicate with beeps in a synchronous communication model. Brandes et al. [9] designed an algorithm for approximating the size of a single-hop beeping network. In addition, broadcasting and multi-broadcasting algorithms were developed in [7,13]. In [5], Andriambolamalala and Ravelomanana developed both a random bit generator and a binary consensus protocols that are energy efficient and fault-tolerant. Noisy beeping channels were studied by Efremenko et al. in [15]. They showed that resilient beeping protocols require significant overhead.

3

Technical Preliminaries

A system of a fixed number of anonymous processes that have no identifiers is considered. All processes are indistinguishable and begin in identical states. The communication is synchronous such that time is equally divided into rounds. The processes communicate using multiple channels with beeps. The weak model is implied in which a process can listen to all available channels simultaneously. A process can also beep on one or more channels simultaneously in a single round. Thus, in a single round, a process can choose to beep on one or more channels and listen for beeps on all channels. The beeping channels do not have collision detection capability. One or more beeps that were transmitted on a channel c during a round r result in the feedback of a beep on that channel during the same round. When a process receives a beep as feedback on a specific channel, it cannot identify the source of the beep as well as whether one or more processes transmitted the beep. There is a number of channels given names beginning from 1. All processes have consistent/global channel names such that channel c for process p is also referenced by process q as c. It is assumed that channels can be divided into k groups with a specific number m of subchannels. Such division implies that group y consists of the channels (y ∗ m, y ∗ m + m − 1). Referencing a channel of a group is instantaneous (e.g., a process can reference Channel 1 of group y as channel (y ∗ m). In addition, it is assumed that a process can read the feedback from all channels instantaneously in the same round.

4

Naming in the Known Number of Processes Model

We developed a distributed Las Vegas algorithm for naming anonymous process in multiple channels with beeps when the number of processes is known. When this algorithm terminates, its result is guaranteed to be correct and it assigns unique names to the anonymous processes. In this model, it is assumed that all processes have knowledge about the total number of processes n and that information is used by the algorithm. The design for the presented algorithm resembles assigning balls to bins, where the bins correspond to labels and balls represent processes. The key idea is to have each ball assigned uniformly at

122

L. S. Aldawsari and T. Altman

random to one of the bins during each attempt. This procedure is repeated until the random assignment yields all bins having at most one ball. Next, the naming algorithm for assigning unique names to anonymous processes communicating with beeps in the weak model. Here, n processes communicate through multiple channels with beeps is described. The channels are divided into k groups such that k = logn n . Each group of channels consists of m subchannels, where m = log n, see Fig. 1. The k groups of channels represent the bins, while the m subchannels are used for identifying the potential process. The procedure for identification and collision detection is performed through a randomly generated number known as the CodeWord. The Pseudo code of the MMBN is shown in Algorithm 1. It is carried out as follows: In the beginning, all processes first initialize an internal variable counter set to 0. The variable counter is utilized for tracking the number of processes that are currently assigned names. Then, the processes execute the assignment loop for several iterations α, where α ≥ 1. Each iteration of the assignment loop consists of two phases, where each phase comprises a single round. The first phase is known as the labeling phase in which a process is assigned to a random group of channels. The second phase is called the announcement phase, in which a winner process notifies all processes about its winning status. Before beginning an iteration, a process first chooses both a random group y and a random number r, which we call CodeWord. The random group y is chosen randomly from the range (1, k), whereas the random number is chosen from the range (1, n). Channels Group 1

Group 2

Channel 1

Channel m+1

Channel (k-1)m+1

Channel 2

Channel m+2

Channel (k-1)m+2

Channel m

Channel 2m

Channel km

1

2

Group k

n

Fig. 1. The system overview for the MMBN and the UMMBN algorithms.

Naming Processes in Multichannels with Beeps in the Weak Model

123

In the labeling phase of an iteration, a process beeps its CodeWord r on the subchannels of group y. The procedure for transmitting the CodeWord is performed by first transforming r into its binary form representation denoted as Binary(r). Therefore, the binary string Binary(r) requires log n bits, which matches the total number of subchannels m. The transmission of the binary string Binary(r) is carried out during a single round. The process transmits the binary string Binary(r) on the subchannels of group y. The bit bi is mapped to the channel ci of the set of subchannels m where i is the order of the bit. For each bit bi , the process transmits a beep on channel ci when bi = 1. At the same time of transmission, a process keeps listening for beeps on the m subchannels of group y. The binary string received as feedback from the channel is denoted as feedback y . Based on the feedback received on the m subchannels, the process checks whether feedback y matches the binary representation of r. The process is designated as a winner process of the group y when feedback y = Binary(r). Otherwise, a process fails to win the group y, so it attempts to win another random group in the next iteration of the assignment loop. Next, a process goes through the announcement phase, and its purpose is to notify other processes about the winning status of a process and to synchronize the counter between all processes. To avoid assigning identical names, the value of the counter must be synchronized between all processes because it is used for assigning names when a process is designated as a winner. The announcement procedure is executed as follows. Each winner process beeps on Channel 1 of its selected group y, which we call a winner group. During the announcement procedure, each process keeps listening for beeps on Channel 1 of all k groups. A process increments its counter variable by 1 for each beep that was received. A winner process assigns itself a name based on the counter value. Thus, a winner process is assigned a name based on its ascending order of the winning groups. Assuming that a process p has won group y and its order of the winning groups is j, then the process p assigns itself (counter + j) as a name. Therefore, at the end of the second phase, all winning processes are assigned names based on the winning group order, and the variable counter is updated for all processes. Similarly, in the next iteration of the assignment loop, all remaining processes, if there are any, continue executing both phases. This continues until α iterations of the assignment loop have elapsed. Once the assignment loop has ended, all processes check whether the value of counter matches n to verify that all processes have been assigned unique names. If they do not match, then the algorithm is repeated with all variables being re-initialized. Otherwise, the algorithm terminates, and all processes are assigned unique names in the range (1, n). 4.1

Proof of Correctness and Analysis

The correctness of the algorithm is proven and its time complexity is estimated in this section. Lemma 1. An execution of a randomized algorithm for naming n anonymous processes requires Ω(n log n) random bits.

124

L. S. Aldawsari and T. Altman

Algorithm 1: Multichannel Multi Beeps Naming repeat /* algorithm main loop n ←number of processes k ← n log n round ← 0 is winner ← false counterp ← 0 feedbackp ← ∅ namep ← ∅ yp ←random(1, k) rp ←random(1, n) repeat /* assignment loop /* first phase if is winner = false then Send Binary Number(r, y) feedbackp = Get Feedback(y) if feedbackp = Binary(r) then is winner ← true /* second phase if is winner = false then yp ←random(1, k) rp ←random(1, n) else namep ← Name By Winning Order(counterp ) for i = 1, k do if Feedback On Channel(1,i) = beep then counterp ← counterp + 1 /* iteration endedd /* end of segment until round = α until counterp = n

*/

*/ */

*/

*/ */

Proof. We examine the naming problem as a balls-into-bins problem, such that balls represent the processes and bins represent groups of channels. The set of all possible correct assignments of names to the processes is obtained through the permutation of all names in the range (1, n). The probability distribution of an algorithm is determined based on the set of all possible correct assignments that produce correct output. This probability distribution is uniform due to the symmetry where all processes begin with the same initial state and execute the same code where the execution path is determined by flipping a coin. The set of all correct outcomes is determined as n!, and the probability of obtaining one of 1 . The average level of information needed for one set of the correct answers is n! correct output is log (n!) = Θ(n log n) bits. Therefore, a randomized algorithm

Naming Processes in Multichannels with Beeps in the Weak Model

125

for assigning unique names requires at least Ω(n log n) bits, which are obtained as knowledge by all n processes.   Theorem 1. The MMBN algorithm is guaranteed to terminate and it has no error on termination such that unique names are assigned to a set of n anonymous processes with a time complexity of O(log n) rounds and a probability of success which is at least 1/k. Proof. The execution of the MMBN algorithm can be defined as a series of repeated segments consisting of two rounds. Such repeated segments have a minor influence on the time complexity with a factor of O(2). The segments begin from 0 and are repeated α times before rerunning the MMBN algorithm. In each segment, a process chooses one of the k groups uniformly at random, denoted as y, and transmits its random CodeWord r on the group k as a combination of beeps and silence based on the binary representation of r. The process also keeps listening for beeps on the group y and obtains a binary string of length m denoted as feedback y . The procedure of transmitting and receiving takes constant time, as it happens instantaneously during the same round. When feedback y = r, the process designates itself as a winner process. Otherwise, it attempts to win a different group that is selected uniformly at random in the next segment. Each winning process then announces its winning status on the second round of a segment through beeps on Channel 1 of the selected groups. At the same time, all processes discover the winning status of other processes through the identification of beeps on Channel 1 of all groups (1, k). Each detected beep is used for updating the counter variable by adding 1, which is treated as obtaining knowledge of the random information generated by all processes. The procedure of identifying the winning statuses and updating the counter variable takes exactly O(1) round. The winning process assigns itself the value of counter as a name after updating the counter value for its selected group. Now, the correctness of assigning unique names to all processes is proved. The proof is demonstrated through induction where it is shown that knowledge about names is shared and synchronized through the counter variable and terminates when counter = n. Initially, the invariant holds for Segment 0 where all processes start with identical states. All processes have no knowledge of other processes, and the counter value is the same for all processes, which is initialized to 0. Next, it must be shown that the invariant holds through the inductive step. Suppose at segment j, all processes have obtained knowledge about the winning number of processes such that counter = c where c ≥ 0 for all processes. Then, in segment j+1, all processes update the counter value when the number of winning processes w is greater than 0. This new knowledge is obtained by all processes as they keep listening to beeps on Channel 1 in all k groups. All processes keep track of this knowledge by setting counter = counter + w at the end of the segment. At the same time, the winning processes update their names based on the counter value and their winning group order. Thus, all processes still have the same knowledge at the end of the segment, and so the invariant is true. Next, we show that, at the end of the MMBN algorithm, all processes still obtain the same knowledge in which counter = n. Assuming that segment j is

126

L. S. Aldawsari and T. Altman

the last segment, the number of winning processes is w. All processes update their knowledge and set counter = counter + w, which is denoted as counter . At the same time, the winning processes are assigned names based on their winning group order and the counter value. A process can identify whether the knowledge of all processes has been obtained by checking the algorithm’s termination condition counter = n. Suppose the termination condition is false; it implies that two or more processes have obtained identical names. Therefore, all processes repeat the MMBN algorithm. Otherwise, when the condition is true, it entails that all processes have gained knowledge about all processes and that each process was assigned a unique name. All processes then terminate once the termination condition is true. Consequently, it is guaranteed that all processes eventually terminate, and that, on termination, all processes acquire the knowledge of all processes, such that counter = n. This knowledge implies that all of them are assigned unique names from the range (1, n). Thus, the invariant holds on the termination of the MMBN algorithm, which concludes the correctness part of the proof. Next, the time complexity for the MMBN algorithm is computed including the probability of error for assigning a unique name. By Yao’s principle, a randomized algorithm is bounded by the best time of the deterministic algorithm [28]. From Lemma 1, we know that at least Ω(n log n) bits must be shared as knowledge among the processes before they are assigned unique names. All processes learn new knowledge for every iteration of the assignment loop. Each group of channels provides, at most, log n bits of knowledge. The amount of knowledge is dependent on the number of groups, which is, at most, k. Furthermore, a process can send its binary CodeWord r on one of the k groups in constant time, where all bits are sent as beeps on the subchannels of its selected group simultaneously. Thus, for a single iteration, the amount of information learned is, at most, k log n. Therefore, the expected number of iterations for a deterministic algorithm is Ω(n log n/k log n) = Ω(log nk ). By substituting k = logn n , the time complexity is Ω(log n) rounds. Next, we prove this lower bound on the time complexity. The execution of the randomized algorithm can be understood as a probability distribution on a finite number of deterministic algorithms. The input for the deterministic algorithms are binary strings with the same length. The algorithm is run as a series of segments, and the lower bound is determined based on the number of segments required for the best deterministic algorithm. This deterministic algorithm works as a tree where the leaves represent the names for the processes, whereas the root node represents the total count of the names. It terminates when the count at the root matches the total number of processes. The tree size is multiplied by 2 when the count does not match n. In Segment 0, the number of groups of channels is 0. Therefore, at the end of the segment, the processes know about at most 20 = 1 process. Next, it must be shown that in Segment i all processes learn about at most 2i processes. At the beginning of Segment i, a process will know about at most 2i processes. In this segment, the number of available groups of channels is 2i . Thus, by the end of the segment a process will know about 2i + 2i = 2i+1

Naming Processes in Multichannels with Beeps in the Weak Model

127

processes. A deterministic algorithm will assign unique names to all the processes at segment j when each process has knowledge about every other process, which is determined by the counter. Therefore, the best deterministic algorithm will terminate on segment j when 2j ≥ n. The algorithm terminates on segment j when j ≥ log n. Each segment consists of 2 rounds, so the algorithm has a lower bound of Ω(log n) rounds, which is also a lower bound for any randomized algorithm for that problem. Thus, the algorithm has an optimal run time of O(log n) rounds.  

5

Naming in the Unknown Number of Processes Model

We developed the UMMBN algorithm for naming anonymous processes when the number of processes is unknown. The algorithm is Monte Carlo because it has a probability of error of less than 12 on termination. This algorithm resembles the assignment of balls to bins; however, the number of bins is unknown. The idea for the algorithm is to keep repeating the algorithm with an increased number of bins when duplicate names are detected. The pseudo code of UMMBN is shown in Algorithm 2. First, all processes begin with k groups of channels, where k = 2t such that t = 2. Each group of channels comprises t + 1 subchannels. Channel 1 is reserved for announcing the winning status, whereas the remaining channels are used for either transmitting the unique CodeWord or collision-checking. Furthermore, initially, the variable collision is defined and set to false. It is used for informing all processes that a collision was detected so that the UMMBN algorithm is repeated with larger groups of channels. Next, all processes begin executing the assignment loop, which is run for αk iterations, where α ≥ 1. At the beginning of the execution for the assignment loop, each process chooses a random group y uniformly from the range (1, k). Additionally, each process generates a random CodeWord from the range (1, 2t ). A single iteration of the assignment loop consists of two phases. The first phase is referred to as the labeling phase in which processes are assigned to groups through the transmission of the random CodeWord r. The second phase is known as the announcement and collision checking phase where processes notify other processes about their winning status and perform a collision-checking procedure. Each of these phases takes a single round, so the time complexity of the algorithm is affected by, at most, a factor of O(2) rounds. In the labeling phase, each process transmits its CodeWord by first transforming it into its binary form representation Binary(r). The transmission is performed by transmitting a beep on channel ci when the bit bi = 1 for the binary string Binary(r). During the same phase, each process keeps listening for feedback on its selected group. The feedback from the subchannels of a group represents a binary string where a beep represents 1 and silence represents 0.

128

L. S. Aldawsari and T. Altman

Algorithm 2: Unknown Multichannel Multi Beeps Naming t←2 repeat /* algorithm main loop k ← 2t t←t∗2 index ← 0 is winner ← false collisionp ← false feedbackp ← ∅ namep ← ∅ yp ←random(1, k) rp ←random(1, 2t ) repeat /* assignment loop /* first phase if is winner = false then Send Binary Number(r, y) feedbackp = Get Feedback(y) if feedbackp = Binary(r) then is winner ← true /* second phase if is winner = true then Beep On Random Channel(y) for i = 1, k do if Feedback On Channel(1,i) = beep then if Check Collision In Group(i)= true then collisionp ← true else counterp ← counterp + 1 if is winner = false then yp ←random(1, k) rp ←random(1, 2t ) else namep ← Name By Winning Order(counterp ) /* iteration endedd /* end of segment until index = αk until collisionp = false

*/

*/ */

*/

*/ */

The feedback received from the subchannels of the group y is denoted as feedback y . A process designates itself as a winner when Binary(r) = feedback y . Otherwise, it again attempts to win a new group in the next iteration of the assignment loop.

Naming Processes in Multichannels with Beeps in the Weak Model

129

In the second phase, the winning process announces its winning status by transmitting a beep on Channel 1 of its selected group. All winning processes perform the collision-checking procedure. In this procedure, a process chooses a random channel cr from the range (1, log t). The winning process transmits a beep on channel cr of the subchannels of the group. All processes check the feedback from the subchannels and count the number of beeps, which is denoted as count beeps . A collision is detected when count beeps > 1, which invalidates the winning status of the winning processes. All processes set collision = true when such collision is detected. Otherwise, when no collisions are detected, the winning process assigns itself a name based on the order of the group y among all winning groups. In addition, the counter is incremented by 1 for each winning group. Thus, at the end of the phase, all processes set counter = counter + w, where w is the number of winning groups. At the end of the assignment loop, all processes conduct a termination condition check. They check whether a collision was found through the variable collision. When no collision is found such that collision = false, then all processes terminate, which marks the end of the UMMBN algorithm. Otherwise, if collision = true, it implies that a collision was detected. Thus, all processes rerun the UMMBN algorithm with larger groups of channels by setting t = t ∗ 2. The UMMBN algorithm is repeated until no collision is found in a single run. 5.1

Proof of Correctness and Analysis

The proof of correctness for the UMMBN algorithm is presented in this section. Because it is a Monte Carlo algorithm, it must be shown that the algorithm assigns unique names with a probability of success that is at least 12 . Theorem 2. The UMMBN algorithm assigns unique names to anonymous processes that has a time complexity of O(log n) rounds and a probability of success which is at least 1 − (n log n)−1 . Proof. In this algorithm, the processes do not know n, the number of processes. The processes run the UMMBN algorithm in a series of repeated stages where, in each stage, the number of groups k is incremented by multiplying t by a factor of 2. All processes begin with k groups of channels, where k = 2t and t = 2. The algorithm outputs are the names assigned to the processes. An error occurs in the output when two or more processes have the same name, which is called a duplicate name. The assignment of a duplicate name occurs when the collisionchecking procedure fails such that two or more processes choose the same group y with identical CodeWord r. The collision-checking procedure fails to detect a duplicate name when the winning processes choose to beep on the same channel with a probability of log1 t . The probability of this event is dependent on the number of groups k. Therefore, two scenarios must be considered in which either k < n or k ≥ n. For each of these scenarios, the probability of error is computed based on a collision event.

130

L. S. Aldawsari and T. Altman

We consider the first scenario when k < n. The probability of failing to identify a collision is maximized when n4 processes select the same group because t = 2 =⇒ k = 4. Therefore, the probability of error is computed as follows: 1 n 1 n 1 5n 1 1 n P r(error) = ( ) 4 ∗ ( ) 4 ∗ ( ) 4 = ( ) 4 < ( )n 4 4 2 2 2

(1)

Therefore, the probability of error is ( 12 )n , which is less than 12 . In the second scenario, we consider the case in which k ≥ n such that t = log n and n ≥ 4. The probability of error is the highest when exactly two processes select the same group, and their CodeWords are identical. Thus, the probability of error is computed as follows: 1 1 2 1 1 2 1 1 ) = ( )4 ∗ ( ) < P r(error) = ( )2 ∗ ( )2 ∗ ( n n log n n log n n log n

(2)

The UMMBN algorithm has a probability of success which is at least 1 − (n log n)−1 . Furthermore, the probability of termination increases in each stage, that implies that the algorithm eventually terminates. This concludes the correctness of assigning unique names to the processes. Next, we estimate the time complexity of the algorithm. The algorithm terminates when k = n, which is reached when the value t = log n. At the initial stage, Stage 1, the value t = 2, and it is multiplied by 2 for every new stage. Each stage takes 2αk rounds, so the time complexity is computed as follows: log n

αt = O(log n)

(3)

t=2

Therefore, the time complexity of the algorithm is O(log n) rounds. This time is optimal using the same proof as in Theorem 1 which states that Ω(log n) rounds is a lower bound.  

6

Conclusions

We proposed two novel naming algorithms for processes communicating with beeps through multiple channels in the weak model. The MMBN algorithm is a Las Vegas algorithm designed for known n, which has an optimal time complexity of O(log n). The UMMBN algorithm was designed for the model with unknown n. It has an optimal time complexity of O(log n) rounds with a probability of success at least 1 − (n log n)−1 . These algorithms solve the naming algorithm in multiple channels and have lower run time compared to existing algorithms. These algorithms can be useful for developing solutions to other related problems in distributed systems. This paper provides insight into exploiting multiple beeps in the weak model that can help design solutions to problems, such as leader election and counting. For future work, it will be interesting to determine whether a better performing algorithm can be developed that has an improved energy efficiency.

Naming Processes in Multichannels with Beeps in the Weak Model

131

Other distributed communication models are yet to be analyzed. These include the model with extended beeps that allows more than one signal type to be sent. Developing fault-tolerant naming algorithms for the noisy beeping model is another possible direction, where a faulty process can falsely detect or not detect a beep on the channel.

References 1. Afek, Y., Alon, N., Bar-Joseph, Z., Cornejo, A., Haeupler, B., Kuhn, F.: Beeping a maximal independent set. In: Peleg, D. (ed.) DISC 2011. LNCS, vol. 6950, pp. 32–50. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24100-0 3 2. Afek, Y., Alon, N., Barad, O., Hornstein, E., Barkai, N., Bar-Joseph, Z.: A biological solution to a fundamental distributed computing problem. Science 331(6014), 183–185 (2011) 3. Aldawsari, L.S., Altman, T.: Naming in multichannel with beeps in the strong model. Appl. Sci. 10(20), 7164 (2020) 4. Andriambolamalala, N.A., Ravelomanana, V.: Energy efficient naming in beeping networks. In: Palattella, M.R., Scanzio, S., Coleri Ergen, S. (eds.) ADHOC-NOW 2019. LNCS, vol. 11803, pp. 355–369. Springer, Cham (2019). https://doi.org/10. 1007/978-3-030-31831-4 25 5. Andriambolamalala, N.A., Ravelomanana, V.: Fault-tolerant energy efficient consensus on ad hoc beeping networks (2020) 6. Angluin, D.: Local and global properties in networks of processors (extended abstract). In: Proceedings of the Twelfth Annual ACM Symposium on Theory of Computing, STOC 1980, pp. 82–93. ACM (1980) 7. Beauquier, J., Burman, J., Davies, P., Dufoulon, F.: Optimal multi-broadcast with beeps using group testing. In: Censor-Hillel, K., Flammini, M. (eds.) SIROCCO 2019. LNCS, vol. 11639, pp. 66–80. Springer, Cham (2019). https://doi.org/10. 1007/978-3-030-24922-9 5 8. Bordim, J.L., Cui, J., Ishii, N., Nakano, K.: Doubly-logarithmic energy-efficient initialization protocols for single-hop radio networks. In: Proceedings of the 16th International Parallel and Distributed Processing Symposium, IPDPS 2002, USA, p. 123. IEEE Computer Society (2002) 9. Brandes, P., Kardas, M., Klonowski, M., Pajak,  D., Wattenhofer, R.: Approximating the size of a radio network in beeping model. In: Suomela, J. (ed.) SIROCCO 2016. LNCS, vol. 9988, pp. 358–373. Springer, Cham (2016). https://doi.org/10. 1007/978-3-319-48314-6 23 10. Buhrman, H., Panconesi, A., Silvestri, R., Vit´ anyi, P.M.B.: On the importance of having an identity or, is consensus really universal? Distrib. Comput. 18(3), 167–176 (2006) 11. Chlebus, B.S., De Marco, G., Talo, M.: Naming a channel with beeps. Fund. Inform. 153(3), 199–219 (2017) 12. Cornejo, A., Kuhn, F.: Deploying wireless networks with beeps. In: Lynch, N.A., Shvartsman, A.A. (eds.) DISC 2010. LNCS, vol. 6343, pp. 148–162. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15763-9 15 13. Czumaj, A., Davies, P.: Communicating with beeps. J. Parallel Distrib. Comput. 130, 98–109 (2019) 14. Dufoulon, F., Burman, J., Beauquier, J.: Beeping a deterministic time-optimal leader election. In: 32nd International Symposium on Distributed Computing (DISC 2018). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2018)

132

L. S. Aldawsari and T. Altman

15. Efremenko, K., Kol, G., Saxena, R.R.: Noisy beeps. In: Proceedings of the 39th Symposium on Principles of Distributed Computing, PODC 2020, pp. 418–427. Association for Computing Machinery, New York (2020) ¨ Singh, A.K.: Naming symmetric processes using shared variables. 16. Eˇ gecioˇ glu, O., Distrib. Comput. 8(1), 19–38 (1994) 17. F¨ orster, K.-T., Seidel, J., Wattenhofer, R.: Deterministic leader election in multihop beeping networks. In: Kuhn, F. (ed.) DISC 2014. LNCS, vol. 8784, pp. 212–226. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-45174-8 15 18. Kutten, S., Ostrovsky, R., Patt-Shamir, B.: The las-vegas processor identity problem (how and when to be unique). J. Algorithms 37(2), 468–494 (2000) 19. Lipton, R.J., Park, A.: The processor identity problem. Inf. Process. Lett. 36(2), 91–94 (1990) 20. Myoupo, J.F.: Dynamic initialization protocols for mobile ad hoc networks. In: The 11th IEEE International Conference on Networks, ICON 2003, pp. 149–154, October 2003 21. Nakano, K., Olariu, S.: Energy-efficient initialization protocols for radio networks with no collision detection. In: Proceedings 2000 International Conference on Parallel Processing, pp. 263–270, August 2000 22. Nakano, K., Olariu, S.: Randomized initialization protocols for ad hoc networks. IEEE Trans. Parallel Distrib. Syst. 11(7), 749–759 (2000) 23. Navlakha, S., Bar-Joseph, Z.: Distributed information processing in biological and computational systems. Commun. ACM 58(1), 94–102 (2014) 24. Panconesi, A., Papatriantafilou, M., Tsigas, P., Vit´ anyi, P.: Randomized naming using wait-free shared variables. Distrib. Comput. 11(3), 113–124 (1998) 25. Schneider, J., Wattenhofer, R.: What is the use of collision detection (in wireless networks)? In: Lynch, N.A., Shvartsman, A.A. (eds.) DISC 2010. LNCS, vol. 6343, pp. 133–147. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3642-15763-9 14 26. Teng, S.-H.: Space efficient processor identity protocol. Inf. Process. Lett. 34(3), 147–154 (1990) 27. Toh, C.-K.: Ad Hoc Mobile Wireless Networks Protocols and Systems. Prentice Hall PTR, Hoboken (2002) 28. Yao, A.C.: Probabilistic computations: toward a unified measure of complexity. In: 18th Annual Symposium on Foundations of Computer Science (SFCS 1977), pp. 222–227 (1977)

Distributed Evolution of Deep Autoencoders Jeff Hajewski1,2(B) , Suely Oliveira2 , and Xiaoyu Xing3 1

Salesforce, San Francisco, USA [email protected] 2 Department of Computer Science, University of Iowa, Iowa City, USA 3 Amazon, Washington, D.C., USA Abstract. Autoencoders have seen wide success in domains ranging from feature selection to information retrieval. Despite this success, designing an autoencoder for a given task remains a challenging undertaking due to the lack of firm intuition on how the backing neural network architectures of the encoder and decoder impact the overall performance of the autoencoder. In this work we present a distributed system that uses an efficient evolutionary algorithm to design a modular autoencoder. We demonstrate the effectiveness of this system on the tasks of manifold learning and image denoising. The system beats random search by nearly an order of magnitude on both tasks while achieving near linear horizontal scaling as additional worker nodes are added to the system.

Keywords: Distributed deep learning Evolutionary algorithms

1

· Neural architecture search ·

Introduction

Autoencoders are an unsupervised, deep learning technique used in a wide range of tasks such as information retrieval (e.g., image search), image restoration, machine translation, and feature selection. These applications are possible because the autoencoder learns to distill important information about the input into an intermediate representation. One of the challenges in moving between application domains (e.g., from manifold learning to image denoising) is that it is not clear how, or even if, one should change the neural network architecture of the autoencoder. Designing a neural network is difficult due to the time required to train the network and the lack of intuition as to how the various layer types and hyper-parameters will interact with each other. In this work, we automate the design process through the use of an efficient evolutionary algorithm coupled with a distributed system that overcomes the computational barrier of training and evaluating a large number of neural networks. Our system is designed to be robust to node failures, which are particularly common in neural architecture search, and offers elastic compute abilities, allowing the user to add or remove compute nodes without needing to pause or restart the search process. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 133–153, 2022. https://doi.org/10.1007/978-3-030-80119-9_6

134

J. Hajewski et al.

Elastic compute abilities are particularly important in the domain of neural architecture search due to the high computational demands. For example Zoph et al. [36] used 800 GPUs over the course of about a week during their architecture search of over 12,000 different deep neural networks. Similarly [22] used 250 GPUs for over 10 days and [16] used 200 GPUs for a day and a half. At these levels of compute nodes and experiment duration, node failure is not surprising. Additionally, the required level of computational resources may not be immediately apparent until later stages of the search where potentially more complex architectures are being explored; having to restart the experiment with additional compute nodes, rather than simply adding nodes while the experiment is running, is costly both in terms of research time and money. We apply this system to the domains of manifold learning [18, 28] and image denoising [32, 33], which are two common domains of application for autoencoders. We explore the effect of varying the number of epochs during the evolutionary search model evaluation, the scalability of the system, and the effectiveness of the search. The primary contributions of this work are: – An efficient and scalable evolutionary algorithm for neural architecture search applied to the evolution of deep autoencoders. – A distributed architecture for the efficient search and evaluation of neural networks. This architecture is robust to node failures and allows additional compute resources to be added to the system while it is running. – A demonstration of the effectiveness of both the search algorithm and the distributed system used to perform the search against random search. This demonstration is performed in the domains of manifold learning and image denoising. The rest of this work is organized as follows. We discuss related and prior work in Sect. 2. In Sect. 3 we give a brief overview of autoencoders and formally introduce the applications of manifold learning and denoising. We follow the background material with a description of our search algorithm along with the architecture and features of the distributed system we built to efficiently run our search algorithm. Section 5 contains a description of our experiments, followed by a discussion of the results, in Sect. 6.

2

Related Work

Neural architecture search has recently experienced a surge of interest, with a number of clever and effective techniques to find effective neural network architectures [3, 7,17,21, 36]. A number of recent works have explored the use of reinforcement learning to design network architectures [13, 21, 36]. Other recent work [11,19, 35] has used a similar strategy to exploring the search space of network architectures, but use evolutionary algorithms rather than reinforcement learning algorithms. The reinforcement learning and evolutionary-based approaches, while different in method of search, use the same technique of building modular networks—the search algorithm designs a smaller module of layers and this

Distributed Evolution of Deep Autoencoders

135

module is used to assemble a larger network architecture. Liu et al. [16] use this modular approach at multiple levels to create motifs within modules and within the network architecture itself. Despite the wealth of prior work on evolutionary approaches to neural architecture search, to the best of our knowledge there is relatively little work exploring the application of these techniques to autoencoders. Most similar to our work is that of Suganuma et al. [26], which uses an evolutionary algorithm to evolve autoencoder architectures. We use a parent population of 10 rather than the single parent approach used by [26]. We are able to do this efficiently for any number of parents by horizontally scaling our system. Perhaps the most striking difference between our work and Suganuma et al. is the improved efficiency we achieve by caching previously seen genotypes along with their fitness and thus reducing the computational load by avoid duplicate network evaluations. Additionally, our approach searches for network architectures asynchronously and on a much larger scale using a distributed system. The graphical approach used by Suganuma et al. allows them to evolve non-sequential networks while our work only considers sequential networks – this is a shortcoming of our approach. Lander and Shang [14] evolve autoencoders with a single hidden layer whose node count is determined via an evolutionary algorithm based on fitness proportionate selection. Rivera et al. [4] also propose and evolutionary algorithm to evolve autoencoders. Their work differs dramatically from both ours and that of Suganuma et al. in that they use a uniform layer type throughout the network. They also add a penalty to the fitness function that penalizes larger networks and thus adding a selective pressure to simpler network architectures. Sciuto et al. [25] argue that a random search policy outperforms many popular neural architecture search techniques. As our data shows, this is not the case, at least for neural architecture search as applied to finding autoencoder architectures for the tasks of manifold learning and image denoising. In fact, we were somewhat surprised at how poorly random search performed on the given tasks.

3

Autoencoders

Module x N Module x N

Reduction Layer

Module x N Dilation layer

Module x N

Input

Output

Fig. 1. Diagram of modular autoencoder.

136

J. Hajewski et al.

Autoencoders are a type of unsupervised learning technique consisting of two neural networks, an encoder network, denoted by ψe , and a decoder network, denoted by ψd , where the output of the encoder is fed into the decoder. The two networks are trained together such that the output of the decoder matches the input to the encoder. In other words, for an input x: x ˆ = ψd (ψe (x)) and the goal is for the difference between x and x ˆ to be as small as possible, as defined by the loss function. For a loss function L, the encoder and decoder (collectively referred to as the autoencoder) optimize the problem given by Eq. (1) Ψ (x; ωe , ωd ) = min L(x, ψd (ψe (x; ωe ); ωd )) ωe ,ωd

(1)

where Ψ (x; ω) represents the encoder-decoder pair Ψ (x; ω) = (ψe (x; ωe ), ψd (x; ωd )) and ωe and ωd are the weights of the encoder and decoder networks, respectively. Typically the encoder network is a mapping to a lower-dimensional space 

ψe : Rd → Rd

and the decoder network is a mapping from the lower-dimensional space to the original input space  ψd : Rd → Rd where d  d. The intuition behind this lower dimensional representation, or embedding, is that the autoencoder transforms the input into a representation that captures essential information and removes non-essential information. In practice, the size of this lower-dimensional representation, which we refer to as the intermediate representation, is a hyper-parameter of the autoencoder. The autoencoders in this paper are constructed from repetitions of layer modules separated by reduction (dilation) modules, as shown in Fig. 1. A module is an ordered collection of layers, separated by ReLU activations, as shown in Fig. 2. Layer modules do not change the spatial dimensions of their input through the use of zero padding. Reducing the spatial dimensions of the input is left to the reduction module. Maintaining spatial dimension within layer modules allows us to guarantee properly formed networks during the neural network construction phase. A reduction module consists of a 1 × 1 convolutional layer followed by a 2 × 2 max pooling layer with stride two. The 1 × 1 convolutional layer doubles the depth of the input layer’s filters, in an attempt to reduce the information loss resulting from the strided max pooling layer. The strided max pooling layer reduces each spatial dimension by a factor of two, resulting in an overall reduction in information of 75% (a quarter of the original spatial information is retained). Doubling the number of filters means the aggregate information loss is at least 50% rather than 75%. In the decoder network, reduction modules are replaced with dilation modules. The dilation modules replace the max pooling layer with

Distributed Evolution of Deep Autoencoders

137

a 2D convolutional transpose layer [34]. The effect of this layer is to expand the spatial dimension of its input. Modular network designs, where the neural network is composed by repeating a layer module a set number of times, are a common technique in neural architecture search [3,7, 11,17, 19,21, 35,36]. It can also be seen in many popular network architectures such as Inception-v3 [27] and ResNets [9]. One of the primary advantages of using a modular network architecture is that we are able to reduce the size of the search space. For example, consider a modular network where the module has a maximum size of five layers and is repeated twice and there are two reduction layers. This network structure will result in a maximum network of 30 layers; however, because the search is performed at the module level, we only search the space of five layer networks. In our work we consider four different convolutional layers with four different filter counts as well as a 2D dropout layer; this gives 17 different options for each layer of the module. Using a modular network architecture enables us to reduce the search space from ∼1085 to ∼1015 .

5x5 Conv2D 32 filters

ReLU

7x7 Conv2D 16 filters

ReLU

3x3 Conv2D 64 filters

ReLU

1x1 Conv2D 128 filters

Layer Module

2x2 Max Pooling

Reduction Module

Fig. 2. Example of a single layer module, followed by a reduction module.

3.1

Image Denoising

Image denoising solves the problem Ψ = min L(X, ψd (ψe (X + ; ωe ); ωd )) ωe ,ωd

(2)

where the loss function, L, is typically the mean squared error (MSE) loss shown in Eq. 3. We choose MSE in this setting because it is a more appropriate choice for regression. If you consider the input and output of the autoencoder is two separate functions, the goal of training is to make them as similar as possible. While we could use binary cross entropy from a technical point of view, we find it is better suited to classification problems, of which autoencoders are decidedly not. On the other hand, one could extend our algorithm to include the loss function as a parameter of the search, as done by Rivera et al. [4].  ||x − ψd (ψe (x + ))||2 (3) L(X, ψd (ψe (X + λ))) = x∈X

and  ∼ N (0, Σ) where Σ = σ 2 I and I is the identity matrix in Rd×d . We tune the difficulty of the denoising problem by modifying σ, larger σ leads to noisier

138

J. Hajewski et al.

images. The intuition of denoising is rather simple, learn weights that allow the decoder to remove noise from a given image. There are a number of applications of this technique, the most obvious application being image restoration. However, initial work on denoising autoencoders actually used this technique as a method of pre-training the neural networks and then fine-tuning the encoder to be used in a classification task. 3.2

Manifold Learning

Manifold learning [15, 18,28] is a form of non-linear dimensionality reduction. Manifold learning algorithms assume the data, X, is sampled from a low dimensional manifold M ⊂ Rd embedded in a higher dimensional space Rd , where d  d. Some of the more popular techniques are Isomap [29], locallylinear embeddings [24], and Laplacian eigenmaps [2]. Autoencoders are another technique for manifold learning. In this case, the encoder network ψe is trained such that for input X ⊂ Rd ψe : X ⊂ Rd → M and the decoder network is trained such that it is a mapping ψd : M → X ⊂ Rd This technique is useful in high-dimensional settings as a feature selection technique, similar to Principal Component Analysis (PCA) [1] except that PCA is a linear dimensionality reduction technique and manifold learning is non-linear. A major difference between linear and non-linear manifold learning algorithms is that the linear algorithms attempt to preserve global structure in the embedded data while the non-linear algorithms only attempt to preserve local structure. The primary challenge in manifold learning with autoencoders is two-fold: we must choose reasonably performant neural network architectures for the encoder and decoder networks and we must also choose an appropriate dimension d . In this work we consider two d values of d = (1/4)d and d = (1/8)d, which we manually set for each experiment. The architecture of the networks is found via evolutionary search. In practice, we would chose d as the smallest d value that minimized the reconstruction loss L(X, ψd (ψe (X))).

4

Evolving Deep Autoencoders

The two main approaches to neural architecture search are based on reinforcement learning or evolutionary algorithms. In this work we focus on the evolutionary approach, which consists of two primary components: evolution and selection. The evolution step is where new individuals (autoencoders in our case) are generated. The selection step is where the algorithm determines which individuals should survive and which individuals should be removed from the population. We use a generational selection mechanism where a population of network architectures goes through mutation, evaluation, and selection as a group. Specifically, we use (μ + λ) selection where a parent population of size μ generates λ offspring and we select the top μ of the μ + λ individuals.

Distributed Evolution of Deep Autoencoders

4.1

139

Network Construction

As discussed in Sect. 3, the autoencoders we consider in this paper consist of a layer module, followed by a reduction layer made up of a 1 × 1 convolution and a 2 × 2 max pooling layer—this structure may be optionally repeated, with each repetition reducing the dimension of the intermediate representation by 50%. We represent the module as a list of sequentially connected layer objects, encoded in a human-readable format. Each layer is defined by a layer token and a filter count, separated by a colon. Layers are separated by commas. For example, a module consisting of a 5 × 5 convolution layer with 16 filters and two 3 × 3 convolution layers with 32 filters would be represented: 5x5conv2d:16,3x3conv2d:32,3x3conv2d:32 This encoding is easy to work with in that it can be stored as a variable length array of strings, which allows us to handle layer mutation via indexing and addition of a layer by simply appending to the list. Of course, this could be condensed to a numerical coding scheme to reduce the size of the encoding. In our experience, using a more verbose encoding greatly simplified debugging and analyzing the results. Using a sequential network architecture simplifies the construction process of the autoencoder when compared to a wide architecture such as the Inception module [27], where the module’s input can feed into multiple layers. Networks are constructed by assembling 1×1, 2×2, 3×3, 5×4, and 7×7 2D convolutional layers as well as a 2D dropout layer [30] with dropout probability p = 0.5 We use convolutional layers, rather than dense layers, primarily because we use color image inputs in our experiments, which have three color channels for each pixel. Convolutional layers are useful for feature detection because they work on patches of pixels in the input. Additionally, they are an effective technique to reduce the total number of network parameters, resulting in smaller and faster training networks. Our algorithm can work with any layer or activation function types and it would be appropriate to consider other layer types in different problem settings. For example, in a feature selection setting with dense data vectors it would make more sense to consider dense layers with evolving node counts rather than different types of convolutional layers. Figure 3 illustrates a 3×3 convolutional layer with zero padding. We zero-pad all convolutional layers to maintain the spatial dimensions of the input, using the reduction(dilation) layers to modify spatial dimensions. Each layer can have 8, 16, 32, or 64 filters and the final output of the model is passed through a tanh activation function. The sequential approach also has a major advantage in that it is guaranteed to construct a valid network. Because each layer feeds into the next there will always be a path from input to output. This is not the case when allowing arbitrary connections between layers and letting evolution select these layers.

140

J. Hajewski et al. 0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

3x3 conv2D

0

Fig. 3. Illustration of a 2D 3 × 3 convolutional layer with 0 padding to maintain the spatial dimensions of the input.

Algorithm 1. Evolutionary Neural Architecture Search. 1: procedure EvolveNN(n, ngen ) 2: P ← new population of size n 3: for i = 0 to ngen do 4: Mutate(P ) 5: for p ∈ P do 6: if p.isEvaluated then 7: continue 8: else if p ∈ cache.keys() then 9: p.setFitness(cache.get(p)) 10: end if 11: t ← Task(p) 12: Send task t to broker 13: end for 14: Wait for |P | results from broker 15: Update cache with received fitness values 16: sort(P ) 17: P ← P [: n] 18: end for 19: end procedure

4.2

 Select top n individuals

Network Evolution

Network architectures are evolved, as described by Algorithm 1, by starting with a minimal layer module (e.g., a single convolutional layer) and either mutating the module or by performing crossover with another layer module. The specific algorithm used to evolve the population a single generation is described in Algorithm 2 while Algorithm 3 describes the specific algorithms used to mutate individual genotypes and perform crossover between two genotypes. Mutation works by either appending a layer or modifying an existing layer. Crossover between two modules involves taking two network architectures and splicing them together. All of this is performed by the model, which is described in greater detail in Sect. 4.3. The autoencoder is packaged into a network task Protocol Buffer and sent to the Broker, which forwards them to a worker. The workers perform the task of training and evaluating the autoencoder, where evaluation is performed on validation data that the network as not previously

Distributed Evolution of Deep Autoencoders

141

Algorithm 2. Population Evolution Algorithm. 1: procedure Mutate(P ) 2: P  ← P .clone() 3: for p ∈ P do 4: o ← RandomSample(P \ p) 5: u1 ← U (0, 1) 6: u2 ← U (0, 1) 7: if u1 < .5 then 8: p ← MutateGenotype(p) 9: end if 10: if u2 < .5 then 11: o ← MutateGenotype(o) 12: end if 13: c ← Crossover(p, o) 14: // Only add p and o to P  if they were mutated. 15: P  ← P  ∪ {p, o, c} 16: end for 17: return P  18: end procedure

 Mutated clone of p

 Mutated clone of o

Algorithm 3. Mutation and Crossover Algorithms used by the Genotypes. 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20:

procedure MutateGenotype(g) c ← g.clone() z ← SampleUniform(0, 1) if |c| < MaxNumLayersandz < 0.5 then AppendRandomLayer(c) else ReplaceRandomLayer(c) end if return c end procedure procedure Crossover(g1 , g2 ) i1 ← U (0, |g1 | i2 ← U (0, |g2 | c ← g1 .layers[: i1 ] + g2 .layers[i2 :] if |c| > MaxNumLayers then //Drop layers exceeding size limit. c ← c.layers[: MaxNumLayers] end if return c end procedure

 Random layer index for g1  Random layer index for g2

seen. The validation loss is packaged in a result Protocol Buffer and sent back to the Broker, who then forwards the result to the model. We define fitness as the reciprocal of the validation loss, which has the nice property that low loss results in high fitness.

142

J. Hajewski et al.

One notable aspect of our evolution algorithm is that we make it very likely one offspring is produced from each genotype in the parent population. We force crossover as long as the two selected genotypes have more than one layer each. This is similar to the forced mutation of Suganuma et al. [26]. The advantage is that we are constantly exploring new architectures; however, this approach also results in very large offspring populations that can be as large as 3× the size of the parent population. If the parents were mutated during crossover their mutated selves are added into the original population. This differs from a more classical evolutionary algorithm in that mutation and crossover are strongly encouraged and forced. This adds variance to the overall population fitness, which we counter by using (μ + λ) selection, making sure we always maintain the best performing network architectures. To cope with potentially large offspring population sizes, we make two modifications to the standard evolution algorithm to improve efficiency by reducing the total amount of work. These can be seen in lines 6 and 8 in Algorithm 1. In line 6, the model checks if the candidate autoencoder architecture (referred to as a genotype) has previously been evaluated. If the genotype was previously evaluated the model decides not to send it off for re-evaluation. This is possible because we make genotypes immutable—if a genotype is selected for mutation or crossover, a copy is made and mutated rather than modifying the original genotype. This modification alone can save anywhere from 25% (offspring population size 3× that of the parent population) to 50% (offspring population size 2× that of the parent population) because we use (μ + λ) selection, so the parent population is always included when evaluating the individuals of a population. Similarly, in line 8, we check if the architecture of the genotype as been seen previously. Caching previously seen architectures and their respective fitness is particularly important because of the stochastic nature of the evolutionary search algorithm. We noticed during the experiments, especially at later generations of evolution when the population has started to homogenize, that previously encountered architectures would be rediscovered. This is expected because evolutionary search stochastically explores the neighborhood of the current position, which means it may explore previously seen locations (architectures) simply due to chance. Let ψ(x; ω) represent a trained neural network with a fixed architecture. We denote training data as Xtr and validation data as Xval . We define an autoencoder Ψ = (ψe , ψd ) as a tuple containing an encoder network, ψe , and a decoder network, ψd . Formally, the problem we solve in this work is shown in Eq. (4), Ψ ∗ = arg min L(Xval , ψd (ψe (Xval ))) ψ∈A

(4)

The optimal autoencoder for the given problem is given by Ψ ∗ , where optimality is defined with respect to an architecture in the space of all neural network architectures, A. The loss function L is mean-squared error, as defined in Eq. (5), where X, Y ∈ Rn×d . n 1 ||x − y||2 (5) L(X, Y) = n i=1

Distributed Evolution of Deep Autoencoders

Nameserver

Model

Broker

Worker

Worker

143

To Broker

Worker

Worker

Fig. 4. Overview of the system architecture.

4.3

System Architecture

We designed a system to efficiently find and evaluate deep autoencoders using the evolutionary algorithm described in Algorithm 1. We use the system proposed in [8] and shown in Fig. 4, which consists of a model, one or more brokers, an arbitrary number of workers, and a nameserver. Flow of information through the system is rather simple – data flows from the model to the workers and then back to the model, all through the broker. Communication within the system is handled via remote procedure calls (RPC) using gRPC [6] and messages are serialized using Protocol Buffers [31]. The system infrastructure is written in Go while the model and worker implementations are written in Python and are available on GitHub1 . This is possible because of gRPC—we specify the system APIs in gRPC’s interface description language (IDL) and then implement them in Go and Python. The model and workers make use of the Python stubs and clients generated by gRPC from the API specification. The system moves the serialized Python objects from the model to the worker without having to worry about the contents of the messages. The advantage is that we can build the system infrastructure in a statically type-checked language such as Go while we can use Python and PyTorch [20] to build, train, and evaluate the autoencoder architectures. This is particularly important for the broker implementations; using Go gives the broker a high level of concurrency and performance when compared to Python. Heartbeat messages are RPCs used by a client to inform a server that the client is still functioning. This is important when a client requests a resource within the system, such as a worker requesting a task from a broker. The resource may be lost if the client fails and no heartbeat mechanism is used. The heartbeat message informs the server if the client has failed, allowing the server to properly handle the shared resources. Model. The model drives the evolutionary search for autoencoders by running the evolutionary algorithm that designs the layer modules. Once a population of layer modules has been generated (e.g., mutated or mated via crossover), the model packages the individual designs into tasks and sends them to the broker 1

https://github.com/j-haj/brokered-deep-learning.

144

J. Hajewski et al.

via RPC. The model then waits for the broker to send back the results from each of the layer module designs. Broker. The broker forms the data pipeline of the system by moving data between the model and the workers. Unevaluated network architectures received from the model are stored in a queue. These network architectures–referred to as tasks–are removed from the queue and sent to a worker when a task request RPC is received from the respective worker. At this point the task is moved to a pending-completion state until the finished result is returned from the worker, at which point the result is forwarded to the model. Brokers can connect (or link) with other brokers to share available compute resources. When connected with another broker, the broker stores shared tasks in a work queue, giving preference to its own tasks (referred to as owned tasks). Linked brokers are typically in different data centers because a single broker is capable of handling a large number of connected workers. Each network architecture evaluation takes on the order of minutes, giving the broker ample time to distribute tasks to other workers. Worker. The worker is the computational engine of the system. Workers register with a broker when they request a task from the broker but are otherwise able to join or leave the system at-will. A worker begins a heartbeat RPC with the broker from which it requested a task. This heartbeat RPC ends once the task is complete and returned from the broker. In this sense, workers lease tasks from the broker. As a result, if a worker leaves the system after returning a task, no other components in the system will know the worker left and there will be no state within the system waiting for additional information from the worker. Nameserver. The nameserver is a central store for the addresses of the brokers in the system. Models or workers can query the central store for a broker address prior to joining the system. This is not strictly required for the system, but improves the scalability of the system as more nodes are added. Rather than manually specifying the broker address for each worker, which may vary if there are many brokers, each worker can query the nameserver and get the address of a broker to connect to. Similarly, brokers can query the nameserver for the addresses of other brokers with whom they can form a link for work-sharing. As a result, the nameserver establishes a heartbeat with connected brokers. Brokers are removed from the nameserver’s list of available brokers when they fail to send a heartbeat within a specified time window. If the heartbeat is late, the nameserver will reply with a reconnect request to the broker, causing the broker to re-register with the nameserver.

Distributed Evolution of Deep Autoencoders

145

Table 1. Fitness of top 3 found architectures for image denoising, trained for 20 epochs. # of epochs during search Rank Architecture

5

Fitness

2

1 2 3

64-3x3conv2d 6.10 64-7x7conv2d 5.97 64-7x7conv2d - 64-3x3conv2d 5.86

5

1 2 3

64-5x5conv2d 6.02 64-7x7conv2d 5.97 64-7x7conv2d - 64-5x5conv2d 5.96

Random

–

–

1.8 ± 1.52

Experiments

We explore two areas of application in the experiments: manifold learning and image denoising. For all experiments we use the STL10 [5] dataset. We chose this dataset for its difficulty—the images are higher resolution than the CIFAR10/100 [12] datasets, allowing more reduction modules. Additionally, the higherresolution images are better suited for deeper, more complex autoencoder architectures. The layer module is restricted to at most ten layers. This gives a sufficiently large search space of around 10 billion architectures. We also compare the found architectures against random search. Over-fitting is a challenge with deep autoencoders, so we restrict our models to only one layer module, rather than repeating the layer module multiple times. In our initial experiments we found autoencoders with multiple layer modules performed dramatically worse (due to over-fitting) than their single layer module counterparts. The networks are implemented and trained using the PyTorch [20] framework. Evolutionary search is performed over 20 generations with a population size of 10 individuals. We use (μ + λ) selection as described in Algorithm 1. Random Search. The random search comparisons are performed by sampling from U [1, 10], and using the sampled integer as the number of layers in the layer module. Each layer is determined by sampling a random layer type and a random number of filters. The number of layer modules and the number of reductions are hyper-parameters and set to the same values as the comparison architectures found via evolutionary search. We sample and evaluate 30 random architectures for each experiment. Image Denoising. We used the image denoising experiments to test how well the evolutionary search algorithm performs when the candidate network architectures are trained for either two or five epochs. If the search can get away with fewer training epochs it will save resources, allowing the algorithm to explore more architectures. We set σ 2 = 1/3 in these experiments because we found that too much noise would cause the system to collapse and output blank images.

146

J. Hajewski et al.

Manifold Learning. In the manifold learning setting we focus on a restricted set of reduced dimensions, namely those dimensions that are reduced by a factor of (1/2)k for k = 2 and 3. This greatly simplifies the construction of both the encoder and decoder networks. Each reduction module, consisting of a 1 × 1 2D convolutional layer followed by a stride 2, 2 × 2 max pooling layer, reduces the each of the spatial dimensions by a factor of two. Thus the input dimension is reduced by 75% and 87.5%, respectively. This is a fairly drastic reduction in dimension and limits the total number of reduction modules used in a given autoencoder architecture.

Fig. 5. Average fitness over 20 generations.

6

Discussion

We present the results form our experiments along with a discussion of their implications. All experiments were performed on Amazon Web Services (AWS) using Nvidia K80 GPUs. Fitness is defined as the reciprocal of the loss, lower loss leads to greater fitness. We train the final architectures for 20 epochs in the image denoising experiments and 40 epochs in the manifold learning experiments. In practice, this is a very small number of epochs for a production model; however, we kept this number low as it did not affect the results and reduced costs. Random Search. Random search performed dramatically worse, on average, than evolutionary search on both the manifold learning and denoising tasks. This is unsurprising given the size of the search space, but differs from the claims of [25]. We train each random architecture for 20 epochs in the image denoising experiments and 40 epochs in the manifold learning experiments. Image Denoising. Table 1 summarizes the results from the denoising experiments. We list the top three network architectures for the 2-epoch and 5-epoch searches, along with their fitness values after 20 epochs of training. Interestingly,

Distributed Evolution of Deep Autoencoders

147

both the 2-epoch and 5-epoch approaches found similar architectures and, perhaps more surprisingly, their third place architectures were combinations of the second and first place architectures. It is unsurprising that both searches favored smaller architectures—the smaller epoch count means networks that train faster will have a better evaluation on the validation data. Smaller networks typically train faster than larger networks with respect to number of epochs, so it makes sense that these networks would rank higher during our searches using a limited number of epochs. On the other hand, over-fitting is an issue with the larger networks so favoring smaller layer module architectures had a positive impact on the overall performance of the autoencoder. Figure 5 shows the average fitness across 20 generations for both the 2-epoch and 5-epoch searches. Both graphs share two similar features: they plateau at the second generation and maintain that plateau until around the 18th generation, when they both find better architectures. Although the 5-epoch graph has an unsurprisingly higher fitness, it is surprising that both approaches appear to improve at about the same pace. This reinforces the idea that the 2-epoch search does a decent job at exploring the space. The un-noised input image is shown in Fig. 6a, the noised input images shown in Fig. 6b, and the denoised images shown in Fig. 6c.

Fig. 6. Denoised images on unseen data.

148

J. Hajewski et al.

Table 2. Fitness of top 3 found architectures for manifold learning, trained for 40 epochs. # of reduction modules Rank Architecture

Fitness

2

1 2 3 –

64-5x5conv2d 32-3x3conv2d 32-7x7conv2d Random

9.05 6.75 4.99 1.17 ± 1.22

3

1 2 3 –

64-3x3conv2d 3.42 64-5x5conv2d 3.38 32-3x3conv2d - 64-5x5conv2d 2.4 Random 0.36 ± 0.30 Speedup over a Single Worker

16 14

Speedup

12 10 8 6 4 2 0 2

4

6

8 10 12 Number of Workers Actual Ideal

14

16

Fig. 7. Scaling factors with respect to a single worker. These are calculated as the geometric average of the number of candidate architectures evaluated per generation, across five generations.

Manifold Learning. Table 2 summarizes the results from performing evolutionary search on autoencoders using two and three reduction modules compared to random search. The search was performed over 20 generations, as is done in the denoising experiments. Because these networks are larger, we train them for 40 epochs instead of the 20 epochs used in the denoising experiments. Similar to the denoising experiments, random search performs worse than evolutionary search in all scenarios. A single 5×5 2D convolutional layer as the layer module ranked highly in both the two and three reduction experiments at first and second place, respectively. As shown in Table 2, the 5x5conv2d layer with 64 filters performed nearly 50% better than the runner-up 3x3conv2d layer in the two reduction experiment. Interestingly, the 3x3conv2d and 5x5conv2d layers performed nearly identically

Distributed Evolution of Deep Autoencoders

149

Table 3. Autoencoder architecture for dropout example. Image effect Layer module architecture Greyscale

16-3x3conv2d – Dropout2D – 32-1x1conv2d – 2x2max-pool

Color

64-7x7conv2d – Dropout2D – 32-7x7conv2d – 2x2max-pool

Fig. 8. Comparison of best architectures found after 20 generations.

in the three reduction experiment with the 3x3conv2d layer taking a slight lead. Perhaps more notably, the 5x5conv2d architecture performed nearly an order of magnitude better than random search in both experiments. Figure 8c shows the results of the best autoencoder architectures found after 20 generations of search. Here we compare the resulting images of two and three reduction autoencoders. The reconstructed images using the two reduction autoencoder is shown in Fig. 8b while the reconstructed images using the three reduction autoencoder is shown in Fig. 8c. System Scalability. Fig. 7 shows the scalability of the system. We were only able to test the system with a maximum of 16 nodes due to resource constraints. The system shows near linear scaling—the curve falls off from linear not due to scalability issues with the system architecture but due to node failures. We would

150

J. Hajewski et al.

Fig. 9. Three layer module with a 2D dropout layer in the middle. Color information is preserved after 10 epochs of training.

occasionally experience GPU out-of-memory errors and this was more prevalent with larger node counts. Using an orchestration system such as Kubernetes [23] or Apache Mesos [10] would improve these scaling numbers as well as making the system more resilient. Impact of Dropout Layers Dropout layers have an interesting, if not somewhat unsurprising, effect on the output of the autoencoder. We give an example in Fig. 9 of two different networks, whose architectures are described in Table 3, after 10 epochs of training. The output of the first network is greyscale while the output the second network is color (although it does exhibit some greyscale properties on some images). We use a 2D dropout layer [30] to regularize the autoencoders—during our experiments we noticed it was fairly easy to overfit the data so we decided to add a dropout layer to the list of potential layers used by the evolution algorithm. Note that, as described in [30], the 2D dropout layer drops out an entire channel at random during training. Despite both networks having a 2D dropout layer in the same position, the network that produces color output has a greater learning capacity due to its larger parameter count. In

Distributed Evolution of Deep Autoencoders

151

other words, this network has more filters (and thus more parameters) and uses these additional filters to store additional learned features of the dataset. This is what appears to the characteristic that allows the network to maintain coloring in the images. Somewhat surprisingly given the ease with which we noticed the autoencoders would overfit the data, none of the best architectures utilized a dropout layer.

7

Conclusion

We have introduced an efficient and scalable system for neural architecture search and demonstrated its effectiveness in designing deep autoencoders for image denoising and manifold learning. An interesting avenue for future work would be to extend the evolutionary algorithm based on the observations of the denoising experiments—rather than search a fixed space for all generations, use the first few generations to find smaller architectures that work well. After these architectures are found, reduce the search space to combinations of these architectures. This is similar, in a sense, to the work of [16]. Other important future work is improving individual worker efficiency, allowing the system to achieve similar performance using fewer workers. Although the large number of workers enables fast search of network architectures, each worker consumes valuable resource (e.g., monetary, compute, environmental, etc.). This work is important in enabling self-improving and automatic machine learning.

References 1. Abdi, H., Williams, L.J.: Principal component analysis. WIREs Comput. Stat. 2(4), 433–459 (2010) 2. Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15(6), 1373–1396 (2003) 3. Brock, A., Lim, T., Ritchie, J.M., Weston, N.: SMASH: one-shot model architecture search through hypernetworks. CoRR arXiv:1708.05344 (2017) 4. Charte, F., Rivera, A.J., Mart´ınez, F., del Jesus, M.J.: Automating autoencoder architecture configuration: an evolutionary approach. In: International WorkConference on the Interplay Between Natural and Artificial Computation, pp. 339– 349. Springer, Cham (2019) 5. Coates, A., Ng, A.Y., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Gordon, G.J., Dunson, D.B., Dud´ık, M. (eds.) Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, 11–13 April 2011, volume 15 of JMLR Proceedings, pp. 215–223. JMLR.org (2011) 6. Google. grpc 7. Ha, D., Dai, A.M., Le, Q.V.: Hypernetworks. CoRR arXiv:1609.09106 (2016) 8. Hajewski, J., Oliveira, S.: A scalable system for neural architecture search. In: IEEE Computing and Communication Workshop and Conference, CCWC 2020 (2020) 9. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778. IEEE Computer Society (2016)

152

J. Hajewski et al.

10. Hindman, B., et al.: Mesos: a platform for fine-grained resource sharing in the data center. In: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation, NSDI 2011, pp. 295–308, Berkeley, CA, USA. USENIX Association (2011) 11. Koutn´ık, J., Cuccu, G., Schmidhuber, J., Gomez, F.: Evolving large-scale neural networks for vision-based reinforcement learning. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, GECCO 2013, pp. 1061–1068. ACM, New York (2013) 12. Krizhevsky, A., Nair, V., Hinton, G.: CIFAR-10 (Canadian Institute for Advanced Research) 13. Kyriakides, G., Margaritis, K.G.: Neural architecture search with synchronous advantage actor-critic methods and partial training. In: Proceedings of the 10th Hellenic Conference on Artificial Intelligence, SETN 2018, pp. 34:1–34:7. ACM, New York (2018) 14. Lander, S., Shang, Y.: EvoAE - a new evolutionary method for training autoencoders for deep learning networks. In: 2015 IEEE 39th Annual Computer Software and Applications Conference, vol. 2, pp. 790–795, July 2015 15. Lin, T., Zha, H.: Riemannian manifold learning. IEEE Trans. Pattern Anal. Mach. Intell. 30(5), 796–809 (2008) 16. Liu, H., Simonyan, K., Vinyals, O., Fernando, C., Kavukcuoglu, K.: Hierarchical representations for efficient architecture search. CoRR arXiv:1711.00436 (2017) 17. Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. CoRR arXiv:1806.09055 (2018) 18. Ma, Y., Fu, Y.: Manifold Learning Theory and Applications. CRC Press, Boca Raton (2011) 19. Miikkulainen, R., et al.: Evolving deep neural networks. CoRR arXiv:1703.00548 (2017) 20. Paszke, A., et al.: Automatic differentiation in PyTorch. In: NIPS Autodiff Workshop (2017) 21. Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture search via parameters sharing. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4095–4104, Stockholmsm¨ assan, Stockholm Sweden. PMLR, July 2018 22. Real, E., et al.: Large-scale evolution of image classifiers. In: Precup, D., Teh, Y.W. (eds.) Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6–11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2902–2911. PMLR (2017) 23. Rensin, D.K.: Kubernetes - scheduling the future at cloud scale (2015) 24. Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000) 25. Sciuto, C., Yu, K., Jaggi, M., Musat, C., Salzmann, M.: Evaluating the search phase of neural architecture search. CoRR arXiv:1902.08142 (2019) 26. Suganuma, M., Ozay, M., Okatani, T.: Exploiting the potential of standard convolutional autoencoders for image restoration by evolutionary search. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4771–4780, Stockholmsm¨ assan, Stockholm, Sweden. PMLR, July 2018 27. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. CoRR arXiv:1512.00567 (2015)

Distributed Evolution of Deep Autoencoders

153

28. Talwalkar, A., Kumar, S., Rowley, H.: Large-scale manifold learning. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008) 29. Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000) 30. Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: Efficient object localization using convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, 7–12 June 2015, pp. 648–656. IEEE Computer Society (2015) 31. Varda, K.: Protocol buffers: Google’s data interchange format. Technical report, Google, June 2008 32. Vincent, P., Larochelle, H., Bengio, Y., Manzagol, PA.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, ICML 2008, pp. 1096–1103. ACM, New York (2008) 33. Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.A.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11, 3371–3408 (2010) 34. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-31910590-1 53 35. Zhang, C., Ren, M., Urtasun, R.: Graph hypernetworks for neural architecture search. CoRR arXiv:1810.05749 (2018) 36. Zoph,B., Le, Q.V.: Neural architecture search with reinforcement learning (2017)

Computational Power of a Hybrid Algorithm for Solving the Multiple Knapsack Problem with Setup Samah Boukhari1 , Isma Dahmani2 , and Mhand Hifi3(B) 1

LaROMaD, USTHB, BP 32 El Alia, 16111 Alger, Alg´erie AMCD-RO, USTHB, BP 32 El Alia, 16111 Alger, Alg´erie EPROAD EA 4669, Universit´e de Picardie Jules Verne, 7, rue du Moulin Neuf, 80000 Amiens, France [email protected] 2

3

Abstract. The Multiple Knapsack Problem with Setups (MKPS) belongs to the well-known NP-hard knapsack family, which represents an extended version of the binary multiple knapsack problem. In this paper, we study MKPS in which a set of families of items and a set of knapsacks are available. Each item is characterized by a knapsack-dependent profit and each family is associated with a knapsack-dependent cost. An item can be selected only if the corresponding family is activated and a family can only be setup in one knapsack. A key feature is that the activation of a family incurs a knapsack-dependent setup cost that should be considered both in the objective function and constraints. The setup cost varies with the knapsack and the goal consists in selecting appropriate items, from different families, to enter a knapsack while maximizing its value with respecting its capacity. We first propose a hybrid algorithm that combines the solution related to the mixed integer linear relaxation and a series of knapsack problems. The mixed linear relaxation can be viewed as the driving problem, where it is solved by using a special black-box solver while the series of knapsack try to build the solutions provided by calling the state-of-the-art Cplex solver. We then propose an enhanced version of the first method, where a series of valid constraints are added for providing a powerful method. The performance of the proposed methods are evaluated on benchmark instances of the literature, where their provided results are compared to those reached by the stateof-the-art Cplex solver and the best methods available in the literature. The computational power of the proposed method shows the importance of hybridization, especially for that type of problems. Keywords: Knapsack

1

· Setups · Local branching · Relaxation

Introduction

The Knapsack Problem with Setup (KPS) can be viewed as a variant of the binary single Knapsack Problem (KP), where a set of items is considered which All authors are listed in alphabetical order. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 154–168, 2022. https://doi.org/10.1007/978-3-030-80119-9_7

Hybrid Methods

155

is divided into a set of classes. Each class is characterized by both fixed cost and fixed capacity while an item can be selected if the class containing that item is activated. KP arises in several situations, like cutting and packing (Hifi [10]), cryptography (Merkle and Hellman [14]), logistics (Perboli et al. [16]), railway stations, telecommunications and others. The Multiple Knapsack Problem with Setup (MKPS) can be viewed as a more complex version of KPS, where almost of one knapsack, there are T knapsacks and the setup cost is related to its corresponding knapsack constraint. These types of problems can be encountered in many real-world industrial and financial applications, such as order acceptance and production scheduling, aviation security system, resource allocation and transportation (for more details, the reader can refer to Chebil and Khemakhem [6], Khemakhem and Chebil [11], Lahyani et al. [12]) and, McLay [13]. MKPS can be encountered in several real-world applications, like the shaping related to the manufacturing process (as described in Lahyani et al. [12]). Indeed, on the one hand, in order to make one product family after another, the production machines must be scheduled and its corresponding production materials must be replaced for an adequate moulding. In this case, the company needs to choose the best order that is able to provide a better profit; that is a KPS’s instance. On the other hand, if demands can be distributed in different periods while at least a product family should respect a specific period, the cost would depend on the completion time related to the demand. Therefore, a cost may be attributed to a request delivered over a specific period requested by the customer and penalties may be assigned for postponing or delaying delivery. Moreover, the company’s production schedule should be planned for reaching a maximum total profit such that the company should take into account the capacity of the production machinery, the profit related to the different products, and the cost related to each family at considered period. In this case, we have a MKPS’s instance. Generally, a instance of MKPS is characterized by a set of T knapsacks of different capacities rt and a set of m disjoint families (classes) of items. Each family i, i ∈ {1, . . . , m}, consists of ni items and is characterized by a knapsackdependent integer setup cost fit and an integer parameter si representing its capacity consumption. Each item j, j ∈ {1, ..., ni }, of a family i is associated with a knapsack-dependent profit pijt and a capacity consumption wij . An item can be selected only if the corresponding class is activated and a class can only be setup in one knapsack. Furthermore, activating a class induces a knapsackdependent setup cost that should be considered both in the objective function and related constraints. The goal of the problem is to maximize selected items from different disjoint classes without violating the capacity constraints.

2

Related Works

Guignard [9] tackled the setup knapsack by using a Lagrangean decomposition for the setup knapsack problem, where no restrictions on the non-negativity of both setup cost of each class and the profit of each item are considered.

156

S. Boukhari et al.

A special case of KSP has been studied by Akinc [2] and Altay et al. [3], where only the setup cost of each class is taken into account (called the fixed charge knapsack problem). In Akinc [2], an exact algorithm has been designed, which is based upon a classical branch-and-bound scheme, while in Altay et al. [3] the authors tackled the case where items can be fractionated by cross decomposition. Michel et al. [15] addressed an extended multiple-class integer knapsack problem with setup. For that case, the weights associated to items are related to the classes and the total weight is bounded by both lower and upper weight bounds. Different integer linear programming formulations were proposed and an extended branch-and-bound algorithm, that is based upon Horowitz and Sahni method, was proposed such that nonnegative setup costs were favored. The knapsack problem with setup has been studied by Chebil and Khemakhem [6] who proposed a dynamic programming procedure, within pseudopolynomial time complexity. A special converting formulation was considered in order to reduce the size of the storage capacity, which remains quite expensive when using such type of approach. Khemakhem and Chebil [11] designed a special truncated tree-search for approximately solving KPS. The method applies an avoid duplication technic that consists in reformulating the original problem into a particular integer program. The experimental part showed the effectiveness of that method, especially the effect of the avoiding duplication technic in terms of improving the quality of the provided solutions. Furini et al. [8] developed linear-time algorithms for optimally solving the continuous relaxation of different integer linear programming formulations of the knapsack with setup. As mentioned in their experimental part, it has been shown that their algorithms outperform both dynamic programming-based approach and blackbox solver. Della et al. [7] designed an exact method which handles the structure of the formal description of KSP, where the search process explored the partitioning strategy that is based on splitting the decision variables into two levels. A fixation strategy has been applied for reducing the current subproblem to solve while the blackbox solver is applied for solving the reduced subproblem. The experimental part showed that method remains competitive when compared to Chebil and Khemakhem’s [6] dynamic programming method. Amiri [4] developed a Lagrangean relaxation-based heuristic for solving largescale knapsack problem with setup. His method follows the standard adaptation of the Lagrangean relaxation, where a series of local optima are provided by using a descent method and transformed into feasible solutions. The behavior of the method was evaluated on both the standard set of benchmark instances and very large-scale ones (containing at most 500 classes and two million of items) and its achieved results were compared to those achieved by the best available method in the literature. Boukhari et al. [5] designed a spacial local branching-based heuristic for the knapsack problem with setup. Their method hybridizes both mixed linear relaxation and local branching. The mixed linear relaxation was solved by calling a

Hybrid Methods

157

special black-box solver while the local branching tried to intensify each solution at hand by adding a series of local-branching constraints. The performance of that method was evaluated on benchmark instances of the literature and new large-scale ones, where its achieved results were compared to those reached by the Cplex solver and the best methods available in the literature. Regarding the MKPS, Yanchun [17] designed a branch-and-bound procedure for tackling the problem withe three knapsack constraints. Their method applies the well-known greedy procedure for providing a starting solution (lower bound) of the search process while a linear relaxation formulation was used for bounding the search with an upper bound. The experimental part showed that their method had a good behavior, especially several problem instances. Lahyania et al. [12] proposed a matheuristic for tackling large-scale MKPS’ instances. The principe the aforementioned method consists of two stages: (i) reducing the original problem MKPS into a series of problems, where each class contains one item, (ii) the linear relaxation of the problems built were solved and a current feasible solution is collected. In order to enhance the quality of the solution at hand, a tabu strategy-based matheuristic was designed. The computational results showed that the proposed method remains competitive when their provided results were compared to those achieved by the best methods available in the literature. Recently, Adouani et al. [1] developed another matheuristic, where both variable neighborhood descent and integer linear programming cooperate for tackling large-scale instances. The local search-based procedure was applied for assigning classes to knapsacks while the integer programming-based procedure was used for selecting the items in each knapsack. The computational part studied the behavior of their method on randomly generated instances, where their provided results were compared to those achieved by the state-of-the-art Cplex solver. The remainder of the paper is organized as follows. First, a formal description of the multiple knapsack problem with setup is presented in Sect. 3.1. Second, Sect. 3.2 describes the hybrid method that is based on combining the solution extracted from the linear relaxation of the original problem with the solution related to a series of knapsack. Section 3.3 discusses an enhanced version of the hybrid method, where a series of local valid constraints are added for tightening the upper bound of the original problem. Section 4 exposes the experimental part, where the computational power of the proposed method is evaluated on several sets of benchmark instances containing medium-sized and large-scale instances, taken from the literature. Thus, the provided results are compared to the best (average) lower bounds published in the literature. Finally, Sect. 5 concludes the paper.

158

3 3.1

S. Boukhari et al.

Hybrid Methods for PM K P S The Model

First, let xijt equal to 1 if item j of family(class) i is placed in knapsack t, and equal to 0 otherwise. Setup binary variables yit equal to 1 if family i is placed in knapsack t, and equal to 0 otherwise. Thus, the formal description of MSKP (noted PM KP S ) can be described as follows: PM KP S : max

ni m  T  

pijt xijt −

t=1 i=1 j=1

s.t.

ni m  

(1)

fit yit

t=1 i=1

wij xijt +

i=1 j=1

m 

si yit ≤ rt

(2)

∀t ∈ {1, ..., T }

i=1

xijt ≤ yit T 

m T  

(3)

∀ i ∈ {1, ...m}, ∀ j ∈ {1, ..., ni }, ∀t ∈ {1, ..., T }

yit ≤ 1

(4)

∀ i ∈ {1, ...m}

t=1

xijt ∈ {0, 1}, yit ∈ {0, }, ∀ j ∈ {1, ..., ni }, ∀ i ∈ {1, ...m}, ∀t ∈ {1, ..., T },

(5)

where the objective function (1) maximizes the profit of selected items minus the fixed setup costs of selected families(classes). Constraints (2) ensure that the weight of selected items in the knapsack t, including all setup capacities related to the activated classes, does not exceed the knapsack capacity rt . Constraints (3) ensure that an item j is selected in the knapsack t iff it belongs to a family i with a setup in the knapsack t. Finally, constraints (4) indicate that any family i is setup at most in one knapsack. 3.2

A Hybrid Method

A starting configuration for MKPS can be provided by solving the linear relaxation of the original problem PM KP S . Let RPM KP S denote the problem provided by relaxing all binary variables, i.e., setting xijt ∈ [0, 1], ∀ j = 1, ..., ni , ∀ i ∈ {1, ...m}, ∀ t ∈ {1, ...T } such that RPM KP S : max

ni m  T  

pijt xijt −

t=1 i=1 j=1

s.t.

ni m  

xijt ≤ yit T 

yit ≤ 1

fit yit

(6)

t=1 i=1

wij xijt +

i=1 j=1

m T  

m 

si yit ≤ rt

∀t ∈ {1, ..., T }

(7)

i=1

∀ i ∈ {1, ...m}, ∀ j ∈ {1, ..., ni }, ∀t ∈ {1, ..., T }. ∀ i ∈ {1, ...m}.

(8) (9)

t=1

xijt ∈ [0, 1], yit ∈ {0, 1}, ∀ j ∈ {1, ..., ni }, ∀ i ∈ {1, ...m}, ∀t ∈ {1, ..., T }.

(10)

Hybrid Methods

159

To build a constructive approximate solution, we first proceed by fixing step by step the variables yit to their binary values in PRM KP S . Second, let α be the number of variables yit fixed to one; thus, let α knapsacks problems whose variables xijt correspond to the classes yit = 1. Then the optimal solution of the series of knapsack achieve a complete feasible solution for MKPS. The procedure used, for providing such a solution, can be described as follows: – The relaxation RPM KP S is optimized by using the simplex method. Then, we collect the current primal solution and let (X, Y ) be the provided configuration. – Let S denote the set of indices of classes fixed to one such that S = {i | yit = 1} and, K denote the set of the knapsack’s index fixed to one; that is,  K = {t | yit = 1}, |K| = α and rh = rh − i∈S si , h ∈ K. – Let PKPh , h = 1, . . . , α, be the knapsacks with capacity rh and the set of elements xijh , i ∈ S. Each element j belonging to the class i, i ∈ S, is characterized by both profit pijh and weight wij . Thus, PKPh can be formally described as follows: PKPh : max s.t.

ni 

pijh xijh

i∈S j=1 ni 

wij xijh ≤ rh

i∈S j=1

xijh ∈ {0, 1}, ∀ j = 1, ..., ni , ∀ i ∈ S. – Let Xh be the optimal solution of PKPh , h = 1, . . . , α. Then, (X  , Y  ) denotes the (starting) feasible solution of M KP S such that (X  , Y  ) =    ∪h∈K Xh , Y = Y . Algorithm 1 describes the main steps of the aforementioned approach; that is used (i) for providing a starting configuration and, (ii) a complete solution of the proposed method, as shown in the rest of the paper.

Algorithm 1. First Hybrid Constructive Method for MKPS Solve RPM KP S with the simplex method and let (X, Y ) be its optimal solution. Let Y  be the solution extracted from Y , such as Set S = {i ∈ I  | yi = 1} and K = {t | yit = 1}. Let rh = rh − i∈S si , h ∈ K. Let PKPh , h = 1, ...|K|, be the knapsack problems with capacity rh and Xh , h = 1, . . . , |K|, be the achieved optimal solution. 6: Set X  = ∪h∈K Xh and Y  = Y . 7: return (X  , Y  ). 1: 2: 3: 4: 5:

160

3.3

S. Boukhari et al.

Enhancing the Hybrid Method

Herein, we propose to enhance the hybrid method by including a series of valid constraints to the original problem PM KP S . Let RPM KP S be the following linear relaxation program provided by relaxing all variables xijt , i.e., xijt ∈ [0, 1]: RPM KP S : max

s.t.

ni m  T   t=1 i=1 j=1

ni m   i=1 j=1

wij xijt +

xijt ≤ yit T  t=1

pijt xijt − m  i=1

m T   t=1 i=1

si yit ≤ rt

∀ i ∈ {1, ...m},

yit ≤ 1

(11)

fit yit

∀t ∈ {1, ..., T }

∀ j ∈ {1, ..., ni },

∀t ∈ {1, ..., T }.

∀ i ∈ {1, ...m}.

xijt ∈ [0, 1], yit ∈ [0, 1], ∀ j ∈ {1, ..., ni }, ∀ i ∈ {1, ...m}, ∀t ∈ {1, ..., T }.

(12) (13) (14) (15)

In order to build an approximate solution for M KP S, we fix all variables yit whose values are integrals and to one the variables having a fractional value in the optimal solution related to RPM KP S . Let dt denote the number of yit with nonnegative values in the tth knapsack. Thus, the following constraint remains valid for the original problem: m 

yit ≤ dt , ∀t ∈ {1, ..., T }.

(16)

i=1

In fact, the constraint (16) is related to each knapsack t, where the number of classes fixed in the solution must not exceed dt . Let RPM KP S−V be the model built by combining RPM KP S (as defined above) and the valid constraints of type (16). Thus, 1. First, fix all variables yit with integral values in the optimal solution of RPM KP S−V . 2. Second, let β be the number of variables yit fixed to one, where β knapsack problems can be represented by their variables xijt ; they are related to the classes yit = 1. 3. Third and last, the optimal solution of KPs induces a completed solution for the original problem. The main principle of the above approach can be described as follows: – We optimize the linear relaxation RPM KP S by using the simplex method. Then, we collect the current primal solution and, let (X 0 , Y 0 ) be the provided solution. – Let D be the set related to the values dt ; that is associated to the number of 0 nonnegative values in the tth knapsack. yit

Hybrid Methods

161

– Let RPM KP S−V be the new model obtained by combining RPM KP S and the new additional constraints (16) such that RPM KP S−V : max

s.t.

ni T  m   t=1 i=1 j=1

ni m   i=1 j=1

wij xijt +

xijt ≤ yit T  t=1 m  i=1

pijt xijt − m  i=1

T  m  t=1 i=1

si yit ≤ rt

∀ i ∈ {1, ...m},

yit ≤ 1

yit ≤ dt

(17)

fit yit

∀t ∈ {1, ..., T }

∀ j ∈ {1, ..., ni },

∀t ∈ {1, ..., T }.

∀ i ∈ {1, ...m}.

∀t ∈ {1, ..., T }

xijt ∈ [0, 1], yit ∈ {0, 1}, ∀ j ∈ {1, ..., ni }, ∀ i ∈ {1, ...m}, ∀t ∈ {1, ..., T }.

(18) (19) (20) (21) (22)

– We optimize the linear relaxation RPM KP S−V by using the simplex method.  Y ) denote the Then, we collect the current primal solution and let (X, obtained solution. – Let S  be the set of indices of classes i fixed to one such that S  = {i | yit = 1} to one such that K  = {t | yit = and, K  be the set of indices of knapsacks fixed    1}, |K | = β, and rh = rh − i∈S si , h ∈ K . – Define PKPh , h = 1, ..., β, as the series of knapsack problems with capacities rh and the set of elements xijh , i ∈ S. Each element j of the i-th class, i ∈ S, is characterized by both profit pijh and weight wij . Thus, PKPh may be described as follows: PKPh : max s.t.

ni 

pijh xijh

i∈S  j=1 ni 

wij xijh ≤ rh

i∈S  j=1

xijh ∈ {0, 1}, ∀ j = 1, ..., ni , ∀ i ∈ S  . – Consider Xh as the optimal solution of PKPh , h = 1, ...β. Then, the complete solution of M KP S is represented by the vector (X ∗ , Y ∗ ) such that X ∗ = ∪h∈K  Xh and Y ∗ = Y . Algorithm 2 describes the main steps of the enhanced hybrid approach, where the valid constraints are added to the original problem.

162

S. Boukhari et al.

Algorithm 2. Enhancing the Hybrid Method 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12:

4

Solve RPM KP S with the simplex method and let (X 0 , Y 0 ) be its optimal solution. Let Y  bethe solution extracted from Y 0 such that if yi0 > 0 then Set yi = 1 else Set yi = 0 end if m      Y ) be the optimal solution of RPM KP S−V . and, let (X, yit Set D = dt | dt = i=1



 Set S  = i | yit = 1 , K  = t | yit = 1 and, rh = rh − i∈S si , h ∈ K  .  Let P LKPh , h = 1, . . . , |K |, be the knapsack of capacity rh and Xh (h = 1, ...|K  |) be an optimal solution. Set X ∗ = ∪h∈K  Xh and Y ∗ = Y . return (X ∗ , Y ∗ ).

Experimental Part

The objective of the computational investigation is to assess the performance of the Hybrid Method with and without adding the valid constraints and, by comparing their provided bounds (objective values) to the best bounds available in the literature. Because two versions are considered, we then noted the fist version as HM (Hybrid Method) and EHM (Enhanced Hybrid Method) for the second version employing a series of valid constraints. Both versions of the proposed method are evaluated on the set containing 360 instances extracted from Chebil et al. [12], where it represents groups of ten instances for each combination T, m and, ni , respectively. For these instances, the total number of knapsacks T varies in the discrete interval {5, 10, 15, 20}, the number of classes m belongs to the interval {10, 20, 30} while the number of items in each class varies in the interval [40, 60], [60, 90] and [90, 100]. We note that these instances are considered as the strongly correlated ones, where they are randomly generated applying the following generator: – Both profits and weights were generated as follows: wij is randomly generated in [10, 10000] and pijt = wij + e1 , where e1 is uniformly distributed in [0, 10]. – The setup cost (resp. capacity) of each class i is a random number correlated to the sum of profits (resp. weights)  of the items in the considered class: the m ni wij ). fit = −e1 × i=1 pijt (resp. si = −e1 × i=1 m ni wij )+ – The knapsack capacity was setting equal to rt = 0.5×m×( i=1 j=1   m n i w ij e2 , where e2 is uniformly distributed in 10, i=1 mj=1 . Note that all proposed solution procedures were coded in C and performed on a computer with an Intel Pentium Core i3 with 2 GHz.

Hybrid Methods

163

Table 1. Behavior of the hybrid approach with and without adding the valid constraints: ten instances of the group with objects in [40, 60]. #Inst. T m

Hybrid method LBHM tHM

Enhanced hybrid method tGap LBEHM tEHM

5 5 5

10 20 30

337066.4 258447.7 421243.0

2.57 3.63 4.64

337066.4 258447.8 421243.3

2.91 3.78 4.04

0.34 0.15 −0.60

10 10 10 20 10 30

563000.4 457902.8 715892.9

6.45 27.36 30.67

563000.4 457902.8 715892.9

4.31 7.78 10.92

−2.14 −19.58 −19.76

15 10 15 20 15 30

691190.4 33.90 691190.5 9.33 658556.5 101.50 658556.5 26.92 1047318.3 202.27 1047318.7 28.88

−24.58 −74.58 −173.39

20 10 20 20 20 30

812956.9 43.58 812956.9 12.35 774283.7 152.90 774283.8 60.13 1359228.6 361.75 1359228.7 92.84

−31.22 −92.77 −268.90

Average 674757.32 80.94

4.1

674757.39 22.02

−58.92

Effect of the Valid Constraints

Herein, we evaluate the effect of the valid constraints when added to the first version of the hybrid method (as exposed in Sect. 3.3). Table 1 (resp. Tables 2 and 3) reports the results achieved by both HM and EHM on all tested instances: columns 1 and 2 show the instance information (the number of knapsacks T and, the number of classes m), the average lower bound LBHM (displayed in column 3 over the ten instances of the corresponding group) achieved by HM and the runtime tHM needed by HM (displayed in column 4) for reaching the final bound, both values (the lower bound LBEHM and the runtime tEHM ) related to EHM, reported in columns 5 and 6 and, the gap tGap between the average runtime between both versions (the negative value means that EHM performs better than HM and, the value of bold-space means that the method performance better than the other one). In what follows, we comment on the results of Tables 1, 2 and 3: 1. From Table 1 (the group of instance with the number of items belongs to [40, 60]), one can observe that EHM performs better than HM. Indeed, one the one hand, it is able to provide an global average bound of 674757.39, which is better than that achieved by HM, i.e., 674757.32. On the other hand, adding the valid constraints induces a powerful method since the average runtime limit is considerably decreased (the gap between both versions is equal to −58.92). Finally, more the size of the instance increases, more the average gap (tGap ) increases.

164

S. Boukhari et al.

2. From Table 2 (the group of instance with the number of items belongs to [60, 90]), we can observe that both versions of the method match the same average lower bounds while the average runtime is much smaller for the enhanced version EHM; in this case, tEHM = 48.35 s while tHM = 501.24 s, which means that EHM is 10.37 faster than HM. 3. From Table 3 (the group of instance with the number of items belongs to [90, 100]), we can notice that for large-scale instances, EHM definitely performs better than the initial version. Indeed, on the one hand, EHM provides better average bounds in 10 occasions over the twelve tests (we recall that each line of the table represents ten instances), that represents 83.33% of the groups. On the other hand, more the instances are complex to solve, more EHM surpasses the initial version. In this case, tEHM varies from 4.89 s (line 1, column 6) to 838.49 sec (line 12, column 6) while tHM varies from 6.24 s (line 1, column 4) to 3600 s (line 12, column 4)

Table 2. Behavior of the hybrid approach with and without adding the valid constraints: ten instances of the group with objects in [60, 90]. #Inst. T m

Hybrid method LBHM tHM

Enhanced hybrid method tGap LBEHM tEHM

5 5 5

766841.8 871105.1 808314.8

6.83 8.13 11.23

766841.8 871105.1 808314.8

3.33 4.02 4.56

−3.50 −4.11 −6.68

10 10 10 20 10 30

1480401.9 1713720.2 1551705.6

40.19 22.51 48.68

1480401.9 1713720.2 1551705.6

8.60 12.26 18.56

−31.59 −10.25 −30.12

15 10 15 20 15 30

1818395.9 2456682.6 2302192.5

68.40 1818395.9 958.99 2456682.6 1086.89 2302192.5

18.44 48.34 18.44

−49.97 −910.64 −1068.46

20 10 20 20 20 30

1945856.9 3055665.7 2922242.6

77.86 1945856.9 288.87 3055665.7 3396.34 2922242.6

23.40 226.87 193.45

−54.46 −61.99 −3202.89

10 20 30

Average 1807760.47 501.24

4.2

1807760.47 48.35

−452.89

Behavior of EHM Versus Other Available Methods

The results provided by EHM are compared to those reported in Adouani et al. [1] (VND&IP: VND combined with IP), in Lahyani et al. [12] (a two phase matheuristic, noted MTS) and the state-of-the-art Cplex solver. Tables 4 and 6 report the solution values achieved by the Cplex solver, MTS, VND&IP and the proposed algorithm EHM on two groups of instances (objets

Hybrid Methods

165

Table 3. Behavior of the hybrid approach with and without adding the valid constraints: ten instances of the group with objects in [90, 100]. #Inst. T m

Hybrid method LBHM tHM

Enhanced hybrid method tGap LBEHM tEHM

5 5 5

550191.7 493256.0 559080.4

6.24 10.37 22.67

550191.9 493256.0 559080.5

4.89 8.86 9.10

−1.35 −1.51 −13.57

10 10 10 20 10 30

1029033.8 938596.4 1086842.3

34.56 227.22 215.09

1029034.0 938596.5 1086842.3

7.70 13.04 29.78

−26.87 −214.17 −185.31

15 10 15 20 15 30

1370361.9 1251095.4 1613249.8

80.78 1370362.1 2059.53 1251095.5 2191.55 1613676.9

28.43 35.14 65.687

−52.35 −2024.39 −2125.86

20 10 20 20 20 30

1616358.4 1686890.0 2144848.6

110.79 1616358.5 1933.36 1686950.3 3600 2146707.6

28.3 243.64 838.49

−82.76 −1689.72 −2761.51

10 20 30

Average 1194983.73 874.35

1195179.34 109.40

−764.95

Table 4. Computational power of EHM versus MTS, VND&IP and the Cplex solver on instances of the group with objects in [40, 60]. #Inst.

Cplex solver

MTS

T

m

LBCplex

tCplex

LBMTS

tMTS

GapMTS

VND& IP

EHM

5

10

337066

2253

337066.4

25.7

−0.000119 337066.4

1.42

20

258448

2965

258447.8

29.8

0.000077 258447.8

30

421243

1731

421243.3

28.6

−0.000071 421243.3

LBVND&IP tVND&IP GapVND&IP LBEHM

tEHM GapEHM

−0.000119

337066.4

2.91

1.57

0.000077

258447.8

3.78

0.000077

1.54

−0.000071

421243.3

4.04

−0.000071

−0.000119

10 10

562993

3204

563000.4

37.7

−0.001314 563000.4

2.86

−0.001314

563000.4

4.31

−0.001314

15 20

457898

2902

457902.8

73.9

−0.001048 457903

4.63

−0.001092

457902.8

7.78

−0.001048

30

715887

3600

715893.1

84.9

−0.000852 715893.1

3.28

−0.000852

715892.9

10.92

−0.000824

10

691178

2967

691190.5

60.0

−0.001809 691190.5

4.59

−0.001809

691190.5

9.33

−0.001809

20

658534

3210

658556.5

115.2 −0.003417 658548.3

6.66

−0.002171

658556.5

26.92

−0.003417

1046226

3600

1047317.9 113.9 −0.104366 1047317.2

7.59

−0.104299

1047318.7 28.88

−0.104442

10

812934

2991

812956.9

111.3 −0.002817 812955.2

4.71

−0.002608

812956.9

12.35

−0.002817

20

774016

2380

774283.8

147.3 −0.034599 774283.8

5.56

−0.034599

774283.8

60.13

−0.034599

30

1358070

3487

1359224.1 150.5 −0.084981 1359224.8

7.26

−0.085032

1359228.7 92.84

−0.085320

Average 674541.08 2940.83 674756.96 81.57 −0.032003 674756.15

4.30

−0.031883

674757.39 22.02

−0.032067

20 30

varying in the interval [40, 60] and [90, 100]) while Table 5 shows the behavior of Cplex, VND&IP and EHM on the group of instances tested in Adouani et al. [1] (objects belonging to the interval [60, 90]). Columns 1 and 2 of Tables 4 and 6 show the instance’s label, columns 3 and 4 reports the Cplex solver’s integer bound (LBCplex ) and the runtime consumed (tCplex ). Columns from 5 to 7 tally the best lower bound reached by MTS (LBMTS ) extracted from [12], the runtime consumed (tMTS ) and the gap between the LBMTS and that achieved by the Cplex solver (LBCplex ). Columns from 8 to 10 display the best lower bound reached by VND&IP (LBVND&IP ) extracted from [1], the runtime consumed (tVND&IP ) and the gap between the LBVND&IP and that achieved by the Cplex

166

S. Boukhari et al.

solver (LBCplex ). Columns from 11 to 13 show the best lower bound reached by EHM (LBEHM ), the runtime consumed (tEHM ) and the gap between the LBEHM and that achieved by the Cplex solver (LBCplex ). Finally, the last line of each table displays the average lower bounds over all tested instances and those related t the average runtimes. Table 5. Computational power of EHM versus MTS, VND&IP and the Cplex solver on instances of the group with objects in [60, 90]. #Inst. Cplex solver

VND& IP

EHM

T m

LBCplex

tCplex LBVND&IP tVND&IP GapVND&IP LBEHM

5 10

766837

2310

766841.8

2.91

−0.000626 766841.8

3.33

−0.000626

20

871105

2051

871105.1

3.69

−0.000011 871105.1

4.02

−0.000011

30

808314

2104

808314.8

16.97

−0.000099 808314.8

4.56

−0.000099

1010

1480338

2497

1480401.9 15.09

−0.004317 1480401.9

8.60

−0.004317

1520

1713710

3170

1713720.2 12.49

−0.000595 1713720.2

12.26 −0.000595

30

1551262

3471

1551705.6 11.70

−0.028596 1551705.6

18.56 −0.028596

10

1818377

1971

1818395.9 12.02

−0.001039 1818395.9

18.44 −0.001039

20

2455626

3373

2456682.6 12.12

−0.043028 2456682.6

48.34 −0.043028

2030

2301745

3603

2302190.8 12.02

−0.019368 2302192.5 18.44 −0.019442

10

1945837

2465

1945856.9 8.28

−0.001023 1945856.9

23.40 −0.001023

20

3051810

3442

3055665.7 22.82

−0.126341 3055665.7

226.87−0.126341

30

2918388

3575

2922242.6 22.20

−0.132080 2922242.6

193.45−0.132080

Average1806945.752836.001807760.33 12.69

tEHM GapEHM

−0.045080 1807760.4748.35 −0.045088

For Table 5, the same notations are used when comparing the behavior of Cplex solver, VND&IP and EHM. We note, for all the tables, that the negative value means that the corresponding method provides a better average lower bound when compared to that achieved by the Cplex solver whereas the value on the bold-space means that the new average bounds are achieved by the corresponding method. In what follows, we comment on the results reported in Tables 4, 5 and 6, where we compare the results provided by EHM (the average lower bounds) with the best average lower bounds provided by the other methods. 1. Cplex versus other methods (all the tables): MTS, VND&IP and EHM dominate the Cplex solver despite several tunings applied to the solver settings. 2. EHM versus MTS (Tables 4 and 6): one can observe that EHM outperforms MTS. Indeed, EHM is able to achieve better global average lower bounds than those reached by MTS; EHM provides two (new) greatest average values for the instances whose objects belong to [40, 60] (Table 4) than those achieved by MTS, and six (new) greatest average values for the instances whose objects belong to [90, 100] (Table 6).

Hybrid Methods

167

3. EHM versus VND&IP: over all tested instances (Tables 4, 5 and 6), seven new average solution values (bounds, in bold-space) are provided by EHM, and it globally matches all the best solution values achieved by VND&IP. The average improvement achieved by EHM varies from 0.032067 (Table 4, line Average, last column) to 0.133807 (Table 6, line Average, last column).

Table 6. Computational power of EHM versus MTS, VND&IP and the Cplex solver on instances of the group with objects in [40, 60].

5

#Inst.

Cplex solver

T

m

LBCplex

tCplex

MTS LBMTS

tMTS

GapMTS

VND& IP

LBVND&IP tVND&IP GapVND&IP LBEHM

EHM tEHM

GapEHM

5

10

550192

2479

550191.9

19.4

0.000018

550191.9

4.90

0.000018

20

493255

3241

493256

41.0

−0.000203 493256

5.84

−0.000203

493256

8.87

−0.000203

30

559080

2746

559080.5

90.1

−0.000089 559080.5

16.54

−0.000089

559080.5

9.10

−0.000089

10 10

1029024

3114

1029034

103.4

−0.000972 1029034

6.55

−0.000972

1029034

7.70

−0.000972

15 20

938588

2941

938596.4

121.7

−0.000895 938596.1

4.75

−0.000863

938596.5

13.05

−0.000906

30

1086450

3510

1086842.3

143.9

−0.036108 1086842.3

21.23

−0.036108

1086842.3

29.78

−0.036108

10

1370342

3083

1370358.7

103.1

−0.001219 1370358.7

16.30

−0.001219

1370362.1

28.43

−0.001467

5.96

0.000018

550191.9

20

1250502

3303

1251095.5

149.3

−0.047461 1251095.5

15.80

−0.047461

1251095.5

35.14

−0.047461

20 30

1609642

3444

1613675.0

195.8

−0.250553 1613676.5

17.54

−0.250646

1613676.9

65.69

−0.250671

10

1616332

2820

1616357.4

96.0

−0.001571 1616357.4

15.39

−0.001571

1616358.5

28.03

−0.001640

20

1681049

3339

1686947.9

262.6

−0.350906 1686950.3

19.07

−0.351049

1686950.3

243.64 −0.351049

30

2138531

3602

2146704.4

354.9

−0.382197 2146707.6

30.02

−0.382347

2146707.6

829.20 −0.382347

Average 1193582.25 3135.17 1195178.33 138.17 −0.133722 1195178.90 14.58

−0.133770

1195179.34 108.63 −0.133807

Conclusion

In this paper the multiple knapsack problem with setup is studied; that is a more complex variant of the well-known binary multiple knapsack problem. A hybrid method was proposed for efficiently tackling large-scale problem instances. The first version of the method combines two feature strategies: (i) solving a relaxed linear program and, (ii) a series of knapsack problems. Next, an enhanced version of the method was designed, where a series of valid constraints were added for providing a powerful method. The performance of the proposed methods are evaluated on benchmark instances of the literature, where their provided results are compared to those reached by the state-of-the-art Cplex solver and the best methods available in the literature. According to the experimental part, the computational power of the enhanced hybrid method showed its ability to remain very competitive. Indeed, it outperformed the methods available in the literature and was able to provide improved gaps for large-scale instances in a reasonable average runtime. As the multiple knapsack problem with setup is related to practical situations, there are several possibilities for further investigation involving efficient approaches. Indeed, some specialized methods can be first tailored, where tights bounds and other valid constraints can be proposed for fixing some variables before solving the rest of the problem. Such a preprocessing can also be used into a specialized branch-and-cut for solving the problem to optimality. Also,

168

S. Boukhari et al.

a parallel cooperative version of the proposed method can also be investigated for providing improved bounds and by decreasing the global average runtime, especially for the local branching used in this work. Acknowledgments. The authors thank the anonymous referees for their helpful comments and suggestions which contributed to the improvement of the contents of this paper.

References 1. Adouani, Y., Jarboui, B., Masmoudi, M.: Efficient matheuristic for the generalised multiple knapsack problem with setup. Eur. J. Ind. Eng. (EJIE) 14(5), 715–741 (2020) 2. Akinc, U.: Approximate and exact algorithms for the fixed-charge knapsack problem. Eur. J. Oper. Res. 170(2), 363–375 (2006) 3. Altay, N., Robinson, J.R., Powell, E., Bretthauer, K.M.: Exact and heuristic solution approaches for the mixed integer setup knapsack problem. Eur. J. Oper. Res. 190(3), 598–609 (2008) 4. Amiri, A.: A Lagrangean based solution algorithm for the knapsack problem with setups. Expert Syst. Appl. 143, 113077 (2019). https://doi.org/10.1016/j.eswa. 2019.113077 5. Boukhari, S., Dahmani, I., Hifi, M.: Local branching strategy-based method for the knapsack problem with setup. In: Wyld, D.C., et al. (eds.) Proceedings of the 4th International Conference on Artificial Intelligence, Soft Computing And Applications, CS & IT - CSCP 2020, pp. 65–75 (2020). https://doi.org/10.5121/ csit.2020.101606 6. Chebil, K., Khemakhem, M.: A dynamic programming algorithm for the knapsack problem with setup. Comput. Oper. Res. 64, 40–50 (2015) 7. Della, C.F., Salassa, F., Scatamacchia, R.: An exact approach for the 0–1 knapsack problem with setups. Comput. Oper. Res. 80, 61–67 (2017) 8. Furini, F., Monaci, M., Traversi, E.: Exact algorithms for the knapsack problem with setup. Technical report, University of Paris Dauphine (2017) 9. Guignard, M.: Solving Makespan minimization problems with Lagrangean decomposition. Discrete Appl. Math. 42(1), 17–29 (1993) 10. Hifi, M.: Exact algorithms for unconstrained three-dimensional cutting problems: a comparative study. Comput. Oper. Res. 31(5), 657–674 (2004) 11. Khemakhem, M., Chebil, K.: A tree search based combination heuristic for the knapsack problem with setup. Comput. Ind. Eng. 99, 280–286 (2016) 12. Lahyani, R., Chebil, K., Khemakhem, M., Coelho, C.: Matheuristics for solving the multiple knapsack problem with setup. Comput. Ind. Eng. 129, 76–89 (2019) 13. McLay, L.A.: Designing Aviation Security Systems: Theory and Practice (2006) 14. Merkle, M., Hellman, M.: Hiding information and signatures in trapdoor knapsacks. IEEE Trans. Inf. Theor. 24(5), 525–530 (1978) 15. Michel, S., Perrot, N., Vanderbeck, F.: Knapsack problems with setups. Eur. J. Oper. Res. 196(3), 909–918 (2009) 16. Perboli, G., Gobbato, L., Perfetti, F.: Packing problems in transportation and supply chain: new problems and trends. Proc. Soc. Behav. Sci. 111, 672–681 (2014) 17. Yanchun, Y.: Knapsack problems with setup. Dissertation, Auburn University, USA (2006)

Accelerated Quantum Computation based on Quantum Coherent Dynamics of Evanescent Photons in Liquid Water Luigi Maxmilian Caligiuri(B) Foundation of Physics Research Center (FoPRC), Cosenza, Italy [email protected]

Abstract. It has been shown how evanescent photons, produced in highlycoherent excited quantum states of liquid water, could be considered in order to perform quantum computations in a completely novel and still unexplored fashion by considering the formation of excited coherent quantum domains in liquid water, associated to cold vortex of quasi-free electrons, and their interaction through the mutual exchange of virtual evanescent photons, by quantum tunnel effect. Furthermore, the use of metamaterials to enclose water molecules, in order to form suitable waveguide for the evanescent photons generated inside water coherent domains, could allow for the implementation of a superfast network of interacting coherent domains able to represent a basic architecture for a novel kind of quantum hyper-computer based on the coherent dynamics of liquid water. This introduces a new frontier in the field of quantum computation, whose applications to both theoretical and advanced-technology fields (from the simulation of complex quantum systems to biotechnology, artificial intelligence, data encryption and decryption, etc.) would be very deep and nowadays unimaginable. Keywords: Quantum electrodynamics coherence · Liquid water · Excited coherent domains · Evanescent photons · Metamaterials · Quantum hypercomputing

1 Introduction According to the theory of QED coherence in condensed matter [1], liquid water can be described as a two – phases system, respectively associated to a coherent and noncoherent fraction of molecules. The coherent fraction is composed by arrays of the socalled “coherent domains” (CDs), namely macroscopic quantum domains in which all the molecules oscillate in phase with each other and with a self-trapped electromagnetic field, while the uncoherent one is composed by vapor (an ensemble of uncorrelated molecules) filling the interstices between coherent domains. Inside every coherent domain, such dynamics generates an electromagnetic field spreading across its boundaries in the form of evanescent field whose tail extends far from it, being therefore able to overlap the analogous evanescent field produced by the other CDs in the surroundings. It has been shown [2–6] this overlapping allows the exchange among close CDs, through quantum © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 169–187, 2022. https://doi.org/10.1007/978-3-030-80119-9_8

170

L. M. Caligiuri

tunnel effect, of virtual photons belonging to the evanescent modes of coherent e.m. field associated to the water coherent domains, provided the involved CDs have a spectrum of excited states as always occurs in the case of coherent liquid water [5, 7]. A very interesting features of such interaction is that the resulting probability transition between quantum states of the coupled interacting coherent domains, can be described by a “Rabi -like equation” [5]. This suggests the fascinating and novel idea to use this still unexplored “evanescent tunneling coupling” interaction to perform quantum computation by using coherent liquid water as a physical substrate since, as it will be shown in the following, such interaction can be used to realize every type of one- and two-qubit gates and then every type of quantum computation at all [6]. The use of coherent water as a physical substrate for quantum computation could offer several meaningful advantages with respect the present-day state of the art in quantum computation scheme as, first of all, its stability against the environmental decoherence, obtained by enclosing coherent water inside cavities made of metamaterials [2, 6, 8]. The employment of metamaterials also determines a meaningful amplification of the evanescent e.m. field crossing the boundaries of every coherent domain so allowing the possibility for each of them “to communicate”, through the evanescent tunnelingcoupling mechanism, with other and farther CDs, so generating an extended network of mutually interacting coherent domains [2–4, 6]. One of the most interesting features of quantum computation is its alleged extraordinary computational power with respect its classical counterpart. Nevertheless, the actual computational speed of a quantum system is limited by several factors that drive the system far apart the largest theoretically reachable speed. This “ultimate” limit would be determined by the energy spread E of the physical system performing computation [9, 10]. Consequently, for a given quantum system, if we could “indefinitely” increase the value of E we would, in principle, “indefinitely” boost the calculation speed to its limit. As it will be proven in the following, this opportunity could be given us just by the dynamics of water coherent domains. Due to the energy gap per molecule characterizing the water coherent state (with respect to the uncoherent one) and the very high number of molecules contained in a single coherent domain, the energy storable by its very easily excitable levels could be very high so that the resulting spectrum would have virtually no upper bound [7]. Furthermore, the arising of the so-called “supercoherence”, namely the onset of a coherence between coherent domains caused by the tunneling coupled interaction [11, 12], further increases the stability of the system and the associated energy gap. In this way the increase of the number of possible logic gates (in turn proportional to the number of the involved CDs) would be accompanied by an increase of the energy available for each gate (due to a wider range of possible excited energy levels for each CD). Supercoherent dynamics opens remarkable chances to further boost the computational power of the proposed model of quantum computation based on water CDs [6]. Just to cite one of most remarkable ones, we consider the implementation of an extended network of correlated water CDs, oscillating in phase with each other, could reduce to nearly zero the overall parallel computational time, when increasing the number N of the

Accelerated Quantum Computation Based on Quantum Coherent Dynamics

171

tuned oscillating elementary systems, so recovering, in principle, the concept of quantum “hyper-computing” or “accelerated” quantum computation (namely the execution of a countable infinite number of computational steps within a finite time interval) already investigated in some previous publications [2–4, 13]. Finally, the use of a particular physical system to perform quantum computation assumes we can control its dynamical evolution in a precise fashion (related to both the computational features and storage/retrieval of information). In the case of water, the supercoherent behavior can be controlled by “tuning” the quantum phase of coherent domains, through the action of e.m. vector potential, or by modifying the coherent oscillation frequency exploiting the interaction between coherent domains and macromolecules different than water [6, 11]. In this paper we discuss how the above model could be used to realize a quantum computing system, whose physical substrate is composed by coherent liquid water, characterized by a very huge computational speed (very close to the maximum speed theoretically reachable), opening the door, in principle, to the realization of quantum hypercomputing, without incurring in some of the most critical issues affecting the quantum computational model so far proposed.

2 A Brief Overview of QED Coherence in Liquid Water According to QFT, the interacting components of a system (atoms or molecules) and the quanta of field that correlate them are considered as objects of the systems as well as the interaction itself. The close correlation between particles and fields produces quasi-particles for which the division between particles and interaction vanishes. In this framework a fundamental role is played by the quantum phase operator  [1, 7, 11, 14] related to the number operator of quanta N by the indetermination relation (we assume, unless differently specified, the natural units system  = c = 1)  · N ≥ 1

(1)

so, for a system composed by a very high number of quanta, N is completely undetermined while  = 0 and the phase is so well-determined and fixes the dynamical evolution of the whole system. An ensemble of large number of components made up of electrically charged particles is always coupled with the quantum fluctuations of physical vacuum (as occurs in the Lamb – shift effect [1]). If we consider an ensemble of two-levels atoms having an energy separation E1 − E2 ≡ E0 = ω0 , then a quantum vacuum fluctuation having the energy E0 is able to excite an atom within the region covered by the wavelength λ=

2π 2π = E0 ω0

(2)

The dynamical evolution of the interacting matter + e.m. field system is described by the Lagrangian function and corresponding Eulero – Lagrange equations for the matter and  It has been shown [1, 11] that, e.m field, the latter described by the potential vector A. when the temperature isless  than  a threshold value Ts and the density of the particles exceeds a critical value N V c , the quantum fluctuations of the charged crepuscules produce a noticeable e.m. field that:

172

L. M. Caligiuri

1) attracts atoms/molecules that resonate with it (generally of the same species), producing a further increase of density until the short-range repulsion is able to stop the process; 2) in turn, drives the fluctuations of the particles, that become “phase-locked”, so that all of them oscillate in tune with each other and with the e.m. field itself between two definite energy configurations within a definite macroscopic spatial domain, called, for this reason, “coherent domain” (CD) whose extension is of the order of λ given by (2) The system then spontaneously moves from an un-coherent state in which all the particles perform uncorrelated oscillations (the e.m. field is nearly zero) to a “coherent” state in which a non-vanishing autogenerated e.m. and the matter fields oscillate in tune having, at any given instant, a well-defined phase. From a QFT standpoint this means the wavefunctions describing coherent domains are eigenstates of a suitable quantum phase operator . Inside CDs, photons and particles lose their individuality to become part of a whole macroscopic object composed by entangled quasi-particles. The oscillation frequency ωcoh of the photons belonging to such coherent e.m. field is lower than the corresponding frequency in vacuum ω0 so that we have for the photon “mass” 2 m2 = ωcoh − k 2  ω02 − k 2 = 0

(3)

Equation (3) implies [1] the coherent e.m. is self-trapped inside CD and it doesn’t propagate outside CD as a progressive wave so ensuring the stability of the system. Nevertheless, as a consequence of the coherent dynamics, the field generated inside a CD spreads outside the CD itself through its own evanescent tail leaking out, in the form of non-radiating evanescent wave, far beyond the CD’s borders. The spatial behavior of the coherent field amplitude A(x, t) is represented in Fig. 1 where r0 is the “radius” of the smallest CD (supposed having spherical symmetry) in equilibrium with its own coherent e.m. and A(0) is the vector potential at the CD’s center (r = 0).

Fig. 1. Spatial profile of coherent e.m. field A(r) of a CD.

The transition from the uncoherent state to the coherent one can be considered as a “condensation” in which the system releases outwards a given energy, previously

Accelerated Quantum Computation Based on Quantum Coherent Dynamics

173

“borrowed” from quantum vacuum zero-point oscillations.  Every coherent domain is then characterized by an energy gap per molecule E N that makes it more stable with respect the uncoherent state. The value of this gap depends on the couple of levels involved in the coherent oscillations. For absolute temperatures different than T = 0, thermal collisions could transfer to molecules an energy higher than the energy gap and push them out of tune. In the general case we can identify a coherent and a non - coherent fraction of the system Fcoh (T ) and Fcoh (T ), respectively indicating the number of molecules belonging, at each instant, to the coherent and non - coherent phases. satisfying, at a given temperature, the following constraint: Fcoh (T ) + Fnc (T ) = 1

(4)

For every T  = 0, the system oscillates between the coherent and the non-coherent state, according to whether the coherent electrodynamic attraction is overtaken or not by the thermal collisions transferring to the CD the energy Ether . The coherent behavior of water has been deeply studied in previous publications [1, 2, 5, 7, 11, 15] to which we’ll refer to for the details. The specific energy level driving the transition towards the coherent state is that characterized by the shortest time τcoh required by the coherent oscillation to build up and consolidate which, in the case of water, is of the order of 10−14 s. The smallest value of critical density in water is associated to the electronic transition between the ground state and the 5d excited stated of the molecule corresponding to the level at about En = 12.06 eV. The corresponding energy gap per molecule is E = 0.260 eV corresponding to the coherent oscillation frequency ωcoh ∼ = 0.26 eV.

3 Quantum Tunneling-Coupling Interaction Between Water Coherent Domains 3.1 “Supercoherence” and the Interaction Between Coherent Domains The idea to use liquid water CDs to perform quantum computations refers to the possibility for coherent domains to “interact” each other [2–6] giving rise to the socalled “supercoherence” [5–7, 11, 12]. It is conceptually similar to coherence between atoms/molecules and e.m. field oscillations just described above in the sense that a certain, generally high, number of CDs, considered as elementary systems, become able to oscillate in phase with each other and the resulting coherent oscillation could cover a much more extended region. As a consequence, the CD changes its own frequency of oscillation and the system also further increases its stability against the damaging effect of thermal fluctuations so further stabilizing the value of Fcoh (T ). For the occurrence of supercoherence, a CD should then discharge its excitation energy outwards in order to give the required oscillations. It could be the case when the coherent CD admits a spectrum of excited energy levels with respect the coherent ground state (at energy ωcoh ) as just occurs in the case of water due to the particular couple of levels involved in the coherent oscillation, namely the ground state and the excited level at 12.06 eV that lies just below the ionization threshold of 12.60 eV. For

174

L. M. Caligiuri

this reason, in the water coherent state more than more than 0.10 of electron per water molecule can be considered as quasi-free particles [7]. A water CD can include, at the density of liquid water, a number of molecules of the order of n  2 · 105 , corresponding to a number of quasi-free electrons of about ne  2 · 104 , meaning that each water CD acts as an enormous reservoir of quasi-free electrons. These electrons can be easily excited by external, even weak, energy source so originating metastable coherent excited levels in the form of “cold electron vortices” [19] as far as this energy supply is lower than the energy gap per molecule Ecoh of the water CD. This energy, because of the coherence, is absorbed by the CD as a whole and such vortices cannot be excited nor decay thermally since they are coherent and have a quantized magnetic moment that allows them to align to an external static magnetic field. For this reason, they are characterized by a very long lifetime. It is important to observe that, being the spectrum {En } a collective property of the entire CD, it has practically no upper limit since, although every molecule in the coherent state cannot absorb an energy higher than Ecoh , all the N resonant molecules can account for a total energy of excitation of the order of NEcoh a very high quantity  (including frequency up to visible and HV spectrum) since, at room temperature, N V ∼ 1022 cm−3 . According to our results [2–6, 13], it is then possible for two or more CDs to directly interact with each other through the overlapping of the evanescent tails belonging to their respective coherent e.m. fields by considering the quantum tunneling of virtual photons, associated to such evanescent modes. As a matter of fact CDs tend to remain closely packed to each other since in this configuration, when their intercentrum distance is of the order of 2r0 , the energy gap associated to the overlapping of evanescent e.m tails determines an energy gaining up to four times larger than in the isolated CDs at the point r = rcoh [1, 5]. 3.2 Quantum Coupling Between Water CDs From the standpoint of classical physics, evanescent waves usually originate by the “frustrated total internal reflection” (FTIR) of a light beam at the boundary between two media characterizedbydifferent values of refractionindex. Inthis casea“transmitted”wavecomponent is nevertheless present in the second medium, namely just the so-called “evanescent” wave. From the quantum standpoint, the tunneling of evanescent modes of e.m. waves is just interpreted as the tunneling of virtual photons through the corresponding potential barrier. Such tunneling photons are characterized by a negative square mass in the ordinary metric or, equivalently, by a superluminal group velocity [2, 16–18]. This evanescent e.m. field acts as a “propagation channel” of virtual photons between water CDs, allowing the realization of quantum computing in liquid water [6]. Despite they are not directly observable, virtual particles represent necessary intermediate states between real observable states having a direct influence on them. When two CDs are sufficiently close each other (or when the evanescent amplitude is sufficiently high), their respective e.m. evanescent fields can overlap. The width of the resulting overlapping zone, through which tunneling of virtual photons takes place, will depend on the distance between the two CDs and on the spreading of the two evanescent fields outside the respective CDs, whose “extension”, described by their tales, is a

Accelerated Quantum Computation Based on Quantum Coherent Dynamics

175

function of coh (Fig.  ω  2). For a water CD in its ground state it has been calculated [2] dp = 103 2π ω0  1.03 · 10−4 m.

Fig. 2. Overlapping between the evanescent fields associated to two nearby CDs.

In order to the tunneling coupling interaction to take place, once the coherent state has been achieved, it is needed to ensure: a) the stabilization of the coherent fraction of liquid water against disruptive thermal fluctuations (so that this water can be considered as “fully” coherent Fcoh (T ) ∼ 1 even at room temperature) while it is advisable b) a suitable amplification of the evanescent fields generated by coherent domains. As already shown [2, 5, 6, 8, 13] both these conditions can be assured by enclosing water inside the inner volume of suitable (even symmetrical) waveguides whose walls are composed of metamaterials (MTMs), namely a kind of material characterized by a purely imaginary value of refraction index as, for example, plasmas or ferromagnetic materials. In this way we can also consider the water molecules enclosed in such waveguide as a unique coherent domain, at least with respect to the generation of the evanescent field by the enclosed CDs [2]. Despite it could be enough for a couple of water CDs to be at a mutual distance dCDs ≤ 1 · 10−4 m in order to have their respective evanescent fields overlapping [2, 5], the evanescent field decays quickly with the distance so that, even for short distances, the field intensity could be too low for ensure the needed coupling between CDs. This could be the case, in particular, for the excited states of CD, for which ωex ωcoh , since dp is inversely proportional to ω. This amplification induced by MTMs determines an important increase of the evanescent penetration depth dp allowing a single CD to interact with the other ones even placed far away from it, so forming a spatially extended network of mutually and directly interacting CDs in water that could be used, as we’ll see, to realize superfast parallel quantum computations. The quantum - tunneling “interaction” is made possible by the existence of excited levels in the CDs whose excitation-deexcitation determine the tunneling of virtual coherent photons between the CDs, since in the fundamental state, the “coherent ground state”, the CD is stable. We could also think this interaction to occur between the quasi-free coherent electrons, respectively pertaining to different excited levels in the two CDs, as the result of the exchange of a virtual photon through quantum tunneling. The analysis of such process at a microscopic scale, can be found in [5, 19]. Macroscopically [5], the “tunneling” coupling between two CDs can be described by the

176

L. M. Caligiuri

following total Hamiltonian: H = H0 + H 

(5)

where H0 and H  respectively indicate the “free” and the “interaction” Hamiltonian of the system (corresponding to the exchange of one photon). If we consider an isolated water CD as a two-levels system with two accessible eigenstates (normalized to unity), named |1 and |2 , satisfying the stationary Schrodinger’s equation: H0 |1 = E1 |1 ; H0 |2 = E2 |2

(6)

with E1 and E2 energy eigenvalues of the system, we can decompose the Hamiltonian as H0 = E1 |1 1| + E2 |2 2|;

H  = (|1 1| + E2 |2 2|)

(7)

where  ∈ R represent the “strength” of the interaction. We then obtain [5] the energy eigenvalues of the system   EA = E1 (1 + T ) 2 + E2 (1 − T ) 2 (8) EB = E1 (1 − T ) 2 + E2 (1 + T ) 2   and T = 1 + 4 2 (E1 − E2 )2 with E1 + E2 = EA + EB . The time-evolution of the system is described by | (t) = U (t; t0 )| (t0 ) = exp[−iH (t − t0 )]| (t0 )

(9)

if we now suppose the initial state of the system is just | (t = 0) = |1 we obtain [5] the transition probability from the initial state |1 at t = 0 to the state |2 at time t as ⎞ ⎛ 2 2 2  (E1 − E2 ) + 4 ⎠ (10) sin2 ⎝ t P12 = | 2|U (t)|1 |2 = 4  2 + (E1 − E2 )2 that is the so-called “Rabi equation”. This solution is equivalent to a Rabi oscillation in which a photon wave packet of given wave number is coupled back and forth between the two CDs. Even more interestingly, we can express the quantum time evolution operator of such system in terms of Pauli’s matrices in the form [5, 20] H = n0 I + n · σ   where I is the 2 × 2 identity matrix, σ = σx , σy , σz and n0 =

1 1 (E1 + E2 ), n1 = , n2 = 0, n3 = (E1 − E2 ) 2 2

the time-evolution operator then becomes   n (E1 + E2 ) t cos(|n|t)I − i sin(|n|t) U (t) = exp −i · σ |n| 2

(11)

(12)

(13)

Accelerated Quantum Computation Based on Quantum Coherent Dynamics

177

On the other hand, any arbitrary unitary operator U can be expressed  as a rotation of an angle θ about an axis defined by the real unit vector uˆ = ux , uy , uz , namely 

σ (14) U = exp(iα)Ruˆ (θ ) = exp(iα) exp −iθ uˆ · 2 where the rotation operator Ruˆ (θ ) can be expanded as



 θ θ I − i sin uˆ · σ Ruˆ (θ) = cos 2 2

(15)

and we have, by comparing (15) and (13) α=−

n E1 + E2 t; θ = 2|n|t; uˆ = |n| 2

(16)

The very important result obtained above is that the time-evolution operator of system of two interacting CDs through tunneling coupling can be expressed as the product of a rotation and an overall phase shift. By effect of this coupling, the two CDs cannot be further considered as independent entities but as two interconnected parts of the same system. The wavefunctions describing the coherent domains, 1 = ψ1 exp(i1 ) and 2 = ψ2 exp(i2 ), are eigenstates of the phase operator (whose eigenvalue is related to the coherent frequency of oscillation including the angular frequency of the excited vortices) and also describe the state of tunneling photon as “being” in the one CD or in the other, through the shift  in the value of their relative phase.

4 Quantum (Hyper) Computation by Means of Water Coherent Domains 4.1 The Concept of “Universal” Quantum Computation Quantum computation manipulates quantum information through dynamic transformation of quantum systems that maps the state space of quantum system in itself. Such transformations are described by unitary linear operators (or matrices) since geometrically they correspond to rotations of the complex vector space associated with the quantum state space. In classical computation, a small set of gates (like, for example, AND, OR, NOT gates) can be used to compute an arbitrary classical function and then such set is said to be universal for classical computation. A similar result holds for quantum computation where a set of gates can be found to be used to realize universal quantum computation. This result is based on the fact that arbitrary unitary transformations can be implemented from a set of primitive transformations, including the two-qubits CNOT (controlled-NOT) gate in addition to three kinds of single qubit gates. A suitable set of gates that also allows a fault-tolerant construction is composed by the following ones [20]:

178

a) b) c) d)

L. M. Caligiuri

the Hadamard gate H (not to be confused with the Hamiltonian);  the π 8 gate T ; the phase gate S; the CNOT (Controlled-NOT) gate.

In particular the CNOT gate, which is a fundamental element of the set of universal 1/ 2 . quantum gates, can be realized by starting from the square-root of SWAP gate USWAP This gate is defined so that when applied twice it realizes the SWAP gate given, in the computational basis, by the matrix ⎛ ⎞ 1+i 00 0 ⎟ 1 ⎜ 1/ 2 ⎜ 0 1i 0 ⎟ = (17) USWAP 1 + i⎝ 0 i 1 0 ⎠ 0

001+i

It can be shown the CNOT gate can be written as [6, 20, 21]       σˆ ZA σˆ ZB σˆ ZA 1/ 2 1/ 2 cNOT = (IA ⊗ HB ) exp iπ exp iπ USWAP exp iπ USWAP (IB ⊗ HA ) 4 4 2 (18) where A (B) indicates the operator acts on the first (second) qubit. Since the CNOT gate 1/ 2 gate can be considered a universal two-qubits gate for quantum computing, the USWAP can then represent the building block of any quantum computer when properly joined with the “universal” sequence of single qubit gates above considered. 4.2 The Realization of Two-Qubits Gates by Means of Water Coherent Domains The computational features of the water CDs are based on the evanescent coupling between two water CDs. We can describe the tunneling exchange of virtual photons between two water-filled waveguides 1 and 2 by using the dual-rail representation for the photon, according to which we would adopt the following encoding for the two logic states |0 L and |1 L |0 L → |0 1 |1 2 = |01 ; |1 L → |1 1 |0 2 = |10

(19)

where |0 i and |1 i respectively represent the state in which a photon is absent and present as spatial mode in the i-th waveguides. If we consider the exchange of one photon, it can be shown the Hamiltonian of the system has the same form of (5), then by assuming |0 L → |1 and |1 L → |2 , where the states |1 and |2 are those defined in (6), we have P12 = | 2|U (t)|1 |2 = |L 1|U (t)|0 L |2 ≡ P(0L  1L )

(20)

In our theoretical framework the propagation of a photon in an optical fiber is likened to the presence of a photonic spatial mode, due to tunneling, in one or the other waveguide as sketched in Fig. 3.

Accelerated Quantum Computation Based on Quantum Coherent Dynamics

179

Fig. 3. Evanescent coupling and virtual photon exchange between two waveguides filled with water CDs.

This can be shown if we consider, assuming a not too strong coupling, the total Hamiltonian for the system (we limit ourselves to the photon dynamics) [6, 22]:      † † † † + a a a + a a ωk a ak,1 +  a (21) H=        k,2 k,1 k,2 k

k,1

k,2

k,1

k,2

where a (a† ) is the annihilation (creation) operator for the mode k in the i-th waveguide k,i k,i (i = 1, 2) acting on the vacuum state |0 and γ is a real parameter quantifying the coupling strength. With this notation |0 L → |ψ1 = a1† |0 , |1 L → |ψ2 = a2† |0 and, if γ is  the total Hamiltonian has the same form approximatively independent of the value of k, of (7) and we can deduce the (21). We then easily note that, if we start the system in the state |0 L , after a time T given by ⎤−1 ⎡  2 − E 2 (E ) 1 2 (22) + 2 ⎦ T =⎣ π 4 it has the maximum probability to be in the state |1 L as results from (10). For a given value of , by suitably choosing the values of E1 , E2 we can obtain P(0L → 1L ) = 1 then realizing the transformation Xˆ such as Xˆ |0 L = |1 L

(23)

that switches the quantum state of the system and then acts like a SWAP gate on its logic state, so we can assume Xˆ = USWAP .  Even more interestingly, for T¯ = T 2, namely ⎡

⎤−1

π (E1 − E2 ) + 2 ⎦ T¯ = ⎣ 4 4 2

we can define a new operator Xˆ¯ by   Xˆ¯ Xˆ¯ |0 L = Xˆ¯ 2 |0 L = |1 L

(24)

(25)

180

L. M. Caligiuri

namely (Fig. 4) 1/ 2 Xˆ¯ ≡ USWAP

(26)

whose action on the computational basis is given by |0 1 |0 2 → |0 1 |0 2 1−i |0 1 |1 2 → 1+i 2 |0 1 |1 2 + 2 |1 1 |0 2 1−i |1 1 |0 2 → 2 |0 1 |1 2 + 1+i 2 |1 1 |0 2 |1 1 |1 2 → |1 1 |1 2

(27)

Fig. 4. Realization of quantum SWAP gate by means of virtual photons exchange in waveguides filled with water CDs.

The deep importance of such result relates to the possibility to realize, in principle, the building block of any quantum computer by using the virtual photon – evanescent coupling interaction between water CDs. 4.3 The Realization of One-Qubit Gates by Means of Water Coherent Domains Any generic one-qubit gate can be represented by a unitary transformation given by the product of a rotation and an overall phase shift in the space of quantum states [20]. On the other hand, as we have also shown, the time evolution of a system of two interacting CDs can be described exactly in this way by Eq. (13). This means the evanescent tunneling coupling dynamics above described can be used to realize any single qubit quantum gate. We  can limit ourselves to realize just the quantum gates representing the Hadamard, the π 8 and the phase gates from whom every one-qubit quantum gate can be constructed [20]. Our goal is to find the values of the parameters α, θ and uˆ (i.e. the three real numbers ux , uy and uz ) such as the unitary operator 

σ (28) U = exp(iα) exp −iθ uˆ · 2

Accelerated Quantum Computation Based on Quantum Coherent Dynamics

181

 respectively represents the Hadamard, the π 8 and the phase gate and then express them as a function of the physical quantities that characterize the dynamics of our systems namely, in our model, E1 , E2 and . These values are shown in Table 1 [6]. Table 1. Values of the parameters of operator U to implement quantum gates. Quantum gate

α

θ

 Hadamard π 2 π T-gate S-gate

nx

ny

nz

√ 2 0

1

0

1

0

1

1

  π 8 π 4 0   π 4 π 2 0

√ 2

We have proven a very important result namely that, by a suitable selection of the physical parameters E1 , E2 ,  and by letting the system evolving for an appropriate time t, the system composed by interacting water CDs can realize, in principle, any generic quantum gate. Such dynamics is also able to reproduce any controlled unitary operations cU (such as, for example, CNOT), so allowing the realization of any quantum computing circuits. This can be realized by considering the circuit shown in Fig. 5.

Fig. 5. Circuit representation of controlled unitary operation on single qubit.

The reason is that any generic controlled operation on one-qubit can be written as U = eiα AXBXC where X = σˆ X and 

 γ  γ  δ +β δ−β ; C ≡ Rz A ≡ Rz (β)Ry − ; B ≡ Ry − Rz − 2 2 2 2

(29)

(30)

with unitary operators A, B, C acting on a single qubit such that ABC = I and that, as we have shown, any unitary transformation can be expressed in terms of rotations in the space of quantum states which, in turn, can be simulated through interacting water coherent domains for a suitable choice of the transformation parameters (see Table 1). The mathematical relations between the parameters in (30) and the angle θ in Table 1 have been calculated in [6]. The previous discussion can be easily generalized to realize multiple - qubits conditioned unitary operations [6].

182

L. M. Caligiuri

5 A Novel Computer Architecture for Quantum Hypercomputation A feasible quantum computing scheme must satisfy, in particular, some requirements [20, 21]: a) b) c) d) e)

the physical realization of well-defined qubits; the initialization in a well-defined state, representing the qubit |0 ; the implementation and control of a set of universal quantum gates; the efficient storing and retrieving of information; a sufficiently long decoherence time.

The issue of environmental decoherence can be solved by invoking the stability of the coherent state of water due to energy gap and the stabilization of the coherent fraction against the thermal fluctuations obtained by enclosing water inside MTM waveguides [2, 5, 6]. The features a) – d) require, in our case, the ability to control the coherent states of liquid water. Within the proposed framework, this means the skill to control the value of quantum phase  of any CD. The control of this phase with precision allows us to exactly defining the quantum state of the system including its excited states. A change in the phase φ then modifies the coherent oscillation frequency and vice versa. Even  more   interesting, such a change would also affect the electromagnetic potentials Aμ = V , A of the system and its dynamical evolution [1, 5–7, 11, 23]. The interaction among two or more CDs or between a CD and the surrounding ˆ “environment” can be then considered as mediated by the action of phase operator . In the coherent quantum state of water, the wavefunction (both of particles and e.m. field) acquires a macroscopic meaning, a situation quite similar to that occurring in laser or in superconductivity where the entire collection of microscopic objects (respectively photons and super-electrons) are described in terms of an electromagnetic field with a well-defined phase. By exploiting such analogy [6, 7], we can state a relationship between phase and potentials in the coherent quantum state of water through the Josephson’s equations:   = eA ∇ 

(31)

e ∂ = V ∂t 

(32)

where the quantum phase  includes, in the excited states, the information about the “rotational” frequency of the associated cold vortices. The above equations show e.m. potentials are able to modify the phase of the coherent state, including the frequency of excited vortices, and vice versa even in absence of the “physical” fields as occurs in the Aharonov – Bohm effect. The e.m. potentials can then establish a long - range as well as superfast (actually faster than light being, driven by a phase velocity involving only a local supply of energy arising from quantum vacuum fluctuations) interaction. The manipulation of vector potentials can be then used to “program” and control the quantum gates implemented by means of the evanescent

Accelerated Quantum Computation Based on Quantum Coherent Dynamics

183

tunneling coupling between water CDs. Furthermore, any coherent state corresponds to an eigenstate of quantum phase operator and then is always well-defined. The time required to obtain a given quantum state of water CDs, after the action of vector potential has taken place, can be very short (we recall that the onset of the coherent state of water would occur in about 10−14 s), while the time required to set the quantum gates, both for one- and two-qubits transformations, can be roughly estimated by the equation ⎡ ⎤−1 θ ⎣ (E1 − E2 )2 (33) + 2 ⎦ Tgate = 2 4 for a suitable choice of the rotation angle, depending on the particular gate to be realized. The time needed to perform a gate operation (a complete CD oscillation in our model), that we could define as the “operational time”, is then given by [6] T = tch + tdsc + Tgate

(34)

where tch is the time required to excite the CD (related to the rate of energy storage from the “environment”), tdsc the time required to de-excite it and Tgate is given by (33). In summary, we could take control of the whole process “turning-on” and “turning-off” the quantum gates, by adjusting the various terms in (34). It is commonly believed that quantum computing could be intrinsically much more powerful than classical computation using traditional silicon processors. We also show that, in principle, the discussed model of quantum computation (based on water coherent domains), could be even much more powerful of the quantum computation schemes so far proposed. According to the proposed model, the overall computational time includes: the operational time T ; the time required by each quantum gate to “communicate” with any other; the decoherence time of the system; the delay time of memorization and readout of the information; the implementation of parallel and/or serial computational schemes. Computational time, expressed in term of the number of logical operations performed per unit of time, is given by [9, 10] 1 2E ≤ (35) T π where E is the energy spread of the computing system, so that there would be a theoretical upper bound to this number given by (35). In the case of two tunneling – interacting water coherent domains we can assume nop =

(E1 − E2 )2 (36) + 2 (E)2 ∼ = 4 and, by comparing (36) with (33), we can deduce the very important result that quantum gates realized in such a way could operate very closely the theoretical limit during the time needed to perform the elementary quantum operation both for one and two-qubits gates, namely  (E1 − E2 )2 (37) + 2 nop,MAX  4

184

L. M. Caligiuri

Even more interesting, for a given value of , we could reduce the time required to implement a gate (so boosting its speed) by increasing the difference E1 − E2 , so increasing nop,MAX . In the case of excited water CDs, it would be possible, at least until the energy gap per molecule per CD is not exceeded. The maximum energy gap per molecule is about E  0.26 eV and then, by assuming an average number of water molecules per CD of about 105 , we have a single water CD could gather in its own excited states up to about 105 eV of energy. The value of nop can be further increased by raising the interaction strength  that in turn depends on the “speed” of coupling interaction, being inversely proportional to the tunneling time of evanescent photons. We have shown [6] this time could be as short as ttun  10−20 s corresponding to a superluminal velocity vmax  10 c [2, 6] for the water coherent ground state and it increases with the energy of excited levels of water CD. Another source of time delay in the computational process is represented by the time required to set the quantum state of coheret domains to a definite value before and after the gate running operation. These states, being eigenstates of phase operator, are entirely characterized by the phase value  whose modification can be realized through the action of suitable e.m. vector potentials, Such “phase interaction”, as already discussed above, being pure information, would propagate between any two points of a coherent environment in a nearly zero time, without posing any difficulty from the standpoint of Special Theory of Relativity [2, 16]. In this way it could be possible to eventually reduce to a very low value the time required to set the state of a quantum gate as well as the delay time occurring during the communication between two quantum gates. Finally, the overall computational speed is also influenced by the computational configuration, namely if the logical operations are performed in serial or in parallel. According to the commonly accepted conception, this limit would be independent from the computational architecture [9]. Nevertheless, according to our model of computation, the onset of the coherence is characterized by an energy gap that implies the coherent state to have an energy smaller than the non-coherent one. A further increase of the coherence of the system, as those occurring in supercoherence, would in turn raises the stability of the system and the correlated energy gap. In this case the use of a greater number of logic gates (proportional to the number of CDs) would be accompanied by an higher energy available for each gate (due to a wider range of possible excited energy levels for each CD). Such dynamics opens a very remarkable perspective to further boost the computational power of the proposed model of quantum computation based on water CDs. The implementation of an extended network of correlated (through the tunneling and/or phase interaction) water CDs oscillating in phase with each other, could then reduce to nearly zero the overall parallel computational time, when increasing the number N of the tuned oscillating elementary systems (water molecules and even other chemical species in the form of macromolecules). In fact, in this case, we could rewrite (35) in the form 2 ¯ El (N ) π N

nop (N ) ≤

l=1

(38)

Accelerated Quantum Computation Based on Quantum Coherent Dynamics

185

in which, when N increases, each term of the sum increases as well while the time required to execute each step proportionally reduces, so that, if N1 N2 we recover, in principle, the concept of “hyper-computing”, already analyzed, in a preliminary form, in several previous publications [2–6, 12, 13]. Finally, in a real computer architecture, it is very important to consider the question of scalability, namely the possibility to perform quantum operations on a large number of qubits as well as the memorization and readout of quantum information in the specific physical system used to compute (the coherent network of water coherent CDs in our case), and the related time required to perform such operations. All these features are strictly related to the specific processes used to implement memorization and readout of quantum information that, in the above model, are linked to the mutual phase interaction between water CDs and to the interaction between them and their surrounding electromagnetic-thermodynamics environment which they “probe” through e.m. vector potentials. A more detailed study of role of these interactions in the storage and retrieval of quantum information is not trivial and it will be carried out in a forthcoming work.

6 Summary and Outlook In this paper, starting from the results of QED coherence in matter applied to liquid water, we have suggested to use the energy-excited states in water coherent domains to realize universal quantum gates for one and two-qubits able, in principle, to perform any type of quantum computing. The computational layer exploits a so far unexplored interaction arising between the evanescent e.m. fields generated by water coherent domains, involving the exchange of virtual photons by tunneling effect as well as a “phase” interaction ruled by the electromagnetic vector potentials to set the communication between quantum gates and the physical states used for computation. The quantum gates so realized can operate at a very high speed and very closely to the ultimate limit imposed by the fundamental principles of quantum mechanics so far generally accepted, by adjusting the energy levels of the water coherent domains (corresponding to its excited levels) used to realize them and the parameters ruling the tunneling interaction between them. One of the most important features of the proposed model is that, by considering a system composed by a very high number of interacting water coherent domains, in turn oscillating in phase with each other (i.e. they are “supercoherent”), it could be possible to realize an extended network of quantum gates acting in parallel whose computational speed would be proportional to number of the matter components that join the common coherent oscillation and then much higher than the operational speed of a single quantum gate. Despite some aspects (like, for example, storing and retrieving of the quantum information and the study of other interaction mechanisms apart the tunneling of virtual photons already considered) would need further closer examinations, the proposed model opens very fascinating perspectives towards a new conception of quantum computing using water molecules as physical substrate. The possible applications of such model to the realization of a physical machine are hardly in progress and, when further improved, could be hugely interesting both from a

186

L. M. Caligiuri

theoretical and applicative standpoint, allowing the development of unimaginable technologies that would require extraordinarily high computational power (from the discover of new physical laws to artificial intelligence, from the simulation of high complex systems to the elaboration of forecasts in environmental, health and epidemiological fields, etc.).

References 1. Preparata, G.: QED coherence in matter. World Scientific, Singapore, London, New York (1995) 2. Caligiuri, L.M., Musha, T.: The Superluminal Universe: from Quantum Vacuum to Brain Mechanism and beyond. NOVA, New York (2016) 3. Caligiuri, L.M., Musha, T.: Quantum hyper-computing by means of evanescent photons, J. Phys. Conf. Ser. 1251(1), 012010 (2019) 4. Caligiuri, L.M., Musha, T.: Accelerated quantum computation by means of evanescent photons and its prospects for optical quantum hypercomputers and artificial intelligence. In: Proceedings of the 2019 International Conference on Engineering, Science, and Industrial Applications (ICESI), Tokyo, Japan, 22–24 August 2019. https://doi.org/10.1109/ICESI.2019.886 2999 5. Caligiuri, L.M.: Quantum (Hyper)computation by means of water coherent domains – Part I: the physical level. In: Caligiuri, L.M., (ed.), Frontier in Quantum Computing, NOVA, New York (2020) 6. Caligiuri, L.M.: Quantum (hyper)computation by means of water coherent domains – Part II: the computational level. In: Caligiuri, L.M., (ed.), Frontier in Quantum Computing, NOVA, New York (2020) 7. Del Giudice, E., Tedeschi, A.: Water and autocatalysis in living matter. Electromagn. Biol. Med. 28, 46–52 (2009) 8. Buzzacchi, M., Del Giudice, E., Preparata, G.: Coherence of the glassy state. Int. J. Mod. Phys. B 16(25), 3771–3786 (2001) 9. Llyod, S.: Ultimate physical limit to computation. Nature 406, 1047–1054 (2000) 10. Margolus, N., Levitin, L.B.: The maximum speed of dynamical evolution. Physica D 120(1–2), 188–195 (1998) 11. Del Giudice, E., Spinetti, P.R., Tedeschi, A.: Water dynamics at the root of metamorphosis in living organisms. Water 2, 566–586 (2010) 12. Caligiuri, L.M.: Super-coherent quantum dynamics of zero-point field and superluminal interaction in matter. In: Amoroso, R.L., Kauffman, L.H., Rowlands, P., Albertini, G. (eds.) Unified Field Mechanics II: Formulation and Empirical Tests, pp. 331–343. World Scientific, Singapore, London, New York (2018) 13. Caligiuri, L.M., Musha, T.: Superluminal photons tunneling through brain microtubules modelled as metamaterials and quantum computation. In: Tiwari, A., Arul Murugan, N., Ahula, R., (eds.) Advanced Engineering Materials and Modeling, pp. 291–333. Wiley Scrivener Publishing LLC, New Jersey (2016) 14. Caligiuri, L.M.: The Quantum Phase Operator and its Role in Quantum Computing. In: Caligiuri, L.M., (ed.), Frontier in Quantum Computing, NOVA, New York (2020) 15. Bono, I., Del Giudice, E., Gamberale, L., Henry, M.: Emergence of the coherent structure of liquid water. Water 4, 510–532 (2012) 16. Caligiuri, L.M.: A new quantum – relativistic model of tachyon. J. Phys. Conf. Ser. 1251, 012009 (2019) 17. Nimitz, G.: On virtual phonons, photons and electrons. Found. Phys. 39, 1246–1355 (2009)

Accelerated Quantum Computation Based on Quantum Coherent Dynamics

187

18. Enders, A., Nimitz, G.: 1992 On superluminal barrier traversal, J. Phys. I, France 2, 1693 (1992) 19. Andrews, D., Bradshaw, D.S.: The role of virtual photons in nanoscale photonics. Ann. Phys. 3–4, 173–186 (2014) 20. Nielsen, M.A., Chuang, I.L.: Quantum Computation and Quantum Information (10th, Anniversary Cambridge University Press, Cambridge (2016) 21. Le Bellac, M.: A short Introduction to Quantum Information and Quantum Computation. Cambridge University Press, Cambridge (2006) 22. Franson, J.D., Jacobs, B.C., Pittman, T.B.: Quantum computing using single photons and the Zeno effect. Phys. Rev. 70, 062302 (2004) 23. Brizhik, L., Del Giudice, E., Jorgensen, S.E., Marchettini, N., Tiezzi, E.: The role of electromagnetic potentials in the evolutionary dynamics of ecosystems. Ecol. Model. 220, 1856–1869 (2009)

Computer Scientist’s and Programmer’s View on Quantum Algorithms: Mapping Functions’ APIs and Inputs to Oracles Ilie-Daniel Gheorghe-Pop1(B) , Nikolay Tcholtchev1 , Tom Ritter1 , and Manfred Hauswirth1,2 1 Fraunhofer Institute for Open Communication Systems (FOKUS) Kaiserin-Augusta-Allee 31,

10581 Berlin, Germany [email protected] 2 Technische Universität Berlin, Straße des 17. Juni 135, 10623 Berlin, Germany

Abstract. Quantum Computing (QC) is a promising approach which is expected to boost the development of new services and applications. Specific addressable problems can be tackled through acceleration in computational time and advances with respect to the complexity of the problems, for which QC algorithms can support the solution search. However, QC currently remains a domain that is strongly dominated by a physics’ perspective. Indeed, in order to bring QC to industrial grade applications we need to consider multiple perspectives, especially the one of software engineering and software application/service programming. Following this line of thought, the current paper presents our computer scientist’s view on the aspect of black-box oracles, which are a key construct for the majority of currently available QC algorithms. Thereby, we observe the need for the input of API functions from the traditional world of software engineering and (web)services to be mapped to the above mentioned black-box oracles. Hence, there is a clear requirement for automatically generating oracles for specific types of problems/algorithms based on the concrete input to the belonging APIs. In this paper, we discuss the above aspects and illustrate them on two QC algorithms, namely Deutsch-Jozsa and the Grover’s algorithm. Keywords: Quantum computing · Grover’s algorithm · Oracle · API

1 Introduction The recent developments in Quantum Computing have brought about a great deal of interest in the domain. Apart from increased activity in the scientific community via thematic conferences, publications and workshops, invested companies as well as universities publish educational material like online courses [1] and promote ambassadors at an international level. Another metric that can serve as a good indicator for increased activity is the number of patents published on the topic of Quantum Computing [2], which is constantly increasing. From the perspective of hardware advances and breakthroughs, companies have presented plans to increase the number of Qbits per processor tenfold over the next three years [3]. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 188–203, 2022. https://doi.org/10.1007/978-3-030-80119-9_9

Computer Scientist’s and Programmer’s View on Quantum Algorithms

189

The massive increase in interest attracts investors and researchers alike. As QC promises technological advantage in multiple domains, old as well as new quantum algorithms have been implemented and tested on the various newly accessible Quantum hardware resources. Starting with the Deutsch-Josza and Grover’s algorithms, many new other implementations targeting advanced applications such as molecular chemistry, traffic optimization and finance have since been tested. Most of the proposed algorithms that have a significant theoretical advantage over classical algorithms such as logarithmic or quadratic speed-ups require a so called quantum oracle. In this paper we focus on this element in particular and analyze its functionality and characteristics from a computer scientist’s and programmer’s perspective. We would like to focus on: (1) How such a quantum oracle circuit is implemented?, (2) How this process takes place ? and (3) What is required, in order to achieve programmers’ like APIs for Quantum Computing ? Answering these questions will help bring a more clear perspective on the advantages and challenges for the future QC applications. Additionally, this work is aimed at providing the key concepts and approach towards the development of applications using a quantum oracle. Moreover, this research is aimed at further reducing the knowledge gap for computer scientists interested in the field of QC applicability. The rest of this paper is organized as follows: Sect. 2 provides a general overview of the history of Quantum Computing as well as previous and current work related to the relevant algorithms. Section 3 attempts a formulation of the current challenges for applying quantum computing to real-world applications, with a discussion relating to quantum oracles. Sections 4 and 5 provide a brief description of the Deutsch-Josza and Grover’s algorithms. Section 6 relates to quantum oracles, their characteristics and provides our view on the implementation process of such oracles. Section 7 introduces a more in-depth look into quantum oracle implementation for the Grover’s algorithm and its’ implementation implications. Next, Sect. 8 describes our current view and approach to the Quantum Oracle implementation research challenges, summarizes the current work and provides a series of conclusions.

2 History and Related Work 2.1 Quantum Computing Quantum Computing has been receiving significantly increased attention over the past two years. However, the first quantum computer proposal dates back to 1980, when Manin first mentioned such an apparatus [4] and Benioff proposed a model of a Turing Machine based on quantum mechanics [5]. Around this time, Feynman proposed the simulation of physical processes using quantum computers [6] due to the limitations of classical computing at the time. Shortly afterwards Benioff proposed using quantum computation for reversible computing [7], which is a paradigm allowing to always identify the correct input based on the available results.

190

I.-D. Gheorghe-Pop et al.

2.2 Quantum Algorithms It was not until 1992 when quantum oracles were applied in a quantum algorithm by Deutsch and Josza demonstrating an advantage over classical algorithms. In this case, the oracle represented a black-box function f(x) that could be either constant or balanced. The Deutsch-Josza algorithm [8] implemented around the oracle was later improved [9] to determine the result in only one query. Another type of oracle was used by Grover in 1996 to obtain a quadratic speedup over classical computing in unstructured database searches [10]. The oracle used in this case behaves as a binary output solution validator, which means that it can influence the quantum algorithm result depending on whether or not a proposed solution in an oracle query is the correct one with a certain probability. First algorithm implementations on quantum hardware followed soon after Grover’s publication with an experimental demonstration in 1998 [11] using a nuclear magnetic resonance (NMR) machine. Over the next decade (2000–2010), several hardware technologies were used, in order to demonstrate quantum computation. These technologies ranged from photonics [12], trapped ions [13], nuclear magnetic resonance and superconductors [14]. In the current decade (2010–2020), several technological advancements enabled practical realization and commercialization of quantum hardware. Nowadays, quantum computers are already commercially available through cloud quantum computing providers [15–21]. At the same time, open source programming languages and libraries have become available for the new generation of QC software developers [22]. 2.3 Quantum Oracles Since the first results and implementations of the Grover’s algorithm, the scientific community has further improved and expanded upon the applications and developments of quantum oracles. Most notable among these are the applications into cryptography [23], constraint optimization problems [24, 25], new materials [26] and of course database searches [27]. However, there is much more work expected and required on the further development and implementation of oracles. Bringing QC to effective real-world applications requires from the current standpoint feasible methodologies for developing, deploying and automatically generating quantum oracles, such that the quantum algorithms and expected speed-ups can be made applicable to relevant problems.

3 Problem Statement Quantum Computing hardware and software technology is currently accelerating with respect to its development [28]. The current proposed applications and underlying algorithms however are based on the presumption of a pre-existing quantum oracle implementation. Whether the target application domain is logistics [30], molecular and material chemistry [31], cryptography [32] or finance [29], the translation of the real-world problem to its’ corresponding quantum algorithm requires significant efforts – very often because of the need to construct a specific quantum oracle for each particular problem instance.

Computer Scientist’s and Programmer’s View on Quantum Algorithms

191

The increasing number of technological advancements on the hardware side, as well as new approaches to the NISQ computing makes it difficult to have a stable methodological process to quantum algorithm implementations. Noisy Intermediate-Scale Quantum (NISQ) stands for the effort of utilizing error prone qbits for computations thereby exploiting the specific nature of algorithms and implementing procedures such as qbit error-correction and error-mitigation. Furthermore, the knowledgebase requirements for deriving and implementing quantum oracles from classical problem formulations are advanced mathematics, statistics and physics, which is often not in-depth available on the side of the programmers. Indeed, we see the following challenges with regard to quantum algorithms, programmers’ APIs and quantum oracles: 1. There is a strong need for clear approaches to mapping classical functions, APIs and their inputs to quantum oracles and belonging quantum circuits. 2. The process of oracle creation as well as specific parametrization in relation to its’ embedding into quantum circuits needs to become an automatic process. 3. A comprehensive model of the target problems needs to be established, i.e. to model the problem class in question, to list its characteristics and to understand the complexity of deriving the belonging quantum oracles. 4. A process for modelling and defining of the specific associated quantum oracles needs to be outlined. This process should be automated as far as possible. We try to illustrate the above problems by starting with two of the iconic quantum algorithms (Deutsch-Jozsa and Grover’s) and their associated quantum oracles. Thereby, we aim at elucidating and illustrating the above listed problem statements, which in our view is one of the paramount steps towards bringing Quantum Computing to real life products and services.

4 Deutsch-Jozsa Algorithm The Deutsch-Jozsa Algorithm was first published in 1992 [8] proposing a class of problems that would be faster solved using QC than classical computing. The principle employed to achieve this speed-up is called “quantum parallelism” [33, 34], which stands for evaluating multiple inputs at the same time. The exact problem statement according to [8]: Let N be a natural number and Uf an oracle for a function f: Z2N → Z2. Identify the true statement from (1) and (2): (1) f is not a constant function (at 0 or 1) N (2) The sequence f(0), ... ,f(2 -1) of values of f does not contain exactly N zeros At least one of the two statements is true at a time. It is possible that (1) and (2) are both true.

192

I.-D. Gheorghe-Pop et al.

In other words, given an unknown function that takes as input 2n bits and gives outputs of 0 or 1, determine if the function is either constant or balanced. Clearly, this problem and the belonging algorithms do not have any great applications in real world scenarios. However, this was one of the first problems, for which a quantum computer intrinsically performs better than a classical computer. A classical computing solution would be to evaluate the function 2n−1 + 1 times, in order to determine if it is balanced or constant given the only two possible conclusions. In contrast, the corresponding type of quantum computation is called “Black Box Model” [39] where the computation output is obtained by querying a black-box also known as an oracle in the QC context. The complexity of the computation is derived from the number of required queries to the oracle together with the input preparation complexity. In the case of the DJ Algorithm, the complexity computed by the authors is O(ln(N)) to set up the input and polynomial for solving the problem. The algorithm is using quantum superposition, quantum entanglement and quantum parallelism to achieve this improvement over classical computing.

Fig. 1. Deutsch-Josza algorithm quantum circuit [9, 41]

In later years, an extended generalized version of the DJ algorithm was proposed [9]. It requires only one evaluation of the function f . The QC implementation of the generalized DJ algorithm available in [16] is depicted in Fig. 1. According to [9], we consider the case for a binary function f:{0,1}⊕n → {0,1}, with {0,1}⊕n being the set of all possible combinations of 0 and 1 with length n and with x•y = x0 y0 ⊕ x1 y1 ⊕ … ⊕ xn−1 yn−1 being the sum of the bitwise product for vectors x and y (as parameters on the left side in the oracle box). The circuit in Fig. 1 consists of n qbits set to |0 and one auxiliary qbit set to |1 as input - i.e. we have a start input vector |ψ 0  = |0⊗n |1 which is then put into superposition using Hadamard gates (H ⊕n ) as follows. |ψ1  = √

n -1 2

1 2n + 1

|x(|0 − |1)

x=0

The next step consists of applying the oracle function, which may be either constant or balanced: |ψ2  = √

1 2n + 1

n -1 2

x=0

|x(|f (x) − | 1 ⊗ f(x) ) =

Computer Scientist’s and Programmer’s View on Quantum Algorithms

193

and as f(x) is either 0 or 1: = √

n -1 2

1 2n + 1

|− 1f(x) |x (|0 − |1)

x =0

Finally, the qbits are rotated back to the original states using Hadamard gates which results in the following vector [35]: ⎤ ⎡n n −1 2 −1 2 1 |ψ3  = n |−1f (x) ⎣ (−1)x∗y |y⎦ 2 y=o x=0 ⎡ ⎤ n −1 2n −1 2  1 ⎣ = n (−1)f (x) (−1)x∗y ⎦ |y 2 y=0

x=0

This final value is measured and yields either 0 or 1 as an answer. After repeating this process for a number of shots, we arrive at a probability distribution which characterizes the function f:{0,1}⊕n → {0,1} in terms of it being constant or balanced. As for the oracle of the solution, according to the problem, this can be either constant or balanced. While the only purpose of this problem was to demonstrate a case for quantum advantage, the type of the oracle represented is still a matter of interest to the scientific community. One such work [35] refers to comparing quantum oracles for the DJ algorithms as well as one application for quantum set intersection [36].

5 Grover’s Algorithm The Grover’s algorithm [10] was first published in 1996 and represents a quantum version of the search through an unsorted database. A possible corresponding algorithm from traditional computing is given by the simple linear search. From the classical perspective, given an unsorted dataset of N elements, it would take on average N/2 iterations to find a specific value. Within the quantum world however, an oracle is utilized. This black-box oracle holds the promise that given a dataset element index as input, it will check the element value against the searched item, and return that index value as marked. The marking is represented by flipping the corresponding qbit state from |1 to −|1. Using this oracle conjoined with a phase amplification of the marked state sequence (also referred as a diffusion step), the Grover’s √ algorithm returns the marked state as output precisely enough after approximately N iterations. Simple representations of 2 qbit oracles for the Grover’s algorithm are depicted in Fig. 2. These oracles are responsible for marking different values (e.g. |00, |10 …). The specific Grover’s algorithm for two qbits and the |00 oracle is depicted in Fig. 3. In the following we explain the structure of the circuit in Fig. 3 thereby relating to the oracles described above. The oracle circuits (see Fig. 2) depict the initial steps of the Grover’s algorithm with the belonging oracles attached.

194

I.-D. Gheorghe-Pop et al. Oracle 00

q0

H

X

q1

H

X

Oracle 10

Z

X

q0

H

X

q1

H

H

q1

H

Z

X

Oracle 11

Oracle 01

q0

X

X

X

Z

q0

H

q1

H

Z

Fig. 2. Grover oracles for 2 Qbit inputs [41]

• The process starts with putting the qbits q0 and q1 in a superposition. • The following step is the application of the corresponding oracle. • For the state |00 this yields the following equation:   1 1 U00 |s = U00 (|00 + |01 + |10 + |11) = (−|00 + |01 + |10 + |11 2 2

q0

H

X

q1

H

X

Z

X

H

Z

X

H

Z

H

Z

H

Fig. 3. 2-Qbit grover algorithm circuit with a |100 marking oracle [41]

After the belonging oracle from Fig. 2 has been executed, the algorithm has to apply a so called diffusion step, which basically amplifies the qubits’ states in a way that the search space is correspondingly explored (see the right side of Fig. 3). • In preparation for the diffusion step, it is required to reapply the reversible Hadamard gate to all qbits (i.e. H⊗n |s) such that we project the qbits – except the marked one – to their original state. • Next, we apply the diffusion operator U s = 2|ss|−1 so that we can further explore the search space before starting another iteration with the oracles above. A simple way to describe this operation would be subtracting each input coefficient from twice the average. In the case of the marked state, since it is flipped and negative, its absolute value √ will be increased, while for the others, it will be decreased. • After N iterations (i.e. one iteration in case of 2 qbits and |00) the solution of the search problem has been identified precisely enough.

Computer Scientist’s and Programmer’s View on Quantum Algorithms

195

The Grover’s algorithm as described above has expanded its application domain since the first publication and consequently more similar algorithms have been applied in quantum cryptography [32, 37] and optimization [38]. All of the variations make use however of a quantum oracle. In the next section we will try to elucidate on the functionalities and characteristics of quantum oracles in general.

6 Quantum Oracles The notion of an oracle is a very often utilized construct for quantum algorithms. The main presumption in this context is that we have a black box at our disposal, that can answer questions relating to whether a particular proposed solution for a complex problem is indeed satisfying the predefined criteria. Indeed, oracles are used to span a whole group of quantum algorithms, which are called black-box models [39], where the quantum computing properties are basically employed to optimize the way questions are asked to the corresponding black box oracle. 6.1 The Role of Quantum Oracles It is important to note that the oracle itself doesn’t bear the capability to solve the problem in question. Hence, it is not possible to ask a query of the type “Please give me a solution to the problem” – the queries which are possible resemble the following type of questions “I have a proposed solution, can you tell me if it is also a solution of the problem?”. Thus, the quantum computations in this line of thought are used to systematically generate new proposed potential solutions, for which the oracle decides if they satisfy all conditions/constraints for being indeed a solution to the problem in question. Indeed, the oracle corresponds to the concept of decision problems [40] from the scientific field of complexity theory, where a decision problem is defined as the possibility to solve particular problem requiring a “yes-or-no” answer. However, the theory around decision problems is more complicated as the notion of quantum oracles, given the aspects that algorithms for decision problems cannot always produce a clear “yes-or-no” depending on the structure and complexity of the specific problem. Based on the above elucidations, Fig. 4 moves on to describe our attempt to provide a process flow structure of a black-box based oracle algorithm and the belonging processes. Each algorithm starts with the initialization of the qbit states. Then, the qbits are brought to a superposition state using Hadamard gates. Afterwards a mathematical processing of the super positioned qbits is conducted using Hadamard and C-NOT gates, in order to systematically generate a potential problem solution. The mathematical processing is the complicated part where a lot of research efforts flow in the solution design resulting in corresponding breakthroughs in quantum algorithms. Having brought the qbits in superposition states representing potential possible solutions, the black-box oracle is then queried. The result leads to an increase in the probabilities for the correct solution(s), whilst decreasing the probabilities for other incorrect candidates. Finally, the algorithm checks if the termination criteria are satisfied which are typically given by the pre-specified number of iterations on the circuit/gate based

196

I.-D. Gheorghe-Pop et al.

quantum computers. If the criteria are not satisfied, the algorithm moves back to generating the next solution candidate followed by the subsequent query of the oracle and resulting probability distribution modifications. In case the termination criteria are met, the final result distribution is compiled and presented as a final result of the algorithm execution.

Fig. 4. Rough structure of black-box oracle based quantum algorithms

6.2 Quantum Oracles as Input Representation Bearing in mind the above description, we can clearly see that the substantial input to such algorithms is hidden within the oracle. Relating back to traditional functions and APIs and taking as an example the linear search of an array that corresponds to the quantum Grover’s algorithm, we can establish the following understanding. The functional signatures of the legacy linear search function API can be written pseudo code wise as follows. int linearSearch (int valueToLookFor, int arrayToSearchIn [])

for an integer value to search for in an integer array and hence return the index of the value in the array. The linear search can be implemented as follows. for (int i=0; i c[0]; measure q[1] -> c[1]; measure q[2] -> c[2]; // Measurement

From a computer scientist perspective, the oracle and amplification circuits can be grouped and reused as separate functions. When the code is put together for a specific algorithm, the number of amplifications can be pre-defined based on a formula and then the whole circuit can be generated. An example for such functions is depicted below using Python pseudocode. Thereby, we have formulated the Grover’s algorithm in a way that it relies on said abstract functions, which are ideally automatically generated based on the API inputs. In addition, the oracle should also be automatically generated based on the inputs, which is research challenge to be addressed for classes of programs within the near future. grover_amplification(nQbits) qc = quantum_circuit(nQbits) for qubit in range(nQbits): qc.add_Hadamard_gate(qbit); qc.add_NOT_gate(qbit); qc.add_CZ_gate(nQbits) for qubit in range(nQbits): qc.add_NOT_gate(qbit); qc.add_Hadamard_gate(qbit); return qc grover_search(nQbits, oracle): qc = quantum_circuit(nQbits) qc.initialise_states(nQbits) # Puts all input Qbits in superposition # calculates the number of needed amplifications k = calculate_num_amplification(nQbits) for iterations in range(k): qc.append(oracle) # adds the oracle qc.append(grover_amplification(nQbits)) results = qc.measure(nQbits) return results

In order to be able to create the oracle circuit, which is depicted as a parameter in the above code, there are several alternatives. In the case of Grover’s search over 3-qbit input, simple oracles can be created in advance for all possible states. Presently, we can only presume that varied generators will be implemented for specific classes of problems like the SAT problem [45]. Based on these considerations and the examples above, the next section concludes our thoughts and draws future research challenges.

200

I.-D. Gheorghe-Pop et al.

8 Conclusions and Final Thoughts Oracle construction and moreover optimized oracle construction will remain challenging for the foreseeable future of Quantum Computing technology. Nevertheless, advancements are made at a very rapid pace. Recent work has also proposed canonical construction of Quantum Oracles [46]. Additionally, there are also works on encoding classical data to quantum registers [47, 48]. In this paper we introduced the current challenges in applying Quantum Computing to real-world problems focusing on quantum oracle implementation challenges. We briefly introduced the DJ and Grover’s algorithm and noted the characteristics of their specific oracles. We then focused on the specifics of the quantum oracle implementation for the generalized Grover’s algorithm and presented a simple approach to its implementation. This leads to the identification of current research challenges with respect to quantum oracle implementations, which will constitute a major challenge towards the accessibility of quantum computing resources over programmers’ APIs. While higher level programming languages allow for a faster implementation of algorithms with many aiding features such as state-vector visualization and phase estimation during the process of building the circuit [22], computer scientists need to solve some challenges before executing their code. Apart from the challenge of mapping a problem to an oracle, the programmer needs to also take into consideration the currently constrained hardware architecture. Moreover, given a particular quantum instance, the circuit needs to be efficiently embedded onto a QPU in order to obtain relevant results. Last but not least, given the current NISQ resources, the target running infrastructure needs to be properly selected. According to our previous work, this challenge can be met by using Quantum DevOps [49] principles. In our vision, the processes of mapping classes of classical problem specific formulations to quantum oracles will be done automatically. Additionally, various features will be available for optimizing the resulted circuits to different QC platforms. Moreover, the intermediate layer between a classical problem and its’ quantum algorithm correspondent should be agnostic to programming languages and support various translating APIs, so that each problem can be tackled using different quantum approaches (annealing/universal) and quantum computing instances. Finally, from our perspective, the upcoming work on efficiently mapping classical problems to quantum oracles will be a key element to bringing the full potential of QC to production.

References 1. QOSF Learning Resources on Quantum Computing: https://qosf.org/learn_quantum/. Accessed 20 Oct 2020 2. Haney, B., Quantum Patents (March 16, 2020). 27 B.U. J. Sci. Tech. L. (2020). (Forthcoming). Available at SSRN: https://ssrn.com/abstract=3554925 3. IBM Quantum Roadmap: https://www.ibm.com/blogs/research/2020/09/ibm-quantum-roa dmap/. Accessed 20 Oct 2020 4. Manin, Yu.I.: Computable and Noncomputable Sov. Radio (1980) 5. Benioff, P.: The computer as a physical system: a microscopic quantum mechanical hamiltonian model of computers as represented by turing machines. J. Stat. Phys. 22(5), 563–591 (1980). https://doi.org/10.1007/bf01011339

Computer Scientist’s and Programmer’s View on Quantum Algorithms

201

6. Feynman, R.P.: Simulating physics with computers. Int. J. Theore. Phys. 21, Nos. 6/7 (1982) 7. Benioff, P.: Quantum mechanical models of turing machines that dissipate no energy. Phys. Rev. Lett. 48(23), 1581–1585 (1982) 8. Deutsch, D., Jozsa, R.: Rapid solutions of problems by quantum computation. Proc. R. Soc. London, 439, 53–558 (1992). https://doi.org/10.1098/rspa.1992.0167 9. Cleve, R., Ekert, A., Macchiavello, C., Mosca, M.: Quantum algorithms revisited. Proc. R. Soc. London 454(1969), 339–354 (1998). https://doi.org/10.1098/rspa.1998.0164 10. Grover, L.K.: A fast quantum mechanical algorithm for database search. In: Proceedings, 28th Annual ACM Symposium on the Theory of Computing, p. 212, May 1996 11. Chuang, I.L., Gershenfeld, N., Kubinec, M.: Experimental implementation of fast quantum searching. Phys. Rev. Lett. 80 (15), 3408–3411 (1998). https://doi.org/10.1103/PhysRevLett. 80.3408 12. O’Brien, J.L., Pryde, G.J., White, A.G., Ralph, T.C., Branning, D.: Demonstration of an alloptical quantum controlled-NOT gate. Nature 426(6964), 264–267 (2003). https://doi.org/10. 1038/nature02054 13. Gulde, S., et al.: Implementation of the Deutsch-Jozsa algorithm on an ion-trap quantum computer. Nature 421(6918), 48–50 (2003). https://doi.org/10.1038/nature01336 14. Clarke, J., Wilhelm, F.K.: Superconducting quantum bits. Nature. 453(7198), 1031–1042 (2008). https://doi.org/10.1038/nature07128 15. Last, T., et.al.: Quantum Inspire: QuTech’s platform for co-development and collaboration in quantum computing. In: Proceedings SPIE 11324, Novel Patterning Technologies for Semiconductors, MEMS/NEMS and MOEMS 2020, 113240J (2020). https://doi.org/10.1117/12. 2551853 16. IBM Q-Experience: https://www.ibm.com/quantum-computing/. 15 July 2020 17. Rigetti: https://rigetti.com/. 15 July 2020 18. D-Wave: https://www.dwavesys.com/services. Accessed 15 July 2020 19. Bristol QCloud, Available: http://www.bristol.ac.uk/physics/research/quantum/engagement/ qcloud/. Accessed 15 Sept 2020 20. Quantum Playground: http://www.quantumplayground.net/. Accessed 15 Sept 2020 21. Amazon Braket: https://aws.amazon.com/braket/. Accessed 15 Sept 2020 22. List of open source QC Software: https://github.com/qosf/awesome-quantum-software. Accessed 15 Sept 2020 23. Jaques, S., Naehrig, M., Roetteler, M., Virdia, F.: Implementing grover oracles for quantum key search on AES and LowMC. In: Canteaut, A., Ishai, Y. (eds.) EUROCRYPT 2020. LNCS, vol. 12106, pp. 280–310. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-457242_10 24. Gilliam, A., Woerner, S., Gonciulea, C.: Grover Adaptive Search for Constrained Polynomial Binary Optimization, arXiv e-prints (2019) 25. Baritompa, W., Bulger, D., Wood, G.: Grover’s quantum algorithm applied to global optimization. SIAM J. Optim. 15, 1170–1184 (2005). https://doi.org/10.1137/040605072 26. Borujeni, S.E., Harikrishnakumar, R., Nannapaneni, S.: Quantum Grover search-based optimization for innovative material discovery. In: 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, pp. 4486–4489 (2019). https://doi.org/10.1109/Big Data47090.2019.9006454 27. Gilliam, A., Pistoia, M., Gonciulea, C.: Optimizing Quantum Search Using a Generalized Version of Grover’s Algorithm, arXiv e-prints (2020) 28. Huang, H., Wu, D., Fan, D., Zhu, X.: Superconducting Quantum Computing: A Review (2020). arXiv:2006.10433 29. Venturelli, D., Kondratyev, A.: Reverse Quantum Annealing Approach to Portfolio Optimization Problems, Papers, arXiv.org (2018). https://EconPapers.repec.org/RePEc:arx:pap ers:1810.08584

202

I.-D. Gheorghe-Pop et al.

30. Kieu, T.D.: The travelling salesman problem and adiabatic quantum computation: an algorithm. Quantum Inf. Process. 18(3), 1–19 (2019). https://doi.org/10.1007/s11128-0192206-9 31. Elfving, V.E.: How will quantum computers provide an industrially relevant computational advantage in quantum chemistry? arXiv e-prints (2020) 32. Boneh, D., Dagdelen, Ö., Fischlin, M., Lehmann, A., Schaffner, C., Zhandry, M.: Random oracles in a quantum world. In: Lee, D.H., Wang, X. (eds.) ASIACRYPT 2011. LNCS, vol. 7073, pp. 41–69. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25385-0_3 33. Deutsch, D.: Quantum theory, the church–turing principle and the universal quantum computer. In: Proc. R. Soc. London. A. Math. Phys. Sci. 400(1818), 97–117 (1985) 34. Richard, J.: Characterizing classes of functions computable by quantum parallelism. Proc. R. Soc. London. Ser. Math. Phys. Sci. 435(1895), 563–574 (1991) 35. Kiss, A., Varga, K.: Comparing two quantum oracles using the Deutsch-Jozsa algorithm. Ann. Univ. Sci. Budapest., Sect. Comp. 50 199–217 (2020) 36. Salman, T., Baram, Y.: Quantum set intersection and its applicationto associative memory. J. Mach. Learn. Res. 13, 3177–3206 (2012) 37. Nagata, K., Nakamura, T.: The Deutsch-Jozsa algorithm can be used for quantum key distribution. Open Access Libr. J. 2, e1798 (2015) 38. Alexandru, C.-M., Bridgett-Tomkinson, E., Linden, N., MacManus, J., Montanaro, A., Morris, H. Quantum speedups of some general-purpose numerical optimisation algorithms. Quant. Sci. Technol. 5(4) (2020). https://doi.org/10.1088/2058-9565/abb003 39. Mosca, M.: Quantum Algorithms, arXiv:0808.0369, submitted: Mon, 4 August 2008. https:// arxiv.org/abs/0808.0369 40. de Jonge, G., Sprij, A.: 10. In: 36 zieke kinderen, pp. 39–42. Bohn Stafleu van Loghum, Houten (2012). https://doi.org/10.1007/978-90-313-8424-2_10 41. Nannicini, G.: Quantum Computing, Lecture 5 (2019). https://researcher.watson.ibm.com/ researcher/files/us-nannicini/8100_lecture_5.pdf 42. Figgatt, C., Maslov, D., Landsman, K.A., et al.: Complete 3-Qubit grover search on a programmable quantum computer. Nat Commun 8, 1918 (2017). https://doi.org/10.1038/s41467017-01904-7 43. Strömberg, P., Blomkvist Karlsson, V.: 4-qubit Grover’s algorithm implemented for the ibmqx5 architecture (Dissertation) (2018). http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva229797 44. Mandviwalla, A., Ohshiro, K., Ji, B.: Implementing Grover’s algorithm on the IBM quantum computers. In: 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA, pp. 2531–2537 (2018). https://doi.org/10.1109/BigData.2018.8622457 45. Wang, P., Liu, G., Liu, L.: A generic variable inputs quantum algorithm for 3-sat problem. In: 2020 IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA), Dalian, China, pp. 308–312 (2020). https://doi.org/10.1109/AEE CA49918.2020.9213471 46. Gilliam, A., Pistoia, M., Gonciulea, C.: Canonical Construction of Quantum Oracles, arXiv: 2006.10656 (2020) 47. Dhawan, S., Perkowski, M., comparison of influence of two data-encoding methods for grover algorithm on quantum costs. In: 41st IEEE International Symposium on Multiple-Valued Logic. Tuusula 2011, 176–181 (2011). https://doi.org/10.1109/ISMVL.2011.29 48. Tsai, E., Perkowski, M.: A Quantum Algorithm for Automata Encoding. https://doi.org/10. 2298/FUEE2002169T

Computer Scientist’s and Programmer’s View on Quantum Algorithms

203

49. Gheorghe-Pop, I.D., Tcholtchev, N., Ritter, T., Hauswirth, M.: Quantum devops: towards reliable and applicable NISQ quantum computing (forthcoming). In:12 IEEE GLOBECOM 2020 50. Bertels, K., Almudever, C.G., Hogaboam, J.W., Ashraf, I., Guerreschi, G. G., Khammassi, N.: (2018–05–24). cQASM v1.0: Towards a Common Quantum Assembly Language. arXiv: 1805.09607v1 (2010)

Orca: A Software Library for Parallel Computation of Symbolic Expressions via Remote Evaluation on MPI Systems Ahmet Artu Yıldırım(B) Issaquah, USA

Abstract. This study describes a Scheme library, named Orca, which is used to compute symbolic expressions in parallel via remote evaluation based on the message-passing interface (MPI) standard. Today, MPI is one of the most used standards, in particular high-performance computing systems. However, MPI programmers are explicitly required to deal with many complexities that render MPI programming hard to reason about. We designed and implemented a set of new APIs to alleviate this complexity by taking advantage of the expressive power of Scheme language using remote evaluation techniques on MPI systems. We introduce the application programming interface (API) of the library and evaluate the implemented model on a real-world application of a common parallel algorithm. Our experiments show that it is practical and useful for a variety of applications to exploit multiple processors of a distributed-memory architecture. Keywords: Parallel computation · Symbolic expression evaluation · MPI · Message-Passing Interface · Scheme

1

· Remote

Introduction

High-performance computing (HPC) has been gaining much attraction for decades among research institutes and industries. Powerful computing capabilities of HPC systems enable us to collectively solve a diverse range of complex problems by leveraging many interconnected processors communicating over fast network mediums. MPI has been a well-known and widely used standard in the high-performance computing community to develop parallel algorithms on computer clusters as well as single machines [8]. MPI provides an abstraction for inter-communication by passing messages between processes through cooperative operations across different machine and network architectures, thus promoting portability and ease of use [15]. A. A. Yıldırım—Independent Researcher c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 204–217, 2022. https://doi.org/10.1007/978-3-030-80119-9_10

Orca: Parallel Computation of Symbolic Expressions on MPI

205

However, parallel programming using MPI is not without complications, requiring programmers to deal with many complexities including multiple code branches in code, difficulty in designing deadlock-free programs, buffer management, and the side-effect nature of the APIs. As a special case of message-passing model, the remote procedure call (RPC) mechanism provides a synchronized type-safe communication between two processes, typically called as client and server processes [22], giving the “illusion” to programmers that the procedure is invoked locally that makes it easy to reason about. Furthermore, remote evaluation (REV) generalizes remote procedure calls by providing the ability to evaluate a program expression at a remote computer [20]. In this technique, a program statement is sent from one computer, client, to another computer called server. When the server receives a REV request, it extracts the procedure and arguments from the message, evaluates the procedure with arguments, and returns the result message to the client. In this work, we present a programming model for the parallel computation of symbolic expressions and implement it as a Scheme [21] library, named Orca. The library allows programmers to evaluate the given symbolic expressions in parallel on multiple processes. In this model, the procedure and arguments of the expression are distributed to processes in different data partitioning methods, by either broadcasting or scattering according to the used API. We make use of functional programming techniques through remote evaluation abstractions, on top of the synchronous and asynchronous communication capabilities of MPI, to alleviate the complexity in parallel programming on MPI systems. The rest of the paper is organized as follows. Section 2 presents the background and motivation of this study. In Sect. 3, we describe the computational model of the implemented library and describe the functionality of the APIs. Section 4 presents an empirical program developed using the library and performance evaluation of turnaround time in broadcasting operation for varying numbers of processors and data sizes. Finally, we conclude in Sect. 5 by discussing future research directions.

2 2.1

Background and Motivation What is Wrong with MPI?

From the perspective of programming, general problems encountered in MPI programs include that of overhead in the decomposition, development, and debugging of parallel programs [19]. We argue that the aforementioned complexity arises largely due to the following reasons: • Multiple Code Branches exist to control the flow of the algorithm per process group in many MPI programs using if-else constructs. One common model is the master/worker pattern [16] which is commonly employed through explicit two groups of conditional branches: one set of conditional blocks for the master process that orchestrates the computation, and performs work distribution; and another set of conditional blocks for worker processes performing

206

A. A. Yıldırım

the common task. Having such segregated execution flows result in complexity in MPI programs and thus making it hard to write and maintain parallel programs. Such complexity exacerbates when the flow of the program is based on random data [4]. • Buffer Management is a fundamental aspect of MPI programming in C and Fortran as part of the standard [15] that programmers have been expected to allocate memory buffers with the specified data type, and then pass the pointers of buffers as arguments to the MPI collective and point-to-point primitives. Such memory management methods are prone to memory corruption and memory leakage, when not managed properly. There are wrapper libraries created that we found useful providing higher-level APIs on top of MPI that abstract away explicit memory management and allow using generic data types [3, 7, 17]. • Side Effects are inevitable in MPI due to its buffer-based communication nature. For example, the basic function MPI Recv accepts a memory buffer as an argument and, in return, fills the memory with the data received from the given process. This way of programming makes it highly susceptible to software bugs that arise when the process modifies the content of the returned buffer unintentionally. The functional programming paradigm solves this problem via pure functions which are idempotent and do not cause any side-effects, independent of the program’s execution environment in producing results, that considerably simplifies how the parallel programs are constructed [9, 11, 12]. • Deadlocks might occur when a process or process group attempts to receive data with no matching send operation causing the receiving counterpart to hang [18]. Finding deadlocks is notoriously difficult in MPI because of the low-level nature of the MPI APIs and high interactivity between processes during the execution of the program [4, 10, 13]. 2.2

Related Works

MPI for Python provides bindings of the Message Passing Interface (MPI) standard for the Python programming language, along with an object-oriented and relatively easier interface on top of the MPI-1/2/3 specifications [3]. In addition to the common MPI functionality, the library allows any Python objects to be communicated between processes. MPI Ruby is another library providing bindings of the MPI standard, exploiting Ruby’s dynamic nature [17]. Since everything is an object in Ruby language, messages are also interpreted as objects, converted to a byte stream that gives more flexibility for message transmission and easier programming than MPI’s C library. It is argued that such libraries do not only simplify the task of writing parallel programs but also make them ideal tools for rapid prototyping which is deemed essential in the scientific computing community. Avalon extends the client/server model and provides Common Lisp constructs for evaluating expressions remotely on any type of Common Lisp types with marshalling and unmarshalling capabilities [2]. Avalon supports transactional remote evaluation in which, when a crash occurs, the system recovers the state to the previous saved consistent state using atomic commits.

Orca: Parallel Computation of Symbolic Expressions on MPI

2.3

207

Symbolic Expression

A symbolic expression, in the context of Lisp (LISt Processor) programming language, is a program statement composed of either a single atomic data (i.e. symbol, number) or an ordered list of atomic data in the form of a binary tree that evaluates to a value [14]. What differentiates Lisp from most other programming languages is that Lisp treats “code as data” in the form of symbolic expressions that provides enormous flexibility for programmers through its advanced macro system. A simple example is as follows. (define input (list 1 2 3)) (map (lambda (x) (* x 3)) input) => (3 6 9)

In this example, map function evaluates to a list of numbers (3 6 9) as the result of the application of (* x 3) to each item in the input list (1 2 3). The function map is pure in the sense that it doesn’t modify any value of arguments, and therefore no side effects are introduced at runtime, so applying the same function to the same arguments always produces the same result without any change in the program state. Such methodology renders the program more concise and less prone to software bugs, with added benefits of improved testability, reliability, and maintainability [1].

3

Programming Model

The programming model of the Orca library is based upon the MPI standard, that defines the foundation for providing communication among a set of processes with separate memory spaces. In this model, many processes are typically launched across the nodes by the job scheduler system (i.e. SLURM [23]) on a computer cluster, or possibly even locally, in a single program, multiple data (SPMD) fashion where the computation differs based on the rank number uniquely identifying a process. Furthermore, our model categorizes processes into two types: master process, and worker process. A worker process waits for an incoming symbolic expression from the master process, executes it in its runtime environment, and then sends the result of the computation back to the master process. Besides, the master process runs a kernel program defining the main computation using the implemented APIs that control the distribution of symbolic expressions among the worker processes and collect the computation results from them into a list, ordered by rank number. Orca library allows all worker processes with separate memory spaces to collectively apply the specified function to the given arguments which are either broadcasted or partitioned among the processes in a way similar to MPI Bcast and MPI Scatter functions in the MPI standard, respectively.

208

A. A. Yıldırım

In Orca, we extend the remote evaluation paradigm by having multiple servers instead of one, and one client acting as a master process driving the execution of the computation, utilizing MPI’s asynchronous communication routines. The kernel code is the main Scheme code for applications, in which programmers implement a parallel program for executing in a computer cluster system as well as a single machine, ideally with multiple processor cores, by explicitly specifying different data partitioning strategies for function and arguments to be evaluated on worker processes in parallel. 3.1

Module Structure

We implement the library using Guile programming language, which is an implementation of the Scheme programming language. Guile conforms to R5RS, R6RS, and R7RS Scheme standards, as well as extends Scheme with a variety of features including a module system, dynamic linking and foreign function interface (FFI) [6].

Fig. 1. Communication layer of Orca library

We benefit from Guile’s module system, which is used to provide a “namespace” for a set of symbols bound to Scheme objects as well as logically organizes the Scheme symbols. Orca library consists of modules (orca) and (orca internal). Binding functions that act as an interface to the MPI C library procedures, along with internal functions reside in the (orca internal) module. On top of the internal module, the module (orca) implements all the public APIs that applications import to consume the APIs. Communication layers of the Orca library are depicted in Fig. 1.

Orca: Parallel Computation of Symbolic Expressions on MPI

209

(rpc-worker-process-id) → integer Listing 1.1. rpc-worker-process-id

(rpc-worker-process-size) → integer Listing 1.2. rpc-worker-process-size

3.2

Overview of APIs

In this section, we briefly explain the implemented APIs of the Orca library. We categorize the APIs into three classes: utility APIs such as rpc-worker-process-id and rpc-worker-process-size; bootstrapping APIs such as rpc-start, and compute APIs performing the given computation. As a utility API, worker processes can determine their id that uniquely determines every worker process using the API rpc-worker-process-id where process numbering begins with 0 and goes up to size − 1. Total number of worker processes (size) is determined via the API rpc-worker-process-size. To bootstrap the execution environment, rpc-start is required to be called by all processes, however, the behaviour of the function differs based on the role of the process. When a worker process reaches the function rpc-start, the process enters a loop and the flow of the execution never goes beyond the rpc-start in the kernel code. On the other hand, the execution continues when the function rpc-start is called in the context of the master process. In the loop, the function rpc-start instructs the worker process to listen on the message channel for the RPC message containing the symbolic expression from the master process. Once the expression is executed in its address space, the worker returns the result into the master process and then again waits for another incoming message from the master process. The function rpc-finalize is called by the master process to terminate the execution environment. When the function is invoked, a finalize message is sent to all worker processes that instruct them to terminate the loop, and then worker processes call MPI Finalize, followed by the exit command causing the process termination with success. Note that calling any Orca function results in an error, as the execution environment is terminated beforehand. The function rpc-make is the most basic compute API so as to execute symbolic expressions collectively. The function evaluates the given symbolic expression in a “quoted” form on the runtime environment of the worker processes,

(rpc-start) → void Listing 1.3. rpc-start

210

A. A. Yıldırım

(rpc-finalize) → void Listing 1.4. rpc-finalize

(rpc-make ) → list Listing 1.5. rpc-make

and then returns a list of values where each i-th element in the list is the result of the computation from the i-th worker.

Fig. 2. Collective execution of the symbolic expression via rpc-make

Figure 2 shows the interaction between processes for the computation of the given symbolic expression from start to the end of execution using rpc-make API. This design generalizes to Orca ’s all computing APIs in a sense that upon the invocation of rpc-start, worker processes execute the given expression, and then master process coalesces the results of the computation received from worker processes into a single list as a returned value. All the compute APIs except rpc-make expect a varying number of parameters that, in turn, are passed to the specified remote function as parameters. We employ pull-based communication for the distribution of function parameters. In this model, the master process distributes function parameters either by broadcasting or scattering to all worker processes that define the type of the compute APIs in the Orca library; broadcast and scatter APIs, respectively. On the other hand, worker processes can only send data to the master process by returning a value from the function invoked remotely.

Orca: Parallel Computation of Symbolic Expressions on MPI

211

(rpc-apply-bcast ...) → list Listing 1.6. rpc-apply-bcast

(rpc-apply-scatter ...) → list Listing 1.7. rpc-apply-scatter

The API rpc-apply-bcast sends serialized function symbol, along with the values of all parameters from the master process to all worker processes in which parameter can be any type of literal value, including a list or a single value. The function symbol needs to be accessible by all execution environments of processes.

Fig. 3. The master process applies given symbolic expression with respect to varying parameter distribution models.

The scatter API rpc-apply-scatter also expects a varying number of parameters, however, every parameter must be a list whose size requires to be the total number of worker processes. In this case, the scatter API sends the i-th element of the list to the i-th worker process in the parameter list. The master process distributes the element of each list in parameters to each worker process in order, determined by the worker process id, and then worker processes apply the given function to its parameters. Note that function parameters are stored in a list, and thus provides flexibility and ease, in constructing parameters at runtime for a varying number of worker processes that can be determined via the API rpc-worker-process-size. Another scatter API is rpc-scatter which is designed to perform task parallelism, instead of data parallelism as applied to other compute APIs. Task parallelism is a technique employed to enhance the locality by executing different parts of a program on available processes [5]. rpc-scatter API accepts a list

212

A. A. Yıldırım

(rpc-scatter ...) → list Listing 1.8. rpc-scatter

of symbolic expressions whose size equals the number of worker processes, where i-th expression is sent to the i-th worker process to be evaluated. As depicted in Fig. 3, rpc-scatter allows to apply disjoint functions to the given parameters on worker processes. In this way, the master process retrieves the results of the disjoint applications executed concurrently in a single call. 3.3

Marshalling

Orca library transforms parameter expressions to UTF-8 format using the Scheme function call-with-output-string for transmission. We verified on the Guile runtime that Orca library supports implicit marshalling of generic data, such as numeric values and strings, and compound data including list, vector, array, byte vector, and association list that might contain generic data as well as compound data in a nested fashion. However, we note that in GOOPS, which is the object-oriented system in Guile, objects are required to be explicitly serialized to the supported symbolic expression (i.e. list) before passing into Orca APIs, and deserialized back into GOOPS object on worker processes in the given remote function, or vise versa. Symbolic expressions are also required to be non-pointer values to be serialized properly. This is a fundamental design requirement, otherwise, the Orca library raises an error and terminates all processes when a pointer value is provided. Furthermore, the literal symbols are allowed to contain any user symbols defined before the rpc-start call as they become global variables across all processes.

4 4.1

Experiments and Discussion A Case Study: Distributed Reduction Operation

As a case study, we implemented a distributed reduction operation using the Orca library. The purpose of this case study is to apply the proposed programming model to one of the common parallel programming problems and see its effectiveness in developing a parallel program that avoids multiple code branches and side effects by leveraging functional programming techniques. As shown in Fig. 4, in a distributed reduction problem, typically we partition input data into disjoint chunks. Then, each chunk of data is processed by each worker process using the same reduction function proc on every element of its input data to reduce them into a single local value. Worker processes perform this reduction operation in parallel using different input data. Once the local results have been computed, each worker process sends the result of the reduction into

Orca: Parallel Computation of Symbolic Expressions on MPI

213

Fig. 4. Worker Processes Apply the Function proc in Parallel on Disjoint Data in Step 1, and Finally the Master Process Reduces the Local Results to Compute Global Results using the Same Function in Step 2.

the master process, and the master process begins to perform global reduction operation on the data whose size is the number of worker processes. Finally, the master process computes the global value using the same reduction function proc. 1 2

#!/usr/bin/env -S guile -s !#

3 4 5 6 7 8

(use-modules (use-modules (use-modules (use-modules (use-modules

(orca)) (srfi srfi-1)) (ice-9 regex)) (ice-9 ftw)) (ice-9 textual-ports))

9 10 11

(define proc max) (define init-value -1)

12 13 14

(define (read-db-numbers db-lst) ...)

15 16 17

(define (apply-distributed-proc db-lst) (reduce proc init-value (read-db-numbers db-lst)))

18 19

(rpc-start)

20 21 22

(define (partition-db-lists) ...)

23 24 25 26 27

(format #t "Result of the distributed reduction operation is ~d~%" (reduce proc init-value (rpc-apply-scatter apply-distributed-proc (partition-db-lists))))

214 28

A. A. Yıldırım (rpc-finalize)

Listing 1.9. Kernel of distributed reduction operation using Orca library. The implementation details of read-db-numbers and partition-db-lists are omitted, three dots is used as a placeholder.

The kernel code of the distributed reduction algorithm is given in Listing 1.9. The first line instructs the operating system to execute the kernel file via the Guile interpreter. Lines [4–8] imports the symbols of the used Guile modules in the kernel, including orca module which is required for all Orca programs. We defined the proc to be the max function that returns the maximum number of given numbers, and init-value to be −1, which has been provided to the reduce function as a default value. The symbols proc, init-value, read-db-numbers and apply-distributed-proc are all defined before the rpc-start call, thus they have become visible in the runtime environments on all processes. We omit the implementation details of the user-defined functions of partitiondb-lists and read-db-numbers for clarification. When the worker processes reach the rpc-start call, they become blocked and wait for receiving the symbolic expressions from the master process to compute. However, master process continues the kernel program and calls reduce function whose parameters are proc, init-value and the result of the rpc-apply-scatter, respectively. In turn, Guile interpreter calls rpc-apply-scatter function that calls the partition-db-lists function; a user-defined function returning a list of lists where every list object at i-th position corresponds to a set of separate input file paths on the disk for ith worker process. Note that every process shares a disk storing input files, and input files contain a collection of numbers. rpc-apply-scatter distributes the paths of input files to worker processes and instructs the Orca library to apply these input files to the apply-distributed-proc function on each worker process in parallel in line 28. As an implementation detail, we utilize Scheme’s macro feature to transform the parameters of rpc-apply-scatter into a respective symbolic expression to be evaluated by the worker processes. When worker processes receive the function and parameters deserialized from the textual form of the symbolic expression, they call the apply-distributed-proc function in line 17. Note that all the numbers on the input files are loaded into memory via read-db-numbers function, and then the reduction operation is performed on these values. When the apply-distributed-proc function returns, the Orca library collects all the local results of worker processes and returns to the reduce function in line 25 for the master process, which in turn computes the global result and prints it. Finally, the master process calls rpc-finalize to terminate the execution environment of Orca. 4.2

Performance Evaluation

We have conducted experiments to evaluate the turnaround time in broadcasting operation using the API rpc-apply-broadcast on varying list size and many worker processes, and compared this time with respect to a corresponding MPI program

Orca: Parallel Computation of Symbolic Expressions on MPI

215

in C language. Our experiments are performed on a single machine using 32 operating system processes. The machine we used is equipped with a 2.6 GHz 6-core Intel Core i7 processor and 16 GB DDR4 memory. We used Guile 3.0.4 to compile Orca library, and MPICH library, version 12.1.8, as an implementation of the Message Passing Interface (MPI) standard. We compiled the MPI program using GCC version 9.3.0.

Fig. 5. Turnaround time in milliseconds with respect to number of worker processes and number of integers in the dataset

For the sake of experiments, we implemented two programs: one is the Orca program using rpc-apply-broadcast API sending an integer list and collects results from worker processes, and the other program is a C program using MPI functions for broadcasting integer list. We conducted experiments using 7 lists with varying lengths where each list consists of 32-bit integers. We measured the turnaround time of the broadcasting operation for both programs, which is defined as wall-clock time between the call of rpc-apply-broadcast and its completion. rpc-apply-broadcast transforms the arguments to the corresponding datum representation, which is then marshalled into textual form for transmission via MPI. Therefore, turnaround time in the Orca program includes datum transformation time, marshaling time, evaluation time, communication time, and the overhead caused by Guile’s runtime environment. However, the turnaround time of the MPI program only contains the communication time. The turnaround times are plotted in the Fig. 5. We observe linear correlations between the size of the list and the number of worker processes in the case of the Orca program. The maximum turnaround time spent in broadcasting a list of 216 integers is 538 ms using 32 processes. In comparison, we obtain a maximum turnaround time of 121 ms in the MPI program. We conclude that experimental results show that the overhead incurred by the runtime and data marshaling is insignificant when compared to the MPI program.

216

5

A. A. Yıldırım

Conclusions and Future Work

In this paper we have presented Orca; a software library implementing a set of RPC APIs which are used for the computation of symbolic expressions in parallel by utilizing MPI library to transmit expressions between processes. We take advantage of the expressive power of Scheme language to represent code as data in the form of symbolic expressions. Scheme language promotes a functional programming paradigm that allows us to define “higher-order” API functions taking a function as an argument, which is also pure in the sense that no side effects are allowed. Such features of the programming model help us alleviate the inherent problems we face in programming on MPI systems which include code complexity due to multiple code branches, buffer management, and inevitable side-effects, by building a new set of APIs based on remote evaluation (REV) technique. As future research, we will investigate an extension to the Orca library by having a new set of APIs to stream expression data between worker processes and the master process. That way we aim to define parallel computations on big data, as well as “infinite” data to process each of the expression chunks via the given function until a given condition is met. Scripting languages are useful for prototyping and integrating different software systems rapidly, but they are not without performance penalties. This is also true for Guile language, as an implementation of Scheme language, in which we use its eval function extensively to execute the given expressions at runtime. In addition to the performance cost coming with the language, there is also a communication cost because Orca library transmits symbolic expressions in a textual form. We will investigate how we can alleviate these problems to be able to achieve higher performance on MPI systems. The source code of the Orca library is made available at https://www. artulab.com/gitweb/?p=guile-orca under LGPL version 3 or later.

References 1. Brown, J.R., Nelson, E.C.: Functional programming. Technical report, TRW DEFENSE AND SPACE SYSTEMS GROUP REDONDO BEACH CALIF (1978) 2. Clamen, S.M., Leibengood, L.D., Nettles, S.M., Wing, J.M.: Reliable distributed computing with avalon/common lisp. In: Proceedings 1990 International Conference on Computer Languages, pp. 169–179 (1990) 3. Dalc´ın, L., Paz, R., Storti, M.: MPI for python. J. Parallel Distrib. Comput. 65(9), 1108–1115 (2005) 4. Forejt, V., Joshi, S., Kroening, D., Narayanaswamy, G., Sharma, S.: Precise predictive analysis for discovering communication deadlocks in MPI programs. ACM Trans. Program. Lang. Syst. 39(4) (2017) 5. Foster, I.: Task parallelism and high-performance languages. IEEE Parallel Distrib. Technol. Syst. Appl. 2(3), 27 (1994) 6. Free Software Foundation (2020). Accessed 21 June 2020 7. Gregor, B., Troyer, M.: (2020). Accessed 21 June 2020

Orca: Parallel Computation of Symbolic Expressions on MPI

217

8. Gropp, W., Lusk, E., Skjellum, A.: Portable Parallel Programming with the Message-Passing Interface. The MIT Press, Using MPI (2014) 9. Hammond, K.: Why parallel functional programming matters: panel statement. In International Conference on Reliable Software Technologies, pp. 201–205. Springer (2011) 10. Haque, W.: Concurrent deadlock detection in parallel programs. Int. J. Comput. Appl. 28(1), 19–25 (2006) 11. Hughes, J.: Why functional programming matters. Comput. J. 32(2), 98–107 (1989) 12. Jones, M.P., Hudak, P.: Implicit and explicit parallel programming in haskell. Dispon´ıvel por FTP em nebula. systemsz. cs. yale. edu/pub/yale-fp/reports/RR982. ps. Z (julho de 1999) (1993) 13. Luecke, G.R., Zou, Y., Coyle, J., Hoekstra, J., Kraeva, M.: Deadlock detection in MPI programs. Concurrency Comput. Pract. Exper. 14(11), 911–932 (2002) 14. McCarthy, J., Levin, M.I.: LISP 1.5 programmer’s manual. MIT Press, Cambridge (1965) 15. Message Passing Interface Forum. MPI: A message-passing interface standard, version 3.1. Specification, June 2015 16. Murthy, V.K., Krishnamurthy, E.V.: Software pattern design for cluster computing. In: Proceedings International Conference on Parallel Processing Workshop, pp. 360–367 (2002) 17. Ong, E.: MPI ruby: scripting in a parallel environment. Comput. Sci. Eng. 4(4), 78–82 (2002) 18. Peter, S.: Pacheco. Parallel Programming with MPI. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1996) 19. Smith, L.A.: Mixed mode MPI/OpenMP programming. UK High-End Computing Technology Report, pp. 1–25 (2000) 20. Stamos, J.W., Gifford, D.K.: Remote evaluation. ACM Trans. Program. Lang. Syst. 12(4), 537–564 (1990) 21. Sussman, G.J., Steele, G.L.: Scheme: a interpreter for extended lambda calculus. High.-Order Symb. Comput. 11(4), 405–439 (1998) 22. Wilbur, S., Bacarisse, B.: Building distributed systems with remote procedure call. Softw. Eng. J. 2(5), 148–159 (1987) 23. Yoo, A.B., Jette, M.A., Grondona, M.: Slurm: simple linux utility for resource management. In: Workshop on Job Scheduling Strategies for Parallel Processing, pp. 44–60. Springer (2003)

Solving High-Dimensional Nonlinear Equations with Infinite Solutions by the SVM Visualization Method Yu-Yuan Lin and Jeng-Nan Tzeng(B) National Chengchi University, No. 64, Sec. 2, ZhiNan Road, Wenshan District, Taipei City 16302, Taiwan R.O.C.

Abstract. There are many standard mathematical methods for solving nonlinear equations. But when it comes to equations with infinite solutions in high dimension, the results from current methods are quite limited. Usually these methods apply to differentiable functions only and have to satisfy some conditions to converge during the iteration. Even if they converge, only one single root is found at a time. However, using the features of SVM, we present a simple fast method which could tell the distribution of these infinite solutions and is capable of finding approximation of the roots with accuracy up to at least 10−12 . In the same time, we could also have a visual understanding about these solutions. Keywords: High-dimensional nonlinear equations SVM

1

· Infinite solutions ·

Introduction

In physics, chemistry and engineering, many problems appear in the form of nonlinear equations. Using numerical methods, the solutions can be computed to a desired degree of accuracy. For equations with one variable, we already have many tools to obtain the approximation of roots. For example, two-point bracketing method like bisection method [1] and false position method (or Regula falsi) [2]. Also we have Fixedpoint iteration [1] and its improved version Wegstein’s method [3,4]. The most famous is Newton’s method which can be seen as a special case of fixed-point iteration. Newton’s method has many extensions and variations. Secant method [5] and Steffensen’s method [6, 7] both replace the derivative in Newton’s method by the slope of secant line. And Halley’s method [8, 9] uses second order of Taylor series instead of first order linear approximation in Newton’s method. For nonlinear equations involve several variables, we don’t have much reference. In [10], Yuri Levin and Adi BenIsrael introduce directional Newton method for differentiable f : Rn → R with conditions for direction vectors, gradient of f , and second derivative (Hessian matrix) of f to reach quadratic convergence. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 218–231, 2022. https://doi.org/10.1007/978-3-030-80119-9_11

Solving Nonlinear Equations

219

In [11], HengBin An and ZhongZhi Bai present directional secant method, a variant of directional Newton method, which also reaches quadratic convergence under suitable assumptions and has better numerical performance. In [12], HengBin An and ZhongZhi Bai give Broyden method for nonlinear equation in several variables, a modification of the classical Broyden method: for f (x) = 0 where f : Rn → Rn is differentiable, the classical Broyden method can be described as follows: given initial value x0 ∈ Rn , initial matrix A0 ∈ Rn×n , and tolerance . Set k = 0. If  f (xk ) > , then 1. 2. 3. 4. 5.

k sk = −A−1 k f (x ) k+1 k k x =x +s y k = f (xk+1 ) − f (xk ) k k k T k s )(s ) = Ak + Ak+1 = Ak + (y −A (sk )T sk k := k + 1

f (xk+1 )(sk )T (sk )T sk

where Ak is an approximation of the Jacobi matrix of f at xk . In one dimension, it is simply secant method since Ak+1 sk = y k . Now apply the above to f (x) = 0 where f : Rn → R. Given initial value x0 ∈ Rn , initial matrix AT0 ∈ Rn , and tolerance . Set k = 0. If |f (xk )| > , then we have 1. 2. 3. 4. 5.

sk = −A†k f (xk ) xk+1 = xk + sk y k = f (xk+1 ) − f (xk ) k k k T k s )(s ) = Ak + Ak+1 = Ak + (y −A (sk )T sk k := k + 1

f (xk+1 )(sk )T (sk )T sk AT

where A†k is the Moore-Penrose inverse of Ak . By A†k = Akk2 and Ak A†k = 1, through some simplifications of the recursive process, [12] rewrites the above in a more concise way: given initial value x0 ∈ Rn , initial matrix AT0 ∈ Rn , and tolerance . Set Δ0 = −f (x0 ) and k = 0. If |f (xk )| > , then 1. 2. 3. 4.

xk+1 = xk + Δk A†0 y k = f (xk+1 ) − f (xk ) k+1 Δk+1 = f (xyk ) Δk k := k + 1

which is the final form of Broyden method. Under suitable assumptions, Broyden method is locally super-linearly convergent and semi-locally convergent. Compared to directional Newton method and directional secant method, numerical experiments show that Broyden method is more efficient and the difference is significant especially in high dimension or the initial point is not ideal. This shows the superiority of Broyden method. Later in Sect. 2, we will give an overview of Monte Carlo method and Support Vector Machine (SVM), after that illustrate our new method. And then in Sect. 3, we will apply the new method to some examples to see its performance and limitations. Finally in Sect. 4, we will summarize the results in Sect. 3 and draw a conclusion about the pros and cons of the new method.

220

2

Y.-Y. Lin and J.-N. Tzeng

Methodology

The above methods for solving nonlinear equations usually need the assumption that f is differentiable. Hence they can’t be applied to functions which don’t satisfy the condition. Moreover, during the iterative process, when we need to compute gradient or difference quotient, it could be complicated, time consuming or even unable to calculate. Sometimes it is also tricky to find a suitable initial point and direction vector. For example, consider this quite simple equation: f (x, y) = x2 + y 2 − 1 = 0. First we randomly choose an initial value x0 = [1, 1] and initial matrix AT0 = [2, 2]. Then apply the Broyden method with tolerance 10−3 as showed in Table 1: Table 1. The broyden method. x0 = [1, 1] and AT0 = [2, 2] xk 1

Function value

x = [0.750.75]

0.125

x2 = [0.785714290.78571429]

0.23469387755102034

x3 = [0.862126250.86212625]

0.48652332755709105

... x20 = [2106.503718872106.50371887] 8874714.835218174

We can see that this combination fails, so does x0 = [1, 1] and AT0 = [−2, −2]. A successful try is x0 = [1, 1] and AT0 = [1, 0]. But even we succeed, every time we apply those methods to our target, only one single root is found. It is not efficient to deal with infinite solutions. And we don’t have a whole picture of our solutions. To avoid the above disadvantages, we try a new method which combines Monte Carlo method and Support Vector Machine(SVM) to find roots. The basic concept of Monte Carlo method is using repeated random sampling and statistical analysis to obtain numerical results. It is often used in the fields of physics and mathematics. One main usage is simulating systems with randomness or modeling phenomena with significant uncertainty. Another usage is transforming the solution of unsolved problem to a parameter (such as expectation) of some kind of random distribution. In mathematics, the most common application of Monte Carlo method is in integration and optimization. Especially when the number of function evaluations grows exponentially in high dimensions. It is useful and powerful for obtaining numerical solutions to problems too complicated to solve analytically. Support vector machine(SVM) is a supervised learning model for classification and regression analysis in machine learning. Basically, it is a binary linear classifier. First we give a set of training data to the model. These data are already marked as belonging to one category or the other according to some valued features. Then SVM model “learns” how to classify from these training data by creating a hyper plane in the feature space. With the hyper plane,

Solving Nonlinear Equations

221

the rules of classification it learned, SVM model can assign new data to one of the two categories. In addition to linear classification, using the skill called “kernel trick”, SVM is capable of performing nonlinear classification. The way it works is mapping featured data into a higher dimension space to apply a linear classification. We can also apply SVM to multiclass problems by distinguish between one category and the rest(one-versus-all) or between every pair of categories(one-versus-one). Now we introduce another method for solving nonlinear equations of several variables. Suppose f : D ⊆ Rn → R is continuous. First we choose a desired region in the domain and randomly select sample points in the region. As to region we refer to a rectangle in two dimensional space, a cuboid in three dimensional space, etc. Compute the function values of these sample points. If we get only positive or negative values, then chances are high that there is no root in this region unless the number of sample points is too small. Now consider the case that we’ve got both positive and negative function values. Here these sample points are like training data with n features, and SVM model classifies them into two classes according to the sign of their function values. Now with the trained SVM model, the hyper plane could be seen as a rough approximation of the roots. Also we have a number of support vectors in each class. These support vectors are supposed to be “close” to the roots in the region. Then we randomly select the same number of sample support vectors in each class and make them into pairs. From every pair we can obtain a “hyper cuboid” by letting each coordinate of the 2n vertices to be the minimum or maximum in each dimension of the paired support vectors. Take n = 2 for example, suppose (a, b) and (c, d) are paired support vectors from different classes, then we can obtain a hyper cuboid (here it is only a rectangle) with the following four vertices: (min{a, c}, min{b, d}), (max{a, c}, min{b, d}), (max{a, c}, max{b, d}) and (min{a, c}, max{b, d}). Now, to decide which hyper cuboid leads to a better opportunity for finding roots, we consider the following factors: – Let r1 be the volumetric ratio of the hyper cuboid to the region. We expect small r1 . – Let n1 and n2 be the numbers of support vectors of the two classes in the 1 ,n2 } hyper cuboid respectively. Let r2 = max{n min{n1 ,n2 } . We expect small r2 . 1 +n2 – Let d = volumn of nthe hyper cuboid be the density of support vectors in the hyper cuboid. We expect large d. Consider the value r1 + r2 + d1 of each hyper cuboid. The smaller it is, the higher probability that we could find roots in this hyper cuboid. So we can choose several candidates to continue. Or we could choose the hyper cuboid with the smallest value to be the new region in the next iteration. And it’s center point is an approximation of a root in this iteration. Continue the process, we can shrink the hyper cuboid and improve our approximation of root until it satisfies demanded accuracy.

222

3

Y.-Y. Lin and J.-N. Tzeng

Examples

Now we apply the method in Sect. 2 with Python programming codes to some examples and see how it works. The equipment we use is a laptop with Intel(R) Core(TM) i5-8265U CPU 1.60 GHz up to 1.80 GHz and RAM 8.00 GB.  Example 1. f (x, y) = x2 + y 2 − 1 + ln(4 − x2 − y 2 )

Fig. 1. Graph of Example 1.

The graph of f is presented in Fig. 1. The roots of f on xy plane is similar to a circle with radius a little less than 2. Now we use Python code to find roots in [1, 3] × [−1, 1] with 1000 sample points in each iteration and with tolerance of diagonal of the rectangle 10−12 . Test results are presented in the order of approximation of a root, function value of the approximation, number of iterations, and time of operations. Table 2. Test results of Example 1 with python code. Approximation

Function value

Number Time

(1.8393485316192288, 0.6556725842999632)

−5.114131340633321e–12

27

(1.8404225031688544, −0.6526519504314735) −4.672262576832509e–12

33

0.59375

(1.941254269782079, 0.21128515949820706)

2.9465319073551655e–13

41

0.71875

(1.7597515054678885, −0.8463948236929669) 3.2591707110896095e–12

32

0.453125

(1.8555758543841576, −0.6082333492989722) −7.824629832953178e–12

32

0.515625

(1.9291048578468786, −0.3027606414355924) 2.4069635173873394e–12

39

0.640625

0.484375

(1.8859593483090873, −0.5062281057965792) −7.256639733554948e–12

37

0.625

(1.8806978984805065, 0.5254378871203835)

37

0.703125

(1.8191277005208264, −0.7098478483235422) −1.8112178423734804e–12 38

0.546875

4.381828233590568e-12

ValueError: The number of classes has to be greater than one; got 1 class

Solving Nonlinear Equations

223

Table 2 shows that out of 10 test results, we have found 9 roots (approximations). Each root takes less than 1 s. And we have one “ValueError: The number of classes has to be greater than one; got 1 class”. That means during the process of iteration, function values of sample points in the rectangle have the same sign. If we draw these roots in Table 2 on xy plane, we may obtain a very rough contour of the roots in [1, 3] × [−1, 1] as showed in Fig. 2.

Fig. 2. Roots founded by python code.

Example 2. f (x, y) = x2 − 2y 2 + xy − 30 sin(x + y) + 20 cos xy The graph and roots of f around the origin are presented in Fig. 3(a) and Fig. 3(b) respectively. Similarly, in example 2 we also use Python code to find roots in [−1, 1] × [−1, 1] with 1000 sample points in each iteration and with tolerance of diagonal of the rectangle 10−12 . Test results are as follows in Table 3.

(a) Graph of the Function

(b) Roots on xy Plane

Fig. 3. Graph of Example 2.

224

Y.-Y. Lin and J.-N. Tzeng Table 3. Test results of Example 2 with python code in [−1, 1] × [−1, 1].

Approximation

Function value

(0.46993790636813626, 0.2622453677771487)

−1.7763568394002505e–14 33

Number Time 0.65625

(0.7684688297613869, −0.012505477809996928) 3.5136338283336954e–12

40

0.484375

−9.78772618509538e–12

30

0.46875

(0.22176752901561408, 0.48857630079124126)

IndexError: index 0 is out of bounds for axis 0 with size 0 (−0.005547432373856567, 0.6929394537660047) −2.9096725029376103e–12 46

0.8125

(0.5621218361256275, 0.17890958192506992)

3.7196912217041245e–12

37

0.703125

(−0.25695433093787506, 0.8894629520497923)

7.059242079776595e–12

32

0.5

(0.8050933184240345, −0.04871131491812492)

−7.226219622680219e–12

35

0.578125

(0.42833405733942814, 0.29992808858241676)

1.0043521569969016e–11

25

0.359375

(0.46601152390386, 0.265797865621633)

4.764189043271472e–12

32

0.65625

One of the test results in Table 3 is “IndexError: index 0 is out of bounds for axis 0 with size 0”, that means the code can’t find the most recommended rectangle during the iterations. When we zoom out to see function f (x, y) = x2 − 2y 2 + xy − 30 sin(x + y) + 20 cos(xy), its graph on xy plane behaves like a hyperbola, see Fig. 4 (a). If we try to find roots in regions that contain no root, we will get “ValueError: The number of classes has to be greater than one; got 1 class”. Now we try to find roots in regions away from the origin, say [50, 51]×[50, 51], see Fig. 4 (b). Table 4 is the test results.

(a) Roots in [−60, 60] × [−60, 60]

(b) Roots in [50, 51] × [50, 51]

Fig. 4. Roots of Example 2 in different scales.

Compare Table 4 with Table 3, the accuracy of function values apparently is worse in Table 4 than in Table 3. If we raise the standard of tolerance, Python code even fails in [50, 51] × [50, 51], see Table 5 where the tolerance of diagonal of the rectangle is 10−15 and ii in the table represents the maximal value allowed in number of iterations.

Solving Nonlinear Equations

225

Table 4. Test results of Example 2 with python code in [50, 51] × [50, 51]. Approximation

Function value

Number Time

(50.37108241597812, 50.30605405694342)

7.59312612785834e–11

36

0.734375

IndexError: index 0 is out of bounds for axis 0 with size 0 (50.216759062176685, 50.209210588582806) 4.789209029354424e–10

25

0.484375

(50.91835623563183, 50.67395719627402)

3.6215297427588666e–10

33

0.484375

(50.08210621719444, 50.15376290895588)

3.0322144795036365e–10

32

0.53125

(50.493655231184896, 50.352278685604674) 9.49995637711254e–11

32

0.5

(50.76882719293579, 50.64478487427914)

2.929940734475167e–10

29

0.5625

(50.060893746994225, 50.18334780684751)

1.6474643871333683e–10

32

0.53125

(50.56335822951033, 50.48002788886255)

−9.930056776852325e–11 41

0.65625

(50.04464454039062, 50.12500465230691)

4.3338976851714506e–10

0.546875

40

Table 5. Raise the standard of tolerance. Approximation

Function value

Number Time

Code 1: root(f, −1, 1, −1, 1, 1000, tol = 10**(−15)), ii=100 (0.7461596882606241, 0.009033344195159302)

3.552713678800501e–15

41

0.578125

(0.2376360543484806, 0.4740324090606552)

0.0

34

0.546875

(0.39138819644970957, 0.3334828756138792)

3.552713678800501e–15

42

0.671875

(0.8130843644528254, −0.056777966137313576) 7.105427357601002e–15

52

0.78125

ValueError: The number of classes has to be greater than one; got 1 class Code 1: root (f, 50, 51, 50, 51, 1000,tol=10**(−15)), ii=100 (50.6987226809441, 50.52614710988952)

5.950795411990839e–14

101

2.078125

(50.3123398028162, 50.34944789599142)

7.105427357601002e–13

101

2.234375

(50.29289226163097, 50.258444639339814)

−8.219203095904959e–12 101

1.953125

(50.25733872551366, 50.21940854261966)

2.327027459614328e–12

101

1.9375

(50.204504343031914, 50.29298780543669)

−7.545963853772264e-12

101

2.03125

1001

21.921875

1001

21.640625

Code 1: root (f, 50, 51, 50, 51, 1000, tol =10**(−15)), ii=1000 (50.31353419225536, 50.295244027029945)

8.739675649849232e–13

ValueError: The number of classes has to be greater than one; got 1 class (50.842699814232994, 50.628859347893254)

6.235012506294879e–13

(50.71581863271457, 50.680914087131555)

−8.562039965909207e–13 1001

22.21875

(50.89625216670158, 50.76244021179818)

−6.986411449361185e–12 1001

21.90625

Code 1: root (f, 50, 51, 50, 51, 1000, tol = 10**(−15)), ii=5000 (50.735470991990084, 50.62593175023604)

3.984368390774762e–12

5001

126.84375

IndexError: index 0 is out of bounds for axis 0 with size 0 (50.297804446513624, 50.254821899553995)

4.4364512064021255e–12

5001

127.078125

(50.30503027546648, 50.30686591842504)

8.107736704232593e–12

5001

128.3125

(50.762567213039844, 50.589032786249454)

5.153211191100127e–12

5001

129.90625

From Table 5, we can see that in region [50, 51] × [50, 51], even if we allow number of iterations up to 5000 times, Python code still failed to obtain an approximation of root that satisfies the required tolerance and the accuracy of

226

Y.-Y. Lin and J.-N. Tzeng

function value doesn’t improve along with the increase in number of iterations. The reason is that sine and cosine function in Python are approximated by series of polynomials and rounding errors will accumulate when independent variables are away from the origin. To avoid this, we can translate the graph so that the independent variables are near the origin. For example, let g(x, y) = f (x + 50, y + 50) and use the periodic property of sine and cosine function, we have g(x, y) = x2 + xy − 2y 2 + 150x − 150y − 30 sin(x + y + 100 − 32π) + 20 cos((x+50)(y +50)−796π). Then apply Python code to g in [0, 1]×[0, 1] with tolerance of diagonal of the rectangle 10−15 , see Table 6. After that we move our approximations of roots back to [50, 51] × [50, 51] and compute their function values, see Table 7. Table 6. Test results of g with python code in [0, 1] × [0, 1]. Approximation

Function value

(0.46818599428272567, 0.38802043952474363) 3.952393967665557e–14

Number Time 45

1.0625

(0.4462804306335044, 0.41798056459405625)

−2.9309887850104133e–13 37

0.8125

(0.4691852199445329, 0.38665487925191777)

2.7533531010703882e–14

41

0.859375

(0.5605807588099267, 0.48208915471610153)

−5.380584866543359e–12

55

1.046875

(0.2017860554505237, 0.2971099319591042)

−5.5209170568559784e–12 42

0.6875

(0.37514876948450493, 0.3632664037200196)

−4.413358567489922e–12

45

1.078125

(0.6542098954593933, 0.5872723444048912)

2.76578759894619e–12

44

0.8125

(0.12206118061828386, 0.09784852619758204) 5.53157519789238e–12

44

0.71875

47

0.875

ValueError: The number of classes has to be greater than one; got 1 class (0.7855399066982386, 0.7079424206421457)

−5.346834086594754e–12

Table 7. Approximations of roots of f in [50, 51] × [50, 51]. Approximation of root

Function value

(50.46818599428272567, 50.38802043952474363) 1.354028000832841e–12 (50.4462804306335044, 50.41798056459405625)

1.91491267287347e–12

(50.4691852199445329, 50.38665487925191777)

1.0031975250512914e–11

(50.5605807588099267, 50.48208915471610153)

2.831068712794149e–12

(50.2017860554505237, 50.2971099319591042)

1.950439809661475e–12

(50.37514876948450493, 50.3632664037200196)

−2.568611989772762e–12

(50.6542098954593933, 50.5872723444048912)

3.382183422218077e–12

(50.12206118061828386, 50.09784852619758204) −1.6697754290362354e–13 ValueError: The number of classes has to be greater than one; got 1 class (50.7855399066982386, 50.7079424206421457)

2.8919089345436078e–12

Solving Nonlinear Equations

Example 3. f (x, y) =

227

cos y+1 y−sin x

By function formula, we know the roots are points on xy plane with ycoordinate equals to (2k + 1)π where k ∈ Z. However, points around the roots belong to the same side of xy plane, see Fig. 5(b). Therefore, when we apply Python code to this function, it fails and we’ll get “ValueError: The number of classes has to be greater than one; got 1 class”. Moreover, for this function, on regions that contain discontinuous points, Python code may still return an answer, but it’s not a root. For example, one test result of Python code is the approximation of root (0.5584914658730595, 0.5299074807530488), but it’s function value is 17034614928732.754, obviously not a root of f (x) = 0.

(a) Graph of the Function

(b) Projection on yz Plane

Fig. 5. Graph of example 3.

Example 4. f : Rm → R defined by f (x) = (x1 , x2 , ..., xm )

m i=1

xi exp(1 − x2i )

where x =

m

   Apparently, the origin (0, 0, ..., 0) is a root of f (x) = 0. First, we apply Python 10

   code to find roots in [−0.9, 1] × [−0.9, 1] × · · · × [0.9, 1] with 250 samples points in each iteration and with tolerance of diagonal of the hyper cuboid 10−12 . Here is the first test result: – The approximation of root : ( 1.2844263924027142e–15, 7.711984581422935e–14, 9.976145297340012e–14, −1.0961014831765263e–14, −1.120713405053965e–14, 3.436442229894976e–14, −8.308598458235813e–14, −9.737252214208976e–14, −1.9765518862372757e– 14, −3.7386907986968744e–14 ) – Function value of the approximation: −1.2843592136232755e–13 – Number of iterations: 310 – Time of operations: 3.328125 s. In Table 8, we only list the last three items of outputs since all approximations of roots suggest the origin.

228

Y.-Y. Lin and J.-N. Tzeng Table 8. Test results of example 4 with python code. Function value

Number Time

1.3779698843748291e–14

292

3.15625

−1.332515734999162e–14

317

3.40625

1.1987897774143587e–13

322

3.4375

−8.386255861714453e–14

310

3.34375

−1.0989147744968376e–13 332

3.515625

−2.0372613787811113e–13 318

3.40625

4.910130259939433e–14

305

3.359375

2.8895734888874934e–13

305

3.265625

−1.2903165704502644e–13 300

3.265625

m

   Next, we apply Python code in [−0.9, 1] × [−0.9, 1] × · · · × [0.9, 1] with dimension m = 15, 20, 25, 30, 35, 40. Table 9 list one test result of each dimension. Since all test results indicate the same root: the origin, we still only present the last three items of the outputs. Table 9. One test result in different dimensions. Dimension Function value

Number Time

15

−1.1705003069935705e–13

471

7.65625

20

−2.1807932674276025e–13

618

13.109375

25

−8.214164474230035e–14

767

19.71875

30

1.7075573329547067e–13

899

28.546875

35

−4.4167018237416875e–14 1063

38.828125

40

−1.7446726440345464e–14 1184

51.265625

In Table 9, time of operations increases to over 50 s when dimension comes to 40. If we want to decrease operation time, we may shrink the region and adjust number of sample points. Now for each dimension m = 10, 15, 20, 25, 30, 35, 40, we take 10 consecutive test results and compute the average and standard deviation of operation time, see Table 10. Also we can find a quadratic function to fit these data points as in Fig. 6.

Solving Nonlinear Equations

229

Table 10. The statistics of operation time of 10 consecutive test results in different dimensions. Dimension Average of operation time SD of operation time 10

3.35

0.10

15

7.52

0.22

20

13.08

0.41

25

19.72

0.38

30

28.41

0.61

35

38.66

1.58

40

50.82

1.71

Fig. 6. Dimension and operation time.

4

Conclusion

We propose a new method which combines Monte Carlo method and Support Vector Machine(SVM) to solve nonlinear equations of several variables. The new method has the following advantages: – It only requires function to be continuous, not necessarily differentiable. – It needs neither to compute gradient or difference quotient nor to find a proper initial value and direction vector.

230

Y.-Y. Lin and J.-N. Tzeng

– For designated region in the domain, it can tell if there are roots in this region as long as we throw enough sample points. And it can give several recommended smaller regions that contain roots and continue to improve the accuracy of roots. – We can cut the region into partition and work in parallel. This could raise efficiency and save time. – Even if it doesn’t achieve the desired accuracy in regions away from the origin because it takes too many iterations or too much operation time, we still have pretty good approximations of roots. – Operation time is not exponentially increasing along with the increase of dimension. However, the method also has some disadvantages: – It can’t deal with multiple roots of even multiplicity. – For regions contain infinitely many roots, it can’t find all roots since it randomly takes sample points in the region. – In high dimensions, it’ll take time to determine suitable number of sample points to have better performance in operation time. There are still some aspects that can be improved. For example, when we select the next smaller region to continue the iteration, we may change the weigh of each factor to raise the efficiency, or maybe there are better filters to decide the next smaller region. Moreover, the Python code we use now in dimension m m    (greater than 2) is designed to find roots in hyper cubic [a, b] × [a, b] × · · · × [a, b]. The following step could be applying the method to more high-dimensional examples in arbitrary region to see its performance.

References 1. Radi, B., El Hami, A.: Advanced Numerical Methods with Matlab 2 Resolution of Nonlinear. Differential and Partial Differential Equations. John Wiley & Sons, Incorporated (2018) 2. Wall, D.D.: The order of an iteration formula. Math. Comput. 10(55), 167–168 (1956). https://doi.org/10.1090/s0025-5718-1956-0080981-0 3. Wegstein, J.H.: Accelerating convergence of iterative processes. Comm. ACM 1(6), 9–13 (1958). https://doi.org/10.1145/368861.368871 4. Gutzler, C.H.: An iterative method of Wegstein for solving simultaneous nonlinear equations. (1959) 5. Ezquerro, J.A., Grau, A., Grau-S´ anchez, M., Hern´ andez, M.A., Noguera, M.: Analysing the efficiency of some modifications of the secant method. Comput. Math. Appl. 64(6), 2066–2073 (2012). https://doi.org/10.1016/j.camwa.2012.03. 105 6. Liu, G., Nie, C., Lei, J.: A novel iterative method for nonlinear equations. IAENG Int. J. Appl. Math. 48, 44–448 (2018) 7. Kumar, M., Singh, A.K., Srivastava, A.: Various newtontype iterative methods for solving nonlinear equations. J. Egypt. Math. Soc. 21(3), 334–339 (2013). https:// doi.org/10.1016/j.joems.2013.03.001

Solving Nonlinear Equations

231

8. Alefeld, G.: On the convergence of Halley’s method. Am. Math. Mon. 88(7), 530 (1981). https://doi.org/10.2307/2321760 9. George, H.: Brown: on Halley’s variation of newton’s method. Am. Math. Mon. 84(9), 726 (1977). https://doi.org/10.2307/2321256 10. Levin, Y., Ben-Israel, A.: Directional Newton methods in n variables. Math. Comput. 71(237), 251–263 (2001). https://doi.org/10.1090/s0025-5718-01-01332-1 11. An, H.-B., Bai, Z.-Z.: Directional secant method for nonlinear equations. J. Comput. Appl. Math. 175(2), 291–304 (2005). https://doi.org/10.1016/j.cam.2004.05. 013 12. An, H.-B., Bai, Z.-Z.: Math. Num. Sin. 26(4), 385–400 (2004)

A Novel GPU Implementation for Image Stripe Noise Removal Pasquale De Luca1,2(B) , Ardelio Galletti2 , and Livia Marcellino2 1

2

Department of Computer Science, University of Salerno, via Giovanni Paolo II, Fisciano, Italy [email protected] Department of Science and Technology, University of Naples “Parthenope”, Centro Direzionale C4, Naples, Italy {ardelio.galletti,livia.marcellino}@uniparthenope.it

Abstract. Image processing is a class of procedures very helpful in several research fields. In a general scheme, a starting image generates a output image, or some image features, whose values are composed by using different methods. In particular, among image processing procedures, image restoration represents a current challenge to address. In this context the noise removal plays a central role. Here, we consider the specific problem of stripe noise removal. To this aim, in this paper we propose a novel Gaussian-based method that works in the frequency domain. Due to the large computational cost when using, in general, Gaussian related methods, a suitable parallel algorithm is presented. The parallel implementation is based on a specific strategy which relies the newest powerful of graphic accelerator such as NVIDIA GPUs, by combining CUDA kernels and OpenACC’s routines. The proposed algorithm exhibits good performance in term of quality and execution times. Tests and experiments show the quality of the restored images and the achieved performance.

Keywords: Image processing Noise removing · Stripe noise

1

· Parallel computing · Gaussian filter ·

Introduction

The image processing has the aim of improving the visualization or accentuating, or reducing, specific features of images through ad hoc methods [1, 11]. The purposes of the processing can be various: exaltation of details, extraction of information, restoration of image, etc. In particular, in the first case, suitable filters or modifications to the image’s histogram are often used in order to enhance the specific requests such as: increase or decrease the contrast, brightness, highlight some colors at the expense of others, eliminate levels of gray or distribute them evenly, i.e. as a uniform distribution [2]. Other important operations in this field are, for example, the extraction of information such as edges or bit-plane c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 232–243, 2022. https://doi.org/10.1007/978-3-030-80119-9_12

A Novel GPU Implementation for Image Stripe Noise Removal

233

and the image segmentation to partition it into regions or isolate objects of interest [3]. Moreover, the majority of image processing algorithms are addressed to image restoration, and a large part of them focuses on noise-affected images. The removal of noise in order to improve images’ sharpness and visualization plays a main role in this field [4]. The algorithms used for this purpose are applied in different domains depending on the nature of the noise. In the spatial domain, smoothing filters are widely used to reduce noise. Since the noise is randomly distributed, it causes strong changes in image brightness; therefore, the smoothing filters perform a weighted average of the neighborhood pixels inside a squared sliding window. In this way, the overall effect is to make the image less sharp than the original one. For example, median filters are very effective in the presence of impulsive or salt and pepper noise, avoiding the typical blurring effect of smoothing filters [5]. Nevertheless, other methods can involve not only neighbors pixels but also include farther values [6]. However, the frequency domain is a powerful tool for image analysis and to better understand the filtering performed in the spatial domain. In this domain, moreover, it is possible to study the various frequency components of the image and, therefore, of the superimposed noise in order to identify and remove it. In this paper we consider a special case of noise affected images: images in which the noise is present in some frequency bands. An example of this kind of noise is illustrated in Fig. 1 where a periodic-like noise is present making viewing unpleasant.

Fig. 1. Striping noise

There are numerous vertical lines of alternating brightness at a certain frequency f0 , that is therefore a sinusoidal noise in the spatial domain which turns out in a rectangular frequency added in the frequency domain. This kind of noise is usually called stripe noise and several methods have been designed to remove it (see [7,8] and references therein). To this aim, some approaches apply a cylindrical filter, in the frequency domain, that mimics the Fourier Transform of the added noise [9]. In general, the key idea is to perform a good approximation of such a filter and, in this sense, here we adopt a suitable Gaussian-based

234

P. De Luca et al.

method. In particular, our contribution is the GPU-parallel implementation of such method. In other words, we exploit the massive parallelism of Graphics Processing Units (GPUs), in the CUDA environment [19], for designing a parallel approach based on both the domain decomposition and a specific memory access management. Moreover, our software uses as additional feature the OpenACC’s routines in order to achieve the data consistency. This approach allows us to obtain interesting results in terms of performance. Following two sections deal with the proposed method, and its implementation, and the related experimental results, respectively. Conclusions close the paper.

2

Stripe Denoising Strategy and Algorithm

In this section we give a detailed description of the main contribution of this work. In particular we describe the proposed strategy for denoising images affected by stripe noise, then we discuss a related algorithm and the details about its implementation in the GPU environment. Let us assume that x = z +  is an image with stripe noise, that is the noise is modeled through a suitable sinusoidal function . Our aim is to recover the output denoised image z starting from x e some other a priori information. The approach we apply is based on the following scheme: (i) firstly we represent the input image in the frequency domain. This is done by computing X, the bidimensional Fourier Transform (FT-2D) of the input signal x; (ii) then we apply to X a filter to recover Z which is the Fourier Transform of the target denoised output z. In other words, we need to define a filter H, with the property: Z = X · H. (1) The point-wise operation in (1) is possible thanks to the convolution theorem; (iii) finally, the recovered image z in obtained by applying the bidimensional inverse Fourier Transform (IFT-2D) to Z. This scheme is summarized in Fig. 2.

Fig. 2. Frequency domain based scheme for image denoising

Therefore, to recover a good approximation of the denoised signal z we need to find, or approximate, a suitable filter function H, with the optimal property:  H · X = X, (2) H · E = 0,

A Novel GPU Implementation for Image Stripe Noise Removal

235

where E represents the Fourier Transform of ε (see [10, 12] for more detail). The shape of H, or its approximation, must depend on the a priori information about the kind of noise in the frequency domain. Indeed, the frequency components of the input image will be attenuated or amplified depending on the H filter choice. As we can see in the example in Fig. 3, referred to the spectrum in the frequency domain for image in Fig. 1, a stripe noisy image, with vertical lines of alternating brightness at a certain frequency f0 , typically presents frequencies localized around three areas: a central part involving the denoised signal frequencies, and two opposite circular regions of a given radius ρ, mainly representing the noise frequencies, centered in two symmetrical peaks: F1 = (−f0 , 0),

F2 = (f0 , 0).

(3)

Here f0 and ρ denote suitable frequency parameters depending on the noise distribution (notice that for horizontal lines noise peaks take positions (0, −f0 ) and (0, f0 )). Hence, a good choice for a filter should have a shape that tries to delete only frequencies around the peaks in (3).

Fig. 3. Left: spectrum of a typical stripe noise image in the frequency domain (the same of Fig. 1). Right: a slice along the main noise direction

This can be obtained, for example, by taking the filter: HC = 1 − (C1 + C2 )

(4)

which eliminates all frequencies localized in the two cylinders with radius ρ and centers in the peaks in (3). More specifically, C1 and C2 represent the cylindrical filters, surrounding the frequencies to be deleted (see left-hand side of Fig. 4), and they take the expression: C1 = CF1 ,ρ

and

C2 = CF2 ,ρ

where Cc0 ,r denotes the generic characteristic function of the circle centered at c0 and having radius r, i.e.:  1 if d(c0 , P ) ≤ r Cc0 ,r (P ) = (5) 0 otherwise and d(c0 , P ) is the is the Euclidean distance between points c0 and P .

236

P. De Luca et al.

Unfortunately, the use of a simple filter as in (4) can generate final images unclear (blurred) due to the fact that cylindrical filters eliminate an excessive frequency information content [1]. To avoid this effect we propose to substitute the cylindrical filters by 2D Gaussian functions that are smoother kernels (see right-hand side of Fig. 4).

Fig. 4. Example of 2D filters. Left: cylindrical. Right: Gaussian.

This approach relies in the following Gaussian based filter for stripe noise removal: (6) HG = 1 − (G1 + G2 ) where: G1 = GF1 ,σx ,σy

and

G2 = GF2 ,σx ,σy

(7)

are Gaussian bells mainly surrounding the frequencies to be deleted. Here G is the bidimensional Gaussian function centered at c0 = (c0x , c0y ) and having standard deviations σx and σy with respect to axes x and y, respectively, i.e.:    (Py − c0y )2 1 (Px − c0x )2 Gc0 ,σx ,σy (P ) = exp − (8) + 2 σx2 σy2 The proposed filter and strategy give raise to the following Algorithm 1. Algorithm 1. Sequential implementation Input: x % noised image % Gaussian parameters f0 , σx , σy Output: z % de-noised image % Input in the frequency domain. 1: compute X as the Fourier Transform of the input image x % Gaussian filters construction 2: build G1 ,G2 following (7) % Gaussian-based filter construction 3: build HG following (6) % Application of the filter in the frequency domain 4: compute Z as point-wise product between X and HG % Output in the spatial domain 5: compute z as the inverse Fourier Transform of Z

A Novel GPU Implementation for Image Stripe Noise Removal

2.1

237

Parallel Implementation via GPU-CUDA

The Algorithm 1 has been firstly implemented in sequential version on CPU, by using the C language. More in details, the images’ management is achieved by suitable routines of the OpenCV library [15], while the image storing is performed by a dynamic matrix allocation, through a buddy system allocator, as in [16]. In fact, due to contiguous memory address usage, the access memory time is considerably reduced. Moreover, the Fourier Transform and its inverse operation are carried out by means of a free software available at [22]. In particular, for the inverse Fourier Transform, in order to extract the real components of the final filtered image, we use the C function creal() [17]. Despite the achieved good quality in our experiments, we observed that, when the software runs on a latest-generation CPU, a very large execution time is required. Then, in order to improve the performance, we propose a parallel approach. To be specific, we have chosen to parallelize the main computational kernels of Algorithm 1, by exploiting the well-known powerful of most recent GPUs architectures, widely used across many application domains. Moreover, we combine the usage of two frameworks for the GPU environment: the OpenACC package [18] and the NVIDIA standard architecture CUDA [19]. The first one provides the upload and the mapping of the input images, while the second one performs the basic linear algebra computations. In addiction, for the operations related to the Fourier Transform and inverse Fourier Transform we used the cuFFT library [20], available on CUDA. The parallel implementation follows a similar flow of the Algorithm 1 with additional suitable routines offered by CUDA environment. The following steps summarize the parallel algorithm: Step 1: an ad-hoc memory management is performed, based on a no-pageable (pinned) memory space strategy. We force to build a bidirectional channel which is able to reduce the time in moving, reading and writing data among CPU and GPU memories [21]; Step 2: the combination of OpenACC and CUDA frameworks is considered. Starting from Algorithm 1, first parallel operations are related to loading and storing image by exploiting the high reliability of OpenCV library boosted with OpenACC directives; Step 3: the strong computational kernels (i.e. FT and IFT) are computed by the cuFFT library. We perform in-place direct Fourier Transform with the cufftExecC2C routine, while the Inverse FFT is computed by the same routine by setting CUFFT INVERSE as parameter; Step 4: overall computation is characterized by using a virtual memory environment where CPU and GPU communicate in a fast way. In particular, Step 1 allows us to reduce the access time in reading/writing both on the CPU and the GPU. More in details, in order to gain in term of performance and to overcome the limited sources of the Operating Systems, we have chosen a suitable memory strategy access, which works as follows: host data allocations are pageable by default, so the Operating System can swap

238

P. De Luca et al.

them out to the disk by decreasing the performance. More specifically, the Direct Memory Access (DMA) transfers to/from GPU without access to data directly from pageable host memory, as shown in Fig. 5, on the left.

Fig. 5. Left: Canonical DMA flow operations. Right: achieved memory structure.

Moreover, the data transfer to the GPU firstly requires that the host data has to be temporarily allocated as page-locked. This kind of memory access is also known as pinned memory paradigm. For this reason we design an accurate set of instructions in order to request the locking data (permanent pinned memory allocation). To do this, we allocate the dynamic memory on the host by using the CUDA routine cudaHostAlloc and we force the GPU kernel to read directly on a block page-locked host memory, by setting as parameter cudaHostAllocMapped. Figure 5, on the right, shows how the memory access is modified with our strategy. In addiction, we force the pinned memory also on GPU. Therefore, we have to menage two pointers: the first pointer is located in the host memory, by means of the routine cudaHostAlloc(), and the second one is located in the device memory, by the routine cudaHostGetDevicePointer(), and used to access to the locked memory from the kernel. In order to perform the page-locked memory mapping on GPU we must enable it by calling the routine cudaSetDeviceFlags(), with the cudaDeviceMapHost parameter, before any other CUDA call. Finally, our page-locked host memory strategy allows us to obtain several benefits: – the copy among page-locked host memory and device memory is performed concurrently with the kernel execution (Asynchronous Concurrent Execution); – the page-locked host memory can be mapped into the address space of the device, eliminating the need to copy data to/from device memory by mapping the memory.

A Novel GPU Implementation for Image Stripe Noise Removal

239

Then, we build a Write-Combining Memory (WCM) buffer, by forcing the data to be combined and temporarily stored after being written in fast way (burst modality). This approach does not guarantee an order in writing the memory, but it is not a problem for the correct computation and it improves the performance.

3

Results

This section shows some experimental results in order to confirm the efficiency and reliability of the parallel implementation proposed. Following, the technical specifications, where the GPU-parallel algorithm has been implemented, are shown: – two CPU Intel Xeon with 6 cores, E5-2609v3, 1.9 Ghz, 32 GB of RAM, 4 channels 51Gb /s memory bandwidth – two NVIDIA GeForce GTX TITAN X, 3072 CUDA cores, 1 Ghz Core clock for core, 12 GB DDR5, 336 Gb/s as bandwidth. Thanks to GPUs’ computational power, our algorithm exploits a combination of CUDA with OpenACC frameworks in order to take best advantage of parallel environment. Our approach relies on an ad-hoc memory strategy which provides to increase the size of local available stack memory for each thread and for each threads’ blocks. More precisely, we force the storing of data into local stack in order to decrease the memory bandwidth on global memory. Exploiting this technique, when a large amount of input data are assigned to each thread, the memory access time is reduced. Previous operations are executed by using the CUDA routine: cudaDeviceSetLimit, by setting as first parameter cudaLimitMallocHeapSize and as second one cudaLimitStackSize; according to hardware architecture, the size value 1024 × 1024 × 1024 is fixed [13, 14, 16]. This trick unlocks the dynamic memory allocation directly on the GPU device by using the malloc system-call. 3.1

Quality Measures

The important property of a good image denoising model is that it should completely remove noise as far as possible as well as preserve edges. To measure this issue, for our algorithm outcomes, we can use both qualitative and quantitative metrics. The test we consider involves the input image in Fig. 1 which is artificially altered by adding different noise thresholds. The output results are obtained using Algorithm 1. Both sequential and parallel implementations are used and provide the same results: we can observe the achieved quality in Fig. 6. In particular, for all noise values, the recovered images seem to be appreciable.

240

P. De Luca et al.

Fig. 6. Qualitative results for a fixed image sequence: on top the original image (without noise); on first row the images corrupted by stripe noise, respectively 2%, 4%, 8%, 10%; on second row the corresponding restored images.

This impression can be highlighted by using a quantitative measure: the peak signal-to-noise ratio (PSNR), which is defined as follows: P SN R = 10 log10

L2 M SE

where L is the maximum possible pixel value of the image and MSE is the Mean Square Error defined as: n

M SE =

1 (yj − yˆj )2 n j=1

The value n represents the number of pixels, while yj and yˆj are the pixel values referred to the original image (ground truth) and to the considered image, respectively. Notice that the larger PSNR the better the quality image should be. In Table 1, the PSNR values performed for the images of Fig. 6 are shown. The PSNR obtained confirm the perceived quality for all noise values. In particular, we highlight that for the image corrupted with a 10 % of noise, a good denoising outcome is also computed. Indeed, the high presence of noise allows to the algorithm the detection of noised region to restore. Hence a good restoration is performed also if a high noise level is present.

A Novel GPU Implementation for Image Stripe Noise Removal

241

Table 1. PSNR values Noise % Noised image Restored image 2

29.42

35.91

4

27.11

33.01

6

23.73

31.28

8

20.09

29.87

10

17.31

28.91

Finally, we are compared the proposed filter with the Cylinder-based one in terms of MSE, for the previous image with noise 10 %. We obtained M SEHC = 129.14 and M SEHG = 84.25. This result suggests that the filter (6) can offer better quality with respect to the Cylinder-based filter in (4). 3.2

Performance Analysis

In this subsection we carry out some comparisons among the sequential CPUalgorithm with respect to the GPU-parallel version in order to prove the better performance of the last one. More in details, two different metrics are used: first we compare the two implementations in terms of execution times; after a comparison in terms of Gflops is also presented. Table 2 highlights the gain of performance, in time terms for both serial (CPU) and parallel version (GPU), achieved by executing the parallel version of the proposed algorithm. The CUDA configuration is fixed to 1 × 512 (block × threads). This choice is strictly induced by our GPU hardware resources. More in details, we observe how the execution time of CPU-sequential versions gradually increases as the input data size is growing. The same situation happens for the GPU-parallel version. However, we get a considerably reduced execution time with respect to the sequential version. In fact, we save up to 97.2% of time (last row of Table 2). This comparison confirms the reliability of the accelerated implementation. Table 2. Execution times: CPU vs GPU for different input sizes Size (bytes) Sequential time (s) GPU time (s) 198 942

67.24

3.83

4 116 234

189.12

7.88

7 119 883

292.24

10.21

9 109 983

521.93

13.93

Table 3 shows the gain in terms of Gflops, comparing the operation number per seconds (CPU vs GPU) in several execution by varying the input size. We

242

P. De Luca et al.

observe an appreciable enhancement of performance obtained by exploiting the GPU architecture. Indeed, the table shows a increasing of performance, in terms of Gflops, for all execution of about 35×. Table 3. Performance in terms of Gflops Input size (Bytes)% Gflops CPU Gflops GPU

4

198 942

9.42

354.91

4 116 234

6.90

253.31

7 119 883

5.63

192.23

9 109 983

7.09

261.35

Conclusions

In this paper we proposed a novel method for stripe noise removal. The idea consists in the application of a Gaussian-based filter in the frequency domain. The related algorithm has been implemented in both sequential and parallel versions. Since the high computational cost, the parallel implementation has been designed using GPU architecture combined with the CUDA kernels and the OpenACC’s routines. Tests have proved the reliability of the method in terms of quality and execution times. Acknowledgment. This paper has been supported by project Algoritmi innovativi per interpolazione, approssimazione e quadratura (AIIAQ) and project Algoritmi numerici e software per il trattamento di dati su larga scala in ambienti HPC (LSDAHPC).

References 1. Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 3rd Ed., Prentice Hall, Hoboken (2008) 2. Buades, A., Coll, B., Morel, J.M.: A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4(2), 490–530 (2005). ISO 690 3. Rahman, Z., Jobson, D.J., Woodell, G.A., Hines, G.D.: Image enhancement, image quality, and noise. In: Proceedings of SPIE 5907, Photonic Devices and Algorithms for Computing VII, 59070N, 15 September 2005 4. Boncelet, C.: Image noise models. In: The Essential Guide to Image Processing, pp. 143–167. Academic Press, Cambridge (2009) 5. Kuan, D.T., Sawchuk, A.A., Strand, T.C., Chavel, P.: Adaptive noise smoothing filter for images with signal-dependent noise. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-7, no. 2, pp. 165–177, March 1985 6. Cuomo, S., De Michele, P., Galletti, A., Marcellino, L.: A GPU-parallel algorithm for ECG signal denoising based on the NLM method. In: 2016 30th International Conference on Advanced Information Networking and Applications Workshops (WAINA), Crans-Montana, pp. 35–39 (2016). https://doi.org/10.1109/WAINA. 2016.110

A Novel GPU Implementation for Image Stripe Noise Removal

243

7. Chen, S.W., Pellequer, J.L.: DeStripe: frequency-based algorithm for removing stripe noises from AFM images. BMC Struct. Biol. 11, 7 (2011). https://doi.org/ 10.1186/1472-6807-11-7 8. Dou, H., Huang, T., Deng, L., Chen, Y.: Stripe noise removal of remote sensing image with a directional l0 sparse model. In: 2017 IEEE International Conference on Image Processing (ICIP), Beijing, pp. 3505–3509 (2017). https://doi.org/10. 1109/ICIP.2017.8296934 9. Teramoto, A., Fujita, H.: Fast lung nodule detection in chest CT images using cylindrical nodule-enhancement filter. Int. J. CARS 8, 193–205 (2013) 10. Cuomo, S., De Pietro, G., Farina, R., Galletti, A., Sannino, G.: A revised scheme for real time ECG signal denoising based on recursive filtering. Biomed. Sign. Process. Control 27, 134–144 (2016) 11. Bovik, A.: The Essential Guide to Image Processing. Academic Press, Cambridge (2009) 12. Proakis, J.G., Manolakis, D.G.: Digital Signal Processing, 3rd Ed. Prentice Hall, Hoboken 13. De Luca, P., Galletti, A., Marcellino, L.: A Gaussian recursive filter parallel implementation with overlapping. In: 2019 15th International Conference on SignalImage Technology & Internet-Based Systems (SITIS), Sorrento, Italy, pp. 641–648 (2019). https://doi.org/10.1109/SITIS.2019.00105 14. De Luca, P., Galletti, A., Ghehsareh, H.R., Marcellino, L., Raei, M.: A GPU-CUDA framework for solving a two-dimensional inverse anomalous diffusion problem. In: Foster, I., Joubert, G.R., Kuˇcera, L., Nagel, W.E., Peters, F., (eds.) Parallel Computing: Technology Trends, Advances in Parallel Computing. vol. 36, pp. 311–320. IOS Press (2020) 15. https://opencv.org 16. De Luca P., Galletti A., Giunta G., Marcellino L., Raei M.: Performance analysis of a multicore implementation for solving a two-dimensional inverse anomalous diffusion problem. In: Sergeyev, Y., Kvasov, D. (eds.) Numerical Computations: Theory and Algorithms. NUMTA 2019. Lecture Notes in Computer Science, vol. 11973. Springer, Cham (2020) 17. https://pubs.opengroup.org/onlinepubs/009695399/basedefs/complex.h.html 18. https://www.openacc.org 19. https://developer.nvidia.com/cuda-zone 20. https://docs.nvidia.com/cuda/cufft/index.html 21. https://man7.org/linux/man-pages/man2/mlock.2.html 22. https://www.math.wustl.edu/∼victor/mfmm/fourier/fft.c

DevOps for Open Source Multimedia Frameworks Juan Zhao(B) , Ullysses Eoff, Focus Luo, Guangxin Xu, Haihao Xiang, and Hongcheng Zhong Intel IAGS, Shanghai, China {juan.j.zhao,ullysses.a.eoff}@intel.com https://01.org/linuxmedia

Abstract. DevOps is well deployed in “Software As A Service”, “Platform As A Service” or a lot of closed-loop software services and products. “DevOps” implies the collaborations between the “Development” and “Operations” team to achieve the same goal of a better development cycle and customer experience. In the closed-loop software, the operations are fully controlled by one party or federation. But in the open source development world, the codes contributed by different individual contributors are maintained by several key maintainers, handling the complexity of the target customers in different domains. One example is the frameworks of the multimedia middleware that provide the media capability to Clouds, Clients and Internet of Things (IoT) products: FFmpeg and GStreamer with a fully open source working model. A better way is required to feed the difference brought by the open source working model. This paper proposed Cartwheel, pre-checkin with DevOps, and the open source tool VAAPI-fits to address these new issues. We introduced the background of Intel’s contributions to the multimedia middleware, as well as the methods engaging with the open source working model and DevOps methodologies. Keywords: DevOps · Multimedia framework integration · Automated testing · VAAPI-Fits

1

· Continuous

Introduction

With the fast-growing and accelerating pace of the world’s connectivity over the last decade, DevOps has become a hot topic in the technology industry. The “DevOps” implies the collaboration between “Development” and “Operations” teams to achieve the same goal of a better development cycle and customer experience [5]. It also involves integrating long-standing development practices—such Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and noninfringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 244–254, 2022. https://doi.org/10.1007/978-3-030-80119-9_13

DevOps for Open Source Multimedia Frameworks

245

as continuous integration, continuous delivery, automated testing, and infrastructure as code. As such, it becomes a catch-all term of development process improvements both new and old [2]. Middleware bridges the gap between various hardware and software products. It provides the consistent behavior cross hardware. And software products or service providers are deploying the DevOps process, so it’s very important for the middleware to adapt to DevOps process. Agile, Scrum, Kanban, testing automation, continuous integration were used in the early days, but all individual processes are not unified. DevOps could link these fragile tools together and bring in more delivery methods [1, 3, 4].

Planning

New Feature

Development

New Patches

Backlog

Program office work out the plan

Continuous Integration

Developers prepare the code include validation

Continuous Delivery

Upstream Release

Upstreaming

Developers upstream the code

Community make the release

Fig. 1. Idea open source development process: the source code development goes through planning, development, continuous integration and continuous delivery process.

Ideally, the open source development process is quite simple: new feature planning, developing, upstreaming and releasing, please refer to Fig. 1. But there are three main issues when linking the open source development model with middleware DevOps: 1) Off-tree patches: There are some patches that cannot be “upstreamed” or the upstream process takes a longer time than expected. 2) Complex contributors: Lots of different developers are contributing to common middleware at the same time, so the pre-checkin and post-checkin life-cycle need better operations. Pre-checkin means the code commit is submitted out, but not merged. Post-checkin means the code commit is merged into upstream mainline. Sometimes, it’s called pre-commit or post-commit. 3) Different domains’ users: Upstream releases’s cadence are not always enough to feed the different use cases in different domains spreading from Cloud, Edge to clients. Meanwhile, multimedia is a complex world with various codecs that including: MPEG2, VC1, AVC, HEVC, AV1 etc. Each codec includes various extensions. Take HEVC for example, it supports 4:2:0, 4:2:2 and 4:4:4 pixel formats at 8-bit, 10-bit, and 12-bit pixel depths. And each codec has various ways to accelerate them, like GPU acceleration, CPU acceleration and Application Specific Integrated Circuit (ASIC) acceleration. Therefore, the DevOps for the open

246

J. Zhao et al.

source multimedia middleware is becoming very complex. A good testing tool is required to fix all these complexities. Below methods are proposed to address the above three issues (off-tree patches, complex contributors, different domains’ users): 1) Cartwheel provides a downstream tool to accumulate the off-tree patches which can’t be upstreamed in time, quality assurance of these codes, and a way to the final product releases/deliveries; 2) Pre-checkin and VAAPI-fits help to address the complex contributors and different users. They can help to guarantee the consistency of the codes’ quality before merged. The testing automation for pre-checkin can help to address the regression issues as early as possible, reducing the release/delivery burdens a lot. And VAAPI-fits (https://github.com/intel/vaapi-fits) is the proposed open source tool to provide the test automation, supporting the DevOps in the open source middleware world. This paper is organized as follows: Sect. 2 presents the principles and the proposed architecture of DevOps in opensource working model. In Sect. 3, we use FFmpeg/GStreamer as the example to explain the DevOps deployment. Then the proposed methods are described in detail: VAAPI-fits, pre-checkin and cartwheel. Finally, Sect. 4 provides the conclusion of the learning and the methods we used for DevOps in the opensource world.

2

DevOps Architecture in the Open Source

In the traditional DevOps workflow, lots of different hardware generations are required to be enabled and supported. “Backward compatibility” and “Seamless upgrade” are required to meet the functional and performance requirements from customers. In the open source world, the code may not be fully controlled by the production group. There are 3 common problems: 1) Unmerged patches which require downstream maintenance; 2) Unexpected integration issues which require pre-checkin and post-checkin lifecycle and release version management; 3) Mixed requirements from different domains which require better release and community support to feed different use-cases in different domains (e.g. IoT, Cloud, etc.). In the middleware domain, the customers vary a lot, and may not have the same criteria for the same middleware. For example, some customer may develop a new product through its API (e.g. OEMs, ISVs or Edge/Cloud Computing groups); some customer may use the demo application in the middleware directly; Some customers may pursue 1000 features, while some may pursue the simple and lite feature set. So, the middleware layer is the partner of these Application/Service/Product providers. And this requires the delivery must varies

DevOps for Open Source Multimedia Frameworks

247

Table 1. Principles Domain

Principles

Actions

Legal & Security

- Fully Compliance 1. Planning the license early - First Priority 2. Security policies training early to ensure no trade-off 3. Various security tools to ensure safety; e.g. static code analysis

Development

- Decent - Testability - Agile

1. Per patch regression tracking and reviewing 2. Architecture review early 3. Step by step improvement

Delivery

- Decent - Testability - Agile

1. Different delivery types to feed the different users 2. Good connection with dependencies 3. Actions in a fast cadence

Functionality & Stability

- Engage Early - Consistent - User obsessed

1. Unit, function, sanity, smoke, and full test 2. Stress, performance test 3. Test cases along with function development

according to the customers’ domain. To resolve the above listed issues, we should follow the principles (showed in Table 1) of the DevOps model in the open source development world. The code in the open source world is different, which includes two different status: “upstreamed” and “not upstreamed”. But the reason for “not upstreamed” (called off-tree patches) should not always be “not decent” or “not good”. Sometimes, it’s just because the maintainer is busy, or the review process needs to wait for the available hardware to validate. Therefore, the management of the “off-tree patches” is also important for a product release. Cartwheel is designed to feed these useful off-tree patches and can also be included in the final delivery in a way. Based on the above principles, Fig. 2 designed this lifecycle of the codes through DevOps in the open source world. The pre-checkin will be monitored before cartwheel, and post-checkin will be monitored in the delivery, and every activity should be well tracked through tools like JIRA. Based on the above analysis, to ensure the health of the codes’ lifecycle, the architecture in Fig. 3 is designed for the DevOps in the open source middleware. The target hardware, the tools for automation testing, agents, storage and build are set up. Through these tools, different types of testing, integration, and deliveries are provided by the DevOps process. The Scheduler will be responsible for all the tests automation. There are four kinds of tests:

248

J. Zhao et al. A line of code's real life Commnity Feedback Upstream

Reviewing and Validating

Coding

Upstream releases

Downstream (Cartwheel CI)

Delivery

Customer Feedback

Fig. 2. A line of code’s real life: the developer prepared the source code at first. Then the reviewer and QA will test the code. Before the developer submits the code to upstream, the code will be stored at downstream (cartwheel). The code will go through the upstream review process. This code may be merged into upstream or send back to developer to modify. If the product required this code but this code isn’t merged in upstream, The code will go through the internal merge process to get into the release as a patch from downstream (cartwheel). Binary Release

Devops

CI Server Scheduler

Source Code Release

Container Release

Full Tests

Smoke Tests

Conformance Tests

Post-commit Test

Sanity Tests

Resources

Build Agents

Unity Tests

Tools

Security Scan Static Analysis

Network Attached Storage

Configurations

Cartwheel (Downstream) Management

Pre-commit Test

Test Agents

Different Hardware Types

Fig. 3. DevOps and resources architecture: to support various releases and development lifecycle, different test types and scheduler should be designed and implemented. And the system should also provide the build agents, various tools, storage, test agents, and hardware as resources to support DevOps.

1) 2) 3) 4)

Unit Tests are used for each feature validation. Sanity Tests are usually used for pull in new platform enabling. Smoke Tests are executed regularly to track regressions (e.g. daily). Full Tests are executed regularly to run all the test cases (e.g. weekly).

DevOps for Open Source Multimedia Frameworks

249

Pre-commit Tests will leverage the unit/full tests but be executed before patches are integrated to cartwheel or mainline. If the test cases in Full Tests can be executed in one day, the smoke tests can be consolidated with Full. In the Delivery part, besides the binary and source code release, the container release and configurations should also be added for various users. Cartwheel, pre-checkin is proposed in this architecture to monitor the off-tree patches status and pull in the health detection. The VAAPI-fits is the automation tool, in the resources layer, to ensure every stages test. Sanity, smoke, full, precheckin tests are based on VAAPI-fits but with different test cases for different stages’ quality assurance. The delivery methods are also extended from the middleware release to different types of deployment, the container, source code, configurations etc. The users may use Openstack, Kubernetes to deploy their service, and they will get the releases/containers regularly to deploy their DevOps lifecycle.

3

Methodology

In this section, we will take FFmpeg and GStreamer for example to describe the detailed methodology of the proposed DevOps method. 3.1

Example of FFmpeg/GStreamer DevOps Deployment

In this section, the example will be introduced for FFmpeg (http://ffmpeg. org/) and GStreamer (http://freedesktop.gstreamer.org/) which are broadly adopted by the multimedia industry. Figure 4 shows the architecture of FFmpeg/GStreamer. VAAPI, OneAPI (was QSV) and some of the filter (including dnn) plugins are the main codes contributed to FFmpeg/GStreamer communities. The development work is to provide and refine the plugins in FFmpeg and GStreamer, enable new Hardware capability, improve the performance or enhance the user experience. The developers need to provide the patch, then go through the codes’ life cycle: 1) 2) 3) 4) 5)

Review and regression test cycle. Functionality test cycle. Cartwheel (downstream) management. Upstreaming cycle. Releasing cycle.

The review session varies a lot for these two community. The community’s culture should be followed to go through the public review session. For example, GStreamer community is using GitLab, and merge request way, while FFmpeg community uses git, email review and patchwork. And the tools preferred by

250

J. Zhao et al.

Edge/Cloud/IoT Services or applications FFmpeg/GStreamer Shared Layer: Base-class/Utility/Interface/other common layers VAAPI plugin

OneAPI plugin

Filter plugins

... plugin

... plugin

Drivers/Libraries CPU

GPU

AI

FPGA

Memory

Interconnect

Security

Fig. 4. FFmpeg/GStreamer architecture: FFmpeg/GStreamer provided the base layer for different and plugin methods. Then those plugins will call the system’s libraries and drivers.

community may also change, GStreamer switched from email review process to GitLab merge request in 2018. For the detailed execution on the DevOps, one solid staging area will be created to consolidate the automation interface. For FFmpeg/GStreamer automation, intel-media-ci (https://github.com/intelmedia-ci) is created to link the different cycles. Continuous Integration. The codes’ integration needs to be executed step by step, with regression check in each stage. And one line of code needs to be healthy in every commit-to-support hardware platform, say, “per-platform validation”. Based on the principles and architecture described in Sect. 2, below activities are designed in cartwheel: 1) 2) 3) 4)

Per-platform per patch pre-checkin test (in the smoke level). Per-platform daily sanity (hundreds of automated test cases). Per-platform daily smoke (thousands of automated test cases). Per-patch, Per-platform 3rd party required testing like FATE in FFmpeg, GStreamer meson unit test. 5) Per-platform daily IP scan. 6) Per-platform daily security static analysis. 7) Per-platform weekly full test (tens of thousands of automated test cases). Once the patches are merged to upstream, the post-checkin full round test will be monitored daily. And those patches will be removed in cartwheel. Figure 5 shows the test automation method. All the automated tests are included in the container or virtual machine to execute automatically. Most of the middleware part don’t need to change the kernel spaces of the operating system. The container could help the automation more efficiently.

DevOps for Open Source Multimedia Frameworks

GitHub

Poll

Build

251

Test

GitLab

Fig. 5. Test automation: The system will poll and pull the required source code from GitHub or GitLab, then it will build the related component to run a new test. The test result will be emailed to stakeholders and could be checked via web.

Usually the testing nodes are used to describe these running containers. Figure 6 shows one node’s architecture. Continuous Delivery. The delivery depends on the target users. For middleware, the target users vary a lot in different domain. There are service providers in the cloud side, application developers in the edge or device side, integration developers from Operating System Vendors (OSV), Independent Software Vendors (ISV) and so on. Consistency is important in the open source world. To avoid mixed version in the open source world, there are several major methods to deliver the contributions: 1) 2) 3) 4)

Regular releases along with upstream releases. Regular releases along with driver releases. On-command releases Supporting the users’ releases. Container releases with all in one.

Some of the users defined their community well (e.g. The OS Ubuntu), and the operation channel is very clear. Two different operation team can collaborate well. For example, they prefer the open source community releases, and provide the defined channel to submit the un-merged patches (from cartwheel) to avoid system crashes or pull in new features. Those un-merged patches in cartwheel are required to be upstreamed eventually. But some of the users don’t have a solid channel and their styles vary a lot. In this case, the middleware provider can’t set up every channel to talk with 1000 different users. In this case, the public information needs to be provided to describe about the releases and how to get the source code and libraries automatically.

252

J. Zhao et al. FFmpeg/GStreamer Target OS

OneAPI

Container

Low Layer API

Host OS

Device Driver

Hardware

Mem management

Fig. 6. The architecture of test nodes: The node will run its target OS in a container above the host OS over hardware. And the software stack of FFmpeg/GStreamer will be installed in the container.

In summary, below actions will be required: 1) Go along with community release, (e.g. https://github.com/FFmpeg, https://gstreamer.freedesktop.org/news/). 2) Connection with OSV/ISV providers, well align in cartwheel management. 3) Containers/stacks release, (e.g. https://clearlinux.org/sw/stacks, https://github.com/OpenVisualCloud/Dockerfiles). 4) Documentations about versions, dependencies, how-to. 3.2

VAAPI-Fits, Pre-checkin and Cartwheel

In the first Sect. 3 main issues (off-tree patches, complex contributors, different domains’ user) are discussed, and then the proposals are described about how to fix it in the open source DevOps: the “cartwheel” and the tool of VAAPI-fits, and added the actions in pre-checkin in the DevOps process. The details will be described in this section.

VAAPI-Fits-Smoke (Pre-check/Sanity test cases)

VAAPI-Fits-Full (Full test cases)

VAAPI-Fits (Python Scripts + Test Framework) Decode/Encode/Transcode/ Video Postprocessing

Fig. 7. VAAPI-Fits: VAAPI-Fits is written in python and open sourced. It provides encode, decode, video postprocessing and transcode tests. It supports different levels of tests, for example: sanity tests, smoke tests and full tests.

DevOps for Open Source Multimedia Frameworks

253

VAAPI-Fits. VAAPI-fits in Fig. 7 is developed in python to provide tools to test media middleware with its dependencies together. VAAPI-fits stands for VA-API Functional Integration Test. VA-API is the standard API proposed by Intel, but it can be used in combination with other devices (https://trac.ffmpeg. org/wiki/Hardware/VAAPI). The VA-API Functional Integration Test Suite is a configurable test suite for VA-API-based media software/middleware. It is written in Python and currently provides tests for VA-API hardware accelerated encode, decode and video processing pipelines for GStreamer and FFmpeg. It can be easily extended to support additional VA-API-based software/middleware. The tests execute the video pipelines and validate their outputs using common image and video metrics such as checksum, SSIM and PSNR. VAAPI-fits is written in python and uses slash to build up the testing framework. The source code is in https://github.com/intel/vaapi-fits. Pre-Checkin. Collaborated with VAAPI-fits, patches will have a healthy check before being committed. This process reduced the regression emergency cases a lot. Cartwheel. For the unmerged/off-tree patches, though they are not included in the mainline, they also need to be well managed. Figure 8 shows how cartwheel working with upstream and CI system.

GitHub

CI Full Test git clone (master)

repository Pull Request Reviewer

.git

Merge

Review Patches

Pull Request Submit PR/ Fix Conflicts

wget

Patches Conflicts?

Email Report

Pull Request Owner

PR Check

Build/ Full Test/ Report

PR Check Result

CI FFmpeg PR Pre-Check

Fig. 8. Cartwheel: cartwheel is designed to manage the not-upstreamed patches and provides the full test cycle for them.

After adding VAAPI-fits, Pre-checkin and Cartwheel, and with various delivery types in the open source DevOps development, these three main issues (offtree patches, complex code contributors, and various users) are well addressed. These regression emergency cases dropped 70%; the new feature delivery to customer is pulling in; the feedback from end users is more satisfied with this updated DevOps.

254

4

J. Zhao et al.

Conclusion

DevOps is deployed in a lot of closed source projects or closed-loop projects, but in the open source world, the development working model is different. In this paper, three main issues (off-tree patches, complex contributors, different domains’ user) are discussed for DevOps in the open source world. Cartwheel, Pre-checkin and VAAPI-fits are proposed in the DevOps of open source world with the real example of FFmpeg/GStreamer deployment. With these methodologies described in this paper, the open source middleware benefits a lot. The off-tree patches may bring value but need more time to review, cartwheel provides a way to deliver to customers with CI/CD. Besides the benefits brought by DevOps, adding pre-checkin into the methodologies can reduce the occurrences of integration and regression issues. VAAPI-fits provide the test automation framework. It can be used in different testing lifecycle and report out results timely, which improves efficiency. The practice in FFmpeg/GStreamer prove that DevOps can be deployed to open source middleware projects, with the community’s free resources, lowering the cost of development, operations, testing and deployment. Acknowledgment. Thanks to the architect Matthew Curfman, who provided his suggestions in the daily work after the media framework team’s reorganization. Thanks to Lena Zhuang, the security expert, who has been helping us in the Legal and Security domain. Thanks to IAGS managers (Michael Fu, Vladimir Travkin and more), who supported the CI-CD in the early days when it’s just an idea in the early days, and thanks to our media framework team members, and our partners/customers. Thanks to Shane Wang and more cloud experts to help us in OpenStack, Kubernetes and cloud domain. Thanks to FFmpeg/GStreamer community’s strong support in Intel’s contributions to the community.

References 1. Agrawal, P., Rawat, N.: Devops, a new approach to cloud development testing. In: 2019 International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT), vol. 1, pp. 1–4 (2019) 2. Davis, A.: Mastering Salesforce DevOps: A Practical Guide to Building Trust While Delivering Innovation. Apress, Berkeley (2019) 3. DevOps DevOps Working Group. IEEE draft standard for devops: building reliable and secure systems including application build, package and deployment. IEEE P2675/D1, pp. 1–95, June 2020 4. Wahaballa, A., Wahballa, O., Abdellatief, M., Xiong, H., Qin, Z.: Toward unified devops model. In: 2015 6th IEEE International Conference on Software Engineering and Service Science (ICSESS), pp. 211–214 (2015) 5. Wettinger, J., Breitenb¨ ucher, U., Leymann, F.: Standards-based devops automation and integration using TOSCA. In: 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, pp. 59–68 (2014)

Designing a Cost-Efficient Network for a Small Enterprise Lucky Osemwengie, Fahimeh Jafari(B) , and Amin Karami University of East London, University Way, Royal Docks, London 16 2RD, UK {u1600211,f.jafari,a.karami}@uel.ac.uk

Abstract. Reducing the cost of running a computer internet network in business cannot be over emphases in this current time of rising inflation. This paper assesses the cost of running an existing small enterprise network, called BCT Services Ltd, and suggests ways of reducing them. This will be done by; the introduction of solar panel, change in more scalable and less costly hardware, using more wireless access point in some areas and a different topology that is less costly to run. These changes have been designed in a proposed network scenario with Cisco Packet Tracer software for comparison with the existing one in the enterprise. In this paper, the parameter of interest is to know if the changes in the proposed network scenario will bring about the same amount of throughput in data or more. Our analyses and evaluations show that the proposed network scenario was more costefficient, as there was an improvement of over 100% data throughput compared with the current network, a cost savings of over £5,000 for five years period and other benefits like; reduce environmental pollution benefit from the solar panel. Keywords: Cost efficiency · Communication network · Small enterprise

1 Introduction This paper investigates the critical parameters which are most effective in lowering the cost of running a network in small enterprise and what can be done to further improve the reduction of these costs with contemporary ideas [1]. Most of the organizations that ventured into the network business are faced with the high cost of setting up and running a network despite the advancement and knowledge in network design and management. Global inflation and escalating energy costs have provided the impetus for a worldwide movement to reduce the cost of designing and running a network and the associated impacts on the customers arising from the high cost of subscribing to these networks [2]. What makes this project different is that existing research studies focused on analyzing critical parameters in minimizing the network cost while this work introduces other contemporary ideas that reduce network costs like the use of solar panels [3], wireless access point and modern network hardware [4]. These changes used to design a proposed network scenario with Cisco Packet Tracer for comparison with the existing network in the enterprise in terms of data throughput and latency. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 255–273, 2022. https://doi.org/10.1007/978-3-030-80119-9_14

256

L. Osemwengie et al.

This project uses a qualitative and quantitative method of data collection because the data needed for this research will the structured and unstructured information in other to arrive at an all-around decision. The small enterprise that this project will use as a reference point is BCT Services Ltd and they are currently running their network at a very high cost. This research addresses the following objectives; investigate the existing network infrastructure at the enterprise (BCT Services Ltd) and to identify the topology used, to identify areas that could be responsible for the high cost of running network in the enterprise (these parameters were selected from past literature and ones already used by the enterprise), to recommend measure on how to make the network cost-efficient for the enterprise, to design and implement a cost-efficient propose network scenario in a Cisco packet tracer, to evaluate the hardware and topology used in the network scenario and to draw conclusion about the effectiveness of the proposed changes carried out in the new scenario. In summary, this project makes these contributions using experimental and conceptual development. Experimental development means acquiring, combining, shaping, and using relevant knowledge and skills to produce plans and arrangements for new improved design, in this case the new proposed network design. Conceptual research is using a methodology wherein research is conducted by observing and analysing already present information on a given topic. Here, it is used during the critical parameter analyses of hardware [5]. First, it outlines the proposed designed network efficiency, which achieves significantly higher throughput and lower latency than the current network. This was made possible with the adjustment of the topology of the business. Second, we introduce solar panel energy for reducing the high energy consumption of the business network. Finally, the reduction in the quantity of equipment, through large-scale collapsing, adjustment of network topology and introduction of more Wi-Fi access point which lead to a reduction of maintenance cost and low initial capital investment [6]. In this project, the methodologies used is to analyse related works on different critical parameters that are most effective in minimising network cost, how it affects small enterprise running cost and the environment in general. It will also analyse the problem statements, the design of a new network, network configuration, evaluation, simulation of data throughput in the new network, an estimate of cost-benefit of the proposed network to the enterprise, contributions to this project and conclusion will be outlined in this research [7].

2 Related Works The following are critical parameters in minimising the network cost and additional contribution was researched on using various means like Google book, UEL Library and its Athens affiliated database. 2.1 Reduce the Cost of Labour The authors in [8] propose that we can reduce the network operator visit for network maintenance, where a visit is a time where scheduled maintenance is performed by the visit of a network operator, example of such visit is restoring energy to nodes in the

Designing a Cost-Efficient Network for a Small Enterprise

257

network. Each visit is called visit expenditure, while the visit lifetime is a time until the next visit. Visits can be adjusted by the network operator reducing visits to the nodes in order to minimize the cost of labour. In [8] they further imply that the total cost of labour can also be reduced by improving the reliability of the network. Doing so will require either by purchasing more robust hardware, this will reduce the frequent operator maintenance visit which will, in turn, reduce the labour cost. In [9] they suggested a method of balancing the cost of maintaining Network with network performance loss to reduce labour cost. Performance refers to the probability of the network detecting an event which prompts a decrease in performance as the number of failed nodes increases. Misra predicted a long-term cost of a network using a probabilistic model and maximum trade-off between node replacement cost and performance loss is determined by the provided algorithm. This technique ignoring the costs of node hardware and the labour required to initially deploy the network algorithm. 2.2 Adjusting the Network Layers The importance of the right network topology cannot be emphasised in reducing network cost, [10] talk about the different area and ways network topology can be modified for an enterprise. Some of the literature contributions in this area are as follows: According to [11], the goal of designing cost-efficient networks is not only for low initial capital investment but also to have good scalability and decreasing cost-pernode-per-unit-traffic for users as transaction size increase. That is why different topology design has helped reduce the cost of operation. For example, optical-cross-connect-based light-path switching and other enabling technologies are currently embedded in existing electronic routing and switching to lower operational cost and capital cost. The full potential of these technologies can be realized by good design of network architecture. In support to [11, 12] wrote that reducing network design layers comes with the desire to simplify its infrastructure and reduce costs. Two-tier architecture instead of three-tier reduces the cost of network ownership in terms of capital and running expenses. As shown in Fig. 1, by collapsing network layers, the business will be left with few numbers of equipment with which to run the network and Less equipment means lower costs of running the network. In this work, we will be using the idea of [13] which state that topology adjustment to consolidate infrastructure will take advantage of cheaper bandwidth. By consolidating your server and application infrastructure, the bandwidth and distributed architectures can be restructured to take advantage of cheaper bandwidth. Example, if your company pays more for WAN bandwidth and less for servers/server administrators, the extra telecom costs will be offset by the gain in the savings from the servers. This topology adjustment will also lead to a reduction of equipment resulting in short latency, good throughput of data and cost savings. 2.3 Use Wireless More There are several literature reviews involving academia and industry partners that are involved in the study of energy efficiency of wireless networks and other equipment in reducing the energy consumption of a business, thereby reducing the cost of operations.

258

L. Osemwengie et al.

Fig. 1. A main site-collapsed core network design [20]

For example, [14] takes a view on the topic, that providing energy-efficient wireless access services has been the focus of most work in the field. The literature has two main directions: Reducing the power consumption, for example, by employing power-saving network deployment strategies and using more efficient equipment or more advanced software to adapt the power consumption to the traffic situation. Example of such equipment is one developed by the Zig-Bee Alliance, Zig-Bee 802.15.4 which is a wireless personal area network and based on the IEEE 802.15 standards. It provides communication in a short range with a low data rate at low cost and less energy consumption. Furthermore, users will experience even higher performance-with fast Ethernet speeds delivered almost anywhere with the introduction of high-speed 802.11n WLAN technology, while installing cable only where it is cost-effective to do so will allow organizations to confidently deploy mission-critical applications and communications [5]. An example can be seen in Fig. 2 below:

Fig. 2. A main site network design

This work adopts energy-efficient and smart wireless device in line with [25] to replace some wired network in the business.

Designing a Cost-Efficient Network for a Small Enterprise

259

2.4 Cheaper and Scalable Hardware The authors in [5] discuss how equipment vendors can build high-speed interfaces by integrating more than one port on a single interface card which is allowed with advances in switching systems technology. The cost per port will decrease in this equipment. The practical situation in which ports are fabricated on a single blade is addressed by the study, interfaces with an aggregate capacity of OC-768 (40 Gbps), in which four OC-192 ports are integrated on a single card has been built by several equipment vendors. This reduces the cost to the business compared to buying more port. Furthermore, the research of [15] scale down cost has also been suggested, the use of cheap hardware in the network design as there is no shortage of second-tier hardware manufacturers which are eager to sell routers and switches at a fraction of the cost compared to Cisco equipment and these cheap equipment function as that of the rated ones. In this project, an effort will be made to use low cost and scalable equipment in line with the suggestions of [15], to design the proposed network. This will be explained further in my proposed section later. 2.5 Cheaper and Scalable Hardware Power consumption in an enterprise office network is a major issue due to its running cost. Several approaches have been proposed such as Energy Efficient Ethernet in order to reduce the power consumption, solar panel installation and Energy Efficient Wireless aggregation will bring about much cheaper energy in the long run. A lot of work has been done on conservation of power consumed by IT equipment, in the literature of [2], which suggested Energy Efficient Wireless aggregation which focuses on the power consumed by office networks and tries to save the power by aggregating low-utilized users to the wireless network and turning off the switches that do not have active users. In addition, [16] In their paper applied the power consumption model for office networks to evaluate the performance of Energy Efficient Ethernet and Energy Efficient Wireless, a further possibility of conserving power by combining the existing power saving techniques was also explored by them. The result of the analysis showed that for the average throughput up to 0.5 Mbps, combining Energy Efficient Ethernet and Energy Efficient Wireless, if using only Energy Efficient Ethernet, the business can save more than 66% of the power consumed and the percentage energy saved is more if the link-sleep technique is put into the combination. In contrast, [7] wrote about the combination of two energy-saving approaches. The first is based on adapting the rate of a network in response to the offered workload, the other by conserving excessive power consumption during idle times which involve putting idle network components to sleep. A solar panel might not be producing energy during the night hours. However, concentrating solar operations have the option to store energy as heat to use for generation during the night hours which is flexibility when planning solar installations. In addition, concentrating solar power plants have the capability to run off fossil fuel or gas as a backup power source [17]. This paper adopts the combination method of solar panel and subscription to an energy provider as suggested by the research papers above.

260

L. Osemwengie et al.

3 Problem Statement BCT Services Ltd is a small company involved in the sale, installation and delivery of electronics in London, and on customer premises in England. The business is staffed by the Sales manager, ten sales staff, two marketing staff, and two accounts staff in its head office in London. The Managing Director (MD), his secretary and receptionist are also located in the head office. At the Branch Office, the staffs are a manager and two technicians and a warehouse where their goods are kept have two staffs. The sales staffs make the most sales over the telephone and computer order template. The marketing department has two computers, the accounts department uses two PC running on accounting software and the Sales department has four PC and one Laptop for preparing quotes. The MD’s secretary also has a computer and all these computers are out-of-date and need to be replaced. The network topology currently in use is mesh topology, a two-tier hierarchical design of core and distribution layers and an exorbitant energy bill every month. 3.1 Building Layout The building is a single story with solid floor 25 m by 20 m, suspended ceiling and studpartition walls. It is in the City while the branch is in an Estate 2 miles apart. There are three external doors, a main entrance to the front, delivery entrance to corridor adjacent to store room and fire exit to rear as shown in Fig. 3.

Fig. 3. BCT office layout and branch office [18]

3.2 Services The building has recently been rewired, with power being supplied to sockets on both interior and exterior walls and the walls have conduits running from the ceiling. There is air conditioning and all staff has a telephone socket which is located on walls. The switchboard is in the reception.

Designing a Cost-Efficient Network for a Small Enterprise

261

3.3 Functional and Non-functional Requirements It requires a robust infrastructure that supports voice, video, data, virtualization, wireless and ability to provide solutions for the traffic of guests and internal staff for both wired and wireless which must comply with the standard security policy of the business; capability to support adding new remote sites to the network without any redesign. The Managing Director of BCT Ltd views the current system as inefficient and worries about the high cost of running the network. He requires a smart network with fewer nodes, more wireless access points, energy-saving initiatives and modern network to reduce the ever-increasing running cost of the business. From the investigation of BCT Ltd business shown in Fig. 4, it was revealed that the topology was too robust for the small business leading to high latency of data, too much energy bills from energy provider, no regular network staff training in the business, too much duplication of equipment and low wireless usage.

Fig. 4. The current BCT network

4 Proposed Approach Some other contemporary critical parameters that this project will be implementing will bring about a cost-efficient network that has a very lower running cost and more data throughput. The factors are changes in network topology, using more wireless, reducing energy through the installation of solar panel and how the network elements can be replaced to minimize cost with the same network throughput [10].

262

L. Osemwengie et al.

4.1 Topology A network topology is very important in network design; it determines how data passes the nodes, the cost of maintenance and the time it takes to its destination. The main three types of topology are shown in Fig. 5. Tree topology is particularly appropriate for low-power and low-cost wireless sensor networks as it sustains power saving operations and lightweight routing. IEEE 802.15.4 MAC super frame structure manages the power saving operation [19]. However, any link failure will abort the data delivery completely which makes the recovery operation incurs significant overhead, restricted routing and poor bandwidth utilization are the drawbacks of this topology.

Fig. 5. Types of topology [12]

The tree topology, which is particularly appropriate for low-power and low-cost wireless sensor networks as it sustains power saving operations and light-weight routing will be used in this research network scenario because it is the most cost-efficient topology to a small enterprise, and it gives the same throughput compared to the mesh topology used in the business. Whenever the energy-saving method is required, the tree topology is preferred compared to star, mesh or bus topology as shown in Fig. 6 [16]. This topology was also chosen in line with the research paper of [12] in Sect. 2.2 of the literature review, where it was demonstrated that topology adjustment in line with the business need can reduce cost. 4.2 Wireless Networks are breaking free of wired networks and going wireless; this allows a certain amount of freedom when using devices as Wireless Laptops, printers, cameras and many other new devices. Temporary office, and some hardware will be reduced due to a change in network technology. Today wireless technology has grown to the extent that it made provision of communication within a short distance possible, one of the wireless equipment that makes these things easier is Zig-Bee technology which is a wireless personal area network (WPAN) base on IEEE 802.15.4 standard to provide communication with low data rate at low cost [20]. In this project, wireless wi-fi access point was used more in line with the suggestion of [14] because there is modern strong wireless access point as discussed in Sect. 2.3 of the literature review that works effectively to reduce wastage of wires, consume less energy and flexible connections. The weak point is that the signal can be low if not strategically positioned.

Designing a Cost-Efficient Network for a Small Enterprise

263

4.3 Energy In business, the reliability and capacity of power source directly influence the Cost of running a network. That is why Power saving technology, energy efficient Ethernet and smart wireless equipment were recommended; in addition, the solar panel will be installed to bring about a reduction of cost of energy in the long run and for a much cleaner environment. In [21], they report that since the invention of the PV cell, it has become the preferred method of power generation for small scale solar panel installations because PV cells are low to maintenance, ideally requiring no attention at all after the initial setup. The system (shown in Fig. 6) which utilize a charging system and battery for backup power is called Off-grid systems and the one (shown in Fig. 7) which utilize the grid for backup power, as well as feeding excess power back into the grid are two basic setups for small scale PV arrays is called grid-connected systems [21].

Fig. 6. Off-grid PV system setup [8]

Fig. 7. Grid-tied PV system setup [17]

This project uses solar panel method as describe in [17] for more than half of the energy need of the business as researched by [7] in Sect. 2.5 of this project and the other half of energy need of the business by public energy provider to design the proposed network, because it costs less compare to their current dependent on only public energy provider. This proposed design is shown in Fig. 8.

264

L. Osemwengie et al.

Fig. 8. The proposed network design

5 Simulation Results The testing of the current and the proposed network infrastructure was done in cisco packet tracer. Static routing procedures was used in the network simulations and In dividing this IP addresses among the subnets, we use variable length subnet mask (VLSM) to allocate it subnet by subnet, some bits for the subnet and some for the host which will create additional bits for the for the prefix. The result from the test carried out shows that all the network nodes were all working, and this can be seen from the ping and simulation results. I was able to ping from the main office to the branch office, from the branch office to the main office and within the main office. With respect to the Throughput, the time it took a packet of 32 bytes to move from the computer (pc8) in MD’s office from the main building to the computer (pc20) in the branch office was 0.889 s. which is less than that of the current network of 2.974 s. of the same distance as seen in Fig. 14. 5.1 Simulations and Ping Result of BTC Old Network The simulations and pinging were successful from computers in the Branch office to others in the main office and within the main office as shown in Fig. 9 and 10.

Designing a Cost-Efficient Network for a Small Enterprise

265

Fig. 9. Simulation of the old network

Fig. 10. Ping results

5.2 Simulations and Ping Result of BTC New Network The simulation and ping of the newly design network were also successful as shown in Fig. 11 and 12 below.

266

L. Osemwengie et al.

Fig. 11. Simulation of the new network

Fig. 12. Ping results

6 Evaluation The evaluations of the two networks are based on the following criteria and the analyses of the criteria are as follows. 6.1 Throughput Throughput which is the rate of successful message delivery over a network and its value is expressed in bits per second (bps). The proposed network has been improved on by the above contributions which bring about efficient, reliable and more throughput compared to the BCT services Ltd current Network scenario [22]. As can be seen in Fig. 13, the

Designing a Cost-Efficient Network for a Small Enterprise

267

time taken by a packet of 32 bytes to move from the computer (pc8) in the main building MD’s office to the computer (pc20) in the branch office was 0.889 s. (as indicated by the arrow) which is less than that of the current network of 2.974 s. (as indicated by the arrow) of the same distance as seen in Fig. 14.

Fig. 13. The proposed network throughput

Fig. 14. The current network throughput

268

L. Osemwengie et al.

6.2 Energy Consumption Energy is saved when some equipment was reduced or collapsed to a smaller one, the old equipment which used to consume high energy were replaced with modern equipment that is more powerful and consumes less energy. The introduction of solar panel which reduces the cost of energy on the long run has contributed to the savings from high bills (£2,000 from BCT Ltd Records) being paid before now. With the installed solar panel, the monthly energy bills have reduced to £500. The rate of kWh was taken from [23] and how it was calculated was taken from [13]. Table 1 presents the amount saved in energy bills due to less equipment. Table 1. Energy hours gained from equipment table. Sources of kWh rate (first choice power, 2019 and SaveOnEnergy, 2018) Current network (Energy at 10 h/Day) Desktops (200 w/h 13 = 26,000 Each)

Proposed network (Energy at Total cost of 10 h/Day) hrs. gained 11 = 22,000

Laptops (75 w/h Each)

1 = 750

3 = 2,250

Wireless Access p. 20w/h @

1 = 200

3 = 600

Servers 250 w/h Each

3 = 7,500

2 = 5,000

Routers 20 w/h Each

4 = 800

2 = 400

Total Hours/1000 Kw

35,250/1000 = 35.25 x300 Days = 10,575

30,250/1000 = 30.25 x300 Days = 9,075

Total Hours/1000 Kw

1,163

998

165

6.3 Cleaner Environment The introduction of solar panel will reduce the level of pollution and make the environment cleaner. This will bring about cost savings through low or no energy subscription payment as electricity is saved and use from the installation of the solar panel. Solar energy is currently growing at an unprecedented rate as a source of energy for offices and home for some time now. If every business use cleans energy like solar, there will be a reduction in the production of energy by nuclear power which released enormous amounts of radiation into the surrounding communities to endanger the lives of people in that environment. Desktop computer energy consumption corresponds to CO2 emissions of about 175 kg per year why that of Laptops and wireless routers is 50 kg according to [16]

Designing a Cost-Efficient Network for a Small Enterprise

269

report. This indicates that using less desktop and routers reduce amounts of radiation into our environment, which is the aim of this project. 6.4 Topology Adjustment The topology of the business has been changed from mesh topology to tree topology. Tree topology is most appropriate for a small business with a small network of low-power equipment, as discussed in Sect. 4.1 above. From the test, the proposed network with the tree topology was faster in throughput and it is also cheaper to maintain compared to the current network. Labour cost to visit each node, according to [24] average computer technician pay was £15 per hour and it took the expert to visit each equipment for maintenance one days of 5 h for a month according to the record of BCT Services Ltd. But with less equipment to visit, some hours are gained, see Table 2 below. Table 2. Labour hours gained table sources of hour pay rate (Indeed, 2019) Hours per month

Hours saved at £15 per hr (x £15ph)

Current hour spend per month

5

75

Proposed hour spend per month

3.5

52.5

Total cost saved for a year

Total save per year (£) (x12 months)

630

6.5 Running Cost/Saving Savings was gained when the network topology was adjusted, and some hardware were replaced with a wi-fi access point. These savings also include reduced IT service manpower, as less time will now be used for repair and supervise less equipment, reduction in energy consumption from low energy consumption modern equipment and savings from the installation of solar panel [25]. The total saving or gain in five years was over £5,000.00 which represents about 10% reduction in expenses as compared to the business total expenses as shown in Table 3. In summary, the intrinsic benefit of the proposed network scenario cannot be quantified, these are the reduction in air pollution and easy to maintain equipment. This savings might look small to a multi-million pounds business, but this becomes a large percentage saving to a small business with rising running cost.

7 Discussion The result from the calculations above shows that the proposals implemented, and adjustment of the network has yielded a good result of reducing the business network running

270

L. Osemwengie et al.

Fig. 15. Running cost comparison of current and proposed network sectors

Fig. 16. Running cost comparison of current and proposed network

cost. Example: Energy bills which was £2,000 per month was reduced to £500 due to the installation of solar panel which cost £6,000 initially and no other cost on the long run and a total of £1,500 pounce was saved in this area for a year. The equipment that was reduced due to topology adjustment has also yielded some savings which is as a result of less power now consumed by the appliances. A total of £850 pounce was saved for a year as a result of the adjustments. IT maintenance manpower being incurred by the business currently was £900 pounce, has now been reduced to £270 due to reduced equipment now maintained after the proposed topology adjustment. A gain of £630 was also gained in this area for a year. In Table 3, Fig. 15 and 16, you can easily see that the running cost of the business has reduced due to the factors analysed and implemented from the literature review and the proposed contributions in line with the objective of this project, which is to design a network that will be cost-efficient. The strength of the approach used in the project lies in the numerous literature reviews suggestions, the proposed solutions used and the successful execution of this approach in cisco packet tracer which resulted to a low data latency rate, higher data throughput and reduced running cost of the business. Some of the weaknesses of the approach are

Designing a Cost-Efficient Network for a Small Enterprise

271

Table 3. Running cost/savings summary table sources of solar panel and energy bills rate (Green Match, 2019) Changes

Current network Cost over 5 years Proposed in (£) P/A in (£) network in (£) PA

Cost over 5 years in (£)

Energy Bills/Solar 2,000 (BCT Ltd Panels (4KW) Record) (Green match, 2019)

10,000

6,500 (6,000/ Panel Cost)

8,500

Savings from less equipment (Energy Hour Table 1)

1,163

5,816

998

4,991

IT manpower Reduction (Table 2)

900

4,500

630

1,350

Saving for a period of 5 yrs Total cost Saved for a year

20,316

14,841 5,475

the low internet signal rate in some areas due to high rate of wireless access point used in the business and the huge capital outlay for the purchases of solar panel.

8 Conclusion The paper presents a cost-efficient network for small business as mentioned in aims and objectives and the various factors that make it more efficient compared with the network of the already established business (BCT Services Ltd) used as a reference network. The critical factors highlighted in the literatures are applied to the proposed network scenario. The evaluation, performance and analyses indicate an improved performance on data throughput and lower latency, and a savings of over £5,000 for a projection of five years period in network running cost. In conclusion, we have efficiently designed a business network that reduces the business running cost; the proposed approach that enhances the achievement of a costefficient network solution and have achieved nearly 10% reduction in running cost. This work also serves as a one-stop shop for critical parameters that need to be looked at in other to reduce any business network running or capital cost. Some implication in the findings shows that the resulting overall cost reduction is not high as anticipated before the research and this can lead to business investment in this area with the hope of ripping immediate profit rather than long-term gain and this research finding may lead to high investment cost in the short run. To achieve high percentage reduction in running-cost the project recommends that: this research expand to involve large numbers factors, including more works of literature views, more use of modern computer components, use of network topology that fit the needs of the business and used of powerful simulating equipment to bring out a clearer

272

L. Osemwengie et al.

images/figures, diagrams and pictures. In the future, we will evaluate more performance metrics such as delay, packet loss and develop a simulation-based on real traffic to verify the validity of the analysis.

References 1. Dawson, C.W.: Projects in Computing and Information Systems: A Student’s Guide, 3rd edn. Pearson Education, Harlow (2015) 2. Gunaratne, C., Christensen, K., Nordman, B., Suen, S.: Reducing the energy consumption of ethernet with adaptive link rate (ALR). IEEE Trans. Comput. 57(4), 448–461 (2008). https:// doi.org/10.1109/TC.2007.70836 3. Awad, M., Khair, O., Hamdoun, H.: An energy optimization model (EOM) to reduce mobile service providers network costs: a multi-objective optimization approach. In: 2015 IEEE International Telecommunications Energy Conference (INTELEC) (2015). https://doi.org/ 10.1109/intlec.2015.7572466 4. Cottrell, S.: Dissertations and Project Reports: A Step by Step Guide. Palgrave Macmillan, Basingstoke (2014) 5. Miller, H., McNeal, K., Herbert, B.: Inquiry in the physical geology classroom: supporting students’ conceptual model development. J. Geogr. High. Educ. 34(4), 595–615 (2010) 6. Eyal, I., Gencer, A.E., Sirer, E.G., Van Renesse, R.: Bitcoin-NG: A Scalable Blockchain Protocol. Cornell University (2016) 7. Nedevschi, S., Popa, L., Iannaccone, G., Ratnasamy, S., Wetherall, D.: Reducing network energy consumption via sleeping and rate adaptation. In: Proceedings of 5th USENIX Symposium on Networked Systems Design and Implementation (2008) 8. Dorling, K., Messier, G.G., Valentin, S., Magierowski, S.: Minimizing the net present cost of deploying and operating wireless sensor networks. IEEE Trans. Netw. Serv. Manag. 12(3), 511–525 (2015). https://doi.org/10.1109/tnsm.2015.2464071 9. Misra, S., Mohan, S., Choudhuri, R.: A probabilistic approach to minimize the conjunctive costs of node replacement and performance loss in the management of wireless sensor networks. IEEE Trans. Netw. Serv. Manag. 7(2), 107–117 (2010) 10. Forouzan, B.: Data Communications and Networking. McGraw-Hill, New York (2013) 11. Guan, K., Chan, V.: Cost-efficient fiber connection topology design for metropolitan area WDM networks. J. Opt. Commun. Netw. 1(1), 158 (2009) 12. Nourildean, S.W.: Study of zig-bee network topologies for wireless sensor network with one coordinator and multiple coordinators. Tikrit J. Eng. Sci. 19(4), 65–81 (2012) 13. SaveOnEnergy. Calculate your energy consumption (2018). https://www.saveonenergy.com/ energy-consumption. Accessed 30 Mar 2019 14. Tombaz, S., Vastberg, A., Zander, J.: Energy- and cost-efficient ultra-high-capacity wireless access. IEEE Wirel. Commun. 18(5), 18–24 (2011). https://doi.org/10.1109/mwc.2011.605 6688 15. Zhu, H., Zhu, K., Zang, H., Mukherjee, B.: Cost-effective WDM backbone network design with OXCs of different bandwidth granularities. IEEE J. Sel. Areas Commun. 21(9), 1452– 1466 (2003). https://doi.org/10.1109/jsac.2003.818232 16. Nguyen, P., Morohashi, T., Imaizumi, H., Morikawa, H.: A performance evaluation of energy efficient schemes for green office networks. In: IEEE Green Technologies Conference (2010) 17. Furman, A., Rashid, M.: Solar feasibility-can solar energy compete economically? In: CONIELECOMP 2011, 21st International Conference on Electrical Communications and Computers (2011). https://doi.org/10.1109/conielecomp.2011.5749323

Designing a Cost-Efficient Network for a Small Enterprise

273

18. Kahode. Kahode office Design Ideas (2019). https://www.kahode.com/office. Accessed 08 Apr 2019 19. Batayneh, M., Zhu, H., Song, L., Mukherjee, B.: OPN02-5: cost-efficient WDM mesh network design with line cards of multiple ports. In: IEEE Globecom 2006 (2006). https://doi.org/10. 1109/glocom.2006.368 20. Kumah, M., Singh, S., Taneja, B.: Relative study of zig-bee/IEEE 802.15.4 WPAN performance under different topology. In: Fifth International Conference on Advanced Computing & Communication Technologies (2015) 21. El-Sharkawi, M.A.: Electric Energy: An Introduction, 2edn. CRC Press, Boca Raton (2009) 22. Cisco: Small Enterprise Design Profile Reference Guide - Small Enterprise Design Profile (SEDP)—Network Foundation Design [Design Zone for Security], Cisco (2018). https:// www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Small_Enterprise_Design_Profile/ SEDP/chap2.html. Accessed 1 Dec 2018 23. First Choice Power: Energy-Education (2019). https://www.firstchoicepower.com/the-lightlab/energy-education. Accessed 08 Apr 2019 24. Indeed: Salaries/Computer-Technician (2019). https://www.indeed.co.uk/salaries/ComputerTechnician-Salaries-at-IT-Support.support. Accessed 08 Apr 2019 25. Green Match: Installation Cost of Solar Panels (2019). https://www.greenmatch.co.uk/blog/ 2014/08/what-is-the-installation-cost-for-solar-panels. Accessed 08 Apr 2019

Staff Rostering Optimization: Ideal Recommendations vs. Real-World Computing Challenges Kimmo Nurmi(B) , Jari Kyngäs, and Nico Kyngäs Satakunta University of Applied Sciences, Satakunnankatu 23, 28130 Pori, Finland [email protected]

Abstract. Staff rostering is a difficult and time-consuming problem that every company or institution that has employees working on shifts or on irregular working days must solve. The Finnish Institute of Occupational Health, which operates under the Ministry of Social Affairs and Health, published their recommendations for shift work in 2019. The recommended values for these individual factors are well justified. However, problems arise when all these recommendations should be satisfied together in real-world staff rostering. This paper shows what can be done to reach the best compromise considering the ideal recommendations, the employer’s point of view and the employees’ point of view. We use the PEAST metaheuristic, a computational intelligence framework, to show how the recommendations and employer requirements compete with each other. We give justification of why we can safely rely on the practical findings given by the metaheuristic. Keywords: Staff rostering · Nurse rostering · Workforce scheduling · Shift scheduling · Shift work · Workforce optimization · Computational intelligence · PEAST metaheuristic · Real-world computing

1 Staff Rostering Staff rostering is a difficult and time-consuming problem that every company or institution that has employees working on shifts or on irregular working days must solve. Staff rostering assigns employees to shifts, and determines working days and rest days. Application areas include hospitals, retail stores, call centers, cleaning, home care, guarding, manufacturing and delivery of goods. The main goals of staff rostering are the performance of staff on financial efficiency and customer and employee satisfaction. Good rosters lower costs, utilize resources more effectively and balance workload more fairly. Workforce schedulers should also address the health, safety, and well-being of the employees. The workforce scheduling process has five major components (see Fig. 1). First, workload prediction is the phase of determining the workload based on known and predicted events. For example, the arrivals of customers can be predicted using models based on queuing theory, simulation and statistics. Second, in shift generation (see e.g. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 274–291, 2022. https://doi.org/10.1007/978-3-030-80119-9_15

Staff Rostering Optimization

275

[1] and [2]), the target is to create shifts that match the known and determined workload as accurately as possible. The result includes the shift structure, the tasks to be carried out in particular shifts, the competences required for different shifts and the break times required. Third, the staffing requirements must be carefully considered in resource and holiday planning. Holidays, training sessions and other short-term absences as well as long-term sick-leaves and forthcoming retirements have major impact to actual staff rostering.

Fig. 1. The workforce scheduling process.

Fourth, the generated shifts form an input for the staff rostering (see e.g. [3]), where employees are assigned to the shifts. The rostering includes the scheduling of days-off and other vacations. The working and resting times are laid down by the collective labor agreements and government regulations. Other important targets include employees’ competences and preferences. The fifth major component, daily rescheduling, deals with ad hoc changes that are necessary due to sick leaves and other no-shows. The workforce management system should recommend suitable substitutes considering the competences, employment contracts, legal limitations and salaries. The goal is to find the most economical candidates. Finally, the completed working times will be booked and made available for payroll accounting system. The four basic requirements of the staff rostering are given in Table 1. Working hours acts and employment contract acts set regulations for shift work. Furthermore, several practices and objectives are generally agreed at the company level. These regulations and agreements provide the basis for staff rostering. Particular attention should be paid to the first requirement. This means that the workforce requirements for every single shift must be fulfilled and guaranteed. At the turn of the 80s, Garey and Johnson [4] and Bartholdi [5] showed that the staff rostering problem was NP-hard. The following two decades introduced several new

276

K. Nurmi et al. Table 1. The four basic requirements of the staff rostering.

# Requirement 1 A minimum number of employees with particular competences must be guaranteed for each shift 2 An employee’s total working time should be within the given limits 3 An employee can only be assigned to a shift she/he has competence for 4 The planned number of holidays must be respected The government regulations, labor agreements and work contracts must be respected

application areas, models and algorithms for various staff rostering problems. A good review can be found in Ernst et al. [6]. At the turn of the millennium, a large number of commercial software packages were available to assist with rostering. The first significant academic-based optimization packages began to rise in the middle of the 2000s, see for example the works by Bard and Purnomo [7], Burke et al. [8], Bilgin et al. [9] and Beddoe et al. [10]. Staff rostering algorithms have made convincing advances in the last ten years. Dozens of objectives can be simultaneously optimized, and impressive solutions can be reached within reasonable runtime. For recent research-oriented workforce scheduling software, see Burke and Curtois [11], Nurmi et al. [12], Jin et al. [13], Kingston [14] and Gärtner et al. [15]. An excellent literature review of the latest staff rostering studies can be found in [3]. The Finnish Institute of Occupational Health has published their recommendations for shift work [16]. The goal of this paper is to show that the recommendations and employer requirements compete with each other. Section 2 presents some legal and human aspects of shift work, as well as the ideal recommendations for shift work. Section 3 describes the computational intelligence framework used for the staff rostering optimization test runs. Section 4 shows how the ideal recommendations and employer requirements compete with each other. We show the difference between the theoretical expectations and the every-day reality. In Sect. 5, we show what can be done to reach the best compromise considering the ideal recommendations, the employer’s point of view and the employees’ point of view. We believe we can give new ideas to the HR managers and the shift schedulers using manual, autonomous, automated and/or optimizationbased planning tools.

2 Ideal Recommendations The efficient use of workforce is a balanced outcome of realizing employer, employee and customer satisfaction. The overall objective is to meet daily staffing requirements and personal preferences without violating government regulations, collective labor agreements and personal work contracts. A shift scheduler should consider human aspects as well as legal aspects of shift work. The most important targets are working hours, resting times, weekend work and rest, and different rule sets referring to work content.

Staff Rostering Optimization

277

From an employee perspective, shift work is associated with pressure on social and family life, health issues, motivation and loyalty. Humans live in a biological rhythm and in a social rhythm. These rhythms and irregular shift work are often in conflict. This can damage humans’ health and well-being. Rapid changes to our sleep-wake cycle leads to misalignment (desynchrony) of rhythmic physiological systems, such as sleep, alertness, performance, metabolism and the hormones melatonin and cortisol [17]. Longterm issues with sleep may cause heart disease or Alzheimer’s disease. The sleep-wake cycle has an extensive influence on us also from a social perspective. Social activities occur mostly in the evenings and on weekends forming a social rhythm [18]. Working hours on evenings and at night impair social well-being and work-life balance [19]. Several studies have found that short rest between shifts (quick returns) and night work is associated with work-related accidents. For example, the study by Vedaa et al. [20] examined whether less than eleven hours between shifts and night shifts is associated with self-reported work-related accidents and near accidents. Shift working is associated with severe health issues and significant pressure on social life. Some individuals are more influenced by these effects than others are. Therefore, it is extremely important that employees have control over their working hours. This buffers the negative impact of shift work on their health and work-life balance. This understandably decreases sickness absences, increases working capacities and lengthens careers. High levels of work time control have been shown to positively influence mental health outcomes such as affective well-being and perceived stress [21]. The Finnish Institute of Occupational Health (FIOH), which operates under the Ministry of Social Affairs and Health, published their recommendations for shift work [16] in May 2019. They presented twenty-two risk and stress factors grouped under five headings: length of the workday, timing of the working hours, recovery and rest times, reconciling work and family life and possibilities to influence personal working times. The institute introduced traffic lights for these factors to indicate the degree of overload. The recommendations of FIOH are based on the ten-year longitudinal study between 2008 and 2018 (see e.g. [22] and [23]). The study followed the effects and consequences of shift work on 13 000 hospital workers. Based on the extensive cooperation, the institute has daily working time statistics of more than 150 000 employees working in healthcare and social services sector. Table 2 shows fourteen of the twenty-two risk and stress factors presented by FIOH. The table shows the FIOH’s recommendations for shift work when the scheduling timeframe is three weeks. Based on the current scientific data on health and recovery issues [16], the three most important factors to consider are the number of night shifts, the rest time between two shifts and the longest work period between two days-off. The importance of these factors is denoted by one. The second most important factors (denoted by two) are long enough days-off, the number of free weekends, the number of night shifts in a row and the length of the shifts. Furthermore, it is important that an employee has an influence on her/his personal working times, i.e. she/he can make shift and days-off requests. The rest of the issues to consider are the number of working days in a row, the number of evening shifts in a row, the number of single working days, the longest rest period between two shifts and the rest time after a night shift.

278

K. Nurmi et al.

Table 2. Fourteen stress and risk factors and the recommendations for shift work published by the Finnish institute of occupational health for a planning horizon of three weeks.

3 Computational Intelligence Framework In this section, we describe the computational intelligence framework used for the staff rostering optimization. We give somewhat detailed justification of why we can safely rely on the comparison runs and the practical findings given later in Sect. 5. A definition of the word metaheuristic is still somewhat disputed. We adopt here a clear definition of Sörensen and Glover [24]: A metaheuristic is 1) a high-level problem-independent algorithmic 2) framework that provides 3) a set of guidelines or strategies to develop heuristic optimization algorithms. The definition implies that a metaheuristic is not an algorithm, i.e. it is not a sequence of actions that needs to be followed. Instead, a metaheuristic offers several tools that are problem independent and that could be used as the skeleton for the development of a heuristic. The following five metaheuristics have indisputably introduced true novelties to the repertoire of optimization methods (see e.g. [25]): MH1. Simulated annealing (1983), allows random moves that deteriorated the solution with a certain probability that decreases both with the size of the deterioration and over time as the search progresses [26]. MH2. Tabu search (1986), accepts deteriorating moves if no improving move is available, but prohibits the search from coming back to previously-visited solutions [27].

Staff Rostering Optimization

279

MH3. Genetic algorithm (1989), equates solutions to individuals in a population competing for survival, selects individuals from the population favoring better objective function values, and combines solutions into new ones by processes called crossover and mutation [28]. MH4. Ant colony optimization (1992), uses a set of agents (called ants) that heuristically construct solutions in parallel exchanging information on the quality of their solutions ensuring that elements that often appear in good solutions obtain a higher probability of being selected for inclusion in future solutions [29]. MH5. Variable neighborhood search (1997), uses different local search operators to work on a single solution, together with a perturbation operator once all neighborhoods have reached a local optimum [30]. We believe that also the following two metaheuristics can be added to the list. They have no doubt introduced new heuristic ideas. Furthermore, their performances have been scientifically and experimentally justified. MH6. Ruin and recreate method (1981, 2000), which obtains better solutions by a reconstruction (destruction) of an existing solution and a following improvement (rebuilding) procedure [31, 32]. MH7. Ejection chain method (1992), where a chain is initiated by selecting a set of elements to undergo a change of state, and the result of this change leads to identifying a collection of other sets where elements must be “ejected from” their current states, and so on [33]. By combining and carefully tuning the most efficient operators of these seven metaheuristics, an experienced heuristic designer can solve real-life optimization problems efficiently. Even though some metaheuristic can be more powerful than another for a given problem type, it is not guaranteed to work better for a different problem type. The No Free Lunch Theorem [34] implies that there cannot exist a superior optimization method. The PEAST metaheuristic [35] combines features from six metaheuristics described earlier. There is evidence that PEAST can successfully solve different problem domains. It has been used to schedule sports leagues (see e.g. [36] and [37]) and staff rosters [38] and to generate shifts [39, 40] and school timetables [41]. Furthermore, it has been used to solve academic problems, such as balanced incomplete block design [42], single round robin tournaments with balanced home-away assignments and pre-assignments [42], days-off scheduling [43] and constraint minimum break problems [44]. The pseudo-code of the PEAST metaheuristic is given in Fig. 2. The metaheuristic components of PEAST are the following: − Population-based (MH3) − Ejection chain search (MH5, MH7) − Simulated Annealing refinement (MH1) − Shuffling operators (MH5, MH6) − Tabu search (MH2).

280

K. Nurmi et al.

Each of the five metaheuristic components in the PEAST name is crucial to produce good-quality solutions. This was verified for three different problem domains in [35]. Recently, the results in solving the General Task-based Shift Generation Problem [39] showed that when any one of the components was removed, the results were clearly worse. The same held even if PEAST was given twice as much time to run without one of the components.

Fig. 2. The pseudo-code of the PEAST metaheuristic.

4 Recommendations vs. Practice In this section, we show how the ideal recommendations given in Sect. 2 and employer’s requirements compete with each other. We use the PEAST metaheuristic (see Sect. 3) to show the difference between the theoretical expectations and the every-day reality. We can solidly rely on the results and findings generated by the PEAST metaheuristic. There is evidence that the metaheuristic generates high-quality staff rostering solutions, see e.g. [12, 45–47]. Table 3 shows four different settings for the fourteen risk and stress factors shown earlier in Table 2. The values in the first column correspond to the safe (green) recommendations of the Finnish Institute of Occupational Health (FIOH). The second column shows a setting where four safe values are relaxed to increased overload (yellow) values and one to heavy overload (red) value. The third and fourth columns show settings where six and eight values are relaxed to increased overload and one to heavy overload. Next, we analyze the effects and consequences of these four different settings. The study was carried out in close cooperation with the managers of the staff rostering software CGI Titania®. The software is widely used by the public sector. As test instances, we use two real-world instances from the Turku University Hospital in Finland: the operating theater unit (OTU) and the intensive-care unit (ICU). The hospital offers specialized medical care services for almost 500 000 residents of the Southwest Finland. Furthermore, patients who require demanding specialized health care are admitted to

Staff Rostering Optimization

281

Table 3. Four different settings for the fourteen risk and stress factors.

the Turku University Hospital from all around Finland. More than 200 000 people use the services of the hospital each year. Table 4 shows the basic characteristics of the test instance. It should be noted that the instances are not exactly the same as those rostered by the shift schedulers. The instances are so hard to solve manually that the schedulers had to considerably relax the instances. For example, they were not able to schedule all the shifts, and they had to relax the competence demands of several shifts. Even though we can rely on the optimization results generated by the PEAST metaheuristic, we should realize the limitation of the study. This first study of the ideal recommendations and the real-world computing challenges deals with only two instances. However, based on our ten-year business experience, we can safely state that the two instances are fair and complex representatives of the real-world staff rostering instances. The optimization task is to look for the best possible roster that considers the legal and human aspects of shift work. The roster should be acceptable to hospital managers and to nurses. Both of them are looking for rosters that are transparent, ergonomic and fair, and efficiently use the available working time. The optimization task consists of four components: BASIC: The basic requirements given in Table 1 must be respected. WORK: Each nurse’s total working time should be within ±4 h compared to her/his target time (the second basic requirement in Table 1).

282

K. Nurmi et al.

Table 4. Basic characteristics of the real-world instances from the Turku university hospital in Finland: Operating Theater Unit (OTU) and the Intensive-Care Unit (ICU). OTU

ICU

Length of the planning period (number of days)

21

21

Number of nurses

88

155

Number of different competences used in rostering

26

30

Number of different shift types

52

73

Competence% of night shifts(*)

43%

42%

Competence% of all other shifts

74%

55%

Total number of shifts

1098

1840

Total number of minutes in shifts(1)

540 900

922 242

Total number of employees’ target minutes(2)

536 977

930 886

The difference between (2) and (1) per employee (mins)(**)

−45

194

Total number of employees’ lower limit minutes(3)

516 194

892 156

The difference between (3) and (1) per employee (mins)

−281

−56

Cover% of nurses’ requests for days-off and shifts(***)

13%

11%

(*) a randomly chosen nurse has competence to carry out a randomly chosen night shift with

probability 0.42 (**) each nurse has on the average 194 minutes shortage on working time compared to her target

time (***) the nurses have made requests towards 11% of all the shifts and days-off in the planning

horizon

RISK: The thirteen risk and stress factors should be considered as given in Table 3 for each of the four settings (note that 1B is always valid due to the shift structure). REQUESTS: The nurses’ requests should be fulfilled as well as possible (5A in Table 3). For each of the four settings, we use the same computational intelligence configuration, i.e. the PEAST metaheuristic uses the same configuration and parameter values for each test run. The most important parameter values (see [35]) for the main components are the following: − the population size is 100 − the maximum length of the ejection chain search is 10 − the starting temperature for the simulated annealing refinement is 0.95 − the shuffling interval is 5000 − the length of the tabu list is 10, which equals the length of the ejection chain search.

Staff Rostering Optimization

283

The optimization task in terms of the PEAST setup is given in Table 5. The hard values (targets) indicate that the solution may not include any violations for those risk and stress factors. The soft values (targets) indicate that violations may exist, but they should be minimized. The soft targets are assigned fixed weights 1, 2 and 3 (most important) according to their importance. PEAST uses a traditional penalty method, which assigns positive weights to the hard and soft targets and sums the violation scores to get a single value to be optimized (see [35]). Table 5. The Optimization Task in Terms of PEAST Setup.

5 Academic and Practical Findings In this section, we show how the ideal recommendations, employer’s requirements and employees’ requests compete with each other. We present the most important academic and practical findings that we believe will deepen the understanding of the complexity of staff rostering and the challenges the shift schedulers encounter. Tables 6A and 6B show the results of the PEAST runs. The solution shown for each setting is the median of eleven runs. The computation time for each run was five hours. All test runs were carried out using an Acer Laptop on an Intel Core i7–8550 at 1.8 GHz with 8 GB RAM operating Windows 10. The first glance of the results clearly shows that the solutions significantly differ from each other. The reason is the multifactorial

284

K. Nurmi et al.

and interdependent relations of the optimization targets, i.e. the targets compete with each other. A small change in one target may lead to inconsistency in another target. On the other hand, a small relaxation in one target may lead to substantial improvement in another target. Table 6A. The Results of Rostering the Intensive-Care Unit Instance using Four Different Risk and Stress Factor Settings. The Solution Shown for Each Setting is the Median of Eleven PEAST Runs.

The main observation is that there is no sense to roster the nurses using the safe values for all the optimization targets. Comparing the results between the FIOH1 and FIOH2 settings indicates that the five relaxations are not sufficient to produce feasible rosters. Even with the most relaxed values in the PRACT3 setting, the rostering task is extremely challenging for both units.

Staff Rostering Optimization

285

Table 6B. The Results of Rostering the Operating Theater Unit Instance using Four Different Risk and Stress Factor Settings. The Solution Shown for Each Setting is the Median of Eleven PEAST Runs.

The results indicate that the total working time for several nurses sinks below the lower limit (minus 4 h) when using the FIOH1, FIOH2 and PRACT1 settings. It is extremely important to understand here, that for PEAST, even more important than to minimize the hard violations is to guarantee the first basic requirement in Table 1 i.e. that the workforce requirements for every single shift is fulfilled. The most intractable requirement is to fulfill the night shift targets. Three other challenging requirements are clearly: − Single working days are not allowed − At least two free weekends must be guaranteed − A maximum of five consecutive working days are allowed.

286

K. Nurmi et al.

For example, in the ICU instance with the FIOH2 settings, the prize for having two free weekends for 26 nurses, causes 57 nurses to have no free weekends. To further analyze the effects and consequences of these three requirements, we carried our new runs using the PRACT3 setting as a basis, and changing the target of each of these three requirements to safe value. Table 7 shows the original PRACT3 setting and the three modifications to safe values. The PEAST setup was the same as given in Table 5. The three safe values were run as a hard optimization target, one by one. Each of the new settings was once again run eleven times. Table 7. The Original PRACT3 Setting and the Three Modifications to Safe Values.

Table 8 shows the results of the test runs for the ICU instance, which is the more challenging of the two test instances. The immediate observation from all the three modifications is that none of the three safe values should be used as a hard optimization target. Each of them significantly increases the number of night shifts and reduces the probability to fulfill the nurses’ shift and days-off requests. Furthermore, the results indicate that the total working time for several nurses sinks below the lower limit for all the three modifications.

Staff Rostering Optimization

287

Table 8. The Results of Rostering the ICU Instance using the Original PRACT3 Setting and the Three Modifications to Safe Values. The Solution Shown for Each Setting is the Median of Eleven PEAST Runs.

Table 9 summarizes the differences between the ideal recommendations published by the Finnish Institute of Occupational Health (FIOH) and our practical recommendations. The ideal recommendations are well justified. In the real-world staff rostering, we have to find a so-called practical optimum by relaxing some of the ideal recommendation values. Our practical recommendations are such that a sophisticated metaheuristic, such as PEAST, can generate high-quality staff rosters so that the computation time is still acceptable considering the release time of the rosters. Table 8 shows the results of the test runs for the ICU instance, which is the more challenging of the two test instances. The immediate observation from all the three modifications is that none of the three safe values should be used as a hard optimization target. Each of them significantly increases the number of night shifts and reduces the probability to fulfill the nurses’ shift and days-off requests. Furthermore, the results indicate that the total working time for several nurses sinks below the lower limit for all the three modifications.

288

K. Nurmi et al.

Table 9. The ideal recommendations for shift work published by the Finnish Institute of Occupational Health (FIOH) compared to our practical recommendations.

6 Conclusions and Future Work The Finnish Institute of Occupational Health have published their recommendations for shift work based on twenty-two risk and stress factors. The institute introduced traffic lights for these factors to indicate the degree of overload. The recommended values for these individual factors are well justified. However, we showed that problems arise when all these recommendations should be satisfied together in real-world staff rostering. We used the PEAST metaheuristic, a computational intelligence framework, to show that our recommendations are well justified. We believe that the presented academic and practical findings will give new ideas to the HR managers and the shift schedulers using manual, autonomous, automated and/or optimization-based planning tools. We hope that we were able to bridge the gap between theory and practice. Our business experience has shown that some of the employees’ requests can be such that they are contrary to some government regulations, collective labor agreements or personal work contracts, or they are violating the green and yellow targets of the ideal recommendations. The personal life conditions differ and therefore the temporary working time preferences as well. In addition, some of us are morning persons and some of us night owls by nature. As a result, each individual human being experiences his/her working life conditions differently. However, the ideal recommendations by FIOH should act as a solid guidance for general staff rostering rules and practices. The recommendations are well justified and they are based on the numerous reliable studies. Therefore,

Staff Rostering Optimization

289

notable differences to recommendations should be deployed only when both employer and employees accept them. Our near future direction is to consider the findings of the extensive 5-year project “Working hours, health, well-being and participation in working life (WOW) − Creating new working time models and solutions for Nordic countries” [48]. In the project, they hypothesize that chronic diseases, occupational and traffic accidents, and sickness absences can be reduced by implementing good working time models and individual coping mechanisms that support recovery and a better work-life balance.

References 1. L. Di Gaspero, J. Gärtner, N. Musliu, A. Schaerf, W. Schafhauser and W. Slany, “Automated Shift Design and Break Scheduling”, In: Uyar A., Ozcan E., Urquhart N. (eds) Automated Scheduling and Planning. Studies in Computational Intelligence 505, Springer, Berlin, Heidelberg (2013) 2. Nurmi, K., Kyngäs, N., Kyngäs, J.: Workforce optimization: the general task-based shift generation problem. IAENG Int, J. Appl. Math. 49(4), 393–400 (2019) 3. Van den Bergh, J., Belien, J., De Bruecker, P., Demeulemeester, E., De Boeck, L.: Personnel scheduling: a literature review. Eur. J. Oper. Res. 226(3), 367–385 (2013) 4. Garey, M.R., Johnson, D.S.: Computers and Intractability: A Guide to the Theory of NPCompleteness. W. H. Freeman and co., New York (1979) 5. Bartholdi, J.J.: A guaranteed-accuracy round-off algorithm for cyclic scheduling and set covering. Oper. Res. 29, 501–510 (1981) 6. Ernst, A.T., Jiang, H., Krishnamoorthy, M., Sier, D.: Staff scheduling and rostering: a review of applications, methods and models. Eur. J. Oper. Res. 153(1), 3–27 (2004) 7. Bard, J., Purnomo, H.: Hospital-wide reactive scheduling of nurses with preference considerations. IIE Trans. 37(7), 589–608 (2005) 8. Burke, E., De Causmaecker, P., Petrovic, S., Vanden Berghe, G.: Metaheuristics for handling time interval coverage constraints in nurse scheduling. Appl. Artif. Intell. 20, 743–766 (2006) 9. Bilgin, B., De Causmaecker, P., Rossie, B., Vanden Berghe, G.: Local search neighbourhoods to deal with a novel nurse rostering model. In: Proceedings of the 7th International Conference on the Practice and Theory of Automated Timetabling, Montréal, Canada (2008) 10. Beddoe, G.R., Petrovic, S., Li, J.: A hybrid metaheuristic case-based reasoning system for nurse rostering. J. Sched. 12, 99–119 (2009) 11. Burke, E.K., Curtois, T.: New approaches to nurse rostering benchmark instances. Eur. J. Oper. Res. 237, 71–81 (2014) 12. Nurmi, K., Kyngäs, J., Kyngäs, N.: The PEAST algorithm - the key to optimizing workforce management and professional sports league schedules. Int. J. Process Manage. Benchmarking 4(4), 406–423 (2014) 13. Jin, H., Post, G., van der Veen, E.: ORTEC’s contribution to the second international nurse rostering competition. In: Proceedings of the 11th International Conference on the Practice and Theory of Automated Timetabling, pp. 499–501 (2016) 14. Kingston, J.H.: KHE18: a solver for nurse rostering. In: Proceedings of the of the 12th International Conference on Practice and Theory of Automated Timetabling, pp. 113–127 (2018) 15. Gärtner, J., Bohle, P., Arlinghaus, A., Schafhauser, W., Krennwallner, T., Widl, M.: Scheduling matters - some potential requirements for future rostering competitions from a practitioner’s view. In: Proceedings of the 12th International Conference on Practice and Theory of Automated Timetabling, pp. 33–42 (2018)

290

K. Nurmi et al.

16. The Finnish Institute of Occupational Health, “Recommendations for shift work. https:// www.ttl.fi/tyontekija/tyoaika/tyoaikojen-kuormittavuuden-arviointi/tyoaikojenkuormittavu uden-arviointi-jaksotyossa/. Accessed 23 Oct 2020. (in Finnish) 17. Arendt, J.: Shift work: coping with the biological clock. Occup. Med. 60(1), 10–20 (2010) 18. Hinnenberg, S., Zegger, C., Nachreiner, F., Horn, D.: The utility of time - revisited after 25 years. Shiftwork Int. Newsletter 24(2) (2009) 19. Wirtz, A., Giebel, O., Schomann, C., Nachreiner, F.: The interference of flexible working times with the utility of time: a predictor of social impairment? Chronobiol. Int. 25, 249–261 (2008) 20. Vedaa, Ø., et al.: Short rest between shifts (quick returns) and night work is associated with work-related accidents. Int. Arch. Occup. Environ. Health 92(6), 829–835 (2019). https://doi. org/10.1007/s00420-019-01421-8 21. Nijp, H.H., Beckers, D.G., Geurts, S.A.: Systematic review on the association between employee worktime control and work-non-work balance, health and well-being, and jobrelated outcomes. Scand. J. Work Environ. Health 38, 299–313 (2012) 22. Karhula, K., et al.: Are changes in objective working hour characteristics associated with changes in work-life conflict among hospital employees working shifts? A 7-year follow-up. Occup. Environ. Med. 75(6), 407–411 (2018) 23. Karhula, K., Hakola, T., Koskinen, A., Ojajärvi, A., Kivimäki, M., Härmä, M.: Permanent night workers’ sleep and psychosocial factors in hospital work. A comparison to day and shift work. Chronobiol. Int. 35(6), 785–794 (2018) 24. Sörensen, K., Glover, F.: Metaheuristics. In: Gass, S.I., Fu, M. (eds.) Encyclopedia of Operations Research and Management Science, vol. 62, pp. 960–970 (2013) 25. Sörensen, K., Sevaux, M., Glover, F.: A History of Metaheuristics. In: Martí, R., Pardalos, P.M., Resende, M.G.C. (eds.) Handbook of Heuristics, pp. 791–808. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-07124-4_4 26. Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220, 671–680 (1983) 27. Glover, F.: Future paths for integer programming and links to artificial intelligence. Comput. Oper. Res. 13(5), 533–549 (1986) 28. Goldberg, D.: Genetic Algorithms in Search, Optimization and Machine Learning. Addison Wesley, USA (1989) 29. Colorni, A., Dorigo, M., Maniezzo, V.: Distributed optimization by ant colonies. In: Proceedings of the First European Conference on Artificial Life, pp. 134–142 (1992) 30. Mladenovic, N., Hansen, P.: Variable neighborhood search. Comput. Oper. Res. 24(11), 1097– 1100 (1997) 31. Dees, W.A., Smith II, R.: Performance of interconnection rip-up and reroute strategies. In: 18th Design Automation Conference, pp. 382–390 (1981) 32. Schrimpf, G., Schneider, K., Stamm-Wilbrandt, H., Dueck, W.: Record breaking optimization results using the ruin and recreate principle. J. Comput. Phys. 159, 139–171 (2000) 33. Glover, F.: New ejection chain and alternating path methods for traveling salesman problems. Computer Science and Operations Research: New Developments in Their Interfaces, pp. 449– 509 (1992) 34. Wolpert, D.H., Macready, W.G.: no free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67–82 (1997) 35. Kyngäs, N., Nurmi, K., Kyngäs, J.: Crucial components of the PEAST algorithm in solving real-world scheduling problems. J. Lect. Not. Software Eng. 1(3), 230–236 (2013) 36. Nurmi, K., Kyngäs, J., Järvelä, A.I.: Ten-year evolution and the experiments in scheduling a major ice hockey league. In: Hak, D. (ed.) An in Depth Guide to Sports, Nova Science Publishers, pp 169–207 (2018)

Staff Rostering Optimization

291

37. Kyngäs, J., Nurmi, K., Kyngäs, N., Lilley, G., Salter, T., Goossens, D.: Scheduling the Australian football league. J. Oper. Res. Soc. 68, 973–982 (2017) 38. Kyngäs, N., Nurmi, K., Kyngäs, J.: Workforce scheduling using the PEAST algorithm. In: Ao, S.-I. (ed.) IAENG Transactions on Engineering Technologies. Lecture Notes in Electrical Engineering, vol. 275, pp. 359–372. Springer, New York (2014) 39. Kyngäs, N., Nurmi, K., Goossens, D.: The general task-based shift generation problem: formulation and benchmarks. In: Proceedings of the 9th Multidisciplinary Int. Scheduling Conference: Theory and Applications (MISTA), Ningbo, China (2019) 40. Kyngäs, N., Nurmi, K., Kyngäs, J.: Solving the person-based multitask shift generation problem with breaks. In: Proceedings of the 5th International Conference on Modeling, Simulation and Applied Optimization, Hammamet, Tunis, pp. 1–8 (2013) 41. Nurmi, K., Kyngäs, J.: A conversion scheme for turning a curriculum-based timetabling problem into a school timetabling problem. In: Proceedings of the 7th Conference on the Practice and Theory of Automated Timetabling (PATAT), Montreal, Canada (2008) 42. Nurmi, K., Goossens, D., Kyngäs, J.: Scheduling a triple round robin tournament with minitournaments for the Finnish national youth ice hockey league. J. Oper. Res. Soc. 65(11), 1770–1779 (2014) 43. Nurmi, K., Kyngäs, J.: Days-off scheduling for a bus transportation company. Int. J. Innovative Comput. Appl. 3(1), 42–49 (2011) 44. Nurmi, K., et al.: A framework for scheduling professional sports leagues. In: Ao, S.-I. (ed.) IAENG Transactions on Engineering Technologies, vol. 5, pp. 14–28. Springer, Heidelberg (2010) 45. Nurmi, K., Kyngäs, J., Kyngäs, N.: Synthesis of employer and employee satisfaction - case nurse rostering in a Finnish hospital. J. Adv. Inf. Technol. 7(2), 97–104 (2016) 46. Nurmi, K., Kyngäs, J., Kyngäs, N.: The core staff rostering problem. In: Ao, C., Katagiri and Xu (eds.) IAENG Transactions on Engineering Sciences - Special Issue for the International Association of Engineers Conferences, World Scientific (2016) 47. Kyngäs, N., Nurmi, K., Ásgeirsson, E.I., Kyngäs, J.: Using the PEAST algorithm to roster nurses in an intensive-care unit in a Finnish hospital. In: Proceedings of the 9th Conference on the Practice and Theory of Automated Timetabling, pp. 83–93 (2012) 48. Finnish Institute of Occupational Health: Working hours, health, well-being and participation in working life – WOW (2015–2020). https://www.ttl.fi/en/research-and-development-pro jects/wow/. Accessed 20 Feb 2020

Resolution of the Frobenius Problem with an Adiabatic Quantum Computer J. Ossorio-Castillo1(B) and Jos´e M. Tornero2 1

2

´ Departamento de Algebra, Universidad de Sevilla, 41012 Sevilla, Spain [email protected] ´ Departamento de Algebra – IMUS, Universidad de Sevilla, 41012 Sevilla, Spain [email protected]

Abstract. The (Diophantine) Frobenius problem is a well-known NPhard problem (also called the stamp problem or the chicken nugget problem) whose origins lie in the realm of combinatorial number theory. In this paper we present an algorithm which solves it, via a translation into a QUBO problem of the so-called Ap´ery set of a numerical semigroup. This algorithm was specifically designed to run in an adiabatic quantum computer (a D-Wave 2X machine), and the performance problems for this precise setting are also discussed. Keywords: Quantum computation · Adiabatic quantum computation · Numerical semigroups · Frobenius problem

1

Numerical Semigroups and the Frobenius Problem

The study of numerical semigroups has its origins at the end of the 19th Century, when James Joseph Sylvester (1814–1897) and Ferdinand Georg Frobenius (1849–1917) were both interested in what is now known as the Frobenius problem, which we proceed to enunciate. Definition 1. Let a1 , a2 , . . . , an ∈ Z≥0 with gcd(a1 , a2 , . . . , an ) = 1, the Frobenius problem, or FP, is the problem of finding the largest positive integer that cannot be expressed as an integer conical combination of these numbers, i.e., as a sum n 

λi ai with λi ∈ Z≥0 .

i=1

This problem, so easy to state, can be extremely complicated to solve in the majority of cases, as will be seen later. It can be found in a wide variety of contexts, being the most famous the problem of finding the largest amount of money which cannot be obtained with a certain set of coins: if, for example, we have an unlimited amount of coins of 2 and 5 units, we can represent any quantity except 1 and 3. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 292–310, 2022. https://doi.org/10.1007/978-3-030-80119-9_16

Resolution of the Frobenius Problem with an Adiabatic Quantum Computer

293

In order to understand the relationship between this problem and numerical semigroups, we shall first define the latter. Definition 2. A semigroup is a pair (S, +), where S is a set and + is a binary operation + : S × S → S that is associative. Definition 3. A numerical semigroup S is a subset of the non-negative integers Z≥0 which is closed under addition, contains the identity element 0, and has a finite complement in Z≥0 . From now on, we shall denote numerical semigroups as S, taking for granted that they are commutative and that their associated operation is the addition. As it can be easily noted, a numerical semigroup is trivially a semigroup. In other words, a numerical semigroup is a semigroup that, additionally, is a monoid (i.e., it also has an identity element) and has finite complement in Z≥0 . In order to work with numerical semigroups, it will be necessary to characterize them somehow. For that let us set forth the following lemma. The proof of this result (and of any other result in this section, unless otherwise stated) can be found in [24]. Lemma 4. Let A = {a1 , ..., an } be a nonempty subset of Z≥0 . Then, S = A = a1 , ..., an  = {λ1 a1 + ... + λn an | λi ∈ Z≥0 } is a numerical semigroup if and only if gcd(a1 , ..., an ) = 1. The previous lemma tells us that, drawing from a finite set A ⊆ Z≥0 , it is possible to generate a semigroup S = A as long as the elements of A satisfy a certain condition. In this context, any set A such that S = A for a certain numerical semigroup S is called a system of generators of S. Even more, it can be proved that any numerical semigroup can be expressed that way, as will be shown next. Theorem 5. Every numerical semigroup S admits a unique minimal system of generators, which can be calculated as S ∗ \ (S ∗ + S ∗ ) with S ∗ = S \ {0}. Corollary 6. Let S be a numerical semigroup generated by A = {a1 , . . . , an } with 0 = a1 < a2 < ... < an . Then, A is a minimal system of generators of S if / a1 , a2 , . . . , ai , for all i ∈ {1, . . . , n − 1}. and only if ai+1 ∈ It will be stated later in this section that the minimal system of generators of a numerical semigroup is in fact finite. Hereinafter, if we say that S = A with A = {a1 , a2 , . . . , an } is a numerical semigroup, then we shall assume without loss of generality that a1 < a2 < · · · < an , gcd(a1 , a2 , . . . , an ) = 1, and that A is the minimal system of generators of S. Some examples of numerical semigroups, which will be used for the rest of the section, are: 3, 7 = {0, 3, 6, 7, 9, 10, 12, →}, 4, 9 = {0, 4, 8, 9, 12, 13, 16, 17, 18, 20, 21, 22, 24, →}, 5, 8, 11 = {0, 5, 8, 10, 11, 13, 15, 16, 18, →}, 5, 7, 9 = {0, 5, 7, 9, 10, 12, 14, →},

294

J. Ossorio-Castillo and J. M. Tornero

where → means that all integers thenceforth are also included in the numerical semigroup. Having thus defined and characterized what numerical semigroups are, we proceed to describe some of their combinatorial invariants. Definition 7. Let S = a1 , a2 , . . . , an  be a numerical semigroup, then m(S) = a1 and e(S) = n are called respectively the multiplicity of S and the embedding dimension of S. Lemma 8. Let S be a numerical semigroup, then m(S) = min S ∗ Definition 9. The set of gaps of a numerical semigroup S is defined as G(S) = Z≥0 \S. Its cardinal, g(S) = |G(S)|, is called the genus of S; and its maximum, f (S) = max G(S), is called the Frobenius number of S. In other words, as by definition any numerical semigroup S has a finite complement in Z≥0 , we can define the maximum of such complement as f (S), known as the Frobenius number. In fact, the Frobenius problem described at the beginning of this chapter in Definition 1 can be enunciated as the problem of finding f (S) for a certain numerical semigroup S. We shall expand the concepts related to the difficulties that surround the calculation of the Frobenius number later on. Table 1 shows the combinatorial invariants associated for the previously given examples of semigroups. Table 1. Combinatorial invariants for some examples of semigroups S = A m(S) e(S) G(S)

g(S) f (S)

3, 7

3

2

{1, 2, 4, 5, 8, 11}

6

11

4, 9

4

2

{1, 2, 3, 5, 6, 7, 10, 11, 14, 15, 19, 23} 12

23

5, 8, 11 5

3

{1, 2, 3, 4, 6, 7, 9, 12, 14, 17}

10

17

5, 7, 9

3

{1, 2, 3, 4, 6, 8, 11, 13}

8

13

5

Roger Ap´ery (1916–1994), better known for proving in 1979 the irrationality of ζ(3) [2], also laid the background in the context of the resolution of singularities of curves [1] for an important set associated to a numerical semigroup S and one of its elements.

Resolution of the Frobenius Problem with an Adiabatic Quantum Computer

295

Definition 10. The Ap´ery set of a numerical semigroup S with respect to a certain s ∈ S ∗ is defined as Ap(S, s) = {x ∈ S | x − s ∈ / S}. A possible characterization of the elements of the Ap´ery set one by one, which will come to special use in the third section, is given by the following lemma. Lemma 11. Let S be a numerical semigroup and let s ∈ S ∗ . Then, Ap(S, s) = {ω0 , ω1 , ..., ωs−1 } where ωi is the least element of S congruent with i modulo s, for all i ∈ {0, ..., s − 1}. Consequently, |Ap(S, s)| = s. By means of an example on how to calculate the Ap´ery set of a semigroup with respect to a certain element, let S = 5, 8, 11 = {0, 5, 8, 10, 11, 13, 15, 16, 18, →}. Then Ap(S, 5) = {ω0 , . . . , ω4 }, where ω0 = min{x ∈ S | x ≡ 0 ω1 = min{x ∈ S | x ≡ 1 ω2 = min{x ∈ S | x ≡ 2 ω3 = min{x ∈ S | x ≡ 3 ω4 = min{x ∈ S | x ≡ 4

mod 5} = 0 mod 5} = 11 mod 5} = 22 mod 5} = 8 mod 5} = 19

We proceed to hint how the Ap´ery set proves some elementary (but nevertheless important) results of numerical semigroups. Proposition 12. The minimal system of generators of a numerical semigroup S is finite. Proof. Let s ∈ S ∗ . Then, it is easy to see that for every t ∈ S there exists a unique pair (u, v) ∈ Z≥0 × Ap(S, s) such that t = us + v. Thus, S is generated by A = Ap(S, s) ∪ {s} and, as A is finite, the unique minimal system of generators must be finite. Lemma 13. Let S be a numerical semigroup, then e(S) ≤ m(S). Proof. Let a = m(S) and let A = Ap(S, a) ∪ {a}. Thus, as S can be generated by A \ {0} and |A \ {0}| = a, we can conclude that e(S) ≤ m(S). The Ap´ery set is noteworthy in the context of the Frobenius problem as there exists a relationship between its members (regardless of the element s ∈ S ∗ we choose) and the Frobenius number, which we proceed to enunciate.

296

J. Ossorio-Castillo and J. M. Tornero

Theorem 14 (A. Brauer – J. E. Shockley, 1962) [7]. Let S be a numerical semigroup and let s ∈ S ∗ . Then f (S) = max Ap(S, s) − s ⎛ ⎞  1 s−1 g(S) = ⎝ ω⎠ − s 2 ω∈Ap(S,s)

Now we exemplify the relationship between numerical semigroups and combinatorial optimization, as one of the most important problems in the latter branch of mathematics, known as the knapsack problem or rucksack problem, and more concretely one of its variants [21] (p. 374), can be seen as the problem of deciding if a given integer t belongs to a certain numerical semigroup S. Definition 15. The numerical semigroup membership problem, or NSMP, is the problem of determining if, given a certain integer t ∈ Z≥0 and a numerical semigroup S = a1 , ..., an , the integer t is contained in S. That is to say, if there exist non-negative integers λ1 , . . . , λn ∈ Z≥0 such that n 

λi ai = t.

i=1

The numerical semigroup membership problem is in NP-complete, as shown in [21]. This fact was used by Jorge Ram´ırez-Alfons´ın in 1996 [22] to finally prove that the Frobenius problem is in NP-hard (under Turing reductions). For that, he defined a polynomial algorithm ΛNSMP for solving the NSMP which uses as a subroutine an unknown algorithm ΛFP that solves the Frobenius problem. Thus, he proved that the NSMP can be Turing reduced to the FP. As the NSMP is in NP-complete, he concluded that the FP is in NP-hard.

2

The Ising Spin Problem in Adiabatic Quantum Computing

Our tackle of the Frobenius problem uses adiabatic quantum computing, more specifically that related to the Ising spin problem. This problem has a story where NP-hard problems are no strangers. A form of adiabatic quantum computation [19] is quantum annealing, where a known initial configuration of a quantum system evolves towards the ground state of a Hamiltonian that encodes the solution of an NP-hard optimization problem. The Canadian company D-Wave Systems announced in 2011 the first commercially available quantum annealer, composed of arrays of eight superconducting flux quantum bits with programmable spin–spin couplings, and published their results [18]. Subsequently in 2013, S. Boixo et al. [4] published their experimental results on the 108-qubit D-Wave One device. Their last chip (by the date of this work), released in 2017 and called D-Wave 2000Q, has 2,048

Resolution of the Frobenius Problem with an Adiabatic Quantum Computer

297

qubits in a Chimera graph architecture [11], and can be seen as a computer that solves the Ising spin problem, a particular type of quantum annealing which we proceed to describe. The Ising spin model, originally formulated by Wilhelm Lenz and first solved by his student, Ernst Ising [17], consists of a model of ferromagnetism in statistical mechanics in which we have to find the ground state of a system of n interacting spins. If we represent the spins of these particles as binary variables si ∈ {−1, 1} with i ∈ {1, . . . , n}, then the Ising Spin problem can be expressed as an integer optimization problem whose objective is to find the spin configuration of the system that minimizes the function H(s1 , . . . , sn ) =

n 

hi si +

i=1

n−1 

n 

Jij si sj ,

i=1 j=i+1

where hi ∈ R is the energy bias acting on particle i (i.e. the external forces applied to each of the individual particles) and Jij ∈ R is the coupling energy between the spins i and j (i.e. the interaction forces between adjacent spins). This problem was proved to be in NP-hard by Francisco Barahona [3], and can be effectively solved with the hardware implemented by D-Wave, whose chip permits to program independently the values of hi and Jij [4, 18]. It will be shown in the next section how to embed a certain optimization problem inside the D-Wave machine.

3

Resolution of the Frobenius Problem with an Adiabatic Quantum Computer

The next algorithm shows the possibilities of calculating the Ap´ery set and the Frobenius number of a numerical semigroup with an actual adiabatic quantum computer. As described in the previous section, current D-Wave quantum annealers solve a certain kind of mathematical optimization problems known as Ising spin problems, namely H(s1 , . . . , sn ) =

n  i=1

hi si +

n−1 

n 

Jij si sj

i=1 j=i+1

with si ∈ {−1, 1}, where the objective is to find the minimum of H(s1 , . . . , sn ). On the other hand, we recall that the Ap´ery set with respect to s ∈ S \ {0}, where S = a1 , . . . , an  is a numerical semigroup, is defined as Ap(S, s) = {x ∈ S | x − s ∈ / S}. The question is, how could we be able to translate the problem of computing Ap(S, s) to the Ising spin model solved by the D-Wave machine? We have already shown in Lemma 11 that Ap(S, s) = {ω0 , . . . , ωn−1 }, where ωi = min{x ∈ S : x ≡ i mod s}.

298

J. Ossorio-Castillo and J. M. Tornero

How about transforming the definition of ωi into the answer of a mathematical optimization problem? Definition 16 [21]. An integer linear program, or ILP, is defined in its canonical form as the optimization problem: min cT x subject to: Ax = b where x ∈ Zn≥0 , A ∈ Zn × Zm , b ∈ Zm and c ∈ Zn . Thus, it is straightforward to see that the calculation of each of the ωi can be redefined as the following ILP: min subject to:

n  j=1 n 

aj xj aj xj = i + sk,

j=1

with xj ∈ Z≥0 for all j = 1, ..., n and k ∈ Z≥0 . This mathematical optimization problem represents a way of calculating the Ap´ery set in its own right and, although integer linear programming is in NPhard and its recognition version (deciding whether Ax = b has a feasible solution or not regardless of its optimality) is in NP-complete [21], this computational complexity arises in the general case. In our context, it may prove to be an easier problem; however, details on this remain to be worked out. This approach for the calculation of the Ap´ery set first appeared in [15], although Greenberg’s work dealt with the direct calculation of the Frobenius number by means of Ap(S, a1 ), as will be discussed later. We have tested this algorithm using state-of-the-art optimization software for integer optimization. In our case, the optimization problem for ωi was modeled using AMPL [12, 13], an algebraic modeling language for solving mathematical optimization problems. AMPL has a clear advantage, as our problem is entirely parameterized and AMPL allows us to describe the generic problem in a .mod file while defining the actual values for the parameters in a separate .dat file. This way, we have just to change the parameters inside the .dat file in order to solve a new instance of the Ap´ery set. The .mod file we propose is shown in Fig. 1 (the names of the parameters and variables of the problem are maintained so that no further explanation is required). Let us suppose that we are interested in the numerical semigroup S = 11, 19, 23, and that we want to calculate the Ap´ery set of s = 30 (i.e., Ap(S, 30)). The calculation of, for example, ω5 ∈ Ap(S, 30), will have the associated .dat file depicted in Fig. 2. In order to compute ω5 , we also need a .run file that will load the model and the data of the problem, and which will also call the solver for solving the problem. In our case, the solver we have chosen is Gurobi [16] which, among other things, solves integer linear problems. The .run

Resolution of the Frobenius Problem with an Adiabatic Quantum Computer

299

file we have used is shown in Fig. 3, and the corresponding output we obtain can be seen in Fig. 4. This output tells us that ω5 = 65 and also that a representation of 65 with respect to the generators of the semigroup is 65 = 0 × (11) + 1 × (19) + 2 × (23). We can also write an alternative .run file that will directly calculate and display the whole content of the Ap´ery set for a certain s ∈ S. First, we have to drop the param i := 5; line from the .run file, as it will be changed in each iteration of the main loop, and modify the .run so that it will look like the one in Fig. 5. Thus, we obtain Ap(S, 30) = {0, 11, 19, 22, 23, 33, 34, 38, 42, 44, 45, 46, 55, 56, 57, 61, 65, 66, 67, 69, 77, 78, 80, 84, 88, 89, 92, 100, 103, 111} (see Fig. 6 for the actual output).

Fig. 1. File apery set member.mod

Fig. 2. File numerical semigroup.dat

This algorithm also provides a way for calculating the Frobenius number of a numerical semigroup. We recall that, for any numerical semigroup S and any integer s ∈ S \ {0}, then f (S) = max {Ap(S, s)} − s.

300

J. Ossorio-Castillo and J. M. Tornero

Thus, as the difficulty of obtaining f (S) this way increments with respect to the number s we choose (we have to solve s ILPs), the smartest way of proceeding is by solving it in the case where s = min(S \ {0}) (i.e., s = a1 , as done by [15]). The AMPL file that represents this approach is shown in Fig. 7. In our case, the output (Fig. 8) tells us that f (S) = 81. All these files can be found in the public GitHub repository [20]; however, a license for both AMPL and Gurobi is needed in order to run them and obtain the same results (or any result at all). What we have shown is a classical algorithm for obtaining the Ap´ery set and the Frobenius number in a general manner that depends on a black box that solves an ILP with global optimality. From now on we will explain the steps we have followed in order to solve this ILP with an adiabatic quantum computer and, specifically, with a D-Wave 2X machine (and also the obstacles we have encountered in that path). In order to transform this ILP problem into the Ising model solved by the D-Wave hardware, one step further involves changing its integer variables into a new set of binary variables with at most a polynomial cost. Definition 17 [21]. A binary linear program, or BLP, is defined in its canonical form as: min cT x subject to: Ax = b where x ∈ {0, 1}n , A ∈ Zn × Zm , b ∈ Zm and c ∈ Zn . Both problems are polynomially equivalent, as shown in [21] (Theorem 13.6), where an upper bound for the number of binary variables representing each integer variable from the original ILP problem is given. However, we can tight this number of binary variables by using our knowledge of the problem and one of the lower bounds of the Frobenius number given in [23] (ideally, we could use f (S)). In 1935, Issai Schur proved in a lecture in Berlin [6, 23] the following result:

Fig. 3. File apery set member.run

Resolution of the Frobenius Problem with an Adiabatic Quantum Computer

301

Fig. 4. Output for apery set member.run

Fig. 5. File apery set.run

Fig. 6. Output for apery set.run

Theorem 18 (I. Schur, 1935). Let S = a1 , ..., an  be a numerical semigroup. Then, f (S) ≤ (a1 − 1)(an − 1) − 1. Thus, if we define 

 (a1 − 1)(an − 1) + s − 1 tj = 1 + log2 aj

302

J. Ossorio-Castillo and J. M. Tornero

for every j ∈ {1, . . . , n}, and

  (a1 − 1)(an − 1) + s − i − 1 u = 1 + log2 s we have given an upper bound for the number of bits needed to describe every variable xj in the worst case (we can also run every known bound for the Frobenius number, and stick to the minimum of them, but for now let us just use Schur’s result). It follows that the transformation of our problem from ILP form to BLP delivers the minimization problem shown below: min

n 

aj

j=1

subject to:

n  j=1

tj 

2l xjl

l=0

aj

tj  l=0

2l xjl = i + s

u 

2m km

m=0

with xjl ∈ {0, 1} and km ∈ {0, 1} for all j, l and m contained in the summation indices. Let us come back to the definition of the Ising model. In it, we have binary variables which can have the values −1 or 1. On the other hand, in our BLP for calculating the elements of the Ap´ery set the variables can be in 0 or 1. It is usually more advantageous to redefine the Ising spin problem as a quadratic unconstrained binary optimization problem, or QUBO, which consists of the

Fig. 7. File frobenius number.run

Fig. 8. Output for frobenius number.run

Resolution of the Frobenius Problem with an Adiabatic Quantum Computer

303

minimization of the objective function Q(x1 , . . . , xn ) = c0 +

n 



ci xi +

i=1

qij xi xj ,

1≤i 0. In BLP form, we obtain the following QUBO problem:

min

n  j=1

aj

tj  l=0

⎛ 2l xjl + λ(ν) ⎝

n  j=1

aj

tj  l=0

2l xjl − i − s

u 

⎞2 2m km ⎠ ,

m=0

with its corresponding penalty updating formula:     tj n u     (ν+1) (ν) l m  λ =λ +α· aj 2 xjl − i − s 2 km   j=1 l=0  m=0 with α > 0.

304

J. Ossorio-Castillo and J. M. Tornero

Fig. 9. File apery set member lagr.mod

Now, we are going to test the performance of this new procedure for obtaining the Ap´ery set (and the Frobenius number) via a classical solver. For that, we have just to modify the original apery set member.mod file into a new one with an unconstrained ILP, with the constraint penalized in the objective function as previously explained. This new AMPL file can be found in Fig. 9, along with its corresponding .run file in Fig. 10. Additionally, we are going to keep track of the number of iterations of the algorithm (i.e., ν), and on the values of λ(ν) needed for the algorithm to stop. This way, if we start with λ(0) = 0, the output for this algorithm is found in Fig. 11. On the other hand, if we start with λ(0) = 100, the output is shown in Fig. 12. Foreseeably, the Ap´ery set we obtain for S and s = 30 is correct, but that should not be a surprise as both optimization problems are equivalent to the original one. What is of interest here is that a low starting point for λ implies a relatively large number of iterations, which translates in more calls to our integer linear programming solver than in the original constrained version. However, if we start with a large enough value of λ, we can obtain the same results without additional calls to the solver and just one iteration for each of the elements of the Ap´ery set. On the other hand, if we exceed in the value for λ, we would also need just one iteration per ωi , but the solver would need more time in order to find the global optimum. The latter behavior can be observed when taking into account the number of simplex iterations and branch-and-cut nodes that Gurobi needs for each of the problems. Nevertheless, this unconstrained reformulation of the problem is not meant to be solved by Gurobi or any other integer linear programming solver, but rather by an ideal adiabatic quantum computer, and more concretely by one of the few D-Wave machines that currently exist worldwide. Our unconstrained problem, in QUBO form, is shown below (please note that if x ∈ {0, 1}, then x = x2 ):

Resolution of the Frobenius Problem with an Adiabatic Quantum Computer

c0 +

u 

cm km +

m=0

tj n   j=1

+

cjl xjl

l=0

⎛ ⎝

1≤j 1) { int mid = (lowEnough + tooHigh) / 2; cobegin { mutateTops(A, lowEnough, mid); mutateTops(A, mid, tooHigh); } } else if (tooHigh > lowEnough) { if (A[lowEnough].size() > 0) { R e = A[lowEnough].pop(); mutate(e); A[lowEnough].push(e); } } }

 h´1 

 list ei · αi (A0 [i], nil)

i“l

mutateTops(A, l, h);  h´1   i i list e · α (A0 [i], nil)

(1)

i“l

In plain  this specification is as follows. The precon English, the meaning of h´1 i i dition, i“l list e · α (A0 [i], nil) , states that for each l ď i ă h, the initial value of the i-th element of array A is a nil-terminated (singly linked) list with abstract value ei · αi , and that each of these lists is separate in the heap (that is, they share no nodes). The postcondition states that the only thing that has changed about each element of A is the abstract value of the first node in the list (and, still that the lists are separate). The first challenge posed by this example with this specification is that it does not preclude aliasing between elements of the stacks in A. For example, if the top element of A[0] is an alias to the top element of A[1], the picture might look like Fig. 1. Note that it is still true that {list α (A[0], nil) ˚ list β (A[1], nil)}, so the precondition is satisfied. The problem with this scenario is that if the mutation performed by the mutate operation is not idempotent, the result will not be correct (in fact, if the top of one stack is an alias to an element of a

An Array Abstraction to Amortize Reasoning

351

stack that is not the top, even an idempotent mutation will cause problems). In a language with clean semantics, we can rely on the fact that separate variables act as separate objects to guarantee there is no dangerous aliasing.3

Fig. 1. An array with separate (in the heap) lists, but whose member stacks have elements that are aliases if x and y are references and x “ y.

The Shared Representation Problem. A much more subtle issue arises when instances of T use a shared representation. For example, the precondition as written would not be satisfied if the stack implementation were swapped out for one based on a shared cactus stack, as in Fig. 2, even though such an implementation could provide correct behavior. This lack of modularity demonstrates a need for abstraction in the specification of concurrent programs (e.g. an interference contract) to facilitate reusable code that can remain proven to be correct even when different underlying data structures are used.

Fig. 2. A cactus stack using a partially-shared realization. Stack S1 “ xF, B, Ay, S2 “ S3 “ xG, D, C, Ay, S4 “ xE, C, Ay, and S5 “ xHy. Note that it is not the case that list α(S1 , nil) ˚ list β(S2 , nil) for any α, β.

3

The Splittable Array Abstraction

The Splittable Array abstraction is a novel array abstraction that amortizes the cost of reasoning about parallel divide-and-conquer algorithms such as the 3

While a modified specification could be written in separation logic to handle this particular situation, it becomes extremely complex in the general case [9].

352

A. Weide et al.

one presented in Listing 2. The Splittable Array component provides the client with operations that divide the array at some split point into two sub-arrays with contiguous indices. By virtue of RESOLVE’s clean semantics the resulting sub-arrays may be reasoned about as totally independent objects. The abstract specification of this component is shown in Listing 3. Notation. The notation used in the remaining listings of this paper is based on the RESOLVE language [1,2]. A concept is the specification of a type, including a mathematical model and specifications for operations on that type in terms of that model. In the operation contracts, the keywords restores, updates, replaces, clears, and preserves are parameter modes, which summarize the effect of the operation on that parameter. The difference between restores and preserves is that a restores-mode parameter might have its value changed temporarily by the implementation of the operation while a preserves-mode parameter may not; both express the fact that the value of the parameter after the operation is the same as it was beforehand. In the ensures clause of the operation contracts, the notation # denotes the “old” value of a parameter—roughly equivalent to a zero-subscript in other specification languages. In the specifications hereafter, we use traditional mathematical notation for the clauses to improve readability; the language has text-based equivalents that would appear in real programs. Listing 3. Abstract specification for Splittable Array

concept Splittable Array Template(type Entry) var Ids: finite set of integer initialization ensures Ids “ H type family Splittable Array is modeled by ( Id: integer Lower Bound: integer, Upper Bound: integer, Contents: integer ´> Entry, Split Point: integer, Parts In Use: boolean ) exemplar A constraint A.Lower Bound ď A.Upper Bound ^ A.Lower Bound ď A.Split Point ď A.Upper Bound ^ A.Id P Ids initialization ensures A.Lower Bound “ 0 ^ A.Upper Bound “ 0 ^ A.Parts In Use ^ A.Id R #Ids end

An Array Abstraction to Amortize Reasoning

operation Set Bounds( restores LB: integer, restores UB: integer, updates A: Splittable Array) requires LB ď UB ^ A.Parts In Use ensures A.Lower Bound “ LB ^ A.Upper Bound “ UB ^ A.Parts In Use ^ A.Id R #Ids ^ #A.Id R Ids operation Set Split Point( restores i: integer, updates A: Splittable Array) requires A.Lower Bound ď i ^ i ď A.Upper Bound ^ A.Parts In Use ensures A.Split Point “ i ^ [[everything else about A stays the same]] operation Swap Entry At( restores i: integer, updates A: Splittable Array, updates E: Entry) requires A.Parts In Use ^ A.Lower Bound ď i ă A.Upper Bound ensures E “ #A.Contents(i) ^ A.Contents(i) “ #E ^ [[everything else about A stays the same]] operation Split( updates A: Splittable Array, replaces L: Splittable Array, replaces U: Splittable Array) requires A.Parts In Use ensures A.P arts In U se ^ L.Incl Lower Bound “ A.Incl Lower Bound ^ L.Excl Upper Bound “ A.Split Point ^ U .Lower Bound “ A.Split Point ^ U .Upper Bound “ A.Excl Upper Bound ^ L.Id “ A.Id ^ L.Contents “ A.Contents ^ U .Id “ A.Id ^ U .Contents “ A.Contents ^ L.Parts In Use ^ U .Parts In Use ^ [[everything else about A stays the same]] operation Combine( updates A: Splittable Array, clears L: Splittable Array,

353

354

A. Weide et al.

clears U: Splittable Array) requires A.Parts In Use ^ L.Parts In Use ^ U .Parts In Use ^ L.Incl Lower Bound “ A.Incl Lower Bound ^ L.Excl Upper Bound “ A.Split Point ^ U .Incl Lower Bound “ A.Split Point ^ U .Excl Upper Bound “ A.Excl Upper Bound ^ L.Id “ A.Id ^ U .Id “ A.Id ensures A.Parts In Use ^ @(i : Z)( (i ă A.Split Point) ñ (A.Contents(i) “ #L.Contents(i)) ^ (i ě A.Split Point) ñ (A.Contents(i) “ #U .Contents(i))) ^ [[everything else about A stays the same]] operation Lower Bound(preserves A: Splittable Array): integer ensures Lower Bound “ A.Lower Bound operation Upper Bound(preserves A: Splittable Array): integer ensures Upper Bound “ A.Upper Bound operation Split Point(preserves A: Splittable Array): integer ensures Split Point “ A.Split Point operation Parts Are In Use(preserves A: Splittable Array): Boolean ensures Parts Are In Use “ A.Parts In Use operation Ids Match( preserves A1: Splittable Array, preserves A2: Splittable Array): Boolean ensures Ids Match “ (A1.Id “ A2 .Id ) end Splittable Array Template The Splittable Array component is modeled in part as a function from integers to entries (Contents). The Incl Lower Bound and Excl Upper Bound give the addressable range of indices. The operations allow the client to set a split point within the addressable range, and to split/combine the array into two subarrays, split at A.Split Point. The Parts In Use flag indicates whether the array has been split into subparts, and controls access to those parts. Figure 3 visualizes how a splittable array is split up by the Split operation. Given three splittable arrays A, A1, and A2, the operation call Split(A, A1, A2) places the contents of A below the split point into A1 and the contents of A at and above the split point in A2. After A is split, a client cannot access anything in A until A1 and A2 are combined back into A; that is, until A.Parts In Use “ true.

An Array Abstraction to Amortize Reasoning

355

Fig. 3. The state after Split(A, A1, A2) is executed on a Splittable Array.

3.1

Divide-and-Conquer Client Code Using Splittable Array

A natural application of Splittable Array is in a parallel divide-and-conquer algorithm such as the mutateTops operation from Listing 2. Using the component in such a context dramatically simplifies the reasoning involved in formally verifying the correctness of such code. Listing 4 shows how such an algorithm might be implemented and specified using Splittable Array. Listing 4. A recursive, parallel divide-and-conquer solution using Splittable Array

uses Stack with [[some interference contract]]; uses Splittable Array with [[some interference contract]]; uses Entry with [[some interference contract]]; facility Stack Fac is Stack(Entry); facility Array Fac is Splittable Array(Stack Fac.Stack); operation Mutate Tops(updates A: Splittable Array); requires A.Incl Lower Bound < A.Excl Upper Bound and not A.Parts In Use; ensures [[the top of each stack in A has been mutated]]; interference spec affects A@∗; recursive procedure decreasing A.Excl Upper Bound ´ A.Incl Lower Bound; if (Excl Upper Bound(A) ´ Incl Lower Bound(A) > 1) then var A1, A2: Splittable Array var mid: Integer := (Incl Lower Bound(A) + Excl Upper Bound(A )) / 2 Set Split Point(mid, A) Split(A, A1, A2) cobegin Mutate Tops(A1) Mutate Tops(A2) end;

356

A. Weide et al.

Combine(A, A1, A2) else var stack: Stack var index: Integer := Incl Lower Bound(A) Swap Entry At(A, index, stack) if Length(stack) > 0 then var e: Entry Pop(stack, e) Mutate(e) Push(stack, e) end Swap Entry At(A, index, stack) end end Mutate Tops As discussed in Sect. 2, keeping verification of this code relatively simple involves showing that the operations inside the cobegin block are non-interfering as defined in [8]. First, by the interference spec (a local, operation-level version of the component-level interference contract) of Mutate Tops, we know that it affects all partitions of A (whatever those partitions may be—this particular interference spec is agnostic to the interference contract used with the array). If there were a shared array parameter between the two recursive calls, it would be impossible to show non-interference. Fortunately, however, the two calls to Mutate Tops inside the cobegin statement operate on different array variables, so they are necessarily independent because of RESOLVE’s clean semantics. 3.2

Reusability and Modifiability

The code in Listing 4 is highly reusable. A client can use any implementation of the Stack concept as long as it respects the interference contract defined in the solution. For example, if there is a cactus stack realization of Stack that has appropriate concurrency properties, the client may use it without having to reprove Mutate Tops or write a new specification. In fact, this particular operation is entirely agnostic to any of the interference contracts that may be supplied. It is also highly amenable to modifications, requiring only minimal proofs to be discharged in most cases. Consider an alternate approach to Mutate Tops where the array is split into four parts instead of two. Now the parallel section of the code might look like Listing 5. Thanks to clean semantics, it is still a simple syntactic check to show that the four parallel calls are non-interfering. The one-time proof of disjointness in the intervals falls on the implementer of the Splittable Array specification—but is trivial unless the implementer opts for a shared realization such one discussed in Sect. 3.3.

An Array Abstraction to Amortize Reasoning

357

Listing 5. The parallel section of a divide-and-conquer solution which splits A into four parts via consecutive calls to Split.

cobegin Mutate Mutate Mutate Mutate end

3.3

Tops(A1) Tops(A2) Tops(A3) Tops(A4)

Efficiently Realizing Splittable Array

A combination of clean semantics, careful component design, and robust specification has reduced the potentially complicated reasoning problem of showing non-interference in Listing 2 to a purely syntactic check in Listing 4. This is a clear advantage of our approach over the traditional one. Importantly, these reasoning advantages can be achieved without compromising performance. Although clean semantics allows the two sub-arrays A1 and A2 to be reasoned about as if they were totally separate arrays, an efficient implementation of this concept would not make any copies of the array. The interface for this component was designed with a shared implementation in mind so that a realization could employ an underlying (traditional) array that is shared among all Splittable Array instances with the same Id. This design choice manifests itself in the use of the Split and Combine operations as pseudo-synchronization points by flipping Parts In Use and preventing access to the array while it is split. Doing so ensures that at any time, there is only one array with each Id that can access any given index. In this way, a realization can share an underlying array among instances with the same Id without introducing any interference. Enabling such a shared implementation is important for preserving the performance benefits that programmers expect from parallel software; that is, the operations Split, Combine, and Swap Entry At can all be done in constant time.

4 4.1

A Hierarchy of Array Abstractions Three Distinct Array Abstractions

The Splittable Array abstraction presented here is one member of a hierarchy of concurrency-ready array abstractions that can be used in multiple contexts [6]. The most general abstraction in this family, Index Partitionable Array, may be partitioned on arbitrary indices rather than contiguous portions of the array. A third abstraction, Distinguished Index Array, allows a client to isolate a distinguished entry and operate on it separately from the rest of the array.

358

4.2

A. Weide et al.

Layered Implementations

The Index Partitionable Array (and Distinguished Index Array) can be efficiently implemented in a similar manner to Splittable Array; that is, by sharing a single underlying array amongst all instances with the same ID. Once an efficient realization for Index Partitionable Array is provided, however, more specialized realizations can be built by layering on top of it. For example, Splittable Array can be realized with an underlying Index Partitionable Array and mapping the two sets of indices of the Index Partitionable Array to the low and high parts of the Splittable Array and maintaining the invariant that each set of indices in the underlying Index Partitionable Array is contiguous (and corresponds to the appropriate indices for abstraction to a Splittable Array).

5

Related Work

One central challenge in sequential, object-oriented software verification involves objects, aliasing, and properties about the heap [10, 11]. Separation logic is an extension of Hoare’s logical rules to address these challenges. Examples of verification using separation logic in Coq include [12,13] and in VeriFast to verify Java and C programs include [14,15]. Automating verification with separation logic is the topic of [16–20]. Concurrent separation logics have attempted to expand the capabilities of separation logic by adding rules for reasoning about concurrent programs [21]. An important new direction for abstraction and simplification in concurrent separation logic is the focus of [22]. Our approach sidesteps the concerns of both separation logic and concurrent separation logic first by using a language which has clean semantics and then by abstracting away most implementation details to avoid reasoning directly about the state of the heap. Other approaches have guaranteed determinism in the presence of parallelism using region logic or a variant thereof. In Deterministic Parallel Java [23] (DPJ), regions are defined explicitly by the programmer. These annotations allow a DPJ compiler to guarantee, syntactically, that two concurrent operations are non-interfering and thus will produce a deterministic result. ParaSail [24] is an extension of the Ada programming language that relies on value semantics to verify that concurrent statements are non-interfering. Both approaches are limited by what can be syntactically checked, and ParaSail in particular is limited by the fact that objects must be reasoned about as a whole and cannot be subdivided. By leveraging the full functional verification capabilities of RESOLVE, our approach can increase the expressiveness of method effects over that of DPJ by including conditional effects, and clean semantics let us preserve the simplicity afforded by ParaSail. Both DPJ and ParaSail offer examples of divide-andconquer solutions similar to the one presented here and have verified them to be deterministic (their correctness is informally argued) [25, 26]. DPJ additionally provides the DPJArray and DPJPartition [27, 28] families of classes to attack many of the same problems as the various partitionable

An Array Abstraction to Amortize Reasoning

359

arrays presented in this paper. A DPJArray allows the client to define subarrays, and operate on them as if they were their own array—without making any copies (shallow or otherwise). Because there are no disjointness requirements placed upon subarrays, the implementer of a divide-and-conquer algorithm in DPJ should prefer to use DPJPartition which splits an array into two disjoint, contiguous sections based upon a client-provided index. However, there is nothing to stop a client from accessing elements of a partitioned array while a parallel thread is accessing either of its sub-parts, potentially compromising determinism. In contrast, there is always exactly one instance of a Splittable Array that can access any given element. Other languages that provide array slicing or partition operations typically make shallow copies of the underlying array [29, 30]. This poses two immediate problems. First, it does not eliminate the potential for aliasing between elements of several arrays that have been sliced. Second, the operation has a runtime that is linear in the length of the slice. Determinism guarantees in our research and others’ amount to showing data race freedom, though our approach deals with high-level programming constructs. There is a large body of work on showing low-level race freedom, including both static approaches [31–36] and dynamic ones [37–40]. Like DPJ and ParaSail, these are limited to guaranteeing determinism and do not claim to formally verify full functional correctness.

6

Conclusions

Verifying the correctness of a parallel divide-and-conquer algorithm in a modular way using traditional arrays is a difficult problem for automated verification engines. To solve this problem, we develop a new abstraction, the Splittable Array, with an interference contract that abstractly defines how data can be accessed. Combined with careful component design and clean semantics, this approach allows us to write software that is easy to reason about even in the presence of concurrency. The Splittable Array allows reusable verification of non-interference of array partitions in one place instead of for every client. The abstraction allows frame conditions to be captured in a novel way that reduces repeated verification costs, as demonstrated through the divide-and-conquer examples in this paper. The development of this new data abstraction allows standard client verification machinery to be used, such as tools developed for the verification of RESOLVE programs [41,42]. Acknowledgments. We thank the members of our research groups at Clemson, Ohio State, and other institutions who have contributed to the discussions on topics contained in this paper. This research is funded in part by US NSF grant DUE-1022941. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the views of the NSF.

360

A. Weide et al.

References 1. Sitaraman, M., et al.: Building a push-button RESOLVE verifier: progress and challenges. Formal Aspects Comput. 23(5), 607–626 (2011) 2. Harms, D.E., Weide, B.W.: Copying and swapping: influences on the design of reusable software components. IEEE Trans. Softw. Eng. 17, 424–435 (1991) 3. Kulczycki, G.: Direct Reasoning. Ph.d. dissertation, Clemson University, School of Computing (2004) 4. Kulczycki, G., et al.: A language for building verified software components. In: Favaro, J., Morisio, M. (eds.) ICSR 2013. LNCS, vol. 7925, pp. 308–314. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38977-1 23 5. Weide, A., Sivilotti, P.A.G., Sitaraman, M.: Enabling modular verification with abstract interference specifications for a concurrent queue. In: Blazy, S., Chechik, M. (eds.) VSTTE 2016. LNCS, vol. 9971, pp. 119–128. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48869-1 9 6. Weide, A., Sivilotti, P.A.G., Sitaraman, M.: Array abstractions to simplify reasoning about concurrent client code. Technical Report RSRG-17-05, Clemson University - School of Computing, Clemson, SC 29634, November 2017 7. Weide, A., Sivilotti, P.A., Sitaraman, M.: Toward a modular proof rule for parallel operation calls. Technical Report OSU-CISRC-3/20-TR01, The Ohio State University, Columbus, OH, March 2020 8. Weide, A., Sivilotti, P.A.G., Sitaraman, M.: Enabling modular verication of concurrent programs with abstract interference contracts. Technical Report RSRG-16-05, Clemson University - School of Computing, December 2016 9. Tschannen, J., Furia, C.A., Nordio, M., Meyer, B.: Usable verification of objectoriented programs by combining static and dynamic techniques. In: Barthe, G., Pardo, A., Schneider, G. (eds.) Software Engineering and Formal Methods, (Berlin, Heidelberg), pp. 382–398, Springer, Heidelberg (2011). https://doi.org/10.1007/ 978-3-642-24690-6 26 10. Leavens, G.T., Leino, K.R.M., M¨ uller, P.: Specification and verification challenges for sequential object-oriented programs. Form. Aspects Comput. 19, 159–189 (2007) 11. Reynolds, J.C.: Separation logic: a logic for shared mutable data structures. In: Proceedings of the 17th Annual IEEE Symposium on Logic in Computer Science, LICS 2002, Washington, DC, USA, pp. 55–74. IEEE Computer Society (2002) 12. Chlipala, A., Malecha, G., Morrisett, G., Shinnar, A., Wisnesky, R.: Effective interactive proofs for higher-order imperative programs. In: Proceedings of the 14th ACM SIGPLAN International Conference on Functional Programming, ICFP 2009, New York, NY, USA, pp. 79–90. ACM (2009) 13. Chen, H., Ziegler, D., Chajed, T., Chlipala, A.,. Kaashoek, M.F, Zeldovich, N.: Using crash Hoare logic for certifying the FSCQ file system. In: Proceedings of the 25th Symposium on Operating Systems Principles, SOSP 2015, New York, NY, USA, pp. 18–37. ACM (2015) 14. Jacobs, B., Smans, J., Piessens, F.: Verifying the composite pattern using separation logic. In: Proceedings of the 7th International Workshop on Specification and Verification of Component-Based Systems, SAVCBS 2008, pp. 83–88 (2008) 15. Jacobs, B., Smans, J., Piessens, F.: A quick tour of the VeriFast program verifier. In: Proceedings of the 8th Asian conference on Programming Languages and Systems, APLAS 2010, pp. 304–311. Springer, Heidelberg (2010). https://doi.org/10.1007/ 978-3-642-17164-2 21

An Array Abstraction to Amortize Reasoning

361

16. Bobot, F., Filliˆ atre, J.-C.: Separation predicates: a taste of separation logic in firstorder logic. In: Aoki, T., Taguchi, K. (eds.) ICFEM 2012. LNCS, vol. 7635, pp. 167–181. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-342813 14 17. Botinˇcan, M., Parkinson, M., Schulte, W.: Separation Logic Verification of C Programs with an SMT Solver. Electron. Notes Theor. Comput. Sci. 254, 5–23 (2009) 18. Piskac, R., Wies, T., Zufferey, D.: Automating separation logic using SMT. In: Sharygina, N., Veith, H. (eds), CAV 2013. LNCS, vol. 8044, pp. 773–789. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39799-8 54 19. Piskac, R., Wies, T., Zufferey, D.: Automating separation logic with trees and data. In: Biere, A., Bloem, R. (eds.) CAV 2014. LNCS, vol. 8559, pp. 711–728. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08867-9 47 20. Bouillaguet, C., Kuncak, V., Wies, T., Zee, K., Rinard, M.: Using first-order theorem provers in the Jahob data structure verification system. In: Cook, B., Podelski, A. (eds.) VMCAI 2007. LNCS, vol. 4349, pp. 74–88. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-69738-1 5 21. Brookes, S.: A semantics for concurrent separation logic. In: Gardner, P., Yoshida, N. (eds.) CONCUR 2004. LNCS, vol. 3170, pp. 16–34. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-28644-8 2 22. Krebbers, R., Jung, R., Bizjak, A., Jourdan, J.-H., Dreyer, D., Birkedal, L.: The essence of higher-order concurrent separation logic. In: Yang, H., (ed.) Programming Languages and Systems. LNCS, vol. 10201, pp. 696–723, Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-662-54434-1 26 23. Bocchino Jr., R.L., et al.: A type and effect system for deterministic parallel Java. In: Proceedings of the 24th ACM SIGPLAN Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA 2009, New York, NY, USA, pp. 97–116. ACM (2009) 24. Taft, T., Hendrick, J.: Designing ParaSail, a new programming language (2017). http://parasail-programming-language.blogspot.com/ 25. The Deterministic Parallel Java Tutorial, Version 1.0 (2010). http://dpj.cs.illinois. edu/DPJ/Download files/DPJTutorial.html 26. Taft, T.: Tutorial: multicore programming using divide-and-conquer and work stealing (2013). https://drive.google.com/file/d/0B6Vq5QaY4U7uUktTcUpySm VjaW8/edit 27. Class DPJArray. https://dpj.cs.illinois.edu/DPJ/Download files/DPJRuntime/ DPJArray.html 28. Class DPJPartition. https://dpj.cs.illinois.edu/DPJ/Download files/DPJRuntime/ DPJPartition.html 29. Array.prototype.slice(). https://developer.mozilla.org/en-US/docs/Web/JavaScri pt/Reference/Global\ Objects/Array/slice 30. An informal introduction to Python. https://docs.python.org/3/tutorial/introd uction.html\#lists 31. Engler, D., Ashcraft, K.: RacerX: effective, static detection of race conditions and deadlocks. In: Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP 2003, New York, NY, USA, pp. 237–252. ACM (2003) 32. Naik, M., Aiken, A., Whaley, J.: Effective static race detection for Java. In: Proceedings of the 27th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2006, New York, NY, USA, pp. 308–319. ACM (2006) 33. Freund, S.N., Qadeer, S.: Checking concise specifications for multithreaded software. J. Object Technol. 3(6), 81–101 (2004)

362

A. Weide et al.

34. Flanagan, C., Qadeer, S.: A type and effect system for atomicity. In: Proceedings of the ACM SIGPLAN 2003 Conference on Programming Language Design and Implementation, PLDI 2003, New York, NY, USA, pp. 338–349. ACM (2003) 35. Naik, M., Aiken, A.: Conditional must not aliasing for static race detection. In: Proceedings of the 34th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2007, New York, NY, USA, pp. 327–338. ACM (2007) 36. Kawaguchi, M., Rondon, P., Bakst, A., Jhala, R.: Deterministic parallelism via liquid effects. In: Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2012, New York, NY, USA, pp. 45–54. ACM (2012) 37. Pozniansky, E., Schuster, A.: MultiRace: efficient on-the-fly data race detection in multithreaded C++ programs: research articles. Concurr. Comput. Pract. Exper. 19, 327–340 (2007) 38. Pozniansky, E., Schuster, A.: Efficient On-the-fly data race detection in multithreaded C++ program. In: Proceedings of the Ninth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP 2003, New York, NY, USA, pp. 179–190. ACM (2003) 39. Flanagan, C., Freund, S.N.: FastTrack: efficient and precise dynamic race detection. In: Proceedings of the 30th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2009, New York, NY, USA, pp. 121–133. ACM (2009) 40. Savage, S., Burrows, M., Nelson, G., Sobalvarro, P., Anderson, T.: Eraser: a dynamic data race detector for multi-threaded programs. In: Proceedings of the Sixteenth ACM Symposium on Operating Systems Principles, SOSP 1997, New York, NY, USA, pp. 27–37. ACM (1997) 41. ResolveOnline. http://resolveonline.cse.ohio-state.edu/ 42. RESOLVE Web IDE. https://resolve.cs.clemson.edu/teaching

Parallel Accurate and Reproducible Summation Farah Benmouhoub1(B) , Pierre-Loic Garoche2 , and Matthieu Martel1,3 1

LAMPS Laboratory, University of Perpignan, Perpignan, France {farah.benmouhoub,matthieu.martel}@univ-perp.fr 2 ENAC, Toulouse, France [email protected] 3 Numalis, Montpellier, France

Abstract. Floating-point arithmetic is prone to accuracy problems due to the round-off errors. The combination of the round-off errors and of the out of order execution of arithmetic operations due to the scheduling of parallel tasks, introduces additional numerical accuracy issues. In this article, we address the problem of improving the numerical accuracy and reproducibility of summation operators. We propose two efficient parallel algorithms for summing n floating-point numbers. The first objective of our algorithms is to obtain an accurate result without increasing the linear complexity of the naive algorithm. The second objective is to improve the reproducibility of the summations compared to those computed by the naive algorithm and this regardless of the number of processors used for the computations. Keywords: Floating-point arithmetic · Parallel computing summation · Numerical accuracy · Reproducibility

1

· Accurate

Introduction

Due to the progresses of high performance computers, parallel computing has become a key technology to efficiently tackle complex scientific and engineering problems. Almost all the computations are performed with floating-point arithmetic as defined by the IEEE754 Standard [1, 7, 17] and this is why users have to pay attention to numerical accuracy. Because they use a limited number of digits, the floating-point numbers cannot represent real numbers accurately without introducing round-off errors. The first goal of our contribution consists of improving the numerical accuracy of the summation results thanks to the algorithms introduced in this article. Indeed, there has been many research efforts related to the numerical accuracy of the sum and the dot product of floating-point numbers [4, 5, 13, 16, 18, 20–22]. In this article, we focus on floating-point summation of sets of floating-point numbers. Some work is based on compensated methods [13, 16, 18, 20–22]. Others use accumulators in high precision and manipulate the mantissa and the exponents of floating point numbers [4, 5]. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 363–382, 2022. https://doi.org/10.1007/978-3-030-80119-9_21

364

F. Benmouhoub et al.

Our second goal is to focus on reproducibility of the summation results, when the summation is done by a parallel machine. We note that reproducibility and accuracy are not synonymous but improving the accuracy often improves the reproducibility. In fact, it is hard to debug programs when the results of several executions of the same code on different or even similar parallel architectures are different. A first solution to ensure the reproducibility is to determine a deterministic order of computations and how they are assigned to processors. This approach was considered impractical and does not scale well with the number of processors. A second solution aims at reducing or eliminating the round-off errors, i.e. by improving the numerical accuracy of computations that we will further see in this article. To overcome these issues, we propose two summation algorithms with the following properties: 1. They improve the numerical accuracy of floating-point summations without increasing the working precision, in sequential or parallel. 2. They improve the reproducibility of summation results independently on how the summands are assigned to processors. 3. They do not increase the complexity compared to the naive algorithm. This article is organized as follows. Section 2 introduces briefly some elements of floating-point arithmetic followed by related work of some existing summation methods proposed to improve the numerical accuracy and reproducibility of computations. We finish by detailing the contribution which motivates this article. Section 3 describes our algorithms to compute accurate and reproducible sums. The experimental results related to the numerical accuracy and reproducibility provided by our algorithms are given in Sect. 4. Finally, in Sect. 5 we conclude and discuss future work.

2

Notations and Background

In this section, we introduce some useful notions used in the rest of this article. In Sect. 2.1 we start with a brief description of the floating-point arithmetic as defined by the IEEE754 Standard [1,7, 17]. Section 2.2 discusses related work. Demmel and Hida algorithms [5] are explained in details in Sect. 2.3. 2.1

Floating-Point Arithmetic

Following the IEEE754 Standard [1, 7, 17], a floating-point number x in base β is given by Eq. (1). x = s · (d0 · d1 · df −1 ) · β exp = s · mβ exp−f +1 where – – – –

s ∈ {−1, 1} is the sign, m = d0 d1 ....df −1 is the mantissa with digits 0 ≤ di < β, 0 ≤ i ≤ f − 1, f is the precision, exp is the exponent with expmin ≤ exp ≤ expmax .

(1)

Parallel Accurate and Reproducible Summation

365

The IEEE754 Standard defines binary formats with some particular values for f , expmin and expmax which are summarized in Table 1. Moreover, the IEEE754 Standard defines four rounding modes for elementary operations over floatingpoint numbers. These modes are towards +∞, towards −∞, towards zero and towards nearest denoted by ◦+∞ , ◦−∞ , ◦0 and ◦∼ , respectively. The behavior of the elementary operations  ∈ {+, −, ×, ÷} between floating-point numbers is given by Eq. (2) (2) v1 ◦ v2 = ◦(v1  v2 ) where ◦ denotes the rounding mode ◦ ∈ {◦+∞ , ◦−∞ , ◦0 , ◦∼ }. By Eq. (2), we illustrate that in floating-point computations, performing an elementary operation ◦ with rounding mode ◦ returns the same result to the one obtained by an exact operation , then rounding the result using ◦. The IEEE754 Standard also specifies how the square root function must be rounded in a similar way to Eq. (2) but does not specify the round-off of other functions like sin, log, etc. Table 1. Binary formats of IEEE754 standard Format

#total bits f bits exp bits expmin expmax

Half precision

16 bits

11

5

−14

+15

Single precision

32 bits

24

8

−126

+127

Double precision

64 bits

+1223

Quadruple precision 128 bits

53

11

−1122

113

15

−16382 +16383

The Eqs. (3) and (4) define two functions denoted by uf p and ulp which compute the unit in the first place and the unit in the last place of a floatingpoint number, respectively. These functions are used further in this article to compute the round-off errors caused by the summation algorithms. The uf p of a number x is given by (3) uf p(x) = logβ | x | The ulp of a floating-point number x with f significants bits is ulp(x) = uf p(x) − f + 1

(4)

In this article, without loss of generality, we consider that β = 2. Moreover, we assume that no overflow occurs and we only consider the rounding mode to the nearest. In floating-point computations absorption and cancellation may affect the numerical accuracy of computations. An absorption occurs when adding two floating-point numbers with different orders of magnitude. The small value is absorbed by the large one. For example, let us consider the sum s = ((x + y) + z) such that x, y and z are three floating-point numbers stored in binary 32 format, where x = 1.2 × 20 , y = 1 × 224 and z = −1 × 224 , respectively. Then, the

366

F. Benmouhoub et al.

computation of s produces the result s = 0.0 which has no correct digits, as the correct real result should be s = 1.2 × 20 . A cancellation occurs when two nearly equal numbers are subtracted and the most significant digits cancel each other. The severity of the cancellation differs from one computation to another depending on the number of useful bits lost. For example, let x = 1.5 × 20 and y = 1 × 224 two floating-point numbers in single precision. The computation of the sequence of operations s = ((x + y) − y) produces the result s = 0.0 while the real result should be 1.5 × 20 . 2.2

Related Work

Summation of floating-point numbers is one of the most basic tasks in numerical analysis. To compute the sum S of n floating-point numbers such that S = n s using the naive algorithm [12] of Eq. (5), i i=1 S = (((s1 + s2 ) + s3 ) + ... + sn )

(5)

we know that the absolute error between the exact sum and the approximate one can be bound by Eq. (6). This bound will be used in the rest of this article to compare the worst-case round-off error between the naive algorithm and our algorithms. n n−1 n   (n − 1)u  si |≤ | ei |≤ | si |, (6) | Sˆ − 1 + u i=1 i=1 i=1 where – | ei | denotes the absolute error of the ith floating-point addition, – u = 12 β 1−p = 12 ulp(1). Research work has focused on improving the numerical accuracy [4, 5,13, 16, 18, 20] or reproducibility [6] of the computations involving summations. There are many sequential and parallel algorithms for this task, a good survey of them is presented in [10,11]. Floating point summation is often improved by compensated summation methods [13,16, 18, 20–22] with or without the use of the error-free transformations to compute the error introduced by each accumulation. We are going to detail some of the compensated sum algorithms further in this section. The accuracy of summation algorithms can also be improved by manipulating the exponent and the mantissa of the floating-point numbers in order to split data before starting computations [4, 5]. This approach is the one employed in this article and it is explained in details in Sect. 2.3. Compensated Summation Methods: Kahan [13] introduced a compensated summation method. The idea is to compute the exact rounding error on an addition as given by Algorithm 1. Compensated summation algorithms accumulate these errors and add the result to the result of the summation. The compensation process can be applied recursively yielding cascaded compensated algorithm.

Parallel Accurate and Reproducible Summation

367

Algorithm 1. Compensated summation Summing n floating-point numbers s = 1: s=0;c=0 2: for i=0 to n do 3: temp=s, y=xi +c 4: s=temp+y, c=(temp-s)+y

n

i=1

xi

Malcolm [16] describes cascading methods based on the limited exponent range of floating-point numbers. He defines an extended precision array ei where each component corresponds to an exponent. To extract and scale the exponent, Malcolm uses an integer division, without requiring the division to be a power of 2. If the extended precision has 53+k bits then obviously no error n in the mantissa, n occurs for up to 2k summands and i=1 pi = i=1 ei . The summands pi are added with the respect to decreasing order into the array element corresponding to their exponent. Rump et al. [18,20, 21] proposed several algorithms for summation and dot product of floating point numbers. These algorithms are based on iterative application of compensations. An extension of the compensation of two floating-point numbers to vectors of arbitrary length is also given and used to compute a result as if computed with twice the working precision. Various applications of compensated summation method have been proposed [8, 9]. Th´evenoux et al. [23] implement an automatic code transformation to derive a compensated programs. Parallel Approaches: In addition to the existing sequential algorithms, many other parallel algorithms have been proposed. Leuprecht and Oberaigner [15] describe parallel algorithms for summing floating-point numbers. The authors propose a pipeline version of sequential algorithms [2, 19] dedicated to the computation of exact rounding summation of floating-point numbers. In order to ensure the reproducibility, Demmel and Nguyen [6] introduce a parallel floatingpoint summation method based on a technique called pre-rounding to obtain deterministic rounding errors. The main idea is to pre-round the input floatingpoint values to a common base according to some boundary so that their sum can be computed without any rounding error. The error depends only on input values and the boundary, contrary to the intermediate results which depend on the order of computations. All these works focused on the improvement of the numerical accuracy. Some others [3] try to make the best use of the parallel architecture performance in order to speed up the program execution. De Luca et al. [3] present how to efficiently use multicore architecture platforms when solving two-dimensional inverse time fractional diffusion equation. 2.3

Demmel and Hida Algorithms

In this section, we describe the accurate floating-point summation algorithms introduced by Demmel and Hida [5]. These algorithms share a few common

368

F. Benmouhoub et al.

ideas with our algorithms introduced in Sect. 3. Given two precisions f and F with F > f , Demmel and Hida’s algorithms use an accumulator of precision F for summing n floating-point numbers of precision f in decreasing order. In Algorithm 2, n floating-point numbers given in precision f areadded in an extra n precise accumulator of precision F in order to compute S = i=1 si . Algorithm 2. Accurate Summation by Sorting Exponents [[5]] 1: 2: 3: 4: 5:

Sort the si such that exp(s1 ) ≥ exp(s2 ) ≥ · · · ≥ exp(sn ) S←0 //S has F bits of precision. for i ← 1 to n do //si has f bits of precision S ← S + si s ← round(S)

Algorithm 3 requires a fixed array of accumulators A0 , ...., AN of precision F , each Aj accumulating the sum of the si where e leading bits are j. Then these Aj will be added using Algorithm 2. Algorithm 3. Accurate Summation using N Accumulators [[5]] 1: 2: 3: 4: 5: 6: 7:

Choose parameter e (number of leading exponent bits of each si to extract). Initialize Ak = 0 for k = 0, 1, ..., N , where N = 2e . for i=1 to n do j ← leading e bits of the exponent of si Aj ← Aj + si //from N to 0 Add the Aj yielding S of precision F s ← round(S) //round S back to f-bits

This algorithm requires accessing the exponent field of each si to decide to which accumulator Aj to add it. By letting j be the leading exponent bits of si , Algorithm 3 divides the exponent range of the si into groups of powers of two. Demmel and Hida [5] give details on the accuracy of the computed result depending on f , F and the number of summands. In contrast to our work which use only working precision with an access to the exponents, Demmel and Hida’s algorithms need in any case some extra precision floating-point format and also an access to the mantissa. Let us also note that these algorithms have a complexity of O(nlogn) thanks to the sorting step, compared to a naive summation algorithm which has complexity O(n) where n is the number of summands.

3

Accurate Summation

In this section, we introduce new algorithms which offer several advantages compared to the algorithms of Sect. 2.3:

Parallel Accurate and Reproducible Summation

369

1. They improve the numerical accuracy similarly to Demmel and Hida algorithms [5]. 2. Their complexity is linear, just like the naive summation algorithms. 3. They perform all the computations in the working precision without using accumulators of high precision. 4. They can be easily parallelized and they scale up with the number of processors. 5. They improve reproducibility compared to the parallel version of the naive algorithms. The sequential algorithm is introduced in Sect. 3.1. Two parallel versions of the sequential algorithm are presented in Sects. 3.2.1 and 3.2.2, the first one performs local sums of its accumulations while the second concentrates all the final accumulation on a single processor. 3.1

Accurate Summation in Sequential

As mentioned in Sect. 3, we provide a sequential algorithm for accurately summing n floating-point numbers. Algorithm 4. Accurate summation in sequential 1: Initialization of the array sum by exp 2: total sum=0.0 3: for i=0 to exp max do  getExp: function used to compute exponent 4: exp s i=getExp(s[i])+bias 5: in base 2 6: sum by exp[exp s i]=sum by exp[exp s i]+s[i]  Total summation 7: total sum=0.0 8: for i=0 to exp max do 9: total sum=total sum+sum by exp[i]

Algorithm 4 works as follows: Before starting the summation, the algorithm allocates an array called sum by exp which is initialized at 0 for all its elements. Let us assume that the exponents range from exp min to exp max. The array sum by exp has exp max − exp min + 1 elements (note that this number does not depend on the number of summands n). Using a bias such as bias = −exp min, the idea is to sum all the summands whose exponents is 2i in the cell sum by exp[i + bias]. This avoids most absorptions while avoiding to sort explicitly the array. Let n exp (si ) denote the exponent of si in base 2. For computing the sum S = i=1 si , each value si is added to the relevant cell sum by exp[exp si +bias] as described by the first loop. Then a sum is performed over the elements of the sum by exp array to compute a total sum (second loop). The complexity of Algorithm 4 is O(n) like the naive algorithm. We will see in Sect. 4.3 the comparison between Algorithm 4 and Algorithm 2 of Demmel and Hida.

370

3.2

F. Benmouhoub et al.

Accurate Summation in Parallel

In this section, we describe our parallel algorithms which aim to improve the numerical accuracy and reproducibility of the summation results. In Sect. 3.2.1 we give the first algorithm which is characterized by the computation of a local sum at each processor before computing the total result while Sect. 3.2.2 presents the second algorithm that computes the total sum only by reduction. 3.2.1 Accurate Summation with Local Sum per Processor In this section, we introduce our first parallel Algorithm 4. More precisely, let us assume that we have P processors and n summands (we assume n P and that P divides n). We assign n/P summands to each processor. Processor i, (i+1)×n/P −1 sj by the method detailed hereafter. Then 0 ≤ i < P computes j=i×n/P a reduction is done to compute the final result. The values si are added to the appropriate cell sum by exp[exp si +bias] according to their exponents similarly to the Algorithm 4 described in Sect. 3.1. Our algorithm is parallel and, consequently, each processor has an sum by exp array to sum locally. In order to obtain its local final result we add these values in increasing order. Once the final local sums are computed, a reduction is being done and the processor receiving the result of the reduction gets the total sum. The cost of Algorithm 5 is O(n/P ) using P processors, no sorting being performed and the access to the data to be summed is done only once. Algorithm 5. Accurate Summation with Local Sum at Each Processor 1: 2: 3: 4: 5: 6: 7: 8:

Initialization of the array sum by exp total sum=0.0  p row: rows number of each processor for i=0 : p row do  getExp:compute exponent in base 2 exp s i=getExp(s[i])+bias sum by exp[exp s i]=sum by exp[exp s i]+s[i] local sum=0.0  Summing locally in order of increasing exponents for i=0 to exp max do local sum=local sum+sum by exp[i]  Total sum by processor 0 9: Reduce(local sum,total sum)

In the rest of this section, we assess the accuracy of Algorithm 5. We start with the following lemma. i≤n Lemma 1. Let us assume rounding to the nearest and let Sn = i=1 si be the sum in precision f of n floating-point numbers of the same exponent exp, i.e. ∀i, 1 ≤ i ≤ n, 2exp ≤ si < 2exp+1 . If n = 2m then the error εn on Sn is bounded by (7) εn < 2exp+m+1−f .

Parallel Accurate and Reproducible Summation

371

Proof By induction on m. If m = 1 then n = 21 = 2 and S2 = s1 + s2 . By hypothesis, ∀i, 1 ≤ i ≤ n, 2exp ≤ si < 2exp+1 and consequently S2 < 2exp+1 +2exp+1 or, equivalently, S2 < 2exp+2 . Then in precision f with a rounding mode to the nearest, the round-off error ε2 following the Eq. (2) on S2 is bounded by ε2 < 2exp+2−f . Since m = 1 we have ε2 < 2exp+m+1−f . Let us assume that Eq. (7) holds for m and let us demonstrate that it still holds for m + 1, i.e. that ε2n < 2exp+(m+1)+1−f . Recall that n = 2m . We have S2n = Sn + σ

with σ =

2n 

si .

i=n+1

Since σ is a sum of n summands of exponent exp done in precision f , by induction hypothesis the error εσ on σ is bounded by εσ < 2exp+m+1−f . Consequently, the error on S2n is bounded by ε2n < 2exp+m+1−f + 2exp+m+1−f = 2exp+m+2−f = 2exp+(m+1)+1−f .  Lemma 2 extends Lemma 1 to any number of summands.

i≤n Lemma 2. Let us assume rounding to the nearest and let Sn = i=1 si be the sum in precision f of n floating-point numbers of the same exponent exp, i.e. ∀i, 1 ≤ i ≤ n, 2exp ≤ si < 2exp+1 . The error εn on Sn is bounded by εn < 2exp+log2 n+1−f .

(8)

Proof From Lemma 1, if n = 2m then εn < 2exp+m+1−f . Then, if 2m−1 < n ≤ 2m then m−1 < log2 n ≤ m and log2 n = m. Consequently 2exp+log2 n+1−f =  2exp+m+1−f . Finally, based on Lemma 2, Theorem 1 bellow assesses accuracy of Algorithm 5. i≤n Theorem 1. Let us assume rounding to the nearest and let Sn = i=1 si be the sum in precision f of n floating-point numbers such that 2expmin ≤ si < 2expmax − 1, for all 1 ≤ i ≤ n. The error εn on Sn computed by Algorithm 5 is bounded by (9) εn < 2expmax +log2 n+1−f . Proof For Algorithm 5, in the worst case, all the summands have the same exponent expmax . By Lemma 2, the error is bounded by εn < 2expmax +log2 n+1−f .

(10)

 Let us remark that the error bound of Algorithm 5, given in Eq. (10) grows logarithmically in the number n of summands while the corresponding bound for the naive algorithm given in Eq. (6) grows linearly in n. Let us also remark that Algorithm 5 and the naive algorithm have the same time complexity (linear in n). Consequently Algorithm 5 can be considered as being strictly better than the naive algorithm.

372

F. Benmouhoub et al.

3.2.2 Accurate Summation with Global Sum The second parallel algorithm differs from Algorithm 5 in that the final sum is not carried out in the same way. Again, the values si are added into the array Algorithm 6. Accurate summation without local sum at each processor 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11:

Initialization of the array elements sum by exp total sum=0.0 for i=0 : p row do  p row: rows number of each processor exp s i=getExp(s[i])+bias sum by exp[exp s i]=sum by exp[exp s i]+s[i] MPI Gather(sum by exp,exp max,,&sum by exp proc0[rank*exp max],exp max,,0,)  Summing the values by column before total sum by processor 0 for j=0 : exp max do sum by column=0.0; for i=0 : exp max do sum by column=sum by column+sum by exp proc0[i*exp max+j]; total sum=total sum+sum by column

sum by exp according to their exponent exp si . But rather than computing a local sum in each processor, each local array sum by exp is send to processor 0 using a gather function. Once this step is achieved, processor 0 sums up the components of all the array elements (corresponding to all processors) by exponents to compute the total sum given by Eq. (11): S=

E P −1  

sum by exp[i, j]

(11)

j=0 i=0

where E = exp max − exp min + 1. By way of illustration, Fig. 1 summarizes the process of computing the sum by Algorithm 6. exp min and exp max being constants, the complexity of algorithm 6 remains linear in the number of summands n (the number of exponents begin constant). Compared to Algorithm 5, Algorithm 6 has the following advantages and drawbacks: – It is more accurate, the sum by exponents being done globally instead of locally by each processor in Algorithm 5. – It performs more communications since E × P values are communicated instead of P in Algorithm 5. – While being linear the constant of the O(n) complexity is higher than Algorithm 5 since E×P sums are done in the final step instead of P in Algorithm 5. Let us note that, in the worst case, the error bound for Algorithm 6 is identical to the bound given in Eq. (10) for Algorithm 5 (the worst case being when all the floating-point numbers have the same exponent).

Parallel Accurate and Reproducible Summation

373

Fig. 1. Diagram of sum computations.

4

Numerical Experiments

In this section, we first evaluate the efficiency of our algorithms described in Sect. 3 on the improvement of the numerical accuracy Sect. 4.1 and the reproducibility (see Sect. 4.2). Secondly, a comparison is made between our sequential algorithm (Algorithm 4) and Algorithm 2 of Demmel and Hida in Sect. 4.3. In our experiments, we use sets of data from 10000 to 100000 summands using various kinds of values chosen to introduce ill-conditioned sums. The floatingpoint values can be small, medium or large, which means, respectively, of the order of 10−7 , 100 and 107 . This is motivated by the single precision of IEEE754 arithmetic. In order to generate various types of absorptions and cancellations, we introduce eight sets of data with different proportions of large values among small and medium values following the methodology of Th´evenoux et al. [14] as specified by Table 2. Table 2. Sign and proportion of large values among small and medium values for each set of data [Th´evenoux et al. [14]]. Datasets

Dataset1 Dataset2 Dataset3 Dataset4 Dataset5 Dataset6 Dataset7 Dataset8

Sign

+

% large values 10%

4.1

+

+

+

+ and − + and − + and − + and −

30%

50%

70%

10%

30%

50%

70%

Numerical Accuracy of the Algorithms

In this section, we evaluate how our algorithms improve the numerical accuracy of floating-point sums. To evaluate the accuracy of Algorithm 5 and 6 in single precision, we compare their results to a sum done in IEEE754 double precision. The first four data sets contain summands of the same sign. Figure 2 represents the absolute errors computed for these first four data sets. These data sets have different proportions (from 10% to 70%) of large values among middle and small

374

F. Benmouhoub et al.

Fig. 2. Absolute errors between results of summations for the first four data sets using three algorithms: Algorithm 5, Algorithm 6 and the Naive algorithm in single precision with the results of summations of the same data sets using Naive algorithm in double precision (Left), zoom on the absolute errors of our summation algorithms (Right).

Parallel Accurate and Reproducible Summation

375

Fig. 3. Absolute errors between results of summations for the last four data sets using three algorithms: Algorithm 5, Algorithm 6 and the Naive algorithm in single precision with the results of summations of the same data sets using Naive algorithm in double precision (Left), zoom on the absolute errors of our summation algorithms (Right).

376

F. Benmouhoub et al.

values. In our experiments, for each data set we generate sets from 10000 to 100000 that will be summed using one of the algorithms studied. For example, results of Dataset1 show that our summation algorithms improve the accuracy of the floating-point summation compared to the naive algorithm in single precision (see Figs. 2a, 2b). We notice that the absolute errors of the naive algorithm of summation are much larger than those of our summation algorithms for each data set. The absolute errors vary from 300 to 55000 for the naive algorithm unlike our algorithms where the errors are between 0 and 270. Concerning our algorithms, three cases arise: – The results of summations of the two algorithms (Algorithm 5 and 6) are similar, so we obtain the same absolute errors, – the results of summations of Algorithm 5 are better than those of algorithm 6, which is justified by the fact that the intermediate sums per column computed by the Algorithm 6 are not in increasing order of exponents, therefore a loss of accuracy is observed. As a consequence, the absolute errors computed for this algorithm will be greater than those of the Algorithm 5, – the results of summations of Algorithm 6 are better than those of Algorithm 5 due to the gain in accuracy obtained thanks to the increasing order of the intermediate sums computed by the Algorithm 6, so the absolute errors of Algorithm 5 are greater than those of Algorithm 6. In the same way, we note that the results of Dataset2 and Dataset3 are similar to those of Dataset1. However, the results of Dataset4 (Fig. 2g and 2h) show that the absolute errors of the naive summation algorithm are less than that computed for other data sets, which is due to the proportion of larger values contained in each of them. More precisely, in Dataset4 we have 70% of large values among small and medium one, therefore the accumulated errors of this data set caused by absorptions will not be greater than that generated by Dataset1 or even Dataset2 containing respectively 10% et 30% of large values among small and medium values. In fact, the absolute errors of the naive algorithm decrease from 55000 to 3700. In the second part of our experiments, we examine the data sets containing the floating-point values with positive and negative signs. Figure 3 summarizes the results obtained for the corresponding data sets. The purpose of these experiments is to measure the impact of the cancellations on the accuracy of summations computations. As we can observe in Fig. 3, our summation algorithms also improve the numerical accuracy of the computations performed between values of different signs, compared to the naive algorithm which represents many more absolute errors. Indeed, the lack of accuracy was caused by computations involving floating-point values of different signs and magnitude of small, medium and large values. 4.2

Reproducibility

In this section, we evaluate the efficiency of Algorithm 5 and 6 on the improvement of reproducibility. Indeed, the combination of the non-associativity of

Parallel Accurate and Reproducible Summation

377

Fig. 4. Reproducibility of the summation results of Algorithm 5 and of the Naive algorithm in single precision for Dataset3 in function of the number of processors.

Fig. 5. Reproducibility of the summation results of Algorithm 5 and of the Naive algorithm in single precision for Dataset4 in function of the number of processors.

floating-point operations like addition and computations based on parallel architecture may affect reproducibility. Figure 4 and 6 give the results of reproducibility for Dataset3 and Dataset4 using Algorithm 5 and Algorithm 6, respectively with a number P of processors varying from 2 to 16. Although, Fig. 5 and 7 give the results of reproducibility for Dataset1 and Dataset2 using Algorithm 5 and Algorithm 6, respectively also by varying the number P of processors from 2 to 16 During our experiments, the absolute errors are computed between the summation results of Algorithm 5, Algorithm 6 and naive algorithm in single

378

F. Benmouhoub et al.

Fig. 6. Reproducibility of the summation results of Algorithm 6 and of the Naive algorithm in single precision for Dataset1 in function of the number of processors.

Fig. 7. Reproducibility of the summation results of Algorithm 6 and of the Naive algorithm in single precision for Dataset2 in function of the number of processors.

precision by varying the number of processors and their respective algorithm with only one processor. First, the results show that the absolute errors of the naive algorithm are larger (between 2368 and 14144) than those of the two new summation algorithms (between 0 and 488). Second, the results of summations computed by the naive algorithm performed on a large number of processors are those which have a larger error, as an example data set of 100000 values has an error equal to 14144. In other words, the results of summations obtained using the naive algorithm with 2 processors are more reproducible than those obtained

Parallel Accurate and Reproducible Summation

379

with 4, 8 and 16 processors. Contrarily to the naive algorithm of summation, the absolute errors linked to the two algorithms described in this article are small whatever the number of processor used to perform the summation computations. 4.3

Our Sequential Algorithm Vs Demmel and Hida Algorithm

In this section, we establish a comparison between our sequential Algorithm 4 already described in Sect. 4 with Demmel and Hida’s Algorithm 2 given in Sect. 2.3. The first experimentation consists of comparing the accuracy of the summation results carried out using our sequential algorithm (Algorithm 4), Demmel and Hida algorithm (Algorithm 2) and the naive algorithm. Let us take Dataset1 as an example: we remark that the absolute errors of the naive summation algorithm are much larger than those of our summation algorithm and those of Demmel and Hida algorithm as shown in Fig. 8a. To better illustrate, let us consider the value 30000 of the x-axis. We notice that the absolute errors are in the order of 104 , 101 and 100 for the naive algorithm, Algorithm 4 and Algorithm 2, respectively. On the other hand, Fig. 8b shows that the summation results of Demmel and Hida algorithm are more accurate compared to our results due to the sort performed on their values before summations unlike our algorithm where no sorting is performed.

Fig. 8. Absolute errors between results of summations for the first data set using three algorithms: Algorithm 4, Algorithm 2 and the Naive algorithm in single precision with the results of summations of the same data set using Naive algorithm in double precision (Left), zoom on the absolute errors of our summation algorithms (Right).

The second experimentation measures the execution time (in seconds) for each algorithm. By this experiment we want to show the compromise between accuracy and running time of our algorithm (Algorithm 4). Figure 9 displays the time in seconds taken by each program (our algorithm, Demmel and Hida algorithm and the naive algorithm) to compute the result of summations of Dataset1. The results show that Demmel and Hida algorithm needs more time to compute this summations. Besides, our algorithm requires much less time than the Demmel and Hida algorithm and a little more than the naive algorithm.

380

F. Benmouhoub et al.

As an example: for summing data set equal to 10000, it takes only 0.0001s and 0.001s using naive algorithm and Algorithm 4, respectively, while this summation requires 7s for Algorithm 2. Thus, we conclude from the experiments shown in Fig. 8 that the summations computed by our algorithm (Algorithm 4) are more accurate that those of the naive algorithm even though it runs faster than our algorithm. Concerning the comparison with Demmel and Hida algorithm, their summations are more accurate because of the sorting performed on the summands before computations but their major drawback is the computations which take longer (171s for data set equal to 50000) to be performed than our algorithm (0.002s for the same data set equal to 50000).

Fig. 9. Execution time of the Naive algorithm, our sequential algorithm and Demmel and Hida algorithm.

5

Conclusion

In this article, we addressed the issues of accuracy and reproducibility of summation results of n floating-point numbers. We have described our summation algorithms and also specifying the difference between our algorithms and those proposed by Demmel and Hida [5]. Our algorithms are tested on eight data sets of different proportions of large values among small and medium values of positive sign or both signs, they perform all the computations in the working precision without need to increase the precision. The results obtained show the efficiency of our algorithms on the improvement of numerical accuracy and reproducibility of summation results, they also show that our algorithms do not increase the cost of complexity. The improvement was measured by comparing our summation results with those of the naive algorithm. An interesting perspective consists in

Parallel Accurate and Reproducible Summation

381

testing these algorithms on realistic problems arising in mechanics, as an example. We would like also to study the impact of the accuracy improvement on the acceleration of the convergence of iterative methods. Acknowledgments. This work was supported by a regional funding (Region Occitanie) and partially by project ANR-17-CE25-0018 FEANICSES.

References 1. ANSI/IEEE. IEEE Standard for Binary Floating-Point Arithmetic. SIAM (2008) 2. Bohlender, G.: Floating-point computation of functions with maximum accuracy. IEEE Trans. Comput. 26(7), 621–632 (1977) 3. De Luca, P., Galletti, A., Giunta, G., Marcellino, L., Raei, M.: Performance analysis of a multicore implementation for solving a two-dimensional inverse anomalous diffusion problem. In: Sergeyev, Y.D., Kvasov, D.E. (eds.) NUMTA 2019. LNCS, vol. 11973, pp. 109–121. Springer, Cham (2020). https://doi.org/10.1007/978-3030-39081-5 11 4. Demmel, J., Hida, Y.: Accurate floating point summation (2002) 5. Demmel, J., Hida, Y.: Accurate and efficient floating point summation. SIAM J. Sci. Comput. 25(4), 1214–1248 (2003) 6. Demmel, J., Nguyen, H.D.: Parallel reproducible summation. IEEE Trans. Comput. 64(7), 2060–2070 (2015) 7. Goldberg, D.: What every computer scientist should know about floating-point arithmetic. ACM Comput. Surv. 23(1), 5–48 (1991) 8. Graillat, S., Langlois, P., Louvet, N.: Algorithms for accurate, validated and fast polynomial evaluation. Jpn. J. Ind. Appl. Math. 26(2–3), 191–214 (2009) 9. Graillat, S., M´enissier-Morain, V.: Compensated Horner scheme in complex floating point arithmetic. In: Proceedings. 8th Conference on Real Numbers and Computers, pp. 133–146. Santiago de Compostela, Spain (2008) 10. Higham, N.: The accuracy of floating point summation. SIAM J. Sci. Comput. 14(4), 783–799 (1993) 11. Higham, N.: Accuracy and Stability of Numerical Algorithms. Society for Industrial and Applied Mathematics (1996) 12. Jeannerod, C.-P., Rump, S.: On relative errors of floating-point operations: optimal bounds and applications. Math. Comput. 87, 01 (2014) 13. Kahan, W.: A survey of error analysis. In: IFIP Congress (1971) 14. Langlois, P., Martel, M., Th´evenoux, L.: Accuracy versus time: a case study with summation algorithms. In: Proceedings of the 4th International Workshop on Parallel and Symbolic Computation, PASCO 2010, pp. 121–130. Association for Computing Machinery, New York (2010) 15. Leuprecht, H., Oberaigner, W.: Parallel algorithms for the rounding exact summation of floating point numbers. Computing 28(2), 89–104 (1982) 16. Malcolm, M.: On accurate floating-point summation. Commun. ACM 14(11), 731– 736 (1971) 17. Muller, J.M., et al.: Handbook of Floating-Point Arithmetic, Birkh¨ auser (2010) 18. Ogita, T., Rump, S., Oishi, S.: Accurate sum and dot product. SIAM J. Sci. Comput. 26(6), 1955–1988 (2005) 19. Pichat, M.: Correction d’une somme en arithmetique a virgule flottante. Numer. Math. 19(5), 400–406 (1972)

382

F. Benmouhoub et al.

20. Rump, S.: Ultimately fast accurate summation. SIAM J. Sci. Comput. 31(5), 3466– 3502 (2009) 21. Rump, S., Ogita, T.: Fast high precision summation. Nonlinear Theor. Its Appl. IEICE 1, 01 (2010) 22. Rump, S., Ogita, T., Oishi, S.: Accurate floating-point summation part I: faithful rounding. SIAM J. Sci. Compu. 31(1), 189–224 (2008) 23. Th´evenoux, L., Langlois, P., Martel, M.: Automatic source-to-source error compensation of floating-point programs: code synthesis to optimize accuracy and time. Concurrency Comput. Pract. Experience 29(7), e3953 (2017)

Cell Assembly-based Task Analysis (CAbTA) Dan Diaper1(B) and Chris Huyck2 1 DDD SYSTEMS, 26 Street Marks Road, Bournemouth BH11 8SZ, UK

[email protected]

2 Middlesex University, The Burroughs, Hendon, London NW4 4BT, UK

[email protected]

Abstract. Based on an Artificial Neural Network model, Cell Assembly-based Task Analysis is a new method that outputs a task performance model composed of integrated mind-brain Cell Assemblies, which are currently believed to be the most plausible, general organisation of the brain and how it supports mental operations. A simplified model of Cell Assemblies and their cognitive architecture is described and then used in the method. A brief sub-task is analysed. The method’s utility to research in Artificial Intelligence, neuroscience and cognitive psychology is discussed and the possibility of a General Theory suggested. Keywords: Human-centered computing · Artificial intelligence · Neural networks · Cell assemblies · Task analysis

1 Introduction A Cell Assembly (CA) is a collection of cooperating neurons and such neurons may be biological or artificial. CAs can represent both the content and the processes of thought in humans and in Artificial Intelligences (AIs). In contrast to the traditional separation of mind and brain, or hardware and software, CAs provide a single description of the physical and the informational. There is a long history of ideas in computing influencing psychological theories [1] and as an example of this a new Task Analysis (TA) method has been developed. It describes human task performance in terms of a simplified model of the brain CAs involved. These CAs are based on an Artificial Neural Network (ANN) model. The new Cell Assembly-based Task Analysis (CAbTA) method exploits TA’s practical, engineering approach to modelling human cognition. Overall, the approach is similar to that of GOMS – Goals, Operations, Methods and Selection rules [2, 3] – method’s implementation of its output as symbolic AI production systems such as ACT-R [4, 5] and EPIC [6]. Similarly, CAbTA can describe CA-based ANN task performance in a sophisticated, consistent manner that can be directly related to an AI’s CAs and their architecture at lower, operational levels, thus enhancing ANN design, implementation and testing.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 383–400, 2022. https://doi.org/10.1007/978-3-030-80119-9_22

384

D. Diaper and C. Huyck

1.1 Cell Assemblies and Brains The human brain has around 1011 neurons [7] and there is considerable evidence, summarised by [8], that much of the brain is CA-based. There is a tight binding between the physical properties of a CA and its functional, information processing ones. Although there are cortical areas that have specialised functions, defining exact functionality is often difficult because this depends on the tested task range. Such functionality problems remain at whatever physiological level of detail studied. Predating Newell’s [9] work on cognitive architecture, Hebb’s 1949 theory [10] is that each concept is represented by a collection of firing neurons. Hebb’s theory has been considerably extended [8], particularly with the development of CA-based ANNs. Hebb’s concepts now represent Long Term Memory (LTM) and active mental content and processes, e.g. calculation [11], which CAs naturally represent as they must change over time. CAs can also modify processes by providing pre-ignition structure. Powerful cognition in less neurally endowed species may involve as few as two hundred neurons [12]. In humans the size range of CAs is suggested as 103 to 107 neurons, although upper estimates may be “super-CAs composed of many sub-CAs” [8]. Most neurons, at different times, will have membership of different CAs, although type may be restricted, e.g. a neuron in the primary visual cortex might always be involved with visual processing. Once ignited, CAs can persist for longer than it takes for a neuron to fatigue. Therefore, for CA persistence of several seconds or more there must be a pool of non-firing neurons that can replace fatiguing neurons (see PotN, Sect. 2.1). With long term CA persistence, a neuron might fire repeatedly, fatigue, recover, and then fire repeatedly again. The QPID Model. A simple model is that CAs can exist in four states: Quiescent, Priming, Ignited, and Decaying (Q, P, I and D, respectively, pronounced “Cupid”). While physiologically similar, functionally the four states differ significantly: Q-state CAs are structured for permanent storage; P-states prepare a CA for ignition and support cooperative and competitive attentional mechanisms; I-states are active CAs; and Dstates prepare CAs for their LTM Q-state. The reality is complicated, e.g. P-state CA structure will change as priming approaches ignition; or D-state decay may return to some notional resting state, or remain primed for re-ignition, or suffer a refractory period where re-ignition is harder. In QPID, a new Q-state is never quite the same as its precursor; the notionally same CA will differ in the set of neurons involved in each ignition, and neurons themselves change over time, e.g. synaptic growth. Thus, the CA’s functional definition must be at a sufficiently high level of description that such differences can be ignored. Some concept of levels of detail is common in many areas. The super-CA/sub-CA proposal and QPID model fits well with TA’s extensive use of levels. Cell Assemblies and Artificial Neural Networks. CA-based ANNs use simplified neuron models that mimic brain operation. Most are self-organizing, i.e. they can learn. Anderson [5] discusses learning in detail but ACT-R only models human learning like any other task. There is, however, a complete difference between modelling how people learn and the inherent, self-directed, internal learning that is typical in ANNs.

Cell Assembly-based Task Analysis (CAbTA)

385

The Cell Assembly roBots (CABots) demonstrate both new object and environmental learning, and they can solve problems, all without human programmer intervention [13, 14]. They are the starting point for CAbTA’s CA models. CA-based ANNs suffer the same definition problems as biological CAs as once running, and learning, CA functionality is difficult to infer, even though the whole system can be inspected. Software functionality in symbolic AIs is well understood as these use traditional programming methods. Unfortunately, symbolic AIs lack any operational level mapping between brains and computers. A variety of neuron models are available that run on CA-based simulators, e.g. the CABots used spiking neurons, some with adaptation [15]. While smaller than human brains, Furber, et al. [16] used neuromorphic hardware (SpiNNaker) to simulate, in real-time, billion neuron CA-based models. Crucially, even simple CA-based systems have been proved to be Turing machines, given enough neurons, i.e. they can compute any legal mathematical or logical expression [17]. Therefore, anything written using traditional programming, including symbolic AI code, can be done using simulated neuron-based CAs. Currently, run-time efficiency remains a problem, but performance should continue to improve, although Huyck’s CABots already run in real time on a PC. The existing computational modelling work, e.g. [14], and computational theory [17] shows that CAs are a solution to all computable problems, and a reasonably simple implementation to many problems. Thus a CA-based TA could be implemented computationally reasonably easily. 1.2 Task Analysis and Symbolic Artificial Intelligence GOMS output resembles program pseudo-code so is easy to implement as symbolic AI production systems like ACT-R and EPIC. Anderson and Lebiere [4] say such systems “are the only modelling formalism capable of spanning a broad range of tasks, dealing with complex cognition …”, and ACT-R has “a profound sense of psychological reality”; Anderson [5] sees EPIC as an ACT-R 6.0 precursor, contributing “Perceptual-Motor” modules. EPIC’s developers’ claims are more cautious [6, 18]. Anderson [5] defines cognitive architecture as a specification of how “the brain … achieves the function of the mind.” Problematically, symbolic AI systems’ program code makes no attempt to mimic the brain, although Anderson [5] attempts, post hoc, some relationships to brain areas. The theoretical issue concerns simulation fidelity, how well one thing, a symbolic AI, can mimic another, the brain, when operationally they are completely different. The Achilles’ Heel of symbolic AIs is their reliance on human programmers. If a GOMS model changes, then its symbolic AI equivalent must be reprogrammed. While ACT-R, and other cognitive architectures like Soar [19], can learn, these typically work by parameter setting or generating new rules from old ones. Unlike CA-based ANNs, they are not capable of symbol grounding [20]. There are hundreds of TA methods and all have a cognitive, psychological component. As Kieras [3] notes, “a task analysis for system design must be rather more informal and primarily heuristic in flavour compared to scientific research.” Using Card, Moran

386

D. Diaper and C. Huyck

and Newell’s [2] cognitive psychology, GOMS is a psychologically sophisticated TA method, yet is easily criticised scientifically. The basic engineering argument in GOMS, and generally in TA, is that some cognitive representations and processes similar to those identified during an analysis must occur. For example, a TA might identify using Short Term Memory (STM), but whether this is Miller [21], or Baddeley and Hitch [22, 23], or involves other temporary stores [24] is moot. Similarly, TAs will identify when decisions are made, but the decision making mechanisms are left unspecified. GOMS is good at predicting human performance, although Kieras [3, 25] is cautious, and there are exceptions [26]. 1.3 Cell Assembly-based Task Analysis The models of behavioural causality in the Square of Identity (Fig. 1) possess two types of identity: (i_1) horizontally, where AI implementation is identical with biological specification; and (i_2) vertically, where cognitive, functional representations and their operational execution have traditional Identity Theory identity [27, 28]; or Feigl [29] in 1958: that mental events “are identical with certain (presumably configurational) aspects of the neural processes”.

Fig. 1. The square of identity: (A) main figure; (B) with interdisciplinary and cross-level identity functions; and (c) the general theory possibility.

In the top half of Fig. 1A, GOMS output pseudo-code is a mental model of task performance with i_1 type identity with symbolic AIs (e.g. ACT-R, EPIC). Programmers must interpret and convert GOMS requirement specifications into symbolic AI ones, so i_1 type identity will not be perfect. Even a compiler like automation would suffer the logical compilation deduction cheat: producing one legal program does not preclude other legal ones. Since, with enough neurons, a CA-based ANN is a proved Turing machine, then any GOMS-based symbolic AI could be re-implemented using CAs, with similar i_1 caveats at the symbolic level.

Cell Assembly-based Task Analysis (CAbTA)

387

The CAbTA approach covers all of Fig. 1A. Starting with an operational level CAbased ANN model, this was used to design the mental, symbolic outputs of the task analysed. This could be used to build a CA-based symbolic AI with true identity (i_2) with its lower, operational level as there is only one CA, described at different levels. CAbTA’s task models enjoy similar mind-brain i_2 identity. Accepting i_1 identity limitations, Fig. 1B illustrates the logical identities that allow interdisciplinary (biology versus computing) and inter-level (symbolic versus operational) reasoning. Of the figure’s six identity functions, one example could use a simplified ANN-based mental model to reason about brain functions that for human psychology are unknown or too complicated to express. Similarly, as was done developing CAbTA, an operational level CA-based ANN model drove the symbolic level psychological model, which then allowed generation of the task’s operational level brain model. Figure 1C illustrates the General Theory possibility. This would provide description, explanation and models of the same type for human and artificial minds and their operations, because all would be CA-based. The Square of Identity offers no protection from fault, in modelling or implementation. As engineering, TA makes no claims to meeting experimental cognitive psychology’s scientific requirements. TA’s practical approach, however, encapsulates scientifically known psychology, applying such to real task performance. CAbTA offers similar analytical power to GOMS, with similar symbolic level opportunities to GOMS’ relationships to ACT-R and EPIC; but suffering the same problem, to be practical, i.e. good enough for engineering. GOMS, while expensive, is good at modelling task performance and has good predictive adequacy. The same can be expected of CAbTA.

2 The Cell Assembly-based Task Analysis Models CAbTA has two models of task performance. The first models brain CAs and the second relationships between these CAs. 2.1 The Simplified Cell Assembly Model (SCAM) The standard CA graphical representation plots number of neurons firing against time [8, 30]. Using this with the QPID model, a Simplified Cell Assembly Model (SCAM) is defined by eight parameters: four relating to time and four to number of neurons firing (Fig. 2).

Fig. 2. The simplified CA model (SCAM) diagram.

388

D. Diaper and C. Huyck

The four SCAM parameters associated with number of neurons are: PotN – potential number of neurons that could have CA membership; Thresh – threshold where there are sufficient neurons firing for CA ignition; IgMax – maximum number of firing neurons at CA ignition; IgFat – fatigue as number of firing neurons at CA ignition extinction. The four SCAM parameters associated with time are: P50% - time when a CA is primed to 50% of the required neurons firing to reach ignition threshold; IgTIg – ignited CA’s time of ignition; IgTEx – ignited CA’s time of extinction; D50% – time when a CA decays to 50% of the neurons firing at CA extinction (IgFat). Each CA identified has an entry in the SCAM Table defined by these eight parameters. 2.2 The Cell Assembly Architecture Relationship (CAAR) Diagram In the CAAR Diagram, each CA identified is represented by a box and the relationships between CAs by arrows. The latter are complex and analysts must consider: how CAs interact; what information is transferred; and CA priming and decay. Horizontally, the CAAR Diagram separates types of CA: Perceptual; Cognitive; and Motor. Within graphical constraints, sequence and time are represented vertically, increasing downwards. The CAAR Diagram is “architectural”; it shows the changing structure between ignited CAs. This paper’s CAbTA used a default cognitive architecture of Cognitive CA chains (Fig. 3).

Fig. 3. The default CAAR diagram.

CAbTA is not architecturally prescriptive and this paper’s analysis found alternatives, for example, when expert performance binds Perceptual and Motor CAs. 2.3 The Main Analysis Document (MAD) Each CA has a standardised entry in the Main Analysis Document (MAD), Fig. 4. The free text description is created during analysis and eventually the SCAM Table and

Cell Assembly-based Task Analysis (CAbTA)

389

Diagram are added. The CAAR Diagram is derived from the INPUTS/OUTPUTS list. Punctuation indicates time, from none meaning simultaneity and increasing separation using commas, semicolons, and colons; a full stop indicates no overlap in time.

Fig. 4. Each CA’s MAD style.

3 Method An overview is provided of the kettle filling task, data collection, and analysis. 3.1 The Task The task analysed involves the task performer entering his home’s kitchen, moving to the hot water preparation area and then lifting the kettle and moving it to the sink for emptying and refilling. This highly familiar task allows focus on the new TA method. The high level Activity List, see [31], in Fig. 5 summarises the task. This task’s low level CAbTA demonstrates that the cognitive requirements are considerable. The right hand must weave a complex path to the kettle handle, avoiding objects, using overhead visual information (arrow in Fig. 5); when lifting the kettle , its weight must be estimated kinesthetically for effective and efficient kettle location, velocity and acceleration over the drainer . Except when the drainer is empty, each of the kettle’s flights over it is different; the largest pan lid rises 30 cm above drainer base height, 60% above kettle height.

390

D. Diaper and C. Huyck

To emphasise the task’s cognitive complexity, programming a robot using traditional techniques, including symbolic AI, to do what non-psychologists would consider a simple task, would be difficult and expensive because of the variability in the task environment. Being more human-like, ANNs should provide a better robotic solution and CAbTA can provide a task specification that would allow robotic behavior to emulate human task performance.

Fig. 5. ‘The Task’ as a high level activity list.

3.2 Data Collection Data was collected using a repeated trials, self-observation, heuristic approach, i.e. the first author, a TA expert (hence “heuristic”), observed himself more than thirty times over a month, making notes after each trial to create a far more detailed Activity List than the summary in Fig. 5. Some trials recorded time at two task points (see Sect. 4.1). Even after more than two decades of domestic task performance, new things were learned during the CAbTA research. For example, the “approach” to the kettle’s location was found to be invariant, starting with the right foot mid-doorway and taking three strides, and a right foot shuffle. Outside the kitchen was discovered a “shuffle zone” where, from any of four directions, footsteps were adjusted for mid-doorway launch. As another example, the left hand was not adequately considered initially. Careful observation identified complex ballistic, and then visual negative feedback controlled, behaviours tracking the kettle’s lid to interception near the black tile in Fig. 5. Furthermore, the approach behaviour was observed and analysed for the other three house residents, i.e. with a more traditional separation between task actor and observer. Of the two long term residents, one, about the same height as the main subject, took the same three strides, launching on the right foot; the shorter one launched right-footed, but took five steps. In contrast, the third, an athletic man in his early thirties, who had only been in the house half a year, never repeated his approach and often was off target by a foot or more.

Cell Assembly-based Task Analysis (CAbTA)

391

3.3 CAbTA Method Expert task analysts will recognise the basic process of inferring mental states and processes from behavioural data, including inferences from the unobserved, such as estimating the kettle’s weight. This is what makes TA different from mere performance observation, and what makes is so useful in engineering design and evaluation, although the cost is that the quality of a TA is dependent on the psychological expertise of task analysts. CAbTA’s representation of such mental states and processes is no more complicated or difficult from the representations used in any sophisticated TA method. The basic CAbTA analysis cycle starts by identifying a small set of related CAs associated with an Activity List line; some of these may have been suggested as outputs from already analysed CAs. Given a mnemonic name and an acronym, analysis begins with specifying CA type: Cognitive; Perceptual; or Motor. New entries for each CA are started in the MAD (Sect. 2.3) and previously analysed CA inputs may be linked. Each CA has a new line in the SCAM Table, which, when completed, is copied under the MAD’s standard table header. The MAD’s descriptive text is started by specifying each CA’s function and the inputs that will ignite it. The text then records the issues considered and the reasons for the parameter values added to the SCAM table. For an expert, duration parameter estimation is relatively easy, although for priming (P50%) the preceding CAs need considering. In contrast, those relating to number of neurons must be based on a CA-based brain model. Analyst’s decision consistency is more important than the assigned values as these can be later modified, e.g. by multiplying by appropriate constants. Once a SCAM table line is completed, then that CA’s SCAM diagram can be drawn and inserted into the MAD. New inputs between the CAs being analysed, and their outputs, between them and to putative subsequent CAs need determining. The information contained in the inputs and outputs needs specifying and for longer persisting CAs when transmissions occur. There may be cycles of CAs affecting each other, which may be excitatory or inhibitory, and transmitted information may change during cycles. Textual descriptions and the tabulated CA inputs and outputs are added to the MAD. Next, the newly analysed CAs are added to the CAAR Diagram. Within graphic design limitations, this step should be automatic as all the information is already in the MAD. Experience suggests that adding to the CAAR Diagram will necessitate iteration of previous steps. Such cyclical changes are typical in TA and are exploited within CAbTA as part of its extensive cross checking procedures, essential for any manually performed TA, i.e. without specialised software support (Sect. 5.1).

4 Results The main result is the demonstration that a CAbTA is possible. The MAD describes each CA; the SCAM Table (Table 1) and CAAR Diagram (Fig. 8) summarise the analysis, and the results are described in more detail in Sects. 4.2 and 4.3, respectively.

392

D. Diaper and C. Huyck Table 1. The SCAM table for the kettle filling task.

4.1 Time Results From the kitchen entrance, timing data was collected from two steps in the task: (i) at kettle handle grip (CA 23 – MRHH); and (ii) at the end, filling the kettle (CA 64 – MPTSU): these were averaged at 4.1 and 9 seconds, respectively.

Cell Assembly-based Task Analysis (CAbTA)

393

Generally, time data is less important than sequence in most TAs. The time data was problematic in this study because, even though a highly practiced task, if the task was immediately repeated, then times tended to become shorter. Such repeated trials data was not included in the averages quoted above. 4.2 SCAM Results Sixty four CAs were identified: 34.4% (22/64) cognitive; 39.1% (25/64) perceptual; and 26.6% (17/64) motor. Perceptual CAs were 80% (20/25) visual; the others were: 12% (3/25) touch and 8% (2/25) kinaesthetic and were not analysed further. Ignoring those CAs still ignited at the end of the analysis, 48 CAs were analysed (Table 2). Table 2. Average data for the 8 SCAM parameters. CA Type

PotN

Thresh

IgMax

IgFat

P50%

IgTIg

IgTEx

D50%

All

11.1

3.3

6.8

5.7

4.5

4.7

5.4

5.5

Cognitive

10.4

2.8

6.6

5.2

3.8

4.1

5.0

5.2

Visual

14.4

5.0

8.6

7.5

4.5

4.7

5.3

5.4

Motor

9.5

2.6

5.9

4.9

5.3

5.4

6.1

6.2

Fig. 6 shows the average SCAM diagrams.

Fig. 6. Averaged SCAM diagrams: (A) All; (B) Cognitive; (C) Visual; and (D) Motor.

394

D. Diaper and C. Huyck

The differences in Fig. 6(b)–(d) shows that the CA brain model and cognitive architecture appear to have been applied appropriately and consistently, e.g. neuron numbers (PotN) are highest for the large visual cortex (Table 3a). Table 3a. Difference in PotN means. (/ represents vertical parameter divided by horizontal one.) /All Visual/ Cognitive/ Motor/

/Cognitive /Motor

29.7% 38.5%

51.6%

−6.4%

–

9.5%

−14.4%

–

–

Similar, consistent results can be seen for Threshold and IgMax (Tables 3b and 3c). Table 3b. Difference in threshold means. /All Visual/

/Cognitive /Motor

51.6% 78.6%

92.3%

Cognitive/ −15.1%

–

7.7%

−21.2%

–

–

Motor/

Table 3c. Difference in IgMax means. /All

/Cognitive /Motor

Visual/

26.5% 30.3%

Cognitive/

−2.9%

–

1.1%

−13.2%

–

–

Motor/

45.8%

Further, similar results were found for Fatigue (IgMax–IgFat), which overall was 16.2%: Cognitive 21.2%; Visual 12.8%; and Motor 17.0%.

Cell Assembly-based Task Analysis (CAbTA)

395

Ignition Duration (IgTEx–IgTIg) analysis (Fig. 7) suggests two CA types: short (1 s). This result was not anticipated, but it is hardly surprising and is explicable in terms of brief CA chains versus complex CAs persisting.

Fig. 7. CA ignition durations as line graphs and histograms.

4.3 CAAR Results The CAAR Diagram is shown in Fig. 8. Ignoring touch and kinaesthetic CAs (5), inhibitory relationships (7), and external motor outputs, there remained 89 relationships in the CAAR Diagram, as summarised graphically in Fig. 9. The default CAAR model (Sect. 2.2) was adhered to, although the “basic chain” had some branches (18 Cognitive CAs, 26 outputs). The intended, tight binding between cognitive/visual CAs (N = 16) was observed (20 versus 21); only 5% (4/89) were visual/motor relationships. With one exception, all 14 motor CAs had cognitive inputs (N = 17), although many motor CAs cause behavior that then provides sensory inputs back into the system.

396

D. Diaper and C. Huyck

Fig. 8. The CAAR diagram for the kettle filling task.

Cell Assembly-based Task Analysis (CAbTA)

397

Fig. 9. CA relationships results.

5 Discussion While GOMS facilitates implementation of symbolic level AIs, CAbTA completes the Square of Identity (Sect. 1.3) at both symbolic and operational levels. This creates opportunities in AI, neuroscience and cognitive psychology for new intradisciplinary and interdisciplinary research. 5.1 Cell Assembly-based Task Analysis Developing a TA method that produces putative cognitive/brain CAs is a major success. The initial analysis required repeated iterations as CAbTA was developed from scratch, e.g. a dozen different SCAM Diagram graphics were tried, and SCAM parameter redesign also changed these diagrams. The final two thirds of the analysis went smoothly and iteration involved changing CAs but not the method, i.e. in a manner similar to, and no more difficult than, any other sophisticated TA method. The highly structured MAD was the core representation and contains the entire analysis; the SCAM Table and CAAR Diagram were generated from it, although were invaluable for reasoning about CAs. Checking and Software Tools. Psychologically sophisticated TA methods are complicated to do and, when done manually, typographical and transcription errors are common. From the outset, CAbTA was designed with testing mechanisms and these proved necessary, even though copy-and-paste was used whenever possible. There is a necessity for specialised software to support TA methods [32]. Such tools will make TAs easier, faster and less error prone and are likely to change the method. Tool programming can identify poor method specification, often where analysts use craft skills. Also, software can facilitate things impractical in manual analyses, e.g. a CAAR Diagram improvement might provide cycle information, e.g. using multiple arrowheads for the cycles observed between cognitive and perceptual CAs; or, automating SCAM Diagram drawing could easily incorporate desirable, non-linear P50% and D50% parameters.

398

D. Diaper and C. Huyck

The software engineering temptation is to automate CAbTA’s manual method, i.e. tables and diagrams being produced from a MAD editor. More creative approaches could make a SCAM Diagram drawing package central, advantageously reducing analysts’ inputs. Levels. Sixty four CAs may seem a lot of analysis for nine seconds but other sophisticated TAs would probably produce similar amounts of detail. Possibly all CA-based TAs will be low level and therefore be unsuitable for long duration tasks, although many safety critical subtasks need detailed, if expensive, analysis, sometimes within other TA methods. A CA library might allow meta-level task description with only occasional descent to lower levels. A CA meta-architecture needs developing and specifying. This might include distant brain areas, but mostly localised ones, e.g. sets of ignited CAs might initially just share neurons but over time become a super-CA with reliable lower level co-ignition, at least when there is not super-CA competition. Such a model hypothesises a tighter binding between CAs than that of just interacting with each other. In TA, the levels concept is ubiquitous so development would focus on brain and CA-based AI meta-architectures. Some cognitive psychology literature, e.g. on attention, may need redrafting to better fit CA level analyses. 5.2 Artificial Intelligence, Neuroscience and Psychology The Square of Identity (Fig. 1) summarises the integrated modelling of biological minds and brains and AI ones. CAs conflate descriptions of functional, symbolic levels and lower, operational ones as i_2 type identity. Admittedly i_1 biological and AI systems identity is weaker because of alternatives. As a General Theory, a CA-based perspective should profoundly influence AI, neuroscience and cognitive psychology models and theories. Existing experimental cognitive psychology and neuroscience could be reexpressed as similar CA-based systems and redirect future research towards better CA specifications and cognitive architectures for both human and machine intelligences. Modelled on ignited CAs, which although ephemeral, can be studied in minds, brains and machines, the QPID model demonstrates how little is known about the Q, P and D states, e.g. how sub-ignition threshold CAs cooperate and compete. If future AIs are CA-based, then CAbTA can facilitate systems development by suggesting: CA types needed, their properties and operations over time, how they interact, and their cognitive architecture. ANNs can test alternative CAs and architectures and the results fed back to cognitive and neuroscience research. Existing working CA-based systems give confidence in the ability to make CAbTA model simulations from spiking neurons, and would support further exploration of simulated CA behaviour [33].

6 Conclusions CAbTA’s theoretical core is the exploitation of TA as an engineering discipline, i.e. using the standard TA defence that even if not quite right, similar mental operations to those identified must occur during task performance. The consequences are manifold.

Cell Assembly-based Task Analysis (CAbTA)

399

A General Theory that integrates brain and mind in humans and machines is within the approach’s scope. AI, neuroscience and cognitive psychology can all benefit from adopting a CA perspective and CA-based theories and models. Exponential benefits should accrue to interdisciplinary collaborations where the CA view provides a common language, classes of models, and epistemology. The CAbTA research reported here is novel but can only be seen as a first step to the development of Cell Assembly-based Task Analysis methods. The authors are looking to recruit resources, human and financial, to support further development.

References 1. Diaper, D.: Designing expert systems: from Dan to Beersheba. In: Diaper, D. (ed.) Knowledge Elicitation: Principles, Techniques and Applications, pp. 15–46. Ellis Horwood (1989) 2. Card, S., Moran, T., Newell, A.: The Psychology of Human-Computer Interaction. Lawrence Erlbaum, Hillsdale (1983) 3. Kieras, D.E.: GOMS models for task analysis. In: Diaper, D., Stanton, N.A. (eds.) The Handbook of Task Analysis, pp. 83–116. Lawrence Erlbaum Associates (2004) 4. Anderson, J.R., Lebiere, C.: The Atomic Components of Thought. Lawrence Erlbaum, London (1998) 5. Anderson, J.R.: How Can the Human Mind Occur in the Physical Universe? Oxford University Press, Oxford (2007) 6. Meyer, D.E., Kieras, D.E.: A computational theory of executive cognitive processes and multiple-task performance: Part I basic mechanisms. Psychol. Rev. 104, 3–65 (1997) 7. Smith, K.: Settling the great glia debate. Nature 468, 160–162 (2010) 8. Huyck, C., Passmore, P.: A review of cell assemblies. Biol. Cybern., 263–288 (2013) 9. Newell, A.: Unified Theories of Cognition. Harvard University Press (1990) 10. Hebb, D.: The organization of behavior: a neuropsychological theory. J. Wiley and Sons (1949) 11. Tetzlaff, C., Dasgupta, S., Kluvicius, T., Worgotter, F.: The use of hebbian cell assemblies for nonlinear computation. Sci. Rep., 12866 (2015) 12. Buzsáki, G.: Neural syntax: cell assemblies, synapsembles, and readers. Neuron 68(3), 362– 385 (2010) 13. Huyck, C., et al.: CABot3: a simulated neural games agent. In: 7th International Workshop on Neural Symbolic Learning and Reasoning NeSYS 2011 (2011) 14. Huyck, C., Mitchell, I.: CABots and other neural agents. Front. Neurorob. 12, 79 (2018) 15. Huyck, C., Parvizi, A.: Parameter values and fatigue mechanisms for FLIF neurons. J. Syst. Cybern. Inf., 80–86 (2012) 16. Furber, S., et al.: Overview of the SpiNNaker system architecture. IEEE Comput. Soc., 2454– 2467 (2013) 17. Byrne, E., Huyck, C.: Processing with cell assemblies. Neurocomputing, 76–83 (2010) 18. Kieras, D.E., Meyer, D.E.: An overview of the EPIC architecture for cognition and performance with application to human-computer interaction. Hum. Comput. Interact. 12, 391–438 (1994) 19. Laird, J., Newell, A., Rosenbloom, P.: Soar: an architecture for general cognition. Artif. Intell. 33(1), 1–64 (1987) 20. Harnad, S.: The symbol grounding problem. Physica D 42, 335–346 (1990) 21. Miller, G.: The magical number seven plus or minus two: some limits on our capacity for processing information. Psychol. Rev. 63, 81–97 (1956)

400

D. Diaper and C. Huyck

22. Ardila, A.: Working memory. In: Heredia, Roberto R., Altarriba, Jeanette (eds.) Foundations of Bilingual Memory, pp. 223–234. Springer, New York (2014). https://doi.org/10.1007/9781-4614-9218-4_11 23. Oberauer, K., et al.: Benchmarks for models of short term and working memory. Psychol. Bull. 144(9), 885-958 (2018) 24. Morey, C.: The case against specialized visual-spatial short-term memory. Psychol. Bull. 144(8), 849–883 (2018) 25. Kieras, D.E., Butler, K.A.: Task analysis and the design of functionality. In: Computing Handbook: Information Systems and Information Technology, 3rd ed., Ch. 33. Taylor & Francis (2014) 26. Jorritsma, W., Haga, P.-J., Cnossen, F., Dierckx, R.A., Oudkerk, M., van Oojen, P.M.A.: Predicting human performance differences on multiple interface alternatives: KLM, GOMS and CogTool are unreliable. Procedia Manuf. 3, 3725–3731 (2015) 27. Smart, J.J.C.: The mind/brain identity theory. In: Stanford Encyclopedia of Philosophy (2007) 28. Ullin, T.P.: Is Consciousness a brain process? Br. J. Psychol. 47, 44–50 (1956) 29. Feigl, H.: The ‘Mental’ and the ‘Physical’. In: Concepts, Theories, and the Mind-Body Problem. University of Minnesota Press (1958) 30. Kaplan, S., Sontag, M., Chown, E.: Tracing recurrent activity in cognitive elements (trace): a model of temporal dynamics in a cell assembly. Connection Sci. 3, 179–206 (1991) 31. Diaper, D., Stanton, N.A. (eds.): The Handbook of Task Analysis for Human-Computer Interaction. Lawrence Erlbaum Associates (2004) 32. Diaper, D.: Task Analysis for Knowledge Descriptions (TAKD): a requiem for a method. Behav. Inf. Technol. 20(3), 199–212 (2001) 33. Carillo-Reid, L., Yuste, R.: What is a neuronal ensemble? In: The Oxford Research Encyclopedia of Neuroscience. Oxford University Press (2020)

PDSA Computing Shuichi Fukuda(B) SDM Research Institute, Keio University, 4-1-1, Hiyoshi, Kohoku-ku, Yokohama 223-8526, Japan

Abstract. As materials are getting softer, we need to interact with the object directly to find out what it is and how we should handle it. Thus, movement becomes increasingly important. Human movements are divided into two. One is Motion observed from outside and the other is Motor inside of us. In the case of Motion, our trajectories vary widely at the early stage but when we get close to the object, muscles harden and move together with skeleton. So, we can apply mathematical approaches and control the trajectory. Why trajectories vary widely at the early stage is because we need to mobilize many body parts to balance our body to adapt to the environment and situation. So, coordination is important, but the way to secure coordination is not established yet. Further, muscles are analog, so their degrees of freedom are tremendously large. Therefore, we cannot solve the problem mathematically. We need another way of computing to find out how we should handle the object. Thus, strategic decision making is called for. So Pragmatic computing, which repeats the Plan → Do → Study → Act cycle is needed. This paper proposed quantitative performance evaluation tool using Mahalanobis Distance and pattern. The uniqueness and usefulness of this approach is it allows us focus our attention on any location where we wish to, and it helps us understand how we can improve our movements. This is made possible by making the most of instinct and proprioception, which have been ignored in regular computing. Keywords: Softening materials · Movement · Coordination · Strategic decision

1 Changing World Our world is changing rapidly. 1. Changes were smooth yesterday, so we could differentiate them and predict the future. But today, changes are sharp. So, we cannot differentiate them. We cannot predict the future anymore. 2. Yesterday, it was a closed world with boundary. So, it could apply mathematical approaches and could control things. But now it expanded widely and became an open world without boundary. Therefore, we need to look for other approaches. 3. Materials are getting softer and softer. Up to now, non-living things have been hard, and they do not deform. So, we easily understand what it is with our eyes and how we should manipulate it. But when materials become soft, we need to directly interact © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 401–409, 2022. https://doi.org/10.1007/978-3-030-80119-9_23

402

S. Fukuda

with it to find out what it is and how we should handle it. Thus, haptics is getting wide attention these days. But touch is only surface sensation. We need to move muscles, joints, tendons to cope with soft materials.

2 Discover How It Behaves 2.1 Try Models One After Another When things were hard, we could identify what it is with eyes alone. And we know how to handle them. In other words, we could identify the model with easily. But, when things become soft, we need to interact with them directly in search for models one after another. Yes, we need to practice Plan → Do → Study → Act (PDSA) cycle and repeat the cycle until we find out the appropriate model. And even after identifying what it is, we need to repeat another PDSA cycle to find out how to handle it. It escapes, when we try to grip it. Then, we have to scoop it up. Thus, we need to find out how we should handle them, according to their behaviors at the time of interaction (Fig. 1).

Fig. 1. Plan → Do → Study → Act (PDSA) Cycle

This may be compared to Topological Quantum Computing. In regular computing, we develop algorithm and pay efforts to process numbers or data exactly. But in Topological Quantum Computing, they do not care much about the exactness of numbers or data. What matters is the operation. They call it Operator. What is important is to identify the exact Operator. In dealing with soft things, what we do is to identify the Operator, or it may be better to say in this case Operation. As we cannot move our body always exactly the same way, what we need is to find out the appropriate Operation. 2.2 Sensing and Actuation In sensing and actuation, VAK, i.e., visual, auditory and kinesthetics, are important. In the days of hard things, visual or vision worked very effectively. We could identify the object and we could understand how we should handle it very easily with vision alone. But in the case of soft objects, kinesthetics, i.e., movement becomes crucially important as described above. So, next, let us consider human movement.

PDSA Computing

403

3 Human Movement Human movement is divided into two kinds: One is motion, which is the movement observed from outside. The other is motor, which is the movement inside of us. Nikolai Bernstein clarified human motion [1], (Fig. 2). At the early stage, our motion trajectories vary widely. But when we get close to the object, muscles harden and start to work together with skeleton. So, the number of degrees of freedom is reduced to the minimum and we can identify parameters easily. Thus, we can Control the trajectory. Most of the human movement researches are dealing with the movement at this latter stage. Then, why do our trajectories vary widely at the early stage? It is because we mobilize many body parts and coordinate them to balance our body. Depending on the environments and situations, we coordinate our body to adapt to them. Thus, Coordination is important at the early stage [2]. In other words, the latter stage is related to tactics and problem solving, while the early stage is related to strategy and goal finding. Coordination is very important, but when we deal with soft objects, the number of degrees of freedom increases tremendously. So, due to the curse of dimensionality, we cannot solve the problem mathematically.

Fig. 2. Human Motion

4 Euclidean Space and Non-euclidean Space Regular computing is carried out in Euclidean Space. It requires orthonormality and distance with unit. Euclidean Distance is interval scale. So, it is measured with unit. Euclidean Space approach is rigorously defined mathematically. So, when the number of dimensions is small, it solves the problem beautifully. But most cases in the Real World do not satisfy these constraints. So, we need to look for Non-Euclidean Space approaches.

404

S. Fukuda

4.1 Mahalanobis Distance (MD) P. C. Mahalanobis proposed Mahalanobis Distance [3]. He is a researcher of design of experiments. He proposed MD to remove outliers to produce better results in his design of experiments. So, his original idea is Euclidean. But his MD is ordinal scale. So, if we look at it just as a distance in ordinal scale, we can get a Non-Euclidean distance. In fact, MD just indicates how far away the point P is from the mean of the dataset. And this Non-Euclidean dataset does not have to satisfy orthonormality (Fig. 3). The original MD is defined as. MD = (Standard Deviation)/(Mean of Dataset) But this Non-Euclidean MD is defined as. MD = (The distance of the point P from the mean of dataset).

Fig. 3. Mahalanobis distance

4.2 Mahalanobis-Taguchi System (MTS) Genichi Taguchi proposed Mahalanobis-Taguchi System (MTS) [4]. He is a quality researcher, but he knew very well that in real industry, it is practically impossible to control quality element by element. Most companies are controlling quality holistically. He noted the characteristics of MD and realized that if he use MD and pattern together, then he can answer the needs of companies.

Fig. 4. Defining unit space

PDSA Computing

405

Let us explain his approach by taking the number 2. Number 2 is written differently from person to person. But if we average it, we can get standard pattern, which he calls Unit Space, of number 2 (Fig. 4). And if MD is smaller than the threshold, we can regard the number as number 2. If larger, we cannot (Fig. 5). This system was very much welcomed by companies. MTS, however, is a static pattern matching.

Fig. 5. Mahalanobis-taguchi system (MTS)

5 Static to Dynamic Although MTS was very much useful in practical application, it is static. But our goal here is to manage movement. So, we need to process dynamic patterns. MTS does not use neural network, but deep learning-based AI is progressing rapidly, and its applications are increasing. Among them are image processing. So, patterns can be classified using deep learning-based AI. If we change neural network to recurrent neural network (RNN), which is a neural network expanded over time, (Fig. 6), we can process dynamic patterns in the same way. But RNN assigns weights to links randomly. If we introduce Reservoir Computing (RC), we can make adjustments at the output. As Edge becomes important with objects getting soft, such adjustments at the edge are very useful. Further, RC enables us to introduce micro technologies such as microphotonics, so that we can sense and compute at the edge and put these devices on our body. Then, we can enhance our human capabilities. Robots have been working for us, but humans and robots are separate and robots work just as instructed. Ashton Kevin proposed IoT and emphasized the importance of humans working together with machines. And importance of humans working in the machine system is

406

S. Fukuda

Fig. 6. Recurrent Neural Network (RNN)

emphasized, but in such a machine-human system, humans are regarded as machine elements. The characteristics of human to manage movements is completely forgotten. But if we robots become microscale and we can put them on our body and work together, we can use them as we wish. So real Human Augmentation or Human Enhancement is realized (Fig. 7).

Fig. 7. Reservoir computing (RC)

6 Perception → Motivation → Decision Making → Action → Emotion (PMDAE) Cycle Movement management is Pragmatic, as described above. To describe it in more detail, this Pragmatic cycle can be illustrated as Fig. 8. Perception here means holistic perception. First, we must be situational or context aware. Then, we are motivated and make decision on how we will act. If the action works well, we will be happy. If not, we will repeat the cycle. We should remember that emotion and motivation come from the same Latin word “movere” or move. So, emotion means to move out. We would like to move out to do something. That is emotion. Thus,

PDSA Computing

407

Fig. 8. Perception → Motivation → Decision Making → Action → Emotion (PMDAE) Cycle

it is deeply associated with motivation. What characterizes humans is decision making. We have some desire in our mind and to realize it, we need to make a decision. This decision is strategic. So, we need to proceed by trial and error. In other words, we need to improve our performance every time. Thus, how we evaluate our performance is important.

7 Performance Indicator In the sense of strategic decision, movement management and business management are basically the same. We need a performance indicator to evaluate our performance to improve. Both must be Non-Euclidean, and we must prioritize our decisions.

8 Mahalanobis Distance-Pattern (MDP) Approach Mahalanobis Distance-Pattern (MDP) Approach is developed to manage movement. Its basic idea is the same as MTS. Mahalanobis Distance (MD) is used, but in MTS, the whole pattern is compared, and MD provides the closeness of the patterns between Unit Space and a sample. But in MDP, we do not need to compare the whole pattern. We can pick up local movement and compare it with the dynamic pattern change of the ideal pattern. Let us take swimming for example, Water changes every minute and body movements are analog. But if you feel your arm muscle movement is not good, then you can compare the sequential change of your local part with that of ideal swimming. If MD of that part is larger, then it means you need to improve your movement there. Thus, depending on your feeling, or whatever idea may come up, you can select any part of your body and compare its movement with the ideal one. If yours is larger, you must make efforts to reduce. But since our body is analog, if you improve at one part, the whole movement might be necessary to coordinate once again. Then, we can compare the whole patterns.

9 Natural Computing 9.1 Mind-Body-Brain We often talk about Mind-Body-Brain. Mind is deeply associated with decision making. When we ask someone to decide, we say, “Make up your mind”. We opt to think that

408

S. Fukuda

Mind = Brain, But brain does not directly interact with the Real World outside. It is body that directly interact with the outside. Mind contains body and brain, but brain processes body experiences and accumulate them as knowledge. And we must remember that knowledge is an accumulation of personal experiences and our life varies from person to person, so that knowledge is very much personal, contrary to our idea that it is applicable to all cases (Fig. 9).

Fig. 9. Mind-Body-Brain

9.2 The Octopus Speaking of coordination of movement, we should remember the octopus. The octopuses die soon after their babies are born. So, they do not inherit knowledge from their previous generations. They live on their instinct alone. But they coordinate their eight limbs, and they escape from any environments and situations. They are known as “expert of escape”. They escape even from the screwed container. They are interacting with the Real World through their bodies and make strategic decisions how to act. Figure 10 compares octopus and human. We rely on knowledge and we think Mind = Brain, while the octopus uses its body to cope with the changing Real World. The head of the octopus is large, but it is not for processing knowledge, but to coordinating its limbs and body. The octopus is invertebrate, and is positioned on the opposite side to human, vertebrate in evolution tree. Their intelligence is at the same level as dogs, but it is one of the very few animals that can recognize self in the mirror. (MSR). Only the great apes passes MSR test, other than human. This demonstrates how the octopus is excellent in recognizing body movement and cope with the Real World with Wisdom.

PDSA Computing

409

Fig. 10. Octopus and Human R

References 1. https://en.wikipedia.org/wiki/Nikolai_Bernstein (2020) 2. Bernstein, N.A.: The Co-ordination and Regulation of Movements. Pergamon Press, Oxford (1967) 3. Mahalanobis, P.C.: On the generalized distance in statistics. Proc. Nat. Institute Sci. India 2(1), 49–55 (1936) 4. Taguchi, G., Chowdhury, S., Wu, Y.: The mahalanobis-taguchi system. McGraw-Hill Professional, New York (2000)

An Iterative Randomized Rounding Algorithm for the k-Clustering Minimum Completion Problem with an Application in Telecommunication Field Mhand Hifi(B) and Shohre Sadeghsa EPROAD EA 4669, Universit´e de Picardie Jules Verne, 7 rue du Moulin Neuf, 80000 Amiens, France {hifi,shohre.sadeghsa}@u-picardie.fr

Abstract. In this paper, we propose a new iterative randomized rounding algorithm for approximately solving the k-clustering minimum completion problem. The problem is defined in the telecommunication field where an instance of the problem is characterized by a bipartite graph G(S, C, E). S represents a set of services, C denotes a set of clients and E is a set related to the links between elements of C and S. The problem is to partition the initial graph into k-clusters such that each cluster is a complete bipartite subgraph. The goal is to minimize added edges to make a complete sub bipartite graph. The proposed algorithm is based upon two phases: (i) rounding a series of fractional variables, augmented with (ii) a constructive hill-climbing procedure. Both phases are embedded into an iterative search forming an iterative randomized search. The performance of the proposed method is evaluated on a set of benchmark instances taken from the literature. Provided bounds are compared to those reached by the best methods available in the literature. Encouraging results have been provided. Keywords: Optimization Telecommunication

1

· Graph · Rounding method ·

Introduction

The k-Clustering Minimum Bi-clique Completion problem (noted PKCM BC ) arises in a variety of real-world applications such as flexible manufacturing systems, commercial and telecommunication fields. In this study, we focus on the application of the PKCM BC in the telecommunication field. The proposed algorithm is applied on a benchmark dataset from the literature (other studies such as Gualandi et al. [5], Hifi et al. [4], Al-Iedani et al. [1], etc. are also used the same benchmark dataset). An instance of PKCM BC is defined by a bipartite graph G(S, C, E). The bipartite graph is composed of the part S of services, part C of clients, and c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 410–422, 2022. https://doi.org/10.1007/978-3-030-80119-9_24

k-Clustering Minimum Completion Problem

411

E denotes the edges linking the items from C to S. It follows that a complete bipartite graph has |S| × |C| edges. PKCM BC is to divide the initial graph into several multicast sessions (clusters). Each session (cluster) must be a complete sub-bipartite graph. The aim of PKCM BC is to minimize the number of the added edges to the sessions in order to transform them into a complete subbipartite graph. Note that in each cluster, a client from C can be served by at least one service from S. Thus a service s can appear only in one session, however, a customer c can benefit from several sessions. The complexity of PKCM BC is investigated in a study by Duginov et al. in [9]. This study explicitly stated that PKCM BC is Np-complete, thus exact methods are not able to find the optimum solution for the instances in a reasonable time. There are several studies in the literature that address a new solution method for PKCM BC . In this study, we applied the proposed method on the benchmark dataset and we compared the results with the best solution values reported from the studies in the literature. Here we used an approach that was introduced initially as a rounding strategy by Hifi in [8] for a disjunctively constrained knapsack problems. In this study, the rounding strategy is adapted to find solutions for PKCM BC , and also it is combined with a constructive hill-climbing procedure to achieve the enhanced solutions. Formally, PKCM BC can be stated as model 1 (cf., Faure et al. [3]):   ω(x) = min (1) xik yjk ¯ k∈K (i,j)∈E

s.t.



xik = 1, ∀i ∈ S

(2)

k∈K

xik ≤ yjk , ∀(i, j) ∈ E, ∀k ∈ K xik , yjk ∈ {0, 1}, ∀i ∈ S, ∀j ∈ C, ∀k ∈ K,

(3)

where xik and yjk are equal to one if the service i belonging to the partition S or respectively service j in partition C is assigned in the cluster k, k ∈ K, 0 otherwise. The objective function (Equality (1)) minimizes the number of the ¯ refers to the links not belonging to the set E) added to the additional edges (E bipartite subgraph in order to transform it into a complete bipartite subgraph. Constraints (2) ensure the affectation of each service i ∈ S in only one cluster, where as constraints (3) ensure that each client j ∈ C is assigned to the same cluster as its corresponding services in S if there is any edge E that links i to j. Note that in a solution, each service i ∈ S can be assigned only in one cluster while each client j ∈ C can appear in more than one cluster. The rest of this study is as follows: In Sect. 2 the most related studies are investigated. The randomized rounding method with the local search implementation is described in Sect. 3. The structure of the instances is described and the results are presented in Sect. 4 and the last section accomplished this study in Sect. 5.

412

2

M. Hifi and S. Sadeghsa

Related Work

To the best of our knowledge, Faure et al. [3] are the first authors tackled PKCM BC . They addressed that the problem is complex and the optimum solutions are not reachable with exact methods. The authors also proposed the first formal model with different versions. They also tailored an exact column generation-based algorithm and adapted the approach for tailoring a heuristic for solving large-scale instances. The emerged problem attracted attention from several researchers. From one of the most related studies, we mention the study of Gualandi [6]. They developed a hybrid method where constraint programming and semidefinite programming have been combined. In a related study of Gualandi et al. [5], the first branchand-price algorithm for the problem has been designed, where a fast tuning has been proposed for accelerating the convergence of the method. In the same time, Cplex solver has been incorporated in order to enhance the reported upper bounds. Later, Hifi et al. [4] described an adaptive neighborhood search algorithm, where both intensification and diversification strategies where combined in order to either diversify the solution process or to explore the search space. The two strategies are mainly based upon a greedy random procedure in which two phases were implemented. Al-Iedani et al. [1] proposed a variable neighborhood search heuristic for PKCM BC . Their proposed method starts with creating feasible solutions, then the solutions are improved using an iterative local search. The authors also introduced neighbor operators for the local search. The reported results are compared with the available methods from the literature. Saleh and Hifi [2] designed an alternative fast tabu search in which a series of sub-solution spaces were explored using a simple tabu strategy. The proposed tabu strategy is accomplished by using both degrading and rebuilding strategies. The performance of the method was evaluated on the benchmark instances extracted from the literature. The proposed method is verified with the small and medium sets of the benchmark instances. The reported results are compared to those reached by Cplex solver (with limited time) and the available results of the literature. Finally, in Hifi and Sadeghsa [7] a first version using an adaptation of the rounding strategy has been proposed, where the method was enhanced with a neighborhood local search. The performance of the method was evaluated with the instances of PKCM BC . The provided results were compared to the best bounds available in the literature. The authors showed in the experimental part that their method was able to match all existing bounds available in the literature for the small instances. This study can be viewed as an extended version of Hifi and Sadeghsa’s [7] method with the new adjustments and new achieved bounds.

k-Clustering Minimum Completion Problem

3

413

A Randomized Rounding Algorithm

The randomized rounding algorithm performs efficiently with a linear model. Thus we start by linearizing the model although, the quadratic formulation PKCM BC can be solved with tailored solvers. The nonlinear expression of the model 1 occurs in the objective function (i.e., Eq. (1)) at which the two binary variables xik yjk are multiplied. A classical linearization method for such a nonlinear model is to replace both decision variables xik and yjk with the new binary variable zijk (zijk = xik yik ). Herein, we might need to add the following linear constraints: zijk ≤ xik and zijk ≤ yjk xik + yjk ≤ 1 + zijk ,

¯ ∀k ∈ K ∀(i, j) ∈ E, ¯ ∀k ∈ K ∀(i, j) ∈ E,

(4) (5)

However considering the existing constraints in the model 1, Eq. (4) is always  satisfied. The formal reformulation, denoted PKCM BC , is described as follows:   zijk (6) ω(x) = min ¯ k∈K (i,j)∈E

¯ ∀k ∈ K xik + yjk ≤ 1 + zijk , ∀(i, j) ∈ E,  xik = 1, ∀i ∈ S s.t.

(7)

k∈K

xik ≤ yjk , ∀(i, j) ∈ E, ∀k ∈ K xik , yjk ∈ {0, 1}, ∀i ∈ S, ∀j ∈ C, ∀k ∈ K, ¯ ∀k ∈ K, zijk ∈ {0, 1}, ∀(i, j) ∈ E,

(8) (9)

Note that E denotes the set of complementary links between services of S and clients of C. It means that it is composed with the edges that are not in the set  E. One can observe that PKCM BC remains difficult to solve it to optimality and exact solvers may converge very slowly (especially for small and medium-sized instances, it is impossible to solve larger instances with reasonable runtime). We then propose an alternative method in order to overcome to this difficulty, even if the aforementioned method cannot ensure the optimality of the achieved bounds. 3.1

A Basic Rounding Procedure

A Basic Rounding Procedure (BRP) is applied to the mixed linear relaxation of  the problem PKCM BC (denoted MIPKCM BC ), which can be described as follows (model 3.1):

414

M. Hifi and S. Sadeghsa

ω(x) = min

 

zijk

¯ k∈K (i,j)∈E

s.t.

¯ ∀k ∈ K xik + yjk ≤ 1 + zijk , ∀(i, j) ∈ E,  xik = 1, ∀i ∈ S

(10) (11)

k∈K

xik ≤ yjk , ∀(i, j) ∈ E, ∀k ∈ K 0 ≤ xik ≤ 1, 0 ≤ yjk ≤ 1, ∀i ∈ S, ∀j ∈ C, ∀k ∈ K, ¯ ∀k ∈ K, zijk ∈ {0, 1}, ∀(i, j) ∈ E,

(12) (13)

where the variables (xik and yjk ) related to both the sets of clients (S) and services (C) are relaxed in Eq. (13). BRP can induce an approximate solution  for PKCM BC , if all the solution variables (zijk , xik and yjk ) are binary. In order to find the binary variables, BRP “round” the value of the variables having fractional values in the optimal solution of model 3.1. Rounding a variable in BPR means that if the optimum value of the relaxed variables xik is less or equal than 0.5 the corresponding value is considered as zero and if the value is more than 0.5 the corresponding value is considered as one. Considering the constraints of the model 3.1 (from (10) to (12)), BRP only round the variables xik and not yjk . We then tailor a BRP to MIPKCM BC as follows: Set Preduce = MIPKCM BC . Solve Preduce using the Cplex solver, and let (x, y, z) be the achieved solution. Fix the value of each xi , if xi = 1 in x. Select among the non-integer variables, the variable xi having the largest fractional value, and round it. 5. Remove all of the fixed variables from the current problem and let Preduce be the new reduced problem. 6. Repeat steps from (2) to (5) until fixing all the elements of the original problem. 7. Retrieve all the items satisfying constraints from (10) to (12), and fix them to 1, 0 otherwise.

1. 2. 3. 4.

 Finally, BRP provides a feasible integer solution for PKCM BC . Because of  the greedy nature of BRP and of the complexity of PKCM BC ’s precedence constraints inequalities of type (12), BRP may provide moderate-quality solutions. In fact, fixing a decision variable yjk (service) doesn’t induce a direct fixation of its containing items (clients) while fixing a client (xik ) to one imposes the fixation of its services. Note that in some cases, BRP may also eliminate, in a very aggressive way, some items by fixing their corresponding decision variables to one. Hence, BRP is hardly used as a stand alone procedure, but rather as part of a more elaborate computational approach. Thus the process needs some ingredients for enhancing

k-Clustering Minimum Completion Problem

415

the convergence of the method towards high bounds and to accelerating its runtime in the same time. In what follows we propose to modify the choice of the fractional variable to select by applying a random choice of that variable among all resulting fractional variables. 3.2

A Modified Rounding Strategy-Based Method

One can observe that BRP can also be transformed to Randomized BRP (noted RBRP), where almost of selecting the variable to fix to one in a deterministic way, one can make a random choice of an item overall fractional variables; thus, the following steps may describe the first extended version of RBP: 1. Set S  = ∅ and Preduce = MIPKCM BC . 2. Repeat (a) Call BRP on MIPKCM BC by using a random choose on the fractional variable. (b) Let Slocal be the achieved solution. (c) Set S  = Slocal whenever z(S  ) > z(Slocal ). 3. Until the stopping criterium is performed. In addition to this seemingly simplistic method, we propose in what follows a two-stage solution procedure, where a single-randomized BRP is called for providing a starting solution that will be modified/enhanced using hill-climbingbased variable neighborhood search. 3.3

Local Constructive Search

A Local Constructive Search (noted LCS) works as follows (when applied to the original problem): let C = {c1 , . . . , cn } (resp. S = {S1 , . . . , Sm }) be the set of n = |C| clients (resp. m = |S| services), E be the set of edges (existing links) between both sets C and S from c ∈ C to s ∈ S (each existing link is noted (c, s)) and nbK = |K| be the number of clusters. 1. Let K = {K1 , . . . , KnK } be the starting nK dummy clusters (empty clusters). Set (a) S  = S and C  = C; (b) h = 1 (the index h associated to the first cluster). 2. Set δ(si ), si ∈ S  , in decreasing order, where δ(si ) denotes the degree of the si -th vertex  (characterizing the number  of clients-nodes linked to it); Set  = arg max δ(s ) = δ(s1 ) | δ(s ) ≥ δ(si ) ; si ∈S

Set Kh = Kh ∪ {s }, S  = S  \ {s }, and update the set C  whenever s is affected, i.e., new sets S  and C  are recomputed. 3. For the nK − 1 remaining clusters, from the cluster h = nK − 1 down to h = 2 do: (a) Assign a vertex  ∈ S  to the h-th cluster; (b) Update both sets S  and C  . 4. Set h = 1; Repeat steps (2) and (3) till fixing all services in the clusters.

416

M. Hifi and S. Sadeghsa

Algorithm 1. Iterative Randomized Rounding Algorithm  Input. An instance of PKCM BC . Output. A feasible solution S  (x, y, z).

1: 2: 3: 4: 5: 6: 7:

Initialization Step. Set S  = ∅, be the starting solution and Pred = MIPKCM BC . Solve Preduce using an exact method, and let S(x, y, z) be the achieved solution. repeat repeat a) Fix the value of each xi , if xi = 1 in x. b) Among the non-integer variables, make a random selection of a variable xi , and round it to one. c) Let Spartial be the provided solution; d) Remove all the fixed variables from the current problem and let Pred be the new reduced problem. 8: Call RCP for solving the reduced P’KCM BC , where only variables of Pred are considered. 9: Let Sc be the complementary solution achieved by RCP.   10: Update  S by Spartial ∪ Sc whenever z(S ) > z(Spartial ∪ Sc ). 11:

until There is no more non integral variables in x

Set Pred = MIPKCM BC and let S(x, y, z) its configuration reached in Step 3.   13: until the stopping criterium is performed 14: Return S  . 12:

3.4

An Iterative Randomized Rounding Algorithm

The iterative randomized rounding algorithm can be viewed as an extended version of the basic rounding procedure. Its aime is to give a chance for improving the quality of the bounds to reach. As mentioned in [5], the values of both vectors x and y induced when solving the mixed relaxed problem MIPKCM BC is either equal to 0, 1 or 12 . Thus, using a deterministic method, like selecting the fractional variable with the highest service’s degree, we introduce a random selection and we then complete the partial solution achieved by using the local constructive procedure. Algorithm 1 describes the main steps of the Iterative Randomized Rounding Algorithm (IRRA). The initialization step (line 1) starts by setting the initial solution S  to an empty set and the original mixed integer program MIPKCM BC is considered (line 3) as the first reduced problem Pred . In line 3, the reduced mixed problem is solved with a blackbox solver, where the provided configuration is stored in S(x, y, z). IRRA is composed of two loops: the global loop from line 4 to line 13 and, the internal loop from line 5 to line 11. We recall that the global loop was introduced for creating a randomized search, where the internal loop is used for building a series of feasible solutions, hoping that these solutions will be explored in different search spaces.

k-Clustering Minimum Completion Problem

417

The internal loop (lines from 5 to 11), where a partial solution Spartial is built (line 7, a), b) and c)) by fixing all integral variables (those having their values equal to one), selects a client x, from x, with a fractional value and round it to 1 (whenever the resulting solution is feasible; 0 otherwise) and remove (line 7, d)) all fixed variables from the current mixed-integer program. Line 8 builds the reduced problem to solve, where only variables related to Pred are considered, and applies the Hill-Climbing procedure RCP (line 9) for providing a new local solution Sc (line 10). Also, line 10 updates the best current solution if its objective value is better than that reached up to now. The internal loop is iterated until saturating all decision variables, especially those related to the clients. Finally, the procedure exits with the final solution S  .

4

Computational Results

In this section, we evaluate the effectiveness of the proposed Rounding StrategyBased Algorithm (denoted RSBA) on the benchmark instances of the literature. Two sets of instances are considered: (i) the first set contains 27 small-sized instances, extracted from Gualandi et al. [5], where their optimal values are known and, (ii) the second set includes 18 medium and large-scale instances, extracted from Hifi et al. [4]. Note that, on the one hand, for the instances of first set, the number of services (resp. clients) related to S (resp. C) varies in the discrete interval {15,18,20}, |S| = |C|, the number of clusters nK varies in the discrete interval {3,4,5} and, the density of the bipartite graph d was generated using the |E| ; that varies in the interval {0.3, 0.5, 0.7}. On the following formula: d = |S|.|C| other hand, the instances of the second set were generated following Gualandi et al.’s generator, where the number of services (resp. clients) of S (resp. C) varies in the discrete interval {50, 80, 100} while all other parameters are the same. The results reached by the proposed algorithm are compared to the bounds achieved the Cplex solver and the best bounds available in the literature (Hifi et al. [4]). The proposed method was coded in C++ and run on the Intel Pentium Core i7 with 2.1 GHz. 4.1

The First Phase of the Randomized Rounding Procedure

Table 1 reports the results provided by the proposed randomized rounding procedure. The first column shows the information about the instances. The second column represents the value of the solutions for maximum 60 min runtime for the Cplex, the third column displays the best upper bound extracted from the literature, the next column UPhase1 represents the upper bound achieved with the first phase of this study for m repeats, where m denotes the number of services/clients of each instance. APhase1 stands for the percentage experimental approximation ratio for the first phase; that is, calculated by (UPhase1 /Best)-1×100. Positive values in APhase1 means that the objective values for the larger instances are higher than

418

M. Hifi and S. Sadeghsa

the best values in the literature, negative values correspond to the new bounds, and zero means that the value reached with this method matches the best available ones. Finally, the last column represents the runtime in minutes needed by the first phase. The last line represents the average of each column. Table 1. Solution quality of the first phase of the IRRA #Inst.

Cplex Best

UPhase1 APhase1 TPhase1

15-15-5-0.3 15-15-5-0.5 15-15-5-0.7 18-18-5-0.3 18-18-5-0.5 18-18-5-0.7 20-20-5-0.3 20-20-5-0.5 20-20-5-0.7 50-50-5-0.3 50-50-5-0.5 50-50-5-0.7 80-80-5-0.3 80-80-5-0.5 80-80-5-0.7 100-100-5-0.3 100-100-5-0.5 100-100-5-0.7

51 50 39 92 86 63 119 123 82 1423 1091 676 4166 2931 1775 6645 4716 2854

51 50 39 92 86 63 119 123 82 1319 1072 671 3815 2862 1768 6248 4650 2842

51 50 39 92 86 63 119 123 82 1424 1100 683 4244 2928 1792 6733 4753 2863

Average

1499

1441.78 1512.5

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.96 2.61 1.79 11.25 2.31 1.36 7.76 2.22 0.74

0.10 0.09 0.03 0.16 0.16 0.02 0.29 0.26 0.06 0.29 0.23 0.14 1.14 0.76 0.48 1.14 0.72 0.46

2.11

0.36

In what follows, we comment the results of Table 1: – The first phase is able to find the best solutions for the instances with the number of services (resp. clients) equals to 15, 18, and 20, respectively. – The approximation ratio varies from 0.00 to 11.25 (instance 80-80-5-0.3) and the runtime needed by the first phase for achieving these bounds varies from 0.10 (instance 15-15-5-0.3) to 1.14 seconds (instances 50-50-5-0.7 and 80-805-0.3). – In total, the first phase’s average approximation ratio is close to 2 and the average runtime needed by this phase is less than one second.

k-Clustering Minimum Completion Problem

4.2

419

Effect of the Hill-Climbing Strategy

Table 2 shows the effect of the Hill-Climbing strategy on the solutions achieved from the first phase. Labels are defined as in Table 1, where columns represent the instances information, best upper bound from Cplex for a maximum of 60 min runtime, best bounds from the literature, upper bound from the hillclimbing strategy, the percentage experimental approximation ratio (as defined in Sect. 4.1), and the runtime needed in seconds, respectively. As in Table 2: Table 2. Using a Hill-Climbing (HC) Strategy: Solution Quality #Inst.

Cplex Best

UHC

AHC

THC

15-15-5-0.3 15-15-5-0.5 15-15-5-0.7 18-18-5-0.3 18-18-5-0.5 18-18-5-0.7 20-20-5-0.3 20-20-5-0.5 20-20-5-0.7 50-50-5-0.3 50-50-5-0.5 50-50-5-0.7 80-80-5-0.3 80-80-5-0.5 80-80-5-0.7 100-100-5-0.3 100-100-5-0.5 100-100-5-0.7

51 50 39 92 86 63 119 123 82 1423 1091 676 4166 2931 1775 6645 4716 2854

51 50 39 92 86 63 119 123 82 1319 1072 671 3815 2862 1768 6248 4650 2842

51 50 39 92 86 63 119 123 82 1315 1068 671 3805 2862 1768 6247 4650 2832

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 −0.30 −0.37 0.00 −0.26 0.00 0.00 −0.02 0.00 −0.35

0.63 0.60 0.19 0.78 0.74 0.10 1.41 1.26 0.27 20.01 6.45 3.83 21.02 28.22 35.23 55.16 28.38 47.93

Average

1499

1441.78 1440.17 −0.07 17.23

– The provided upper bounds are closer to the best upper bounds in the literature when compared to the ones in Table 1. – The average value of the percentage approximation ratio over the instances is equal to 2.11 in Table 1 and −0.07 in Table 2. Of course, as smaller the approximation ratio the results are closer to the best values. – The negative value (column 5) shows that new bounds are achieved. Indeed, over the 18 instances considered, the method achieved 5 new bounds and it is able to match the rest of the 13 instances (the best solutions of the literature). – The global average approximation ratio is equal to −0.07 and the average runtime needed by the method is close to 17 s.

420

M. Hifi and S. Sadeghsa

Hence, Table 2 shows that the hill-climbing strategy has a positive effect on the solution values founded from the first phase and it decreases the approximation ratio. 4.3

Behavior of IRRA Vs Other Methods

Table 3 shows the behavior of IRRA versus other methods available in the literature. The first column represents the instances, the second column represents the best values provided with Cplex with 60 min as a runtime limit, Method 1 represents the results reached by ANS extracted from Saleh and Hifi [2], the next column represents the results provided with the Method 2, where Method 2 displays the best bound achieved by Al-Iedani et al. [1]. The last column represents the best results achieved by all methods. The last row of Table 3 indicates the number of instances with the best bound for each method. From Table 3, we observe what follows: – IRRA vs Cplex: IRRA dominates the Cplex solver; in this case, IRRA is able to provide 9 better bounds when compared to the results achieved by the Cplex and matches the rest (9 instances) of the bounds. Table 3. Behavior of IRRA Versus Three methods of the literature #Inst.

Cplex Method 1 Method 2 UIRRA Best

15-15-5-0.3 15-15-5-0.5 15-15-5-0.7 18-18-5-0.3 18-18-5-0.5 18-18-5-0.7 20-20-5-0.3 20-20-5-0.5 20-20-5-0.7 50-50-5-0.3 50-50-5-0.5 50-50-5-0.7 80-80-5-0.3 80-80-5-0.5 80-80-5-0.7 100-100-5-0.3 100-100-5-0.5 100-100-5-0.7

51 50 39 92 86 63 119 123 82 1423 1091 676 4166 2931 1775 6645 4716 2854

Nb best bounds 9

51 50 39 92 86 63 119 123 82 1319 1072 671 3815 2862 1768 6248 4650 2842

51 50 39 94 87 63 120 124 90 1485 1090 681 – – – – – –

51 50 39 92 86 63 119 123 82 1315 1068 671 3805 2862 1768 6247 4650 2832

51 50 39 92 86 63 119 123 82 1315 1068 671 3805 2862 1768 6247 4650 2832

13

4

18

–

k-Clustering Minimum Completion Problem

421

– IRRA vs Method 2: IRRA outperforms Method 2 although when considering the number of best bounds provided by each algorithm. Indeed, over the 12 instances tested by both methods IRRA provides 8 better bounds and matches the rest of the bounds, as Method 2. – IRRA vs Method 1: IRRA performs better than Method 1, since it is able to match 18 bounds (over the 18 instances tested by both methods) and is able to reach 5 new bounds.

5

Conclusion

This paper proposed a new randomized rounding algorithm to approximately solve the k-clustering minimum completion problem. The basic rounding procedure is the main idea of the method where it has been used for building a solution. Later a local search was first applied to improve the solutions at hand and finally the enhancing procedure was embedded into an iterative search to implement an efficient and powerful method until reaching the stopping conditions. The proposed method was evaluated on a set of benchmark instances of the k-clustering minimum completion problem available in the literature. The comparison study showed the superiority of the proposed method and was able to provide new bounds when compared to the best bounds available in the literature.

References 1. Al-Iedani, N., Hifi, M., Saadi, T.: Neighborhood search-based heuristic for the kclustering minimum biclique completion problem. In 2016 International Conference on Control, Decision and Information Technologies (CoDIT), pp. 639–643. IEEE (2016) 2. Ali Saleh, S., Hifi, M.: A fast method for optimizing the K-clustering bi-clique completion problem in telecommunication. J. Duhok Univ. 175–183 (2017) 3. Faure, N., Chr´etienne, P., Gourdin, E., Sourd, F.: Biclique completion problems for multicast network design. Discret. Optim. 4(3–4), 360–377 (2007) 4. Hifi, M., Moussa, I., Saadi, T., Saleh, S.: An adaptive neighborhood search for kclustering minimum bi-clique completion problems. In: Le Thi, H., Pham Dinh, T., Nguyen, N. (eds.) Modelling, Computation and Optimization in Information Systems and Management Sciences, pp. 15–25. Springer, Cham (2015). https://doi. org/10.1007/978-3-319-18161-5 2 5. Gualandi, S., Maffioli, F., Magni, C.: A branch and price approach to k-Clustering minimum biclique completion problem. Int. Trans. Oper. Res. 20(1), 101–117 (2013) 6. Gualandi, S.: k-klustering minimum biclique completion via a hybrid CP and SDP approach. In: van Hoeve, W.-J., Hooker, J.N. (eds.) Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems. LNCS, vol. 5547, pp. 87–101. Springer, Heidelberg (2009). https://doi.org/10.1007/ 978-3-642-01929-6 8 7. Hifi, M., Sadeghsa, S.: Adaptation of the rounding search-based algorithm for the kclustering minimum completion problem, In: IEEE, the 7th International Conference on Control, Decision and Information Technologies (CoDIT), Prague, Czech Republic, pp. 1150–1155 (2020). https://doi.org/10.1109/CoDIT49905.2020.9263846

422

M. Hifi and S. Sadeghsa

8. Hifi, M.: An iterative rounding search-based algorithm for the disjunctively constrained knapsack problem. Eng. Optim. 46(8), 1109–1122 (2014) 9. Duginov, O.: On the complexity of the clustering minimum Biclique completion problem (2015)

Non-dominated Ranking Biogeography Based Optimization Algorithm for Virtual Machine Placement in Cloud Computing Asma Bouhank(B) and Mourad Daoudi Electronics and Computer Science Faculty, Laboratory LSI, USTHB, El Alia, BP 32 16111 Bab-Ezzouar, Algiers, Algeria [email protected]

Abstract. In cloud environment, placing virtual machines in appropriate servers with guarantee of high performance level is a crucial problem, known to be NP-hard due to its complexity and large solutions space. In this paper, a Non-dominated Ranking Biogeography Based Optimization algorithm, named NRBBO, is proposed in order to minimize simultaneously the total resource wastage and power consumption of all servers in virtual machines placement. Using Synthetic data from literature, the effectiveness of the proposed approach has been investigated by conducting several experiments. NRBBO is compared to other multi-objective solutions and the obtained results show that NRBBO is more efficient and has better convergence and coverage in finding the optimal solutions for the virtual machines problem. Keywords: Multi-objective optimization · Biogeography based optimization · Virtual machines placement problem · Cloud computing

1

Introduction

Cloud computing [11] is an emerging technology that has marked its place in the IT Industry [29] and research area as well [9]. Cloud computing offers flexible and scalable services on user-demand via a pay-per-use model. It can provide three types of services: Saas (Software as a service), Paas (Platform as a service) and Iaas (Infrastructure as a service). These services are offered with different levels of quality of service in order to meet the specific needs of different users. In order to maximize the use of resources, virtualization is considered as a key. The resources of a single physical machine (PM) are divided into multiple isolated virtual machines (VMs). The use of virtualization in cloud computing is beneficial in terms of flexibility, resource utilization, availability and cost minimization. An inappropriate placement of virtual machines in physical machines may result in VMs performance degradation due to the imbalance use of resources, and some servers overload . . . For these, virtual machines must be c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 423–438, 2022. https://doi.org/10.1007/978-3-030-80119-9_25

424

A. Bouhank and M. Daoudi

properly placed to achieve the performance objectives. Virtual machines placement (VMP) problem is one of the major challenges in cloud computing. It consists of finding the appropriate physical machine for each virtual machine with guarantee of high level of performance. The VMP problem is similar but more complex then the bin packing problem [16]. Due to its complexity and the potential large solution space, the VMP problem is considered NP-hard problem. In any resource allocation policies, several criteria must be satisfied in order to meet the user QoS and provider requirements, which make the VM placement a multi-objective optimization problem. Meta-heuristics are good candidate to solve this kind of problems. In this paper, we propose a pure multi-objective approach for the VMP problem named NRBBO, based on the non-dominated ranking [5]. A pareto-based Biogeography Based Optimization (BBO) algorithm is used aiming to optimize two conflicting objective functions: minimizing both power consumption and resource wastage. According to [15] only few of the literature articles proposed a pure multi-objectives approach, using Pareto dominance for comparing conflicting objective functions. To the best of authors’ knowledge, the proposed BBO approach with the use of a non-dominated ranking is a novel technique. The performance of the proposed approach is compared with other multi-objective evolutionary algorithms [5, 8,29]. The rest of the paper is as follows: Sect. 2 gives a brief review of the existing solutions. Section 3 presents a formulation of the VMP problem. The proposed approach with a background knowledge are provided in Sect. 4. The conducted experiments and results are shown in Sect. 5 to evaluate the effectiveness of the proposed approach and finally Sect. 6 concludes the paper.

2

Background

Many researchers have studied the VM placement problem, and different approaches exist in the literature. The VMP problem was formulated as a bin packing problem. In [13] authors have proposed a novel algorithm called the first last algorithm aiming to minimize power consumption and migration cost. Whereas, in [3] the best fit techniques were used considering power consumption and memory sharing. In order to satisfy the cloud users’demands, a modified round robin algorithm was proposed in [17] to improve the resource allocation by reducing the response time. In [10] the authors proposed an online algorithm to solve the resource placement problem in real-time that can handle in both traditional and distributed cloud systems. They proposed an approximation algorithm for joint allocation of different resources. An algorithm based on the random learning automata (LA) is developed in [9] for the dynamic placement of VMs in datacenters. The aim of this algorithm is to reduce the power consumption with the minimum Service Level Agreement (SLA) violation, based on the live migration and turning off the idle nodes. Heuristics have also been used to solve the VMP problem. In [26], the author solves the multi-objective VMP problem using heuristic algorithms aiming to minimize the total execution time of the PMs. A redundant

NRBBO Algorithm

425

technique to enhance the reliability of cloud services when placing VMs was proposed in [30]. In [19], the authors studies increasing time and proposes a resource efficiency-based approach using a proposed heuristic to minimize total energy consumption. Considering the multiobjective nature of the VMP problem, metaheuristics based solutions have taken a big attention due to NPhardness of the problem. In [21], a multiobjective crow search algorithm was proposed to find an optimal solution for the VM placement problem with reducing both resource wastage and power consumption. In order to manage and schedule a large set of VM requests a queuing structure was proposed firstly. In [2], the authors proposed a Hungarian algorithm aiming to improve the resource utilization taking into account MIPS and PE values. In [14], an interactive memetic algorithm was adopted to solve the many objective virtual machine placement problem. The approach was designed to consider five objective functions: power consumption, network traffic, economical revenue, quality of service and network load balancing. In [20] the authors proposed an adaptative genetic algorithm AGA based on parametric and algorithmic adaptation aiming to minimizing the number of filled PMs and the energy supply cost. An improved genetic algorithm with fuzzy multiobjective evaluation was also proposed in [27] to solve the VMP problem aiming to optimize simultaneously total resource wastage, power consumption and thermal dissipation costs. A VM provision solution is proposed in order to achieve load balancing with the minimum wasting resources. The load balancing model take in consideration both of customer and service provider based on game theory. Then, ant colony optimization based on Nash equilibrium is used [4]. In [8], the author proposed a modified ant colony system algorithm named VMPACS to solve the multi-objective VM placement problem in order to minimize resource wastage and power consumption. The Ant Colony Optimization algorithm was adopted to minimize the total energy consumption for VM placement in [1]. In [24] the authors proposed the direct guidance ant colony system for effective virtual machines mapping with maximal resource utilization and minimal power consumption. An improved ant colony algorithm based on a prediction model was proposed by [7] to minimize the energy consumption with guarantee of Quality of Service (QOS). Particle Swarm was also adopted to solve the VMP problem by [25,28]. In the first work, the authors proposed a modified binary particle swarm optimization taking into consideration two factors: efficient energy consumption and resource utilization. In other hand, A discrete particle swarm optimization approach to minimize the number of active PMs was proposed to solve the VMP problem, called SOWO [28]. Biogeography based optimization was also used. In [29], the authors used the biogeography-based optimization to solve the VM consolidation problem. They formulated the problem as a complex system aiming to minimize resource wastage and power consumption simultaneously. In [18], the authors aimed to optimize three objective functions simultaneously: reducing power consumption, load balancing and VMs’ migration time. They used a hybrid solution called MBBO/DE based on biogeography-based optimization and Differential Evolution. It adopts Cosine migration model with a differential

426

A. Bouhank and M. Daoudi

strategy and Gaussian mutation model in optimization. The VMP problem was solved in [22] using improved biogeography-based optimization. The proposed solution aims to reducing server power consumption, load balancing, and migration resource overhead. Improving migration and mutations model showed its impact on the effectiveness of the proposed method. The review of different VM allocation strategies in cloud environment have been presented and analyzed in [12, 15]. Our main contributions are: – Formulating the VM placement problem as a pure multi-objective optimization problem based on the non-dominated ranking [5]. – Using adapted BBO to optimize two conflicting objective functions: power consumption and resource wastage. – Reducing the energy consumption by minimizing the energy consumed by all servers and by minimizing the number of active servers. – The migration operator used in the BBO algorithm depends on the ranking associated to each solution.

3

Problem Formulation

The VMP problem consists of placing N virtual machines in M physical machines (or servers) with respect of several constraints. Each server j, j ∈ {1, ...M }, is characterized by: – Its CPU and memory capacities and utilization, denoted respectively CjCP U , U em and useM . CjM em , useCP j j U em – Its remaining CPU and memory, denoted, respectively resCP and resM . j j – Its power consumption when it is idle and on full-load, denoted, respectively Pjidle and Pjbusy . – A binary variable yj which indicates wither it is used or not. Further, each virtual machine i, i ∈ {1, . . . , N }, is characterized by its CPU and memory demands denoted, respectively by DiCP U and DiM em , with respect to the following constraints: – Each VM is allocated to only one server. – The resource utilization of VMs allocated in the j th server don’t exceed the server’s capacity: userj ≤ Cjr , j ∈ [1, . . . , M ] , r = {CP U orM emory} We aim to minimize the total resource wastage and the power consumption as defined below.

NRBBO Algorithm

3.1

427

Power Consumption

Previous studies have showed that the power consumption is related to CPU utilization [29]. The power consumption of the jth server is defined as follows:    U idle + P power consumption (j) = yj × Pjbusy − Pjidle · useCP j j When the virtual machines are properly allocated, the energy efficiency in data centers will be enhanced. Moreover, consolidating virtual machines to physical servers and shutting down idle ones can also constitute a solution for the energy problem. On this basis, we consider the following normalized function which represents weighted sum function that aim to optimize both of total power consumption of all servers and the number of active servers:   M N br serv busy j=1 power consumption (j) + M inimize max power consumption M Where N br serv busy is the number of active servers, max power consumption is the maximum value of power consumption that correspond to the value obtained in the worst VM placement scenario (one VM per server). M is the total number of server and also is the number of active servers in the worst scenario. 3.2

Resource Wastage

This function aims to balance the resources left in each server in all dimensions in order to fully utilize these resources [29]. The wasted resources in server j is calculated as follow:   R yj   resrj − reslj  +  × resource wastage (j) = R userj + uselj r=l l=1 Where R is the number of resources in a server (R = 2, CPU and Memory) and  a tiny positive number. Our objective function is defined as follows: ⎞ ⎛ M  resource wastage (j)⎠ M inimize ⎝ j=1

4

The Non-dominated Ranking Biogeography Based-Optimization

In this section, we present our approach, named NRBBO, based on Biogeography based optimization. A brief overview of BBO is discussed and then the modifications performed to BBO for solving the VM placement problem are explained.

428

4.1

A. Bouhank and M. Daoudi

Biogeography Based Optimization

BBO is an evolutionary algorithm introduced in 2008 [23]. It is inspired from biogeography which is the study of migration, extinction and geographical distribution of biological species. This algorithm can provide good performance given enough generations. The BBO algorithm manipulates a population of individuals called islands (or habitats) which correspond to candidate solutions of an optimization problem. Each island is represented by a vector of SIVs (Suitability Index Variables). SIV is an integer that is allowed in a habitat which represents the decision variable. The fitness of each island is determined by its Habitat Suitability Index (HSI) and it is used to measure the goodness of the solution. High HSI indicates good habitat while low HSI indicates poor habitat. Habitats with higher HSI tend to have a large number of species, a low immigration rate and a high emigration rate. Vice-versa, habitats with low HSI tend to have a small number of species, a high immigration rate and a low emigration rate. In general, the immigration and emigration rates, respectively λi and μi , are calculated using the equations below: 

Si λi = I 1 − SM ax E · Si μi = SM ax Where I and E are the maximum immigration and emigration rates, respectively Si is the number of species in the ith habitat and SM ax the maximum number of species. There are two main operators: migration and mutation. Migration is a probabilistic operator used to share SIVs between solutions. The probability that a solution is selected to immigrate or emigrate depends on its immigration and emigration rates. Whereas, mutation modifies probabilistically one or more SIVs in a selected solution by a randomly generated new SIVs. The initial population is maintained and updated by migration followed by mutation from generation to generation until a termination criterion is met [23]. 4.2

The Adapted BBO for the VMP Problem: NRBBO

NRBBO solves the VM placement problem in a multi-objective context. A solution to this problem is represented as a vector of integers {x1 , x2 , . . . , xN } called island, where N is the number of virtual machines and each xi represents a SIV indicating the server number assigned to ith VM, which means xi ∈ [1 . . . M ] where M is the number of servers. A solution is feasible if and only if it meets all the aforementioned constraints (Sect. 3). Our proposed approach works as follows: – The BBO parameters are initialized such as: the number of VMs N , the number of servers M , the number of islands in population p size, mutation

NRBBO Algorithm

– – – – – –

429

probability p mut, SIV migration probability P siv, elitism parameter N elit, and stopping criterion. The initial population is generated. The initial population consists of p size feasible solutions which are generated randomly. Each solution is evaluated according the two objective functions mentioned in Sect. 3. The non-dominated ranking is applied: a rank ranki is associated by applying the non-dominated ranking. HSI of an island is the inverse of its rank. Migration on the actual population is performed. So, a new population is generated called pop mig. The mutation is performed on the pop mig which results a new population called pop mut. If the maximum number of iterations is not reached, the population for the next generation is formed from the best Nelit islands from previous population and islands from the popmut , applying non-dominated ranking. The pseudo-code of NRBBO is illustrated below in Algorithm 1.

Algorithm 1. NRBBO Pseudo-Code initialize parameter and generate the initial population. while Stopping criteria not reached do calculate the values of the two objectives for each habitat. rank each habitat. calculate the immigration and emigration rates of each habitat. for each habitat i do if rand < λi then for each siv of the habitat do if rand < P siv then Select the emigrated habitat j using the roulette wheel selection. Share siv between habitat i and habitat j. end if end for end if end for for each habitat i do if rand < p mut then choose a random siv from habitat i. replace the chosen siv with a randomly generated siv. end if end for calculate the values of the two objectives for each habitat. rank each habitat. process elitism and generate the population of the next generation. end while

430

A. Bouhank and M. Daoudi

The Non-dominated Ranking. The Non-dominated Ranking is based on Pareto fronts and crowding distance [5]. The identification of different fronts in a population is performed using the dominance concept. A solution is nondominated if there exists no other solution that improves on all of the objectives functions at once. The search of non-dominated solutions makes it possible to classify the individuals in several fronts. A solution belonging to a front does not dominate any solution of the same front. The optimal Pareto front corresponds to the front 1, the front 2 is obtained by removing the non-dominated solutions of the front 1 and then searching for the non-dominated solutions, and so on for the next fronts [5]. After classifying the solutions in different fronts, the crowding distance is used to compare individuals of the same front. It is used to estimate the density of solutions surrounding a particular solution on a front [5]. The crowding distance is calculated as follows: – For each front, we consider the objective functions independently. – For each objective function, the individuals are sorted in ascending order. – The boundary solutions, those with smallest and largest objective value, are assigned an infinite crowding distance. – Then, for each intermediary solution, we compute the normalized difference between the following and preceding individuals and sum it to the individual crowding distance. To attribute a rank for each solution we proceed as follows: First, islands are sorted according to their pareto level, island with lower value is considered the best. Islands with the same pareto level are sorted using crowding distance, island with larger value is considered the best. Migration. In migration, we share SIVs between two islands which are chosen according to their immigration and emigration rate. For each island, we calculate the immigration and emigration rates based on the island’s rank using following formulas: ranki λi = p size + 1 μi = 1 − λi Where p size is the total number of islands in the population and ranki the associated rank to the ith solution. The probability of an island being choose as an immigrating island is linearly related to its immigration rate λ. The solution with a low rank has the higher probability to be chosen. Then, an emigrating island is probabilistically selected for each immigrating island using the roulette wheel selection based on emigration rate μ. The solution with a low rank has the higher probability to be chosen. Each SIV from the immigrating island has the probability Psiv of being replaced by a randomly chosen SIV from the emigrating island. In each swap, we check the feasibility of the generated solution, If the replaced SIV results an infeasible solution we set back the original SIV and we move to the next one.

NRBBO Algorithm

431

Mutation. Mutation operator consists of replacing a randomly chosen SIV from a selected island by a new randomly generated SIV. Island’s chance of being chosen for the mutation is proportional to the mutation probability Pmut . If the generated island is not feasible then we repeat the operation with an another chosen SIV until we get a feasible island or all the possibilities are treated, and in this case, we save the initial solution and then move to the next selected island.

5

Tests and Results

Tests and experiments have been conducted to evaluate the effectiveness of the proposed NRBBO for solving the VM placement problem. Two sets of experiments have been designed; the first one consists of comparing our algorithm with two other existing multi-objective solutions: the VMPMBBO proposed in [29] based on BBO/Complex system and the ant algorithm called VMPACS proposed in [8]. The second experiment is to compare NRBBO with the classical multiobjective algorithm NSGA II [5] adapted for the VMP problem. All algorithms were coded in the java language and ran on an Intel Core i5 processor with 2.50 GHz CPU and 4 GB RAM. The values of different parameter in NRBBO have a direct impact on the algorithm’s performance. Preliminary experiments were conducted to determine the suitable parameter values. Table 1 shows the fixed parameters. Table 1. NRBBO parameters Parameter

Value

p size

6

Psiv

0.3

Pmut

0.05

Nelit

2

max iteration 100000

The number of servers and virtual machines is set to 200 Servers and VMs have the same cardinality in order to include the worst scenario where each server hosts only one single VM. For reasons of simplicity, we assumed the homogeneity of servers but our proposed algorithm can also deal with heterogeneous servers. CPU and Memory utilization of each server CjCP U and CjM em are bounded to 90% so that the server overload is avoided. PjCP U and Pjbusy mentioned in Sect. 3 are fixed to 162w and 215w, respectively. The CPU and Memory demands of each VM are generated randomly using the same method presented in [8]. CPU and Memory are linearly correlated. The detailed method is presented in the Algorithm 2 below:

432

A. Bouhank and M. Daoudi

Algorithm 2. VMs Generator for i ≤ N do DiCP U = rand (1) ∗ 2DCP U DiM em = rand (1) ∗ DM em r = rand (1)    if r < P ∧ DiCP U ≥ DCP U ∨ r ≥ P ∧ DiCP U < DCP U then DiM em = DiM em + DM em end if end for

Where DCP U and DM em represent respectively the CPU and Memory Demands’ reference value. rand(1) generates a real random number in the rang [0; 1]. The CPU and Memory demands are controlled by defining the probability P. DCP U and DM em take two values (25%; 45%) and for each reference value, P is varying five times (0 ; 0.25 ; 0.5 ; 0.75 ; 1), so 10 different scenario are obtained with different correlation coefficients. When DCP U = DM em = 25%, DiCP U and DiM em are in the range [0; 50%] and the obtained CPU and Memory demands average correlation coefficients are −0.754, −0.348, −0.072, 0.371, and 0.755. When DCP U = DM em = 45%, DiCP U and DiM em are in the range [0; 90%] and the obtained CPU and Memory demands average correlation coefficients are −0.755, −0.374, −0.052, 0.398, and 0.751. The five coefficients for each reference value refers to strong negative, weak negative, zero, weak positive, and strong positive correlations. For each scenario, the experiments are repeated 20 times and the average results are noted. 5.1

Comparison of NRBBO with VMPACS and VMPMBBO

The performance of the proposed algorithm is compared with two other existing multi-objective algorithms: VMPACS [8] and VMPMBBO [29]. The total resource wastage and power consumption obtained from each of the algorithms under consideration are reported in Table 2. VMBMBBO divided the initial population into four subsystems where each subsystem optimizes a single objective function. In order to share information between subsystems, within subsystem migration operator is used. The operator is based on the rankings of islands within their own subsystem. So, we noticed that the VMPMBBO optimizes each objective function independently without conflict with the second objective function and there is no general comparison between subsystem solutions according to the two objective functions which brings us to a corporation of four mono-objectives algorithms. So, the obtained results don’t belong to the approximation pareto solutions set. The power consumption results obtained from the NRBBO are approximative to those of VMPMBBO for the two-reference value 25% and 45% while the resource wastage is efficiently optimized in NRBBO compared to VMPMBBO in all 10 scenarios with different correlation coefficients.

NRBBO Algorithm

433

Table 2. Comparison with VMPACS and VMPMBBO for resource wastage and power consumption Reference value Corr. 25%

Algorithm

Power (w) Wastage

VMPACS VMPMBBO NRBBO VMPACS VMPMBBO NRBBO VMPACS VMPMBBO NRBBO VMPACS VMPMBBO NRBBO VMPACS VMPMBBO NRBBO

11754,38 11913,23 12246,32 11766,94 11304,71 11809,86 11661,55 10904,85 11920,74 11684,99 10590,78 11859,06 11470,24 10385,42 10937,52

c2,86 1,2 0,65 2,12 1,17 1,1 2,47 0,92 0,51 2,32 0,73 0,47 1,12 0,54 0,38

VMPACS VMPMBBO NRBBO −0,374 VMPACS VMPMBBO NRBBO −0,052 VMPACS VMPMBBO NRBBO 0,398 VMPACS VMPMBBO NRBBO 0,751 VMPACS VMPMBBO NRBBO

23364,88 22380,15 23020,57 23444,02 21297,67 22662,43 20876,32 20646,12 22190,57 22538,2 19890,57 22131,4 21766,89 18770,45 20771,86

18,78 9,8 6,87 13,88 6,53 5,21 13,13 5,59 5,28 9,61 5,54 3,94 5,18 2,59 1,85

−0,754

−0,348

−0,072

0,371

0,755

45%

−0,755

In the other hand, VMPACS is a pure multi-objective algorithm which optimizes the two objective functions resource wastage and power consumption using an ant colony system and returns the set of non-dominated solutions. For DCP U = DM em = 25%, the power consumption results of NRBBO are inferior but approximative to those of VMPACS. NRBBO shows better power consumption results for all scenarios related to the reference value

434

A. Bouhank and M. Daoudi

DCP U = DM em = 45%. However, NRBBO performs better then VMPACS in minimizing the total resource wastage in all 10 scenarios. In general, we can say that our algorithm was able to find solutions of good compromise between the two objective functions and has proved its efficiency in solving the multi-objective VM placement problem. 5.2

Comparison of NRBBO with NSGA II

To evaluate the good behaviour of our approach, we compared the performance of NRBBO to that of NSGA II [5]. NSGA II is a multi-objective evolutionary algorithm that use a nondominated sorting based on crowding distance, introduced for the first time in 2009. It was considered as a reference to find the optimal pareto solutions, with an excellent solutions distribution. Nsga II begins, for each iteration, with population of N individuals. Population of off-springs is created from parents using selection, crossover and mutation operators. The two populations are merged and formed new population with 2N individuals. A non-dominated sorting is conducted on this new population, the best N individuals are chosen to construct the next population for the next generation [5]. For that, we use a special metric called the Hypervolume indicator IH [6]. This indicator is used to quantify both the convergence and diversity of information of the set of solutions given by a multi-objective optimization algorithm without having previous knowledge of the true Pareto front. IH measures the area dominated by the approximated Pareto front given by one algorithm. A reference point is used to bound this area. It can be defined as: ⎞ ⎛ |Q|  IH = V olume ⎝ vi ⎠ i=1

For each solution i ∈ Q, vi represents the hypercube formed with the reference point and the solution i in the diagonal corners. IH is the hypervolume of the all hypercubes union. An algorithm with better convergence and coverage of the optimal Pareto front is indicated with the greater value of IH. In these experiments, the hypervolume indicator IH is calculated at each iteration of each execution of the two algorithms NRBBO and NSGA II. the number of iteration is set to 10000. The reference point is defined as the worstcase solution (one VM for each server).

NRBBO Algorithm

435

(a) DCP U = DM em = 25%, Corr. =(b) DCP U = DM em = 45%, Corr. = −0.754 −0.755

(c) DCP U = DM em = 25%, Corr. =(d) DCP U = DM em = 45%, Corr. = −0.348 −0.374

(e) DCP U = DM em = 25%, Corr. =(f) DCP U = DM em = 45%, Corr. = −0.052 −0.072

(g) DCP U = DM em = 25%, Corr. = 0.371 (h) DCP U = DM em = 45%, Corr. = 0.398

(i) DCP U = DM em = 25%, Corr. = 0.755 (j) DCP U = DM em = 45%, Corr. = 0.751

Fig. 1. Hypervolume indicator relative to the number of iterations in NRBBO and NSGA II

436

A. Bouhank and M. Daoudi

In Fig. 1, the hypervolume indicator for all test sets are shown. From these results, it can be concluded that NRBBO is performing better then NSGA II. The results show that NRBBO converge quickly to the optimum solution then NSGA II. In NSGA II, chromosomes share information with each other without any preferences, and the non-dominated sorting is only used in the selection phase. But in the other hand, NRBBO used the non-dominated ranking to rank islands and then to calculate the immigration and emigration rates for the migration operator. So, in NRBBO, the islands with best ranking shares information with other islands which results in a quick move to the optimal solution. NRBBO has always the greater IH values so we conclude that it has better convergence and coverage of the optimal pareto front.

6

Conclusion

In this paper, we propose a pure multi-objective solution for the VM placement problem in cloud computing called NRBBO. The proposed algorithm uses the Biogeography Based Optimization algorithm based on the Non-dominated Ranking in order to provide a set of non-dominated solutions which provide a good compromise between total resource wastage and power consumption. The Non-dominated Ranking is used to rank islands and then calculates the immigration and emigration rates based on this ranking. The power consumption is minimized by reducing the total power consumed by all servers and minimizing the number of active servers. The efficiency of NRBBO is demonstrated by comparing it with other multi-objective algorithms in term of quality, convergence and diversity of the obtained solutions. The results show that NRBBO is a competitive solution for the virtual machine placement problem.

References 1. Alharbi, P., Tian, Y.-C., Tang, M., Ferdaus, M.H.: Profile-based ant colony optimization for energy-efficient virtual machine placement. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, E.-S.M. (eds.) Neural Information Processing, vol. 10634, pp. 863–871. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70087-8 88 2. Amitha, B., Acharya, S.: Policy for resource allocation in cloud computing. Am. J. Intell. Syst. 7, 95–99 (2017) 3. Batra, G., Singh, H., Gupta, I., Singh, A.K.: Best fit sharing and power aware (BFSPA) algorithm for VM placement in cloud environment, pp. 1–4. IEEE, September 2017 4. Bui, K.T., Pham, T.V., Tran, H.C.: A load balancing game approach for VM provision cloud computing based on ant colony optimization. In: Cong Vinh, P., Tuan Anh, L., Loan, N.T.T., Vongdoiwang Siricharoen, W. (eds.) ICCASA 2016. LNICSSITE, vol. 193, pp. 52–63. Springer, Cham (2017). https://doi.org/10.1007/ 978-3-319-56357-2 6 5. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6, 182–197 (2002)

NRBBO Algorithm

437

6. Deb, K.: Multi-Objective Optimization Using Evolutionary Algorithms. Wiley, Hoboken (2001) 7. Duan, H., Chen, C., Min, G., Yu, W.: Energy-aware scheduling of virtual machines in heterogeneous cloud computing systems. Future Gener. Comput. Syst. 74, 142– 150 (2017) 8. Gao, Y., Guan, H., Qi, Z., Hou, Y., Liu, L.: A multi-objective ant colony system algorithm for virtual machine placement in cloud computing. J. Comput. Syst. Sci. 79, 1230–1242 (2013) 9. Ghobaei-Arani, M., Shamsi, M., Rahmanian, A.A.: An efficient approach for improving virtual machine placement in cloud computing environment. J. Exp. Theor. Artif. Intell. 29, 1149–1171 (2017) 10. Hao, F., Kodialam, M., Lakshman, T.V., Mukherjee, S.: Online allocation of virtual machines in a distributed cloud. IEEE/ACM Trans. Netw. 25, 238–249 (2017) 11. Hogan, M., Liu, F., Sokol, A., Tong, J.: NIST cloud computing standards roadmap. NIST Spec. Pub. 35, 6–11 (2011) 12. Khan, M.A., Paplinski, A., Khan, A.M., Murshed, M., Buyya, R.: Dynamic virtual machine consolidation algorithms for energy-efficient cloud resource management: a review. In: Rivera, W. (ed.) Sustainable Cloud and Energy Services. LNICSSITE, pp. 135–165. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-62238-5 6 13. Kumar, A., Sathasivam, C., Periyasamy, P.: Virtual machine placement in cloud computing. Indian J. Sci. Technol. 9 (2016) 14. Lopez-Pires, F.: Many-Objective Resource Allocation in Cloud Computing Datacenters, pp. 213–215. IEEE, April 2016 15. Lopez-Pires, F., Baran, B.: Virtual machine placement literature review. arXiv:1506.01509 [cs], June 2015 ´ Approximability of virtual machine allocation: much harder than bin 16. Mann, Z.A.: packing, pp. 21–30 (2015) 17. Pradhan, P., Behera, P.K., Ray, B.N.B.: Modified round robin algorithm for resource allocation in cloud computing. Procedia Comput. Sci. 85, 878–890 (2016) 18. Zheng, Q., Li, J., Dong, B., Li, R., Shah, N., Tian, F.: Multi-objective Optimization Algorithm Based on BBO for Virtual Machine Consolidation Problem, pp. 414– 421. IEEE, December 2015 19. Quang-Hung, N., Son, N.T., Thoai, N.: Energy-saving virtual machine scheduling in cloud computing with fixed interval constraints. In: Hameurlain, A., K¨ ung, J., Wagner, R., Dang, T.K., Thoai, N. (eds.) Transactions on Large-Scale Data- and Knowledge-Centered Systems XXXI. LNCS, vol. 10140, pp. 124–145. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-662-54173-9 6 20. Rolik, O., Telenyk, S., Zharikov, E., Samotyy, V.: Dynamic Virtual Machine Allocation Based on Adaptive Genetic Algorithm, pp. 108–114, February 2017 21. Satpathy, A., Addya, S.K., Turuk, A.K., Majhi, B., Sahoo, G.: Crow search based virtual machine placement strategy in cloud data centers with live migration. Comput. Electr. Eng. December 2017 22. Shi, K., Yu, H., Luo, F., Fan, G.: Multi-Objective Biogeography-Based Method to Optimize Virtual Machine Consolidation, pp. 225–230, July 2016 23. Simon, D.: Biogeography-based optimization. IEEE Trans. Evol. Comput. 12, 702– 713 (2008) 24. Srija, J., John, R.R., Kanaga, G.M.: An element search ant colony technique for solving virtual machine placement problem. J. Phys. Conf. Ser. 892, 012007, September 2017 25. Tripathi, A., Pathak, I., Vidyarthi, D.P.: Energy efficient VM placement for effective resource utilization using modified binary PSO. Comput. J. 61, 832–846 (2018)

438

A. Bouhank and M. Daoudi

26. Wu, J., Shen, H.: Efficient algorithms for VM placement in cloud data center. In: Chen, G., Shen, H., Chen, M. (eds.) PAAP 2017. CCIS, vol. 729, pp. 353–365. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-6442-5 32 27. Xu, J., Fortes, J.A.B.: Multi-Objective Virtual Machine Placement in Virtualized Data Center Environments, pp. 179–188. IEEE, December 2010 28. Yan, J., Zhang, H., Xu, H., Zhang, Z.: Discrete PSO-based workload optimization in virtual machine placement. Pers. Ubiquit. Comput. 22(3), 589–596 (2018). https://doi.org/10.1007/s00779-018-1111-z 29. Zheng, Q., et al.: Virtual machine consolidated placement based on multi-objective biogeography-based optimization. Future Gener. Comput. Syst. 54, 95–122 (2016) 30. ZhouZhou, A., et al.: Cloud service reliability enhancement via virtual machine placement optimization. IEEE Trans. Serv. Comput. 10, 902–913 (2017)

Multi-core Aware Virtual Machine Placement for Cloud Data Centers with Constraint Programming Nagadevi(B) and Kasmir Raja SRM Institute of Science and Technology, Chennai, TamilNadu, India [email protected] http://srmuniv.ac.in

Abstract. Creation of a Virtual Machine (VM) in a suitable Physical Machine (PM) is the critical requirement of a Cloud Service Provider in cloud data center. Mapping the VM to an appropriate PM is called VM Placement. The VMP decision is required at different stages of a cloud data center. Finding the best PM to place/create a VM is popularly known as Virtual Machine Placement Problem (VMPP). Efficient VMP techniques improve the resource utilization, power consumption and number of VM migrations in a cloud data center. Due to the significant and intrinsic difficulty, many models and algorithms have been proposed for VMPP. VMP algorithm is executed by considering the resources like CPU, RAM, Disk, etc. of VMs and PMs. Many researchers have designed VMP algorithms by considering the CPU capacity of a PM without considering the number of physical cores (pCPU) available in a PM and physical core capacity of a PM. i.e. VMs are mapped onto PMs, if and only if the sum of their CPU capacity does not exceed the CPU capacity of a PM. Such allocation results in pCPU overload which leads to performance degradation and violation of Service Level Agreement. However, in the real scenario, PM and VM consist of multiple physical cores. So, to place VM on PM, the CPU capacity of VM must be mapped to pCPUcapacity of a PM. i.e. the virtual core (vCPU) of a VM should be mapped to a pCPU of a PM. So, VM to PM mapping must be done based on core to core mapping and not on machine to machine mapping. Also, VMP using the total computational capacity of the PM leads to increase the number of core overloads and thus the host overload. More number of host overload leads to more number of VM migration. So, we have designed a VMP algorithm which will eradicate all the above mentioned problems of core overload, less resource utilization, more number of host overloads and more number of VM migrations. In our proposed work, we have considered a multi-core aware VMP by considering the core capacity of a VM and PM. In multi-core aware VMP, the CPU capacity of VM is checked against the pCPU capacity of PM, so that, the pCPU of a PM not get overloaded. So, that the overall performance is improved when compared to non-core aware virtual machine placement algorithms. Keywords: Multi-core · Virtual machine placement · Data center · Constraint programming

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 439–457, 2022. https://doi.org/10.1007/978-3-030-80119-9_26

440

Nagadevi and K. Raja

1 Introduction A cloud is a set of servers accessed over Internet or intranet and maintained in a multitenant environment facilitated by virtualization. Cloud computing is a term used to indicate the data centers (DC) having computer system resources including servers, data storage, databases, networking, software, and analytics available to many users ondemand over the internet without direct active management by the user. Cloud servers are always available to customers in online mode. Cloud providers back up their services on several machines all over the world. There are mainly three types of cloud services available to the users. They are: Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS) and Infrastructure-as-a-Service (IaaS). In recent years Function-as-a-Service (FaaS) has emerged as the fourth service [1]. 1.1 Virtualization Cloud Service Providers (CSP) like Amazon EC2, virtualizes their hardware resources to meet up the scalability level of applications demand. Virtualization is a key technology in cloud computing for resource sharing. Virtualization is the important building block of cloud infrastructure. Virtualization is the creation of a virtual version of something such as an OS, server, and storage or network resource. DC resources are virtualized to meet the increasing demand of the users [2]. 1.2 Virtual Machine Cloud as an on-demand model provides its resources in the form of Virtual Machines (VM). VMs are online instances of a physical machine (PM). The hypervisor, also known as Virtual Machine Monitor (VMM), allocates physical computing resources such as processors, memory, and storage to each VM.A hypervisor is a lightweight software layer to coordinate between it and the underlying physical hardware. Since the VM is sandboxed from the rest of the system, the software inside the VM cannot tamper with the host computer. VMs can also be used for other purposes such as server virtualization. In virtualization, VMM allows more than one operating system to be executed on the single PM. That is the VMs can share the same PM. VMM virtualizes the physical Central Processing Unit (CPU) to virtual CPU, on which guest operating systems run. VMM sits between the OS (operating system) level and hardware level in the virtualization system. Virtualization technology is used to make efficient use of hardware [3]. 1.3 Data Center Data center (DC) is a pool of resources. Cloud data centers make extensive use of virtualization techniques to meet up the growing demand for computation, storage and networking [4]. A CSP maintains all these resources in an efficient manner to make benefits. CSP provides resources to the users for rent directly using IaaS model. Proficient resource allocation method allows the CSP to grant resources to as many customers. Minimization of number of active PMs and switching of the idle resource will reduce the power consumption.

Multi-core Aware Virtual Machine Placement

441

1.4 VM Placement CSPs afford significant effort in re-optimizing the arrangement of VMs on PMs to save the power, and to improve the resource utilization of a cloud data center. The high resource utilization in a cloud data center leads to save more resources and that is being used for the forthcoming request from the end users which will increase the cost benefit to the CSP. So, the creation of VM in a suitable PM is the critical requirement of a CSP in cloud data center. In this context, deciding the right allocation of VMs into PMs is known as Virtual Machine Placement (VMP). Choosing the destination PM is a challenging task in cloud computing. Mapping the VM to an appropriate PM is called VMP. The VMP decision is required at different stages of a cloud data center. Finding the best PM to place/create a VM is popularly known as Virtual Machine Placement Problem (VMPP) [5]. Efficient VMP techniques improve the resource utilization, power consumption and number of VM migrations [6] in a cloud data center. The optimal placement of VMs into PMs, not only reduces the amount of hardware usage but also reduces the unused PMs by putting it into standby mode or even to shut down [7]. 1.5 Motivation Since the Cloud Computing model is based on pay-per-use on demand model, the VMs are dynamically created, used and destroyed on the go. Because of this dynamism the physical resources may be fragmented in smaller units and that may not be used for the future demands. An unused fragment of physical resources significantly increases the resource wastage and also reduces the resource utilization level of IaaS Cloud. Thus, the overall performance of the DC is affected. PM consolidation plays an important role in DC resource management. It improves the resource usage of a DC to a greatest level. It is very important to match the available PM capacity to the users demand in real time. With proper planning and monitoring of resources it is feasible to map the real time demand for resources to the equivalent server’s capacity. VMP and VM consolidation on the same PM leads to limits the number of active PMs of a DC significantly. An effective VMP and consolidation techniques improves the resource utilization of a DC by packing all the possible VMs into a minimum number of PMs. Underutilization of PMs could be optimized by rearrangement of VMs on the available PMs, called VM migration. An efficient VMP at the initial stage of resource allocation may lead to reduce the number of host overload and host under load. This kind of VMP techniques not only improve the resource utilization, also reduce the number of VM migration. So, that the overall performance of the system is improved. The above mentioned problem of resource wastage, idle servers, over provisioning and under provisioning can be addressed by improving server resource utilization by an effective VMP and Consolidation techniques. Due to the significant and intrinsic difficulty, many models and algorithms have been proposed for VMPP. VMP algorithm is executed by considering the resources like CPU, RAM, Disk etc. of VMs and PMs. Many researchers have designed VMP algorithms [8–11] by considering the CPU capacity of a PM without considering the number of physical cores (pCPU) available in a PM and physical core capacity of a PM. i.e. VMs are mapped onto PMs, if and only if the sum of their CPU capacity does not exceed the CPU capacity of a PM. Such allocation results in pCPU overload which leads to

442

Nagadevi and K. Raja

performance degradation and violation of Service Level Agreement. However, in the real scenario, PM and VM consist of multiple physical cores. So, to place VM on PM, the CPU capacity of VM must be mapped to pCPU capacity of a PM. i.e. the virtual core (vCPU) of a VM should be mapped to a pCPU of a PM. So, VM to PM mapping must be done based on core to core mapping and not on machine to machine mapping. For example, to place a VM with single vCPU requesting 500 MIPS (Million Instructions Per Second), we may consider a PM with two pCPU having 250 MIPS each. According to non-multi-core aware VMP, the PM having a total of 500 (2pCPU * 250) MIPS is sufficient to place a VM with 500 MIPS. But, a single core VM cannot take advantage of a dual core PM. The VM is assigned with a dedicated pCPU. The VM will only make use of a single core of a PM not both. Also, VMP using the total computational capacity of the PM leads to increase the number of core overloads and thus the host overload. More number of host overload leads to more number of VM migration. So, we have designed a VMP algorithm which will eradicate all the above mentioned problems of core overload, less resource utilization, more number of host overloads and more number of VM migrations. In our proposed work, we have considered a multi-core aware VMP by considering the core capacity of a VM and PM. In multi-core aware VMP, the CPU capacity of VM is checked against the pCPU capacity of PM, so that, the pCPU of a PM not get overloaded. So, that the overall performance is improved when compared to non-core aware virtual machine placement algorithms. Considering the multi-core aware VMP algorithms in cloud data centers, very few research works [12–15] have been done by the researchers. Even in the few works either the VMP algorithms does not considered the cloud environment or does not considered the multi-dimensional resources and so on. No definite model is developed and no heuristic measure is suggested to evaluate the performance of the multi-core aware VMP algorithms in cloud data centers. This motivated us to choose the multi-core aware VMPP in a cloud data center. 1.6 Problem Statement Multi-core aware VMP algorithm is required to minimize the core overload, to improve resource utilization, to assign dedicated core for a VM and to reduce power consumption. The problem space consists of n, number of virtual machines, VM-List = {vm1 , vm2 … vmn } and m, number of physical machines, PM-List = {pm1 , pm2, … pmm }. Each VM is represented as a d-dimensional vector of resource capacity as, VM = (ncores, mips, mem). Similarly, PM is represented as PM = (ncores, mips, mem), where mips is the CPU capacity, ncores is the number of cores, mem is the physical memory capacity. The requirement is to find a map F, Map F:

VM-List

PM-List

ncores

between PM-List, VM-List and ncores such that all the constraints and objective functions are satisfied. The mapping of VMs on PMs must satisfy the following constraints:

Multi-core Aware Virtual Machine Placement

443

• All the given VMs must be placed on at-least one PM • The total resource capacity of all VMs that are placed on a PM must be less than or equal to the total resource capacity of a PM in all dimensions • The required resource capacity of a VM that has to be placed on a PM must be less than or equal to the pCPU capacity of a PM. The following contributions are formulated and implemented: 1. To devise a model, based on constraint programming for multi-core aware VMPP. 2. To develop a multi-core aware VMP algorithm (MCA-VMP) for cloud data centers.

2 Related Work In [10], authors have designed multi-objective VMP algorithm using multi-dimensional resource usage model. Resource Utilization Factor (RUF) is used in MOVMP to determine the target PM to place a VM. The goal of the algorithm is to improve resource utilization in a balanced manner and also, to minimize resource wastage, power consumption and SLA violations of cloud data centers. In [11] power usage effectiveness was used to measure and reduce the carbon footprint and cost. Authors didn’t use multicore aware model. In [12], author used multi-core processors, to place the VM on a core of a multi-core PMs with constraint programming to balance solution quality and time compared to non-multi-core aware heuristic. Algorithm considers only the CPU in a single dimension. Multi-dimensional resources are not considered. In [13] illustrates various challenges and issues that must be addressed in multi-core cloud data centers and brings up the programming requirement and parallelism among the tasks, describes how the muti-core model will benefit cloud computing. Here the authors not considered multi-core aware VMP. In [14] the goal of the author is to map VM cores to PM cores and benchmark applications are used to minimize the number of active hosts, to maximize the system throughput and to mitigate the resource contention by conducting worst-fit mapping of VMs to physical cores. But virtualized cloud environment is not considered, no specific heuristic is used, no quantitative model is used and real cloud dataset is not used. In [15] Cloudsim was used to analyse the power consumption, number of VM migrations on synthetic data. But scheduling considers only the total CPU capacity of the PM, not the core capacity. In [16] VMP decisions are made based on the following: Reservation, On-Demand and Spot Market. On-Demand VMP decisions are based on initial VMP and migration of VMs. Both of the above steps can be considered for power conservation, SLA, revenue maximization and reliability. The resource allocation problems in heterogeneous DC are categorized into three types [17]: based on VM, based on PM, and based on application. Their study shows, the proactive resource allocation by predicting the future workload enhances the performance of resource management. It also confirms that finding the minimum number of active PMs to satisfy the resource demand is the optimal solution to reach maximum resource utilization. In [18], authors have presented a publicly available cloud dataset to reflect a real workload from Google cluster traces using Monte Carlo Simulation. The

444

Nagadevi and K. Raja

dataset generated as Google Cloud Jobs (GoCJ) dataset. The GoCJ dataset consists of jobs of different types as small, medium, large, extra-large, and huge in varying compositions. In [19], the authors have presented the stochastic characteristics of resource usages and proposed a solution, probabilistically, to handle resource overload during migration. The proposed stochastic load balancing algorithm used multi-dimensional resource request to measure migration cost by considering network topology and minimizes the migration overhead. In [20] reallocation of VMs performed to minimize the number of physical nodes. Idle nodes switched-off to decrease the power consumption. MBFD used for VM provisioning and VMP. VMs selected for migration based on MBFD. In [21] initial VMP carried out on an unloaded DC to reduce resource wastage (CPU, Memory, Bandwidth), power consumption and to minimize SLA violation. ACO (Ant Colony Optimization) used over GA and heuristic method. In [22] the authors have developed algorithm to increase resource utilization, to meet SLA requirement and to reduce number of PMs. Pearson Correlation Coefficient between a pair of VMs is considered. As a result CPU utilization increased, SLA violation decreased, number of active servers reduced. Authors in [6] maps VMs to PMs as a part of VM Migration to minimize the energy by shutting down services. In [23], designs an online VMP algorithm to increase cloud providers revenue for the multi-dimensional resources. In [24], authors have solved virtual resource provisioning and allocation problem using a constraint programming model and the model is validated through simulation. They have achieved improvisation in QoS violation and resource usage costs.VM provisioning using constraint programming reduces the cost of the utilized resources. Similarly, VM packing using constraint programming reduces the number of active PMs. They have considered CPU, RAM and Bandwidth for the analysis. But, here the number of cores of a PM is not being considered. In [25] designed a constraint programming based VMP. FFD algorithm is to reduce the number of active PMs and to improve the resource utilization as well. But, here the number of cores of PMs is not being considered. In most of the above mentioned VMP algorithms except [12], authors considered the overall CPU capacity of a PM, without considering the number of cores available in a PM and pCPU capacity of a PM. i.e. VMs are mapped onto PMs, if their CPU capacity is less than or equal to the total CPU capacity of a PM. Such an allocation results in PM overload which leads to performance degradation and violation of Service Level Agreement.

3 Multi-core Aware VMP Model using Constraint Programming (MCA-VMP) In our proposed system, we assumed a cloud data center with fixed number of heterogeneous PMs. i.e. each PM is considered to be having varying capacities in the dimensions CPU, memory and number of pCPUs. Multiple pCPUs are contained by a single multicore PM. PMs are virtualized using hypervisor in a cloud data center. Multiple VMs can be allocated to a pCPU of a PM as long as the capacity requested by all VMs is less than the capacity of a pCPU of a PM. VMs are assumed to have a single core (vCPU). Multiple applications can be executed on a single VM. The proposed system architecture is shown in Fig. 1.

Multi-core Aware Virtual Machine Placement

445

3.1 Resource Allocation Manager The request from users for VMs is stored in VM Queue. The resource allocation manager is responsible for fairly allocating the resources for VMs requested by the users from a DC for VMs. Resource allocation manager identifies the pCPU of a PM with the required CPU and memory capacity, to place a new VM. Then, it allocates the required resources for that VM. We have proposed multi-core aware VMP algorithm (MCAVMP) to efficiently allocate VMs in PMs, to reduce the number of pCPU overload and to provide a dedicated pCPU for a VM. To place a VM in a multi-core PM, resource allocation manager invokes our MCA-VMP algorithm. MCA-VMP algorithm consists of two phases. First phase is PM Filter and the second phase is Core Mapper. In step1, the PM filter, selects list of all PMs that has sufficient CPU and memory capacity, to place a requested VM, i.e. the PMs which satisfies resource constraint 1 and resource constraint 2 are filtered out. In step 2, PM Filter chooses the target PM using First Fit (FF) strategy from the list of PMs resulted from the step1, to place a VM.

Fig.1. Proposed system architecture

Core Mapper finds the target pCPU from the target PM using FF strategy such that, the pCPU is sufficient enough to place a new VM in it, along with its current load, i.e. it chooses a target pCPU that satisfies resource constraint 4. Thus, the Core Mapper finds a suitable pCPU of a specific PM to place a new VM.

446

Nagadevi and K. Raja Table 1. Notations and terminologies used. X

Set of variables in the VMP problem

VM-List

Set of VMs, of size n

PM-List

Set of PMs, of size m

v

Number of VMs placed on a PM

vmncores i vmmem i mips vmi

Number of cores in a VM, i

pmncores j

Number of cores in PM, j

pmmem j

Memory capacity of PM, j

Memory capacity of VM, i vCPU capacity of a VM, i

c−mips pCPU capacity of PM, j

pmj xij

1 if VM i placed on PM j, else 0

pj

1 if PM j is active, else 0

zijk

1 if VM i is placed on kth core of a PM j, else 0

cjk

1 if kth core of PM j is active, else 0

cpu

Cj

Total CPU capacity of PM, j

3.2 MCA-VMP Problem Formulation The mathematical formulation of the multi-core aware VMP problem is presented here. The notations and terminologies used in the proposed model are given in Table 1. Relating the VMPP with the constraint programming, and based on the model described by [24, 25], the problem of multi-core VMPP could be modeled as follows: In multi-core VMP, the VMs must be mapped to the specific pCPU of a PM since all the PMs are multi-core machines. Multi-core aware VMP consists of two steps. The first step is to find a suitable PM for a VM and the second step is to find a specific pCPU of a PM to place a VM. The mapping of VM onto a specific processing element/core ensures that, VM is committed to that pCPU. The pCPU assigned for that VM is responsible for executing the application assigned to that VM. So, each and every VM will have a dedicated pCPU for it. So that, no VM will be waiting for resources in queue for other VMs to finish. Because of this dedicated pCPU, no pCPU of PM will be overloaded. So, pCPU overload is reduced which in turn reduces the host overload which in turn reduces the VM migrations. Consider, there are m, number of physical machines, such as PM-List = {pm1 , pm2 … pmm } and n, number of virtual machines such as VM-List = {vm1 , vm2 … vmn } [11, 28–30]. VMPP is the multi-dimensional bin packing problem [26]. Each PM is represented as PM < ncores, core-mips, mem, > where ‘core-mips’ represents the CPU

Multi-core Aware Virtual Machine Placement

447

capacity of a pCPU of a PM, ‘mem’ represents the RAM capacity of a PM and ‘ncores’ represents the number of pCPU of a PM. Each VM is represented as VM < ncores, core-mips, mem, >. ‘core-mips’ represents the CPU capacity of a single vCPU of a VM, ‘mem’ represents the RAM capacity required by a VM and ‘ncores’ is the number of vCPU of a VM. So, given the variables, domains and the decision variables: X = {VM − List, PM − List} where X is the set of all variables of the VMPP to be solved and. VM − List = {vm1 , vm2 , . . . . . . , vmn } is the set of VMs that has to be placed on a set of all PMs in, PM − List = {pm1 , pm2 , . . . . . . , pmm } Each VM, vmi in VM − List is assumed to be having the following configurations as, mips

vmi < vmncrores , vmi i

, vmmem > i mips

where vmncores is the number of cores of vmi and its value is an integer > 0, vmi is i is the memory capacity of a vm in KB. the vCPU capacity of vmi in MIPS and vmmem i i Each PM, pmj in PM − List is assumed to be having the following configurations as, c−mips

, pmj pmj < pmncrores j

, pmmem > j c−mips

is the number of cores of a pmj and its value is an integer > 0, pmj where pmncores j is the pCPU capacity of pmj in MIPS, pmmem is the memory capacity of a pmj in KB. j The requirement is to find a map F, Map F:

VM-List

PM-List

ncores

between PM-List, VM-List and ncores/Processing Elements such that to satisfy all the below mentioned constraints and objective functions. 3.3 MCA-VMP with Constraint Programming MCA-VMP is designed to satisfy the following resource constraints and placement constraints. Resource constraints ensure that the resources allocated to a VM are within the capacity of a DC PMs. The following four decision variables are used to determine the VM placement status. Placement Decision Variable 1: Every VM has to be assigned to some PM. It is represented using a binary decision variable, xij = {0, 1} as,  1, if vmi placed on pmj , ∀ i ∈ VM − List, ∀ j ∈ PM − List xij = (1) 0, Otherwise

448

Nagadevi and K. Raja

Placement Decision Variable 2: A binary decision variable zijk vmi is placed on k th core of pmj ,  zijk =

{0, 1},

=

is

used

indicate

1, if vmi placed on k th core of pmj , ∀ i ∈ VM − List, ∀ j ∈ PM − List, ∀ k ∈ pmncrores j 0,

Otherwise

whether

(2)

Active Decision Variable 1: A binary decision variable pj = {0, 1} is used to denote whether ±j is active or not.  1 if pmj is active, ∀ j ∈ PM − List j p = (3) 0, Otherwise Active Decision Variable 2: A binary decision variable cjk = {0, 1} is used to denote whether k th core of ±j is active or not.  1 if k th core of pmj is active, ∀ j ∈ PM − List k (4) , ∀k ∈ pmncores cj = j 0, Otherwise The objective function of our proposed system is to minimize the Resource Wastage (RW) and Power Consumption (Power) of m PMs and is given by, m RW (5) min j=1

min

m j=1

Power

(6)

subject to the constraints, Resource Constraint 1: v i=1

vmmem · xij ≤ pmmem , 1≤j≤m i j

(7)

It guarantees that the memory demands of all the VMs placed on any PM should not exceed its memory capacity. Resource Constraint 2: mips

vmi

c−mips

· xij ≤ pmj

, 1 ≤ i ≤ n, 1 ≤ j ≤ m

(8)

It guarantees that the CPU demands of a VM i, placed on any PM j should not exceed its pCPU capacity. Resource Constraint 3: v i=1

mips

vmi

c−mips

· xij ≤ pmncores pmj j

, 1 ≤ j ≤ m

(9)

Multi-core Aware Virtual Machine Placement

449

It guarantees that the sum of CPU demands of all the VMs placed on any PM should not exceed its CPU capacity. Resource Constraint 4: v i=1

mips

vmi

c−mips

· zijk ≤ pmj

, 1 ≤ j ≤ m, 1 ≤ k ≤ pmncores j

(10)

It guarantees that the CPU demands of all the VMs placed on any core of any PM should not exceed its pCPU capacity. Service Constraint 1: m j=1

xij = 1, 1 ≤ i ≤ n

(11)

This constraint ensures that each VM can be placed on exactly one PM. Service Constraint 2: m pmncores j j=1

k=1

zijk = 1, 1 ≤ i ≤ n

(12)

It guarantees that each VM can be placed on exactly one core of one PM. 3.4 Performance Evaluation To calculate the overall CPU utilization we have used the formula (13) to represent the multi-cores in a PM and also, according to [8] and [10], where the authors calculated utilization without considering the cores. So, the overall CPU Utilization of m, number of PMs is, n mips m xij 1 cpu i=1 vmi j P · (13) Um,pm = m j c−mips ncores j=1 p pmj pmj j=1 cpu

where Um,pm is the overall CPU utilization of m, number of PMs in a cloud data center. The Normalized pCPU Utilization is calculated as, n  cpu mips c−mips vmi · zijk /pmj Uj,k = P j · cjk · (14) i=1

cpu

where, Uj,k is the Normalized pCPU Utilization of kth core of a jth PM. It is simply stated as the ratio between the sum of CPU capacity of all the VMs placed on a specific active core of a specific active PM to its pCPU capacity. The total number of active cores, in the proposed cloud data center is calculated as, m pmncores j j=1

k=1

pj · cjk

(15)

450

Nagadevi and K. Raja

where pj says whether the pmj is active or not and cjk says whether kth core of a jth PM is active or not. Similarly, the overall Memory Utilization of m, number of PMs is calculated as, n m vmmem xij 1 mem i = m j P j · i=1 mem (16) Um,pm j=1 pm p j=1 j mem is the overall memory utilization of m, number of PMs in a proposed cloud where Um,pm data center. The potential cost of resource wastage of a PM in a data center, in multiple dimensions like CPU, memory, disk and so on, is proposed by [27] is,    p  Lj − Lm j + ∈ (17) Wj = p Uj + Ujm p

p

where Uj and Ujm are the normalized CPU and memory resource usage. Lj and Lm j are the normalized remaining CPU and memory resource. ε is the very small positive real number and its value is set to 0.0001. This is adopted by [8, 10], and reframed for two dimensional and multi-dimensional resources. We adopted [8, 10, 27, 34] to calculate the resource wastage (RW) of all cores in all PMs in two dimensions (CPU, Memory) in a cloud data center and is given as, ⎡ ncores c−mips n ⎤  mips pmj pmj − i=1 vmi xij pmmem − ni=1 vmmem xij j i − + ∈⎥ c−mips ⎢ pmmem m m pmncores pmj j ⎢ ⎥ j RW = Pj ⎢ ⎥ n n mips mem x j=1 j=1 ⎣ ⎦ vm x vm ij ij i=1 i=1 i i c−mips + ncores pmmem pmj

pmj

j

(18)  where m j=1 RW is the sum of resource wastage of m number of PMs in two dimensions in a proposed system. According to [5, 31–33], CPU utilization is the crucial factor in determining the power consumption of a PM. The relationship between power consumption and CPU utilization is linear. Based on this, [27] proposed a power consumption model as follows:  busy Pj − Pjidle ∗ UjP + Pjidle , UjP > 0 (19) Pj = 0, otherwise busy

and Pjidle are the power consumption of jth PM at full utilization and at idle where Pj time. Following [8, 10, 27, 32, 34], we designed, the overall Power Consumption of a cloud data center as, ⎡ ⎤   n vmmips . x m m ij busy i=1 i Power = P j · ⎣ powj − powjidle + powjidle ⎦ c−mips j=1 j=1 pmncores pm j j (20)

Multi-core Aware Virtual Machine Placement

where

451

m

j=1 Power is the power consumption of m, number of PMs in proposed system. busy are the power consumption of PM j at busy and idle time. powj , powidle j

3.5 MCA-VMP Algorithm MCA-VMP algorithm works as follows. Initially, the CPU load of all the pCPU of all the PMs is initialized to zero. Then, we select a VM i, from VM-List from the beginning. Here, VMs are assumed to be having one vCPU. Then, our algorithm filters, a set of candidate PMs from PM-List, that has pCPU and memory capacity greater than or equal to the demand of VM i. Thus, our algorithm ensures resource constraint 1 and resource constraint 2. By checking for the pCPU capacity availability, we are allowing only the PMs with pCPU capacity greater than or equal to CPU demand of VM i, for further processing, i.e. it checks whether the pCPU of any PM is equal to or greater than that needed by a VM i. Also, for the memory, the memory demand of a VM is checked against the memory of a PM. Here, the memory is not fixed for pCPU of a PM. Instead, memory is sharable among all the pCPU of a PM. So it checks that the remaining memory capacity of a PM to place a VM i. Thus our algorithm allows only a candidate PMs. As a next step our algorithm proceeds to find a specific pCPU of a specific PM from the candidate PMs, which is having a sufficient mips (using First Fit method) to place a new VM i, along with its current CPU load, i.e. after placing the VM i, the CPU load of that pCPU should not exceed its capacity. Thus, our algorithm ensures that the pCPU of a PM is not overcommitted and not overloaded. Also, ensures the dedicated core for a VM i. The decision variable zijk , ensures that VM i placed on kth core PM j. The proposed MCA-VMP algorithm minimizes the number of core overloads in a cloud data center. Thus, it reduces the number of host overload.

452

Nagadevi and K. Raja

Algorithm 1 :

Multi-Core Aware Virtual Machine Placement Algorithm(MCA-VMP)

Input: VM-List, PM-List; Output: Placement of VMs in PM-List € PM-List for each for each pCPU , k= (1,…, ) =0; end for end for € VM-List for each € PM-List for each if ( β > γ. Note that the processing time of a task can have any distribution with the given means, but the heavytraffic delay optimality of Balanced-PANDAS algorithm is only proven under Geometric service time distribution, while MaxWeight based algorithm does not have a general heavy-traffic delay optimality under any distribution for service time. Capacity Region Characterization: An arrival rate for all task types is supportable for service by the M servers if and only if the load on each server is strictly less than the capacity of the server. Considering a processing rate of one ¯ ∈ L) is in the capacity region for each server, an arrival rate vector λ = (λL¯ : L of the system if and only if:  λL,m ¯ + α ¯ ¯

L:m∈L

 ¯ ¯k L:m∈ L

 λL,m λL,m ¯ ¯ + < 1, ∀m ∈ M, β γ ¯ ¯ L:m∈Lr

¯ that are processed by where λL,m is the rate of incoming tasks of type L ¯ server m.

Robustness Comparison of MapReduce Scheduling Algorithms

3

499

Load Balancing Algorithms

In this section, we introduce the main three algorithms on scheduling for data centers with more than or equal to two levels of data locality that have theoretical guarantees on optimality in some senses and under some conditions. In order to introduce the load balancing algorithm of each method, we also need to present the queueing structure required for that method. The three algorithms are 1. Priority algorithm by [62], which is best fit for applications with two levels of data locality, e.g. for the cases that only data locality is taken into account. An example is scheduling for Amazon Web Services inside a rack. 2. Balanced-PANDAS by [63, 65, 69], which is the state-of-the-art for scheduling applications with multiple levels of data locality and is observed to perform better in terms of average task completion time by fourfold in comparison to MaxWeight based algorithms. It is proven by [63] that under mild conditions, Balanced-PANDAS is both throughput and heavy-traffic delay optimal for a system with three levels of data locality and a rack structure. 3. MaxWeight based algorithms by [53,58], which can be used for multiple levels of data locality and are throughput optimal, but not heavy-traffic delay optimal, and it is observed that they generally have poor performance at high loads compared to weighted workload based algorithm used in BalancedPANDAS algorithm. The following three subsections present a complete introduction to these three main algorithms. 3.1

Priority Algorithm

The Priority algorithm is designed for a system with two levels of data locality. In other words, it only considers data locality, but not the rack structure. Hence, there are only local and remote servers from the perspective of a task. The queuing structure under this algorithm is to have a single queue per server, where the queue corresponding to a server only keeps tasks that are local to that server. At the arrival of a task, a central scheduler routes the incoming task to the local server with the shortest queue length. An idle server is scheduled to process a task in its corresponding queue as long as there is one, and if the idle server’s queue length is zero, it is scheduled to process a task from the longest queue in the system. The priority algorithm is proved to be both throughput and heavy-traffic delay optimal. However, its extension to more than two levels of data locality is not even throughput optimal, let alone heavy-traffic delay optimal. 3.2

Balanced-PANDAS Algorithm

The Balanced-PANDAS algorithm can be used for a system with multiple levels of data locality, but here we propose the algorithm for a data center with a rack

500

A. Daghighi and J. Q. Chen

structure with three levels of data locality. The queuing structure under using this algorithm is to have three queues per server, one queue for storing local tasks to the server, another queue for storing rack-local tasks to the server, and a third queue for storing remote tasks to the server. Hence, server m has a tuple of three  queues denoted by Qlm , Qkm , Qrm , where they refer to the queues storing local, rack-local, and remote tasks, respectively. The  corresponding queue lengths at time t are denoted by Qlm (t), Qkm (t), Qrm (t) . The workload on server m at time slot t is defined as follows: Wm (t) =

Qlm (t) Qkm (t) Qrm (t) + + . α β γ

¯ is routed to the corresponding queue of the server An incoming task of type L with the minimum weighted workload, where ties are broken randomly, in the set below:

Wm (t) arg min . α · 1{m∈L} ¯ + β · 1{m∈L ¯ k } + γ · 1{m∈L ¯r} m∈M An idle server m at time slot t is scheduled to process a local task from Qlm if Qlm (t) = 0; otherwise, it is scheduled to process a rack-local task from Qkm if Qkm (t) = 0; otherwise, it is scheduled to process a remote task from Qrm if Qrm (t) = 0; otherwise, it remains idle until a task joins one of its three queues. The Balanced-PANDAS algorithm is throughput optimal. It is also heavy-traffic delay optimal for a system with a rack structure of three levels of data locality if β 2 > α · γ, which means that the rack-local service is faster than the remote service in a specific manner. 3.3

MaxWeight Based Algorithm

The MaxWeight algorithm is proposed by [53] and a modification of it called JSQ-MaxWeight algorithm is proposed by [58], which is described below. Consider one queue per server, i.e. server m has a single queue called Qm , where its queue length at time slot t is denoted by Qm (t). The routing policy is as the ¯ is routed to the queue of the Priority algorithm, i.e. an incoming task of type L ¯ This routing policy is called join-the-shortest-queue (JSQ). shortest length in L. An idle server m at time slot t on the other hand is scheduled to process a task from a queue with the maximum weighted queue length, where ties are broken at random, in the set below:   arg max α · 1{n=m} + β · 1{n=m,R(n)=R(m)} + γ · 1{R(n)=R(m)} · Qn (t) . n∈M

The JSQ-MaxWeight algorithm is throughput optimal, but it is not heavy-traffic delay optimal.

Robustness Comparison of MapReduce Scheduling Algorithms

501

Fig. 1. Comparison of the algorithms using the precise value of parameters.

4

Robustness Comparison of Scheduling Algorithms

In this section, we present the results on our extensive simulations on robustness of scheduling algorithms presented in Sect. 3. To this end, we run the algorithms with parameters that have error to study which algorithm can tolerate errors better than others. More specifically, we use incorrect α, β, and γ in the algorithms for calculating weighted workloads or weighted queue lengths and observe the average task completion time under these scenarios. The arrival process is a Poisson process and the processing time has an exponential distribution. We have also tested the algorithms for processing times with heavy-tailed distributions and observed similar results. We make these parameters 5%, 10%, 15%, 20%, 25%, and 30% off their real value, either greater than the real value or smaller than the real value, and evaluate algorithms under these cases. The traffic load is assumed the same under all algorithms so that the comparison makes sense. We further compare all the three algorithms mentioned in Sect. 3 with the Hadoop’s default scheduler which is First-In-First-Out (FIFO). Figure 1 shows the comparison between the four algorithms when precise value of parameters are known by the central scheduler. As we see, Balanced-PANDAS algorithm has the lowest average task completion time at high loads. A closer look of high loads is presented in Fig. 2, where Balanced-PANDAS obviously outperform JSQ-MaxWeight in terms of average task completion time. The performance of the four algorithms are compared to each other when the parameters are lower than their real values by certain percentages, where the results are shown in Fig. 3. As is seen, Balanced-PANDAS has best performance among all algorithms by changing the parameters’ error from 5% to 30%. In fact, Fig. 4 shows that the Balanced-PANDAS has the least sensitivity against change of parameters while JSQ-MaxWeight’s performance varies notably by the increase of error in parameter estimations. Comparison of the algorithms when the parameters are off for some percentages, but higher than their real values are given in Fig. 5. It is again observed that the Balanced-PANDAS algorithm has consistent better performance than

502

A. Daghighi and J. Q. Chen

Fig. 2. Comparison of balanced-PANDAS and JSQ-MaxWeight at high loads using the precise value of parameters.

(a) Parameters are off for 5% lower

(b) Parameters are off for 10% lower

(c) Parameters are off for 15% lower

(d) Parameters are off for 20% lower

(e) Parameters are off for 25% lower

(f) Parameters are off for 30% lower

Fig. 3. Robustness comparison of algorithms when parameters are off and lower than their real values.

Robustness Comparison of MapReduce Scheduling Algorithms

503

Fig. 4. Sensitivity comparison of balanced-PANDAS and JSQ-MaxWeight against parameter estimation error.

(a) Parameters are off for 5% higher

(b) Parameters are off for 10% higher

(c) Parameters are off for 15% higher

(d) Parameters are off for 20% higher

(e) Parameters are off for 25% higher

(f) Parameters are off for 30% higher

Fig. 5. Robustness comparison of algorithms when parameters are off and higher than their real values.

504

A. Daghighi and J. Q. Chen

Fig. 6. Sensitivity comparison of balanced-PANDAS and JSQ-MaxWeight against parameter estimation error.

the JSQ-MaxWeight algorithm. The sensitivity comparison of the BalancedPANDAS and JSQ-MaxWeight algorithms in this case is presented in Fig. 6.

5

Conclusion and Future Work

In this work, we did a literature review on both classical and state-of-the-art scheduling algorithms for the affinity scheduling problem. Data center load balancing is a special case of the affinity scheduling problem. Considering the rack structure of data centers, there are three levels of data locality. The priority algorithm that is heavy-traffic delay optimal is not even throughput optimal for three levels of data locality. The Balanced-PANDAS algorithm is the state-of-the-art in heavy-traffic delay optimality. We investigated the robustness of BalancedPANDAS and JSQ-MaxWeight algorithms with respect to errors in parameter estimation. We observe that Balanced-PANDAS keeps its better performance even in the absence of precise parameter values versus JSQ-MaxWeight. Note that the JSQ-MaxWeight algorithm is also robust under parameter estimation errors, but it is more sensitive than Balanced-PANDAS, specially at high loads close to the boundary of the capacity region. For future work, one can use machine learning tools to estimate the system parameters and make them more precise in the meanwhile that the load balancing algorithm is working with the estimated parameters. The scheduling algorithms presented in this work can also be applied to a vast number of applications including but not limited to healthcare and super market models as discussed by [9, 17, 24, 61], web search engines by [5, 30,50, 51,64], electric vehicle charging by [1,2, 7,8, 15, 18, 44, 45, 56] and so on.

Robustness Comparison of MapReduce Scheduling Algorithms

505

References 1. Alinia, B., Sadegh Talebi, M., Hajiesmaili, M.H., Yekkehkhany, A., Crespi, N.: Competitive online scheduling algorithms with applications in deadline-constrained EV charging. In: 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), pp. 1–10. IEEE (2018) 2. Almalki, M.M., Chehardeh, M.I., Hatziadoniu, C.J.: Capacitor bank switching transient analysis using frequency dependent network equivalents. In: North American Power Symposium (NAPS), 2015. IEEE (2015) 3. Bell, S., Williams, R., et al.: Dynamic scheduling of a parallel server system in heavy traffic with complete resource pooling: asymptotic optimality of a threshold policy. Electr. J. Probab. 10, 1044–1115 (2005) 4. Bell, S.L., Williams, R.J., et al.: Dynamic scheduling of a system with two parallel servers in heavy traffic with resource pooling: asymptotic optimality of a threshold policy. Ann. Appl. Probab. 11(3), 608–649 (2001) 5. Broder, A.: A taxonomy of web search. In: ACM SIGIR Forum, vol. 36, pp. 3–10. ACM (2002) 6. Byers, J.W., Considine, J., Mitzenmacher, M.: Geometric generalizations of the power of two choices. In: Proceedings of the Sixteenth Annual ACM Symposium on Parallelism in Algorithms and Architectures, pp. 54–63. ACM (2004) 7. Chehardeh, M.I., Almalki, M.M., Hatziadoniu, C.J.: Remote feeder transfer between out-of-phase sources using STS. In: Power and Energy Conference at Illinois (PECI), 2016 IEEE. IEEE (2016) 8. Chehardeh, M.I., Hatziadoniu, C.J.: A systematic method for reliability index evaluation of distribution networks in the presence of distributed generators. In: 2018 North American Power Symposium (NAPS). IEEE (2018) 9. Clower, R., Leijonhufvud, A.: The coordination of economic activities: a Keynesian perspective. Am. Econ. Rev. 65(2), 182–188 (1975) 10. Cooper, C., Els¨ asser, R., Radzik, T.: The power of two choices in distributed voting. In: International Colloquium on Automata, Languages, and Programming, pp. 435–446. Springer, Berlin (2014) 11. Daghighi, A.: Application of an artificial neural network as a third-party database auditing system (2019) 12. Daghighi, A., Kavousi, M.: Scheduling for data centers with multi-level data locality. In: 2017 Iranian Conference on Electrical Engineering (ICEE), pp. 927–936. IEEE (2017) 13. Dahlgaard, S., Knudsen, M.B.T., Rotenberg, E., Thorup, M.: The power of two choices with simple tabulation. In: Proceedings of the twenty-seventh annual ACMSIAM symposium on Discrete algorithms, pp. 1631–1642. SIAM (2016) 14. Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008) 15. Deilami, S., Masoum, A.S., Moses, P.S., Masoum, M.A.S: Real-time coordination of plug-in electric vehicle charging in smart grids to minimize power losses and improve voltage profile. IEEE Trans. Smart Grid 2(3), 456–467 (2011) 16. Doerr, B., Goldberg, L.A., Minder, L., Sauerwald, T., Scheideler, C.: Stabilizing consensus with the power of two choices. In: Proceedings of the Twenty-Third Annual ACM Symposium on Parallelism in Algorithms and Architectures, pp. 149–158. ACM (2011) 17. Eisenhauer, E.: In poor health: supermarket redlining and urban nutrition. GeoJournal 53(2), 125–133 (2001)

506

A. Daghighi and J. Q. Chen

18. Gan, L., Topcu, U., Low, S.H.: Optimal decentralized protocol for electric vehicle charging. IEEE Trans. Power Syst. 28(2), 940–951 (2013) 19. Gardner, K., Harchol-Balter, M., Scheller-Wolf, A., Velednitsky, M., Zbarsky, S.: Redundancy-d: the power of d choices for redundancy. Oper. Res. 65(4), 1078–1094 (2017) 20. Gast, N.: The power of two choices on graphs: the pair-approximation is accurate? ACM SIGMETRICS Perform. Eval. Rev. 43(2), 69–71 (2015) 21. Harrison, J.M.: Heavy traffic analysis of a system with parallel servers: asymptotic optimality of discrete-review policies. Ann. Appl. Probab. 822–848 (1998) 22. Harrison, J.M., L´ opez, M.J.: Heavy traffic resource pooling in parallel-server systems. Queueing Syst. 33(4), 339–368 (1999) 23. He, C., Lu, Y., Swanson, D.: Matchmaking: a new MapReduce scheduling technique. In: 2011 IEEE Third International Conference on Cloud Computing Technology and Science (CloudCom), pp. 40–47. IEEE (2011) 24. Hosseini, M., Jiang, Y., Yekkehkhany, A., Berlin, R.R., Sha, L.: A mobile geocommunication dataset for physiology-aware dash in rural ambulance transport. In: Proceedings of the 8th ACM on Multimedia Systems Conference. ACM (2017) 25. Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed dataparallel programs from sequential building blocks. In: ACM SIGOPS Operating Systems Review, vol. 41, pp. 59–72. ACM (2007) 26. Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., Goldberg, A.: Quincy: fair scheduling for distributed computing clusters. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, pp. 261–276. ACM (2009) 27. Jin, J., Luo, J., Song, A., Dong, F., Xiong, R.: Bar: an efficient data locality driven task scheduling algorithm for cloud computing. In: Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 295–304. IEEE Computer Society (2011) 28. Kavousi, M.: Affinity scheduling and the applications on data center scheduling with data locality. arXiv preprint arXiv:1705.03125 (2017) 29. Kreimer, J.: Real-time system with homogeneous servers and nonidentical channels in steady-state. Comput. Oper. Res. 29(11), 1465–1473 (2002) 30. Krishna, K.H., Rani, K.A.: Reducing the energy consumption energy-efficient query processing node in web search engines (2018) 31. Livny, M., Melman, M.: Load balancing in homogeneous broadcast distributed systems. In: ACM SIGMETRICS Performance Evaluation Review, vol. 11, pp. 47–55. ACM (1982) 32. Lowery, J.C., Collins, M.A., Schroeder, B.: Systems and methods for provisioning homogeneous servers. US Patent App. 11/562,921, 22 May 2008 33. Luczak, M.J., McDiarmid, C., et al.: On the power of two choices: balls and bins in continuous time. Ann. Appl. Probab. 15(3), 1733–1764 (2005) 34. Lumetta, S., Mitzenmacher, M.: Using the power of two choices to improve bloom filters. Internet Math. 4(1), 17–33 (2007) 35. Mandelbaum, A., Stolyar, A.L.: Scheduling flexible servers with convex delay costs: heavy-traffic optimality of the generalized cµ-rule. Oper. Res. 52(6), 836–855 (2004) 36. Meyn, S.: Stability and asymptotic optimality of generalized MaxWeight policies. SIAM J. Control Optim. 47(6), 3259–3294 (2009) 37. Mitzenmacher, M.: The power of two choices in randomized load balancing. IEEE Trans. Parallel Distrib. Syst. 12(10), 1094–1104 (2001)

Robustness Comparison of MapReduce Scheduling Algorithms

507

38. Moaddeli, A., Ahmadi, I.N., Abhar, N.: The power of d choices in scheduling for data centers with heterogeneous servers. arXiv preprint arXiv:1904.00447 (2019) 39. Musavi, N.: A game theoretical framework for the evaluation of unmanned aircraft systems airspace integration concepts. arXiv preprint arXiv:1904.08477 (2019) 40. Musavi, N., Onural, D., Gunes, K., Yildiz, Y.: Unmanned aircraft systems airspace integration: a game theoretical framework for concept evaluations. J. Guidance Control Dyn. 40, 96–109 (2016) 41. Musavi, N., Tekelio˘ glu, K.B., Yildiz, Y., Gunes, K., Onural, D.: A game theoretical modeling and simulation framework for the integration of unmanned aircraft systems in to the national airspace. In: AIAA Infotech@ Aerospace (2016) 42. Polo, J., et al.: Resource-aware adaptive scheduling for MapReduce clusters. In: ACM/IFIP/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing, pp. 187–207. Springer, Berlin (2011) 43. Richa, A.W., Mitzenmacher, M., Sitaraman, R.: The power of two random choices: a survey of techniques and results. Comb. Optim. 9, 255–304 (2001) 44. Saadatmand, S., Azizi, S., Kavousi, M., Wunsch, D.: Autonomous control of a line follower robot using a q-learning controller. In: 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), pp. 0556–0561. IEEE (2020) 45. Saadatmand, S., Kavousi, M., Azizi, S.: The voltage regulation of boost converters using dual heuristic programming. In: 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), pp. 0531–0536. IEEE (2020) 46. Saadatmand, S., Sanjarinia, M.S., Shamsi, P., Ferdowsi, M.: Dual heuristic dynamic programing control of grid-connected synchronverters. In: North American Power Symposium (NAPS), 2019, pp. 1–6 (2019) 47. Saadatmand, S., Sanjarinia, M.S., Shamsi, P., Ferdowsi, M., Wunsch, D.C.: Heuristic dynamic programming for adaptive virtual synchronous generators. In: North American Power Symposium (NAPS), 2019, pp. 1–6 (2019) 48. Sadiq, B., De Veciana, G.: Throughput optimality of delay-driven MaxWeight scheduler for a wireless system with flow dynamics. In: 47th Annual Allerton Conference on Communication, Control, and Computing, Allerton, pp. 1097–1102. IEEE (2009) 49. Salehi, M.: Optimal physiology-aware scheduling of clinical states in rural ambulance transport. In: 2017 International Conference on Inventive Computing and Informatics (ICICI), pp. 247–252. IEEE (2017) 50. Salehi, S., Du, J.T., Ashman, H.: Use of web search engines and personalisation in information searching for educational purposes. Inf. Res. Int. Electr. J. 23(2), n2 (2018) 51. Schwartz, C.: Web search engines. J. Am. Soc. Inf. Sci. 49(11), 973–982 (1998) 52. Singh, V.P.: Two-server Markovian queues with balking: heterogeneous vs. homogeneous servers. Oper. Res. 18(1), 145–159 (1970) 53. Stolyar, A.L., et al.: MaxWeight scheduling in a generalized switch: state space collapse and workload minimization in heavy traffic. Ann. Appl. Probab. 14(1), 1–53 (2004) 54. van de Ven, P., Borst, S., Shneer, S.: Instability of MaxWeight scheduling algorithms. In: INFOCOM 2009, pp. 1701–1709. IEEE (2009) 55. Van Mieghem J.A.: Dynamic scheduling with convex delay costs: the generalized c— mu rule. Ann. Appl. Probab. 809–833 (1995) 56. Wang, C.-S., Stielau, O.H., Covic, G.A.: Design considerations for a contactless electric vehicle battery charger. IEEE Trans. Ind. Electr. 52(5), 1308–1314 (2005)

508

A. Daghighi and J. Q. Chen

57. Wang, W., Zhu, K., Ying, L., Tan, J., Zhang, L.: A throughput optimal algorithm for map task scheduling in MapReduce with data locality. ACM SIGMETRICS Perform. Eval. Rev. 40(4), 33–42 (2013) 58. Wang, W., Zhu, K., Ying, L., Tan, J., Zhang, L.: Maptask scheduling in MapReduce with data locality: throughput and heavy-traffic optimality. IEEE/ACM Trans. Netw. (TON) 24(1), 190–203 (2016) 59. Tom White: Hadoop: the definitive guide, yahoo (2010) 60. White, T.: Hadoop: The Definitive Guide. O’Reilly Media Inc., Newton (2012) 61. Winkler, F.: Consumerism in health care: beyond the supermarket model. Policy Politics 15(1), 1–8 (1987) 62. Xie, Q., Lu, Y.: Priority algorithm for near-data scheduling: throughput and heavytraffic optimality. In: 2015 IEEE Conference on Computer Communications (INFOCOM), pp. 963–972. IEEE (2015) 63. Xie, Q., Yekkehkhany, A., Lu, Y.: Scheduling with multi-level data locality: throughput and heavy-traffic optimality. In: INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications. IEEE (2016) 64. Xie, X., Liu, Y., de Rijke, M., He, J., Zhang, M., Ma, S.: Why people search for images using web search engines. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp. 655–663. ACM (2018) 65. Yekkehkhany, A.: Near data scheduling for data centers with multi levels of data locality. Dissertation, University of Illinois at Urbana-Champaign (2017) 66. Yekkehkhany, A.: Risk-averse multi-armed bandits and game theory. Dissertation, University of Illinois at Urbana-Champaign (2020) 67. Yekkehkhany, A., Arian, E., Hajiesmaili, M., Nagi, R.: Risk-averse explore-thencommit algorithms for finite-time bandits. In: 2019 IEEE 58th Conference on Decision and Control (CDC) (2019) 68. Yekkehkhany, A., Arian, E., Nagi, R., Shomorony, I.: A cost-based analysis for risk-averse explore-then-commit finite-time bandits (2020) 69. Yekkehkhany, A., Hojjati, A., Hajiesmaili, M.H.: GB-PANDAS: throughput and heavy-traffic optimality analysis for affinity scheduling. ACM SIGMETRICS Perform. Eval. Rev. 45(2), 2–14 (2018) 70. Yekkehkhany, A., Nagi, R.: Blind GB-PANDAS: a blind throughput-optimal load balancing algorithm for affinity scheduling. IEEE/ACM Trans. Netw. 28(3), 1199– 1212 (2020) 71. Yildiz, Y., Agogino, A., Brat, G.: Predicting pilot behavior in medium scale scenarios using game theory and reinforcement learning. In: AIAA Modeling and Simulation Technologies (MST) Conference, p. 4908 (2013) 72. Zaharia, M., Borthakur, D., Sarma, J.S., Elmeleegy, K., Shenker, S., Stoica, I.: Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling. In: Proceedings of the 5th European Conference on Computer Systems, pp. 265–278. ACM (2010) 73. Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R.H., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: Osdi, vol. 8, p. 7 (2008)

Seamless Update of a Hypervisor Stack Pavel Tatashin1,2(B) and William Moloney1 1

University of Massachusetts Lowell, Lowell, MA 01854, USA pavel [email protected] 2 Microsoft Corporation, Redmond, WA 98052, USA

Abstract. For security, performance, and competitive reasons, the hypervisor must be regularly updated in order to maintain the cloud infrastructure. This work provides an overview of three novel techniques of updating a hypervisor stack with a minimal disruption for the running virtual machines. HVPlexing: The first technique involves using a cooperative multi-OSing paradigm to share the same machine between the two operating systems while the new OS is booting, and the old OS is still servicing the running virtual machine instances. HVInception: The second method introduces a way of starting the new operating system in the virtual machine, and once boot is finished, transfer it from the virtual to physical environment, thus, fully replacing the old hypervisor stack with the new one. HVPartitioning: The third method is to start the new operating system on a subset of a physical machine. This method involves using CPU and memory dynamic reconfiguration in order to change the number of active processing units and the amount of memory in the running operating system. Once the new operating system has finished booting, it is resized to take over the full machine. In all three cases the running VMs are absorbed by the new hypervisor without noticeable interrupts by checkpointing and restoring them in the updated environment. Keywords: Hypervisor

1

· Operating system · Performance · Update

Introduction

Many modern data centers consist of sliced machines each of which runs a hypervisor stack with multiple virtual machines (VMs) on top. In a public cloud, these machines may be shared among a large number of tenants. In the case of a private cloud, they are typically divided into VMs that are used within the same organization to perform dedicated tasks. In both cases, the hypervisor and the underlying operating system must be reliable, secure, and provide high availability. In order for an operating system to be secure and reliable, it must be regularly updated. Also, recent Meltdown and Spectre exploits have shown that security fixes may require fundamental changes to the kernel [5, 6]. Traditional operating c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 509–519, 2022. https://doi.org/10.1007/978-3-030-80119-9_31

510

P. Tatashin and W. Moloney

system servicing involves rebooting the machine hardware in order for a new version of a kernel and in some cases a new userland to take effect, but this means that virtual machines must be temporarily shut down which, in turn, impacts the availability requirement for cloud machines. Depending on the hypervisor type, the hypervisor stack may differ. In this work, we will concentrate on the following common components of the stack: – Kernel: Linux – Hypervisor: KVM – Virtual Machine Monitor: QEMU The goal of this work is to update every component of the stack without a long downtime for the virtual machines. First we will give a brief overview of existing accelerated hypervisor update methods: kernel soft reboot, in-memory suspended VMs, hypervisor multiplexer, hot-patching, and live updates. We will also discuss their limitations. Next we will cover the three proposed upgrade methods, and finally describe the future directions for each of the methods.

2

Background

In this section we will take a look at the existing methods of seamless hypervisor upgrades, and also describe their limitations compared to the goals that are set in this paper. 2.1

Faster Reboot

A faster reboot of a hypervisor mitigates lengthy down times for virtual machines. The following two techniques are currently deployed in public clouds such as Azure. – One way to reduce reboot time is to optimize the kernel’s boot and shutdown procedures and to also skip firmware reset, where the old kernel loads the new kernel directly into memory, quiesces devices and jumps into the loaded kernel image. This functionality is called Fast-Reboot on Solaris [9], Kexec Reboot on Linux [11], and Kernel Soft Restart on Windows [12]. – Another reboot optimization is to keep the virtual machines intact and suspended in memory while the kernel is being rebooted, i.e. memory preserving reboot [13]. After the new operating system is up and running, the suspended virtual machines can be resumed as their memory state does not need to be modified. Both of the methods, while readily available in many existing operating systems, still involve a downtime for the virtual machines. On large machines, with hundreds of CPUs and terabytes of memory, it takes a non-trivial amount of time to boot a new operating system. The kernel contains many CPUs and Memory proportional data structures which must be initialized during boot. Also, during early boot, the kernel is singly threaded and thus at least some of the system initialization work is inherently non-scalable.

Seamless Update of a Hypervisor Stack

2.2

511

Partial HV Stack Update

There are techniques for seamless updating that involve skipping some part of the HV Stack, or relying on another piece of software. – Using another multiplexer layer beneath the hypervisor to facilitate switching between the hypervisors [2]. – Live updating KVM and QEMU performed by using Orthus[16]. The hypervisor multiplexer requires another piece of software underneath it, which, while thin, may also require an update with downtime, thus it is not possible to update the full stack. The Orthus updates QEMU and KVM, but does not live update the Linux kernel. 2.3

Seamless Upgrades

There are also methods that can achieve the goal of this work, such as a seamless upgrade of the Hypervising Stack, but these are not useful as general purpose solutions. – Kernel Hot-Patching [1, 10], allows fixing critical issues without reboot by modifying the kernel text on a live system. – Live migration of VMs, can be used to free up the Hypervisor OS and enable a regular update. Kernel Hot-Patching does not allow upgrading to a new version of an operating system or hypervisor, also, except for limited cases, most of the time it does not allow modifying the data pages, as it is complicated to change the sizes of data structures and the offsets of the fields must remain unmodified. Finally, this method introduces a slight slowdown until the machine is rebooted due to having an extra branch to the new function body that has been patched. A live migration method requires the availability of another server with a shared storage. In some cases, it is not viable to do a fast live migration if a virtual machine uses unique local devices such as a graphics card, FPGA, or local solid state storage. In some cases, live migration may also introduce a slow down because the virtual machines need to be booted with the lowest common hardware specifications in order to be migratable within the data center. In addition, there is currently no reliable way to live migrate generic physical devices even if they are present on a destination machine, as there is no support for pass-through live migration [16]. All of the above methods, while useful in some cases, are not suited for every situation of live updating. The methods that are proposed in this work do not suffer from the limitations of existing techniques and can seamlessly upgrade a full hypervisor stack without the use of extra hardware and without an extra software layer. The trade-off of the proposed methods is that an update procedure itself may take longer, as the underlying hardware is temporarily shared between two hypervisors.

512

3 3.1

P. Tatashin and W. Moloney

Upgrade Methods HVPlexing

With memory preserving reboot, one operating system is able to pass ranges of data to a new operating system residing in volatile memory. The system firmware could assist with this. Another way is to skip firmware and boot into a new OS directly from the old OS. In Linux, the kexec tool is used to load a new kernel image into memory, and jump to it from the old kernel. In the simplest way, the old kernel may pass ranges of memory to preserve to the new kernel by using kernel parameters to store ranges of physical memory. Later the new kernel can pass this data to userland for faster restoration after reboot. Examples can include quickly restoring the state of a VM or a database. A sample Hypervisor Stack upgrade sequence with memory preserving reboot is shown on Fig. 1. While the new OS is booting, the VMs are still experiencing downtime because the time to shutdown the old userland, boot the new kernel and start a new userland is non-trivial. HVPLexing that uses Cooperative multi-OSing is built on top of Memory Preserving Reboot, but takes it farther. Instead of jumping only once to the new OS, the old OS is preserved in memory, and the new OS boots for a short period of time, and then yields back to the previous OS to allow it to fulfill its scheduling duties. If the cycle of jumping to the new OS and then back to the old OS is short, it can be seamless from the perspective of the virtual machine. Figure 2 shows the sample cycle. The cooperation continues until the new operating system and its userstack are fully booted, at which point, the new hypervisor takes over the VMs, and the old hypervisor is discarded by virtue of never jumping back to it.

Fig. 1. Memory preserving reboot

Seamless Update of a Hypervisor Stack

513

Fig. 2. Cooperative multi-OS boot

While promising, implementing HVPlexing requires a significant effort. First, suspending and resuming a running hypervisor must be significantly optimized, so the quantum of time between jumps is small enough not to be noticeable by guest virtual machines. The biggest challenge is to reduce the time it takes to pause and resume CPUs. Currently, this operation is performed serially in Linux and scales poorly on large CPU count machines, but it could be improved by allowing CPUs to suspend/resume concurrently and allow each CPU to perform the necessary tasks independently. Second, we found that the kexec jump feature itself does not have active users because of a lack of use-cases and that it has been broken in the mainline kernel for at least two years, based on bugs that we found and when they were introduced. 3.2

HVInception

This method of seamlessly updating a hypervisor involves booting the new version of the hypervisor in a virtual machine and later converting it to a physical instance by using a special entry point such as the kexec syscall in Linux. While the new hypervisor stack is being booted, the old hypervisor stack keeps providing the necessary services to the running VM instances and only when the new hypervisor is fully booted and is ready to take the responsibilities of the old hypervisor, does the switch occur. The basic principle is shown on Fig. 3. The challenge with this approach is that, normally, the virtual machine is different from the underlying hardware. In particular, the VM has different firmware, the memory mappings are different, the devices are different and even the virtualized CPU might be different from the native CPU. All of these issues, however, can be resolved. Resolving the device issue for this method is actually similar to the other upgrade methods covered in this work and a possible solution is described in

514

P. Tatashin and W. Moloney

Fig. 3. Hypervisor inception update

Sect. 4 of this document. One possible solution is to boot the new hypervisor with the same device tree (device trees are used for simplicity) as the underlying hardware, but the devices are migrated from the old hypervisor during the transition phase. The difference in firmware can be resolved either by renegotiating firmware runtime services with the OS during transition from the VM to the physical environment, or to just not rely on any firmware run-time services. An example of this can be an ARMv8 that is booted with FDT instead of EFI/ACPI. On such platforms, once the kernel is booted, there are no services that are provided by the firmware and everything is handled by the kernel itself. A CPU can be emulated to be the same as an underlying physical CPU, but this is inefficient and slow. Therefore, it is better to use virtualization extensions that are provided by newer CPUs. However, by using these extensions, CPUs, while mostly identical, have some notable differences when they are running in a physical versus virtual mode. The differences can be resolved by the fact that the hypervisor must be aware that eventually, it will be transitioned to the bare metal and all the necessary changes to the hypervisor’s knowledge of the CPU can be done during the switch time. When an OS is booted in a virtual machine, its physical addresses are not the same as the physical addresses of the bare metal machine. The underlying hypervisor performs the conversions from a virtual physical address to the real physical address. Therefore, we cannot simply transition from a virtual to a physical environment without taking care of these translation differences. The

Seamless Update of a Hypervisor Stack

515

memory mapping issue might seem to be a hard problem to solve, but we will show that it actually can be achieved in a simple way. One possible solution would be to update page tables in the OS during the transition so virtual addresses are translated to real physical addresses instead of the virtualized physical addresses. This approach however, is not feasible in real world scenarios. First, it breaks the identifying map’s page tables (where any kernel VA == PA + offset), second it breaks the kernel allocators which manage memory in physical memory lists (such as the memblock allocator in Linux), finally it breaks any hash functions where a physical address value was used as a key. The second possible solution is to relocate every page from the bare metal physical location to where the virtual machine expects these pages to reside. In this scenario, a hypervisor, while running in a virtualized environment, is booted with exactly the same memory list as a bare metal hypervisor (it cannot touch all the memory until the switch is done, otherwise it will run out of memory or pages will need to be swapped). During the switch, the pages are moved from where they are physically located to where the virtual machine thinks they should be located. For example, if during boot, the hypervisor touched a page at physical address 0 × 20000, and this page corresponds to bare metal physical address 0 × 400000, during the switch from the virtual to physical hypervisor, the page is moved from 0 × 400000 to 0 × 20000. While this method is straightforward and does not require many changes to the hypervisor itself, it is inefficient: moving pages is slow, which means that the update is not going to be instantaneous, since the switch operation will take a substantial amount of time. Another drawback is that the hypervisor that is being updated must keep track of pages that were touched by the new hypervisor during boot in order to pass the list to the switch function. The third solution, which we think is the most practical, involves setting up memory with a one-to-one correlation between the bare metal hypervisor and the virtualized hypervisor. This means that virtual PAs are always equal to bare metal PAs. In this method, the virtualized hypervisor is also booted with the same memory list as the physical machine, but the virtualized hypervisor is allowed to touch only those physical pages that were pre-allocated for it. For example, if the physical hypervisor allocated page 0x400000, the virtual hypervisor is allowed to touch page 0x400000. This way, when the switch happens, no physical page relocations are needed. The pages which were not accessible to the virtual hypervisor become accessible as it becomes a physical hypervisor. HVInception requires a significant amount of development in order to make this feature practical in a cloud environment. The biggest challenges include the differences between virtualized and physical CPUs, even with hardware virtualizations and support of nested virtualization in guest Virtual Machines. While possible to do a software only implementation of HVInception, in our opinion, it is best if a new CPU is designed with this feature in mind, where there is

516

P. Tatashin and W. Moloney

another virtualization option available for a Hypervisor: the instructions in a virtualized CPU and a bare metal CPU are identical. 3.3

HVPartitioning

This method is similar to HVInception, but instead of first starting the new hypervisor in a virtual machine to later escalate it to bare metal, we start the new hypervisor right next to the running hypervisor on the physical machine. The HVPartitioning principle is shown on Fig. 4. This technique involves partitioning the hardware: hot-remove some CPUs and some amount of memory from the running hypervisor and allow the new hypervisor to boot and only use the available hardware resources. Once the new hypervisor is fully booted and is able to accept the running virtual machine instances, it can do so by check-pointing and restoring them after taking the rest of the hardware resources from the previous hypervisor. This method is easiest to implement on architectures that do not have runtime firmware services, since most of the firmware does not expect two Operating Systems to be running on the same machine simultaneously.

Fig. 4. Hypervisor partitioning update

The new hypervisor is booted only on a subset of the machine, but knows about the rest of the machine resources. This means that during boot, it can allocate all the hardware resource proportional data structures according to the way they will be needed once it is allowed to use the rest of the machine.

Seamless Update of a Hypervisor Stack

517

Once boot is finished, the new hypervisor will request the active hypervisor to suspend and will take over all the rest of the hardware resources, online the memory that was not accessible during boot and hot add CPUs that were not part of the boot partition. There were a number of attempts to perform hardware partitioning using the Linux kernel [3,8, 14] but none of them were merged or are officially supported by the Linux Kernel today. There are some partitioning hypervisors that are supported by the Linux kernel today such as Jailhouse [15] and Bao [7], but they allow only one hypervisor that manages the entire machine and hardware partitions for virtual machines. We have implemented hardware partitioning on the Linux kernel 5.9 that allows booting two hypervisors on the same machine. However, more work is needed in order to support proper interrupt handling between the two kernels and to support passing guest VMs between them.

4

Device Migration

In all three methods described in this paper, we did not mention how to handle the IO devices. There is no hardware independent way to pass the handling of a device from one hypervisor to another if the first hypervisor has an internal driver state that corresponds to this device. It is possible to add a new device driver interface that would be used to pass the information from the old hypervisor to the new one, in a fashion similar to how it has been done to perform live migration of direct-access-devices with show device drivers [4]. There is however, a simpler solution that already exists. A hypervisor does not have to handle the IO devices. Instead, an IO-Domain, which is a virtual machine that runs on a hypervisor, can handle all the physical devices and the other virtual machines would use the services of this IO-Domain for virtual devices or virtual functions. With this approach, the hypervisor simply does a pass-through of all the PCIe devices to the IO-Domain and only needs to pass device driver independent information to the new hypervisor about the state of the pass-through devices.

5

Conclusion

This work shows three hypervisor update methods that can be used to hide the boot latency of the new kernel or of a full hypervisor stack boot and can be used to live migrate all services to successfully perform an uninterrupted update of a machine in a cloud environment. These new update methods have advantages compared to existing techniques in that they do not require extra hardware resources, do not permanently slow down the running VM instances and, except for the virtual machine preserved memory ranges and device passthrough information, the new and old kernels do not need to share intrinsic information. Finally, these methods do not require another layer of software beneath the hypervisor.

518

P. Tatashin and W. Moloney

There are a number of ways these methods can be improved and optimized. Once two kernels are up and running, the checkpoint restore mechanics can be used to add support of live migration of containers and individual processes. Shutting down and resuming devices is expensive between OS switches, therefore, a new quiescing function can be added to the driver interface that only blocks interrupt activity from the device while another OS is active.

References 1. Arnold, J., Kaashoek, M.F.: Ksplice: automatic rebootless kernel updates. In: Schr¨ oder-Preikschat, W., Wilkes, J., Isaacs, R. (eds.) Proceedings of the 2009 EuroSys Conference, Nuremberg, Germany, 1–3 April 2009, pp. 187–198. ACM (2009) 2. Bagdi, H., Kugve, R., Gopalan, K.: Hyperfresh: live refresh of hypervisors using nested virtualization. In: Proceedings of the 8th Asia-Pacific Workshop on Systems, Mumbai, India, 2 September 2017, pp. 18:1–18:8. ACM (2017) 3. Joshi, A., Pimpale, S., Naik, M., Swapnil, R., Pawar, K.: Twin-linux: running independent linux kernels simultaneously on separate cores of a multicore system. In: Proceedings of the 2007 USENIX Annual Technical Conference, Santa Clara, CA, USA, 17–22 June 2007. Linux Symposium (2010) 4. Kadav, Asim, Swift, Michael M.: Live migration of direct-access devices. Oper. Syst. Rev. 43(3), 95–104 (2009) 5. Kocher, P., et al.: Spectre attacks: exploiting speculative execution. CoRR arXiv:1801.01203 (2018) 6. Lipp, M., et al.: Meltdown. CoRR arXiv:1801.01207 (2018) 7. Martins, J.: Bao: a lightweight static partitioning hypervisor (2020). https://lwn. net/Articles/820830/. Accessed 8 Aug 2020 8. Nomura, Y., Senzaki, R., Nakahara, D., Ushio, H., Kataoka, T., Taniguchi, H.: Mint: booting multiple Linux kernels on a multicore processor. In: 2011 International Conference on Broadband, Wireless Computing, Communication and Applications, BWCCA 2011, Barcelona, Spain, 26–28 October 2011, pp. 555–560. IEEE Computer Society (2011) 9. Oracle. Oracle solaris administration: Common tasks: accelerating the reboot process (2011). https://docs.oracle.com/cd/E23824 01/html/821-1451/gldoi.html. Accessed 5 Dec 2020 10. Poimboeuf, J.: Live kernel patching. LinuxCon North America, August 2014 11. Kexec Tools Project. Kexec(8) — Linux manual page (2006). https://www.man7. org/linux/man-pages/man8/kexec.8.html. Accessed 5 Dec 2020 12. Russinovich, M.E.: Memory-preserving reboot. Patent US20150178097A1, January 2018 13. Russinovich, M.E., Raghuraman, M.R.: Virtual machine-preserving host updates. Patent US20140157264A1, August 2015 14. Shimosawa, T., Matsuba, H., Ishikawa, Y.: Logical partitioning without architectural supports. In: Proceedings of the 32nd Annual IEEE International Computer Software and Applications Conference, COMPSAC 2008, 28 July–1 August 2008, Turku, Finland, pp. 355–364. IEEE Computer Society (2008)

Seamless Update of a Hypervisor Stack

519

15. Sinitsyn, V.: Understanding the jailhouse hypervisor (2014). https://lwn.net/ Articles/578295/. Accessed 8 Aug 2020 16. Zhang, X., et al.: Fast and scalable VMM live upgrade in large cloud infrastructure. In: Bahar, I., Herlihy, M., Witchel, E., Lebeck, A.R. (eds.) Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2019, Providence, RI, USA, 13–17 April 2019, pp. 93–105. ACM (2019)

Developing a Framework of Critical Factors Affecting the Adoption of Cloud Computing in Government Systems (ACCE-GOV) Naif Al Mudawi1 , Natalia Beloff1,2(B) , and Martian White2 1 University Najran, University of Sussex, Brighton, UK

{N.Al-mudawi,N.Beloff}@sussex.ac.uk 2 University of Sussex, Brighton, UK [email protected]

Abstract. Many governments are investigating strategies for the adoption of cloud computing to improve service quality in e-government systems. The decision to adopt cloud computing for government systems without identifying and examining critical factors is a problem facing most e-government organizations in developing countries. This study has developed and examined a technology– organization–environment model with the addition of social context. The model (ACCE-GOV) investigates the influential factors that impact the adoption of cloud computing for Saudi e-government systems within Saudi government organisations. The study identifies 12 hypotheses that have been evaluated using structural equation modelling through regression analysis. Data was gathered, from 838 higher level managers and IT employees from Saudi government organisations. The analysed data provides rich insights into how attitude, trust, top management support, security, technology readiness and competitive pressure have a significant direct effect on the adoption of cloud computing. The findings imply that positive attitudes and trust in adopting cloud computing should be supported and encouraged by IT administrations through designing appropriate IT infrastructure environments to help the top management in government organisations adopt such cloud services. Further, this study suggests some recommendations for government organisations considering the adoption of cloud computing for e-government systems. Keywords: Adoption · Cloud computing · e-Government system

1 Introduction With the explosion of Internet technology, there is increasing pressure on existing storage in computing facilities. This has led providers of Internet services to use commoditycentric computers as their platform [1]. Cloud computing has thus become viewed as a useful strategy for Information Technology (IT) users and Internet service providers (ISPs) [2]. Furthermore, some countries around the world have invested in this technology: the United States of America and Japan have both built cloud computing frameworks © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 520–538, 2022. https://doi.org/10.1007/978-3-030-80119-9_32

Developing a Framework of Critical Factors

521

as a national strategy [9]. Cloud computing is a progressive technique that has introduced a new paradigm by providing a logical model that has changed the means of consuming IT by providing on-demand services over the Internet [4]. Cloud computing is one of the best alternative solutions to overcoming both the high cost of establishing an IT infrastructure and other obstacles that face e-government [6]. Moreover, developed countries have implemented their applications in cloud computing and have achieved maximum benefits from it. This has contributed to reducing the cost of developing e-government systems and more accessibility, authentication and ease of use [7]. The developing world should heed the results that developed countries have achieved in adopting cloud computing for e-government systems. Thus, one of the most common challenges facing e-government is to overcome the established e-government infrastructure, especially for a developing country such as Saudi Arabia. Innovation in cloud computing have been introduced into several sector such e-commerce, e-health [3]. However, cloud computing has not been sufficiently made use of for e-government systems for their benefits to have become apparent to many governments [2]. This study highlights some of the benefits of the Cloud as well as barriers that hinder its adoption. The purpose of this study is to understand and investigate which factors influence cloud adoption for e-government systems. This study has discussed the adoption of cloud computing from an organizational perspective based on the TOE model. This study however conducts a more holistic evaluation by also considering the impact of social context in the adoption of the Cloud and seeks to develop a research model that assists top management in government organisations to understand the critical factors influencing the adoption of cloud computing by Saudi government organisations. The study attempts to fill the knowledge gap in the field of the adoption of cloud computing for e-government systems by investigating it from the perspective of government organization in the Saudi context of Saudi Arabia; and creating a more appropriate model to evaluate the critical factors of cloud adoption for e-government systems.

2 Theoretical Background 2.1 Overview of Cloud Computing There is no one definitive definition of cloud computing by expert researchers in the field. This section will provide some definitions from different researchers’ perspectives and explore the advantages of cloud computing. Studies over the past few years have provided important information on cloud computing as a model enabling a network that is available everywhere, with easy access and shared resources, such as networks, servers, storage, applications and services; and which allows users to access them quickly and reduces the effort or cost in service management [8, 9]. Cloud computing is becoming increasingly popular worldwide, and organisations have used it to provide online services [10]. This started with e-business, e-commerce, and e-government, and provided benefits which included improving the quality of service delivery to public sectors [13]. Centralised data storage will be of great benefit to governments in providing e-services more effectively.

522

N. Al Mudawi et al.

2.2 Adoption of the Cloud The research reviewed theoretical foundations for the adoption of the cloud computing; which is briefly discussed below. There is consistent empirical support for using the technology-organization-environment (TOE) framework in the domain of understanding IT use in government organisations [12]. Some researchers have explored the definition of cloud computing based on The TOE framework. Cloud computing is the creation of a new model based on providing computer infrastructure [14]. The field of information system (IS) innovation is often concerned with the relationship of IS applications and organizational context. According to the IT innovation literature [22] suggested that IT innovation refers to the response to changes in the external and internal environment of organizations and adoption of new technology to make processes more effective. Some studies have addressed issues about technological investment. Hansen et al., (2018) proposed a model that identified issues with cloud computing based on reducing expenditure cost and security risks in private industry. Almutairi and Thuwaini (2015) found that top management attitude to direct and indirect expenditure on e-government significantly influenced the implementation of cloud computing. Lal and Bharadwaj (2016) developed a research model to examine the factors influencing the adoption of cloud services for organisations in the Indian private sector. The study found that there is a strong relationship between the adoption of cloud and security concerns. A study conducted by [24] proposed a model for public organisations using cloud computing. This study found that the factors influencing the implementation of cloud computing in public sector organisations were technical issues, especially security and IT infrastructure readiness. In the next section, we provide a background to cloud computing including relevant studies on the adoption of cloud computing for e-government systems. In the third section, we provide the research model along with the hypotheses. The research method and design are then presented in more detail; and the fifth section describes the analysis of the results followed by a discussion of the major findings of our study, its theoretical contributions and limitations. Finally, the paper concludes by recommending future research.

3 Research Model and Hypotheses 3.1 Theoretical Models Over the past two decades, intensive research has been conducted using different models to investigate adoption of different types of information technology in organisations [15]. Some research considers the adoption of new technologies by using the TOE Model, which allows identification of the most important factors affecting adoption [5]. • Diffusion of Innovation Theory (DOI) The Diffusion of Innovation theory (DOI) was first designed by Rogers in the 1960s, to study the dissemination of innovation in society. In 1995, the DOI theory was further developed by Rogers to analyse and discuss adopting new technology through its implementation of electronic systems [18].

Developing a Framework of Critical Factors

523

• Technology-Organisation-Environment (TOE) Framework This framework aims to identify the components of organisations that have an impact on the adoption and implementation of technological innovations [23]. The TOE model consists of three elements, which revolve around technological problems, internal and external factors, and organisations [19]. The first element refers to the technological issues facing organisations that intend to adopt a technology, such as IT infrastructure and security and. The second factor refers to internal and external factors that have an impact on organisations, such as competition in the market and regulations and policies, while the third factor describes organisations in terms of the nature of their activity, size and structure [20].

4 Research Model Given that empirical studies on the factors influencing the adoption of innovations such as the Cloud for e-government systems have successfully made use of the dimensions in the TOE model, our model adopts these and adds one of its own. The model is thus developed based on four theoretical concepts: • • • •

Technological context Organisational context Environmental context Social context.

The theoretical models used are the Technology, Environment and Organisations (TOE) model along with critical factors from the Diffusion of Innovations (DOI) theory for a better understanding of the factors related to adopting cloud computing for e-government in Saudi Arabia. This study proposes a model (ACCE-GOV) that identifies the factors that most influence the adoption of cloud computing in the Saudi egovernment sector. This integrated model is also utilized to discover ways to enhance the adoption of cloud computing in the government sector in Saudi Arabia. The model and its factors are shown in Fig. 1. 4.1 Technological Context Relative Advantage: refers to “… the degree to which an innovation is perceived as being better than the idea it supersedes” [21]. Comparative advantage drives organisations to switch to cloud computing if its adoption can reduce operational costs and improve service delivery [28]. One of the most important contributions of innovation theory is perceived relative advantage [29]. These issues have generated the following hypothesis: H1. Perceived relative advantages have a direct and positive influence on the intention to adopt cloud computing for e-government systems. Compatibility: refers to “… the degree to which an innovation is perceived as consistent with the existing values, past experiences, and needs of potential adopters [21].” The compatibility of processes with the standards of organisations is considered an influential technological factor in the adoption of new technologies in an organisation [25].

524

N. Al Mudawi et al.

Fig. 1. The conceptual model

Based on the technological innovation literature, compatibility is one of the important factors that predict the adoption of information systems in organisations [30]. These issues lead to the following hypothesis: H2. Higher compatibility positively influences the intention to adopt cloud computing for e-government systems. Complexity: is described as “…the degree to which an innovation is perceived as relatively difficult to understand and use” [21]. Recent studies state that understanding the level of perceived difficulty in adoption of cloud computing will enhance the organisation’s intention to either adopt or ignore it [26]. In fact, the adoption of cloud computing contributes to e-systems, saving time on performing tasks, which increases the efficiency of data transfer [31]. Based on the discussion above, the following hypothesis is proposed: H3. Less complexity positively influences the intention to adopt cloud computing for e-government systems. Service quality: Refers to the result of the comparison between the system quality and level of satisfaction with the service and an understanding of the way the system was implemented [27]. Service quality is an important factor for top management in the adoption of new technologies [32]. It is therefore hypothesised that: H4. There is a positive relationship (direct correlation) between the high service quality of the Cloud and the intention to adopt cloud computing for e-government systems. Security: many researchers have identified that security issues are one of the most influential elements of users’ trust in technology. Security refers to the degree to which a user and organisation’s information is safe and protected by the cloud storage provider [44]. Governments need to put in place penalties to deter illegal access to data [33]. This type of legislation contributes to increasing the trust of governments’ organisations in adopting e-government services. The following hypothesis is proposed: H5. High level of security (data protection) positively influences the intention to adopt cloud computing for e-government systems.

Developing a Framework of Critical Factors

525

4.2 Organizational Context Top management support: considers how the role of top management has an important influence on the adoption of cloud computing [17]. Top management support refers to the role of top managers in supporting decision-making that adopts and implements information technology in order to seek to develop the organisation with technological investments [36]. Based on the literature review, top management support is one of the most important factors affecting the adoption of IT investment at the organisational level [38]. It is suggested, therefore, that: H6. Top management support influences positively the intention to adopt cloud computing for e-government systems. Technology readiness: is one of the critical factors that influences the adoption of cloud computing in organisations [39]. Technology readiness refers to the maturity level of the network technologies and enterprise systems that can adopt cloud computing. Technology readiness has a significant impact on leadership when it comes to making a decision on adopting an innovation [35]. Governments in developing countries face difficult social, economic and political constraints that limit their ability to invest in expensive information systems to compete on the world stage. The following hypothesis can be drawn: H7. Technology readiness influences the intention to adopt cloud computing for e-government systems. 4.3 Environmental Context Regulations: refer to the rules and policies that are applied by organisations and which influence the adoption of IT innovations [16]. Governments have the ability to promote the adoption of cloud computing by drafting regulations in order to encourage organisations to invest. Government regulations have a positive effect by reducing the restrictions related to the adoption of new technologies [37]. It is suggested, therefore, that: H8. Existence of a competitive pressure influences positively the intention to adopt cloud computing for e-government systems. Competitive pressures: refers to the level of pressure felt by organisations to increase demand by effectively providing e-government systems [42]. Therefore, one of the most important steps to achieving the objectives of the organisation should be keeping pace with information technology [40]. The competitive pressure to achieve the vision of the organisation is an important factor in the adoption of information technology. The following hypothesis is therefore proposed: H9. A less stringent regulatory environment will have a positive influence on the intention to adopt cloud computing for e-government systems. 4.4 Social Context Trust: can be generally understood as “The degree to which people trust e-government policy and service that can be received from another party” [29]. Although trust in technology differs from the concept of trust in general, researchers point out that trust

526

N. Al Mudawi et al.

in technology exists when users’ expectations are met by interacting with technology having reliability [28]. These issues are addressed by the following hypothesis: H10. High level of trust in cloud computing positively influences the intention to adopt cloud computing for e-government systems. Awareness: refers to “An understanding of the activities of others, which provides a context for your activity” [27]. Awareness plays an important role in identifying the characteristics of new technologies compared to older ones and how they are presented to investors [43]. Moreover, top management in government organisations seeks to increase knowledge of the benefits of new technologies they are considering to adopt [41]. Based on the discussion above, the following hypothesis is proposed: H11. High level of awareness positively influences the intention to adopt cloud computing for e-government systems. Attitude: refers to “positive attitudes toward the system and willingness to participate in the implementation and to accept the change brought about by the system” [46]. Cloud computing dramatically expands data storage capacity, making it easier to manage and synchronise data in different business environments, and this encourages organisations to adopt cloud computing to obtain these benefits [47]. Attitudes towards investment in new technologies thus influence organisational behaviour [53]. The following hypothesis can be drawn: H12. Positive attitude on cloud computing positively influences the intention to adopt cloud computing for e-government systems.

5 Research Methodology 5.1 Measurement This study has been conducted using a quantitative (survey) method. In the IS research field, quantitative analysis is more common. The quantitative method will be the main method of data collection utilising a survey questionnaire. The questionnaire items were developed based on a comprehensive review of the literature. All constructs were measured using a minimum of four closed-ended items. 5-point Likert scales were used for responses, where 1 refers to ‘strongly disagree’ and 5 refers to ‘strongly agree’. The questionnaire was distributed in different government organisations in Saudi Arabia, electronically and manually to be more comprehensive. The target sample size was around 1000+ responses. This study considered respondents who were mostly IT employees and top managers of organisations. Therefore, to increase content validity, target respondents were key informants in that they were knowledgeable about cloud computing. We encouraged all participants that filled out this survey to have some level of familiarity with cloud computing by providing a description of cloud computing. The data analysis of the survey is based on the research model; and results were validated by using descriptive statistics as well as the structural equation model (SEM). 5.2 Data The empirical data examined the hypothesised relationships of the factors that influenced adoption of cloud computing for e-government systems in Saudi organisations; and

Developing a Framework of Critical Factors

527

were obtained by using an internet survey with respondents from Saudi government organisations. The content validity of questionnaire items was established, reviewed, and revised by experts and the pre-test (a pilot study) achieved all requirements for validity. The pilot study was conducted with 33 participants to make sure the questionnaire had clear instructions no errors in the wording of questions and validity. The item numbers for each variable and the associated references are shown in Table 4. Each item was translated by language and IT experts who gave feedback about whether the questionnaire was appropriate for the research objectives. This study selected participants randomly that were from more than 60 government organisations such as (education sector, finance sector and Industry sector) during three months from 1st of March 2019 to 31st of May 2019 to ensure that IT managers and employees were able to receive the questionnaire and to increase the response rate. These questionnaires were sent to around 1,000 people in IT departments in every government organisation; 878 responded, with 838 having complete data available for subsequent analysis, yielding an effective response rate of 95.4%. The target sample size was focused on top management, IT managers, and employers in Saudi government organisations. IT managers have direct contact with systems and understand the issues about IT infrastructure and are normally in collaboration with IT investors to provide better IT services to government organisations.

6 Results 6.1 Confirmatory Factor Analysis (CFA) CFA refers to a statistical technique where the researcher can verify the factor structure of a set of observed variables related to another construct. This technique of statistical CFA allows the researcher to test the hypotheses and examine any results that may be received. CFA can be applied to examine the theoretical pattern [49]. CFA enables the decision of whether the theory that has been studied should be accepted or rejected. However, this research also chose to use the SEM statistical technique [46]. 6.2 Validity and Reliability Assessment Measurement items were created based on a comprehensive review of the literature and were modified to examine the adoption of cloud computing for e-government systems. The research model was measured for reliability, convergent validity, and discriminant validity. Furthermore, Cronbach’s alpha was used to assess construct reliability prior to making any advanced analysis [42]. The measurement model also demonstrated construct reliability greater than 0.7 for all of the constructs [52]; hence, exhibiting adequate reliability. This study has measured reliability by calculating Cronbach’s alpha and composite reliability (CR) which should achieve the minimum requirement threshold of 0.60 and 0.70. Table 2 shows Cronbach’s alpha ranged from 0. 761 to 0. 831 for all constructs and (CR) was greater than .05 (as shown in Table 1). Convergent validity, an excellent model fit requires the value of AVE to be over 0.5 [51]. The AVE achieved the standard minimum acceptable value of 0.5, as shown in Table 1. The categories of acceptable model are as follows: the value of CMIN/DF

528

N. Al Mudawi et al.

should be between 1 and 5, the value of CFI ≥ 0.90, the value of RMSEA ≤ 0.06, and the value of SRMR ≤ 0.08. These results with the required level of the model fit. In a study collecting data from in excess of 250 participants; the value of chi-square should be less than 3 to achieve model fit [42]. This study has collected data from 838 participants with a chi-square value of 1.758. All measures resulted in model fit, including CFI = 0.953, SRMR = 0.036 and RMSEA = 0.030. These results have received the required level of the model fit. Thus, at this stage, model fit was achieved as all measures met the requirements, as indicated in Table 1. Discriminant validity refers to the extent of the distinction between a construct and other constructs. This is measured by associating the square of correlation estimates with AVE values for any two constructs [51]. Moreover, the AVE value should be higher than the squared correlation estimates. This study revealed no concerns regarding discriminant validity, as presented in Table 1. The researcher used cross-loadings matrix and FornellLarker criterion [47]. The model fit is not considered as having discriminant validity if the correlations are lower than the square root of the AVE by a construct [52]. Furthermore, the square roots of the AVEs “diagonal cells” show scores that are higher than the correlations between constructs which refer to discriminant validity. Table 1. Reliability and construct validity Constructs Cronbach (above 0.7) C.R AVE

MSV

RA

0.769

.770 0.530 0.224

CM

0.820

.824 0.540 0.244

CO

0.809

.812 0.523 0.089

SQ

0.82

.829 0.549 0.129

SE

0.829

.821 0.549 0.089

TM

0.765

.777 0.525 0.310

TR

0.761

.762 0.517 0.170

RE

0.757

.757 0.510 0.310

AC

0.801

.807 0.515 0.241

CP

0.769

.776 0.537 0.296

TU

0.796

.807 0.511 0.257

AW

0.831

.835 0.560 0.224

AT

0.775

.781 0.544 0.224

Developing a Framework of Critical Factors

529

Table 2. Discriminant validity RA

CM

CO

SQ

SE

TM

TR

RE

AC

CP

TU

AW

RA

0.728

CM

−0.401

0.735

CO

0.129

−0.144

SQ

−0.359

0.298

0.042

0.740

SE

0.021

0.042

−0.298

−0.144

0.741

TM

0.277

−0.278

0.124

−0.221

−0.029

0.724

TR

−0.237

0.353

0.129

0.200

0.014

−0.413

0.719

RE

0.379

−0.339

0.121

−0.182

−0.016

0.557

−0.384

0.714

AC

0.427

−0.231

0.166

−0.117

0.033

0.382

−0.143

0.490

0.718

CP

0.410

−0.494

0.189

−0.237

−0.007

0.271

−0.296

0.544

0.430

0.733

TU

0.281

−0.388

0.184

−0.297

−0.129

0.486

−0.321

0.507

0.434

0.468

0.715

AW

0.250

−0.406

0.134

−0.208

0.085

0.294

−0.222

0.296

0.317

0.374

0.473

0.748

AT

0.473

−0.082

−0.069

−0.064

0.028

0.131

0.001

0.256

0.426

0.251

0.114

0.189

AT

0.729

0.737

7 Data Analysis This research used SEM. As discussed earlier, CFA is one of the theoretical patterns of the variables loading test in order to identify the relation between the actual data and the theoretical specification of the factors [47]. The researcher can determine influence factors based on selecting the appropriate theory being examined before obtaining any results. The researchers applied a CFA to take into account the unidimensionality that should be made first of these measurements include the reliability, discriminant, and convergent validity before examines the hypotheses [49]. The researcher should take into account that before modelling the interrelationship in the SEM, it is necessary to perform CFA for all latent constructs. The assessment of items for unidimensionality indicates that factors with a latent construct loading of 0.6 and above are acceptable [8]. The CFA results revealed 13 latent constructs 12 constructs comprising 55 items to examine the perceived effectiveness of adopting cloud computing for e-government systems. 7.1 Structural Equation Model (SEM) The structural equation model (SEM) is a powerful multivariate tool that allows the researcher to study the interrelationship between the latent and observed variables. SEM is very popular in behavioural, attitude, educational, psychological, and social research [50]. Latent variables (constructs) are those variables that cannot be directly observed or measured such as behaviour, attitude, intelligence, etc. In establishing a model to reflect reality, assessing the interrelationships between latent and observed variables is necessary. This research has applied the SEM procedures in order to examine the proposed conceptual model by the AMOS 25 software. This study has ensured that the collected data was clean and normally distributed. This required checking the data via a confirmatory factor analysis. The main components of the SEM are the structural model and measurement model. The path that relates the independent to the dependent variables

530

N. Al Mudawi et al.

is the structural model. This study has collected data that exceeds 838 participants with the chi-square value of 1.758. All measures show model fit including CFI = 0.953, SRMR = 0.036, RMSEA = 0.030; and AVE achieved the standard low of 0.50. Table 2 shows these results with the required level of the model fit. Table 3. Descriptive analysis and factor loading for items Measure

Chi-square (CMIN)/degrees of freedom (DF)

Comparative fit index (CFI)

Standardised root mean square residual (SRMR)

Root mean square error of approximation (RMSEA)

P of close fit (PClos)

Estimate

1.758

.953

.036

.030

1.000

Threshold

Between 1 and 5

>0.95

max assert then max assert = partial percentage assert; WF = wF ; WR = wR ; WL = wL ; Z = z; end end end Algorithm 1: Interactions Weight Algorithm (IWA) 5.2

IWA Evaluation

This section describes the results and discussion of the IWA. A training sample set ST composed of 1210 users was used to run the IWA algorithm in order to select appropriate weights.

How Twitter Interactions Leak Political Trends

603

However, in order to validate the goodness of these weights, a second sample of users SC ∈ O was also considered. The control set SC was composed of 101 Users. The tendency of these users was also manually obtained, using the same procedure when analyzing ST . So, the estimated tendency can be contrasted to the real one. Taking into account the weights obtained with the training set ST , the assert of the algorithm was tested over the second set SC . Thus, the IWA algorithm run over the training set ST returned the following weights and Z value: WF = 0.49, WR = 0.41, WL = 0.10, Z = 3. According with these values, for each user u in ST and SC , the weight Wux was calculated for each political party x ∈ X: Wux = (|Fux | · WF ) + (|Rux | · WR ) + (|Lxu | · WL ). Then, for the Z = 3 parties for which Wux is highest, political affinity of u versus x is estimated as positive. This estimation is contrasted with the real affinity, obtained manually a priori. This comparison retrieves an assert percentage for each set, which are shown in Table 3. This table shows that the success rate for the training sample ST is 96.23%, which validates the suggested weights and Z parameter. An equivalent procedure was performed over the control sample of users SC to check whether the weights and the number of political parties Z were suitable enough. As shown in Table 3, the assertion was 86.51%, which can be considered fine. As expected, the training set ST obtained better results than SC because the weighs were obtained adhoc with the matrix TST corresponding to the set ST . Table 3. Results Interaction Weight Algorithm (IWA) Users Sets Assert

6

Error

ST (118)

96.23% 3.77%

SC (101)

86.51% 13.49%

Users Interaction Analysis Strategy (UIAS)

The UIAS strategy is focused on retrieving the Twitter political interactions between users in O and users in P , in order to infer the political tendency of each user. For each tweet launched, the system retrieves several interactions of the tweet owner. In particular, for each Twitter user analyzed, the system obtains their relevant interactions (Friends, Hashtags, Mentions, Retweets and Likes) concerning their political affinity. From these interactions, the UIAS system computes the affinity parameter of user u to quantify their trend for each political party x. Using this parameter, the system output is the estimation of

604

M. Sol´e et al.

Fig. 7. UIAS workflow diagram

the users’ political tendency, which is then used to update the global tendency of each political party. This strategy is divided into three main blocks, shown in Fig. 7. ETL Engine block is focused on managing the streaming Twitter data and prepare data for the final calculation. Data Warehouse block manages the database that stores data generated by the system. Tendency Processing block is focused on data processing process to retrieve the individual users trends as well as the global political tendencies. Each block is explained in detail below. 6.1

ETL Engine

This block is devoted to contextualize the analysis and retrieve users’ interactions. This block is made up of the three main stages detailed below. Data Initialization Context Definition: In order to focus on a particular context and prior to starting the streaming process, several sets of keywords are defined to set up the UIAS strategy. These sets contain significant words related to the Spanish political context, which are used to classify or extract the tweets and interactions. In this work, these words are extracted from official web pages, Twitter accounts or campaign hashtags related to different political parties. Once the Keywords Set K is defined, it will be used to filter those tweets which contain political words in K. Then, the users that launch these tweets will be placed in O.

How Twitter Interactions Leak Political Trends

605

Secondly, the family of sets named Collection Keywords C is considered. For each party x ∈ X, the set Cx contains user identifiers for relevant political users in Px along with hashtags in Hx . Then  C= Cx . x∈X

These sets will be used to classify interactions between users in O and P . Data Extraction Twitter API: To extract the tweets and user interactions, an interface to retrieve data from Twitter is required. In this work, the Twitter API2 has been used, more specifically the Streaming API, because it provides real-time information on tweets and users. Twitter API is used to extract those tweets which contain words in K and their owners are placed in O. Interactions Extraction: The goal of this process is to collect, and periodically update, all the interactions between users in O and U (not exclusively with P ). According to this, for each analyzed user, this stage extracts and updates their relevant political interactions (Friends, Hashtags, Mentions, Retweets and Likes) with any other Twitter user. It is worth pointing out that this phase does not take into account the political affinity of the user analyzed. To collect the Retweet interactions, the system uses two different mechanisms. On the one hand, the system saves the retweets received in the streaming process. On the other hand, every politician in P is monitored in such a way that when they launch a tweet, the users retweeting it are retrieved. So, the output of this block is the interaction sets Fu , Hu , Mu , Ru and Lu for each user u analyzed. Data Cleaning Interactions Classifier: This process is focused on identifying the relevant interactions between users in O and users in P and classifying the interactions in relation to the political affinity. So, as an input, this process receives a user u and its interaction sets Fu , Hu , Mu , Ru and Lu . Hence, the goal of this process is to filter the interactions exclusively related to users in P . To classify the interactions, the set Cx is used. From the point of view of the streaming architecture, this block is separated with the aim to provide flexibility to update Cx in real time, if necessary. Thus, the output of this block is the interaction sets of user u for each political party x: Fux , Hux , Mux , Rux , and Lxu .

2

The Twitter API platform provides three tiers for searching Tweets, https://dev. twitter.com/.

606

6.2

M. Sol´e et al.

Data Warehouse

The aim of this block is to unify all data in one repository that stores the data required by all the processes of the system. These data is managed with a MongoDB system deployed in one node isolated of the other system clusters. The main data collections managed by the system are: Keywords: The keywords set K used to extract those users that launch tweets demarcated in a particular political context. Collection Keywords: The keywords sets Cx used to classify the users interactions according to their affinity to the different political parties. Users: It stores all available information about each user, such as the user information provided by Twitter, her political interactions and her political tendency (according to the different time periods it has been analyzed. Global Political Tendencies: It stores the global political tendency retrieved along different time periods. Notice that the information managed in this block is not static, since it is constantly updated along the overall process. 6.3

Tendency Processing

This block receives sets of users interactions compressed between two dates and then, by means of two different processes, it calculates the users and the global political tendencies. Processing data is computed in a Hadoop cluster solving a costly computation by distributing it in a MapReduce programming mode. The Hadoop cluster is made up by 6 nodes. Each node has two Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70 GHz 8 cores. The frontend node has 128 GB RAM and the 5 slaves have 64 GB RAM. User Tendency Update: Prior to calculate the global political tendency, it is necessary to assign a political tendency to each user. For this purpose, a community detection algorithm over interaction graphs [3] has been used to identify which are the more relevant interaction types. This procedure has revealed that Friends (Fu ), Retweets (Ru ) and Likes (Lu ) are the most significant interactions (whilst hashtags and mentions interactions can be dismissed). Furthermore, since each interaction might be not equally relevant, a weight for each one has been empirically established by using a sample population. This evaluation has determined that a proper assignation might be WF = 0.49, WR = 0.41, WL = 0.10. These weights satisfy that WF + WR + WL = 1 and 0 ≤ WF , WR , WL ≤ 1. Then, for each user u and each political party x ∈ X, the affinity parameter Wux is computed as Wux = (|Fux | · WF ) + (|Rux | · WR ) + (|Lxu | · WL ).

How Twitter Interactions Leak Political Trends

607

From these values, the affinity percentage for user u concerning party x is computed as Wx Axu =  u j Wu j∈X

As an output, the User Tendency Update process returns the three political parties with highest affinity for user u. For these three parties, the user’s political trend is Bux = Axu , while for the rest of parties, Bux = 0. Table 4 shows an example of the User Tendency Update for a given user u. The first column presents the political parties analyzed, the three next columns show the number of interactions (Friends, Retweets and Likes) between user u and each political party, the next column shows the affinity parameter Wux and finally the last two columns contain the affinity percentage Axu and user political trend Bux . In this example, user u will be considered affine to parties x1 , x2 and x3 since they present the highest affinity to this user. Global Political Tendencies Update: Finally, this process updates the global political tendency for a given time period. The process to calculate the global political tendency for party x is obtained as the mean of the users’ political trends Bux . However, notice that not every user might be considered relevant for the final computation, since users with sporadic interactions could distort the result. In particular, a minimal number of interactions for each user (UNI) should be requested and also at least one type of interaction should attain a given threshold (MIT). These restrictions reduce the set of objective users from O to OM . Hence, the global political tendency for a political party x is computed as  Bjx Gx =

j∈OM

|OM |

.

A proper selection of thresholds for UNI and MIT are discussed in next section. Table 4. User tendency update calculation example for one particular user u. Political Parties x |Fux | |Rux | |Lxu | Wux x1 = PP

15

18

x2 = PSOE

12

x3 = CS

15

Axu

Bux

16

16,7 35,38% 35,38%

20

3

10,9 23,00% 23,00%

8

12

10,7 22,56% 22,56%

x4 = VOX

10

4

8

6,6 13,88% 0,00%

x5 = Podemos

13

1

1

2,2

4,66%

0,00%

x6 = JuntsXCat

4

1

2

1,8

3,79%

0,00%

x7 = ERC

1

0

0

0,1

0,21%

0,00%

x8 = EHBildu

0

0

0

0,0

0,00%

0,00%

x9 = PNV

0

1

0

0,4

0,87%

0,00%

x10 = CC

2

0

1

0,7

1,46%

0,00%

608

7

M. Sol´e et al.

UIAS Evaluation

This section highlights the results obtained by the implementation of UIAS. A set of 616,235 users, along with its interactions, has been analyzed. This streaming data has been extracted during one month, between 27/03/2019 and 28/04/2019. The system supports a massive and continuous flow of data on the streaming data process, which on average deals with 1,153,023.3 tweets and 154,059 users per week. All processes are deployed in a Storm machine. The use of this technology enables streaming processes to manage and operate huge volumes of tweets, as well as provide the scalability of the whole system. The Storm machine has been deployed on a virtual machine hosted in an Open Nebula server and is composed by 16 GB RAM, 16 CPU (AMD opteron (TM) processor 6274) and 30GB Disk. The processing time for users interactions in the streaming flow on Storm, regardless the Twitter API waiting time, is 3795 milliseconds and the Hadoop MapReduce to calculate global trends operations spends 29 min. As pointed out in the previous section, the set of objective users O is reduced to OM when restrictions concerning the minimal amount of interactions are required (UNI and MIT parameters). These parameters had to be tunned in order to maximize the assert of the estimation. To select proper parameters, several combinations for these parameters have been tested. The global political tendencies obtained with each combination have been contrasted with a Spanish elections results realized on 28/04/2019. Table 5 shows the results for four different combinations of these parameters. The third column shows the number of users for each assignation filtered by the Map Reduce operations. The fourth column shows the average of differences (AVG). To compute this parameter, the absolute difference between the global political tendency Gx and the percentage of voting on the elections results is computed for each party x. Then, AVG is the average of these differences. Finally, last column shows the standard deviation (SD). The analysis of the AVG reveals that all cases obtain an average difference lower than 5 points, which can be considered satisfactory. Likewise, the SD for all the cases is lower than 5.5 points with slight differences between the analyzed cases. Therefore, Sample 1 seems to be the best candidate to select the UNI and MIT parameters since it obtains the smallest AVG and SD. However, note that since UNI and MIT are more restrictive for this sample, the number of sample users is reduced to 2.95% of the total sample. Table 5. UNI and MIT combinations results. UNI MIT Sample 1

20

15

Sample 2

15

Sample 3

10

Sample 4

3

|OM |

AVG SD

18,201

3.5

3.5

10

42,492

4.05 4.11

5

147,113 4.13 4.36

2

389,200 4.81 5.35

How Twitter Interactions Leak Political Trends

609

Therefore, according to the previous analysis, UIAS strategy has been implemented considering the UNI and MIT values 20 and 15, respectively. Taking these parameters into account, the assert of the user tendency calculation has been analysed with a sample of 300 random users, which has been contrasted with their real political tendencies (manually retrieved). The results achieve 88.2% of assert, which confirms that this approach is suitable for being used to obtain global calculation. Finally, the global tendency for each political party predicted by UIAS has been contrasted both with the general election results and also with CIS3 , which is an official electoral statistical estimation elaborated one month before the elections4 . This comparison is shown in Fig. 8. In general, the accuracy of the UIAS strategy related to the elections results is very high for the majority of the parties (Ciudadanos, EHBildu, Podemos and CC and PP). Likewise, compared with the CIS Directed Vote results, UIAS has a better approach in several political parties (Ciudadanos, EHBildu, PP, Podemos and VOX). Furthermore, the average and deviation of the differences among CIS and the elections results are AVG = 4.65 and SD = 4.20, which are worst than the parameters attained by UIAS (in fact, are also worst that the results obtained in Sample 2). Hence, these results confirm that the accuracy in relation to the CIS Directed Vote is performed better by UIAS.

Fig. 8. Elections results, UIAS and CIS directed vote comparison

3 4

The center of psychologies researches of the Spanish Goverment http://www.cis.es/ cis/opencms/ES/index.html. Macrobarometer March 2019 pre-election http://www.cis.es/cis/export/sites/ default/-Archivos/Marginales/3240 3259/3242/es3242mar.pdf.

610

8

M. Sol´e et al.

Conclusions

This paper proposes a new streaming strategy, named Users Interaction Analysis Strategy (UIAS), to analyze the Twitter users’ political trends. The political tendency of each user is determined by analyzing and classifying the user interactions. From these results the global political tendency is obtained. The UIAS framework extracts interactions in a streaming process and analyses the relationship between each user and politicians from the different parties. Taking into account the user interactions, an affinity rate is defined for each user in relation to each political party. In order to compute the global political tendency, only relevant users are considered. Experimentation suggests that a proper selection would be users with a minimal number of interactions over 20 (UNI) and at least 15 interactions of one type (MIT). In order to validate the goodness of the global political tendency results of UIAS, they were contrasted with the results of elections and the CIS Directed Vote. The UIAS forecast differs by only 3.50 points on average in relation to the real elections results, while CIS obtains a deviation of 4.65 points. As future work, user tendency behavior could be tracked on a timeline and, eventually, determine whether it might be linked to significant political facts. Likewise, it would be worth studying whether the UIAS strategy might be refined by means of integrating sentiment analysis.

References 1. Bastian, M., Heymann, S., Jacomy, M.: Gephi: an open source software for exploring and manipulating networks. In: International AAAI Conference on Web and Social Media (2009) 2. Baviera, T.: Influence in the political Twitter sphere: authority and retransmission in the 2015 and 2016 Spanish general elections. Eur. J. Commun. 33, 321–337 (2018) 3. Blondel, V.D., Guillaume, J.-L., Lambiotte, R., Lefebvre, E.: Fast unfolding of communities in large networks. J. Stat. Mech: Theory Exp. 2008(10), 10008 (2008) 4. Conover, M.D., Goncalves, B., Ratkiewicz, J., Flammini, A., Menczer, F.: Predicting the political alignment of twitter users. In: 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, pp. 192–199, October 2011 5. De Silva, L., Riloff, E.: User type classification of tweets with implications for event recognition. In: Proceedings of the Joint Workshop on Social Dynamics and Personal Attributes in Social Media, pp. 98–108, June 2014 6. Demchenko, Y., Grosso, P., De Laat, C., Membrey, P.: Addressing big data issues in scientific data infrastructure. In: 2013 International Conference on Collaboration Technologies and Systems (CTS), pp. 48–55, May 2013 7. Dorle, S., Pise, N.: Political sentiment analysis through social media. In: 2018 Second International Conference on Computing Methodologies and Communication (ICCMC), pp. 869–873 (2018) 8. Fruchterman, T.M.J., Reingold, E.M.: Graph drawing by force-directed placement. Softw. Pract. Exp. 21, 1129–1164 (1991)

How Twitter Interactions Leak Political Trends

611

9. Guo, X., Li, J.: A novel twitter sentiment analysis model with baseline correlation for financial market prediction with improved efficiency. In: Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS) (2019) 10. Kruikemeier, S.: How political candidates use twitter and the impact on votes. Comput. Hum. Behav. 34, 131–139 (2014) 11. Lee, K., Palsetia, D., Narayanan, R., Patwary, M.M.A., Agrawal, A., Choudhary, A.: Twitter trending topic classification. In: 2011 IEEE 11th International Conference on Data Mining Workshops, pp. 251–258, December 2011 12. Mangal, N., Niyogi, R., Milani, A.: Analysis of users’ interest based on tweets. In: Gervasi, O., et al. (eds.) ICCSA 2016. LNCS, vol. 9790, pp. 12–23. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-42092-9 2 13. Pennacchiotti, M., Popescu, A.-M.: A machine learning approach to Twitter user classification. In: International AAAI Conference on Web and Social Media, pp. 281–288 (2011) 14. PreoT ¸ iuc-Pietro, D., Liu, Y., Hopkins, D., Ungar, L.: Beyond binary labels: political ideology prediction of Twitter users. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 729–740 (2017) 15. Schroeder, D.T., Pogorelov, K., Langguth, J.: Fact: a framework for analysis and capture of twitter graphs. In: 2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS), pp. 134–141 (2019) 16. Sol´e, M., Gin´e, F., Valls, M., Bijedic, N.: Real time classification of political tendency of Twitter Spanish users based on sentiment analysis. Int. J. Comput. Inf. Eng. 12(9), 697–706 (2018) 17. Tumasjan, A., Sprenger, T., Sandner, P., Welpe, I.: Predicting elections with twitter: what 140 characters reveal about political sentiment. In: International AAAI Conference on Web and Social Media (2010)

Public Reaction on Social Media During COVID-19: A Comparison Between Twitter and Weibo Tian Wang, Ian Brooks(B) , and Masooda Bashir University of Illinois at Urbana-Champaign, Champaign, IL 61820, USA [email protected]

Abstract. COVID-19 has been spreading across the world starting from early 2020, and there are numerous and varied discussions on social media platforms related to the COVID-19 outbreak. In this study, we analyzed and compared the data on Twitter and Weibo at different times based on the public’s understanding of COVID-19 to ultimately understand the characteristics of social media reaction in U.S. and in China during the pandemic. Results show that both similarities and differences existed when comparing the public reaction on social media in the U.S. and in China. The study suggests that data from social media could be used as a good reflection of the public’s reaction, especially in a pandemic like COVID-19. It is important for the government to understand people’s timely reaction during the pandemic in order to ensure the authorities are on the right direction to provide services and accurate information to the public. Keywords: Social media · Social response · Data analytics

1 Introduction Since December 2019 with the first confirmed case in Wuhan, China, the novel coronavirus, officially named COVID-19 by the World Health Organization (WHO) on February 11, 2020, has quickly spread across the globe. WHO officially declared it as a public health emergency of international concern (PHEIC) on January 30, 2020. By the end of June, 2020, there were over 10 million confirmed cases worldwide. Although different countries have different trends on the spread of COVID-19 because of the size of population, government regulations, effectiveness of medical care services, etc., the early stage of the outbreak in most countries has the similar characteristic of starting with a relatively small number of initial cases and then seeing a significant increase in the number of cases. Because the coronavirus was not contained and has now impacted nearly every country in the world, individuals and communities around the world may share similar experiences. Meanwhile, as the coronavirus itself circulates the globe, numerous discussions have been taking place regarding the wide range of information being presented, as well as the legitimacy of the sources presenting said information. Traditional news media, like The New York Times and The Washington Post, have published a number of related © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 612–625, 2022. https://doi.org/10.1007/978-3-030-80119-9_38

Public Reaction on Social Media During COVID-19

613

reports and have developed separate sections on their websites dedicated to keeping people updated on the latest information about COVID-19, including case maps, reopening trackers, and instructions on living at home. However, most of the information presented to the public was done so through social media platforms. During the early stage of the COVID-19 outbreak while the virus was mainly contained to Asia, there were a lot of posts related to the outbreak on Weibo (a Chinese social media platform), but only few discussions on Twitter and Facebook, presumably since there were no many confirmed cases in the United States at that time. Things changed dramatically in early March, 2020. The volume of social media posts related to COVID-19 almost quadrupled, as the number of confirmed cases in the United States quickly skyrocketed. Some of the most commonly posted content on social media includes sharing latest news or updated information related to COVID-19, expressing personal attitudes or emotions, and discussing about current trends. Although social media could be used to deliver up-to-date news during the outbreak, unlike reputable news sources (e.g. newspaper) that need to go through the editing process to publish the content, posts on social media might include controversial content given the large number of personal expressions being posted and shared [1]. For example, at the early stage of COVID-19 outbreak, an overwhelming trend appeared with uses of racist terms like “Wuhan Virus” and “Chinese Flu” [2] in the social media posts to describe the coronavirus, and it might bring the topics into racial discrimination instead of focusing on the outbreak itself. Considering people on social media had various reaction towards COVID-19, and such reactions may be reflective of how the public deals with such a serious outbreak, we want to take a closer look at the public reaction on social media. China and the United States are selected as the specific countries to observe in this study because of their distinctive characteristics: China was the first country that experienced the COVID-19 outbreak, and the United States has the most confirmed cases currently. In this research study, we observed and analyzed how people in the United States and people in China reacted on social media during the early stage of the COVID-19 outbreak. Twitter and Weibo are selected as the social media platforms in this study since these social media platforms are most commonly used and popular in their respective country. The research questions addressed and discussed are described as the following: 1. What are the characteristics of Twitter and Weibo content during the early stage of COVID-19 outbreak? 2. Specifically, what kind(s) of content do people post on Twitter and Weibo as COVID19 spreads? Does it change over time? 3. Are there any similarities and differences between Twitter and Weibo content? This research study aims to understand how information related to COVID-19 spread by observing public reaction on social media in different regions (U.S. and China) during the outbreak. Results from this study could be helpful for governments and organizations to build better strategies to provide information to public during a pandemic. It is also important for governments and public health departments to communicate with the public effectively and appropriately to prevent and control the spread of COVID-19.

614

T. Wang et al.

2 Background 2.1 Power of Social Media Analysis Since December 2019 with the Researchers have conducted data analysis on social networks for many years. Social media obtains massive amount of information about the users; Not only the basic demographics of social media users [3], but also the way users interact with information on social media. A previous study [4] suggested that social network users tend to acquire information based on their worldviews. Different models were built for content analysis by using information on social media, and previous research showed that such information could be used for identification and classification of social media users. Examples include using machine learning method to make a classification on users’ profile and Tweet content [5], identifying hateful users in terms of activity and word using pattern [6], and analyzing political manipulation during 2016 Russian interference using Twitter users’ geolocation and political ideology [7]. To conduct social network analysis, some software toolkits are frequently used, including R, SNA library, JUNG, Guess, NodeXL and Gephi. These software toolkits allow researchers to collect and browse the data with a broad overview, as well as visualize and interpret data in a graphic way [8]. 2.2 Importance of Social Media After Crisis Previous literature has pointed out the importance of social network analysis in research studies, especially after a significant event happens [9]. Furthermore, social media can also help people prepare for an upcoming disaster by warning the community prior to the event and bringing people together afterwards [10]. In the early development stages of social networking, after 2010 Haiti earthquake happened, information spread on social media allowed the Red Cross to receive over 8 million US dollars with 2 days, as well as brought people’s attention to the disaster, which proved the benefit of social networks in information propagation [11]. The US Federal Emergency Management Agency utilizes several social network platforms to offer the public emergency information about a catastrophe. 2.3 COVID-19 Related Works Only a few months after the initial COVID-19 outbreak, there were already a large number of studies related to social networks and the outbreak. Researchers from Rochester University studied the controversial terminology on social network platforms during the outbreak [12]. Some studies focused on using information from social media to make estimation or predictions. Cinelli et al. [13] addressed the diffusion of information about COVID-19 by using data from Twitter, Instagram, and YouTube, to provide an estimate of rumors’ amplification. Another work by Li et al. [14] conducted a retrospective analysis with social media data, trying to figure out a method of predicting the outbreak. Since COVID-19 started in China, many previous research studies were conducted based on Chinese social media. A previous study by Zhao et al. [19] observed Weibo (a Chinese social media platform) to see Chinese public’s attention to the COVID-19 epidemic on

Public Reaction on Social Media During COVID-19

615

social media. Similar work by Li et al. [18] analyzed Weibo data to understand how the outbreak has been propagated on social media. In addition, there were some research studies observed the characteristics of social media information. A previous study found that the pandemic of social media panic travels faster than coronavirus outbreak itself [15].

3 Method 3.1 Data Collection To study how the public reaction changed on social media during the COVID-19 outbreak, Twitter and Weibo were selected as the representative sources of information as typical examples of social media platforms in the U.S. and in China to gather data in this study. By searching keywords “COVID-19” and “coronavirus” on Twitter and Weibo (those keywords were translated into Chinese when searching on Weibo), posts related to COVID-19 during the early stage of the outbreak were retrieved and analyzed. Since only public social media reaction in the U.S. and in China are compared in this study, restrictions on language (English) and location (the United States) were added as filters when retrieving data from Twitter. For Twitter data retrieval and analysis, Crimson Hexagon, a data analytics tool, was used to automatically present the results with specific filters or settings. For Weibo data, multiple APIs supported by Weibo were applied for data analytics. In this study, we will focus on the results from the following five categories to describe the characteristics of Twitter and Weibo posts during the early stage of the outbreak: • • • •

Trends on Volume: Trends on the number of posts during the time period. Sentiment: If the post is positive, negative, or neutral sentiment. Most Posted Topics: The primary topics being posted on the social media. Top Mentions: The most mentioned social media accounts, calculated by the number of re-posts and posts that mention that author. • Content: types of content being posted during the time period, especially focusing on the percentage of official announcement and news on the social media posts in order to analyze the public’s awareness of the COVID-19 outbreak. 3.2 Data Analysis In order to observe and understand the change of public reaction on Twitter and Weibo over time, four timestamps were defined based on the occurrence of certain events with possible significant influence on public reaction. The four timestamps chosen for COVID-19 in the United States are described as following: • December 31, 2019: Official announcement from Wuhan Municipal Health Commission on 27 confirmed cases of viral pneumonia, which was further named as Coronavirus. • January 20, 2020: First case confirmed in the United States, who is a 35-year-old man in Snohomish County, Washington.

616

T. Wang et al.

• March 11, 2020: President Trump decided to ban all travel from 26 European countries. • April 1, 2020: The U.S. surpasses 200,000 cases. The four timestamps chosen for COVID-19 outbreak in China are described as following: • • • •

December 8, 2019: First confirmed COVID-19 case in China. January 23, 2020: Wuhan lockdown. February 12, 2020: China surpassed 50,000 confirmed cases. April 8, 2020: Wuhan officially announced to stop the lockdown.

Although there were differences between the spread of COVID-19 in the U.S. and in China, similarities still existed in the early stage of the outbreak, especially on public’s understanding and attitudes towards the outbreak. Therefore, based on the public reaction at different time periods, the outbreak was divided into three stages: 1) Stage 1, as the initial stage, at which people didn’t know much about the situation and the impact of COVID-19 was unknown; 2) Stage 2, at which the outbreak started to spread, and people became aware of COVID-19. People began to experience changes in their personal lives due to the outbreak, which led to different kinds of feelings and emotions; 3) Stage 3, the governments and organizations started to take actions and had basic controls over the outbreak. People got to know more information about the outbreak. In this stage, people would have more responses on the policies published, and had actual reactions to deal with the outbreak. Table 1 shows all the three stages in the United States and in China. Table 1. Time period for each stage in the United States and in China Public reaction

The United States

China

Stage 1: Seek for Information

Dec 31–Jan 20

Dec 8–Jan 23

Stage 2: Express Personal Feelings

Jan 21–Mar 11

Jan 24–Feb 12

Stage 3: Follow Government Instructions

Mar 12–Apr 1

Feb 13–Apr 8

Data from the social media platforms for the three stages was collected separately. For each stage, results of all five categories mentioned in the above section was recorded first, and then the results from all the three stages in these two countries will be compared, analyzed, and discussed. Additionally, a set of original posts from each stage was randomly selected as a sample to observe the content that people post on Twitter and Weibo during each stage of the outbreak.

Public Reaction on Social Media During COVID-19

617

4 Results 4.1 Trends on Volume The number of posts on Twitter and Weibo in each day during the three stages was recorded to analyze the trends on volume. Overall, the number of Twitter posts related to COVID-19 has increased since it was first confirmed in December, 2019. For stages 1 and 2, the number of posts continually increased and reached the peak at the end date of the stages (January 20 and March 11), at which major event(s) happened. For stage 3, although there were ups and downs as shown in Fig. 1, the number of posts began to decrease starting March 12, 2020. However, there is a huge increase between the first two stages and stage 3. As shown in the figure, the number of posts in stage 3 is much higher than the numbers in stages 1 and 2.

Fig. 1. Overall Twitter volume from 12/31/2019 to 4/1/2020

Figure 2 shows the overall Weibo volume during the specified time range. As shown in the trend, there were not many Weibo posts related to the outbreak in stage 1. After Wuhan locked down, the number of posts increased significantly in stage 2, and reached the peak value in mid-February when COVID-19 had been spreading in all the provinces in China. Since stage 3, for each day, the number of posts almost stayed the same, or slightly increased or decreased without dramatic changes.

Fig. 2. Overall Weibo volume from 12/8/2019 to 4/8/2020

To have a better understanding of the difference between the volume trends on Twitter and Weibo, Fig. 3 compared the trends on Twitter and Weibo in all three stages (Twitter: orange, Weibo: blue line, from the left to right: stage 1, stage 2, and stage 3). Since the number of posts on Twitter and Weibo varied from each other and different scales were applied, Fig. 3 is more focused on the trends or changes on the volume, instead of the

618

T. Wang et al.

actual number for each day. As Fig. 3 shows, there were only a slight number of posts in stage 1 for both Weibo and Twitter. The pattern of the volume trends was also similar in stage 2 and stage 3 for both social media platforms, while it increased significantly at the end of stage 2, and started to decrease since stage 3.

Fig. 3. Comparison between Twitter and Weibo volume in all three stages

4.2 Sentiment Figure 4 showed the percentage of posts with positive, negative, and neutral sentiment for each stage in the United States and in China. As shown, in the United States, the percentage of positive and negative posts increased from stage 1 to stage 2 (2% to 8% for positive posts, and 13% to 34% for negative posts), but the percentage of positive and negative posts didn’t change significantly from stage 2 to stage 3. It is notable that there was a slight increase on positive posts from stage 2 (8%) to stage 3 (11%), with decreased negative posts concurrently (from 34% to 31%). Meanwhile, the percentage of positive posts started with a high number (30%). It kept increasing as the outbreak

Fig. 4. Percentage of posts with sentiment for each stage

Public Reaction on Social Media During COVID-19

619

continued (from 30% to 33%), but dropped from stage 2 (33%) to stage 3 (22%). The percentage of negative posts was almost the same for stage 1 and stage 2, but increased slightly in stage 3 (19%). During all the three stage, over half of the posts stayed neutral. Table 2 provides some examples of Twitter and Weibo posts during the three stages with different sentiments (the original Weibo posts were in Chinese, and were translated into English). In positive posts, people were trying to stay optimistic and encourage others to deal with the situation, while people expressed their frustration and stress in negative posts. Most neutral posts were about personal experience and statement without any emotions, or facts and news without any bias. Table 2. Examples of posts with different sentiments in all three stages Stage

Sentiment

Example of posts

1

Positive

“We should all also thank Professor Yong-Zhen Zhang, from Fudan University for releasing the Wuhan Coronavirus sequence. I am sure his group was under lots of pressure…” (Twitter) “I wish everyone could be safe and healthy, and if we stay together, we could overcome the outbreak.” (Weibo)

Neutral

“What is Coronavirus and what are the symptoms?” (Twitter) “Can we have a test kit for everyone?” (Weibo)

Negative

“uhhhhh I do not feel good about this coronavirus situation, folks” (Twitter) “The novel coronavirus seems so terrible!!” (Weibo)

Positive

“Yes, our prayers are with all the ppl affected by this #coronavirus outbreak…” (Twitter) “I felt like hope still exists when I saw the latest news that there were new recovered cases.” (Weibo)

Neutral

“Has anyone written about how coronavirus may interfere with the 2020 census?” (Twitter) “The first thing I did when I woke up is to check my cellphone on the COVID-19 updates.” (Weibo)

Negative

“This is an absolutely terrible idea. Why bring these people to Alabama when we can quarantine them where they are?” (Twitter) “I felt so upset about the current situation on the outbreak.” (Weibo)

Positive

“THANK YOU TO ALL THE MEDICAL PROFESSIONALS, first responders, and all workers still working to keep us all safe.” (Twitter) “I must use the time to study more when I stayed at home.” (Weibo)

Neutral

“It’s more important than ever to prioritize your mental health….” (Twitter) “This is the 7th day for my self-quarantine.” (Weibo)

Negative

“Getting so frustrated w/my community! They are congregating outside and many kids are playing together!…” (Twitter) “I had to wait until 3pm to check in and wait for the quarantine, very frustrated.” (Weibo)

2

3

620

T. Wang et al.

4.3 Most Posted Topics For U.S., the most frequently discussed topics on Twitter were “new virus”, “new cases”, and “novel coronavirus” while people didn’t have enough information on the coronavirus at the first stage, and there wasn’t an official name for the virus yet. The primary topics changed to “Wuhan”, “China”, and “Trump” in stage 2, and people were more focusing on “Getting”, “Testing”, “Americans” in the last stage. Since China confirmed the first case of COVID-19, people on Weibo have been searching topics related to the coronavirus. In stage 1, the most posted topics on Weibo were mainly focused on COVID-19 in China, including “coronavirus”, “outbreak”, “Wuhan”, “test”, and “confirmed case”. Moving onto stage 2, topics on Weibo were more about how to deal with the situation, and topics like “prevent”, “control”, “arrange”, and “supplements” were being posted. In stage 3, people were talking about COVID-19 in other countries. Some examples included “Russia” and “Sierra Leone”. People were also caring about post-COVID care at this stage, topics like “antibodies” and “healthcare system” were also being posted and discussed a lot. 4.4 Top Mentions For the U.S., in stage 1, top mentions included government and authorized organization like WHO and CDC, as well as some news websites like Bloomberg (@business). Since stage 2, the President has been the most mentioned Twitter user, and more politicians, such as Vice President (@VP) and Nancy Pelosi, Speaker of the House (@SpeakerPelosi) were involved in the top mentions. However, types of top mentioned users in China were slightly different. In stage 1, the most mentioned Weibo author (5549 reposts) was the official account for a public platform operated by Chinese Public Welfare Fund with the primary goal of raising funds for charity. The second and third most mentioned users are official accounts for news websites, while the fourth and fifth were verified personal accounts with large number of followers. In stage 2, all the top mentions were official accounts for news websites. In stage 3, there were more official accounts, either from the government directly (top 1 mentioned author), or operated by the government (second, third, and fifth mentioned users). 4.5 Content of Posts Different kinds of content were posted on Twitter and Weibo during the outbreak. To have a general understanding of the content, we exported the dataset and generated a random sample of original posts from the two social media platforms in each stage. After observation, most of the posts could be categorized into the four types described as below: 1. Personal feelings or experiences: people express their personal feelings or describe their personal experiences related to COVID-19 outbreak. 2. Personal statement or opinions: people’s attitude or judgement on COVID-19 or related events.

Public Reaction on Social Media During COVID-19

621

3. Sharing external link: the post is created because users chose to share something from external website instead of the social media itself. 4. Official announcement: original posts from verified users related to government or authorized organizations like WHO, CDC, or news websites. Since there was a huge number of posts in each stage and it required manually reading and understanding the content for categorization, especially the content including personal attitudes and feelings, this research study would only focus on the content that was official announcement or updated news related to COVID-19 (including shared link of announcement or news). Content with official announcements or news updates would imply how much the public knew about the situation of the outbreak, in other words, the public’s awareness of the outbreak. Content of announcements and news was identified in two ways: 1) original posts from a user account that is a verified account for the governments, authorized organization, or news website, and re-posts of such content; 2) posts including an external link to official organizations’ websites (for example, sharing a latest news published by WHO on Twitter). Since the total volume for both Twitter and Weibo posts were too large, for each social media platform, 10,000 posts in each stage were selected randomly for data analytics. Figure 5 shows the results on the percentage of Twitter and Weibo posts which included official announcements and news in its content for all three stages. From the results, the percentage of content involving official information were different in the United States and in China. For Twitter posts, the percentage of content including official announcements and updated news was very high at the initial stage, and then decreased from 85.40% to 48.57% when the outbreak moved to stage 2. From stage 1 to stage 2, people were more interested in expressing their personal feelings

Fig. 5. Percentage of posts including official announcement and news

622

T. Wang et al.

and experiences on Twitter, instead of simply sharing information on official announcements and news. However, the percentage of shared official announcements and news increased in stage 3, at which the government started to take more actions and guidance, and there were more information about testing and treatment. Unlike Twitter, the percentage of Weibo posts with content including announcement and news started with a low rate (only 23.61% at stage 1), and slightly increased as time passed by. Even though the percentage kept increasing, it was still less half of the posts in stage 3. From further observation, people on Weibo were more likely to express their personal feelings, or discuss about their personal experiences, instead of simply sharing the latest news or official information.

5 Discussion Results showed both similarities and differences between public social media reaction in the United States and in China. In general, people from those two countries had more responses on social media as the COVID-19 outbreak continued spreading. For both countries, one of the primary reflections was the number of social media posts, which increased significantly from stage 1 (started initially) to stage 2 (continued spreading). Even though the volume didn’t keep increasing rapidly after stage 2, the average number of posts per day was much higher in stage 3 comparing with stage 1. For Twitter posts, the volume reached the highest point when President Trump announced the travel bans for European countries (which was also the same day that WHO declared COVID-19 as a pandemic). For Weibo posts, the volume reached the peak value on February 18. While the COVID-19 continued spreading mainly in areas close to Wuhan, February 18 was the first time that there were more recovered cases than new confirmed cases in China. Comparing the trends of volume on Twitter and Weibo, it might imply that the public in U.S. could possibly have more reaction if there are official announcements or regulations from the governments and authorities, while people in China may be more focusing on the situation of COVID-19 itself. The top mentioned authors in Twitter and Weibo also had similar characteristics. There were more governments and authorities accounts being mentioned on Twitter, but accounts for news website (mostly only published latest news without any personal opinions) were more likely to be mentioned on Weibo. One of the reasons behind this situation might be the difference on general social media regulations. People in China would have more freedom to express their personal feelings and opinions when commenting on or re-posting from non-governmental Weibo accounts, since the official government accounts sometimes will not allow individual users to make comments on their Weibo posts. However, the government and authorized organizations post content on Twitter more frequently, and public Twitter posts are available for all the Twitter users to interact with. From the results of the Sentiment and Emotion category, it is obvious that people’s reaction on social media changed drastically during different stages of the outbreak. At the earlier stages, before WHO defined an official name for COVID-19 and the U.S. saw its first confirmed case, there wasn’t much discussion related to COVID19 on Twitter. Most people on Twitter tended to have a neutral attitude towards the

Public Reaction on Social Media During COVID-19

623

outbreak during this time. However, after the first confirmed case, people in the U.S. were expressing more personal feelings on social media. Some of them were worried about the spread of outbreak by posting more about outbreak tracking or getting tested. Some of them expressed negative feelings due to the effect on their daily life by the outbreak and quarantine restrictions. Results showed that there were more people on Weibo than Twitter that were trying to stay positive during COVID-19 outbreak by encouraging others or praying (with higher percentage of positive posts in all three stages). Overall, while the public became more sensitive after COVID-19 started to spread in both countries, people in China might be more positive on the spread of the outbreak than people in the United States. Previous research found that the way media communicates and the way each person reacts to an event could be different and conditioned by unique factors and the local circumstances of the region that the event occurred [16], which could explain the difference between people’s emotions on Twitter and Weibo. The comparison between posts sentiment on Twitter and Weibo Similar characteristics could be found on the topics being posted and discussed as well. The topics may not only depend on the situation of the outbreak, but also be affected by the governments’ regulations and the culture of the region. With the government taking actions and control over the outbreak, people on Weibo were following the specific instructions made by the governments when the outbreak continued, and started to discuss about situations in other countries when it was under control in China. In the United States, even though the spread of COVID-19 continued to grow, the focus of topics shifted from healthcare to more general topics like politics and economics. After the first confirmed case in the United States, President Trump has been the top 1 mentioned author on Twitter posts related to COVID-19. It is understandable that the public in U.S. wants to hear from a powerful authority like the President or Speaker of the House during such a pandemic. However, unlike China, different states in the U.S. were establishing different policies towards the outbreak without nationwide consistency. Because of this, people on Twitter may have received different information which lead them to make their own judgements and conclusions rather than follow a clear instruction outlined by government authorities.

6 Limitations The first limitation of this study is the difference between how the COVID-19 outbreak grows in the United States and in China. While it has been declining and is nearly controlled in China, COVID-19 is still an ongoing outbreak in the United States. Although there were similarities during the early stage of the outbreak, the three stage described in this study were not exactly equal because of the difference between the outbreak in these two countries. Additionally, different social media platforms may have different targeted group of users. For example, a research conducted by Pew Research Center indicated that Twitter users are young and more educated than general public [17]. Therefore, this specific case study on Twitter can only reflect to a part of the U.S. population’s reaction on social media during the outbreak. This is the same for using Weibo to gain a representation of the Chinese public’s reaction to the coronavirus on social media. In further study, data analytics on multiple social media platforms are recommended in order to

624

T. Wang et al.

have a more comprehensive understanding. Additionally, there was bias on categorizing the content of social media posts since all the posts from Twitter and Weibo were reviewed manually. One possible way to reduce the influence of personal bias towards the results is to ask more reviewers with different backgrounds to categorize the content in the future study. Another limitation is that major cultural and regulatory differences exist in these two countries. Since the population in U.S. and in China is from different national origin, they may have various beliefs, behaviors, and ways of expression. In this case, it might be difficult to simply measure and compare the social media reactions because of too many possible factors that might influence the public reaction other than the outbreak itself. There were also limitations raised from using data analytics tools to retrieve information. Since the algorithm of those data analytics tools are unknown, it is possible that sentiments of some posts may be misclassified (for example, sometimes the keyword “happy” was used for sarcasm, but it was classified as positive sentiment), and it could have a negative impact on the accuracy of the results. Also, the accuracy of the inclusion and exclusion of the social media posts was not perfect. Some social media posts were not included even though it had the keyword “coronavirus” or “COVID-19” in the content, while there were some included posts was completely irrelevant to the topic.

7 Conclusion It is important for government and organizations to observe and understand public reaction during such a pandemic like COVID-19 in order to ensure that they are providing helpful services and accurate information to the public. Social media platforms like Twitter and Weibo could be powerful tools to reflect public reaction considering the huge amount of social media users and information shared on social media. In this study, relevant Twitter and Weibo data from the early stage of COVID-19 outbreak was collected and analyzed in order to understand the public reaction on social media in different countries during the outbreak. Results showed that the public had different reactions at different time periods in general, and people were more involved and gave more responses as the situation became serious or major events happened. However, although there were similarities on social media reaction, people in U.S. and in China may still behave differently towards similar situations because of cultural and regulatory differences. Given the limitation of this study, further research is recommended to have more comprehensive observation on overall public reaction on social media, especially in the United States, by extending the time period. Also, it is necessary to compare multiple social media platforms (for example, Facebook in U.S. and WeChat in China) since different platforms may have different targeted group of users. By monitoring how people react on social media during the COVID-19 outbreak, government and healthcare organizations could use such information to take further actions efficiently to provide necessary services and avoid social panic.

Public Reaction on Social Media During COVID-19

625

References 1. Agrawal, A.J.: It’s Not All Bad: The Social Good of Social Media. Forbes, March 2016. https://www.forbes.com/sites/ajagrawal/2016/03/18/its-not-all-bad-the-social-good-of-soc ial-media/#6db6c224756f 2. Lin, C.Y.: Social reaction toward the 2019 novel coronavirus (COVID-19). Soc. Health Behav. 3(1), 1 (2020) 3. Chang, J., Rosenn, I., Backstrom, L., Marlow, C.: epluribus: ethnicity on social networks. In: Fourth International AAAI Conference on Weblogs and Social Media, pp. 18–25, May 2010 4. Bessi, A., Coletto, M., Davidescu, G.A., Scala, A., Caldarelli, G., Quattrociocchi, W.: Science vs conspiracy: collective narratives in the age of misinformation. PLoS ONE 10(2), e0118093 (2015) 5. Pennacchiotti, M., Popescu, A.M.: A machine learning approach to twitter user classification. In: Fifth International AAAI Conference on Weblogs and Social Media, vol. 5, no. 1, July 2011 6. Ribeiro, M.H., Calais, P.H., Santos, Y.A., Almeida, V.A., Meira Jr, W.: Characterizing and detecting hateful users on twitter. In: Twelfth International AAAI Conference on Web and Social Media, vol. 12, no. 1, June 2018 7. Badawy, A., Ferrara, E., Lerman, K.: Analyzing the digital traces of political manipulation: the 2016 Russian interference twitter campaign. In: 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pp. 258–265. IEEE, August 2018 8. Butts, C.T.: Tools for social network analysis. R package version, 2 (2010) 9. Kim, J., Hastak, M.: Social network analysis: the role of social media after a disaster. In: 10th Anniversary Homeland Defense/Security Education Summit, vol. 38, no. 1, pp. 86–96 (2017) 10. Houston, J.B., et al.: Social media and disasters: a functional framework for social media use in disaster planning, response, and research. Disasters 39(1), 1–22 (2015) 11. Gao, H., Barbier, G., Goolsby, R.: Harnessing the crowdsourcing power of social media for disaster relief. IEEE Intell. Syst. 26(3), 10–14 (2011) 12. Lyu, H., Chen, L., Wang, Y., Luo, J.: Sense and Sensibility: Characterizing Social Media Users Regarding the Use of Controversial Terms for COVID-19. arXiv preprint arXiv:2004.06307 (2020) 13. Cinelli, M., et al.: The covid-19 social media infodemic. arXiv preprint arXiv:2003.05004 (2020) 14. Li, C., Chen, L.J., Chen, X., Zhang, M., Pang, C.P., Chen, H.: Retrospective analysis of the possibility of predicting the COVID-19 outbreak from Internet searches and social media data, China, 2020. Eurosurveillance 25(10), 2000199 (2020) 15. Depoux, A., Martin, S., Karafillakis, E., Preet, R., Wilder-Smith, A., Larson, H.: The pandemic of social media panic travels faster than the COVID-19 outbreak. J. Travel Med. 27(3) (2020) 16. Mejía, F., Brooks, I., Marti, M., David, N.O., Cosio, G.D.: Fear on the networks: analyzing the 2014 Ebola outbreak. Revista Panamericana de Salud Pública 41, e134 (2018) 17. Wojcik, S., Hughes, A.: Sizing Up Twitter Users. Pew Research Center (2019). https://www. pewresearch.org/internet/2019/04/24/sizing-up-twitter-users/ 18. Li, L., et al.: Characterizing the propagation of situational information in social media during covid-19 epidemic: a case study on weibo. IEEE Trans. Comput. Soc. Syst. 7(2), 556–562 (2020) 19. Zhao, Y., Cheng, S., Yu, X., Xu, H.: Chinese public’s attention to the COVID-19 epidemic on social media: observational descriptive study. J. Med. Internet Res. 22(5), e18825 (2020)

Word Frequency Analysis of Community Reaction to Religious Violence on Social Media Samah Senbel1(B) , Carly Seigel2 , and Emily Bryan1 1 Sacred Heart University, Fairfield, CT 06825, USA

[email protected] 2 Weston High School, Weston, CT 06883, USA

Abstract. Researchers in data science, psychology, and linguistics have recognized the value in assessing community reactions to traumatic experiences on social media. Twitter in particular, is popular for sharing opinions and discussing current events. In this work, we collect and analyze the reaction to three shootings at different houses of worship. Three events were chosen: the massacre at the Emanuel Church in Charleston, NC, the massacre at the Tree of Life Synagogue in Pittsburg, PA, and the massacres at two mosques in Christchurch, New Zealand. The events were all committed by shooters with a similar supremacist mentality. We used the python programming language to collect the data from twitter using popular hashtags for the events. We then cleaned it up by removing punctuation, web links, stop words, hashtags, and foreign letters. We then did a word frequency count on the remaining words, and generated word clouds for each event. We also did a search for well-known hate words in the corpus. We found out that the three events generate a different amount of chatter and people use very different words to describe each of them. The church dataset had the most varied set of popular words with the shooter’s name being the most used. The synagogue dataset had the least amount of tweets and two words dominating the discussion: “shooting” and “synagogue”. The mosque dataset had the least discussion about the shooter in comparison to the other events. On the positive side, very few hate tweets were found in any dataset. Keywords: Word frequency · Social media · Religious violence

1 Introduction Religious violence has unfortunately become a recurring event around the world, with houses of worship directly targeted. The purpose of this work is to study the reaction of the public to these events. Regular media is a good way to measure the reaction of the media and government, but the community’s reaction are more of interest, as they measure the true sentiment on the street. Social media sites are a great way to measure people’s reaction to these event through their posting, sharing, comments and “likes”. It would be of interest to see how long the subject is discussed on social media, what words are used in the discussion, and whether the discussion is different based on the religious house of worship being targeted. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 626–633, 2022. https://doi.org/10.1007/978-3-030-80119-9_39

Word Frequency Analysis of Community Reaction to Religious Violence

627

For this study, we chose three events of religious violence based on their high volume of media coverage and people’s reactions. The first event was the Emanuel African Methodist Episcopal Church shooting on June 17th , 2015. Nine African-American worshippers were shot dead during a Bible study by declared white supremacist Dylann Roof [1]. This event has a racial as well as a religious aspect to it. The second event was the attack on the Tree of Life Synagogue in Pittsburg, PA on October 27th , 2018. The shooter was Robert Bowers and he killed 11 mostly elderly worshippers [2]. The third event was the shooting of 51 worshippers at two mosques in Christchurch, New Zealand. Once again, the shooter was a declared supremacist, Brenton Tarrant [3]. The victims were mainly Muslim emigrants to New Zealand.

2 Background Twitter is being increasingly used to publish both, information about an event as well as the online community reaction to it. This enormous cache of information can manifest itself in a number of ways ranging from supporting social resistance through calls for change and assistance and spreading of advice [4] and [5]. Researchers have always studied the reactions on social media to important world events to gain insight about people’s reaction. Burnap et al. studied the reaction on Twitter to a terrorist attack in London, UK. They built a model to predict information flow size and survival using data derived from Twitter. The found that tweets with sentimental content had a positive effect on its spread through re-tweeting. Also, tweets by the press was a good predictor of the size of the reaction of the community [6]. Another interesting study was by Dore et al. who studied the aftermath of the elementary school shooting at Sandy Hook, CT. Using a large Twitter data set, they identified temporal and spatial patterns in use of emotional and cognitive words in tweets about the Sandy Hook Elementary School shooting. Although use of sadness words decreased with time and spatial distance, use of anxiety words showed the opposite pattern and was associated with concurrent increases in language reflecting causal thinking [7]. Human emotion in reaction to violence in college campuses on Twitter was measured by Jones et al. They used longitudinal Twitter data across 3 case studies to examine the impact of violence near or on college campuses in the communities of Isla Vista, CA, Flagstaff, AZ, and Roseburg, OR, compared with control communities, between 2014 and 2015. They observed an increase in post-event negative emotion expression among sampled followers after mass violence, and show how patterns of response appear differently based on the timeframe under scrutiny [8]. Building on these previous work, we concentrate on comparing the reaction of the public to very similar events, the only difference being the faith of the victims. Are the emotions and wording used dependent on the event itself, the faith or the victims, or the prejudices of the tweeters.

3 Method Several social media sites are used by the public to share their opinions about current affairs. Twitter was chosen as the sample of social media reaction to hate events, because

628

S. Senbel et al.

of its popularity and large number of subscribers. Twitter also enables the community the collect data from it for analysis. Our data was collected over three batches, one for each event. The software Python library “Twint” [9] was used to collect historic tweets for the events on October 19th , 2019. Table 1 shows the number of tweets collected for each event and the hashtags used. Each tweet data contains the tweet text, number of likes, number of retweets, tweeter ID, tweeter name, and tweet date as well as other fields. Table 1. Number of tweets collected for the three events Event

No. of tweets Hashtags used

Charleston Church

383,288

#CharlestonMassacre, #CharlestonShooting, #EmanuelAME, #Charleston9, #AMEshooting, #AMEterrorism, #AMEmassacre, #PrayforCharleston, #StandwithCharleston, #DylannRoof, #Emanuelnine, #CharlestonStrong, #Emanuel9, #MotherEmanuel, #CharlestonTerrorist, #DylannStormroof

Pittsburg Synagogue

147,320

#TreeofLifeSynagogue, #TreeofLife, #SynagogueShooting, #PittsburgSynagogue, #PittsburgShooting, #RobertGregoryBowers, #Bowers, #RobertBowers, #RobBowers, #PittsburgMassacre, #PittsburgShooter, #PittsburgSynagogueShooting, #SquirrelHill, #SynagogueMassacre, #PittsburgStrong

Christchurch Mosque 400,901

#ChristChurchShooting, #Christchurchmosqueshooting, #Mosqueshooting, #Christchurchterrorattack, #Christchurchattack, #Christchurchmosque, #Christchurchmassacre, #Brentontarrant, #Newzealandmosqueattack, #Newzealandattack, #Newzealandshooting, #Nzmosqueshooting, #Christchurch, #Tarrant

The data was then cleaned by removing numbers, punctuations, web links, hashtags, and foreign letters. For this work, we only concentrated on the tweets themselves, and ignored the number of likes, replies, and retweets. This is because our goal is to study the words used in the tweets. A word frequency count was then performed for all the data of the three events as well as for each dataset separately using the python code shown in Fig. 1. It was then charted into a simple bar graph, as shown in Fig. 2. The next step was to search for the occurrences of any hate words in the tweets. A list of common hate words associated with each religion can be found in reference [10]. Word clouds were also generated for the events, as shown in Fig. 3. The data and the output word frequency chart and word cloud were used as a teaching tool in two classes “CS650 Big Data Analytics” and “ENG201 Understanding Hate”.

Word Frequency Analysis of Community Reaction to Religious Violence

629

word_count = {} for word in final: word = word.lower() # make all words lowercase if len(word)>=4 and word[-1] == 's' and not(word[len(word)-2]=='e') and not(word[len(word)-2]=='s'): # make plural words into singular word = word[:len(word)-1] if '#' in word: # Remove hashtags but keep the word word = word[1:] if ('.' in word) or ('!' in word) or (',' in word): # remove punctuations word = word[:len(word)-1] if word in word_count: word_count[word]+=1 else: word_count[word]=1

Fig. 1. Python code for word frequency count

4 Results The word frequency analysis provided some interesting findings about the community’s choice of words in reaction to each event. The number of tweets varied for each event, with the synagogue shooting getting a much lower tweets counts than the other shootings (Table 1), in spite of the great similarity of the way the shooting was performed. For the Charleston massacre, the most frequent words was actually the name of the killer himself, both as a hashtag and in regular text, as shown in Fig. 2a. The general discussion was also concentrated on race issues with popular words like “white”, “black”, “confederate”, and “flag”. So clearly, this event generated a community discussion about both the killer and the underlying racial aspect of this shooting. In the synagogue shooting, two words clearly dominated the discussion: “Synagogue” and “shooting”, as shown in Fig. 2b. This was followed by the name of the shooter and that of president Trump. Another interesting observation is that the victims themselves was relatively unpopular in both the Charleston and the Pittsburg massacres, as shown by the frequency of the words “victim”, “jewish”, “jews”, and “black”. The mosque massacre shows a different pattern of community reaction, with more emphasis being put on the victims compared to the killer. The words “muslim” and “victim” are more popular than that of the shooter “tarrant”. There has been a media push to not use his name or photo in media coverage of this event. The emphasis was placed on the victims and gun legislation, and this was clearly reflected on the community tweets. Another interesting observation was the very popular use of the word “attack”, and not “shoot”, as the word to describe the event. After searching for the hate words in reference [10], it was interesting that few of them was found in our three datasets. We conclude that this is due to twitter’s strong policy against hateful conduct. Figure 3 shows the word clouds of religious and violence words found in each dataset, and hate-filled words and tweets are quite rare. Table 2 shows a sample of the few hate tweets found.

630

S. Senbel et al.

a) Word frequency count of the tweets about the Charleston church shooting

b) Word frequency count of the tweets about the Pittsburg synagogue shooting

c) Word frequency count of the tweets about the New Zealand shooting

Fig. 2. Word frequency count for the reaction to the three religious events

Word Frequency Analysis of Community Reaction to Religious Violence

a) Word Cloud for religious/hate words in the church massacre

a) Word Cloud for religious/hate words in the synagogue massacre

a) Word Cloud for religious/hate words in the mosque massacre

Fig. 3. Word clouds of hate-related words in the three events

631

632

S. Senbel et al. Table 2. Sample of hate-filled tweets found in the datasets

Event

Sample hate-filled tweets

Church massacre

“There’s already too many black ghetto #PorchMonkeys & thugs in North Carolina, we don’t need more(from South Carolina) #CharlestonShooting”

Synagogue massacre “You know that the Jewish people couldn’t let Christians get all of the media attention, because of Sri Lanka. So they decided to shut the Christians down with a #synagogueshooting. To them, one Jewish life is worth more than 250 goyim lives” Mosque massacre

“I need a Tarrant I’m holding out for a Bowers to defend what is white He’s gotta remove a kebab or a jew And he’s gotta be streaming it live I need a Tarrant I’m holding out for an Earnest to strike fear in a kike He’s gotta be wasting a taco or asian And inspire our people to”

5 Conclusion In this work, we used the python programming language and the Scikit-learn library to analyze the content of over 900,000 tweets posted in reaction to three religious violence event. Three houses of worship of the Christian, Jewish and Islamic faiths were attacked in an almost identical fashion by supremacist shooters. The data was collected from Twitter, cleaned by removing hyperlinks, punctuations, stop words, and foreign words. Then, we performed a word frequency count for the tweets text, and then searched them for a list of hate words used for each faith. Our observation was that the response was different in each case: the number of tweets varied, as well as their content. The shooter got a lot of attention in the church and synagogue shootings, but much less in the mosque one, where the victims got more attention. Also, the word “shoot” was common in the church and synagogue reactions, whiles “attack” was more common in the mosque reaction. Another observation was the general lack of non-sympathetic or hateful tweets, although there was a lot of discussion about the motives of the killer and most of these was about how hate became a. Future work on this project includes looking at the frequency and time-line of the tweets to discover how long each event is of interest to the public, as well as comparing the tweet texts to the number of likes they get, the number of retweets and the number of followers of the tweeter.

References 1. Cassidy, W., La France, B.: Routine adjustments: how journalists framed the Charleston shooting. Int. J. Commun. 12, 4668–4688 (2018) 2. McIlroy-Young, R., Anderson, A.: From “welcome new gabbers” to the Pittsburgh synagogue shooting: the evolution of gab”. In: Proceedings of the Thirteenth International AAAI Conference on Web and Social Media ICWSM, pp. 651–654 (2019) 3. Every-Palmer, S., Cunningham, R., Jenkins, M., Bell, E.: The Christchurch mosque shooting, the media, and subsequent gun control reform in New Zealand: a descriptive analysis. J. Psychiatry Psychol. Law (2020)

Word Frequency Analysis of Community Reaction to Religious Violence

633

4. Burnap, R., et al.: Detecting tension in online communities with computational twitter analysis. Technol. Forecast Soc. Change 95, 96–108 (2013) 5. Procter, R., Vis, F., Voss, A.: Reading the riots on Twitter: methodological innovation for the analysis of big data. Int. J. Soc. Res. Methodol. 16(3), 197–214 (2013) 6. Burnap, P., et al.: Tweeting the terror: modeling the social media reaction in the Woolwich terrorist attack. Soc. Netw. Anal. 4, 206 (2014) 7. Doré, B., Ort, L., Braverman, O., Ochsner, K.: Sadness shifts to anxiety over time and distance from the national tragedy in Newtown, Connecticut. Psychol. Sci. 26(4), 363–373 (2015) 8. Jones, N., Wojcik, S., Sweeting, J., Cohen Silver, R.: Tweeting negative emotion: an investigation of Twitter data in the aftermath of violence on college campuses. Psychol. Methods 21(4), 526–541 (2016) 9. Poldi, F.: Twint Python library. https://github.com/twintproject/twint. Accessed 21 Sept 2020 10. Hatebase project. https://hatebase.org/search_results. Accessed 24 Sept 2020

Humans Digital Avatar Reconstruction for Tactical Situations Animation Ilya Zuev1 , Alexander Gilya-Zetinov1(B) , Alexander Khelvas1 , Egor Konyagin2 , and Julien Segre3 1

Moscow Institute of Physics and Technologies, Institutskii per. 9, Dolgoprudnii, Russian Federation {zuev.iv,gilya-zetinov}@phystech.edu 2 Skolkovo Institute of Science and Technology, Bolshoy Boulevard 30, bld. 1, Moscow, Russian Federation 3 Stanford University, Stanford, CA 94305, USA http://www.mipt.ru

Abstract. Today video surveillance systems are omnipresent and increasingly complex, requiring the simultaneous recording, storage, and transfer of information in real time. Such requirements necessitate the efficient storage of large quantities of data and corresponding techniques to optimize the flow of pertinent information. We propose a novel algorithm for the conversion of video data into a simple vector format which is then used to reconstruct a video of the given scene including all needed actionable information for both security personnel and automation. Neural networks are used to detect and localize human keypoints (corresponding to human joints), which can then be stored to render the scene depicting the actions of individuals and more complex events relevant to security concerns. Keywords: Artificial intelligence · Computer vision · Action recognition · Dimensionality reduction · Data compression

1

Introduction

Today, high volumes of raw video streams are collected from surveillance cameras and subsequently stored on servers. The most important information of interest is stored in the interest of public security. For such purposes, data are often stored and required for further processing by either human security personnel or by computer vision algorithms. The former implies that the video stream must be stored for a given amount of time in order to be reviewed in person. The stored video files may prove useful for further analysis - namely, the identification of security concerns: the occurrence of complex events emerging from the actions of one or more individuals’ behavior. Pose estimation algorithms, such as [4] and [5], were chosen for the analysis of individuals’ behavior. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 634–644, 2022. https://doi.org/10.1007/978-3-030-80119-9_40

Humans Digital Avatar Reconstruction for Tactical Situations Animation

635

We propose an algorithm that can solve these kinds of tasks, i.e. extract and store useful information from video files instead of storing the raw video file, thereby economizing storage space. This useful information is stored as a feature vector containing the following data: – coordinates of 12 keypoints corresponding to human joints (e.g. neck, shoulders, etc.), – color description of human appearance (this item is described in section...). Additionally, we propose a method for the reconstruction and display of video files from the extracted feature vectors. The algorithm consists of two main components: an encoding, as well as a decoding process. The encoding process extracts the specified feature vectors from the current frame and stores them. During the decoding part, the stored information is used to render an image depicting the corresponding scene using keypoint and color data of the individuals present. If the video is recorded by a static video camera (the most common surveillance case), the static background image (i.e. image describing the location and containing no people) can be extracted and reinserted in order to compress the output video file without any significant loss of vital information. In contrast, the detected individuals are represented only by the keypoints necessary for three dimensional pose estimation and aggregate color data - the output image is then composed by superimposing the encoded feature vectors onto the static background image. The output image is not lossless in comparison to the original image, but all detected individuals in the original image are rendered with enough detail to support later identification or analysis. The information extracted during the encoding part is following: – keypoints corresponding to human joints, – color description of human outlook. The algorithm shows impressive results in terms of video compression ratio, as only 50 float variables are used for description of each person present in the scene, so these variables consume less than 1 kB per person per frame. Moreover, the algorithm is invariant in terms of memory consumption to the video files of different dimensions, it depends only on the number of people in the frame and on the frame rate. This feature may decrease the time of request processing if the video needs to be transferred from the server to the local machine. The information extracted to determine human keypoints for pose estimation may theoretically prove useful if used as input in another computer vision algorithm: human keypoint localization may be used for tracking purposes, appearance may be useful for human re-identification algorithms, and the spatial and postural relations of multiple individuals can be related to derive higher level event detection i.e. fighting, stealing.

2

Related Work

One of the most popular approaches for video compression is the implementation of Deep variational autoencoders [1]. These algorithms belong to a class of gener-

636

I. Zuev et al.

ative models, i.e. they create a high-level output from a sampled vector. Several models have proven to show very impressive results. Both encoding and decoding in variational autoencoders is performed by neural networks. This means that these processes are equivalent in terms of resource consumption. In this method, the processes of encoding and decoding asymmetrical, meaning image decoding consumes is notably more computationally intensive than image reconstruction. This allows for the use of the algorithm on distributed systems with multiple computing nodes, each tailored for encoding or for decoding. This work was highly inspired by the DensePose [2] algorithm developed by the Facebook Artificial Intelligence Research (FAIR) group. This work is devoted to specific keypoint extraction. These keypoints can be considered as necessary information for further pose reconstruction. The architecture of the model consists of two interdependent steps, one refining the other. We wanted to implement a similar approach, while including additional color data, specifically the aggregate color of the regions defined by the selected keypoints. Our method thus can reconstruct more fully detailed models in comparison to DensePose algorithm, which extracts exclusively pose data. However, neither DensePose nor our algorithm yield fully realistic renders of humans upon reconstruction. Another generative approach for human image synthesis was proposed in [3]. In this model it was proposed to use two separate inputs: 1) pose description vector, 2) learnable human descriptors. The latter vector was used in the model to build the realistic image given the target pose. This algorithm can also be used for 3D human synthesis. However, it also requires neural network deployment for both encoding and decoding parts, therefore it requires more powerful computations to be done.

3

Formal Task Description

From the mathematical point of view, the task of data compression is the problem of finding corresponding mapping f from the class of mappings F from original high dimensional space to final low dimensional space: F = {f |f : Xorig → Xf inal }, dim Xf inal < dim Xorig . In this case the mapping class F is not invertible, i.e. it is not possible to restore the actual image after mapping is applied, but for the purposes of the work the necessary attributes can be reconstructed from that mapping, while improving efficiency. In this case the necessary attributes are some descriptors related to humans present in the frame. As a set of such descriptors a keypoints array was chosen, where each keypoint corresponds to a human joint, and the colors of 10 specific regions defined by the keypoints (e.g. color of human limbs). The regions are distributed in the following way: – 4 regions define color appearance of arms (2 regions per each arm); – 4 regions define color appearance of legs (2 regions per each leg);

Humans Digital Avatar Reconstruction for Tactical Situations Animation

637

– one region defines the color appearance of the main body; – one region defines the color appearance of the head. These features are extracted and stored after the mapping f from class F is applied. Now F can be considered as a parametrized class of mappings f , where the parameters define the location of limbs as a function of corresponding joint locations. In this work the good parametrizations of F was empirically found so that the reconstructed image allows us to understand the pose of original humans as well as their limb colors.

4

Method Description

The goal is to form a vector representation of a scene containing a large number of people in motion, taking into account their movements, and then restore the image without losing the information about captured human actions. The approach comprises two main parts. The first one is detection of human figures in the field of view of a high-resolution camera and the formation of a vector of parameters describing the pose and a set of additional attributes for each detected person in the frame. In the simplest case, the only parameter is color of body parts. The second part is the creation of a representation for the scene based on skeletal models, in which the skeletons are replaced with simple graphic primitives. Figure 1 describes the algorithm of the developed solution. Source video is processed and separated to individual frames and then they are used to estimate human pose via an existing method, e.g. [4, 5], and color of body parts using the designed algorithm. Optionally, the frames are also used for background image creation if it is not provided by external tools. Poses and colors for each individual are stored for all frames in a standardized JSON (JavaScript Object Notation) file along with path to background image. Files produced by this method consume less than 1% of memory space than original video. 4.1

Video Preprocessing

The source video is a sequence of N frames Akuv , k ∈ 0, N . Example of the video frame is shown in the Fig. 3 as the initial frame. Preprocessing of the source video includes the extraction of single frames. 4.2

Extracting the Background Image

This application includes default background extraction algorithm based on the median frame of the video. All the available frames Akuv are used to make pixelwise median image Cuv , where each pixel cij = median(akij ). This method is k∈0,N

useful when the background is static without moving instances such as escalators, digital billboards and advertisements, stars, etc. In other cases other approaches are recommended, e.g. photo editing, selection of the frame without foreground objects or taking a still background picture with minimal interference.

638

I. Zuev et al.

Fig. 1. Scheme of the algorithm

4.3

Human Poses Detection and Estimation

Each frame Akuv is processed by the human posture assessment system. For this work the freely available Detectron2 framework was used for key-point estimation [5]. This system performs two tasks: → 1. the search and localization of the specified human joints − r l , l ∈ 0, 17 of the skeletal model (nose, ears, eyes, shoulders, elbows, hands, hips, knees and ankles) in the frame; 2. the assembly of the found joints into M k separate skeletons (i.e. matching the found joints to people in the frame), where M k represents the number of people in k-th frame. Thus, a vector Xjk,m , m ∈ M k , j ∈ 0, 34 with the coordinates of the specified → 17 keypoints − r l of the skeletal model in the COCO format [7], presented at Fig. 2, was obtained for each person in the frame. The Detectron2 framework also provides information on joint visibility, contained in a vector of scores, Slk,m , for each of 17 joints. Joint visibility can be determined by the thresholding of the corresponding score. Examples of the output human posture are shown in the Fig. 3 as keypoints detection outputs. 4.4

Calculation of the “Color” Parameter of Body Parts

The obtained Xjk,m vector is used to locate keypoints in a frame. Dominant color for each of the 12 joints (shoulders, elbows, hands, hips, knees and ankles) and

Humans Digital Avatar Reconstruction for Tactical Situations Animation

639

Fig. 2. 17-keypoint skeletal model used in the work

10 body parts (head, body, arms, forearms, thighs and shins), which is localized between particular joints (e.g. left shin is localized between left ankle and left knee), is calculated with the following color estimation algorithm. It takes into account a set of pixels P that is distributed, depending on the instance which color is calculated, either around single joints or between joints which constitute the respective body part. For each person limbs are defined by the position of two keypoints, representing human joints (e.g. knees and ankles), the head is defined by the position of both ears and the body is defined by the position of two keypoints with the mean coordinates of shoulders and hips respectively. In case of single joints, coordinates of pixels Px on x-axis are taken from a horizontal interval with the pixelwise width w and the center in the correspond→ ing joint − r l , l ∈ 0, 12 in the amount of max(0.2 · w, 5): Px ∈ [rxl − 0.5 · w, rxl + 0.5 · w],

(1)

→ → w = αq · dist(− r le , − r re ),

(2)

→ → where − r le , − r re represent coordinates of the left and the right ears of the → → r re ) is considered as the width of human head, which person (therefore dist(− r le , − is the reference size for joints’ drawing sizes in the algorithm, as will be described in Sect. 4.5), αq is the coefficient that is passed to the algorithm in a configuration file and is used for further drawing Sect. 4.5. Coordinates of pixels Py on y-axis are sampled from the normal distribution on the axis, perpendicular to the xaxis: (3) Py ∼ N (0, 0.15 · w), When determining the color of a body part, defined by two component joints, → → r 2 , the relative pixel coordinates, Px , on x -axis which are represented with − r 1, − for a pixel contained by P are directed along the line segment passing through the joint positions and are taken in the amount of max(0.2 · l, 5): Px ∈ [−0.5 · l, 0.5 · l],

(4)

640

I. Zuev et al.

→ → l = dist(− r 1, − r 2 ),

(5)

Coordinates of pixels Py on y  -axis for are sampled similarly from the normal distribution on the axis, perpendicular to the x -axis: Py ∼ N (0, 0.15 · h),

(6)

h = γp · l,

(7)

where h corresponds to the pixelwise height of the body part, γp is the coefficient that is passed to the algorithm in a configuration file and is used for further rendering of the body part ellipses Sect. x - and y   4.5. Obtained  cos θ − sin θ r 2 −r 1 coordinates are modified with rotation matrix R = , tan θ = rx2 −rx1 y y sin θ cos θ in order to get true frame coordinates. Px , Py → Px , Py R

(8)

The pixels P are processed in order to extract the dominant color of the body part. The four algorithms that are useful for color determination were considered: 1. Color space algorithm: each pixel from P is located in three dimensional RGB color space P ⇐⇒ RGB[255, 255, 255], which are then sorted according to the number of pixels corresponding to the color. RGB coordinates of the largest number of pixels is the required ones. 2. MFV (most frequent value) algorithm is an improvement of the above algorithm: set of pixels P is listed and then processed with the unique elements finding algorithm from [6] that counts number of times each unique value comes up in the list. The most frequent value is considered as the required one. 3. K-means clustering is a method of vector quantization that aims to partition n observations into k clusters. The clusters are parametrized by a single point per cluster called cluster centroid (kn ). After this parametrization is performed, the point xi belongs to cluster Kj if dist(xi , kj ) = min dist(xi , kp ). p∈K

The K-means algorithm is an iterative method that minimizes the inertia value, or the within-cluster sum-of-squares criterion: k1...K = min

kp ∈K

n 

dist(xi , kp )2 .

i=1

In this case, the K-means algorithm is applied to the list of pixels, which are considered as elements of three-dimensional linear space, from P in order to identify the central color of the body part. The K-means algorithm can be also considered as a method of variance reduction: only k unique values of colors are defined.

Humans Digital Avatar Reconstruction for Tactical Situations Animation

641

4. PCA (principal component analysis) [8]: This approach implies that color space is a three-dimensional linear space. The idea of PCA is to find optimal subspace in the initial space which describes the variance of the data in the best way. The dimension of optimal subspace was selected to be equal to one. After that transform, the color space will be transformed to 1D subspace, in which it is less computationally expensive to find the most frequent value. After this value is obtained, it is necessary to apply inverse transform to this element to find the element in initial color space corresponding to the most frequent color in the transformed space. The guarantees for optima of the transformed subspace are given by properties of SVD (singular value decomposition), which lies behind PCA. Obtained dominant colors for each joint and body part are stored in RGB format as vector Yik,m , i ∈ 0, 66. Examples of obtained dominant colors are presented in the Fig. 3. 4.5

Constructing the Graphical Representation of the Scene

Fig. 3. Stages depiction. From left to right, top to bottom: initial frame, keypoints detection, ellipses rendering, ellipses filling with captured colors

Developed drawing algorithm aims to reconstruct video in the most clear way based on extracted attributes Cuv , Xjk,m , Slk,m , Yik,m (background, keypoints coordinates, their visibility scores, colors) from compression file. Examples of algorithm outputs are shown on Fig. 4. The main goal is to render humans and their movements with enough detail to allow for identification. This approach employs asymmetrical encoding-decoding

642

I. Zuev et al.

model which leads to faster image reconstruction on the end devices as information is encoded on video capture or video processing devices and does not have to be decoded on output devices. Human body is rendered according to the rules of artistic people drawing with practical improvements: there are 9 body parts (body, arms, forearms, thighs and shins), represented by ellipses with joints Xjk,m (or its mean values in case of body, as described in Sect. 4.4) on their major semi-axes at a distance of 0.9 · ap , where ap , p ∈ 0, 8 is a length of the major semi-axis; human head, represented by a circle with the middle between the ears in the center and the → → → → r re as a diameter Dh = dist(− r le , − r re ) and 12 joints distance between ears − r le , − (shoulders, elbows, hands, hips, knees and ankles) represented by circles with the joints in the center and the diameter d = αq · Dh , q ∈ 0, 11. Examples of drawn ellipses are shown in the Fig. 3. Ratio γp between semiaxes’ lengths of different ellipses and αq between joints’ circles diameter and head diameter can be changed by the user and passed to the algorithm in a configuration file. Each body part and joint is monochrome and filled with the corresponding color from Yik,m , as shown in the Fig. 3 with the filled ellipses. All the ellipses are drawn only if all the corresponding joints are visible (its visibility score sk,m exceed the threshold) in order not to cover significant spots on the background and limbs of other people in the frame. All the figures are drawn over the background image Cuv and constructed images Gkuv constitute the resulting video.

5

Results

The main result of the work is the confirmed possibility of skeleton - based avatars usage for scenes reconstruction and visualisation. The resulting data size for scenes recording is about 1000 times smaller than initial video file size. This estimation lead us to the possibility of scenes recording and obtaining the data set with scenes for a long (several years) period. This data set can be used for tactical situations recognition and estimation without source video. Color estimation algorithms for this work has been implemented on the Intel(R) 8-core Xeon(R) Silver 4108 CPU @ 1.80 GHz system, where four approaches were tested: – – – –

Color space MFV K-means PCA

MVF and PCA algorithms proved to be the most efficient and, moreover, they can be run in parallel for even better performance in further studies. As a result of proposed approach the possibility to represent the video scene with only a background image and a vector of human attributes is gained, as shown at Fig. 4.

Humans Digital Avatar Reconstruction for Tactical Situations Animation

643

Fig. 4. Examples of the algorithm output (right) compared to initial frames (left)

There are no baggage units or trolleys the resulting image. This is separate part of scenes recognition and reconstruction algorithm and is out of presented research. Acknowledgments. The reported study was funded by Russian Foundation of Basic Researches, project number 19-29-09090.

References 1. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, 14–16 April 2014, Conference Track Proceedings (2014) 2. G¨ uler, R.A., Neverova, N., Kokkinos, I.: DensePose: dense human pose estimation in the wild. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, pp. 7297–7306 (2018). https://doi.org/10.1109/CVPR. 2018.00762 3. Dong, H., Liang, X., Gong, K., Lai, H., Zhu, J., Yin, J.: Soft-gated warping-GAN for pose-guided person image synthesis. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS 2018), pp. 472–482. Curran Associates Inc., Red Hook (2018) 4. Cao, Z., Hidalgo Martinez, G., Simon, T., Wei, S.-E., Sheikh, Y.A.: OpenPose: realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans. Pattern Anal. Mach. Intell. 43(1), 172–186 (2019). https://doi.org/10.1109/TPAMI. 2019.2929257 5. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2 (2019). https:// github.com/facebookresearch/detectron2

644

I. Zuev et al.

6. Harris, C.R., Millman, K.J., van der Walt, S.J., et al.: Array programming with NumPy. Nature 585, 357–362 (2020). https://doi.org/10.1038/s41586-020-2649-2 7. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1 48 8. Mackiewicz, A., Ratajczak, W.: Principal components analysis (PCA). Comput. Geosci. 19(3), 303–342 (1993)

Action Recognition in Handball Scenes Kristina Host, Marina Ivasic-Kos(B) , and Miran Pobar Department of Informatics, Center for Artificial Intelligence and Cybersecurity, University of Rijeka, 51000 Rijeka, Croatia {kristina.host,marinai,mpobar}@inf.uniri.hr

Abstract. Action recognition in sports, especially in handball, is a challenging task due to a lot of players being on the sports field performing different actions simultaneously. Training or match recordings and analysis can help an athlete, or his coach gain a better overview of statistics related to player activity, but more importantly, action recognition and analysis of action performance can indicate key elements of technique that need to be improved. In this paper the focus is on recognition of 11 actions that might occur during a handball match or practice. We compare the performance of a baseline CNN-model that classifies each frame into an action class with LSTM and MLP based models built on top of the baseline model, that additionally use the temporal information in the input video. The models were trained and tested with different lengths of input sequences ranging from 20 to 80, since the action duration varies roughly in the same range. Also, different strategies for reduction of the number of frames were tested. We found that increasing the number of frames in the input sequence improved the results for the MLP based model, while it didn’t affect the performance of the LSTM model in the same way. Keywords: Human action recognition · Action recognition in sport · Handball · Inception v3 · CNN · MLP · LSTM

1 Introduction In the modern way of life, more and more attention is paid to healthy lifestyle, recreation and physical activity. The development of technology, especially mobile phones with various sensors and sophisticated cameras that can record fast movement, have made it possible to record the activities and performance of athletes during sports schools and recreation, and not just top athletes. The collected data and their analysis can help the athlete or his coach to get a better insight into statistics related to the athlete’s activity, but more importantly, to analyze the performance of an action, finding key elements of technique that should be adopted to improve player performance. This is especially important in team sports where a large number of players are present in the field and perform different actions at the same time, so it is difficult for the coach to follow everyone. Also, the need and opportunity for efficient ways of indexing and using the huge amount of available sports visual data has © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 645–656, 2022. https://doi.org/10.1007/978-3-030-80119-9_41

646

K. Host et al.

led to a growing interest in automatic analysis of visual data and sports scenes, such as automatic player detection, tracking and recognition of actions the players perform. There are different sports on which researchers are focused today, such as basketball [1, 2], soccer [3–5], baseball [6], hockey [7, 8], volleyball [9–11], etc. In general, to recognize human actions, one can use visual data, data obtained with sensors, or a combination of the two. Many researchers collected visual data from different online sources or recorded it themselves, defined some human actions in the domain on which they focused, and tried or proposed different approaches for action recognition. For example, in [12], the focus was on two basic player actions in broadcast tennis video, left and right swing, that were recognized using motion analysis and Support Vector Machine (SVM). Their main challenge were the far-view frames when a player figure might be only 30 pixels tall. In [7], the researchers focused on recognizing four hockey activities (penalty corner, goal, long corner, and free hit) based on video samples which they collected from International Hockey Federation and YouTube. They proposed a model that utilizes the pre-trained VGG-16 Convolutional Neural Network (CNN) which was fine-tuned for classification of hockey activities. The authors report good results, with model sometimes confusing between free hit and long corner, as both activities share mostly similar visual pattern in terms of player’s position and appearance. In [2], a subset of the NCAA games available on YouTube was used to detect eleven events, and key actors in basketball games using only static information and basing their work on the Inception-7 [13] network. Authors of [14] extracted five broadcast videos from BadmintonWorld.tv channel on YouTube to compare the performance of different feature extraction methods for the task of recognizing five actions (clear, drop, lift, net shot, and smash) with a modification of Alexnet CNN and SVM. In [15], the Siamese spatio-temporal convolutional neural network that takes as an input RBG data images and Optical Flow was suggested to detect and classify actions in table tennis on a video dataset taken by GoPro cameras. Except in sports, human action recognition is used in various domains such as video surveillance, abnormal activity identification, healthcare monitoring, and education, as detailed in a survey [16]. In this paper we focus on handball action recognition in video, for which different computer vision tasks should be combined. Firstly, the object or person detection can be applied to detect the players and the ball on the field [17], and secondly, the object tracking can be applied to follow the players’ movements across the field [18, 19]. The active players are determined so that the key elements in a match can be followed and carriers of the game identified [20, 21]. Lastly, the action recognition can be applied on video sequences containing only the player of interest to recognize different actions across the field, in order to extract statistical data or analyze the players’ performances. All experiments are performed on video sequences from the dataset of handball training sessions [22]. Since different handball actions can have different durations, here we explore how the number of frames that are input to the classifier and the way the frames are sampled from the original video affects the performance of two different classifiers for the action classification task. Furthermore, there are not many researches focusing on analysis of handball, so our idea is quite unique, but follows the main concepts from other action recognition approaches in sports mentioned above.

Action Recognition in Handball Scenes

647

The paper is organized as follows: in the next section, there is an introduction on handball game, problem definition, and a description of the prepared handball dataset. The experiment, the results of the experiment and observations are given in Sect. 3, followed by the conclusion and future research directions in Sect. 4.

2 Experiment Workflow for Action Recognition in Handball Handball is a team sport played by 14 players divided into two teams. The point of the game is to use hands to pass the ball to each other in order to score a goal. During the game, every player is moving around the field and performing different actions. The analysis of these actions, during or after matches and practices, would be simpler if an automated system could recognize them. For that reason, different algorithms for human action recognition are being developed [18]. Before applying them, it is necessary to collect and process visual data like video, to make same dimensionality reduction if needed, and to extract features from the data. Here, features are automatically extracted using the InceptionV3 [23] deep neural network and two approaches were used to recognize actions, one model that does not take into account the sequential steps of which an action consists but classifies each frame separately and two models which take into account the time information and the sequence of frames in action. To evaluate the model performances, we have used validation loss and accuracy metrics. 2.1 The Dataset The dataset contains a set of short high-quality video recordings of actions in handball, recorded indoors during a handball school [22]. The recordings were made using a stationary Nikon D7500 DSLR camera, with a Nikon 18–200 mm VR lens, in full HD resolution (1920 × 1080) at 30 to 60 frames per second. To get the subset with actions in it, it was necessary to label the recorded data and to extract parts of the video containing the chosen actions. The obtained subset consists of 2,991 short videos in.mp4 and.avi format, belonging to 11 different action classes. Considered classes are: Throw, Catch, Shot, Jump-shot, Running, Dribbling, Defense, Passing, Double-pass, Crossing and Background where action is not happening. The subset was then split into training and testing sets in a ratio of 80:20. The distribution of videos through classes and sets is presented in Table 1. Table 1. Distribution of videos through classes and train/test sets Class name

No. videos (train)

No. frames (train)

No. videos (test)

No. frames (test)

Throw

184

4907

32

Catch

202

4120

50

937

Shot

83

6097

22

1655

270

18018

83

5676

Jump-shot

913

(continued)

648

K. Host et al. Table 1. (continued)

Class name

No. videos (train)

No. frames (train)

No. videos (test)

No. frames (test)

Running

56

6049

14

Dribbling

42

3549

11

753

Defense

97

6027

30

1668

Passing

509

30618

121

7252

Double-pass

1424

35

2122

11

654

Crossing

238

18204

59

4482

Background

684

24929

158

8342

2400

124640

591

33756

Total

The average number of frames in the short videos, depending on the action class the video belongs to, is shown in Fig. 1. It can be seen that most actions contain an average number of frames of around 60 or higher, with only Throw and Catch actions that are significantly shorter, that is, the average number of frames in which those actions occur in is approximately two or more times shorter than the other ones.

Fig. 1. Average number of frames per action from the handball dataset

An example of frame sequence for the Catch action, that is consecutive frames where a player is reaching out with his hands to catch a ball, is shown in Fig. 2. Because of the different typical lengths of actions, we tested models with different input lengths ranging from 20 to 60 frames. Since the Passing action consists of Throw and Catch actions in sequence, we also performed all experiments excluding these two classes, so we have a set of models trained on all 11 classes and a set of models trained on 9 classes.

Action Recognition in Handball Scenes

649

Fig. 2. An example of frame sequence for the catch action.

2.2 The Metrics The loss function represents a measure of errors made for each example during training and validation. It is used to optimize the algorithm and to conclude how well the model is performing after each iteration of optimization. If the model’s predictions are good, the loss is equal or close to zero, otherwise the loss is greater. Accuracy is a metric for evaluating the classifier’s performance, it shows how many predictions our model got right, that is, how accurate the model’s prediction is compared to the true data shown in percentage or in interval [0, 1]. If the model’s predictions are good the value increases towards a 100%. The point is to have low value of loss and high value in accuracy. 2.3 The Action Recognition Models As the baseline model, the InceptionV3 network [23] was fine-tuned on our handball dataset using the ImageNet [24] pre-trained weights as the starting point. The fine-tuning was performed so that the final output layer in the original network was replaced with a dense layer with 1,024 neurons followed by an output layer with 11 neurons with SoftMax activation function. The newly added layers were trained for 10 epochs with 12,4640 images per epoch using the RMSprop optimizer in the Keras framework with learning rate of 0.0001. Then, the top two inception layers and the newly added layers were trained for additional 25 epochs using the SGD optimizer with the learning rate of 0.0001 and momentum value of 0.9. In our experiment, this network is called CNN. CNN Voting Model The fine-tuned network was used to classify each frame in the video into one of the 11 classes, and then a majority voting scheme was used to determine the classification label for the whole sequence, so that the class label that occurs the most often among the predicted labels for each frame was selected as the label for the whole sequence. This step is presented in experiment as CNN voting. Additionally, for comparison with other networks that only see a fixed number of frames per video, the CNN network was tested so that the same number of frames that is used as input to other networks is classified. Then the majority vote is used again to obtain the class label for the video.

650

K. Host et al.

LSTM and MLP Models To capture the temporal information in the video sequence, two networks with different configurations were defined. The first is a LSTM-based network with one LSTM layer with 1,024 units, one fully connected layer with 512 neurons, each followed by a dropout layer with 0.5 dropout rate and a final output layer with 11 neurons. The second, hear called MLP, is a neural network with two fully connected hidden layers, with 512 neurons in each hidden layer, followed by a dropout layer with dropout rate of 0.5 The input to both networks consists of a sequence of features extracted from video frames using the baseline (InceptionV3) model, tapped at the final pooling layer, yielding 2,048 features per video frame. The networks were trained using the Adam optimizer with a learning rate of 0.00001 and decay of 10–6 for up to 100 epochs, stopping early if the validation loss does not improve for more than 20 epochs. Frames Selection Strategy Because the input to the network must be of equal length for each example, and the source videos containing actions consist of a different number of frames, ranging from 20–100, input videos have to be reduced to a fixed number of frames. Therefore, as a network input, we tested several different video sequences of 20 to 80 frames in length with different frame selection strategies. In the videos that contained fewer frames than the network expects, copies of existing frames were inserted between frames to extend the number of frames. Conversely, to reduce the number of frames of long videos, the chosen number of frames were selected consecutively from either beginning, middle or the end of the video, or from the whole video by decimation, i.e. by skipping some frames at regular intervals. This was tested since different actions might have most distinctive characteristics in different parts of the sequence, e.g. the beginning of a jump-shot action might be similar to the running action and thus not the most informative, while for some other action like throw the beginning when the player lifts the hand with the ball might be more important. Both networks were trained for each combination of string lengths and selection strategies.

3 Experimental Results and Observations The results of the experiment and some observation about the performance of different models are presented below. 3.1 Comparison of the Performances of Action Recognition Models In the first part of the experiment, we trained the CNN, LSTM and MLP models on all 11 classes and on 9 classes with different number of frames in the input sequence, obtained from the whole video by skipping frames at regular intervals (decimation of frames). The results in the terms of validation accuracy are shown in Fig. 3. The results for the MLP models are marked with dots, for the LSTM models with downward pointing triangles, and with upward triangles for the CNN models. The validation accuracy values are

Action Recognition in Handball Scenes

651

Fig. 3. Validation accuracy values obtained by skipping frames for 9 and 11 classes.

shown on the vertical axis, and on the horizontal axis are shown the number of frames used as input to the classifier (20, 30, 35, 40, 45, 60, 80). Out of all the models, the MLP model achieves the highest validation accuracy for both 9 and 11 classes. The best accuracy score of 81.95% the MLP model achieves with 60 input frames, but it achieves a similar score of 80.01% on average for all numbers of frames taken in the case of 9 classes. For 11 classes, MLP again has the highest accuracy score of 75% with the same number of input frames and the average accuracy for all tested number of input frames of approximately 72.25%. It can be seen that the validation accuracies for the model trained on 9 classes are higher than the ones trained on 11. In addition to the simpler problem with smaller number of classes, the reason for this is that Throw and Catch are fundamental parts of other actions (passing, double-pass, crossing, shot, jump-shot), which makes it more difficult to recognize the right action. The LSTM model achieved significantly lower validation accuracy values compared to the MLP model. It achieved the best score for 9 classes of 67.58% with 45 input frames (67.05% on average), while for 11 classes the best accuracy was 65.31% for 35 input frames (60.88% on average). The worst performance is achieved by the CNN model that has no temporal dimension with the best score of 44.5% for 30 input frames, but with minimal difference between scores for different input lengths (44.37% on average). With the majority voting scheme, the score improved for about 4% to 49.1% for 45 input frames, or 48.54% on average. From this result, it can be concluded that the temporal dimension plays a major role in action recognition. 3.2 Analysis of Model Performance with Respect to the Frame Selection Strategy In the second part, we analyzed the performance of the MLP and LSTM models based on the chosen number of frames sampled from either the beginning, the middle or the end of the video, or by skipping frames at regular intervals.

652

K. Host et al.

In the following Figs. 4, 5, 6 and 7, the different frame selection strategies are represented with different colors. The strategy with frames selected from the beginning is marked as first, the one with frames from the middle is marked as middle, with frames selected from the end of the video as last, and with frames selected by skipping frames at regular intervals as def . Figures 4 and 5 show the results obtained with the LSTM model trained for 11 classes, and 9 classes, respectively, taking into account different frame selection strategy. The maximum validation accuracy is obtained with the model trained on 11 classes with 45 frames taken from the middle of video, with the value of 70.94%, followed by 70.55% obtained with the model trained on 9 classes with 20 frames in the middle, and 70.47% for the last 45 frames and 9 classes.

Fig. 4. Validation accuracy for the LSTM models, 11 classes

Fig. 5. Validation accuracy for the LSTM models, 9 classes

If we take into consideration the number of frames for both 9 and 11 classes, the best results are obtained with 45 frames followed by 20 frames. In most cases, (except the best result overall), for the LSTM model, additional frames in the sequence don’t improve the result much over the models with 20 frames. Looking at the way the sequence is taken and not at the number of frames, the highest average accuracy (67.69%) is achieved by the model with 9 classes taking into consideration only the last frames, followed by 67,05% by skipping frames. Figure 6 and 7 show the results obtained with the MLP model trained for 9 and 11 classes, respectively.

Action Recognition in Handball Scenes

653

Fig. 6. Validation accuracy for the MLP models, 11 classes

Fig. 7. Validation accuracy for the MLP models, 9 classes

Model MLP achieves the maximum validation accuracy for 9 classes when is trained with 45 frames from the beginning (82.89%) followed by 82.11% obtained with 60 frames in the middle, and by 82.03% with the last 45 frames. In the case of 11 classes, the MLP model achieves the best results when it is trained with 35 frames, from the end of the video sequence (last), but comparative results achieve with the same number of frames from the middle and from the beginning of the video sequence (first). It is evident from the graph that much better validation accuracies are achieved with 9 classes than with 11 and also that the best results in case of 9 classes were achieved when 35 frames were used, and in the case of 11 classes when 45 frames were used. As opposed to the LSTM model, for the MLP model, the higher number of input frames in general increased the achieved accuracy. Looking at the way the sequence is taken and not at the number of frames, the highest average accuracy (80.33%) is achieved by the MLP model with 9 classes taking into consideration only the last frames, followed by 80.08% looking at the first frames. From the obtained results it is not possible to conclude exactly which frame selection strategy is the best and how many frames gives the best results because the results differ between models, and even for the same model for a different problem (9 vs. 11 classes). The number of frames and frame selection strategies appear to be highly dependent on the type of action being performed, so we will explore this in a future research.

654

K. Host et al.

In Fig. 8, validation accuracies for all combinations of MLP and LSTM models, frame selection strategies and number of frames for 9 and 11 classes, are presented. The validation accuracy values for the MLP models trained on 9 classes are marked with red dots, and for the models trained with 11 classes are marked with blue cross. The results for LSTM models are marked with upward pointing triangles for the model trained on 9 classes, and downward pointing triangles for the model with 11 classes. The number of frames used as input to the classifier are shown on the horizontal axis.

Fig. 8. Validation accuracy for MLP and LSTM

We can see that the MLP algorithm trained on 9 classes obtains best results regardless of the number of frames, or the way they are taken. The validation accuracies for the model trained on 9 classes are generally higher than the ones trained on 11 classes but MLP model with 11 classes performs better than LSTM with both 9 and 11 classes. The best overall result of 82.89% of accuracy is the one obtained with MLP model for 9 classes with 45 input frames. If we observe the average accuracy per number of frames for all observed strategies of frame selection, the highest average accuracy (81.62%) is achieved by the MLP model with 9 classes and 45 frames, followed by 60 frames (81.47%). By the same principle looking at the number of frames rather than the way the frame sequence is taken, the LSTM model achieves the highest average accuracy (68.28%) with 9 classes and with 20 frames, followed by 45 frames (68.07%).

4 Conclusion The goal of this paper was to test the performance of different algorithms for action recognition on our handball dataset containing short videos of 11 action classes. We compared the validation accuracies of a CNN-based classification model used as baseline that doesn’t use the temporal dimension of input videos but classifies the frames individually, with two models that do (LSTM and MLP). Both LSTM and MLP models are built on top of the CNN model, so that the input features to the LSTM and MLP are calculated using the CNN model. For LSTM and MLP models we tried different

Action Recognition in Handball Scenes

655

numbers of classes (9 and 11), different numbers of frames (20, 35, 40, 45, and 60) taken in different ways (from the beginning, the middle, the end, and by frame decimation). Overall, we obtained the best results with the MLP model with validation accuracy 82.89% for 9 classes trained on the first 45 frames of the video, significantly higher than the best result obtained with the baseline classifier (49.1%). We can conclude that the MLP and LSTM models successfully exploit the information in the temporal dimension to for recognizing handball actions. Our data set contains handball actions that differ significantly in the duration of the performance, so to train the model they had to be reduced to the same length. Experiments have shown that the number of frames and frame selection strategies can significantly affect the accuracy of the action recognition, however, an exact conclusion about the best strategy for frame selection from the obtained results is difficult because the strategy that gives the best results differs between models, and even for the same model for a different problem (9 vs. 11 classes). Nevertheless, for the LSTM model it can be concluded that increasing the number of input frames does not contribute to a better result, regardless of the frame selection strategy, while for MLP it could be concluded that a larger number of frames can positively affects the results of action recognition. The number of frames and frame selection strategies appear to be highly dependent on the type of action being performed, so the question of the ideal number of frames, and the question of best selection strategies remains opened. Therefore, it will be more deeply analyzed in further work, where each action will also be considered separately. Likewise, we plan to enlarge our dataset, especially for those actions for which we have fewer videos, to achieve a more balanced distribution of videos for different action classes. Acknowledgment. This research was fully supported by the Croatian Science Foundation under the project IP-2016-06-8345 “Automatic recognition of actions and activities in multimedia content from the sports domain” (RAASS) and by the University of Rijeka under the project number uniri-drustv-18-222.

References 1. Ji, R.: Research on basketball shooting action based on image feature extraction and machine learning. IEEE Access 8, 138743–138751 (2020) 2. Ramanathan, V., Huang, J., Abu-El-Haija, S., Gorban, A., Murphy, K., Fei-Fei, L.: Detecting events and key actors in multi-person videos. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas (2016) 3. Sanford, R., Gorji, S., Hafemann, L.G., Pourbabaee, B., Javan, M.: Group activity detection from trajectory and video data in soccer. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle (2020) 4. Gerats, B.: Individual action and group activity recognition in soccer videos. Faculty of EEMCS, University of Twente, Twente (2020) 5. Bonenkamp, K.: Action recognition in soccer videos. Informatics Institute, Faculty of Science, University of Amsterdam, Amsterdam (2014) 6. Piergiovanni, A., Ryoo, M.S.: Fine-grained activity recognition in baseball videos. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City (2018)

656

K. Host et al.

7. Rangasamy, K., As’ari, M., Rahmad, N., Ghazali, N.F.: Hockey activity recognition using pre-trained deep learning model. ICT Express 6(3), 170–174 (2020) 8. Sozykin, K., Protasov, S., Khan, A., Hussain, R., Lee, J.: Multi-label class-imbalanced action recognition in hockey videos via 3D convolutional neural networks. In: 2018 19th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), Busan (2018) 9. Haider, F., et al.: A super-bagging method for volleyball action recognition using wearable sensors. Multimodal Technol. Interact. 4(2), 33 (2020) 10. Ibrahim, M.S., Muralidharan, S., Deng, Z., Vahdat, A., Mori, G.: A hierarchical deep temporal model for group activity recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas (2016) 11. Bagautdinov, T., Alahi, A., Fleuret, F., Fua, P., Savarese, S.: Social scene understanding: endto-end multi-person action localization and collective activity recognition. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, (2017) 12. Zhu, G., Xu, C., Huang, Q., Gao, W., Xing, L.: Player action recognition in broadcast tennis video with applications to semantic analysis of sports game. Association for Computing Machinery, New York (2006) 13. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167 [cs] (2015) 14. Rahmad, N.A., As’ari, M.A.: The new Convolutional Neural Network (CNN) local feature extractor for automated badminton action recognition on vision based data. J. Phys. Conf. Ser. 1529(2), 022021 (2020) 15. Martin, P., Benois-Pineau, J., Péteri, R.: Fine-grained action detection and classification in table tennis with siamese spatio-temporal convolutional neural network. In: 2019 IEEE International Conference on Image Processing (ICIP), Taipei (2019) 16. Pareek, P., Thakkar, A.: A survey on video-based Human Action Recognition: recent updates, datasets, challenges, and applications. Artif. Intell. Rev. 54(3), 2259–2322 (2020). https://doi. org/10.1007/s10462-020-09904-8 17. Buri´c, M., Pobar, M., Ivaši´c-Kos, M.: Adapting YOLO network for ball and player detection. In: 8th International Conference on Pattern Recognition Applications and Methods (ICPRAM 2019), pp. 845–851 (2019) 18. Host, K., Ivasic-Kos, M., Pobar, M.: Tracking handball players with the DeepSORT algorithm, In: ICPRAM, pp. 593–599 (2020) 19. Buric, M., Ivasic-Kos, M., Pobar, M.: Player tracking in sports videos. In: 2019 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pp. 334–340 (2019) 20. Pobar, M., Ivasic-Kos, M.: Active player detection in handball scenes based on activity measures. Sensors 20(5), 1475 (2020) 21. Ivasic-Kos, M., Pobar, M., Gonzàlez, J.: Active player detection in handball videos using optical flow and STIPs based measures. In: 2019 13th International Conference on Signal Processing and Communication Systems (ICSPCS), pp. 1–8 (2019) 22. Ivasic-Kos, M., Pobar, M.: Building a labeled dataset for recognition of handball actions using mask R-CNN and STIPS. In: 7th IEEE European Workshop on Visual Information Processing (EUVIP), Tampere (2018) 23. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 2818–2826. IEEE (2016) 24. Deng, J., Dong, W., Socher, R., Li, K.L.L., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami (2009)

Application of Robotics to Domestic and Environmental Cleanup Tasks Amritha Ananthanarayanan1 , Chase G. Frazelle1 , Sowmya Kethireddy1 , Chen-Ho Ko2 , Rohan Kumar1 , Vignesh Prabhu3 , Bhargav Srinivas Vasudevan1 , and Ian D. Walker1(B) 1 Department of Electrical and Computer Engineering, Clemson University, Clemson,

SC 29634, USA [email protected] 2 Planning, Design and the Built Environment, Clemson University, Clemson, SC 29634, USA 3 Department of Mechanical Engineering, Clemson University, Clemson, SC 29634, USA

Abstract. Robots are often used to perform repetitive, dangerous, and undesirable tasks for humans. In this paper, we discuss the current potential of robots for assisting humans with a variety of mundane but important tasks. Robotics technology has reached the point where relatively inexpensive robotic solutions can be created for many commonly encountered effort-intensive tasks humans perform across the globe. Specifically, we discuss the application of robotics to supporting the handling of laundry, as well as conducting cleanup activities, both indoors and outdoors. The discussion is supported and detailed by case studies. The robot prototypes proposed in this research can be used as a reference for the development and application of domestic and environmental cleanup robots. Keywords: Robotics · Assisted living · Environmental cleanup

1 Introduction Robotics technology has advanced significantly during the past 20 years [8]. However, with the notable exception of the Roomba home vacuum cleaner robot, and, more recently, some unmanned aerial vehicles (i.e. drones), few robots have made their way into the everyday lives of ordinary people. This is in part because traditional robotic technologies have been expensive and required specialists to operate. However, in the past several years a variety of new, cheaper and easier to use robotics technologies have emerged [7]. In this paper, we discuss and explore, using several case studies of important tasks currently performed by humans around the globe, the potential of the new robotic capabilities enabled to assist people in their daily domestic activities, and in important clean-up tasks. These case studies include the adaptation of current robotic solutions to new tasks as well as exploring the design of new platforms to address solution needs. This research was supported in part by the U.S. National Science Foundation under grant IIS1718075. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 657–665, 2022. https://doi.org/10.1007/978-3-030-80119-9_42

658

A. Ananthanarayanan et al.

The paper is organized as follows. The following section discusses the application of robotics to laundry handling, both pre- and post-washing. Section 3 discusses robotic solutions for indoor clean-up, specifically focused on sorting and cleaning up toys. Sections 4 and 5 introduce simple but effective robotic solutions for environmental clean-up in outdoor environments (for gutter cleaning in Sect. 4, and for more general environmental clean-up scenarios in Sect. 5). Conclusions are presented in Sect. 6.

2 Inside the Home: Case Study - Laundry Consumer robots have been part of popular culture for decades, fueling visions of having robots living alongside humans in our homes to assist us with daily tasks, entertain, educate, and socialize [2]. Disabled and elderly people are often unable to handle the work of laundry by themselves [6]. Robots provide possible solutions to assist them in doing laundry [3]. Previous studies in laundry automation focus on the improvement of washing machine control, ignoring that the major inconvenience for people is the process of putting clothes in the washing machine and taking them out of it [5]. In this section, we explore replacing two steps of the laundry handling process with previously developed robotic solutions: one created previously in our lab, and one readily found in industry. 2.1 Clothes to the Washer First, we consider a scenario in which a robot assists with laundry by picking up discarded clothing and transporting the clothing to the washer. The robot depicted in Fig. 1, which we term Lamp+, is part of a suite of robots we are prototyping for the home [1, 4]. Lamp+ consists of a mobile base combined with a long continuum arm that serves as a light source as well as a manipulator through a gripper located at the arm’s end-effector. Thus, Lamp+ is well suited for the task of retrieval and transport of laundry given its mobility and extended reach.

Fig. 1. Lamp+ continuum arm demonstrates reach and manipulation while transporting clothing.

A Raspberry Pi single-board computer, mounted along with a camera module, helps in detection of clothes on a surface after portioning a table into three parts. Computer

Application of Robotics to Domestic and Environmental Cleanup Tasks

659

vision was implemented by using the Raspberry Pi and camera module. The camera takes a snapshot of the clothes on the table, partitioning the surface into a discrete grid with known centroid locations. A template matching algorithm is used to determine whether the pre-defined discrete positions on the table contain clothes or not. When the robot arm reaches its specified spot above the table, the Raspberry Pi is triggered. Two programs are executed by the Raspberry Pi. The first takes a snapshot and converts the image to grayscale (for ease of image processing). The second program takes the snapshot as input and checks the pre-defined discrete slots for clothes by matching it with available templates. The final output consisted of whether each slot contained clothes or not, as demonstrated in Fig. 2. The Lamp+ then proceeds to either pick up the clothes in a given slot (Fig. 3) or skip the slot.

Fig. 2. Computer vision detects presence of clothing on discretized surface.

Servomotors help in picking and placing the clothes as shown in Fig. 3 and Fig. 4, respectively. Lithium Polymer (LiPo) batteries power the Lamp+ and the motors help running the robot. An Arduino Due and a Raspberry Pi act as the brain of the system.

Fig. 3. Lamp+ grabs cloth from table surface

660

A. Ananthanarayanan et al.

Fig. 4. Placement of laundry by Lamp+

2.2 Clothes from the Washer Getting wet clothes out of the washer can be challenging to automate due to the constrained environment and need for precise movement. In an effort to support this, we propose a design which uses an industry grade robotic arm that can reach into the washer and pick out the clothes. The primary challenge here is to identify if there is still clothing in the washer while unloading and where the remaining articles of clothing are located.

Fig. 5. Removing wet clothes from washer.

The proposed solution makes use of an Arduino Uno, a Kinova MICO (6 Degrees-ofFreedom) robotic arm and a PixyCam. Once the user sends in a text over Bluetooth, the Arduino sends a signal over a serial connection to initiate the laundry retrieval process. The PixyCam interacts with the arm over a serial connection by indicating the presence and location of laundry in the washer. The PixyCam camera was chosen for its ability to identify and track various objects based on their color. We affix the PixyCam to a stand and focus the camera on the inside of the washer. Coordinates for clothing positions are estimated relative to a fixed origin position for the robot. The robot arm moves to each coordinate and picks out each item of clothing one at a time, depositing the retrieved clothing in a basket outside the washer, as seen in Fig. 5.

Application of Robotics to Domestic and Environmental Cleanup Tasks

661

3 Inside the Home: Case Study – Cleaning Up Toys Sorting and organizing may not always be the most difficult tasks for humans, but they can certainly be tedious. These tasks may seem like prime examples of things that a robot could do, but it is non-trivial to teach a robot to perform tasks like picking up, recognizing, and sorting objects. Although picking and placing is a common application in industrial robots [5], their application to toy sorting has rarely been discussed. The toy sorter robotic solution proposed herein sorts toys automatically if there is no human movement in the play area for more than five minutes. The toy sorter communicates with parents through a smartphone via a Bluetooth module and sends a message requesting permission to activate the sorting mode. Once the “toy-sorting function” is triggered, the system recognizes toys using computer vision, again realized with the help of a PixyCam. Location coordinates of toys are calculated using a personal computer. As shown in Fig. 6, the toy sorter then picks up the toys one by one according to the dynamic status. It sends messages to the parent once the toys have been sorted as in Fig. 7.

Fig. 6. Left: Toy sorter recognizes images using computer vision. Right: Robot arm picks up the toys and places them into preassigned baskets.

Fig. 7. Toy sorter message screen for parents.

The toy sorter features a motion sensor used to detect human movements around the toy-playing area. The PixyCam is used to identify the location of the toys and enables the picking of objects according to their colors and shapes, and then calculating their coordinates relative to the robotic arm. The MICO robot grips each toy recognized by

662

A. Ananthanarayanan et al.

the PixyCam and drops it into a pre-designated bin according to the sorting method designated by the parent (such as sort by color or shape of toys).

4 Outside the Home: Case Study – Gutter Cleaning Every fall, across the globe, trees start shedding their leaves in preparation for winter. This mass of fallen debris combines with other debris, such as small twigs and dirt, and often collects in gutter systems. This can clog up the gutter and put undue pressure on it, which in turn can make the structure collapse or cause the overflow of drain water. It is a taxing and cumbersome process to clean gutters, and not without the risk of falling related injuries. The primary challenge in designing a robot to automate this task is the space constraint, given that gutters are quite narrow. The mobile robot proposed herein is designed to determine when the gutter needs clearing using computer vision and then, using a continuous rotation servo motor, clear the gutter of any debris. An attachment at the end of the robot pushes the debris towards an outlet. The prototype uses a Raspberry Pi camera, Raspberry Pi 3 single-board computer and an Arduino Uno. The design consists of multiple stacked layers, a gear arrangement, and one wheel that is located at the rear of the robot. These components are 3D printed. The design also has a caster wheel in the front to support the robot. The servo motor powers the back wheel and consequently the entire robot (Fig. 8).

Fig. 8. Gutter cleaning robot design

The Raspberry Pi camera takes a picture of the gutter line when it is clear of debris and saves this as a reference image. A second picture is taken at frequent intervals and is compared to the reference image. When debris is detected, the Raspberry Pi publishes a message over a ROS (Robot Operating System) terminal. A subscriber, which is the Arduino in this case, receives the message and actuates the servo motor to drive the robot forward and remove the debris. The robot in operation, cleaning leaves, is shown in Fig. 9.

Application of Robotics to Domestic and Environmental Cleanup Tasks

663

Fig. 9. Gutter cleaning robot operation.

5 In the Wider World: Case Study – Environmental Cleanup Every single year, there are about 8 million tons of plastic that end up in the oceans [9]. A part of this problem is due to plastics discarded at shores and beaches. To tackle this, and to address litter that washes up on shore, there are numerous volunteer beach cleanups carried out at various places. Here, we propose a robotic solution that extends help to such volunteers in ‘pick and place’ operations at beaches. However, this is not a simple task as the robot has to pick up trash of various sizes and place the trash in a trash box/bag while picking up minimal amounts of sand.

Fig. 10. Environmental clean-up robot.

To solve this, our solution features an inexpensive mobile robot. The four-wheeled robot uses a blade-like sieve upfront to help pick trash (Fig. 10). The robot is controlled via a mobile phone application and provides a first-person view using the phone’s built-in camera. The rear view from the robot, and subsequently the view for the robot’s operator, is given in Fig. 11.

664

A. Ananthanarayanan et al.

Fig. 11. Rear view of the beach cleaning robot

The mobile phone was augmented with a tripod stand to raise the camera’s view over the trash storage bin. The wheel support and the blade were 3D printed to give adaptive motion on sand and other terrains, and better support for the sieve structure, respectively. An Arduino Uno was instrumental in controlling the robot by receiving commands for motor movement via a Bluetooth communication module. 9 V batteries are used as the power supply. The mobile application was developed using MIT App Inventor and transmits commands to the Arduino Uno through a Bluetooth connection between the on-board mobile phone and Arduino. According to the video-feed received by the operator’s mobile phone, the operator sends inputs to the robot using the mobile application. Once the robot encounters trash, the command over the mobile application directs the blade to pick and place trash into the on-board storage bin. The blade movement is controlled using a servo motor. The placement of the gear and the blade is placed at an angle such that it buries the sieve below the sand. The picked trash is then deposited in the trash box and the operation repeats until the trash box is full.

6 Conclusions We have presented a series of relatively simple and inexpensive robotic solutions for common tasks. Future considerations focus on path planning for the robots or using different control strategies to efficiently control robot mobility. Variants of the toy sorter system could be further used for tasks like warehouse sorting or cleaning up a disaster area by including various modules like GSM or GPRS. The environmental cleanup robot concept could be extended and implemented to pick trash, pick and place objects at domestic households, and also at factories.

References 1. De Aguiar, C., et al.: The networked, robotic home+ furniture suite: a distributed, assistive technology facilitating aging in place. In: Proceedings of the 12th IEEE International Conference on Automation Science and Engineering (CASE), Fort Worth, TX, pp. 1067–1072 (2016)

Application of Robotics to Domestic and Environmental Cleanup Tasks

665

2. Forlizzi, J., DiSalvo, C.: Service robots in the domestic environment: a study of the roomba vacuum in the home. In: Proceedings of HRI 2006, 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction, Salt Lake City, USA, pp. 258–265 (2006) 3. Hata, S., Hiroyasu, T., Hayashi, J.I., Hojoh, H., Hamada, T.: Robot system for cloth handling. In: 2008 34th Annual Conference of IEEE Industrial Electronics, pp. 3449–3454 (2008) 4. Hawks, Z., Islam, M.S., Lastinger, M., Rutland, H., Trout, M., Walker, I.D., Green, K.E.: Robots in the home: a novel suite of interactive devices for assisting with disease prevention and meal preparation. In: Proceedings of the IEEE Global Humanitarian Technology Conference, Seattle, WA, pp. 715–718 (2019) 5. Nejad, M.G., Shavarani, S.M., Güden, H., Barenji, R.V.: Process sequencing for a pick-andplace robot in a real-life flexible robotic cell. Int. J. Adv. Manufact. Technol. 103(9–12), 3613–3627 (2019). https://doi.org/10.1007/s00170-019-03739-6 6. Smarr, C.A., Prakash, A., Beer, J.M., Mitzner, T.L., Kemp, C.C., Rogers, W.A.: Older adults’ preferences for and acceptance of robot assistance for everyday living tasks. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 56, No. 1, pp. 153–157. SAGE Publications, Los Angeles (2012) 7. Verma, S., et al.: Design and evaluation of two robotic furnishings partnering with each other and their users to enable independent living. In: Proceedings of 12th EAI International Conference on Pervasive Computing Technologies for Healthcare, pp. 35–44. ACM, New York (2018) 8. Walker, I.D., Green, K.E.: Architectural robotics: unpacking the humanoid. In: Proceedings of 11th International Conference on Ubiquitous Computing (UBICOMP’09), Orlando, FL, pp. 28–29 (2009) 9. Parker, L.: The World’s Plastic Pollution Crisis Explained. Plastic Pollution Facts and Information, National Geographic, 7 June 2019. www.nationalgeographic.com/environment/habitats/ plastic-pollution. Accessed 27 Aug 2020

RF Mapping for Sensor Nodes Connectivity and Communication Signal Recovery Mustafa Ayad1(B) and Richard Voyles2 1

State University of New York, Oswego, NY 13126, USA [email protected] 2 Purdue University, West Lafayette, IN 24907, USA [email protected]

https://www.oswego.edu/electrical-computer-engineering/content/mustafa-ayad

Abstract. The network connectivity is vital for operative and proficient mobile robot team’s maneuvers. In search and rescue scenarios, many obstacles are preventing effective wireless communication. A promising solution to the problem is combining the RF mapping method and the gradient method. The RF mapping endeavors to measure the effects of RF obstacles in the physical environment that may attenuate signals as a transmitter/receiver pair moves through the field. By mapping these attenuation patterns, nodes can avoid known “RF shadows.” The gradient method, based on extracting the components of the multidimensional gradient as the robots move around a source of RF interference to find the locations of strong signal strength (direct line of sight) and weakest signal strength (dead zones). This paper presents a connectivity approach based on the radio frequency (RF) mapping and the robot team’s gradient estimation methods. The physical and simulation results of the experiments promise to achieve reliable network connectivity for mobile nodes. Keywords: Connectivity maintenance recognition

1

· Gradient · RF mapping

Introduction

Most wireless communication networks operate in adverse environments with various RF interference sources, resulting in signal attenuation, loss of line of sight (LOS), and interferences [5]. A typical example of such adverse environments is a collapsed building, where a team of small robots cooperates to achieve assigned tasks [4, 6]. Figure 1 demonstrates the scenario of a collapsed structure and a swarm of urban search and rescue (USAR) robots communicate with each other to transmit data back to the destination [2, 3, 12]. A significant problem for the robot’s network is preserving a communication path for message delivery among robots. The robots perform multiple tasks based on their capabilities, c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 666–684, 2022. https://doi.org/10.1007/978-3-030-80119-9_43

RF Mapping for Signal Recovery

667

Fig. 1. Robotic Network in a Collapsed Structure.

including searching for survivors, maintaining communication networks in collapsed structure scenarios, and transmitting data from all peripheral areas of the network to the network base station (BS). Robot teams provide potential solutions to surveillance, monitoring, search and rescue operations, and eliminate the need for human intervention in dangerous areas [1,8]. However, robots have limited mobility, power, communication range, and computation capabilities [7]. Therefore, to accomplish the mission, system resources should be distributed through multiple robots in the network. Reliable wireless network communication strategies are essential for these environments where a robot can quickly lose contact with the rest of the team [9]. Therefore, the variation in the field’s signal strength measurements can be employed to guide mobile robots to ensure their network connectivity. In this work, the RF mapping recognition approach is based on signal strength measurements to recognize hostile environments enclosing RF obstacles. The approach motivates us to explore the correlation between known RF obstacle types and their impacts on RF signal strength by addressing various scenarios’. Based on RF mapping recognition, a gradient algorithm to control directions of movements is presented. Current gradient approaches achieve two-dimensional (2-D) gradient concerning a fixed source of a signal while not

668

M. Ayad and R. Voyles

addressing mobile nodes gradient estimation, which requires assessments of gradient in four-dimensional (4-D) space, we combine the RF mapping recognition, and gradient algorithm methods are augmented to maintain connectivity among mobile robots in RF environment [10]. Several simulation experiments are conducted in this work. Two robots are moved around known obstacles of different sizes (wall, cylinder, and cage made of a perfect conductor ) and measure RF signal strength. The goal is to examine the effect of the conducting obstacle on the RF signal measurements while the robots move around them. Robots will move on known trajectories around the obstacle and measure the signal strength. The signal strength measurements along the robot trajectory hold adequate information, allowing classification of the obstacle type.

2

RF Mapping Versus Physical Mapping

35

N

Y

RF Mapping is different from physical mapping. Physical mapping clearly shows where obstacles in the mapped environment are located, while RF mapping shows the effects those RF obstacles have on RF signals. In RF mapping, the sensor measurements require two agents (transmitter and receiver) to transfer and receive the RF signal. These sensor measurements are not uniquely or directly tied to physical objects, but they are affected by multi-path, fading or interference. The configuration space of the RF obstacle consists of (x1 , y1 , x2 , y2 ) coordinates of a 2 Degree of Freedom (DoF) transmitter and a 2-DoF receiver if an Omni-directional antenna is used. Thus, with this scenario a four dimensional configuration space is formed, which makes RF mapping more difficult compared to that of physical mapping. In our work, we simplified the 4-DoF configuration space to a 2-DoF configuration space for visualization. We assumed constant sized types of the RF obstacles, a fixed distance between the transmitter and receiver pair as well as fixed orientation, see Fig. 2.

30

..... .

25

20 d

3

15

2

10

y

1

x 5

0

0

5 1

10 2

15 3

20

25 30 ,,, , , , , , , , ,

(a)

35

40 M

45 X

(b)

Fig. 2. (a) The 2-D space for the simulation experiments, and (b) an application scenario where two robots moving around a wall obstacle.

RF Mapping for Signal Recovery

2.1

669

Modeling of RF Mapping Recognition Method

In our application scenario, two mobile sensor nodes transmit and receive RF signals at 2.4 GHz. The receiver will measure the RF signal strength at its current position. The signal measurements might be affected by different multi-path interference and fading [13]. The experimental 2-D space for the RF mapping method is arranged into equal spaced grids. Δx = Lx /M denotes grid width in x-direction and Δy = Ly /N in y-direction. The length and width of the space are Lx and Ly , and M and N are the number of segments in x-direction and y-direction as in Fig. 2 (a). The width of a grid could be one third, two thirds or one wavelength in length, where the wavelength is 12.5 cm at 2.4 GHz[10]. The sensor nodes are located at (xk , yk ) 2-D Cartesian coordinates at time k as in Fig. 2 (b). The robots will move along the lth trajectories to collect signal strength in the experiments. The lth path is given by (i)

(i)

(i)

(i)

xk,l = x0,l , yk,l = y0,l + kΔy , k = 1, 2, · · · , N,

(1) (i)

(i)

where l is the trajectory index, i ∈ {1, 2} is the robot index, (x0,l , y0,l ) represents the ith robot initial location at time 0. Equation (1) defines the motion (i) (i) of ith robot which starts with (x0,l , y0,l ) and then moves along y-direction with (1)

1 = 0 are a step size of Δy for each time step [10, 11]. Moreover, x0,l = lΔx , y0,l (2)

(1)

assumed for the 1st robot for the lth trajectory. for the x0,l = x0,l + d and (2)

(1)

y0,l = y0,l are assumed for the 2nd robot. Therefore, a fixed distance d will separate the two robots in x-direction while both robots have same coordinate in y-direction. The scenario of two robots is verified in Fig. 2 (b). The received RF signal strength at the receiver location, at time k for the lth trajectory in the presence of obstacle type j, is denoted as (j)

(1)

(1)

(1)

(1)

(2)

(2)

(2)

(2)

Sl (k) = f (x0,l , y0,l , xk,l , yk,l , x0,l , y0,l , xk,l , yk,l , φj ), (j)

(2) (i)

(i)

where Sl (k) represents a function of the initial robot locations (x0,l , y0,l ), (i)

(i)

robots locations (xk , yk ) at time k and the characteristics of the obstacle φj . The trajectory index l = 1, ..., L(j) for each j, where L(j) is the trajectories number around an obstacle of type j. In (2), j ∈ {1, 2, 3} denotes type of the (j) (j) obstacle and φj = {(xc , yc ), θ (j) } represents a set of obstacle characteristic, which contains the shape parameters of the obstacle θ (j) and the the central (j) (j) position of the obstacle (xc , yc ). For example, the wall obstacle has central L coordination ( L2x , 2y ), and the shape parameters θ (j) , which includes width, length and height information [10,11]. The signal strength measurements in the experiment field with the presence of three RF obstacle types are demonstrated in the following sections. 2.2

RF Signal Strength Measurements

The signal strength measurements in the experiment field with the presence of three RF obstacle types are demonstrated. In addition, we presented the

670

M. Ayad and R. Voyles

measurement sequence obtained from different trajectories. By investigating how the signal strength changes at different locations, it is possible to recognize and classify the RF signatures of certain RF obstacle types. For the simulation experiments, we used Computer Simulation Technology (CST), which is a professional tool for the 3-D Electro Magnetic simulation of high frequency components [14]. CST microwave studio enables quick and accurate analysis of high frequency devices such as antennas, filters, planar and multi-layer structures [14]. In our simulation, we used a 60 mm × 60 mm patch antenna to send and receive the communication signal and made the interference source out of pure electric conducting material as shown in Fig. 2(b).

20 10

Wall

0

−80

−10 −60

−20

Dead zones

−30

−40

−40 −50 −60 −70 50

−20

Y cm

Signal Strength (dBm)

30

0

Signal Spikes

20 40

0 −50

10 20

−10 0

Y cm

60 −15

X cm

−10

0

5 X cm

10

15

20

25

(b)

Signal strength (dbm)

(a)

−5

-40

-40

-40

-45

-45

-45

-50

-50

-50

-55

-55

-55

-60

-60

-65

-70 0

-60

-65

-65

2

4

6

8

10

12

14

16

18

20

22

-70 0

2

4

6

8

10

12

14

16

18

20

22

-70 0

-40

-40

-40

-45

-45

-45

-50

-50

-50

-55

-55

-55

-60

-60

-65

-65

-70 0

2

4

6

8

10

12

14

16

18

20

22

-70 0

2

4

6

8

10

12

14

16

18

20

22

-60

-65

2

4

6

8

10

12

14

16

18

20

22

-70 0

2

4

6

8

10

12

14

16

18

20

22

Time

(c)

Fig. 3. (a) RF signal strength measurements for a wall obstacle (b) top down view of Fig. 3(a), and (c) RF signal strength measurement sequences corresponding to different trajectories for the wall obstacle.

Wall Obstacle. In wall experiments, we use three different wall sizes, which are 7 cm × 30 cm × 30 cm, 10 cm × 30 cm × 30 cm and 15 cm × 30 cm × 30 cm. The RF signal strength measurements of the experiment field with a wall obstacle of the size 10 cm × 30 cm × 30 cm are shown in Fig. 3(a). The signal strength becomes extremely low when the transmitter is extremely close to the edge of the RF obstacle at one side, and the receiver is located one meter away from the other side, or vice versa. As the transmitter or receiver moves

Signal strength (dbm)

RF Mapping for Signal Recovery

671

RF deadzones

130

120

110

Y cm

100

90

80

70

60

50

20

40

60

(a)

80 X cm

100

120

(b) -60

-65

-65

-70

-70

-75

-75

-80

-80

-85

-85

-90

-90

-95

-95

-100

-100

-105 0

-105 0

Signal strength (dbm)

-60

5

10

15

20

25

-60 -65 -70 -75 -80 -85 -90 -95 -100

5

10

15

20

25

-105 0

-60

-60

-65

-65

-65

-70

-70

-70

5

10

15

20

25

-75

-75

-75

-80

-80

-85

-85

-90

-90

-80

-85

-90 -95

-95

-100

-100

-105 0

-95

5

10

15

20

25

-105 0

5

10

15

20

25

-100 0

5

10

15

20

25

Time

(c)

Fig. 4. (a) RF signal strength measurements for a cage obstacle (b) top down view of Fig. 4(a), and (c) RF signal strength measurement squences corresponding to different trajectories for the cage obstacle.

away from the RF obstacle, the signal strength becomes significantly stronger as shown in Fig. 3(a). Figure 3(b) shows the top-down view of Fig. 3(a) where the dark red spots represent the four negative spikes of Fig. 3(a). Figure 3(c) shows different signal shapes at different distances from the RF obstacle at the receiver position. The signal shapes reflects the RF obstacle effects on the RF communication signal between the robots when they move around the RF obstacle. Cage Obstacle. The cage obstacle shown in Fig. 4(a) is a Faraday cage, which is made of pure electric conducting material. The cage size is 30 cm × 30 cm × 30 cm. The signal strength decreases and becomes extremely weak when one of the antennas is located inside the cage as shown in Fig. 4(a). The signal strength oscillates when the transmitter or the receiver approaches the entrance of the cage due to conducting material effects. Figure 4(a) shows that the signal strength becomes weak because the cage prevents the line of sight between the transmitter and receiver. The signal strength intensity image of Fig. 4(a) is shown in Fig. 4(b), which shows that the signal strength becomes stronger as the antennas move away from the cage and there is a line of sight.

672

M. Ayad and R. Voyles

Figure 4(a)–(c), demonstrate the effect of the cage obstacle on RF signal strength measurements. Figure 4(c) shows the measurement sequence obtained from different trajectories at different distances from the RF obstacle.

Signal strength (dbm)

Cylinder Obstacle. In cylinder experiments, we use three different cylinder radiuses, which are r = 10 cm, r = 15 cm and r = 20 cm; the height of the cylinders is 30 cm. Figure 5(a) shows the simulation results of signal strength measurements in the experimental field with a cylinder obstacle. The cylinder obstacle is located at the center of the experiment field, with a diameter of 15 cm. As we can see in Fig. 5(a), the signal strength becomes weak as the receiver moves closer to the cylinder. The signal strength intensity image of Fig. 5(a) is shown in Fig. 5(b). Figure 5(a)–(c) show the effect of the cylinder obstacle on the RF signal strength measurements. As the antennas move along a straight line at each side of the RF obstacle, the signal strength measurement sequence shows that it holds sufficient information to allow recognition of the RF obstacle type. The RF obstacle shadow on the RF signal strength measurements produces different signal shapes at different distances as the receiver antenna moves closer to the RF obstacle.

-50 -60 -70 100 80 60 40 20 0

Y cm

-20 -40

20 -60

0 -80

X cm

-20

(a)

(b) -40

-40

-40

-45

-45

-45

-50

-50

-50

-55

-55

-60

-60

-55

Signal strength (dbm)

-60

-65

-70

-25

-65

-65

-70

-70 -20

-15

-10

-5

0

5

10

15

20

-25

-20

-15

-10

-5

0

5

10

15

20

-25

-20

-15

-10

-5

0

5

10

15

20

-40

-45

-45

-45

-50

-50

-50

-55

-55

-55

-60

-60

-60

-65

-65

-65

-70

-70

-70 -75 -25

-75 -20

-15

-10

-5

0

5

10

15

20

-25

-20

-15

-10

-5

0

5

10

15

20

-25

-20

-15

-10

-5

0

5

10

15

20

Time

(c)

Fig. 5. (a) RF signal strength measurements for a cylinder obstacle (b) top down view of Fig. 5(a), and (c) RF signal strength measurement sequences corresponding to different trajectories for the cylinderl obstacle.

RF Mapping for Signal Recovery

3

673

Physical Obstacles Experiments

To conduct regular physical experiments, we started with modeling simple conducting RF obstacles such as walls, cylinders, and cages. We created an nearly clean physical environment to minimize unknown sources of interference. Two Texas Instruments CC2510 development kits were used (one transmitter, one receiver), and the antennas were moved manually around a copper obstacle that laid on top a cardboard box in a laboratory environment. A 2.4 GHz transmitter and receiver antennas were moved around the physical RF obstacles in the experiment field. We recorded signal strength at different antenna positions as they moved around a single source of interference. The source of interference itself had been centered in the middle of the experiment field, and measurements were taken at up to 100 cm on either side, as well as above and below the source of the interference. We constructed different RF obstacle shapes that were similar to the RF obstacles used for the simulation. The physical results were based on the environment, especially the ground floor where we put the antennas and then moved them. The carpet material has different effects than wood or concrete. After many trials, we found that a cardboard box of 15 cm height is more convenient and provide satisfactory results compared to the simulation results.

Receiver

Copper Wall

Transmitter Fig. 6. Copper wall obstacle and two physical devices.

3.1

Copper Wall Obstacle

We created a wooden box frame (10 cm × 30 cm × 30 cm) and covered it with a copper screen as shown in Fig. 6. The physical results of moving the transmitter and the receiver around the wall are shown in Fig. 7(a). The signal strength oscillates between high and low and becomes extremely low (the spikes in Fig. 7(a)) when either the transmitter or the receiver is extremely close to the RF obstacle from one side, and the other Transmitting antenna is 1 meter away

674

M. Ayad and R. Voyles

0 Wall −20

Wall

−40

−10 X cm

Signal strength dBm

20

−60

0 10

20 0

20

−80

−20 Spikes

−60

−40

−20

0

20

Y cm

−40 −60

10

0 −10

−80

Y cm

20

Spikes

X cm

(a)

(b)

-63

-62

-60

-64

-62

-66

-64

-64

Signal Strength dBm

Signal sterngth dBm

-66 -67 -68 -69 -70

Signal Strength dBm

-65

-68

-70

-72

-66

-68

-70

-71 -74

-72

-72 -73 -20

-15

-10

-5

0

5

10

15

20

-76 -20

25

-15

-10

-5

0

5

10

15

20

-74 -20

25

-60

-64

-62

-62

-66

-64

-64

-68

-70

-66

-68

-72

-70

-74

-72

-10

-5

0

5

10

15

20

-74 -20

25

Signal Strength dBm

-60

-15

-10

-5

0

-62

-62

15

20

25

5

10

15

20

25

5

10

15

20

25

-66

-68

-72

-10

-5

0

5

10

15

20

-74 -20

25

-15

-10

-5

0

Time -60

10

-70

-15

Time -60

5 Time

-62

-76 -20

-15

Time

Signal Strength dBm

Signal Strength dBm

Time

Time -58 -60 -62

-66

-68

Signal Strength dBm

-64

Signal Strength dBm

Signal Strength dBm

-64

-66

-68

-70

-70

-72

-72

-64 -66 -68 -70 -72

-74 -20

-15

-10

-5

0

5 Time

10

15

20

25

-74 -20

-74

-15

-10

-5

0

5 Time

10

15

20

25

-76 -20

-15

-10

-5

0 Time

(c)

Fig. 7. (a) Copper wall obstacle physical results (b) top down index of copper wall obstacle of Fig. 7(a), and (c) different signal shapes of copper wall obstacle.

on the other side. As the transmitter moves around the wall between the edges, the signal strength is still low but better than that when the transmitter or the receiver are at the edges, which matches the simulation results shown in Fig. 3(a). Figure 7(b) shows the index image of Fig. 7(a) where the spikes show the lowest signal strength. The spikes on the figure can help to estimate the thickness and the width of the wall. Physical and simulation results show that it is possible to estimate the size and the shape of RF obstacle. The signal strength becomes stable when the transmitter and the receiver move far from the RF obstacle, and there is a line of sight. Figure 7(a)–(c) determine the effectiveness of the copper wall obstacle on RF signals. As the transmitter and receiver move along a straight line at each side of the RF obstacle, the signal strength measurement sequences are shown to contain sufficient information to enable the recognition and classification of the RF obstacle type. Furthermore, the RF obstacle shadow on the RF signal strength measurements produces different signal shapes as the antenna moves

RF Mapping for Signal Recovery

675

toward the copper wall obstacle as a result of RF obstacle effects as shown in Fig. 7(c).

20

Cage

0 Cage 120

−40

Spikes

−60

110

−80

100 Y cm

Signal Stregngth dBm

130

−20

−100 Signal strength inside the cage

−120 −140 Spikes 100 Y cm

80 70 60

100

50

50

50

90

X cm

50

60

70

80

(a) -75

-70

-75

-80

-75

-80

-85

-90 -95

-90 -95 -100

-100 -105 -110

60

70

80 90 100 Distance cm

110

120

130

140

-95

-115 40

50

60

70

80 90 100 Distance cm

110

120

130

-115 40

140

-75

-75

-80

-80

-80

-85

-90 -95 -100 -105

-90 -95 -100 -105

60

70

80 90 100 Distance cm

110

120

130

-120 40

140

-75

-80

-80

80 90 100 Distance cm

110

120

130

140

50

60

70

80 90 100 Distance cm

110

120

130

140

50

60

70

80 90 100 Distance cm

110

120

130

140

-90 -95 -100

50

60

70

80 90 100 Distance cm

110

120

130

140

-115 40 -75 -80

-85

-85

-85

Signal strength dBm

-95

-105

Signal strength dBm

-90

-90

-100

-95 -100 -105

-90 -95 -100

-110 -105

-110

-115

-115 -120 40

70

-110

-115

-75

60

-105

-110

-110

50

-85

Signal strength dBm

Signal strength dBm

Signal strength dBm

-90

-75

-85

Signal strength dBm

-85

-100

-110

-110

50

130

-105

-105

-115 40

120

-80

Signal strength dBm

Signal strength dBm

Signal strength dBm

-85

50

110

(b)

-70

-115 40

90 100 X cm

-110

-120

50

60

70

80 90 100 Distance cm

110

120

130

140

-125 40

50

60

70

80 90 100 Distance cm

110

120

130

140

-115 40

(c)

Fig. 8. (a) Copper cage obstacle physical results (b) top down index of copper cage obstacle of Fig. 8(a), and (c) different signal shapes of copper cage obstacle.

3.2

Copper Cage Obstacle

In this experiment, we created a four sided wood frame cage (30 cm × 30 cm × 30 cm) and then covered it with a copper screen. The cage was centered on a large cardboard box. The transmitter and receiver moved around the cage at a regular 1 meter distance apart. The signal strength was measured for different positions outside and inside the cage. The signal strength decayed and becomes extremely low when the transmitter or the receiver was located inside the cage as shown in Fig. 8(a). The signal strength oscillated when the transmitter or the receiver approached the entrance of the copper cage as a result of the conducting material effects. We got closer results from the simulation when an antenna was at the entrance of the cage as shown in Fig. 8(a). The left side of Fig. 8(a) shows

676

M. Ayad and R. Voyles

how the signal strength goes down as the copper cage prevents line of sight between the transmitter and receiver. The index image of Fig. 8(a) is shown in Fig. 8(b). The signal strength significantly increased as the transmitter and the receiver moved away from the cage and preserved a line of sight, which is consistent with the simulation results as shown in Fig. 4(a). The signal strength became much better at the rear side (see Fig. 8(b)) as the antennas move away from the cage, which is right for the simulation result of Fig. 4(a), but with a difference of −5 dB better than the physical result, as a result of electromagnetic effects that came from the surrounding physical environments. The RF obstacle shadow on the RF signal strength measurements produces different signal shapes as the receiver antenna moves closer to the RF obstacle as shown in Fig. 8(c).

4

Simulation and Physical Results Validation

In this section, we present physical results and CST simulation results validation for cage and wall obstacles. The physical and simulation results are plotted together to compare their accuracy. The physical results are effected by the sources of interference in the experiment environment.

-40

-40

-40 85cm from wall

88cm from wall

CST

-45

-45

-50

CST

-55

CMK205

-45

-50

CST

-55

CMK205

80cm from wall

CST

-50

-55

CMK205

-60

Physical

-60

-60 -65

-65

-70

-70

-75 -20

lab21

-15

-10

-5

0

5

10

15

20

25

-40

Signal strength dBm

-45

70cm from wall

CST

-15

-10

-5

0

5

10

15

20

25

-45

CMK205

-60

60cm from wall

CST

Lab21

-45

CMK205

Lab21

-10

-5

0

5

10

15

20

25

-40

-75 -20

CST

-45

-10

-5

0

5

10

15

20

25

CMK205

-60

CST

Lab21

-75 -20

-55

CMK205

Lab21

-70

-75 -20

5

10

15

20

25

25

CST

CMK205

Lab21

-15

-10

-5

0

5

10

15

10cm from wall

20

25

CST

CMK205

-65

-75 -20

0

40cm from wall

-55

-70

-5

20

-60

-65

-75 -20

-10

15

-50

-70

-15

10

-55

-45

-60

-65

5

-40

20cm from wall

-50

-55

0

-70

-15

-40

30cm from wall

-50

-5

-65

-70

-15

-10

-60

-65

-70

-15

-50

-55

-60

-65

-75 -20

-40

-50

-55

-45

-80 -20

Lab21

-70

-40

-50

-75 -20

-65

lab21

-75

-15

-10

-5

0

5

10

15

20

25

Lab21

-15

-10

-5

0

5

10

15

20

25

Time Fig. 9. Wall obstacle physical and simulation results comparison.

4.1

Wall Results Validation

We carried out different copper wall physical experiments to verify the CST simulation. We moved antennas about the copper wall in an area 2 m2 , in two

RF Mapping for Signal Recovery

677

different places, as shown in Fig. 6. We chose two places in different buildings to perform the experiments. We sought environments with fewer sources of interference to guarantee reliable communication signal between the transmitter and the receiver. Figure 9 shows different wall signal shapes for CST simulation and physical experiments. Signals that are black represent simulation results, whereas signals that are red and blue are physical results. Blue physical signals are close to the CST signals (black) and signal strength difference is between −2 dBm up to −8 dBm, whereas the blue signals cause a difference of −15 dBm and above. The decrease in signal strength results from different interference sources.

-60

-60

-60

60cm from cage

55cm from cage CST

-70

-80

Signal strength dBm

CMK205

CMK205

-90

-90

-90

-100

-100

-100

-110

Lab21

-110

Lab21

-110

-120

-120

-120

-130

-130

-130

-140 -150 70

-140

80

90

100

110

120

130

-60

-150 70

90

100

110

120

130

-150 70

35cm from cage -70

CST

-100

-110

Lab21

-120

-120

-130

-130

-130

-140

110

120

130

-60

-150 70

90

100

110

120

130

-60

-150 70

-100

CMK205

-110

-120

-120

-130

-130

-130

-140

120

CST

-100

Lab21

-120

110

130

-150 70

130

-90

-110

-140

120

-80

-100

Lab21

100

110

10m from cage -70

-90

90

100

CMK205

-90

-110

90

-60

CST

-80 CMK205

80

80

20m from cage -70

CST

-80

-150 70

Lab21

-140

80

25cm from cage -70

CST

-110

-120

-140

130

CMK205

-100

Lab21

120

30cm from cage

-100

100

110

-80

-90

90

100

CMK205 -90

-110

90

-70

CST

-80 CMK205

-90

80

80

-60

40cm from cage

-80

-150 70

Lab21

-140

80

-60

-70

CST

-70

-80 CMK205

CST Physical

50cm from cage CST

-70

-80

Lab21

-140

80

90

100

110

120

130

-150 70

80

90

100

110

120

130

Time Fig. 10. Cage obstacle physical results comparison.

4.2

Cage Results Validation

Different copper cage physical experiments to verify the CST simulation are carried out. We moved antennas around the copper wall in an area 2 m2 in two different places. We choose two places in different buildings to perform the experiments. We sought environments with fewer sources of interference to a guarantee proper communication signal between the transmitter and the receiver. Figure 10 shows different cage signal shapes for CST simulation and physical experiments. Signals that are black represent simulation results, whereas signals that are red and blue are physical results. Blue physical signals are close to the CST signal (black), and the signal strength difference is between −3 dBm up to −20 dBm, whereas the red signals produce a difference of −20 dBm and above. The decrease in signal strength results from different interference sources.

678

5

M. Ayad and R. Voyles

Gradient Descent Method

The gradient represents the steepness and direction of that slope. Gradient descent is popular for most extensive optimization problems because it is easy to implement, and each iteration is reduced. Its major drawback is that it can take a long time to converge [15].

6

Gradient Computation of the RF Signal

In this work, a simple gradient approach is used. Therefore, the gradient descent is used to reduce the error on the signal strength because the robots estimate the signal strength gradient while they are moving. The gradient method is applied in a way that helps in minimizing the total path cost function, and increasing the likelihood of success of controlling the direction and the robots’ motion. In the gradient estimation experiments, a simple scenario was started where two mobile robots transmitted and received RF signals, respectively. The sensor measures RF signal strength at the receiver location. The two robots were separated by distance d along the x-direction as shown in Fig. 2(b) [10, 11]. The (j) signal strength Sl (k) at time k for the lth trajectory in the presence of RF obstacle type j can be measured according to Eq. 2. Robot2

Robot1

4

3 2

1

(xk(1) , yk(1))

(xk(2) , yk(2)) (a)

(b)

Fig. 11. Step wise trajectory of two robots as transmitter and receiver.

7

Gradient Descent Formulation

As the robots move in the RF environment with line of sight between each other, (j) the signal strength Sl (k) remains stable. However, if a conductive RF obstacle (j) appears in the experiment field, the Sl (k) is subject to change as the robots move around the RF obstacle. (j) In this scenario, the RF signal strength measurements, Sl (k), resulted from (i) (i) the two moving robots are measured and recorded for each (xk , yk ), i = 1, 2

RF Mapping for Signal Recovery

679

at time k. We then extract the gradient vector of the RF signal strength corresponding to known trajectories [10]. The gradient vector of the signal strength at time k for the lth trajectory is defined as (j)

 Sl (k) =

 ∂S (j) (k) ∂S (j) (k) ∂S (j) (k) ∂S (j) (k) T l l l l (1)

(1)

∂xk

(2)

∂yk

(3)

(2)

∂xk

∂yk

We compute the signal strength gradient using the trajectories as shown in (1) (1) Fig. 11 where we assume robot 1 and robot 2 are located at positions (xk , yk ) (2) (2) and (xk , yk ) at time k, respectively. If only one robot moves at a time and the other stays still, the gradient can be calculated using the following method. Figure 11 shows the step wise trajectories of the two robots for calculating the gradient vector. During time k and k+1, robot 1 moves along trajectory segment (1) (1) (1) (1) (2) (2) (2) (2) 1, so xk+1 = xk + Δx , yk+1 = yk , xk+1 = xk , yk+1 = yk , and the gradient (j)

element

∂Sl

(k)

is calculated as

(1)

∂xk

(j)

(j)

∂Sl (k)

Sl (k)

≈

(1)

∂xk

(1)

xk

(j)

=

(j)

Sl (k + 1) − Sl (k) (1)

(1)

xk+1 − xk

(j)

=

(j)

Sl (k + 1) − Sl (k) Δx

(4)

During time k + 1 and k + 2, robot 2 moves along trajectory segment 2, so (1) (1) (1) (1) (2) (2) (2) (2) xk+2 = xk+1 , yk+2 = yk+1 , xk+2 = xk+1 + Δx , yk+2 = yk+1 , and the gradient (j)

element is

∂Sl

(j)

∂Sl (k) (2)

∂xk

(k)

(2)

∂xk

calculated as

(j)

≈

Sl (k + 1) (2)

xk+1

(j)

=

(j)

Sl (k + 2) − Sl (k + 1) (2)

(2)

xk+2 − xk+1

(j)

=

(j)

Sl (k + 2) − Sl (k + 1) Δx

(5) During time k + 2 and k + 3, robot 1 moves along trajectory segment 3, so (1) (1) (1) (1) (2) (2) (2) (2) xk+3 = xk+2 , yk+3 = yk+2 + Δy , xk+3 = xk+2 , yk+3 = yk+2 , and the gradient (j)

element

∂Sl

(k)

(1)

∂yk

(j)

∂Sl (k) (1) ∂yk

is calculated as (j)

≈

Sl (k + 2) (1) yk+2

(j)

=

(j)

Sl (k + 3) − Sl (k + 2) (1) yk+3

−

(1) yk+2

(j)

=

(j)

Sl (k + 3) − Sl (k + 2) Δy

(6) During time k + 3 and k + 4, robot 2 moves along trajectory segment 4, so (1) (1) (1) (1) (2) (2) (2) (2) xk+4 = xk+3 , yk+4 = yk+3 , xk+4 = xk+3 , yk+4 = yk+3 + Δy , and the gradient (j)

element

∂Sl

(j)

∂Sl (k) (2) ∂yk

(k)

(2)

∂yk

is calculated as (j)

≈

Sl (k + 3) (2) yk+3

(j)

=

(j)

Sl (k + 4) − Sl (k + 3) (2) yk+4

−

(2) yk+3

(j)

=

(j)

Sl (k + 4) − Sl (k + 3) Δy

(7)

680

M. Ayad and R. Voyles 140

140 R2 moves

Robot2

Robot1 130

130

120

120

110

110

100

100

Y cm

Y cm

Cage

90

90

80

Cage

R1 still

80

70

70 Step wise trajectory

60 50 80

60

90

100

110

120 X cm

130

140

150

50 20

160

40

60

(a)

80 X cm

100

120

140

(b)

Fig. 12. (a) Two robots the cage, and (b) the left robot stays still and the right robot moves in stepwise trajectory. 150 200

R1 moves

140 130 120

150

100 R2 still

Y cm

Y cm

110

90 80

100

70 60 50

0

20

40

60

80

100

120

140

X cm

(a)

50 0

20

40

60

80

100

120

140

160

180

X cm

(b)

Fig. 13. (a) The right robot stays still and the right robot moves in stepwise trajectory, and (b) two robots move at the two sides of the cage.

8

Various Gradient Scenarios

As can be seen in Fig. 12(a), when the two robots hold the line of sight between each other, their receiving signal strength is strong enough for communication, which is indicated by yellow boxes, and their gradient directions point towards each to the direction of a strong RF signal, which is indicated by arrows. However, when one of the robots is trapped inside the cage, the signal strength magnitude become small and the gradient is less effective, which is indicated by green boxes and arrows. The small movement of the robot which is the outside of the cage do not result in a noticeable increase of the signal strength. However, small movement of the trapped robot results in a significant increase of the signal strength once it leaves the cage as can be noticed in Fig. 12(a). In Fig. 12(b), the robot moves extremely close to the cage in a step wise trajectory while the other robot stands still on the left side of the cage. In the presence of the line of sight between the two robots the gradient points in the direction of the strongest signal strength of the other robot. However, the gradient is subject to scatter when the RF obstacle blocks the two robots.

RF Mapping for Signal Recovery

681

Figure 13(a) shows the signal strength gradient direction, and the signal strength magnitude for the scenario when one robot is close to one corner of the cage while the other one moves in a stepwise trajectory around the cage. Similarly, as the previous example the gradient points toward the other robot when there is a line of sight, and it is subject to scatter as the robots are separated by RF obstacle. The impression from Figs. 12(b) and 13(a) shows that gradient directions can lead the robots to the favorable positions to maintain their connectivity. Figure 13(b) shows a scenario where two robots move in parallel taking turns along different trajectories around the cage. The robot at the left side moves two segments in a step wise trajectory, one segment in x-direction and the other one in y-direction while the other robot stands still. Then, the robot who was standing still moves one segment in y-direction along a straight line trajectory and this process repeated along each trajectory in the experiment field. In this experiment, the gradient is calculated for the right robot according to Eqs. 4 and 6. The gradient directions and magnitudes of the signal strength are shown at each robot position. This confirms that the gradients are helpful in finding favorable locations to maintain RF links between robots.

9

Gradient Algorithm Experimental Results

The gradient algorithm works on an experimental field area as shown in Fig. 14(a). The area or the configuration space of the robots is divided into grids as explained in Sect. 2.1. As shown in Fig. 14(a), two robots move in the area defined by the yellow grids, measure the signal strength, and compute the gradient for any two points in the field for both moving robots. A database was established, which contains the robots positions, RF signal strength measurements, and gradient computation results for each time k. The flowchart of the gradient algorithm is shown in Fig. 14(b). The algorithm starts by picking the (1) (1) (2) (2) x0 , y0 position of robot 1 and the x0 , y0 position of robot 2. The algorithm seeks its database for the robots position coordinates. If the robots positions are found in the database, then the algorithm checks the measured signal strength between the two positions of the robots. If signal strength power is above some threshold, then the two robots must move according to their normal velocity. Otherwise, each robot computes its gradient. The robot with higher gradient should proceed first in the direction of its gradient direction. The gradient algorithm is used for various scenarios, to verify its performance and success in driving robots towards favorable positions in the experimental field. In the first scenario, Fig. 15(a) shows two robots’ trajectories, where robot (1) (1) (2) (2) 1 starts at x0 = 20, y0 = 11, and robot 2 starts at x0 = 38, y0 = 14 both at time k = 0 in the experiment field. The gradient algorithm drives the robots through trajectories that avoid the RF obstacle shadow and maintain their connectivity. In the second scenario, Fig. 15(b) shows at time k = 0 two

682

M. Ayad and R. Voyles Start

Get (X1,y1) and (x2,y2) start robots (R1,R2) positions

Read (X1,y1), (x2,y2), signal strength from database yes no

no

Continue

If Found

Stop

140

yes

R2

130

Get Signal Strength

Cage no

120

Signal Strength < Threshold

Move R1 and R2

110

yes

100

Calculate Gradient GR1 and GR2

90 80

Move R2 in the direction of GR2

70

no

GR1> GR2

R1

yes

60 50

80

90

100

110

120 X cm

130

140

150

Move R1 in the direction of GR1

160

(a)

(b)

R1

R2 20 19 18 17 16 15 14 13 12 11 10 9

25

Distance

20

15 14 13 12 11 10 9 8 7 6

Destination

30

25

20 4 3 2

6 5

1

4

R1

R2

30 29 28 27 26 25 24 23 Cage 22 21 20 19 18 7 12 17 126 1516

5

8 7

15

X 5cm

Destination

30

Distance

X 5cm

Fig. 14. (a) Configuration space of two robots, and (b) gradient algorithm flowchart.

15

20 19 18 17 16 14 13 11 10 9 8 5 4 3

3 1

10

Threshold= −60

5

0

0

R1 starts at ( 20 ,

11 )

R2 starts at ( 38 ,

14 )

5

10

10

dbm

15

Threshold= −60

5

20

25 Distance

30

35

40

0

45 X 5cm

0

dbm

R1 starts at ( 17 ,

15 )

R2 starts at ( 38 ,

15 )

5

10

15

(a)

20 25 Distance

30

35

40

45 X 5cm

(b) (1)

(1)

(2)

(2)

Fig. 15. (a) Robots start at x0 = 20, y0 = 11 and x0 = 38, y0 = 14 at time k=0, (1) (2) (2) and (b) robots start at x0 (1) = 17, y0 = 15 and x0 = 38, y0 = 15 at time k=0. (1)

(1)

trajectories, where robot 1 starts at (x0 = 17, y0 = 15), at the entrance of (2) (2) the cage obstacle and robot 2 starts at (x0 = 38, y0 = 15), on the right side of the obstacle. Both trajectories show that the gradient algorithm can assist the robots to approach each other to avoid the effect of the RF obstacle and maintain the robots’ connectivity. The scenarios in Fig. 16(a) and 16(b) confirm the scenarios in Fig. 15(a) and (b) and assure the robots’ connectivity.

R1

R2 18 17 16 15 14 13 12 11 10 9 8 7

25

Distance

X 5cm

Destination

30

20

9 8 6

2 3

3

1

5 4

20 12

683

R2 22 21 20 19 18 17 16 15 14 13 12 11 10

4

10

14 13 12 11 10 9 8 7 3

9 4567

15

10

Threshold= −60

5

0

R1

25

5

6

15

Destination

30

7

Distance

X 5cm

RF Mapping for Signal Recovery

0

dbm

R1 starts at ( 22 ,

17 )

R2 starts at ( 40 ,

23 )

5

10

Threshold= −60

5

15

20 25 Distance

30

35

40

45 X 5cm

0

0

dbm

R1 starts at ( 19 ,

16 )

R2 starts at ( 40 ,

20 )

5

10

(a)

15

20 25 Distance

30

35

40

45 X 5cm

(b) (1)

(1)

(2)

(2)

Fig. 16. (a) Robots start at x0 = 22, y0 = 17 and x0 = 40, y0 = 23 at time k=0, (1) (1) (2) (2) and (b) robots start at x0 = 19, y0 = 16 and x0 = 40, y0 = 20 at time k=0.

10

Conclusion

This paper presented the concepts of RF Mapping and gradient approaches and explained the difference between RF mapping from physical mapping. The conducted numerical simulations showed that the RF signal strength measurements effectively along the robot trajectory in the experimental field to identify obstacle shadow. Several simulation experiments for different RF obstacle types and sizes are carried out and showed the RF obstacle’s effects on the RF signal measurements, producing various signal shapes. Signal shapes for three types of RF obstacles are shown and explained in detail. Besides, different physical RF obstacles experiments verified simulations, and physical results were presented. The proposed methods have presented promising solutions to maintain connectivity in the robotic network and can be a competitive alternative to broken link replacement and maintaining robot connectivity in robotic networks by controlling mobile robots’ movement in the field. Extensive physical and simulation experiments for different RF obstacle shapes, types, and sizes should be conducted to confirm the RF mapping and gradient methods’ ability in achieving robots’ connectivity and controlled motion.

References 1. Ayad, M., Voyles, R.: Physical link maintenance and logical message routing integration for robotic network connectivity. In: IEEE Vehicular Technology Conference (VTC-Fall 2019), Hawaii, September 2019 2. Fink, J., Kumar, V.: Online methods for radio signal mapping with mobile robots. In: IEEE ICRA Conference (2010) 3. Twigg, J.N., Fink, J.R., Yu, P.L., Sadler, B.M.: RSS gradient-assisted frontier exploration and radio source localization. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 889–895, May 2012 4. Ayad, M., Voyles, R.: Locally Selectable Protocol (LSP) for sparse, highly-volatile, robotic wireless video sensor networks. Int. J. Sens. Netw. (IJSNet) 20, 70–83 (2016)

684

M. Ayad and R. Voyles

5. Lindhe, M., Johansson, H., Bicchi, A.: An experimental study of exploiting multipath fading for robot communications. In: Proceeding of Robotics: Science and Systems, Atlanta, CA (2007) 6. Voyles, R.M., Bae, J., Larson, A., Ayad, M.: Wireless video sensor network for sparse, resource-constrained, multi-robot teams. J. Intell. Serv. Robots 2(4), 235– 246 (2009) 7. Michael, N., Zavlanos, M.M., Kumar, V., Pappas, G.J.: Maintaining connectivity in mobile robot networks. In: International Symposium on Experimental Robotics, Athena, Greece (2009) 8. Bae, J., Voyles, R.M.: Wireless video sensor network from a team of urban search and rescue robot. In: International Conference on Wireless Networks (2006) 9. Hahnel, D., Ferris, B., Fox, D.: Gaussian processes for signal strength-based location estimation. In: Proceeding of Robotics: Science and Systems, Philadelphia, PA (2006) 10. Ayad, M., Zhang, J.J., Voyles, R., Mahoor, M.H.: Mobile robot connectivity maintenance based on RF mapping. Presented at the (2013) 11. Ayad, M., Zhang, J.J., Voyles, R., Mahoor, M.H.: Electromagnetic field recognition for proactive robot communication connectivity maintenance. In: Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR) (2012) 12. Fink, J., Kumar, V., Michael, N., Kushleyev, A.: Experimental characterization of radio signal propagation in indoor environments with application to estimation and control. In: IEEE International Conference on Intelligent Robots and Systems (2009) 13. Zeiger, F., Kraemer, N., Schilling, K.: Commanding mobile robots via wireless adhoc networks - a comparison of four ad-hoc routing protocol implementations. In: IEEE International Conference on Robotics and Automation (ICRA) (2008) 14. CST microwave studio. http://www.cst.com/MWS/Overview.aspx 15. Hauser, K.: B553 lecture 4: gradient descent (2012). http://homes.soic.indiana. edu/classes/spring2012/csci/b553-hauserk/gradient-desc

DBA-Filter: A Dynamic Background Activity Noise Filtering Algorithm for Event Cameras Sherif A. S. Mohamed1(B) , Jawad N. Yasin1 , Mohammad-Hashem Haghbayan1 , Jukka Heikkonen1 , Hannu Tenhunen2 , and Juha Plosila1 1

2

University of Turku, 20500 Turku, Finland [email protected] KTH Royal Institute of Technology, 11428 Stockholm, Sweden

Abstract. Newly emerged dynamic vision sensors (DVS) offer a great potential over traditional sensors (e.g. CMOS) since they have a high temporal resolution in the order of µs, ultra-low power consumption and high dynamic range up to 140 dB compared to 60 dB in frame cameras. Unlike traditional cameras, the output of DVS cameras is a stream of events that encodes the location of the pixel, time, and polarity of the brightness change. An event is triggered when the change of brightness, i.e. log intensity, of a pixel exceeds a certain threshold. The output of event cameras often contains a significant amount of noise (outlier events) alongside the signal (inlier events). The main cause of that is transistor switch leakage and noise. This paper presents a dynamic background activity filtering, called DBA-filter, for event cameras based on an adaptation of the K-nearest neighbor (KNN) algorithm and the optical flow. Results show that the proposed algorithm is able to achieve a high signal to noise ratio up to 13.64 dB. Keywords: Event cameras Noise

1

· Background filtering · KNN · Dynamic ·

Introduction

Novel silicon retina or “event cameras” are bio-inspired vision sensors that mimic the neural architecture of the human eyes. The output of event cameras is fundamentally different from traditional sensors (i.e. CMOS sensors). The output of traditional cameras is a sequence of images “absolute brightness” at a constant temporal rate (typically between 30 120 Hz). Images can be either monochrome “gray-scale” images or images that have three channels per pixel, such images are called RGB images. On the other hand, event cameras measure per-pixel brightness changes (called “events”) asynchronously. This results in a stream of events with a high dynamic range, low power consumption, and at a high temporal rate (up to 1 MHz). Due to these characteristics, event cameras offer a great potential to solve classical as well as new computer vision problems [11]. Recently, they become commercially available in various models such as Dynamic c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 685–696, 2022. https://doi.org/10.1007/978-3-030-80119-9_44

686

S. A. S. Mohamed et al.

Vision Sensors (DVS) [7], Dynamic and Active pixel Vision Sensor (DAVIS) [3], and Asynchronous Time-based Image Sensor (ATIS) [14]. Event cameras are similar to all other vision sensors they are noisy because of the inherent shot noise in photons and from transistor circuit noise, and they also have non-idealities [5]. To overcome this undesired effect, a sort of filtering block is required as a pre-processing block for event-based visual odometry (VO) algorithms. Alzugaray et al. [1] proposed a filtering method to remove redundant events caused by sudden and significant contrast change using a constant timestamp threshold k = 50 ms. The mechanism of the proposed filter is that the incoming event is processed if it has a different polarity from the previous one at the same location. However, if it has the same polarity it is only processed if its timestamp (i.e. time of its occurrence) t > t1 + k, where t1 is the timestamp of the latest event triggered in the same location. In [13], the authors propose a filtering algorithm using Neural Network-Based Nearest Neighbor (NeuNN) to remove noise from data captured by an ATIS camera exploiting the neuromorphic architecture of the IBM TrueNorth processor [10]. The filter considers √ the activity of incoming events with its neighbors within the distance D ≤ 2, i.e. the surrounding eight neighboring pixels. This activity, i.e. synaptic activity, is summed on to a neuron and the output of the activity is determined based on the configuration of the neuronal parameters. This summation of the synaptic, leak, and membrane voltage of the neuron is compared with a certain threshold. If it is greater than or equal to the threshold, then the neuron fires a spike. The main drawback of this filtering method is that the total number of required cores to perform the noise filtering increases rapidly with the size of the filter and the sensor, which makes real-time performance infeasible on embedded systems, i.e. resource-constrained systems. In this paper, we propose a dynamic filtering algorithm that is able to achieve a high signal-to-noise ratio and achieve real-time performance on a Jetson TX2 system. The filter consists of two main stages: the timestamp (TS) filter and the BA filter. The TS-filter is based on optical flow and the timestamp of the previous event at the same location. The filter is able to discard noise, i.e. redundant events, caused by sudden movements and significant contrast changes. The BAfilter is based on an adaptation of the KNN algorithm and it is responsible to discard the BA noise caused by the hardware noise, i.e. transistor switch leakage. The proposed approach is able to significantly increase the signal-to-noise ratio and reduce the number of processed events captured by the event-based camera without the loss of relevant information. The rest of this work is structured as follows. In Sect. 2, we illustrate the principle operation of event cameras and the cause of the noise. In Sect. 3, we explain the methodology of the proposed dynamic filtering. In Sect. 4, we present the experimental results. Finally, in Sect. 5, we draw a conclusion of the article.

Hamiltonian Mechanics

2

687

Dynamic Vision Sensors

Recently, DVS (i.e. event-based) cameras offer a huge unprecedented raw performance characteristics and usability over traditional frame-based cameras. Many applications, such as localization and obstacle avoidance [16, 17] can exploit these powerful characteristics to achieve better real-time performance, power efficiency, and high accuracy. In [18], the authors proposed an algorithm to perform obstacle avoidance in low lighting conditions based event-camera. DVS cameras are particularly suitable for applications that require to perform localization or object tracking in challenging conditions such as high-speed maneuvers and low-light conditions. 2.1

Principle of Operation

A simplified circuit diagram of the DVS is shown in Fig. 1. Each pixel in the sensor is composed of three main parts: photoreceptor, differencer, and two comparators. Pixels memorize the log intensity of their previous events and continuously monitors the changes in intensity. As illustrated in Fig. 2, an event ei is triggered at pixel pi = (xi , yi ) and at timestamp tsi when the change of intensity, i.e.  log I(pi , tsi ) = log I(pi , tsi ) − log I(pi , tsi − tsi )

(1)

exceeds a certain threshold C, typically adjusted between 10% and 50%, i.e. | log I| > C

(2)

The Address Event Representation (AER) protocol [9] is used to generate the information for each event (quadruplets), i.e. ei = (xi , yi , tsi , poli )

(3)

where xi and yi denote the position of the pixel, i.e. the column and row of the sensor, respectively. The timestamp of the event is denoted by tsi and polarity poli {+1, −1} is the sign of the intensity change.

Fig. 1. Simplified circuit diagram of the DVS pixel

688

3

S. A. S. Mohamed et al.

Filtering

To reduce the amount of redundant events and increase the signal to noise ratio, we propose a filtering method (called DBA-filter) that is composed of two stages: the timestamp-filter (TS-filter) and the background active filter (BAfilter) (Fig. 4).

Fig. 2. Schematic of the operation of a DVS pixel, converting brightness changes into events

Fig. 3. The timestamp of the events triggered due to the translation of a moving dot. The (red) line represents the naive SAE, i.e. w/o filtering which stores the latest timestamp. On the other hand, our proposed TF-filter discards redundant events and the SAEts (green) stores only the timestamp of unique events which represent the time when the visual stimulus accrued.

3.1

TS-Filter

According to Eq. 2, a sudden movement or a significant contrast change would trigger multiple events at the same pixel in a short time window. Processing those duplicate events increases the computational cost unnecessarily. Moreover, it affects the accuracy since the timestamp stored in the Surface Active Event (SAE) does not represent the time when the visual stimulus was accrued, as illustrated in Fig. 3. SAE is a common way which uses the spatio-temporal

Hamiltonian Mechanics

689

Fig. 4. An overall system architecture of the proposed filtering

Algorithm 1: Dynamic TS-filter Result: ets initialization; if poli = polj then Process event; Update G-SAE; else Select batch ei ∈ PN ; Obtain batch ← OFPN ; Calculate → tr ; if tsi ≤ tsj + tr then Discard event; else Process event; Update G-SAE; end end

domain. It can be considered as an elevation map, which is composed of the two-dimensional sensor frame (x, y) and one dimension to represent the time. The first stage of the proposed algorithm, i.e. TS-filter, is illustrated in Algorithm 1. We use a global SAE (G-SAE) to store the information for filtered events. The G-SAE has the same dimension of sensors which stores the timestamp of events (32-bit) and the polarity of events (1-bit). For instance, using a DVS with 240 × 180 pixels will result in a G-SAE of size 175 KB. The filter checks the polarity of the previous event at the same location in G-SAE. If the polarity of the incoming event differs from the polarity of the previous event in the same location, the incoming event is processed. Otherwise, events triggered within time-window tr are discarded and only events with timestamp bigger than the threshold are processed, i.e. tsi > tsj + tsr

(4)

where the timestamp of the incoming event is denoted by tsi . The timestamp of the previous event at the same location is denoted by tsj , and tsr denotes

690

S. A. S. Mohamed et al.

the timestamp threshold. The optical follow (OF) of each batch of size 15 × 15 is computed using local planar [2]. A suitable timestamp threshold is obtained for each batch, i.e. A ; ei ∈ Pk (5) tri = OFPk + B

3.2

BA-Filter

Background activity (BA) events are triggered without any brightness or movement changes, however, due to hardware limitations. The main causes of BA noise are thermal noise and current leakage of switches [7]. This noise degrades the quality of captured data and increases the computational costs unnecessarily. Hence, a filtering algorithm to eliminate BA noise is necessary to obtain a high-performance localization. The main characteristic of BA events is that they lack correlation with events in their spatiotemporal neighborhood, i.e. ⎧ ⎨ |x1 − x2 | ≤ dP (6) corr(e1 , e2 ) = |y1 − y2 | ≤ dP ⎩ |t1 − t2 | ≤ dT The spatiotemporal correlation of two events e1 (x1 , y1 , t1 ) and e2 (x2 , y2 , t2 ) is denoted by corr(e1 , e2 ) as illustrated in Eq. 6. The spatial window size is denoted by dP and the time size is denoted by dT . The probability of observing BA events on a pixel on the DVS sensor can be considered as a Poisson distribution [6], i.e. P (ei = w) =

(λt)w e−λt ; w

λ>0

(7)

where w is the number of BA events triggered in the time interval t and λ is the BA rate. In [4, 6, 8], the authors have used a fixed size window to filter the BA events. Based on the movement speed of the camera or the object in the scene, two or more BA events might be close enough in the spatiotemporal neighborhood. Hence, using a fixed spatiotemporal neighborhood, see Fig. 5, would pass BA noise or filter real events in some cases. Therefore, we propose a second stage filtering, i.e., a dynamic BA filtering algorithm based on an adaptive spatiotemporal neighborhood window and KNN to filter the BA noise. The size of the neighborhood window is determined based on the optical flow of the batch. The K parameter of the KNN algorithm is obtainedbased on the entropy of the batch,

Hamiltonian Mechanics

691

y

t

x

Fig. 5. Principal of spatiotemporal neighbourhood of size (3 × 3 × 3). Green block indicates the incoming event and red blocks denotes the neighboring events.

i.e., number of events in the last t time interval. The incoming event will pass if it has neighbors more or equal to K threshold within its neighborhood window. The proposed algorithm is summarized in Algorithm 2. The sensor is subsampled into N number of batches, each batch has a size of 15 × 15 pixels, which is used to calculate the optical flow (OF). The OF is computed using the RANSAC algorithm by fitting a local planar [2]. The size of the spatiotemporal neighborhood (STN) window is an odd number and varies from 3 × 3 to 15 × 15. An incoming event in a high-speed batch will have a size of 3 × 3, on the other hand, events fall in slow batches will STN of size 15 × 15. The entropy of each batch is computed by calculating the number of events that occur within a t time interval, which is used to determine the K parameter. If the number of neighboring events with STN of the incoming event is less than K the event is eliminated, otherwise it passes the filter.

4 4.1

Results Experimental Setup

We evaluated the proposed filtering algorithm by running the algorithm on publicly available datasets of [12], [15]. We carefully selected a number of subsets to ensure a fair and comprehensive evaluation scheme. The selected subsets are composed of simple and complex scenes, including low and high textured environments, and slow and fast motions. In a slow-motion scenario, the DAVIS camera generates up to 3 × 105 events per second. On the other hand, in rapid camera/object movements, it can generate up to 3 million events per second. The datasets are recorded by a Dynamic and Active-pixel Vision Sensor DAVIS-240, which contain many sequences of frame-based, i.e., intensity images, and asynchronous events at the resolution of 240 × 180. Note that the intensity images

692

S. A. S. Mohamed et al.

Algorithm 2: Dynamic BA-filter Result: ef subsampling → PN ; Calculate OF → OFPN ; Calculate Entropy → EPN ; while ets do Select batch ets ∈ PN ; Obtain batch ← OFPN ; Compute STN(x,y,t); Obtain batch ← EPN ; Compute K; if neighbours ∈ SN < K then Discard event; else Pass event; end end

are only used to obtain the ground-truth for evaluation purposes. The proposed algorithm has been implemented in software in C++. The application was run on an Nvidia Jetson TX2 board with quad-core ARM Cortex-A57 CPU @ 2 GHz clock frequency. 4.2

Qualitative Evaluation

Figure 6 shows a qualitative comparison of the video output of the DVS camera without using any filter and using our proposed filter. The dataset (i.e. shapes 6dof ) used in this evaluation is recorded indoors using a DVS camera moving freely in 6-DOF with different motion speeds. The shapes appear in the video are 2D pictures mounted on a wall. The results show that the proposed filter is able to cope with both slow-motion and high-speed camera motion to remove background noise and filter redundant events generated from the sudden movement, significant contrast changes, and hardware limitations. In order to comprehensively evaluate the proposed algorithm, we have tested it on the night run dataset. This dataset is recorded outdoors by a static camera, while a human is running in front of the camera. Figure 7, show that the proposed filter is able to filter noise not only in static scenes but also in a dynamic environment without losing valuable information.

Hamiltonian Mechanics

[slow motion w/o filter]

filter]

[fast motion w/o filter]

693

[slow motion w/

[fast motion

w/ filter]

Fig. 6. Shapes sequence. Screenshots of the DVS video output at various time instances.

[moving object w/o filter]

filter]

[no motion w/o filter]

[moving object w/

[no motion w/

filter]

Fig. 7. Night run sequence. Screenshots of the DVS video output at various time instances.

4.3

Quantitative Evaluation

In order to quantify the noise filtering performance using the algorithms detailed in Sect. 3 and their implementation on Jetson TX2 described in Sect. 4.1, we first define the following terms and the metrics below to compute them on the unfiltered and filtered data, see Table 1.

694

S. A. S. Mohamed et al. Table 1. Quantitative metric Abbr.

Definition

Rw/o

Total no. of real events without filtering

Nw/o

Total no. of noise events without filtering

Rf

Total no. of real events with filtering

Nf

Total no. of noise events with filtering

PRR

Per. % of remaining real events w/filtering

PNR

Per. % of remaining noise events w/filtering

RNRw/o Real to noise ratio without filtering RNRf

Real to noise ratio with filtering

The results of the quantitative evaluation are summarized in Table 2. Two different datasets, i.e. shapes 6dof and nigh run, were used for the evaluation. The results show that the proposed algorithm is capable of filtering noise caused by sudden movement and background activity effectively. The filter is able to maintain an RNR, i.e. signal-to-noise ratio (SNR), up to 13.64 dB in a static scene, and up to 6.709 dB in a high dynamics scene. Table 2. The quantitative performance of proposed algorithm on dataset. shapes 6dof night run

5

Rw/o

10.31 × 106 0.34 × 106

Nw/o

7.62 × 106

0.455 × 106

Rf

9.73 × 106

0.3 × 106

6

Nf

0.42 × 10

0.064 × 106

PRR

94.37%

88.2%

PNR

5.24%

14.06%

RNRw/o 1.31 dB

−1.27%

RNRf

6.709 dB

13.64 dB

Conclusion

We have presented DBA-filter, adaptive filtering method that redundant events and noise based on asynchronous events. There are two main sources of noise such as sudden and significant contrast changes and background activities. Our method is composed of two-stage: 1) The first stage, i.e. TF-filter is to discard redundant events using dynamic timestamp techniques. The timestamp is computed based on optical flow for each batch; 2) the second stage, i.e., BA-filter

Hamiltonian Mechanics

695

is designed to discard background events based on an adaptive KNN algorithm. We demonstrated that our approach facilitates execution under different scenarios. Our method is able to discard unnecessary events without loss of relevant information. Acknowledgment. This work was supported by the Academy of Finland under the project (314048).

References 1. Alzugaray, I., Chli, M.: Asynchronous corner detection and tracking for event cameras in real time. IEEE Robot. Autom. Lett. 3(4), 3177–3184 (2018) 2. Benosman, R., Clercq, C., Lagorce, X., Ieng, S.-H., Bartolozzi, C.: Event-based visual flow. IEEE Trans. Neural Networks Learn. Syst. 25(2), 407–417 (2014) 3. Brandli, C., Berner, R., Yang, M., Liu, S., Delbruck, T.: A 240 × 180 130 db 3 µs latency global shutter spatiotemporal vision sensor. IEEE J. Solid-State Circuits 49(10), 2333–2341 (2014) 4. Delbruck, T.: Frame-free dynamic digital vision. In: Proceedings of the International Symposium on Secure-Life Electronics, Advanced Electronics for Quality Life and Society, March 2008 5. Gallego, G., et al.: Event-based vision: a survey. CoRR, abs/1904.08405 (2019) 6. Khodamoradi, A., Kastner, R.: O(n)-space spatiotemporal filter for reducing noise in neuromorphic vision sensors. IEEE Trans. Emer. Top. Comput. 1 (2017) 7. Lichtsteiner, P., Posch, C., Delbruck, T.: A 128×128 120 db 15µs latency asynchronous temporal contrast vision sensor. IEEE J. Solid-State Circuits 43(2), 566– 576 (2008) 8. Liu, H., Brandli, C., Li, C., Liu, S., Delbruck, T.: Design of a spatiotemporal correlation filter for event-based sensors. In: 2015 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 722–725 (2015) 9. Mahowald, M.: An Analog VLSI System for Stereoscopic Vision. Kluwer Academic Publishers, New York (1994) 10. Merolla, P.A., et al.: A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345(6197), 668–673 (2014) 11. Mohamed, S.A.S., Haghbayan, M., Westerlund, T., Heikkonen, J., Tenhunen, H., Plosila, J.: A survey on odometry for autonomous navigation systems. IEEE Access, 97466–97486 (2019) 12. Mueggler, E., Rebecq, H., Gallego, G., Delbr¨ uck, T., Scaramuzza, D.: The eventcamera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM. I. J. Robotics Res. 36(2), 142–149 (2017) 13. Padala, V., Basu, A., Orchard, G.: A noise filtering algorithm for event-based asynchronous change detection image sensors on truenorth and its implementation on truenorth. Front. Neurosci. 12, 118 (2018) 14. Posch, C., Matolin, D., Wohlgenannt, R.: A QVGA 143 db dynamic range framefree PWM image sensor with lossless pixel-level video compression and timedomain CDS. J. Solid-State Circuits 46(1), 259–275 (2011) 15. Scheerlinck, C., Barnes, N., Mahony, R.: Continuous-time intensity estimation using event cameras. In: Asian Conference Computing Vision (ACCV), December 2018

696

S. A. S. Mohamed et al.

16. Yasin, J.N., Mohamed, S.A.S., Haghbayan, M., Heikkonen, J., Tenhunen, H., Plosila, J.: Unmanned aerial vehicles (UAVS): collision avoidance systems and approaches. IEEE Access 8, 105139–105155 (2020) 17. Yasin, J.N., et al.: Energy-efficient formation morphing for collision avoidance in a swarm of drones. IEEE Access 1 (2020) 18. Yasin, J.N., et al.: Night vision obstacle detection and avoidance based on bioinspired vision sensors. In: 2020 IEEE SENSORS, pp. 1–4 (2020)

Gamification in a Learning Resource for the Study of Human Computer Interaction Dawn Carmichael1(B) , Claire MacEachen2 , and Jacqueline Archibald2 1 Glasgow Caledonian University, Glasgow, UK

[email protected] 2 Abertay University, Dundee, UK

Abstract. Human Computer Interaction (HCI) is increasing in relevance as it, in part, concerns the design of digital interfaces such as downloadable apps. Despite the importance of the subject area students frequently find the topic boring, dry and overly theoretical. In order to address this apparent lack of student interest, a gamified learning resource for a HCI module in the second year of University was created. A literature review into the HCI educational requirements was conducted and used to inform the design. A further analysis of the literature on dialogic feedback, simulations and gamification features was conducted and key features were matched to the educational requirements. The resultant gamified learning resource, dubbed the “work simulation”, was evaluated first by a two cohort comparison for module lecture attendance and exam attainment. The results of the first phase of the evaluation showed a statistically significant improvement in both attendance and attainment for the cohort experiencing the “work simulation”. A further evaluation was carried out by means of interviews with students, which suggested that dialogic feedback and the game narrative/fiction, in particular, were useful. Keywords: Educational technology · Gamification · Human Computer Interactions

1 Introduction Human Computer Interaction (HCI), sometimes Computer Human Interaction (CHI), is the study of design and evaluation methods for computer based technology. Traditionally the main concerns of HCI have been efficiency and reliability [1]. However more recently emerging concerns in HCI include; usability, accessibility and User eXperience (UX). In the last few years the availability of easy to use tools for rapidly developing web and mobile applications has led to a proliferation of apps of varying degrees of usefulness. The importance of being able to design effective interfaces and to evaluate their performance has never been more evident. The role of HCI as a subject within ‘computing’ courses is crucial if the next generation of software designers and developers are going to be able to meet the challenges of tomorrow.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 697–717, 2022. https://doi.org/10.1007/978-3-030-80119-9_45

698

D. Carmichael et al.

1.1 HCI Pedagogic Challenges One of the key objectives when teaching HCI is to assist students in preparing for professional practice [2]. However many of the core techniques such as scenario building and storyboarding can seem irrelevant to students and are not always taken seriously by them [3]. For instance in a study by Aberg [3] students described the study of HCI as ‘fuzzy’ and ‘trivial’. HCI has a relatively large theory base relative, that is, to more praxis oriented subject areas such as programming. Factors such as perceived relevance and the study of theory together with the fast pace of change [4] can make HCI a pedagogic challenge. 1.2 HCI Educational Aspirations It has been a long held view that a HCI learning experience should be active for students and should be engaging [1]. Aberg [3] suggests that students also need a sense of realism in HCI learning contexts for example the use of high fidelity prototypes as opposed to paper prototypes that can add to a sense of achievement. Furthermore, students should be engaged in creating meaningful projects [5] that are situated in real contexts [6]. It has also been argued that regularly submitted exercises are a good means of drawing connections to the course literature [3]. In a similar vein the use of ‘case studies’ in HCI education is important as it can add authenticity [1]. Case studies can be problem-based using real world or analogous scenarios which may encompass development projects and papers from the literature. Typically, students are asked to apply methods to clearly defined problem areas. Case base learning also includes decision-making cases where students are asked to arrive at a finding and be able to justify the reasoning. Decisionmaking cases have the advantage of encouraging reflection but are considered by some to be difficult to deliver to students [1]. Active learning and relevant case studies are the main stay of HCI education and in addition there are other aspirations for quality enhancement. For example HCI learning should facilitate both individual and collaborative learning [5]. It has also been suggested that it is useful to include feedback which can be rewarding and improve participation Aberg [3]. Some sources have argued that it is important to have strict requirements on reflection Aberg [3] as reflection is an important aspect of experiential learning [7] which underpins much of contemporary HCI education. Another useful goal for HCI education is to support communication between students, peers and tutors [1]. It has also been suggested that unexpected experiences [7] can add interest and that puzzles and games are useful for supporting students in dealing with difficult concepts [8]. Summary of Requirements for effective HCI education from the literature: • • • • • • •

Active learning [1] Creating meaningful projects [5] real context projects [6]. Regular hand-in exercises and feedback [3]. Problem and decision based cases [1]. Promote realism by working on high-fidelity prototypes [3]. Maintain individual and collaborative learning [5] Promote reflection [3, 7]

Gamification in a Learning Resource for the Study of HCI

699

• Interactions between students and tutors [1]. • Include unexpected experiences [7] • Inclusion of quizzes and games [8].

2 Pedagogy for HCI In the previous section we outlined the general problem which is the improvement of HCI education and arrived at a set of requirements for learning. The pedagogy provides us with a number of powerful ideas which can help to inform the design and creation of a response. 2.1 Blended Learning Having arrived at requirements for HCI education the next step is to consider how pedagogic theory can be applied in pursuit of the goal of creating a solution. The requirements suggest that a mixture of face to face teaching and e-learning resources, commonly referred to as blended learning [9], are appropriate. Blended learning has become more significant due to the prevalence of VLE’s in universities [10]. It has been suggested that blended learning represents a paradigm shift from teaching to learning [11] and that there are advantages of using both, face to face and digital resources, over technology alone [12]. It is thought that adding online resources to traditional delivery can enhance student performance [13–17]. For instance, a study into the implementation of blended learning found a positive effect in decreasing dropout rates, and improving exam results and final grades. The positive outcomes associated with blended learning activities may improve student perception of the subject and effect career decisions [18]. A similar finding suggested that blended learning and attendance was indicative of student commitment and improved final marks [19, 20]. However not all sources are entirely positive about the use of blended learning, for instance Traphagen et al. [21] points out that making learning materials available via a Virtual Learning Environments (VLE) such as blackboard can lead to poor attendance. Specifically finding that 76% of students agreed or strongly agreed that they had skipped classes because materials were available online; however it was also found that webcasting lectures largely offset the effect of non-attendance [21]. Overall there is considerable evidence that a blended learning approach is appropriate for matching our requirements. 2.2 Dialogic Feedback Assessment and feedback is the means by which we measure academic standards and support students learning. They have a major influence on student perception and experience of learning [22]. Feedback is especially valued by students but frequently falls short of their expectations [23]. Feedback takes up a considerable proportion of staff time, effects reputation, impacts on league tables and, most importantly, shapes the way students study and learn. However survey results, across the sector, show students are often disappointed with the feedback that they receive [24]. In response academics react

700

D. Carmichael et al.

by attempting to produce faster feedback rather than look at the whole assessment process. It was been stated in a NUS student experience report [25] that 71% of students would prefer individual verbal feedback but only 25% of students experience this type of feedback. In a study conducted for the Higher Education Academy, Williams& Kane [24] suggested that students required dialogue in order to be able to successfully interpret feedback. One system level approach to organising feedback is the dialogic feedback model which suggests that assessment and feedback should be organised as a dialogue between the tutor and student [26–28]. The dialogic feedback model is a dynamic process which provides scaffolding for the learning process [27]. The rationale for the effectiveness of dialogic feedback is based upon the central idea that the term ‘feedback’ implies a response. In other words, if a student submits work and receives comments that don’t affect their subsequent actions then these comments are not feedback in the true sense of the word. It is not atypical for students to receive ‘feedback’ at the end of a module when coursework is submitted. It is possible that this feedback might affect how students submit subsequent work but it is by no means certain. The dialogic feedback approach suggests that in order to improve feedback to students, the feedback has to enable students to take actions based upon that feedback. The dialogic approach to feedback also suggests that feedback should be personalised to an individual student rather than general feedback that would be applicable to most or the entire cohort. There are several aspects of the dialogic feedback model that have the potential to enhance the student experience such as providing a process for learning. However, for us the key question relates to how the approach may support of the HCI educational requirements. The requirements for active learning and feedback could be accommodated with the dialogic feedback model as it requires activity from the student based upon reflection of feedback. In a similar vein the nature of the dialogic feedback model also requires interactions between students and tutors. The dialogic feedback model is generic, rather than subject based, and does not necessarily require criteria such as real world projects or problem and decision based cases. Overall the dialogic feedback model can provide a useful framework for design a learning experience which could promote learning. 2.3 Simulations The use of work simulations in higher education can be a highly effective means of delivering employability skills [29]. It has been suggested that simulations can increase enjoyment in learning, allow safe experimentation in decision making [30] and help learners to understand theory [31]. Specifically the use of work simulations for management and executive courses is well established and often reported on favourably [32]. Ideally simulations should provide quick and detailed feedback to the learner [33] in the experiential learning tradition. There are caveats to the approach of using a simulation including the fact that some are thoughttobeinefficientingeneral[34]andinparticularforsomesubjectareas[35,36]have a lack of flexibility [29], over simplicity [37]. Simulations should not attempt to simplify or replace real world interactions within a group by a computer interaction [29]. It has been

Gamification in a Learning Resource for the Study of HCI

701

suggested that “topical and enduring Face To Face feedback” between students and tutor discussions can promote reflective practice and are more important than a simulation based interaction [29]. In spite of some limitations simulations can generate insights and deepen learning [38] and overall they can make a valuable addition to blended learning [29]. The key question in this study is ‘how’ and ‘to what extent’ simulations can address the requirements for HCI education which were outlined above. One of the requirements for HCI education concerns providing meaningful projects (problem and decision based cases) in real contexts and this is an affordances of simulations. Another of the key requirements for HCI education is that it should include unexpected experiences as well as quizzes and games. It would not be unreasonable to suggest that it is possible to design into the simulations narrative expected events or changes to circumstances. Although it is less clear that a simulation will provide for quizzes and games. There are other key HCI educational requirements that are not inherently supported by a simulation. For example, not all simulations enable collaborative learning and interactions between both students and tutors. Similarly, not all simulations are designed to support regular hand-in, feedback and reflection. An analysis of gamification was carried out in order to fully match the HCI educational requirements with design features. 2.4 Gamification Gamification is the use of game concepts in non-gaming systems in order to improve the user experience of the system [39]. The idea is proving popular with many organisation in a range of sectors such as business, government, health and education [40]. The reasoning behind the use of gamification is that there are thought to be motivational properties of educational games which include allowing students to communicate and share accomplishments [41, 42]. There have been studies that state that student academic performance is significantly higher amongst those that participated in a gamified resources than those who did not [43]. 2.4.1 Examples of Gamification There are many examples of gamification in education such as Indiana University where course metrics and activities were gamified. Students were required to gain points, to move up the levels (grades), by completing activities presented using gaming concepts such as ‘fighting monsters’ which was used as a metaphor for tackling assessments [44]. In another example, narrative elements were used to target behaviours which were assessed as successes or failures [45]. In another example game elements, were used to incentivise students to complete online practice tests [46]. Although the proliferation of games concepts seems promising it has been suggested that many gamification initiatives will fail due to a lack of understanding of the cause and effect in the use of gaming concepts [40]. 2.4.2 Caveats Although there have been studies going back a number of years suggesting that educational games and gamification may lead to academic success [47], there are also studies

702

D. Carmichael et al.

that present significant caveats. Gamification can lead to mixed results due to design of the resources and their motivational affordances [48, 49]. It has been suggested that gamification helps students improve practical work but not written work [50]. It has also been found that students with prior experience of games had a greater positive valence for gamification [51] than mature students would did not have prior experience of games [43]. In some instances gamification did not have a positive affect thought to be due to cognitive dissonance for students who did not like the mixing of work and play [52]. Whilst it has been suggested that gamification can generate positive, intrinsically motivating learning experiences [53] it has also been asserted that gamified materials led to less intrinsic motivation [54]. The contradiction here is important because it is thought that deep learning is associated with intrinsic motivation whilst surface learning is associated with extrinsic motivation [55]. The concern regarding extrinsic motivation is focused on the potential for the goals of the game to become more important than learning the material. Out of concern for this possibility it has been suggested that gamification should avoid excessive use of the achievement aspects of gamification as it is based upon extrinsic motivation which isn’t sustainable in the long run [56]. In a more recent study it was found that there was a decline in usage of a gamified resource over time [43] there was a similar decline in participation in this study a finding that is discussed below. 2.4.3 Design Features Educational gamification includes the use of game mechanics with a Virtual Learning Environment (VLE) providing a flexible learning environment [41]. The game mechanics can include avatars, badges, leader boards, content unlocking and other virtual goods [57] these game features can act as external rewards [58]. Goal setting in the gamified learning resource can be used to direct the attention of learners and to motivate students [41]. Kofinas [59], proposed a narrative approach to gamification to encompass both formative and summative assessments which can be set as challenges to overcome. A narrative approach was utilised in this study. It has been suggested that in the design stage of a gamified learning resource that the process of feedback to students has to be incorporated. The feedback process should be implemented using a loop of motivation to action to feedback to motivation, in order to encourage engagement with the resource [60]. Another key feature of a gamified system is the ‘novelty effect’ which seeks to intrigue users and to encourage them to progress through the content of the learning resource, however as the novelty wears off the motivation of students may decrease [48, 49, 54, 61]. The ideas of using a ‘feedback loop’ and the ‘novelty effect’ were instantiated in the ‘work simulation’ in this study. The use of games concepts has been used quite informally with individual researchers choosing familiar ideas and applying them to learning [40]. However, Bedwell [62] has created a taxonomy of gamification attributes which are explained in Table 1 [62].

Gamification in a Learning Resource for the Study of HCI

703

Table 1. Gamification attributes Attributes

Explanation

Action Language

Communication between player and game e.g. UI controls

Assessment

Progress tracked e.g. points and/or leader board

Conflict/challenge

Problems and difficulty e.g. inter-team competition

Control

Player alter the game e.g. discussion forum

Environment

Surroundings of the player e.g. move from physical to virtual

Game fiction

Game story e.g. lectures and tests are renamed adventures

Human Interaction

Degree of player interaction e.g. participation noticeable to others

Immersion

Affective & perceptual experience e.g. use of real world imagery

Rules/goals

Information for progress e.g. correct answers

2.5 Summary This section contains a review of a number of approaches to the design of a blended learning resource such as; simulation, dialogic feedback and gamification. In summary Table 2 provides a mapping of requirements to attributes suggested by the approaches. Table 2. Mapping of requirements to pedagogy HCI educational requirement

Simulation

Dialogic feedback

Active learning [1]

Gamification attribute Action language

Creating meaningful projects [5] Case study real context projects [6]

Submit work

Immersion

Regular hand-in exercises and feedback [3]

Feedback

Environment

Problem and decision based cases [1] Promote realism by working on high-fidelity prototypes [3]

Challenge Employability tasks

Assessment

Maintaining individual and collaborative learning [5] Interactions between students and tutors [1] Promote reflection [3, 7]

Human interaction

Response to feedback

Control

Include unexpected experiences [7]

Game fiction

Inclusion of quizzes and games [8]

Rules/Goals

704

D. Carmichael et al.

3 Designing a Gamified Learning Experience The design for the learning resource began with an examination of the modules learning requirements together with the HCI educational requirements. The design process was iterative taking account of good practice from the literature review including the gamified attributes which were articulated into specific features. The work simulation narrative was based on the premise of the student working as an intern at an app design and development business. The narrative was presented via first a job advert, then emails and sms from the ‘boss’. In addition, there were decisions trees which presented students with real world decisions scenarios which they examine using interactive trees. Screen shots in Fig. 1 illustrate some of the interface elements.

Fig. 1. Work simulation

The Dialogic Feedback model was implemented via weekly feedback to students as illustrated in Fig. 2. The feedback was made up of verbal comments and a gamified marking rubric, which used language common to games such as ‘expert’ and ‘newbie’. The final stages of the design process involved ensuring that the HCI educational requirements were mapped to a game attribute [62] and implemented in at least one specifically gamified feature. A summary of the attributes to features is given in Table 3.

Gamification in a Learning Resource for the Study of HCI

Requirements

•Case Study •Job ApplicaƟon

•Scenarios •Feedback on Job ApplicaƟon

•Feedback on Scenarios •Storyboards

705

Design •Visual design •Feedback on storyboards

Prototyping

Start

Fig. 2. Work simulation dialogic feedback Table 3. HCI requirements mapped to gamification features HCI educational requirement

Game attribute

Gamified feature

Active learning [1]

Action language

UI controls

Creating meaningful projects [5] real context projects [6]

Immersion

Incremental tasks as SMS and emails

Regular hand-in exercises and feedback [3]

Environment

Real or virtual world tasks

Problem and decision based cases [1]

Conflict or Challenge

Decision trees

Promote realism by working on high-fidelity prototypes [3]

Assessment

Progress tracking using rubric descriptions

Maintaining individual and collaborative learning [5] Interactions between students and tutors [1]

Human interaction

Individual and group tasks

Promote reflection [3, 7]

Control

Player impact on game

Include unexpected experiences [7]

Game fiction

Narrative of work simulation

Inclusion of quizzes and games [8] Rules/Goals

Accessibility game & ethics quiz

4 Methodology The problem addressed in this research is the creation of an effective gamified learning resource which addresses key requirements for HCI education. In devising an evaluation methodology there are two distinct processes which need to be examined; the first is a holistic measure, of effectiveness for the gamified work simulation approach in general, the second is to examine the constituent parts of the resources. The holistic measurement of the work simulation can be inferred by general data such as student module lecture attendance and attainment in the exam. However, a constituent analysis should look at

706

D. Carmichael et al.

the HCI educational requirements matched to the attributes of the resource, otherwise it is impossible to determine which if any individual element is having a positive effect on learning. The aim of the research is stated in the following research question and hypotheses. RQ – Is there evidence of educational effectiveness for the work simulation? H1 – Module lecture attendance is improved for students experiencing the work simulation. H2 – Attainment in the end of module written exam is improved for students experiencing the work simulation. H3 – Students report favourably on one or all aspects of the work simulation. The module lecture attendance and examination attainment was examined for two cohorts, one of 97 students the other 104 students, both studying an HCI module. The lecture content, assessed by means of an exam, was the same for both cohorts. The control cohort experienced a ‘traditional’ delivery for the practical classes. The traditional practical class delivery provided students with a list of assessment tasks in digital form, responses to which were submitted at the end of the module’s twelve-week delivery period. The experimental cohort experienced the gamified work simulation during the practical classes. The gamified delivery required students to attend in order to receive feedback on the previous week’s task. This in practice meant that attendance at the practicals was mandatory for the experimental group. In summary, both cohorts were assessed by means of an exam, the control cohort submitted a coursework based on the practical tasks (at the end of the module) whilst the experimental cohort submitted the work simulation tasks (each week). All students were required to ‘swipe’ their student ID cards using a card reader in order to register their attendance. The attendance data was examined for the two cohorts over a 10-week period, in respect of student attendance at the lectures only. One cohort had to attend the practicals the other did not, therefore the analysis of attendance data for the practicals would be heavily slanted. However, both cohorts experienced the same block of 10 weekly lectures. The data was examined as a comparison of the percentage of the cohorts that attended. The attainment data in respect of the exam was examined for the two cohorts of students as explained above. Students in both cohorts were given a mark out 100 (operationally a percentage) for the tasks either presented in the traditional manner or the gamified work simulation. The data for each cohort was analysed to compare the distribution of ordinal grades; A, B, C, D, F. Such an approach yields data such as how many ‘A’s in one cohort as compared to another. In the second phase of the evaluation process two sets of structured interviews were conducted in order to determine which aspects of the work simulation, if any, were considered to be useful to students. There were a total of ten student volunteers. In the first set of interviews five students from the control group and five from the experimental group gave responses to question associated with the HCI requirements list. The requirements were operationalised as questions (given in Appendix 1). In the second phase five students from the experimental group were asked questions specific to the work simulation, which the control group did not experience. The questions

Gamification in a Learning Resource for the Study of HCI

707

were based upon the features of the work simulation. The participants were asked to ‘describe your experience of X’ and ‘was your experience of X positive, neutral or negative’. Where X was a specific feature for example the decision trees. The comments from the interviews were summarised by a three person panel.

5 Results There are two result sections; module attendance and attainment in the exam as well as student interviews. 5.1 Attendance and Attainment Data for two cohorts were examined; the first acted as a control and didn’t have access to the resource, the second was the experimental group which did have access. The attendance data was taken over 10 weeks and was in regard to the module’s lectures. The gamified resource, known as the ‘work simulation’ was used in the practical session; this meant that relative rates of attendance between the two cohorts at the practicals would not be meaningful. However, the lecture arrangements were the same for both cohorts. The results for the experimental group showed a mean attendance score of 52% whilst the control group mean was 33%. The data showing a higher rate of attendance for the experimental group is shown in data-set 1 and illustrated in Fig. 3.

90 80 70 60 50

Control

40

Experimental

30 20 10 0 1

2

3

4

5

6

7

8

9

10

Fig. 3. Attendance percentage s for two cohorts over 10 weeks

A two tailed t-test was carried out to test for statistical significance and was found to be P = .05%, which can be expressed as a one in twenty chance that the difference between the two sets of scores occurred by random factors. The result shows that the

708

D. Carmichael et al.

experimental group, which experienced the work simulation, had statistically significant higher rate of attendance at the lecture than the control group. The attainment data was taken from the results of a written closed book examination. Both the control and experimental groups had access to lectures and tutorials made up of the same materials. The control group spent the practical sessions working on the practical assessment in a self-organising way, whilst the experimental group experienced the work simulation. The examination questions differed between the two groups but in both papers the primary aim was to assess the understanding of the same knowledge content related to the lectures. The results were; a difference in the means between the control group of 62.5% and the experimental group of 66% which was statistically significant at P = .05% for a two tailed t-test. The results in data-set 1, which are illustrated in Fig. 4, show the results categorised in an A-F ordinal scale. 80% 70% 60% 50% Control

40%

Experimental

30% 20% 10% 0% A

B

C

D

F

Fig. 4. Examination attainment grades for two cohorts

In the experimental group 72% or students attained a B grade (60–70%) as opposed to the control group which attained 42% B grades. However, the control group had a greater number of students attaining an ‘A’ grade at 27% as opposed to the experimental group which had 20%. 5.2 Student Interviews In order to obtain a more in depth perspective on the student perceptions of the work simulation interviews with a group of student volunteers were conducted. The interview participants were asked about the HCI educational requirements (given above) using the questions set out in Appendix 1. In summary the findings in regard to the HCI requirements from both the experimental and control groups were:

Gamification in a Learning Resource for the Study of HCI

709

• Improvement for the experiment groupoo – – – –

Regular hand-in and feedback Problem and decision based Peer and Tutor interactions Unexpected experiences

• Little or no improvement for the experimental group – – – –

Active learning Promote realism Promote reflection Include quizzes and games

In addition, interview participants were asked to describe their experience of the various features of the work simulation stating whether they were positive, neutral or negative. In summary the key findings in regard to the gamified features from the experimental group were: • Simulation – Case study was broadly useful but the employability tasks were not • Dialogic feedback – Weekly submission was considered by some to be a burden but feedback and response to feedback were very positive • Gamified features – Largely positive concerning; ‘Immersion’ in terms of the weekly tasks, ‘game fiction’ in terms of the narrative, action language in terms of the user interface and ‘conflict or challenge’ in terms of the decision trees. – ‘Control’ and ‘assessments’ in the gamified resource was negatively reported upon. The results are discussed in the next section.

6 Discussion This study involved determining the requirements for HCI education and subsequently the creation of a resource based on a number of ideas from the pedagogy intended to match those requirements. The results of evaluation of the ‘work simulation’ resource are now interpreted in the light of the examined literature.

710

D. Carmichael et al.

6.1 Blended Learning The use of blended learning is thought to be useful for improving student attainment [13–16]. For example Lopez-Perez et al. [18] stated that blended learning could improve results as well as attitudes to learning. In this study, which examined two cohorts, an improvement in the mean percentage for the experimental group was found. A breakdown of the percentages, converted to grades A–F, (see Fig. 4) shows that there was a substantial increase in the number of B grades. However, this was not the case for A grades. The effect found in the results is not a uniform improvement in exam attainment with C grades becoming B grades and B grades becoming A grades. Rather Fig. 4 shows that effect is limited to C grades becoming B grades this suggests that the resource might be supporting ‘average’ students rather better than ‘excellent’ students. Although it should be noted that these are two separate cohorts, we can speculate that the increased number of B grades is due to improvements in base level understanding and that the effect is more pronounced with C grade students. Not all studies lend support to the use of blended learning for example Woltering et al. [20] suggested that such an approach was deleterious to attendance. Furthermore Traphagen et al. [21] found that 76% of students agreed or strongly agreed that they skipped classes when materials were available via a VLE. In this study attendance for some classes was mandatory and was designed that way in part to address such concerns. However, the results of an analysis between two cohorts for lecture attendance, which was not mandatory, showed better results for the experimental group than for the control group. It should be noted that the results showing improvements in exam attainment and attendance could be the effect of inherent differences between the two cohorts. Notwithstanding this caveat H1 (Attendance is improved for students experiencing the work simulation) and H2 (Attainment is improved for students experiencing the work simulation) are supported by these findings. Assuming that there is a positive effect as the results indicate, it is worth considering what elements of the work simulation might be contributing to that success. 6.2 HCI Requirements The central ethos for the design of the work simulation was to match the HCI educational requirements as suggested by the literature review. The HCI requirements, which were mapped to design features, were posed as questions to two groups of students one from the control and one from the experimental group. It was found that the work simulation resulted in more positive responses from the experimental group for; regular hand-in and feedback, problem/decision based learning, peer/tutor interactions and unexpected experiences. Results from students in the experimental group indicated that the weekly hand-in regime was found to be onerous but at the same time the weekly feedback was valued. This finding was echoed in responses to the peer and tutor interactions. The problem and decision based experiences were also positive and there was a clear distinction between the two groups concerning the increased opportunities provided for this by the work

Gamification in a Learning Resource for the Study of HCI

711

simulation. Because the work simulation was controlling the access to the tasks and resources as the narrative progressed, students had ‘unexpected experiences’ and, it is argued, felt more engaged as indicated in de-Marcos [61]. 6.3 Simulation and Dialogic Feedback The experimental group also responded to question concerning the designed features of the ‘work simulation’ grouped into; simulation, dialogic feedback and gamified aspects as set out in the literature review. The dialogic feedback, given on the work simulation coursework tasks was, overall, appreciated by students; they found benefit in the process of obtaining weekly feedback and being able to respond to the feedback in order to improve their exam attainment. The design for the feedback schedule, modelled on Beaumont et al. [27] and the feedback loop suggested by Werbach [60], was valued by students. It is possible that the process of work simulation coursework feedback helped to reinforce learning and led to improved performance in the exam although that can’t be stated definitively. In terms of the simulation aspect it was found that students appreciated the case study but not the employability aspect, which was, for example, the writing of a CV style letter in response to the case study. There is only tacit support for the simulation aspect; however there is an overlap with the simulation case study and the narrative of the game which we will turn to next. 6.4 Gamified Features The gamification aspects of the work simulation were laid out in Table 3 (above). The gamified elements [62] discussed next are highlighted in bold for clarity. Student responses to the interview questions indicated that the immersion and game fiction features, which delivered the narrative, were positively received. This finding supports the literature concerning the use of games style narratives [44, 45, 59]. It was also stated by some student respondents that the game fiction provided context for the weekly tasks. In addition, some students stated that the decision trees were “interesting and engaging” and the ‘action language’, these were calls to actions in the work simulation, were also thought to be a positive aspect. However, it was also found that students did not find the ‘control’ aspects of the work simulation to be positive. The idea of ‘control’ in the work simulation was the way that navigation and content viewing was controlled by the software. Specifically, each week the software revealed the next set of role play tasks. In the future, it would be possible to allow students’ greater levels of user control with a ‘peek forward’ feature aimed allowing students to explore the future content. According to Kofinas [59] assessments in a narrative can be ‘challenges’. In this case study the ‘assessments’ findings were somewhat negative. The student feedback was that the assessments which involved submitting the weekly practical tasks were somewhat onerous. In this study, however, the conflicts and challenges element was intended to be the decision trees which provided the students with an unfolding scenario which presented choice points which affected the next step. Some of these choice points

712

D. Carmichael et al.

were counter intuitive, so for instance, it would not always be optimal to select ‘ask the user’ it might be a better choice to ‘ask the client’. The results from the interviews suggested that students enjoyed exploring the decision trees. The game environment attribute referred to the mixture of real world and virtual world elements in the gamified resource. For instance, some elements were conducted in the real world such as presenting a storyboard whilst others where interactive tasks such as the decision trees. The findings from the interviews were a mixture of positive to neutral comments, with one student reporting that the resource had a good balance between the real and virtual world. The Human Interaction attribute referred to the interactions both between students and with students and tutors. The results of the interviews for this attribute were fairly positive. Students, to the extent they expressed a view, were positive about the interactions with teaching staff but in one instance concerned about workload balance amongst group members. The Rules & Goals gamification feature prompted both neutral and positive comments from students. In the work simulation there were several features that could be described as having ‘rules and goals’ however as the work simulation narrative which had rules and goals was analysed in other attributes, the focus here was on an accessibility empathy game and an ethics quiz. The empathy game was based upon the idea of helping interface designers to consider software from the point of view of a user with a cognitive disability. The game involved the students having to use a navigable web interface to find information whilst preventing the loss of life in an arcade style game in a window to the side. The intention was to emphasise the possible distractions that a user might be experiencing. The ethics quiz took students through a number of ethical scenarios in which they had to make a choice. The inclusion of these features is suggested by Bedwell [62] and also by the ‘novelty effect’ laid out in de-Marcos [61]. The interviews with students suggested that the game and quiz did have a ‘novelty effect’ but the educational purpose was not clear to all students. Overall the gamified features were fairly well received by students, at least as indicated by the interviews, and there is therefore partial evidence to support H3 (Students report favourably on one or all aspects of the work simulation). In reflecting on this study in the light of key ideas from the literature, for instance, the conflicting assertions about whether gamification would lead to increasing intrinsic motivation [53] or decreasing intrinsic motivation [54] over time, there weren’t any comments from interviewed students to support either position. However, the attendance at the lectures did decline over time, as was the case in Tsay et al. [43], which may suggest waning motivation.

7 Conclusions HCI is an important subject because it is concerned with improving usability and accessibility of interfaces to technology. As technology, particularly mobile apps, proliferates the mastery of the subject becomes ever more essential. However, the subject matter can prove problematic for ‘computing’ students as it has a relatively large, and often contentious, theory base. An analysis of the literature was carried out in order to garner the prerequisites for effective HCI education and resulted in a set of requirements. A consideration of the

Gamification in a Learning Resource for the Study of HCI

713

pedagogy produced a set of attributes to aid the design and creation of a HCI learning resource. In particular, blended learning, dialogic feedback, simulation and gamification ideas were used. A mapping of requirements to attributes was carried out. A design for the HCI learning experience was developed in relation to the mapping and taking into account the module learning outcomes. The design was used to create a gamified ‘work simulation’ based on the narrative of working as an intern at a design and development business and completing a set of weekly tasks. Dialogic feedback was implements via weekly feedback to students which comprised verbal comments and incorporating a gamified marking rubric. The evaluation of the HCI learning experience involved two processes; the holistic measures of the effectiveness of the gamified work simulation and examination of the constituent parts. In terms of the holistic measure; data was collected regarding attendance and exam attainment. Interviews were conducted to ascertain from students the usefulness of the work simulation’s constituent parts. The results indicated that the work simulation can be considered to have a statistically significant positive effect in terms of attendance and exam attainment particularly at the C to B grade boundaries. In relation to the individual components as revealed in interviews with students; the employability feature of simulation was not considered helpful whereas the dialogic feedback and gamification elements were deemed beneficial. These positive results for the gamified features are in line with Tsay et al. [43]. In the light of these findings there is tentative support for the research question concerning ‘evidence of educational effectiveness for the work simulation’ and the related hypotheses. In particular, the gamification elements that were useful to students were ‘game fiction’ and ‘action language’. Despite the merits of this study, which has utilised a methodology made up of quantitative and qualitative data, there are a number of limitations. The two cohort study results may be affected by inherent differences in these two cohorts rather than by the learning resource. The interviews were intended to gain insight into the constituent parts of the learning resource but didn’t take account of the inter play between features. In future work it would be useful to examine elements of the gamified features such as ‘game fiction’ in isolation rather than in concert with other features. It has been stated by Landers [40] that there is a lack of understanding of gamification in the literature. This study has demonstrated an innovative way to match HCI educational requirements to a gamified resource in a way that may have contributed to improvements in exam attainment and module attendance.

Appendix 1 Interview questions were derived by an academic using the HCI educational requirements. The questions were then revised by three academics. HCI educational requirement

Interview question – In respect of the practicals and assessment

Active learning [1]

Was your learning experience active (with things to do) or passive (listening)? (continued)

714

D. Carmichael et al.

(continued) HCI educational requirement

Interview question – In respect of the practicals and assessment

Creating meaningful projects [5] real context projects [6]

Did you create projects that were meaningful to you and that seemed realistic?

Regular hand-in exercises and feedback [3]

Did you feel that there was regular hand-in and feedback? Was it useful?

Problem and decision based cases [1]

Did you feel that there were times that you had to problem solve and make decisions? Was it useful?

Promote realism by working on high-fidelity prototypes [3]

Did you find working on the prototype useful?

Maintaining individual and collaborative learning [5]

Did you feel there were opportunities to interact with peers? Was this useful?

Interactions between students and tutors [1]

Did you feel there were opportunities to interact with tutors? Was it useful?

Promote reflection [3, 7]

Did you feel there were opportunities to reflect on your work, for example to thing about ways of improving your work? Was it useful?

Include unexpected experiences [7]

Did you feel there were unexpected aspects to the experience which were either useful or not?

Inclusion of quizzes and games [8]

Did you make use of the quiz and the game? Were they useful?

References 1. McCrickard, D.S., Chewar, C.M., Somervell, J.: Design, science, and engineering topics? Teaching HCI with a unified method. In: SIGCSE’04 (2004) 2. Kurosu, M., (ed.): Human-computer interaction theories, methods, and tools. In: 16th International Conference, HCI International 2014 (2014) 3. Aberg, J.: Challenges with teaching HCI early to computer students. In: ITiCSE’10 (2010) 4. Lazar, J., Gasen, J., Preece, J., Winograd, T.: New issues in teaching HCI: pinning a tail on a moving donkey. In: Abstracts of CHI 2002 (2002) 5. Preece, J., Abras, C.: The challenges of teaching HCI online: it’s mostly about creating community the UMBC online HCI class, vol. 1, pp. 2–5 (2003) 6. Hauser, S., Desjardins, A., Wakkary, R.: Design activism in the HCI classroom. In: CHI’13 Extended Abstracts on Human Factors in Computing Systems (2013) 7. Obrenovi´c, Ž.: Rethinking HCI education: teaching interactive computing concepts based on the experiential learning paradigm. Interactions 19(3), 66–70 (2012)

Gamification in a Learning Resource for the Study of HCI

715

8. Hill, J.M.D., Ray, C.K., Blair, J.R.S., Carver, C.A.: Puzzles and games: addressing different learning styles in teaching operating systems concepts. In: Proceedings of the Technical Symposium on Computer Science Education (SIGCSE 2003) (2003) 9. Garrison, D.R., Kanuka, H.: Blended learning: uncovering its transformative potential in higher education. Internet High. Educ. 7(2), 95–105 (2004) 10. Centre for Educational Research and Innovation (CERI), E-learning in tertiary education: Where do we stand? OCDE Publishing, Paris (2005) 11. Nunan, T., George, R., McCausland, H.: Rethinking the ways in which teaching and learning are supported: the flexible centre at the University of South Australia. J. High. Educ. Policy Manag. 22(1), 85–98 (2000) 12. Harding, A., Kaczynski, D., Wood, L.: Evaluation of blended learning: analysis of qualitative data (2005) 13. Boyle, T., Bradley, C., Chalk, P., Jones, R., Pickard, P.: Using blended learning to improve student success rates in learning to program. J. Educ. Media 28(2–3), 165–178 (2003) 14. O’Toole, J.M., Absalom, D.J.: The impact of blended learning on student outcomes: is there room on the horse for two? J. Educ. Media 28(2–3), 179–190 (2003) 15. Lim, D.H., Morris, M.L.: Learner and instructional factors influencing learning outcomes within a blended learning environment. Educ. Technol. Soc. 12(4), 282–293 (2009) 16. Mitchell, P., Forer, P.: Blended learning: the perceptions of first-year geography students. J. Geogr. High. Educ. 34(1), 77–89 (2010) 17. Gibbs, G.: Student engagement, the latest buzzword. Times High. Educ. 1, 193–210 (2014) 18. López-Pérez, M.V., Pérez-López, M.C., Rodríguez-Ariza, L.: Blended learning in higher education: students’ perceptions and their relation to outcomes. Comput. Educ. 56(3), 818– 826 (2011) 19. Donnelly, R.: Harmonizing technology with interaction in blended problem-based learning. Comput. Educ. 54, 350–359 (2010) 20. Woltering, V., Herrler, A., Spitzer, K., Spreckelsen, C.: Blended learning positively affects students’ satisfaction and the role of the tutor in the problem-based learning process: results of a mixed-method evaluation. Adv. Heal. Sci. Educ. 14, 725–738 (2009) 21. Traphagan, T., Kucsera, J.V., Kishi, K.: Impact of class lecture webcasting on attendance and learning. Educ. Technol. Res. Dev. 58(1), 19–37 (2010) 22. Biggs, J.B.: Teaching for Quality Learning at University: What the Student Does. McGrawHill Education (UK) (2011) 23. O’Donovan, B., Price, M., Rust, C.: Know what I mean? Enhancing student understanding of assessment standards and criteria. Teach. High. Educ. 9(3), 325–335 (2004) 24. Williams, J., Kane, D.: Exploring the NSS: assessment and feedback issues. In: The Higher Education Academy, p. 84 (2008) 25. Survey, N.: NUS student experience report (2008) 26. Woolfolk, A.N., Holquist, M., Emerson, C.: The Dialogic Imagination: Four Essays by M. M. Bakhtin, vol. 12 (1983) 27. Beaumont, C., O’Doherty, M., Shannon, L.: Reconceptualising assessment feedback: a key to improving student learning? Stud. High. Educ. 36(6), 671–687 (2011) 28. Yang, M., Carless, D.: The feedback triangle and the enhancement of dialogic feedback processes. Teach. High. Educ. 18(3), 1–13 (2012) 29. Avramenko, A.: Enhancing students’ employability through business simulation. Educ. Train. 54(5), 335–367 (2012) 30. Fripp, J.: A future for business simulations? J. Eur. Ind. Train. 21(4), 138–142 (1997) 31. Doyle, D.F., Brown, W.: Using a business simulation to teach applied skills – the benefits and the challenges of using student teams from multiple countries. J. Eur. Ind. Train. 24(6), 330–336 (2000)

716

D. Carmichael et al.

32. Musselwhite, C.: University executive education gets real: by Chris Musselwhite University executive education programs are incorporating business simulations into the classroom. T D 60(5) (2006) 33. Adobor, H., Daneshfar, A.: Management simulations: determining their effectiveness. J. Manag. Dev. 25(3), 151–168 (2006) 34. Chang, J., Lee, M., Ng, K.L., Moon, K.L.: Business simulation games: the Hong Kong experience. Simul. Gaming 34(3) (2003) 35. Anderson, P.H., Lawton, L.: Business simulations and cognitive learning: developments, desires, and future directions. Simul. Gaming 40(2), 193–216 (2008) 36. King, M., Newman, R.: Evaluating business simulation software: approach, tools and pedagogy. Horiz. Educ. Train. Iss J. Eur. Ind. Train. 17(6), 368–377 (2012) 37. Mintzberg, H.: Managers, Not MBAs: A Hard Look at the Soft Practice of Managing and Management Development. Berrett-Koehler Publishers (2004) 38. Moon, J.A.: A Handbook of Reflective and Experiential Learning: Theory and Practice. Psychology Press (2004) 39. Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From game design elements to gamefulness: defining gamification. In: 15th International Academic MindTrek Conference: Envisioning Future Media Environments (2011) 40. Landers, R.N.: Developing a theory of gamified learning: linking serious games and gamification of learning. Simul. Gaming 45(6), 752–768 (2014) 41. Landers, R.N., Callan, R.C.: Casual social games as serious games: the psychology of gamification in undergraduate education and employee training. In: Serious Games and Edutainment Application, pp. 399–423. Springer (2011) 42. Nicholson, S.: A user-centered theoretical framework for meaningful gamification. In: Games+Learning+Society, vol. 8 (2012) 43. Tsay, C.H.-H., Kofinas, A., Luo, J.: Enhancing student learning experience with technologymediated gamification: an empirical study. Comput. Educ. 121, 1–17 (2018) 44. Tay, L.: Employers: Look to gaming to motivate staff. itnews for Australian Business (2010) 45. Nicholson, S.: Exploring gamification techniques for classroom management. In: Games+Learning+Society, vol. 9 (2013) 46. Roediger, H.L., Karpicke, J.: Test-enhanced learning: taking memory tests improves long-term retention. Psychol. Sci. 17(3), 249–255 (2006) 47. Lonka, K., Ahola, K.: Activating instruction: How to foster study and thinking skills in higher education. Eur. J. Psychol. Educ. 10(4), 351 (1995) 48. Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work? - A literature review of empirical studies on gamification. In: 2014 47th Hawaii International Conference on System Sciences (HICSS), pp. 3025–3034 (2014) 49. Hamari, J., Koivisto, J.: Why do people use gamification services? Int. J. Inf. Manage. 35(4), 419–431 (2015) 50. Domínguez, A., Saenz-de-Navarrete, J., De-Marcos, L., Fernández-Sanz, L., Pagés, C., MartíNez-HerráIz, J.-J.: Gamifying learning experiences: practical implications and outcomes. Comput. Educ. 63, 380–392 (2013) 51. Landers, R.N., Armstrong, M.B.: Enhancing instructional outcomes with gamification: an empirical test of the Technology-Enhanced Training Effectiveness Model. Comput. Human Behav. 71, 499–507 (2017) 52. Denny, P.: The effect of virtual achievements on student engagement. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 763–772 (2013) 53. Huotari, K., Hamari, J.: Defining gamification: a service marketing perspective. In: Proceeding of the 16th International Academic MindTrek Conference, pp. 17–22 (2012)

Gamification in a Learning Resource for the Study of HCI

717

54. Hanus, M.D., Fox, J.: Assessing the effects of gamification in the classroom: a longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Comput. Educ. 80, 152–161 (2015) 55. Baeten, M., Kyndt, E., Struyven, K., Dochy, F.: Using student-centred learning environments to stimulate deep approaches to learning: factors encouraging or discouraging their effectiveness. Educ. Res. Rev. 5(3), 243–260 (2010) 56. Deci, E.L., Koestner, R., Ryan, R.M.: Extrinsic rewards and intrinsic motivation in education: reconsidered once again. Rev. Educ. Res. 71(1), 1–27 (2001) 57. Iosup, A., Epema, D.: An experience report on using gamification in technical higher education. In: Proceedings of the 45th ACM Technical Symposium on Computer Science Education, pp. 27–32 (2014) 58. Banfield, J., Wilkerson, B.: Increasing student intrinsic motivation and self-efficacy through gamification pedagogy. Contemp. Issues Educ. Res. 7(4), 291–298 (2014) 59. Kofinas, A.: Managing the sublime aesthetic when communicating an assessment regime: The Burkean Pendulum. Manag. Learn. 49(2), 204–221 (2018) 60. Werbach, K., Hunter, D.: For the Win: How Game Thinking can Revolutionize your Business. Wharton Digital Press, USA (2012) 61. de-Marcos, L., Garcia-Lopez, E., Garcia-Cabot, A.: On the effectiveness of game-like and social approaches in learning: comparing educational gaming, gamification & social networking. Comput. Educ. 95, 99–113 (2016) 62. Bedwell, W.L., Pavlas, D., Heyne, K., Lazzara, E.H., Salas, E.: Toward a taxonomy linking game attributes to learning: an empirical study. Simul. Gaming 43(6), 729–760 (2012)

Machine Learning Control for Mobile Robot by Approximation Extremals by Symbolic Regression Askhat Diveev(B) Federal Research Center “Computer Science and Control” of Russian Academy of Sciences, Moscow, Russia Abstract. The control system synthesis problem is considered for mobile robot. It is necessary to find a control as function of state space vector that supplies achievement of terminal position from some compact set of initial conditions with optimal value of quality criterion. For solution of this problem symbolic regression is used. This allows to find a structure of control function mathematical expression. General properties of symbolic regression are presented. Search of mathematical expression structure for control system is performed on the space of codes, therefore here a specific forms of genetic algorithms are used. They are differed crossover and mutation operations in depend on form of mathematical expression codding. At the searching optimal mathematical expression for control system by symbolic regression methods two approaches can be applied two approaches, as in artificial neural network technology, with teacher and without one. For obtaining a training set it is necessary to solve the optimal control problem some times for set of initial conditions. It is shown, that for decreasing the search space the principle of small variations of basic solution can be used and a basic solution should be set as closed to optimal one intuitively. In this case crossover and mutation operations will be performed the same on sets of variation vectors independently on symbolic regression method. In a computational experiment the control system synthesis problem for mobile robot is considered, that must be move from an area of initial conditions to the set terminal position on the plane with phase constraints. Keywords: Control synthesis · Machine learning control · Optimal control · Extremals · Evolutionary algorithm · Symbolic regression

1

Introduction

Machine learning of control is a novel paradigm in the field of control [1]. The main difference this new approach to creating of control system consists of application of symbolic regression to a search of control function structure. Before control system structure was set by an engineer or researcher with accuracy to parameter values, after that values of these parameters were found by a numerical method. Possibility to find structure of control system by computer is appeared c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 718–736, 2022. https://doi.org/10.1007/978-3-030-80119-9_46

Machine Learning Control

719

after creating genetic programming (GP) [2]. Initially symbolic regression was intended for solution of the automatic writing programs problem. Any control system can be written in the set of formulas. Any formula can be program. Therefore, if a numerical method can write a program, then it can write formulas for control systems. GP codes a possible solution in the form of computational tree graph. Node of the graph are connected with computational operations, and leaves of the tree graph are connected with arguments of mathematical expression. For search of optimal solution GP uses special genetic algorithm (GA). Crossover operation in this GA is performed as an exchange of branches of selected trees. Evolutionary search of optimal solution is called learning. The term “learning” came from the artificial neural networks (ANN) technology [3]. Note, that ANN always has a certain structure and a large number of unknown parameters. Learning of ANN is a search of parameter values. A structure of ANN can change by increasing or decreasing a number of ANN layers or by change activation functions, but at the learning ANN always values of parameters are searched with help of optimization algorithm. The term “learning” is used instead of the term “optimization” because for ANN optimal values of parameters don’t need. It’s necessary to find such parameters that the cost function had satisfactory value. In machine learning of control (MLC) a mathematical expression of control system is searched by symbolic regression method. GP is one of the symbolic regression method. GP codes have different lengths that are changed after crossover operations. This is main disadvantage of GP. Now there are more than ten symbolic regression methods [4]. In the work [5] GP was used for synthesis control system. In that work standard units of transfer functions was used as code GP elements. In the work [6] other symbolic regression method was used for control synthesis problem, the network operator method, that codes a mathematical expression in the form of oriented graph. In that work the mathematical expression for a control function was searched on the base of elementary function sets with one or two arguments. For extend a set of elementary functions parameter vector was introduced as an argument of mathematical expression. Optimal values of these parameters were looked for together with searching of mathematical expression structure by the same of genetic algorithm but with other crossover operation. In result, crossover operations for structures of mathematical expression and for parameter vectors four new possible solutions are obtained. In that work the general synthesis problem of control was considered, when one control function is searched for a set of initial conditions. Value of goal function is a sum of values computed for each initial condition. There also for the first time the principle of small variations of basic solution was used, that was formulated later as a general approach for nonnumerical optimization problems [7]. In the works [8–12] control synthesis problems for mobile robots or group of robots were solved by different symbolic regression methods. In the work [13] solutions of control problems by symbolic regression method are considered as reinforcement learning [14], therefore symbolic regression methods can be belonged to kind of ANN, or to development of ANN.

720

A. Diveev

In the most cases at applications of symbolic regression to solution of control problems training data aren’t use. A symbolic regression method looks for optimal structure of control system in the form of mathematical expression code by a goal function without examples of solutions for this problem. Main specific of search optimal solution in the space of codes consists of that this space has no numerical metrics of distance between two possible solutions. This means that for any possible solution it’s can’t determinate a neighborhood. Therefore there are no characteristics of local optimum. If a symbolic regression method has found optimal solution then this solution is the best one from all checked solutions, but it not the best from everyone. More that, symbolic regression method can’t point how near a value of goal function for found solution to the optimal one. In the work [15] to solve the control system synthesis problem an approximation of an extremals’ set is used by a symbolic regression method. Here the extremal is a numerical solution of the optimal control problem for one initial condition. The extremals’ set is a set of numerical solutions of optimal control problem for the given set of initial conditions. This work continues researches on this approach. The control system synthesis problem with phase constraints is considered. It is necessary to move a control object from some area of state space to the given terminal point. Firstly the optimal control problem is solved for the set for initial conditions selected from the given area. Found solutions are control functions of time. For each solution some points in determined moments of time are stored as a value of time and a state space vector. Secondly the approximation problem of these stored points is solved by symbolic regression method. For this purpose the multidimensional function of control is searched, that is inserted in right side of differential equations of the mathematical model of control object. The control function must provide such solutions of the differential equations’ system for the set of initial conditions that they passed near stored points. In this work for solution of the optimal control problem evolutionary gray wolf optimizer algorithm GWO-algorithm is used [16, 17]. For the approximation extremals’ set a symbolic regression method the variational Cartesian genetic programming [4, 18] is applied. In application part the optimal control synthesis problem for mobile robot, which has to move on the plane from some area to the terminal point with circumventing obstacles is considered.

2

Statement of the Control System General Synthesis Problem

A mathematical model of the control object is given x˙ = f (x, u),

(1)

where x is a state space vector, x ∈ Rn , u is a control vector, u ∈ U ⊆ Rm , U is bounded closed set, m ≤ n.

Machine Learning Control

721

A domain of initial conditions is X0 ⊆ Rn ,

(2)

x(tf (x0 )) = xf ∈ Rn ,

(3)

Terminal conditions are

where tf (x0 ) is a time of archiving the terminal conditions xf from initial condition x0 ∈ X0 . ∀x0 ∈ X0 , tf (x0 ) is limited, tf (x0 ) ≤ t+ . A quality criterion is n

   tf(x0 )  f0 (x(t), u(t))dtdx1 . . . dxn → min , J = ... u∈U 0 X0

(4)

It is required to find a control as a function of state space vector u = h(x) ∈ U

(5)

such that any partial solution of the system of differential equations x˙ = f (x, h(x))

(6)

from the domain (2) achieves the terminal conditions (3) and provides an optimal value of the criterion (4). In order to create a numerical computational algorithm for solving this problem the problem statement. is reformulated. The initial conditions (2) is replaced with a finite set of initial conditions, multiple integrals (4) are replaced with a sum of integrals for each initial condition from a finite set and procedure for determination of terminal time. ˜ 0 = {x0,1 , . . . , x0,M }, X J˜ =

M  j=1



where tf (x0,j ) =

(7)

(x0,j ) tf 

f0 (x(t), u(t))dt → min, u∈U

(8)

0

t, ift < t+ and xf − x(t, x0,j ) ≤ ε , t+ , otherwise

(9)

ε is a small positive value, x(t, x0,j ) is a solution of the differential equation ˜ 0 , 1 ≤ j ≤ M , t+ is a given limit system (6) with the initial condition x0,j ∈ X time of control process. TThe control general synthesis problem was state in sixty years of the last century [19]. Now it known some analytical solutions of partial problems. General approach to solve this problem isn’t created. There aren’t a numerical method

722

A. Diveev

for the general synthesis problem (1), (7), (3), (8), (9), except symbolic regression methods. There is a technical approach for solution of this problem, when specialist on control defines control function with the precise to the constant parameters, and then these parameters are determined by an optimization algorithm. This approach also includes a solution of the synthesis problem on base of regulators, for example widely used widely used in control system proportional, integral and differential controllers (PID-controllers). There are half-analytical methods for solution a the control general synthesis problem. These are backstepping method [20] and analytical design of aggregated controller [21]. These methods direct to provide stable, and they don’t consider any quality criterion in general form (8). Solutions by these methods are connected with concrete form of control object mathematical model. They aren’t fit for the control system general synthesis problem. For solution of this problem (1), (7), (3), (8), (9), initially the optimal control problem is solved for each initial condition from the set (7) x(0) = x0,k ∈ X0 , k ∈ {1 . . . , M },

(10)

and one integral functional tf  (x0,j )

f0 (x(t), u(t))dt → min,

J1 =

(11)

u∈U

0

In the result a control as function of time is found u = v(t, x0,k ), k ∈ {1 . . . , M }.

(12)

For each control (12) a solution of the system (1) is found x(t, x0,k ), k ∈ {1 . . . , M }.

(13)

To obtain training data points in specified time moments are stored T = (tk,j , x(tk,j , x0,k ) : j = 1, . . . , sk , k = 1, . . . , M ).

(14)

Then the approximation problem of training data (14) is solved by symbolic regression method. For this purpose the control in form (17) is found on criterion Ja =

sk M   k=1 j=1

(˜ x(tj , x0,k ) − x(tk,j , x0,k ))2 → min, u∈U

(15)

˜ (tj , x0,k ) is a solution of the system (6) for initial condition x0,k ∈ X0 where x in the moment tj , j ∈ {1, . . . sk }, k = 1. . . . , M .

Machine Learning Control

3

723

Solving the Optimal Control Problem

For solution of the optimal control problem on the base direct approach let’s transform this problem to the nonlinear programming problem. An interval Δt on the time axis is set. For a limited time t+ it is obtained not more than

+ t (16) K= Δt intervals and K +1 parameters for one component of control vector on boundaries of the intervals. The control function is described as a piecewise linear one in each time interval subject to restrictions on control ⎧ + ˜ i > u+ ⎨ ui , if u i − ui (t) = ui , if u (17) ˜ i < u− i , ⎩ u ˜i , otherwise where

qr(i,j)+1 − qr(i,j) (t − (j − 1)Δt) + qr(i,j) , (18) Δt r(i, j) = j + (i − 1)(K + 1), i = 1, . . . , m, j = 1, . . . , K. The solution of the optimal control problem is a vector of (K + 1)m parameters (19) qi = [q1 . . . qm(K+1) ]T . u ˜i =

In most cases, it is difficult to determine the unimodality property of the objective function on the parameter space, so it is better to use the evolutionary algorithm to find a solution. All evolutionary algorithms include the following steps [17]: – – – –

generation a set of possible solutions which is called a population; calculation of quality criterion value for each possible solution; variation or evolution of all or some subset of possible solutions; substitution of population elements: if a new possible solution is better than the old one before evolution, then substitution of the old possible solution with the new one.

One generation includes a given number of evolution transformations. After performing all the generations, the best solution is defined on value of a target function. This the best solution is considered as the solution of the optimization problem. In all evolutionary algorithms main parameters are a given number of generations, number of elements in an initial population, and number of evolution transformations in one generation. Today it is known more than two hundreds evolutionary algorithms. Almost all they have unusual for mathematicians names of animals or a nature phenomenons that inspired algorithm’s creators. The most well-known of these algorithms is genetic algorithm.

724

A. Diveev

Evolutionary algorithms are differed by procedure of evolution performing on the third stage. An evolution is a small change of possible solution after that values of goal function for some part of possible solutions is changed on a small value. At the performing evolution, information about values of quality criterion for the current better and other possible solutions is used. Evolutionary algorithms work better in complex optimization problem when they use for evolution various information about values of quality criterion for other possible solutions in population. The genetic algorithm uses information about the values of the quality criterion for the best and the possible solutions selected for crossing only to calculate the probability of performing the crossing operation. For complex optimization problems, it is more advisable to use other evolutionary algorithms that work better than the genetic algorithm. Here to numerical solve the optimal control problem Grey Wolf Optimizer algorithm is applied (GWO) [16, 17]. GWO algorithm uses for evolution transformation of possible solutions three best current possible solutions in population. In this work GWO algorithm was modified. Now a number of the best current possible solutions used for evolution is a parameter of algorithm, that can be set by researcher. GWO algorithm has the following steps: 1. Creating initial population by random generation of possible solutions

where

Q = {q1 , . . . , qH },

(20)

r = ξ(qi+ − qi− ) + qi− , qi,j

(21)

r = 1, . . . , H, j = 1, . . . , K + 1, ξ is a random number from the interval [0; 1], qi− and qi+ are the minimum and the maximum values of parameters of the control function (17) respectively, i = 1, . . . , m. 2. Calculating values of the quality criterion (11) for each possible solution F = {f1 = J1 (q1 ), . . . , fH = J1 (qH )},

(22)

where J1 (qr ) is the value of goal function (11) of the solution of system (1) with control function (17) obtained using vector qr . The current value of generations iterator is set to g = 1. 3. Setting the number N of the best possible solutions I = {r1 , . . . , rN },

(23)

fr1 ≤ . . . ≤ frN ≤ ∀fr , r ∈ {1, . . . , H} \ I.

(24)

where 4. A possible solution is random selected from the population r = ξ(H) where ξ(H) is a random integer number from 1 to H.

(25)

Machine Learning Control

725

Calculating the following value A=

N m(K+1)   i=1

j=1

qjri − L(2ξ − 1)|2ξqjri − qjr |,

(26)

where

2g . P Performing an evolution for the selected possible solution qr L=2−

q˜ir =

A , i = 1, . . . , m(K + 1). N

(27)

(28)

˜ r . If J1 (˜ Calculating the value of criterion (11) for a new vector q qr ) < fr , r r ˜ , then the old vector q is replaced on the new vector q ˜ r , fr ← J1 (˜ qr ). q←q

(29)

Repeating the step 4 R ≥ H times. 5. Increasing the value of generation count g ←g+1

(30)

and go to the step 3. Repeating steps 3–5 P times. After all generations, when g > P , the possible solution with the best value of quality criterion is considered as an optimal solution of the problem. The control function (17) constructed on the base of the optimal parameter vector is the solution of the optimal control problem. To apply the modified GWO algorithm it is necessary to define the following parameters: H is a number of possible solutions in a population; P is a number of generations; R is a number of evolution transformation in one generation; N is a number of possible solutions with best values of goal function for performing evolution tranformations. Experience of using GWO algorithm showed, that this algorithm suit well for direct solving the optimal control problem.

4

Variation Cartesian Genetic Programming

For solution of the approximation problem the symbolic regression method variation Cartesian programming is used. To present a mathematical expression in the form of Cartesian genetic programming code the sets of arguments and elementary functions are given. The set of arguments contains variables and parameters (31) A0 = {x1 , . . . , xn , c1 . . . , cp }, where xi is a variable, i = 1, . . . , n, cj is a parameter j = 1, . . . , p.

726

A. Diveev

The set of elementary functions contains functions with one, two or three arguments F = {f1,1 (z), . . . , f1.W (z), f2,1 (z1 , z2 ), . . . , f2,V (z1 , z2 ), f3,1 (z1 , z2 , z3 ), . . . , f3,S (z1 , z2 , z3 )},

(32)

where f1,i (z) is a function with one argument, i = 1, . . . , W , f2,j (z1 , z2 ) is a function with two arguments, j = 1, . . . , V , f3,k (z1 , z2 , z3 ) is a function with three arguments. k = 1, . . . , S. Cartesian genetic programming code of mathematical expression is a set of calls of functions. Each call of function is coded as an integer vector with four component (33) v = [v1 v2 v3 v4 ]T , where the first component, v1 is a number of element from the set of functions (32), others components v2 , v3 , v4 are the number elements from the set of arguments (31). Results of calculated functions are added to the set of arguments. For example call of function y1 = f3,α (xβ , cγ , xσ ) is codded as

v1 = [W + V + α β n + γ σ]T .

After calculation value of y is added to the set of arguments (31) A = {x1 , . . . , xn , c1 , . . . , cp , y1 }. If the function uses less than three arguments, then the other arguments’ numbers can be anyone. For example call of function y2 = f2,τ (y1 , xδ ) is codded as

v2 = [W + τ n + p + 1 δ 1]T ,

where the last number of argument 1 isn’t use. Consider an example. Let us code a mathematical expression y = x1 exp(−c1 x2 ) cos(c2 x2 + x3 ),

(34)

To code this mathematical expression (34) the following sets are used A = {x1 , x2 , x3 , c1 , c2 },

(35)

F = {f1,1 (z) = z, f1,2 (z) = −z, f1,3 (z) = exp(z), f1,4 (z) = cos(z), f2,1 (z1 , z2 ) = z1 + z2 , f2,2 (zz , z2 ) = z1 z2 }.

(36)

Machine Learning Control

727

Write the mathematical expression (34) as calls of functions from (36) y1 y2 y3 y4 y5 y6 y7 y8

= c1 x2 , = −y1 , = exp(y2 ), = c2 x2 , = y4 + x3 , = cos(y5 ), = y3 y 6 , = x1 y7 .

In result the following Cartesian programming code is received ⎡ ⎤ 62365 4 6 6 ⎢ 4 6 7 5 9 10 8 1 ⎥ ⎥ C=⎢ ⎣ 2 2 4 2 5 8 11 12 ⎦ . 1 3 5 6 7 9 10 11

(37)

(38)

Variation Cartesian genetic programming codes small variation of Cartesian genetic programming code. For writing of small variation code integer vector with three components is used w = [w1 w2 w3 ]T .

(39)

where w1 is a the column number in Cartesian genetic programming code (38), w2 is the line number of code (38), w3 is a new value of code element. For example a variation vector has the following value w = [2 1 3]T . Then a new code is received

⎡

⎤ 63365 4 6 6 ⎢ 4 6 7 5 9 10 8 1 ⎥ ⎥ w◦C=⎢ ⎣ 2 2 4 2 5 8 11 12 ⎦ . 1 3 5 6 7 9 10 11

(40)

This new code corresponds the following mathematical expression y˜ = x1 exp(exp(c1 x2 )) cos(c2 x2 + x3 ),

(41)

According to the principle of small variations of basic solution [7] one basic solution is codded are codded in the form of Cartesian genetic programming code and others possible solutions in the form of the sets of variation vectors W = (w1 , . . . , wd ),

(42)

where d ia a given a number of elementary variations for one possible solution. Crossover and mutation operations are performed on the sets of variation vectors. For crossover two possible solutions are selected

728

A. Diveev

Wα = (wα,1 , . . . , wα,d ), Wβ = (wβ,1 , . . . , wβ,d ).

(43)

Then a crossover point ks ∈ {1, . . . , d} is determined randomly and tails of possible solution WH+1 = (wα,1 , . . . , wα,ks −1 , wβ,ks , . . . , wβ,d ), WH+2 = (wβ,1 , . . . , wβ,ks −1 , wα,ks , . . . , wα,d ),

(44)

where H is a number of possible solutions in a population.

5

Computation Experiment

Consider the solution of the control system general synthesis problem for a mobile robot. A mathematical model of the control object is given in the following form [22] x˙ 1 = 0.5(u1 + u2 ) cos(x3 ), x˙ 2 = 0.5(u1 + u2 ) sin(x3 ), x˙ 3 = 0.5(u1 − u2 ),

(45)

where the control u = [u1 u2 ]T is constrained + u− i ≤ ui ≤ ui : i = 1, 2.

(46)

+ u− i = −10, ui = 10, i = 1, 2. Domain of initial conditions is + X0 = {x− i ≤ xi ≤ xi : i = 1, 2, 3}.

(47)

− − + + + where x− 1 = 9, x2 = 9, x3 = −π/4, x1 = 11, x2 = 11, x3 = π/4. Terminal state is xf = [0 0 0]T .

(48)

Quality criterion shows the time of reaching the terminal state (48) +

+

+

x1 x2 x3 tf(x

0

)⎛

⎝1 + a

J= − − x− 1 x2 x3

r 

⎞ ϑ(φj (x))⎠ dtdx1 dx2 dx3

j=1

0 +

+

+

x1 x2 x3

tf (x0 )dx1 dx2 dx3

= − − x− 1 x2 x3 +

+a

+

+

0

x1 x2 x3 tf(x ) r

x− 1

x− 2

x− 3

0

j=1

ϑ(φj (x))dtdx1 dx2 dx3 → min , u=h(x)

(49)

Machine Learning Control

729

where a is a penalty coefficient, a = 3, r is a number of phase constraints, ϑ(A) is Heaviside step function  1, ifA > 0 ϑ(A) = , (50) 0, otherwise φj (x) is a phase constraints φj (x) = Rj −



(x1 − x1,j )2 + (x2 − x2,j )2 ,

(51)

j = 1, 2, R1 = 2.5, R2 = 2.5, x1,1 = 5, x1,2 = 5, x2,1 = 8, x2,2 = 2. According to the proposed method, initially the optimal control problem should be solved for some set of initial conditions. Let us replace the domain (47) with the set of M = 24 initial conditions ˜ 0 ={x0,1 = [11 11 π/4]T , X x0,2 = [10 11 π/4]T , x0,3 = [9 11 π/4]T , x0,4 = [9 10 π/4]T , x0,5 = [9 9 π/4]T . x0,6 = [10 9 π/4]T , x0,7 = [11 9 π/4]T , x0,8 = [10 10 π/4]T , x0,9 = [11 11 0]T , x0,10 = [10 11 0]T , x0,11 = [5 11 0]T , x0,12 = [9 10 0]T , x0,13 = [9 9 0]T , x0,14 = [10 9 0]T , x0,15 = [11 9 0]T , x0,16 = [11 10 0]T x0,17 = [11 11 − π/4]T , x0,18 = [10 11 − π/4]T , x0,19 = [9 11 − π/4]T , x0,28 = [9 10 − π/4]T , x0,21 = [9 9 − π4/]T , x0,22 = [10 9 − π/4]T , x0,23 = [11 9 − π/4]T , x0,24 = [11 10 − π/4]T }.

(52)

730

A. Diveev

The optimal control problem is solved for each initial condition from the set (52) with the following quality criterion J3 = tf (x0.i )+ max{|xfi − xi (tf (x0.i ))| : i = 1, 2, 3} → min, u(t)

(53)

where tf (x)0,j , j = 1, . . . , 24, is defined in (9) considering t+ = 2.1s, ε0 = 0.01. At the solving the optimal control problem (45)–(53) the time interval was set to Δt = 0.21. Thus the dimension of a vector of parameters is m(K +1) = 20, q = [q1 . . . q20 ]T . The search for solutions of the optimal control problems was performed using GWO algorithm with the following parameters: H = 512, P = 8192, R = 512, N = 8. Found 24 optimal trajectories or in other words extremals onto the plane {x1 , x2 } are shown in Fig. 1, 2 and 3. Table 1 contains values of the quality criterion for each initial condition from the set (52).

Fig. 1. Optimal Trajectories on the Plane {x1 , x2 } for Initial Conditions x0,1 , . . . , x0,8

Machine Learning Control

731

Fig. 2. Optimal Trajectories on the Plane {x1 , x2 } for Initial Conditions x0,9 , . . . , x0,16

Fig. 3. Optimal Trajectories on the Plane {x1 , x2 } for Initial Conditions x0,17 , . . . , x0,24

732

A. Diveev Table 1. Optimal Values of the Quality Criterion for given Initial Conditions. No Initial conditions x0 J3 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

[11 11 π/4]T [10 11 π/4]T [9 11 π/4]T [9 10 π/4]T [9 9 π/4]T [10 9 π/4]T [11 9 π/4]T [11 10 π/4]T [11 11 0]T [10 11 0]T [9 11 0]T [9 10 0]T [9 9 0]T [10 9 0]T [11 9 0]T [11 10 0]T [11 11 − π/4]T [10 11 − π/4]T [9 11 − π/4]T [9 10 − π/4]T [9 9 − π/4]T [10 9 − π/4]T [11 9 − π/4]T [11 10 − π/4]T

1.7779 1.7502 1.7402 1.6302 1.5102 1.5501 1.6602 1.6902 1.8703 1.8604 1.8601 1.8103 1.6503 1.5402 1.6900 1.8018 1.9701 1.9401 1.9772 1.8702 1.8502 1.7801 1.8102 1.9003

According to the proposed method at the second stage found optimal trajectories should be approximated by symbolic regression method. In the considered computational experiment for this purpose the variation Cartesian genetic programming was used. For each optimal trajectory, state vector values were stored at time interval 0.42 s. So each extremal trajectory was presented as five or six points. At the approximation process the functional (15) was used. At solution of approximation problem was found a problem, that approximating control function doesn’t sens phase constraints and badly sens terminal conditions. Therefore the penalty functions considering violations of phase constraints and accuracy of terminal conditions achievement were added to approximation functional (15). Total functional for approximation had the following form

Machine Learning Control sk M  

J˜a =

733

(˜ x(tj , x0,k ) − x(tk,j , x0,k ))2

k=1 j=1

+a

M  k=1

tf  (x0,k )

r 

0

ϑ(φj (x))dt

j=1

 M    n f  (x − xi (tf (x0,k ), x0,k )) → min, +b i i=1

k=1

u∈U

(54)

where a and b are a penalty coefficients, a = 3, b = 2, tf (x0,k ) is determined by formula (9). The approximation of found extremals by variation Cartesian genetic programming the following control function is received ⎧ + ˜ i ≥ u+ ⎨ ui , if u i − (55) ui = ui , if u ˜ i ≤ u+ i , ⎩ u ˜i , otherwise where

u ˜1 = (c1 (xf1 − x − 1) + (xf2 − x2 )2 + A)3  u ˜2 =

where

+ c3 + arctan(xf1 − x1 ), (xf2 − x2 )2 , if2c1 (xf1 − x1 ) + c2 > 0 , (2c1 (xf1 − x1 ) + c2 )2 , otherwise

(56) (57)

A = c3 (xf3 − x3 ) + (xf2 − x2 )2 × (arctan(xf1 − x1 ) + η1 (x3 , (xf2 − x2 ), (xf3 − x3 ))),  β, ifα > 0 , η1 (α, β, γ) = γ, otherwise

c1 = 13.62207, c2 = 7.21387, c3 = 5.85938. The Fig. 4 shows trajectories on the plane {x1 , x2 } obtained by the object movement using the stabilization system (56)–(57) from eight first initial conditions x0,1 = [11 11 π/4]T , x0,2 = [10 11 π/4]T , x0,3 = [9 11 π/4]T , x0,4 = [9 10 π/4]T , x0,5 = [9 9 π/4]T , x0,6 = [10 9 π/4]T , x0,7 = [11 9 π/4]T , x0,8 = [11 10 π/4]T .

734

A. Diveev

Fig. 4. Trajectories on the Plane {x1 , x2 } of 8 Obtained by the Object Movement using Found Stabilization System

To confirm the results of learning let us consider initial conditions, that was not included in the training set of extremals. Let it be the following initial conditions (58) x0,25 = [10 10 0]T . Trajectories on the plane {x1 , x2 } obtained by solution of the optimal control problem, and by control function (56), (57) are presented in the Fig. 5.

Fig. 5. Trajectories on the Plane {x1 , x2 }. Trajectory Obtained by Solution of the Optimal Control Problem (Blue), By Synthesized Control Function (Black)

Machine Learning Control

735

Conclusion Application of machine learning control by symbolic regression methods to solving the problem of the control system general synthesis was considered. To obtain a training set initially the optimal control problem was solved for set of initial conditions by evolutionary Grey Wolf Optimizer algorithm. On the second stage a control function was obtained by approximating the training set of optimal trajectories by variation Cartesian genetic programming. In computational experiment the control system synthesis problem for one mobile robot was solved. The robot must move from one point to other on a plane with obstacles. The training set contained twenty-four optimal trajectories. After learning of control, the obtained control function was checked for initial conditions not included in the training set. Experiments showed that the obtained control function finds a control providing a moving mobile robot from initial conditions to terminal state without violation of phase constraints with almost optimal value of quality criterion. Acknowledgment. The theoretical parts of work, Sects. 1 to 4 were performed at support from the Russian Science Foundation (project No 19-11-00258). Experimental and computational parts of the work, Sections 5, were performed at support from the Russian Foundation for Basic Research (project No. 18-29-03061-mk).

References 1. Duriez, T., Brunton, S.L., Noack, B.R.: Machine Learning Control - Taming Nonlinear Dynamics and Turbulence, Fluid Mechanics and Its Applications, vol. 116. Springer, Cham (2017). 212 p. https://doi.org/10.1007/978-3-319-40624-4 2. Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge (1992). 819 p 3. Goodfellow, I., Benjio, Y., Courville, A.: Deep Learning. The MIT Press, Cambridge (2016). 788 p. ISBN-978-1-491-937990 4. Diveev, A.I.: Numerical methods for solution of the control synthesis problem. Chislennye metody reshenia zadachi sinteza upravlenia. Moscow RUDN Publishing (2019). 192 p. (in Russian) 5. Koza, J.R., Keane, M.A., Mydlowec J.Yu.W., Bennett III, F.H.: Automatic synthesis of both the control law and parameters for a controller for a three-lag plant with five-second delay using genetic programming and simulation techniques. In: Proceedings of the American Control Conference Chicago, Illinois, pp. 453–458, June 2000 6. Diveev, A.I., Sofronova, E.A.: Numerical method of network operator for multiobjective synthesis of optimal control system. In: 2009 IEEE International Conference on Control and Automation, Christchurch, New Zealand, 9–11 December 2009, pp. 701–708. https://doi.org/10.1109/ICCA.2009.5410619 7. Diveev, A.I.: Small variations of basic solution method for non-numerical optimization. IFAC-PapersOnLine 48(25), 28–33 (2015). https://doi.org/10.1016/j.ifacol. 2015.11.054

736

A. Diveev

8. Diveev, A.I., Ibadulla, S.I., Konyrbaev, N.B., Shmalko, E.Y.: Variational analytic programming for synthesis of optimal control for flying robot. IFAC-PapersOnLine 48(19), 75–80 (2015). https://doi.org/10.1016/j.ifacol.2015.12.013 9. Diveev, A.I., Ibadulla, S.I., Konyrbaev, N.B., Shmalko, E.Y.: Variational genetic programming for optimal control system synthesis of mobile robots. IFACPapersOnLine 48(19), 106–111 (2015). https://doi.org/10.1016/j.ifacol.2015.12. 018 10. Diveev, A.I., Shmalko, E.Y.: Automatic synthesis of control for multi-agent systems with dynamic constraints. IFAC-PapersOnLine 48(11), 384–389 (2015). https:// doi.org/10.1016/j.ifacol.2015.09.216 11. Diveev, A.I., Sofronova, E.A.: Automation of synthesized optimal control problem solution for mobile robot by genetic programming, intelligent systems and applications. In: Bi, Y., Bhatia, R., Kapoor, S. (eds.) IntelliSys 2019, Volume 2. Advances in Intelligent Systems and Computing, vol. 1038, pp. pp. 1054–1072. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-29513-4 77 12. Diveev, A.I.: Numerical method of synthesized control for solution of the optimal control problem. In: Arai, K., Kapoor S., Bhatia R. (eds.) Intelligent Computing. SAI 2020. Advances in Intelligent Systems and Computing, vol 1228, pp. 137–156. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52249-0 10 13. Derner, E., Kubal´ık, J., Ancona, N., Babuˇska, R: Symbolic regression for constructing analytic models in reinforcement learning. ArXiv:abs/1903.11483 (2019) 14. Sutton, R.S., Barto, A.G.: Reinforcement Learning. MIT Press. Cambridge (2018). 526p. ISBN 978-0-262-03924-6 15. Konstantinov, S.V., Diveev, A.I.: Solving the problem of the optimal control system general synthesis based on approximation of a set of extremals using the symbol regression method. Herald of the Bauman Moscow State Technical University. Series Instrument Engineering 131(2), 59–74 (2020). https://doi.org/10. 18698/0236-3933-2020-2-59-74 16. Mirjalili, S., Mirjalili, S.M., Lewis, A.: Grey wolf optimizer. Adv. Eng. Software 69, 46–61 (2014). https://doi.org/10.1016/j.advengsoft.2013.12.007 17. Diveev, A.I., Konstantinov, S.V.: Study of the practical convergence of evolutionary algorithms for the optimal program control of a wheeled robot. J. Comput. Syst. Sci. Int. 57(4), 561–580 (2018). https://doi.org/10.1134/S106423071804007X 18. Miller, J.F.: Cartesian Genetic Programming. Springer, Heidelberg (2011). 342 p, ISBN 978-3-642-17310-3. https://doi.org/10.1007/978-3-642-17310-3 2 19. Boltyansky, V.G.: Mathematical Methods of Optimal Control, p. 272. Holt, Rinehart and Winston, New York (1971) 20. Khalil, H.K.: Nonlinear Systems. Prentice Hall, New York (2002). 750 p 21. Podvalny, S.L., Vasiljev, E.M.: Analytical synthesis of aggregated regulators for unmanned aerial vehicles. J. Sci. 239(2), 135–145 (2019). https://doi.org/10.1007/ s10958-019-04295-w ˇ 22. Suster, P., Jadlovska, A.: Tracking trajectory of the mobile robot Khepera II using approaches of artificial intelligence. Acta Electrotechnica et Informatica, 1(11), 38–43 (2011). https://doi.org/10.2478/v10198-011-0006-y

Towards Synchronous Model of Non-emotional Conversational Gesture Generation in Humanoids Aditi Singh(B) and Arvind K. Bansal Department of Computer Science, Kent State University, Kent, OH 44242, USA {asingh37,akbansal}@kent.edu

Abstract. We need social robotics to assist humans with elderly care, health care, child education, entertainment, space exploration and hospitality. This task requires sharing a common space with human beings along with humanlike humanoid-human interactions that involve combining facial-expression analysis, speech understanding, gesture analysis and generation. Facial expression analysis, speech understanding, speech generation and hand-gesture recognition using camera and depth-sensors are increasingly being explored. Limited gesture generation is being explored in avatars (virtual) and more recently in humanoids. However, gesture generation is limited to head-gestures, and random and meaningless hand-gestures, which lack detailed classification of gestures and their translation to humanoid gesture generation. In this paper, we extend the hand-gesture classification in the psychology-domain to include haptic gestures and extend the conceptual dependency model for computational hand-gesture generation. We also present a model to generate different subclasses of hand-gestures based upon the notion of scene, object attributes, shallow anaphora analysis and speech-action temporal alignment at the sentence level. Keywords: Conversational gestures · Gesture generation · Hand-gesture classification · Humanoid-human interaction · Social robotics

1 Introduction In recent years, robotics research has increased in four different areas [1]: 1) tools for search-and-rescue missions, including space exploration [2] and deep-sea search; 2) extending human-life and mobility, including intelligent medical devices such as exoskeletons; 3) avatars for mediated communication [3]; and 4) social robotics to provide human-like services for elderly care, entertainment, health care and child education [4–7]. Social robotics requires humanoids to have human-like appearance, comprehend human behavior, and converse with humans without exhibiting negative emotional behavior [8, 9]. We are specifically interested in non-emotional conversational interactions that involve understanding a combination of facial-expression, speech and gestures © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 737–756, 2022. https://doi.org/10.1007/978-3-030-80119-9_47

738

A. Singh and A. K. Bansal

[10–12] and producing limited non-emotional behavior traits expressed using speech and gesture production [13]. Conversational gestures are visual non-verbal communications based upon combinations of upper-body postures and motions to augment or substitute the speech based on the visibility, context and the shared knowledge of the actors [13]. Conversational gestures also facilitate information-retrieval and speech-production as shown by Iverson and Goldin-Meadow using visually challenged speakers [14]. Conversational gestures convey the intent, enhance referential communication, and augment and clarify co-occurring speech by: 1) symbolically describing entities in the scene [15]; 2) expressing behavior traits such as assertion, agreement, denial, interest, etc.; 3) mimicking human actions; 4) expressing warmth through touch, gentle grasp, holding hands, kiss, hug, etc.; 5) breaking the monotony of speech generation. Gestures also reveal human mental states such as anxiety, impatience, nervousness, frustration, indifference, contemplation, besides emotional states such as anger, sadness, happiness, surprise. There are multiple head-gestures required for human non-emotional conversation such as greeting, appreciation, denial, inclusion, backchanneling. Researchers have developed a declarative knowledge-based process-based transition model to generate 36 head-gestures [20] involved in expressing behavior modes in non-emotional conversation. The declarative model allows for ease of modification, adaptation and learning of head-gestures. However, it needs to be augmented with non-emotional conversational gestures involving arms, palms and fingers; spine; shoulder-gestures; postures; their combinations. Along with head-gestures, hand-gestures play a major role in conversational gestures. The semantic domain of hand-gestures [16–19] include: 1) referring to objects in the scene by pointing to them using hands or gaze; 2) modeling the external world and objects using attributes associated with them; 3) modeling the actions of the actors associated with the co-occurring speech; 4) subjectively expressing actors’ mental states such as nervousness, frustration, hesitation, affection, warmth, dominance, submission; and 5) meaningless repeated or random motions while retrieving information or during unclear articulation. Random repeated gestures also act as fillers, providing additional time to retrieve information. Schank and others [21, 22] pioneered the development of conceptual dependency as language independent knowledge-based mental models to understand stories. Conceptual graphs [21, 23] model the world using role-based templates, the notion of proximity, transfer of objects and information, and various action classes to summarize and infer information from the conceptual graphs. Since hand-gestures are used to model the world and actors’ actions, conceptual dependency model [21, 23] can be extended to generate hand-gestures by associating conceptual dependency graphs with associated gesture-productions. In most of the gesture productions, the interlocutor assumes the role of an actor: the action, proximity and location are acted out with respect to the self; all the references to the actor are played with respect to the body of the interlocutor. Gesture generation involves synchronized motion and integration such as jaw and lip motion for the utterance of speech; head-gestures and gaze; hand-gestures involving

Towards Synchronous Model of Non-emotional Conversational Gesture Generation

739

arms, palms and fingers; shoulder and spine-based postures and motions to describe objects virtually using their attributes such as shape, size, weight and motion, and haptic gestures. The synchronized head-gestures are realized using synchronized motion of DC motors [20] using a subset of Allen’s event synchronization rules [25]. In this paper, we extend hand-based gesture classification in the cognitive psychology domain [16–19], and integrate it with: 1) declarative model of the head-gestures [20]; 2) a fuzzy semantic model of gestures [24]; 3) haptics-based gestures. Our goal for this extended classification is the automated humanoid gesture generation for non-emotional conversation in humanoid-human interaction. Our major contributions in this paper are: 1. Extension of hand-based gesture-classification in the psychological domain to include haptics, mental states, and metaphoric gestures (gestures where objects/actions are depicted using their attributes), 2. Exploiting conceptual-dependency graph to map linguistic phrases to gesture actions, 3. Disambiguation of one-to-many and many-to-one mapping between linguistic phrases and gestures using action and object attributes, 4. Integrating declarative synchronized process and fuzzy semantic algebra to model hand-gestures and its integration with head gestures, and 5. Development of a conceptual dependency based knowledge-based computational framework for linguistic gesture-generation. The overall organization of the paper is as follows. Section 2 describes the background work: a subset of Allen’s event synchronization needed for gesture modeling [20, 25]; fuzzy semantic model of gestures [24]; head-gesture modeling based upon Allen’s synchronization; conceptual dependency [21–23] and related definitions. Section 3 describes the related works and gesture classification in the psychological domain [16–19]. Section 4 describes the extended classification of hand-gestures and its integration with head-gestures. Section 5 describes a model for linguistics correlated gesture generation using the conceptual dependency model. Section 6 describes the future work.

2 Background 2.1 Fuzzy Semantic Algebra for Gestures Fuzzy systems map a large set to a small finite probabilistically overlapping intervalbased set denoted by an integer-scale or discrete values [24]. Using fuzzy domains improves computational efficiency. There are two types of gesture parameters: 1) body-posture; 2) body-movement. Body-posture is a combination of head-posture, hand-posture, shoulder-posture, spineposture and gaze. We model a posture as an N-tuple of fuzzy angular (or linear) positions of the corresponding actuators. We map an angular-position to a fuzzy set {−1, 0, +1} where ‘−’ sign denotes either left direction or downward or backward, and ‘+’ sign denotes right direction or upward or forward. We model movement as a (a

740

A. Singh and A. K. Bansal

sequence of postures, transition-delay). In the simplest form, motion of a single organ is a triple of the form (start-posture, end-posture, transition delay). Upper-body motion is a combination of motions of multiple organs: head-motion, shoulder-motion, hand-motion, palm motion, eye-motion along with the transition-delay. A motion can be asynchronous or synchronous. Speech (including silence) and gesture are often synchronized at the sentence-level; Vergence and eye-tracking are synchronized for left-eye-motion and right-eye-motion; Finger movements are synchronized if they are controlled by the same actuator muscles. Head-posture is a triple (head-rotation angle, head-nod angle, head-tilt angle). Hand posture is a triple (left-hand posture, right-hand posture, proximity of hands). The parameter left-hand posture or right-hand posture is a pair (upper-arm orientation, lower-arms orientation). We model orientation-values for upper-arm or lower-arm using a discrete fuzzy set: {front down, front stretched, front up, sideways down, sideways stretched}. The parameter proximity takes a value from the discrete fuzzy set {finger i , palm} × {finger j , palm, hand, shoulder, nose, face, eyes} × {touching, close, apart}. For simplicity, we model a finger as an indexed entity finger i (0 ≤ i ≤ 4) where i is 0 for a thumb. The parameter shoulder-posture takes a value from the discrete fuzzy-set {0, −1, +1} where 0 denotes neutral position; −1 denotes shoulder pushed downwards; +1 denotes pushed upwards. We model palm as a pair (palm-silhouette, palm-orientation). The parameter palmsilhouette takes a value from the discrete fuzzy-set {fist, open palm, thumb up, thumb down, index finger up, index finger pointing}. The parameter palm-orientation takes a value from the discrete fuzzy set {bent backward, bent forward, facing up, facing down, facing left, facing right}. Finger shape takes a value from the fuzzy discrete set {curled, semi-folded, stretched, pointing}. We model gaze as a triple: (left-eye, right-eye, duration). We model left-eye or righteye as a triple: (iris-position, pupil-shape, eye-shape). The parameter iris-position takes a value from a discrete fuzzy-set {inward, outward, up, down, neutral, invisible}. The parameter pupil-shape takes a value from a discrete fuzzy-set {constricted, normal, dilated}. The parameter eye-shape takes a value from a discrete fuzzy-set {blinking, closed, narrow, normal, round, popping}. The fuzzy values are derived using silhouette analysis [26] and depth perception analysis [27]. 2.2 Synchronized Motions in Gestures Two or more movements are synchronous if the start, and the end of the corresponding threads are associated with temporal constraints that cannot be violated [20, 25]. Six types of Allen’s synchronization [25] used in head-gestures modeling are shown in Table 1. We illustrate synchronous declarative modeling of gestures using extended BNF and multiple concurrency operators as shown in Column 3 of Table 1. The symbol ‘→’ denotes the transition from one fuzzy-posture to another fuzzy-posture; ‘+’ shows right-side, up-side or right tilt; ‘−’ shows left-side, down-side or left tilt; ‘0’ shows the neutral posture as described in Subsect. 2.1. The organ-actions or corresponding actuator motions such as head-rotate, head-nod are shown in italics. An action showing movement in either direction will be denoted as ± or ∓. A transition (±1 → ∓ 1) denotes

Towards Synchronous Model of Non-emotional Conversational Gesture Generation

741

movement from (right-side to left-side) or (left-side to right-side) or (upside to downside) or (downside to upside). In the extended BNF, the notation ()* denotes zero or more occurrence of a ; the notation ()+ denotes one of more occurrence(s) of the ; the enclosure of an action within square-brackets […] denotes an optional . A group of nested-actions or a single transition is enclosed within a pair of parenthesis (…). Table 1. Synchronization for gesture generation

Type

Temporal Constraint

Notation

M2

Sequential

M1

M1 M2

M1

asynchronous

M1 || M2 M1

Start synchronous

M2 M1

End synchronous

M2 M1

Strictly synchronous

M2

During

M1

M2

M2

M1

M2

M1

M2

M1 M1

M2

2.3 Modeling Head-Based Conversational Gestures Head-based gestures exhibit human behavior-traits to augment or substitute the speech. Gestures are generated using a combination of synchronized movements of various organs’ postures and motions realized using DC motors. Researchers [20] have modeled a subset of head-gestures such as acceptance, admiration, agreement, affirmation, appreciation, argument, avoidance, backchannel, confidence, confusion, discourage, defensive, defiance, denial, depression, dominate, disagreement, encouragement, expectation, frustrated, greet, inclusion, interested, interject, interrogate, permit, persuade, plead, question, reject, relaxed, request, ridicule, seek-attention, submit, unsure, and veiled disagreement. Example 1. Modeling the head-gesture disagreement. The head-gesture for disagreement is modeled as one or more head-rotation followed by head resting in a neutral position. It is encoded in the extended BNF and synchronization notations given in Table 1 as (head-rotate: (0 → +1))  (head-rotate: (+1 → − 1 → +1)* ) (head-rotate: (−1 → 0)). The notation shows that the head rotates to right at first. Alternately, a head can first rotate to left. Hence, the alternate encoding can

742

A. Singh and A. K. Bansal

be (head-rotate: (0 → −1))  (head-rotate: (−1 → + 1 → −1)* )  (head-rotate: (−1 → 0)). The two alternate actions are combined to be represented as (head-rotate: (0 → ±1))  (head-rotate: (±1 → ∓ 1 → ±1)* )  (head-rotate: (±1 → 0)). Example 2. Modeling the head-gesture interrogation. Interrogation is modeled as a sequence of three actions: head-tilt, concurrent progressively dampened one or more head-nods and speak-phrase starting simultaneously (start synchronization) followed by bringing the head to a neutral position (0). It is encoded in the extended BNF and notations in Table I as (head-tilt: (0 → −1))  (speak-phrase  (head-nod: (0 → (+1 → −1)+ )))  (head-tilt: (−1 → 0)). Example 3. Modeling the head-gesture interested. This gesture interested is a sequence of actions: head-tilt, followed by a strictly synchronized head-nod and speak-phrase, followed by another head-tilt back to neutral position. It is encoded as: head-tilt: (0 → −1)  ((head-node: (0 → (+1 → 0)+ ) speak-phrase))  head-tilt: (−1 → 0). 2.4 Haptics Haptic gestures are based upon a tactile sense of touching palm, hand, face-parts, fingers using palm or fingers. During gesture generation, touch requires pressure-sensor feedback about applied force based on change in capacitance or impedance with change in pressure [37]. Example 4. Modeling the hand-gesture for the mental state nervousness. The mental state of nervousness is expressed as a haptic hand-gesture. This is modeled as two palms facing each-other changing the proximity-value from (close → touching), where the value close or touching are derived by pressure sensor or silhouette analysis of palms. Nervous: ((lower-left-arm: (front-down, front-stretched), lower-right-arm: (frontdown, front-stretched), left-palm-orientation: facing-right, right-palm-orientation: facing-left, (fingersi [(0 ≤ i ≤ 4)], fingersi [(0 ≤ i ≤ 4)], close)) → ((lower-left-arm: (front-down, front-stretched), lower-right-arm: (front-down, front-stretched), left-palmorientation: facing-right, right-palm-orientation: facing-left, (fingersi [(0 ≤ i ≤ 4)], fingersi [(0 ≤ i ≤ 4)], touching)) → ((lower-left-arm: (front-down, front-stretched), lowerright-arm: (front-down, front-stretched), left-palm-orientation: facing-right, rightpalm-orientation: facing-left, (fingersi [(0 ≤ i ≤ 4)], fingersi [(0 ≤ i ≤ 4)], close)))+ .

2.5 Conceptual Graphs and Dependencies Conceptual graph is a logical formalism based upon entity-classes and their attributes, relations, action-classes and attributes to model the world and activities [18]. Concepts are modeled as nodes, and relations connect two concepts. Concepts can be entities,

Towards Synchronous Model of Non-emotional Conversational Gesture Generation

743

attributes, action-class, states and events. It has been applied for knowledge-based inferencing of stories based upon mental models [17]. Conceptual dependency [16] uses many primitive action-classes such as P-trans, A-trans, M-trans, Mbuild, Attend, Grasp, Propel, Move, Ingest, Expel as given in Table 2 [17]. Table 2. Conceptual dependency primitives for story Primitive Description P-Trans

Transfer of location of an object

A-Trans

Transfer of control of an object

M-Trans Transfer of mental information MBuild

Construction of a new information

Attend

Focusing attention on an object

Grasp

Grasping an object

Propel

Applying physical force on an object

Move

Moving own body-part by an agent

Ingest

Taking in an object (food, air)

Expel

Expelling an object from the self

A sentence is mapped to a conceptual graph comprising slots (vertices) and edges connecting the slots using conceptual dependency parsing [18] as illustrated in Fig. 1. Some examples of slots are actor, primitive action class, object, location, proximity-type shown by a preposition. A conceptual-dependency graph corresponding to a speechsentence is based upon lookup of stored templates in the knowledge base indexed on the primitive action class using similarity-based matching. Ambiguities are resolved using context, and pronoun disambiguation using shallow anaphora resolution. Example 5. Conceptual graph for a sentence “John lifted an orange from the table.” In this sentence, the slots actor, primitive action-class, object, proximity-1type, fromlocation are filled with the values John, PTrans, orange, from, table respectively.

Actor John

Primitive action-class PTrans

Object

Proximity type

From location

orange

from

table

Fig. 1. A template for conceptual dependency

744

A. Singh and A. K. Bansal

3 Related Works We classify research studies in gestures into three major categories: 1. human gesture classification by cognitive psychologists and linguists [16–19]; 2. low-level hand-gesture recognition based upon silhouette analysis [26], depth perception analysis in videos [27], statistical learning models such as HMM (Hidden Markov Model) [11, 28], RNN (Recurrent Neural Networks) [29–31], combination of CNN (Convolutional Neural Network) and RNN [32], combination of CNN with LSTM (Long Short Term Memory) [33], Petri net models [34], and fuzzy semantic algebra model [24]; 3. humanoid gesture-generation for social interaction based upon mental modeling of synchronizing speech generation and gesture generation as in the Honda Asimov knowledge-based model [35], inverse kinematic model as in NAO[36], and declarative synchronized process model [20]. 3.1 Cognitive Hand-Gesture Classification Cognitive psychologists associate linguistics and speech generation to hand-gestures [16–19], and have studied the disambiguation of sentence semantics, temporal alignment of spoken phrase and words with gestures, and linked them with the mental states that cause the speech and gestures [12]. Efron classified hand-gestures [16] mainly as 1) objective gesture - gestures independent of speech; 2) Deictic gesture - referring to objects present in the scene; 3) physiographic gesture - visualizing the form of a visual object, a spatial relationship, or acting out a bodily action; 4) logical-discursive gesture related to the semantic meaning of the speech; 5) baton gestures which are rhythmic; and 6) ideographic gesture that sketches object, direction or paths in the air. McNeil, Ekman and Friesen categorized gestures based on tangible objects [17]. Major classifications of gestures defined by McNeil are iconic; metaphoric; deictic and beat gestures. Iconic gestures model images of concrete entities and/or actions; body movement or movement of objects/people in space. Iconic gestures are used to draw the shape of an object. Metaphoric gestures involve abstract depiction of form and/or major attribute of an entity, including abstract concepts. Deictic gestures are used to refer to objects, people, and locations in an actual scene. These gestures are also used to refer to entities (or people), which are not present in the current scene. For example: “a big pot was here in the room.” We can draw the shape of a big pot in air to show where the pot was placed in the room. Beat gestures are rhythmic hand movements and/or finger movements used to add emphasis in a sentence. Ekman and Friesen [18] classify gestures as: kinetographic gestures; pictographic gestures; spatial gestures; ideographic gestures; deictic gestures; and baton gestures. Kinetographic gestures are used to show bodily action or depict actions/motions, which are verbally described in a co-speech. For instance, in the phrase “he swung at the pitch,” swinging motion with the hand represents the bodily action. Pictographic gestures draw a picture in the air to describe the outline/shape of the referent. Spatial gesture show movement that depicts a spatial relationship. For example, a phrase “come near me”

Towards Synchronous Model of Non-emotional Conversational Gesture Generation

745

is acted by moving hand towards the body. Ideographic gesture, deictic gesture, and baton gesture of Ekman and Friesen are like metaphoric gesture, deictic gesture and beat gesture of McNeil [17]. The gesture-classification by McNeil [17] are limited to the actions, movements, space, and their co-relations to the spoken phrases. The classification lacks domainspecific gesture-types such as pointing an entity/location using hand-based or gazebased gestures. Though Ekman and Friesen extended the classification of McNeil, their classification lacks mental states, enumeration, mental model of actions and haptics. Although beat and baton gestures cover repeated motions, these classifications do not distinguish between meaningful gestures involving repeated motion such as emphasis, nervousness and enumeration and nondeliberate random repeated gestures when a person is retrieving information or has an unclear picture of an articulated topic. These classifications also need to be further augmented by formal specifications of synchronized movements of organs. Kraus, Chen and Gottesman [19] divided the classification into bigger clusters: symbolic gesture, lexical gesture, deictic gestures and motor gestures. Symbolic gestures occur when vocal communication is restricted due to distance or noise disturbances. Lexical gestures occur with speech and have a co-relation with the semantics of the speech. These gestures are non-repetitive. Deictic gestures, like McNeil’s classification [17] and Ekman’s classification [18], refer to an object, person, direction, or space, and occur with or without speech. Motor gestures, similar to beat gestures [19] or baton gestures [17], are repetitive and random, unrelated to the semantics of the speech. These classifications lack haptic gestures, meaningful repeat-gestures, mental states, and mapping of the same action words to different gestures based on the combinations of action-class and attributes of the object. These studies have influenced us. We extend these studies to address the limitations as described in Sect. 4. However, the main goal in this paper is to model gesture-generation of humanoids computationally. We also model synchronized temporal behavior of multiple concurrent transitions during a gesture-generation. 3.2 Recognizing Simple Hand-Gestures Gesture recognition involves image-based temporal analysis of a video (a sequence of image-frames). Hidden Markov Model (HMM) has been used for gesture recognition systems in robot control [28, 29] and hand-gesture recognition [11]. Yang et al. [29] proposed an HMM-based method using skin-color based segmentation of the hand area. RNN (Recurrent Neural Network) has been used for hand recognition [30, 31]. RNN has been used along with a skeleton-based model to recognize dynamic hand-gestures [31]. The model uses finger-motions along with other global for gesture-classification. However, because of long time-lags, the information about the finger-motion can vanish or explode, leading to an inaccurate gesture-classification. RNN has also been coupled with Convolution Neural Network (CNN) for hand-gesture recognition [32]. CNN is used to extract depth-features from a set of skeleton points in each snapshot, and RNN extracts temporal patterns from the movement of skeleton points within a sequence. However, RNN can cause information-loss when considering a dynamic gesture over a

746

A. Singh and A. K. Bansal

long time. Researchers have also combined CNN with LSTM (Long Short-Term Memory neural networks) [33] to overcome the information-loss due to RNN. 3.3 Gesture Generation in Humanoids In recent years, humanoid researchers have addressed head-gesture generation [20, 24], and limited hand-gesture generation [35, 36], and fuzzy semantic algebra for gesture representation [24]. Researchers have also developed a declarative model for generating head-gestures that handles synchronization of multiple comprising movements [20]. Honda Asimov robot uses an integrated mental model for synchronized speech and gesture generation [35]. The model synchronizes expressive gestures such as emblems, iconic, deictic, and beat gestures with speech using contextual and semantic knowledge. It is one of the first models to recreate gestures defined by cognitive psychologists. However, the model cannot disambiguate action-words mapping to multiple gesture actions based upon object-attributes. The notion of gesture for the abstract objects is absent. It cannot handle attribute-based depiction of objects and attribute-based metaphoric gesture. It also lacks the notion of ‘self’ present in conceptual dependency. The synchronization is between the speech and action only at the sentence-level. It lacks Allen’s synchronization between multiple actions, and is restricted to synchronization only between speech and overall gestures. The fuzzy semantic algebra model [24] reduces the search space by mapping gestures to a combination of postures and movements (sequence of transitions between postures). However, it lacks temporal behavior and synchronization in the gestures, does not classify gestures based upon linguistic association, and has no notion of conceptual-dependency based modeling to act out the gestures. Except Ng-Thow-Hing et al. [35], other models do not have hand-gesture classification rooted in the psychological domain, lack haptic hand-gestures and linguistic correlation with hand-gestures. None of the models, including Ng-Thow-Hing et al.’s model, exploit the power of conceptual dependency to handle disambiguation of oneto-many mappings between the action-words (verbs) and the multiple gestures based upon object-attributes. These models also do not support metaphoric gestures that play out an object by its attribute(s). These models also lack Allen’s synchronization [20, 25] between concurrent movements comprising a gesture.

4 An Extended Gesture Classification The classifications by cognitive psychologists [16–19] can be further extended based on haptics and conceptual dependency. Using conceptual dependency, the same sentence can map to alternate gestures based upon the actor’s choice or context. For example, the sentence “Tom threw a ball” has two alternate gestures: 1) throwing a small object; or 2) describing a small spherical object.

Towards Synchronous Model of Non-emotional Conversational Gesture Generation

747

During gesture generation, an interlocutor plays the role of the actor in the phrase using the interlocutor’s body. For deictic gestures, an interlocutor uses the index finger to point to an object in a scene. Objects not in the scene are drawn or pointed away by the interlocutor. Body and palm of the interlocutor become the instruments to exhibit proximity, enumeration, emphasis, mental states, etc.. Actions that bring the object towards the actor are acted out by moving palm (or fingers) towards the body. Actions that take the object away use hand(s) to move away from the body. Changes in proximity are played by moving one or both palms away from a reference-object nearest to the palm. Objects are referred metaphorically by their attributes: shape, size, weight or motionpattern. Motion and force are exhibited using palm movement. Shapes are drawn using a combination of fingers, palms, hands or all of them concurrently. Shapes of smaller objects are displayed using combinations of palm(s) and fingers. Descriptions of bigger objects also involve the use of hands. There is one-to-many mapping of the same action, such as ‘open’ depending upon the associated attributes of the object. For example, gestures for “open the jar,” “open the door,” “opening a bottle” and “open your mind” are different. They all borrow the mental image from the actual actions archived from actual human activities. Hence, many verbs are overloaded and seen in the context of the object they are associated with in the associated co-speech. Similarly, the action taken by the phrase “open the door” is scene specific as opening can be done by pulling or pushing the door or by turning the knob (clockwise or counterclockwise) before pushing or pulling. Similarly, gesture for play varies based upon the object context such as “play hockey,” “play tennis” or “play it out” because of the distinct movements of the associated objects. Figure 2 illustrates one-to-many mapping between a verb and the corresponding gestures. Gestures also have many-to-one mapping because of the limited number of organs to act and the limited action possibilities to express proximity. For example, in Fig. 3 the phrase “go away” and “push” both map to the same gesture: the palm is raised facing outward, and the arm is moved resulting into palm moving away. In both the cases, loss of proximity and application of the force is common. In the phrase, “go away,” the verbal command is the application of virtual force, which may translate to push when materialized. Figure 3 illustrates many verbs/phrases being realized by the same gesture.

Action: Open

Object mind jar door

bottle

Fig. 2. One-to-many mapping

748

A. Singh and A. K. Bansal

Action: Go-away

Action: push

Gesture: palms moving outward

Fig. 3. Many-to-one mapping

Metaphoric gestures use the attributes such as shape, size, or movement of one object in the scene used in the linguistic phrase. For example, a person requesting to write a note makes a gesture for holding a pen and scribbling. The act of holding also makes the gesture as haptic and drawing from the pen makes it kinetographic. Hence, a gesture can be a combination of more than one class of traditional gestures. The presence of referential pronouns such as ‘he/she’, ‘it’, ‘them’, ‘that’ in the associated sentence leads to the use of deictic gestures. However, the way the fingers are pointed depends upon the object’s presence in the scene. This requires a list of attributes of the object such as color, shape and motion and related histograms to identify the object’s location in the scene database. We introduce haptic gestures such as “holding hands”; haptic-kinetic (haptic+kinetographic) gestures like “writing with a pen”; enumeration gestures – gestures showing numbers, which are actually repeated kinetographic gestures. Figure 4 shows a directed acyclic graph modeling different subclasses of handgestures. Gesture-subclasses are connected through ‘is-a’ relationship. Multiple outgoing single-edges to distinct classes denote a gesture-subclass has more than one parentclass. Multiple outgoing double edges denote that a gesture-subclass have different ways of expressing the same gesture that belongs to distinct classes. We classify hand-gestures broadly as three independent classes: 1) meaningful gestures; 2) nondeliberate gestures; 3) emphasis. Meaningful gestures associate a meaning with the gestures and are further sub-classified as: 1) co-speech gestures; 2) haptic gestures. The overall classification of hand-gestures is shown in Fig. 4. Co-speech gestures are linguistically correlated with the current sentence or silence in response to a previous sentence. For example, in response to a person saying “goodbye,” the other person may just wave his hand with no speech. Another example is when asked for an opinion, another person may just shrug her shoulder(s) showing indifference. Co-speech gestures correspond to one or more words (or a phrase) in the current sentence. These words trigger the generation of hand-gestures or retrieve the information needed to generate gestures using attributes and/or conceptual graphs. Linguistically correlated gestures are sub-classified as: 1) deictic that point to an object in the scene or away from the scene; 2) attributional gestures which describe an object by its attributes: size, shape, weight, or movement patterns; 3) spatial gestures that describe the proximity and arrangement patterns of two or more objects (or components of an object).

Towards Synchronous Model of Non-emotional Conversational Gesture Generation

749

retrieving nondeliberate gestures

random / unclear

emphasis

ashamed nervous

Hand gestures

haptic gestures meaningful gestures co-speech Iconic gestures

bored / indifferent anxious Kinetic-haptic frustrated

single parent subclass

M e n t a l

multi parents subclass

S t a t e s

deictic action

linguistically correlated gestures

enumerative spatial attributional

composite Metaphoric

Fig. 4. An extended hand-gestures classification

Non-deliberate gestures are exhibited involuntarily when a person is taking additional time to retrieve the information to articulate during a conversation or is unclear about his/her articulation or explanation. Examples of non-deliberate gestures are subconsciously touching face, nose, forearm, palm, etc.; mutual rotation of two palms circularly around each-other; flipping a palm outwardly. Haptic gestures are gesture associated with touching or holding an object. In haptic gestures, fingers and palms touch: 1) other upper-body parts; 2) another surface including others’ arms, face, or palm; 3) another surface; 4) clasp hand; 5) touch and separate palms repeatedly. Haptic gestures can be of different types: 1) non-deliberative gestures such as involuntary touching body-part while speaking; 2) meaningful actions expressing a mental state with no associated speech; 3) involuntarily expressing mental state while speaking; 4) linguistically correlated gesture as part of the conceptual dependency model of the action such as ‘hold’, ‘kiss’, ‘touch’, ‘grab’.

750

A. Singh and A. K. Bansal

Mental gestures exhibit limited mental states such as nervousness, anxiety, frustration, shame and boredom. Many of them are haptic gestures. These gestures may be correlated with the sentences that describe the mental state of the speaker as an adverb qualifying the action or carried from the context. Nervousness is a haptic gesture where fingers from two hands are repeatedly touching and breaking contact. Anxiety is a haptic gesture where both palms touch each-other with fingers interleaved. Ashamed uses two hands to hide the face while touching the face gently. Bored uses a subset of right fingers to tap on the table or a flat surface. Frustration is expressed by raising the lower hand vertically up with palm facing upwards. In attributional gestures, an entity is described using its attributes such as size, shape, weight and/or motion-type. Pictographic gestures are a subclass of these gestures because pictographic gestures draw out an object-attribute to refer to the object. Emphasis and metaphoric gestures do not have a direct correlation with linguistics. Emphasis is imposed on a spoken set of sentences. It is modeled subjectively by different persons by repeated movement of index finger or folded palms, or both palms facing each other. Metaphoric gestures involve the attributes of an entity. They may or may not involve a spoken phrase. Many times, the notion of proximity is embedded in the notion of self, location attribute and conceptual dependency of action-classes (verbs in the linguistic representation). Other times, the spatial relation is linguistically articulated. For example, the proximity associated with the word “come” is part of the conceptual dependency that triggers the palm movement towards the interlocutor’s body. However, the phrase “two cars were parked next to each other” explicitly describes the spatial relationship with a palm repeatedly counting two or two palms facing close to each other with each palm symbolically modeling a car. Enumerative gestures quantify the entities and are associated with words like ‘all’, ‘collectively’, ‘as a group’ ‘one by one’, ‘one’, ‘two’, ‘many’. Each gesture is played out differently. The words ‘all’, ‘as a group’, ‘collectively’ exhibit cohesion or proximity and are played as two palms facing each other coming closer. The phrase ‘one by one’ is played as a repeated movement of index finger(s) in vertical motion. Numbers are played by raising the equivalent number of fingers. The word “many” is shown by two palms facing each other and diverging outwards. Composite gesture are linguistically correlated, where one word or action causes a combination of two or more gestures. The combination may involve head-gestures, spine gestures, shoulder-gestures, or haptic gestures. An example of composite gesture would be ‘writing with a pen’ that involves holding a pen and making wavelike repeated motion in a single direction. Eyes in combination with head-movements are also used to substitute hand movement to point to objects, express domination, giving/accepting commands silently.

Towards Synchronous Model of Non-emotional Conversational Gesture Generation

751

4.1 Integrated Gestures One of the challenges while designing a social robot is to ensure natural human-like interaction and engagement in the social environment. Many organ-movements are combined to enhance the effect of a gesture. Thus, it is important for robot to show gesture as an integration of head and hand gestures or shoulder and hand-gestures or a combination of all three. They are often accompanied with speech. Many gestures such as apology, denial, jubilation, encouragement and frustration are integrated gestures. For example, encouragement involves lifting the head and lifting an imaginary entity using hand(s). The gesture for jubilation uses both approval (a head-gesture) and vertical repeated shaking of both palms facing each other. Denial is a combination of repeated head-rotation along with repeated right-hand motion with palm raised and facing the listener. Shoulder-shrugging shows indifference or arrogance and is often accompanied with a head-tilt; the behavior indifference and unsure exhibit integration of head-gestures, hand-gestures and shoulder-shrugging. In integrated gestures, synchronization between speech and other gestures becomes quite important. The correlation between gaze and hand motion also has to ensure spatial synchronization. Eyes point toward hand movement, guiding the motion of the head while referring to an object in the scene. Head-gesture and hand-gesture should start and end using strict synchronization with speech at the sentence level to minimize aberration in comprehension by the listener.

5 Linguistic Gestures Modeling The overall gesture modeling comprises scene-objects set, various associated attributes of scene objects, conceptual dependency parser that translates textual sentence onto conceptual graph involving scene-objects, action words and attributes, prepositions and conceptual graph dictionary that use disambiguation of gestures based upon objecttype, object and action combination; syllable and temporal analysis to match the speech time and gesture-generation time, location identifier for deictic gestures that use attribute matching with scene-objects, and gesture generator using a declarative gesture generation library. Meaningful gestures involve objects in a dynamically changing scene. The scene changes due to the insertion, deletion or movement of objects changing their proximity with respect to self or other objects. We give an abstract model for gesture-generation in Fig. 5. We assume that during a conversation, the objects in the scene do not change. However, their locations can change. A scene is modeled as a pair (set of objects, proximity relationship-matrix) where each object is abstracted as a 10-tuple (name, class, location, size, shape, color, motiontype, color histogram, shape histogram, motion histogram). Histograms are needed for similarity-based matching of objects in visual scene-analysis. The attributes are inherited from the generic class in the absence of exact knowledge about the object that can be learned. Proximity relationship matrix is a fuzzy relationship built by a combination of preposition analysis such as on, in, under, over or the use of proximity words and phrases such as near, far, close to.

752

A. Singh and A. K. Bansal spoken sentence S c e n e O b j e c t L i s t

Text-to-speech

conceptual dependency templates

textual sentence Parser + anaphora analysis

object attribute-list best conceptual graph + mental state + proximity Attribute/actiong esture library

Speech + gesture timing filter co-speech gestures + mental state + proximity

Nondeliberate gestures

Low prob

probabilistic selection declarative gestures

generated gestures

Gesture generation engine

Fig. 5. A generalized gesture generation model

Deictic gesture generation requires shallow anaphora analysis of the spoken sentence using the scene-object database. The anaphora resolution matches pronoun qualifiers and attributes with the attributes of the objects in the scene based upon the most-recent-first, to enhance the probability of accuracy. The gesture generation has following steps: 1. 2. 3. 4. 5. 6. 7.

Read the sentence (or listen to the next dialog) and extract the words, their classification and the associated objects. Calculate the spoken time of each word by splitting them in syllables and phoneme equivalent. Parse the sentence into extended conceptual dependency template. Using the set of entities in the scene-object database and parsed sentence, identify the words that are appropriate for gesture generation. Map each potential gesture-generating word into a sequence of primitive gesture using a dictionary lookup. Disambiguate the one-to-many mappings of action-words to a specific gesture using action and object-attributes associations. Calculate the gesture generation time for each gesture generating word. Gesture generation and the corresponding phrase should start and end at the similar time-range. Limited padding silence or dilation of time is added to the speech to synchronize the gesture rendering and speech time.

Towards Synchronous Model of Non-emotional Conversational Gesture Generation

753

8.

For the pronouns, perform the attribute matching in the corresponding scene set. Use the most-similar object in the scene set and compare its location with the location of the self. Calculate the parameters to move arm, finger, head and gaze combination in a synchronized manner. 9. Based upon prior learning, probabilistically pick up one sequence of action comprising gesture(s) where the sum of gesture rendering-time is in the range of the conversational speech-time and any added silence. 10. Activate the motors to play the gesture sequence using the rendering of gestures described in a high-level declarative gesture description language [20]. The gestures require a subset of one or more synchronized motions, including repeated dampened motions [20], to change from a posture to another posture. Each organ motion is modeled using rotations of a synchronized combination of motors or linear actuator motions [20]. Pointing an object in deictic gestures is a complex motion involving a combination of head rotation, eye-movements and raising of hand to point to the object. This movement also requires calculation of the relative position of the objects in three dimensions. Example 6. Modeling objects by attributes. Consider a scene with three objects: box, ball, and pen. Box-class has a shapeattribute-value cube; ball-class has shape-attribute-value spherical; and pen-class has shape-attribute-value cylindrical. The spoken phrase is “This box should be moved closer to the pen.” The presence of the pronoun “this” starts referential gesture. Box is identified by a shape histogram. The location of the box is extracted from the boxobject in the scene set, the difference between the body-location and the box-location is calculated. After this calculation, the hand/wrist is rotated to point to the object, index-finger is raised to point to the box. Example 7. Gesture generation of a sentence “Tom threw a ball.” Sentence parsing yields the scene of two objects {Tom, ball}. The knowledge base lookup yields the attributes of Tom and ball, and the objects along with their attributes are placed in the object-property-list. Conceptual dependency analysis derives that actorvalue is Tom; action-value is throw; object-value is ball. Hence, ‘self’ is substituted for Tom. The property-list for the object ball (taken from the knowledge base) is {size: small, shape: spherical}. The analysis shows two possible gestures: 1) action of throwing, or 2) describing the object ball. There are many gestures for throwing depending upon the size and shape of the object. Looking up the attribute-list of the object ball resolves the gesture. The gesture for the action throw is a combination of simple gestures: lower-arm: (0 → +1)  grasp  upper-arm: (0 → +1)  release  (upper-arm: (0 → + 1)  (lower arm:(+1 → 0)). The gesture for grasp is also a combination of multiple actuators associated with finger movements: finger i (0 ≤ i ≤ 4): curled  transition-delay  uncurled  release. Similarly, the gesture for release is inverse motion to grasp. The value 0 denotes the relaxed downward hanging position of the arm. The gesture to render the spherical shape is to curl all five fingers followed by a perceptible delay before releasing the fingers.

754

A. Singh and A. K. Bansal

Example 8. Gesture generation of a sentence: “Bob picked up the pen.” Sentence parsing yields the scene of two objects {Bob, pen}. The knowledge base lookup yields the attributes of Bob that are placed in the object-property-list. Conceptual dependency analysis derives actor-value as Bob, action-value as pick, object-value as pen. Hence, ‘self’ is substituted for Bob. The property-list for pen (taken from the knowledge-base) is {size: small, shape: cylindrical}. Gestures can be rendered for pick and pen. The gestures for picking can be multiple depending upon the size and shape of the object. Looking up the attribute-list of the object resolves the gesture. The gesture for the action ‘pick’ is a combination of simple gestures: lower arm(0 → + 1)  grasp. The gesture for grasp is also a combination of multiple actuators associated with thumb, index-finger and middle-finger movements: (thumb (0 → +1)  index-finger (0 → −1)  middle-finger(0 → −1)). The gesture to render the cylindrical shape for the pen is grasping the pen, followed by a small number repeated back-and-forth rotation of fingers to show the cylindrical shape. The gesture is modeled as grasp  wrist(0 (→ +1 →−1)+ → 0) where 0 denotes neutral position, +1 denotes right-direction, and −1 denotes left-direction.

6 Future Work We are currently implementing an integrated model of upper-body gesture generation that integrates head-gesture, gaze, and hand-gesture. We are also developing a knowledge base of composite gestures associated with action-verbs and simple objects. We are also developing a model of learning gestures using supervised and reinforced learning.

References 1. Fong, T., Nourbakhsh, I., Dautenhahn, K.: A survey of socially interactive robots. Robot. Auton. Syst. 42(3–4), 143–166 (2003) 2. Radford, N.A., et al.: Valkyrie: NASA’s first bipedal humanoid robot. J. Field Robot. 32(3), 297–419 (2015) 3. Ishii Y., Watanabe, T.: An embodied avatar mediated communication system with virtual actor for human interaction analysis. In: Proceedings of the 16th IEEE International Symposium on Robot and Human Interactive Communication, Jeju, South Korea, pp. 37–42 (2007) 4. Thalmann N. M., Zhang, Z.: Assistive social robots for people with special needs. In: Proceedings of the International Conference on Contemporary Computing and Informatics, Mysore, India, pp. 1374–1380 (2014) 5. Loghmani, M.R., Patten, T., Vincze, M.: Towards socially assistive robots for elderly: an end-to-end object search framework. In: Proceedings of the IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), Athens, Greece, pp. 190–194 (2018) 6. García, D.H., Esteban, P.G., Lee, H.R., Romeo, M., Senft, E., Billing, E.: Social robots in therapy and care. In: Proceedings of the 14th ACM/IEEE International Conference on Human-Robot Interaction, Daegu, South Korea, pp. 669–670 (2019)

Towards Synchronous Model of Non-emotional Conversational Gesture Generation

755

7. Kirstein, F., Risager, R.V.: Social robots in educational institutions they came to stay: introducing, evaluating, and securing social robots in daily education. In: Proceedings of the 11th ACM/IEEE International Conference on Human-Robot Interaction, Christchurch, New Zealand, pp. 453–454 (2016) 8. Duffy, B.R.: Anthropomorphism and the social robot. Robot. Auton. Syst. 42(3–4), 177–190 (2003) 9. Wainer, J., Feil-seifer, D.J., Shell, D.A., Mataric, M.J.: The role of physical embodiment in human-robot interaction. In: Proceedings of the 15th IEEE International Symposium on Robot and Human Interactive Communication, Hatfield, UK, pp. 117–122 (2006) 10. Liu, H., Wang, L.: Gesture recognition for human-robot collaboration: a review. Proc. Int. J. Ind. Ergon. 68, 355–367 (2018) 11. Moni, M. A.: HMM-based hand gesture recognition: a review on techniques and approaches. In: Proceedings of the 2nd IEEE International Conference on Computer Science and Information Technology, Beijing, China, pp. 433–437 (2009) 12. Noroozi, F., Corneanu, C., Kami´nska, D., Sapi´nski, T., Escalera, S., Anbarjafari, G.: Survey on emotional body gesture recognition. IEEE Trans. Affect. Comput. (2018). https://doi.org/ 10.1109/TAFFC.2018.2874986 13. Wagner, P., Malisz, Z., Kopp, S.: Gesture and Speech in Interaction - an overview. Speech Commun. 57, 209–232 (2014) 14. Iverson, J.M., Goldin-Meadow, S.: Why do people gesture as they speak. Nature 396, 228 (1998) 15. Andrist, S., Tan, X.Z., Gleicher, M, Mutlu, B.: Conversational gaze aversion for humanlike robots. In: Proceedings of the 9th ACM/IEEE International Conference on Human-Robot Interaction, Bielefeld, Germany, pp. 25–32 (2014) 16. Efron, D.: Gesture and Environment. King’s Crown Press, Morningside Heights (1941) 17. McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. The University of Chicago Press, Chicago (1992) 18. Ekman, P., Frisen, W.V.: The repertoire of nonverbal behavior: categories, origins. usage and coding. Semotica 1, 49–98 (1969) 19. Krauss, R.M., Chen, Y., Gottesman, R.F.: Lexical gestures and lexical access: a process model. In: Language and gesture, pp. 261–283. Cambridge University Press, New York (2001) 20. Singh, A., Bansal, A.K.: Declarative modeling and implementation of robotic head-based gestures for human-robot interaction. Int. J. Comput. Appl. 26(2), 49–66 (2019) 21. Schank, R.C.: Conceptual dependency: a theory of natural language understanding. Cogn. Psychol. 3, 552–631 (1972) 22. Lytinen, S.L.: Conceptual dependency and its descendants. Comput. Math. Appl. 23(2–5), 51–73 (1992) 23. Chein, M., Mugnier, M.L.: Conceptual graphs: fundamental notions. Revue d’Inteligence Artificielle 6(4), 365–406 (1992) 24. Ghayoumi, M., Thafar, M., Bansal, A.K.: A formal approach for multimodal integration to derive emotions. J. Vis. Lang. Sentient Syst. 2, 48–54 (2016) 25. Allen, J.F.: Maintaining knowledge about temporal intervals. Commun. ACM 26(11), 832– 843 (1983) 26. Wang, L., Tan, T., Ning, H., Hu, W.: Silhouette analysis-based gait recognition for human identification. IEEE Trans. Pattern Anal. Mach. Intell. 25(12), 1505–1518 (2003) 27. Xu, D., Chen, Y.L., Lin, C., Kong X., Wu, X.: Real-time dynamic gesture recognition system based on depth perception for robot navigation. In: Proceedings of the IEEE International Conference on Robotics and Biomimetics, Guangzhou, China, pp. 689–694 (2012)

756

A. Singh and A. K. Bansal

28. Park, H.S., Kim, E.Y., Jang, S.S., Park, S.H., Park, M.H., Kim, H.J.: HMM-based gesture recognition for robot control. In: Marques, J.S., Pérez de la Blanca, N., Pina, P. (eds.) IbPRIA. LNCS, vol. 3522, pp. 607–614. Springer, Heidelberg (2005). https://doi.org/10.1007/114924 29_73 29. Yang, Z., Li, Y., Chen, W., Zheng, Y.: Dynamic hand gesture recognition using hidden Markov models. In: Proceedings of the 7th International Conference on Computer Science & Education, Melbourne, VIC, Australia, pp. 360–365 (2012) 30. Shin, S., Sung, W.: Dynamic hand gesture recognition for wearable devices with low complexity recurrent neural networks. In: Proceedings of the IEEE International Symposium on Circuits and Systems, Montreal, QC, Canada (2016). https://arxiv.org/pdf/1608.04080.pdf 31. Chen, X., Guo, H., Wang G., Zhang, L.: Motion feature augmented recurrent neural network for skeleton-based dynamic hand gesture recognition. In: Proceedings of the IEEE International Conference on Image Processing, Beijing, China, pp. 2881–2885 (2017) 32. Lai, K., Yanushkevich, S.N.: CNN+RNN depth and skeleton based dynamic hand gesture recognition. In: Proceedings of the 24th International Conference on Pattern Recognition, Beijing, China (2018). https://doi.org/10.1109/ICPR.2018.8545718 33. Hakim, N.L., Shih, T.K., Arachchi, S.P.K., Aditya, W., Chen, Y., Lin, C.: Dynamic hand gesture recognition using 3DCNN and LSTM with FSM. Sensors 19(24), 5429 (2019). https:// doi.org/10.3390/s19245429 34. Nam, Y., Wohn, N., Lee-Kwang, H.: Modeling and recognition of hand gesture using colored Petri nets. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 29(5), 514–521 (1999) 35. Ng-Thow-Hing, V., Luo, P., Okita, S., Synchronized gesture and speech production for humanoid robots. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Taipei, pp. 4617–4624 (2010) 36. Fadli, H., Machbub C., Hidayat, E.: Human gesture imitation on NAO humanoid robot using kinect based on inverse kinematics method. In: Proceedings of the International Conference on Advanced Mechatronics, Intelligent Manufacture, and Industrial Automation, Surabaya, Indonesia, pp. 116–120 (2017) 37. Huisman, G.: Social touch technology: a survey of haptic technology for social touch. IEEE Trans. Haptics 10(3), 391–408 (2017)

MQTT Based Appliance Control and Automation with Room Occupancy Monitoring Using YOLO Janine Grace B. Abad, Diana G. Romero, Jerome M. Dolalas, Raymark C. Parocha(B) , and Erees Queen B. Macabebe Department of Electronics, Computer, and Communications Engineering, School of Science and Engineering, Ateneo de Manila University, 1108 Quezon City, Philippines {rparocha,emacabebe}@ateneo.edu

Abstract. Control and automation systems are desired in today’s fast-paced world as they provide users the luxury of accessing and controlling devices remotely and automate their activities to an extent. This eventually produces desired results such as efficient energy and resource management. In this study a control and automation system was developed, specifically a smart air conditioning system guided by room occupancy accounting. The typical architecture of a home automation system was used as the system’s architecture, wherein different inherent and non-inherently smart devices were integrated through the Internet IoT platform OpenHAB and MQTT as communication protocol. The development of this study served as demonstration of control and automation systems’ capability to be scaled to larger paradigms such as smart buildings and facilitate efficient energy and resource management on a larger scale. Keywords: MQTT · Smart buildings · Smart building energy management systems (SBEMS) · You Only Look Once (YOLO) · Edge and cloud computing · Room occupancy · Automation

1 Introduction With the emergence and continuous development of the Internet of Things (IoT), various implementations of control and automation systems with its application are being introduced. A study by Mehrabi, Fung, and Raahemifar indicated that in implementing a home automation system, or other automation systems built on IoT, the success is dependent on choosing the right hardware and communication protocol to be used [1]. Additionally, implementation of control and automation systems that are guided by human presence and behavior have also emerged. Home automation systems are optimized by automatically adjusting itself to the most efficient usage of energy depending on the monitored human behavior or presence in the home. Effective and efficient home automation systems can then be scaled into smart buildings and some of the emerging technologies for the creation of smart buildings include smart metering, smart lighting, smart building energy management systems (SBEMS), and smart HVAC systems [2]. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 757–770, 2022. https://doi.org/10.1007/978-3-030-80119-9_48

758

J. G. B. Abad et al.

Fig. 1. Typical architecture of a home automation system [1].

The current architecture for a home automation system can already be scaled to smart building applications and provide its convenience and benefits to a larger population. Figure 1 shows the typical architecture of a home automation system. However, scaling it to a larger paradigm introduces new problems such as devices having different communication protocols forcing users to depend on multiple applications to gain access and control on the devices [3]. There is also the problem of huge amounts of data being exchanged between various devices and the main control unit wherein the choice of communication protocol to be used by the automation system is crucial. In consideration of the aforementioned problems, this study focused on the integration of the MQTT protocol on non-inherent smart devices to the OpenHAB framework which served as the control and automation system’s architecture. This was tested to demonstrate the system’s capability of non-inherent smart automation. Room occupancy monitoring was also explored using CCTV feed and data processing techniques were employed to accurately detect human presence and take into account the number of people in a room. It should be noted that this study was limited to the development of a smart air conditioning (AC) system in a university room setup. The implemented smart AC system considered the number of people inside the room, and the room’s temperature and humidity.

2 Methodology This section discusses how the objectives of this research were accomplished. The overall methodology is divided into five sections: (1) hardware components and software protocols, (2) room occupancy accounting, (3) implementation of MQTT-based sensors, and (4) appliance control and automation system architecture. 2.1 Hardware Components and Software Protocols Since the term “Internet of Things” has been used back in 1999, it has been defined in many different ways, including that which focuses on different requirements such as connectivity and sensory entities that are entangled in the usual IoT environments [4]. In recent years however, researchers and innovators are focusing on the “smartness” of

MQTT Based Appliance Control and Automation

759

IoT which differentiates it from other similar concepts such as sensor networks. An IoT network is said to be smart if it has standardization and openness in the communication standards used, as well as it has object addressability and multifunctionality [4]. The proposed system focused on implementing this aspect of IoT through the integration of various non-inherent smart devices and the use of a standard communication protocol. The proposed system used a Raspberry Pi 3 Model B+ to serve as its main server, where the MQTT broker and automation software, OpenHAB, were installed. For the system to be fully operational, the Raspberry Pi must always be connected to the network and provided with a stable power supply. Aside from being used as a main server, the same Raspberry Pi model was also used to serve as an edge computing device for the video processing module of the system. Edge computing is a solution proposed by Singh which enables data processing and computations to be done at the edge of a network. This reduces transport cost and latency is avoided by transporting large chunks of data from the IoT devices to the main server and back [5]. Other hardware used for the proposed system includes a DHT 22 temperature and humidity sensor, an infrared receiver LED, and two infrared transmitter LEDs. These were integrated to a NodeMCU board to enable MQTT communication with the main server through the WiFi network for control and automation. For the NodeMCU board to be operational, it must be configured to the network being used and provided with a stable 5 V power supply. Lastly, a TP-Link NC250 IP Camera was used for the room occupancy module of the system. The automation and monitoring of these devices were done using a laptop and the temperature and humidity data are displayed and monitored through it. For the MQTT part of the system, Eclipse Mosquitto was used as the message broker. Mosquitto, together with OpenHAB, were installed in the Raspberry Pi main server. To test the MQTT communication between the sensors and the broker, MQTT.fx was used. It monitored the publication of temperature and humidity data as well as room occupancy data. 2.2 Room Occupancy Accounting Using Video Processing Room occupancy data is also central in regulating an AC system as mentioned by Bhutta [2]. In relation to this, room automation assisted with room occupancy can cultivate a platform for appliances that lifts the burden of being mindful of energy saving habits away from the users. Also, this type of system can be beneficial in maintaining energy resources [6]. Thus, it is vital to properly extract room occupancy data. There are mainly two types of occupancy accounting methods, namely, vision-based techniques which are also seen as a non-intrusive way of accounting for room occupation, while the other type is data-based method wherein environmental data of the area are obtained using installed sensors [7]. This study utilized the vision-based technique as the system used images from video feed for the input to determine room occupancy. The system developed in this study can use either a CCTV or an IP camera as a means of extracting raw data from a classroom or a laboratory in the university. Although CCTV cameras are widely used, an IP camera can provide high quality images and are cheaper than the usual cameras used on video-based occupancy accounting techniques such as the Kinect camera used in the study of Zhang et al. [8].

760

J. G. B. Abad et al.

A study by Lan et al. used images from a camera and implemented the You Only Look Once (YOLO) network model to detect the number of pedestrians [9]. In a paper by Redmon et al. YOLO is the fastest model with an FPS of 155 which is ideal for a real-time detector but its accuracy is mediocre [10]. YOLO does detection by first dividing the image up into a smaller S × S grid. Each cell then predicts bounding boxes and confidence levels for each of the boxes. The thicker the box means the higher the confidence value or the probability that that box contains an object. This step only predicts that there is an object in the cell but not what the object is. Simultaneously, the S × S grid also creates a segmentation map with each cell predicting a class probability. This step does not determine that there is an object in the cell but rather a conditional probability that if there is an object in this cell then this is what the object is. After combining the two, a confidence threshold is set to take away low confidence detections for any class and a non-max suppression is used to remove duplicate detections. This then produces the final detection of the algorithm for that image [10]. A confidence threshold is set to minimize unnecessary bounding boxes created from even the smallest hints of an object. The process of detecting objects in a frame by YOLO is shown in Fig. 2.

Fig. 2. General detection process of YOLO algorithm [10].

In this study, the flowchart for the proposed algorithm for extracting room occupancy data is shown in Fig. 3. The algorithm begins with accessing the livestream of the IP camera and then applies the YOLO network model. The next integral step is applying the detection algorithm to filter and detect humans only. A pre-trained model was used for human detection, which was trained using the 2017 COCO dataset. It contained 123,000 images and 80 unique object categories [11]. Although there are more accurate object detection models, YOLO algorithm, from its name, only looks at an image once which results in faster output times. After properly detecting a person, the algorithm accounts for the total number of people inside the room. The mechanism utilized is a sum function which simply adds up

MQTT Based Appliance Control and Automation

761

Fig. 3. General flow of room occupancy algorithm.

the number of bounding boxes within the frame and stores it in a variable. This variable then is displayed on the upper left of the frame. This data is then sent to the MQTT broker for further use. 2.3 Implementation of MQTT-Based Sensors In IoT, a large number of data from different devices are transferred across networks which make the communication protocol an essential factor in designing a system. MQ Telemetry Transport (MQTT) is very efficient for IoT implementations as it mitigates issues faced by other communication protocols by reducing the protocol overheads and providing high efficiency communication. Tests run by Yokotani and Sasaki concluded that the protocol overhead of HTTP, a mostly used communication protocol, is indeed larger than MQTT and this difference in overhead could be critical in IoT applications [12]. Figure 4 shows the latency comparison between HTTP and MQTT protocols.

Fig. 4. Latency comparison between HTTP and MQTT [12].

Since for IoT implementation, inherently smart devices and non-inherently smart devices communicate with the server, an asymmetric architecture like the MQTT protocol is preferable. In line with this, the system made use of various MQTT-based sensors to collect temperature and humidity data from the environment and to send control signals to the AC units. To collect temperature and humidity data from the environment, a DHT 22 temperature and humidity sensor was used. On the other hand, to send control signals to the AC units, an IR receiver LED as well as two IR transmitter LEDs were used. These sensors were then integrated to a NodeMCU board, which served as the

762

J. G. B. Abad et al.

sensors’ WiFi module for MQTT communication and an edge computing device for the IR control of the AC units. Through the Arduino IDE, the NodeMCU board was configured to connect to the WiFi network as well as the main server, which is also the MQTT broker. The NodeMCU board acts as an MQTT client to which temperature data from the DHT 22 will be sent. IR transmitter LEDs were also programmed to send specific IR control signals to the AC units depending on occupancy and temperature data. An IR receiver LED was also used to obtain the AC units’ IR codes. Figure 5 shows the schematic layout of the setup of the MQTT-based sensors and IR LEDs.

Fig. 5. Schematic layout of the setup.

To collect temperature and humidity data, the algorithm first measures the temperature and humidity data in float data type which is then converted to a character array and published with the dtostrf function. These data are published to the MQTT broker under the topic names “room/humidity” and “room/temperature” every 30 s. The room’s heat index was also computed and can be read through the Arduino IDE’s serial monitor. The IR control was designed by first decoding the IR signals used to control the AC units. In this study, the AC units used were Samsung split type AC units. To do so, an IR receiver LED was used to decode the IR signals from the remote control of the AC units using a program written in Arduino IDE. These decoded IR codes were used to control the AC units depending on the collected temperature and room occupancy data. 2.4 Appliance Control and Automation System Architecture The Appliance and Control System Architecture is based on a temperature control system designed to control and automate air conditioning units to produce a more energy efficient output. To do so, determining thermal comfort is essential. Aniag et al. utilized the Predictive Mean Vote (PMV) sensation scale which makes use of a seven-point ASHRAE scale wherein the participants of the study can rate his or her thermal preference and

MQTT Based Appliance Control and Automation

763

yielded a preferred temperature ranging from 22 °C to 24 °C, which is much lower than the ASHRAE standard of 25 °C [13]. Another study by Anamon in buildings at Makati, Philippines also produced similar results. Their study also supplemented further the effectiveness of a temperature control system in being more energy efficient. According to Anamon, raising the temperature setting from 23 to 24 °C would reduce cooling energy consumption by 1.4% and the total energy consumption by 0.6% once other equipment, such as lights, are taken into account [14]. The block diagram of the system architecture is shown in Fig. 6. The system architecture is divided into three blocks, namely, the sensory block, the control block, and the logging block, which also contains the main server for video processing.

Fig. 6. Block diagram of the appliance control and automation system architecture.

The sensory block consists of the camera, the DHT 22 temperature and humidity sensors, which collect necessary data from the environment that adjust the commands sent by the control block to the AC units. The room occupancy data is obtained using the camera feed and processed in the Raspberry Pi. Aside from employing video processing techniques, the Raspberry Pi also serves as an MQTT client and publishes room occupancy data to the MQTT broker, or the Raspberry Pi main server. On the other hand, the DHT 22 sensor collects temperature and humidity data from the environment and publishes it to the main server with the NodeMCU board. The NodeMCU board serves as an MQTT client, which enables it to publish data to the main server. The published temperature and humidity data were integrated to OpenHAB and can be viewed through OpenHAB’s dashboard by the end user. The control block consists of the IR transmitter LEDs which act as an IR remote that sends IR commands to the AC units. For the proposed system, the control block sends control signals dependent on the gathered temperature and room occupancy data. The

764

J. G. B. Abad et al.

control block regulates the temperature inside the room to the desired room temperature of around 24 °C. Figure 7 shows the flowchart of the simple temperature control algorithm implemented by the proponents. The algorithm works similar to the typical bang-bang control system implementation in conventional thermostats, wherein the system changes its response depending on a predetermined setpoint, as shown in Fig. 7. Moreover, the desired room temperature must be adjusted depending on the number of people inside a room. Hence, the system is integrated with the room occupancy data to automate the control of the AC units.

Fig. 7. Implementation of the temperature control algorithm.

Figure 8 shows the flowchart of the room occupancy control algorithm implemented. The logging block consists of the Raspberry Pi which acts as the main server and MQTT broker of the system, as well as the OpenHAB software. As the main server, all temperature, humidity, and room occupancy data were received by the Raspberry Pi. Temperature data are published under the topic “room/temperature” and humidity data were published under the “room/humidity” topic. Lastly, room occupancy data were published under the “room/people” topic. These published data were then retrieved by devices that subscribe to these topics. In this system architecture, the NodeMCU board, as an MQTT client, subscribes to these topics to retrieve the data. These retrieved data were then used to send

MQTT Based Appliance Control and Automation

765

specific IR control signals to the AC units. On the other hand, the OpenHAB software installed in the Raspberry Pi main server served as the end user interface that may be accessed through the Wi-Fi network. From the OpenHAB dashboard, the end user may be able to view collected temperature and humidity data for monitoring purposes.

Fig. 8. Implementation of the room occupancy control algorithm.

3 Results and Discussion 3.1 Room Occupancy Accounting The YOLO algorithm is usually used to detect different objects in a single frame as shown in Fig. 9. Humans, laptops and chairs with bounding boxes as shown are detected, thus, a filtering technique is used so that only humans are detected. Figure 10 shows the frame where only humans are detected. The sample frames in both figures were generated with the confidence threshold set to 0.40. Initially, there are no humans detected by the algorithm since at the exact start of accessing the feed, the program has not detected any moving objects as there are no previous frames to compare the pixels to. Upon movement of the subject, the algorithm begins to perform background subtraction and also calculate if the detected foreground object is not negligible. Multiple foreground objects detected but most of the extra partitions are interference that could have been caused by changes in lights or movement of other objects in the room. However, due to the set threshold for the size of the detected object only one object is detected by

766

J. G. B. Abad et al.

Fig. 9. Sample frame with bounding boxes on all detection.

Fig. 10. Sample frame after filtering with bounding boxes where only humans were detected.

the program as seen in Fig. 10 where only one bounding box is drawn and it is drawn over the human subjects. It should be noted that for most of the evaluation period, there was only one object detected, that is, humans. While the algorithm can detect partially hidden humans, this is dependent on both the percentage and part that is hidden and the motion of the subject. Figure 11 shows two partially hidden human subjects indicated by the red boxes. Human subject A was detected by the algorithm while human subject B was not detected. In this frame, human subject A was moving while human subject B remained stationary.

MQTT Based Appliance Control and Automation

767

Fig. 11. Partially hidden human subjects with and without bounding box.

Figure 12 shows a frame where all the human subjects were detected by the algorithm. In this frame, human subject B remains partially hidden but the percentage of the exposed part has increased, thus, the algorithm was able to detect the subject.

Fig. 12. All human subjects are detected.

It is also possible that more than one bounding box is drawn over the human subject. This can be improved further by recalibrating the code more. The parameters for the image processing techniques could be adjusted as well as the threshold for the size of the detected object. It should be emphasized that these parameters can be adjusted depending on the environment of the room as well as on the angle of the camera. The number of human subjects in the frame is determined by the number of bounding boxes. The number of people is updated each time a subject is detected or undetected, as indicated by the red text on the upper right portion of the frames in Figs. 10, 11 and 12. 3.2 Appliance Control and Automation System The appliance control and automation system was tested considering these two conditions: (1) if it can turn the AC units ON when a person is detected, and (2) if it can control

768

J. G. B. Abad et al.

and maintain the room temperature significantly close to the desired room temperature. During the test, when a person is detected, the room occupancy module publishes a “1” and sends it to the main server or MQTT broker. This prompts the control module to turn ON the AC unit. Figure 13 shows the performance of the temperature control module. It contains the plots of the temperature, heat index and humidity in a room. The data in Fig. 13(a) were measured in the morning, between 10:48 AM and 12:51 PM while the data in Fig. 13(b) were measured between 3:00 PM and 5:30 PM.

(a)

(b) Fig. 13. Performance of the temperature control module (a) in the Morning and (b) in the Afternoon.

The gray line in Fig. 13 shows the desired room temperature set to 24 °C. As shown in Fig. 13(a), the temperature and heat index, which is the apparent temperature or the human-perceived temperature, were closer to the desired room temperature for comfort.

MQTT Based Appliance Control and Automation

769

The room temperature varied between 23.4 °C to 24.1 °C while the heat index ranged from 23.2 °C to 24.1 °C. The temperature control performed well in regulating the room temperature. In Fig. 13(b), the temperature readings stabilized to a value below the desired room temperature. However, the measured room temperature is within ±1 °C of the desired room temperature. It can also be observed the heat index is slightly lower than the measured temperature. This is attributed to the low percentage of humidity. The heat index readings ranged from 24.5 °C to as low as 22.5 °C by the end of testing. This means that the perceived temperature by the people occupying the room was much cooler and comfortable. The appliance control and automation system was able to control the AC units and regulate the room temperature. The room occupancy module can trigger the AC unit to turn ON/OFF when a person was detected or if the room is empty. On the other hand, the temperature control module was able to lower or increase the room temperature closer to the desired room temperature. Albeit erratic and always oscillating, the controlled room temperature was maintained to be close to the desired temperature by at most ±1 °C.

4 Conclusion The study was able to develop a control and automation system that makes use of temperature and humidity sensors, and room occupancy data. To ensure data privacy, video feeds are processed real-time. Thus, there is no need for the system to store the images from the CCTV or IP camera. It only accepts the number of people in the room from the YOLO algorithm and utilizes this information for its features. The system was evaluated on its two main features. First is the capability of the system to detect human presence using the video from CCTV cameras and the people counting algorithm implemented using YOLO. Second is the ability of the system to control and automate non-inherent smart appliances such as the air conditioning unit while considering different parameters such as room temperature and humidity as well as number of people in the room. Communication between the devices of the system was implemented using the MQTT protocol. Results of the study show that YOLO, as an object detection algorithm, can also detect partially hidden humans in a video frame. The control and automation system was able to regulate the room temperature with variations within ±1 °C of the desired temperature. Future work includes expansion of the architecture such as connecting additional air conditioning units and integrating room lights to the automation system. The ability of the system to add elements for room automation needs to be improved as well. Also, migrating the system to the cloud will help in future deployments. It will create flexibility and increase the computational efficiency of the system for human detection. The system developed in this work can also be coupled with an energy monitoring system to provide relevant information to show the impact and benefits of control and automation of devices in smart buildings. Acknowledgment. This study was funded by the University Research Council (URC) of the Ateneo de Manila University under the URC project, “Development of an Energy Monitoring System for Smart Buildings (Phase 1).”

770

J. G. B. Abad et al.

References 1. Mehrabi, T., Fung, A., Raahemifar, K.: Optimization of home automation systems based on human motion and behaviour. In: Canadian Conference on Electrical and Computer Engineering, n.a. Ryerson University, Canada (2014) 2. Bhutta, F.: Application of smart energy technologies in the building sector – future prospects. In: 2017 International Conference on Energy Conservation and Efficiency (ICECE), pp. 7–10. Techfa Consulting, Pakistan (2017) 3. Parocha, R.: Implementation of Home Automation System Using OpenHAB Framework for Heterogeneous IoT Devices. Ateneo de Manila University, Philippines (2019) 4. Khodadadi, F., Dastjerdi, A.V., Buyya, R.: Internet of Things: Principles and Paradigms. Morgan Kauffman (2016) 5. Singh, S.: Optimize cloud computations using edge computing. In: 2017 International Conference on Big Data, IoT and Data Science (BID), n.a. (2018) 6. Bartido, K.P.: Real-Time Occupancy Monitoring Through Thermal Image Processing. Ateneo de Manila University, Philippines (2018) 7. Pedersen, T.H., Nielsen, K.U., Petersen, S.: Method for Room Occupancy Detection Based on Trajectory of Indoor Climate Sensor Data. Aarhus University, Denmark (2017) 8. Zhang, X., Yan, J., Feng, S., Lei, Z., Yi, D., Li, S.Z.: Water filling: unsupervised people counting via vertical kinect sensor. In: IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance (AVSS), n.a. IEEE Computer Society, China (2012) 9. Lan, W., Dang, J., Wang, Y., Wang, S.: Pedestrian detection based on YOLO network model. In: IEEE International Conference on Mechatronics and Automation, n.a. Changchun, China (2018) 10. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: IEEE Conference on Computer Vision and Pattern Recognition, n.a. Nevada, United States (2016) 11. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/ 10.1007/978-3-319-10602-1_48 12. Yokotani, T., Sasaki, Y.: Comparison with HTTP and MQTT on required network resources for IoT. In: The 2016 International Conference on Control, Electronics, Renewable Energy and Communications (ICCEREC), n.a. Kanazawa Institute of Technology, Japan (2016) 13. Aniag, B., Avenilla, A., Collas, J., Regner, P., Cruz, E.: A study on thermal comfort of office employees in the Philippines. In: DLSU Research Congress 2017, n.a. De La Salle University, Philippines (2017) 14. Andamon, M.: Thermal comfort standards and building energy use in Philippine office environments. In: 40th Annual Conference of the Architectural Science Association, pp. 66–72. The University of Adelaide, Australia (2006)

Multi-scale Defect Detection Network for Tire Visual Inspection Minghua Wei1 , Ren Wang2 , and Qiang Guo3,4(B) 1

3

Jinan Institute of Product Quality Inspection, Jinan, China 2 School of Software, Shandong University, Jinan, China School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, China [email protected] 4 Shandong Key Laboratory of Digital Media Technology, Jinan, China Abstract. Automatic defect detection in industrial inspection remains a challenging task due to the diversity of products and the complexity of textures. In this work, we focus on the detection task in tire industry and develop a multi-scale defect detection network (MDDN), which captures both semantic and texture features using two parallel deep convolutional network. High-level semantic features containing defect shapes and locations are captured via a semantic-aware network, simplified by an offthe-shelf fully convolutional network. A simple yet effective texture-aware network is simultaneously developed to complement the details filtered out by the sub-sampling. Pixel-wise detection results are then obtained by integrating features with semantic and texture information. Moreover, we carefully designed a multi-scale preprocessing strategy to make the model describe defects more accurately with the help of the texture similarity in the tire image. Extensive experiments demonstrate that MDDN can achieve significant performance improvement in detecting defects with different sizes over some existing methods. Keywords: Convolutional neural network Multi-scale representation

1

· Defect detection ·

Introduction

Automatic defect detection techniques have become an indispensable part in industrial processes [1,2], especially in tire manufacturing. In most real-world applications, various defects caused by unclean raw materials and undesired manufacturing facilities can be presented in the X-ray image [3]. The defect detection techniques needed to locate defective regions from inspected images first. Defective products are then graded according to the location and size of defects. Due to miscellaneous defect patterns and insufficient illuminating environments, as mentioned in previous works [4, 5], tire defect detection remains one of the most challenging inspection tasks. Although many tire detection methods have been proposed [6–8], most of inspection processes are still performed by human observers, which reduces the efficiency and increases the risk of improper operation. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 771–782, 2022. https://doi.org/10.1007/978-3-030-80119-9_49

772

M. Wei et al.

Existing detection methods are mostly devoted to distinguishing the differences between defective regions and backgrounds (defective-free regions). A key issue of such methods is how to extract features. Early works generally utilized the hand-engineered filter to separate the texture and background components from inspected images. Then defective regions are derived from the residual image via the adaptive binarization thresholding. Obviously, a major shortcoming of these methods is the limitation of the information contained in low-level clues and spatial domain features. In order to take full advantage of the texture features, Guo et al. [3] constructed an anomaly map of texture distortion by weighted averaging of the dissimilarity between each pixel and its neighbors, and located the defects through thresholding the anomaly map. Besides, since the singularity analysis can capture anomalous textures in images, multi-scale wavelet and curvelet transforms were introduced in defect detection tasks [4]. The optimized edge detection and total variation algorithm were also used to achieve more accurate results [8]. Unfortunately, the representation capability of fixed filter kernels used in these algorithms is not powerful enough, and the computational complexity of the transformation is high. Recently, deep convolutional networks (ConvNets), which have outstanding performance in the recognition and segmentation tasks, were attempted to classify tire defects [9]. Relying on the excellent feature extraction capability of ConvNets, Wang et al. [5] extended the fully convolutional network to implement the detection and segmentation of tire defects. However, due to the existence of pooling layers, fully convolutional network is not sensitive to small defects and edge details. To achieve better performance on detecting small objects, many detection methods have been proposed [10–12]. Most of them are based on multi-scale strategies and can be roughly classified into image pyramids, in-network feature hierarchies, and the network integration. In general, image pyramids are a series of multiscale images obtained from hand-crafted filters, e.g., Gaussian filters. Although the image pyramid is simple to implement and widely used [10], it is not sensitive enough to detect defects in tire images with high texture similarity. In-network feature hierarchies are formed by the forward propagation within ConvNets. Through several sub-sampling layers, in-network hierarchies produce feature maps with pyramid shapes and different spatial resolutions [11]. To our best knowledge, SSD [12] is one of the first attempts at combining predictions from these features maps to detect multi-scale objects. An alternative strategy of using feature hierarchies is to directly fuse hierarchical features rather than prediction combination. For example, fully convolutional network (FCN) [13] defined a skip architecture to generate more accurate segmentation. Similar top-down skip architectures are also used in recent works [14, 15]. The training goal of the networks used in both prediction combination and feature fusion is to extract abstract semantics. Thus some texture details are inevitably neglected [16]. Unlike the in-network feature hierarchies in a single network, network ensembles generally rely on several networks with different structures. CrowdNet [17] combined shallow and deep networks to capture multi-scale features, in which the shallow network is specifically designed

Defect Detection Network

773

Fig. 1. The multi-scale preprocessing operation. As the scale decreases, the proportion of defective areas increases. Owing to the similarity of the background texture in tire images, the blocking strategy can balance the number of samples in the defect and the background area, which allows detailed textures to be fully captured by the model.

to retain more details by reducing the number of pooling layers. So detailed features can be better represented. In this paper, by jointly applying the network ensemble and multi-scale representation strategies, we construct two deep convolutional frameworks called multi-scale defect detection network (MDDN). Multi-scale features are captured by training two individual networks, i.e., semantic-aware network and textureaware network. This work mainly includes two contributions. First, we develop a detection structure for tire visual inspection with the out-network multi-scale feature capture strategy. Coarse-scale features can be mined well by existing deep learning architectures (e.g., FCN), and fine-scale features are captured by a simple yet effective texture-aware network without using pooling layers. Secondly, owing to the similarity of the background texture in tire images, a multi-scale preprocessing method, e.g., image blocking, is designed to highlight the detailed texture. As illustrated in Fig. 1, the relative proportion of defective areas increases as the scale of image patches decreases, which makes defects easier to locate.

2

Multi-scale Defect Detection Network

Our network aims to improve the accuracy of tire defect detection, especially for refining edge features and texture details. To achieve this goal, on the one hand, the proposed MDDN integrates multi-scale features from deep and shallow layers by two parallel learning frameworks shown in Fig. 2. The semantic-aware network, a simplified version of FCN, is used to extract abstract semantic features including shape and location information. To complement the details lost in high-level semantics extraction, a simple texture-aware network is attempted to cover more low-level clues without increasing computational complexity. On

774

M. Wei et al.

Fig. 2. Overall architecture of the proposed MDDN. A simple texture-aware network and a semantic-aware Network with different preprocessing strategies are adopted to extract low-level clues and high-level features, respectively.

the other hand, owing to the regularity of background textures and diversity of defects [3, 5], the multi-scale preprocessing step is exploited to further improve performance. 2.1

Semantic-Aware Network

The proposed method captures the desired high-level semantics of defects using a network architecture similar to the well-known FCN-VGG16 (a FCN with VGG16 as the basic framework), which has been proven to be feasible in dealing with tire defect detection task [5]. The method repeatedly applies a stack of convolution and max pooling layers to extract the representative information in tire images. Each fully connected layer in the VGG16 is replaced by a special convolutional layer to retain sufficient spatial informations. Then prediction results with the same size as input images are output by a softmax layer. Here we simplify the off-the-shelf FCN-VGG16 into a pixel-wise binary-classification model. Specifically, in each convolution layer, both padding and stride size are set to 1 and the size of convolution kernels is set to 3 × 3. The crop layers for aligning feature maps in FCN are removed to keep the network more simple. Note that if the input image size is less than 192 × 192, the size of the final output feature maps will be reduced to 1 × 1 after five pooling layers. In practice, this issue can be addressed by adjusting the resolution of the X-ray imaging device. 2.2

Texture-Aware Network

As mentioned above, with several pooling layers, deep networks can extract highlevel semantic information, while a part of detail clues are inevitably lost. We empirically find that increasing the number of pooling layers will improve the detection precision and drop the recall rate. A possible reason is that some small defective regions are discarded by pooling layers as backgrounds. Therefore, we introduce a shallow texture-aware network that consists of only four groups of convolutional layers. Without pooling layers, the detail information can be completely retained. Specifically, A shrinking layer shown in Fig. 2 is utilized

Defect Detection Network

775

to indirectly reduce the parameter dimension, which is a special convolutional layer with 1 × 1 kernels [18]. The reduction capability of this shrinking layer in the channel dimension can reduce redundant information and effectively retain the spatial resolution. Besides, the first two groups of convolution layers were used to extract features. After the shrinking layer, a group convolution layers are adopted to improve representation capability. Note that each group of convolutional structures consists of two convolutional layers and two non-linear layers alternately. 2.3

Multi-scale Network Integration

Since features extracted by the texture-aware network contain a large amount of detail information, the detection results derived from these features are usually sensitive to texture details and have a high false positive rate. In contrast, the deep semantics obtained by the semantic-aware network contain accurate location information and rough edge details of defects. To generate more accurate predictions, an integration learning scheme is introduced. Concretely, semantic and texture feature maps are concatenated in the channel dimension. After that, the final detection result is automatically learned from connected features by training the additional convolution layer with 1 × 1 kernels. This process can be formulated as Integrationx = conv(concat(sx , tx )), where sx and tx are the outputs of semantic-aware and texture-aware networks, respectively, concat(·, ·) denotes the concatenation operation, and conv(·) is the convolution operation using 1 × 1 kernels. 2.4

Preprocessing and Training

To make the texture-aware network more robust to scale variations, training image are all preprocessed by multi-scale blocking and scaling strategies. Generally, in natural image segmentation tasks, image blocks containing the object are first derived from object proposals [19], and the final results are then obtained by segmenting these areas. In contrast, owing to the similarity of background textures, tire images can be cropped evenly while retaining defect semantic information. As can be seen from Fig. 3, defects remains significant in tire blocks, while most of the semantic information is lost in natural image blocks. The cropping and scaling operations essentially change the proportion of defects in training images, as shown in Fig. 1. Thus we empirically divide an image into 4 and 16 blocks without overlap, respectively, and consider scales of 0.5 and 1.5. Since the two sub-networks use different input data and preprocessing methods, the texture-aware network is first trained using block data, after which MDDN is trained with fixed texture-aware network parameters to output

776

M. Wei et al.

Fig. 3. Comparison of natural and tire image blocks. The object (defect) remains significant in tire blocks (Line 1), while most of the semantic information is lost in natural image blocks (Line 2).

refinement results. As described in the previous subsections, owing to the characteristics of tire image textures, original images can be directly fed to obtain end-to-end results during the test phase.

3

Experiments

3.1

Implementation Details

Dataset. Our dataset consists of 914 tire images including both sidewall and tread images, 714 of which are used for training, 100 for validation, and the remaining 100 as the test set. These images involve various defects such as metal impurities, bubbles, and overlaps. In order to more accurately capture the tire image features, different augmentation strategies are used in two networks. For the semantic-aware network, we use flipping, rotating and mirroring to augment the dataset, where the rotation angle is controlled at −15 to 10. Each operation has half the probability that it will be applied to the image during training. For the texture-aware network, cropping and scaling are considered to enable the model to capture local details. Parameter Setting. The MDDN was coded with Python 3.5 in the Pytorch framework. A GTX-2080Ti GPU and Intel Core i9-9900 3.60 GHz CPU were used for both training and testing. The momentum parameter and weight decay were set to 0.99 and 0.0005, respectively. Moreover, semantic-aware network was implemented on the public FCN code,1 and its parameters are randomly initialized. The texture-aware network is a carefully designed structure whose convolutional bias and weights are initialized using randomly and glorot initialization [20], respectively. 1

Available at https://github.com/shelhamer/fcn.berkeleyvision.org.

Defect Detection Network

777

Evaluation Metrics. To quantify the performance of the proposed network, we employed three metrics widely used in semantic segmentation community, including mean pixel accuracy (mPA) [13], mean intersection over union (mIoU) and weighted intersection over union (wIoU). mPA indicates the pixel-level accuracy of the detection results, which is the rate of correct pixels in a per-class basis. In our experiments, the class merely includes the defect and background. mIoU and wIoU calculate the ratio between prediction and true value with different weights to quantify the accuracy of location. In addition, false alarm rate (FAR) and missing alarm rate (MAR) are reported to evaluate the model performance in real-world application, which are effective metrics in detection community. The pixel-level detection capability of networks can be intuitively reflected through the FAR and mPA. The MAR refers to the proportion of undetected defect areas to the total defect areas, which indicates the reliability of the model. Suppose that TP refers to the number of true positives, FP refers to the number of false positives, and TN and FN represent the number of true negatives and false negatives, respectively. These evaluation metrics can be formulated as follows:   TP TN 1 × + , 2 TP + FP TN + FN   TP TN 1 + , mIoU = × 2 TP + FP + FN TN + FN + FP TP + FN TP wIoU = × , TP + FP + TN + FN TP + FP + FN FP , F AR = FP + TN FN . M AR = TP + FN mP A =

3.2

Ablation Experiments

We conducted two ablation experiments to evaluate the effectiveness of the proposed network structure and preprocessing strategies. Firstly, two parallel networks are trained on the same tire dataset without augmentation. Features derived from the semantic-aware network are directly used to generate the segmentation results, which is equivalent to an improved FCN-32s. The textureaware network is then used alone to detect defects. The detection results and quantitative evaluation of two ablation experiments are shown in Fig. 4 and Table 1, respectively. As shown in Fig. 4c and 4d, the semantic-aware network can roughly detect the location and shape of defects, and the texture-aware network has a high false positive rate due to the noise in extracted features. Another ablation experiment was designed to observe the effect of the block strategy on the detection results. Original images and blocks are separately fed into our network for training and testing. Experimental results, illustrated in Fig. 4e and 4f, verified that tire blocks still has relatively complete semantic information.

778

M. Wei et al.

Fig. 4. Results of ablation experiments. From left to right: (a) Tire images, (b) Ground truths, (c) Semantic-aware network, (d) Texture-aware network, (e) MDDN with original images, and (f) MDDN with blocks. Table 1. Quantitative evaluation of ablation experiments. Method

mPA

mIoU wIoU

Semantic

67.94

62.28

96.11

Texture

73.03

68.88

97.03

MDDN

75.43

70.56

97.66

MDDN+blocks 85.19 79.47 97.78

3.3

Comparison with State-of-the-Art Methods

We compare our MDDN with several state-of-the-art object detectors, includes FCN-8s [13], FCN-Tire [5], SegNet [21], DRN [22], and ResNetDUC [23]. The first three utilize VGG16 as the basic framework to extract features, aiming to improve the segmentation accuracy by fusing in-network hierarchy features. Among them, FCN-8s and FCN-Tire developed a skip structure concatenating multi-scale feature map to supplement the filtered detail textures. SegNet attempts to obtain the pixel-by-pixel mapping through the encoding-decoding strategy, which constructs an encoder with identical topology as VGG16, and fuses sparse up-sampling features via trainable convolutional layers in the decoder. The latter two methods adopt ResNet as the basic framework to mine deep features and focus on the adjustment of the pooling layer in the network. Concretely, pooling layers are usually replaced by dilated convolutional layers to reduce the filtering of texture details. These methods were all trained using the SGD optimizer with mini-batch of size 4. Note that, we trained the network from scratch in the same enhanced tire dataset instead of using ImageNet pre-trained parameters for a fair comparison. The comparative experimental results are shown in Fig. 5. It can be observed that using the same in-network feature fusion strategy, SegNet is more sensitive to edge details, but its accuracy is lightly worse than FCN-8s and FCNTire in the tire dataset, which is consistent with the results in [24]. A main

Defect Detection Network

779

Fig. 5. Comparison of experimental results. From top to bottom: Tire X-Ray images, ground truth and results of different networks in turn, including FCN-8s, FCN-Tire, SegNet, DRN, ResNetDUC, and our MDDN.

reason is that tire defects are usually insignificant so that too many convolutional layer parameters cause trouble for detection in the encoding-decoding structure. FCN-Tire fine-tunes the network structure and hyperparameters for tire images, which allows it to obtain better visual performance than FCN-8s. Unlike the in-network feature fusion, MDDN uses two independent sub-networks so that detailed features can be fully preserved, and obtains more accurate detection results. In addition, as can be seen from Fig. 5, the dilated convolution structure effectively improves the detection capability of the model, and makes the network more sensitive to small defects. Since dilated convolution introduces zeros in the convolutional kernel, the segmentation result has obvious “gridding”. In contrary, the shrink layer in MDDN filters redundant information in the feature dimension, which completely preserves the spatial feature of defects. The quantitative evaluation results of the MDDN versus aforementioned detection models are listed in Table 2, where the best results are highlighted in boldface. It is clear that the overall score of our model is higher than other networks. Compared with DRN and ResNetDUC, MDDN integrates multi-scale features, which makes it perform better in defect location detection. Moreover,

780

M. Wei et al.

the proposed method exhibits the best performance in terms of mIoU and FAR scores. In a word, MDDN can effectively detect different size defects with lower false alarm rate. Table 2. Quantitative results of different methods.

4

Method

Training Evaluation metrics Backbone Iterations mPA mIoU wIoU

FAR MAR

FCN-8s [13]

VGG

260K

82.01

75.23

97.56

0.57 35.40

FCN-Tire [5]

VGG

260K

88.39

77.48

97.24

0.67 22.54

SegNet [21]

VGG

360K

88.42 74.38

97.16

1.26 21.90

ResNetDUC [23] ResNet

400K

85.75

76.79

97.64

0.73 27.76

DRN [22]

ResNet

360K

81.31

78.16

97.67

0.52 37.14

MDDN (our)

VGG

300K

85.19

79.47 97.78 0.47 29.15

Conclusion

In this paper, we proposed an MDDN model for tire defect detection. By combining a semantic-aware network and a texture-aware network, MDDN can obtain the essential detailed features while extracting the semantic information hidden in high-level features. Specifically, pooling layers are replaced by a shrinking structure in the texture-aware network to filter redundant information, which completely preserves the spatial features. It is worth noting that, due to the high similarity of background textures in tire images, the blocking operation is exploited to enlarge the dataset and retain more information. The experimental results demonstrate that the detection capability of MDDN has significantly improved over existing tire defect detecting methods, which can locate small defects more accurately. Acknowledgment. This work was supported by National Natural Science Foundation of China (61873145, U1609218), Natural Science Foundation of Shandong Province (ZR 2017JL029) and Science and Technology Innovation Program for Distributed Young Talents of Shandong Province Higher Education Institutions (2019KJN045).

References 1. Kumar, A.: Computer-vision-based fabric defect detection: a survey. IEEE Trans. Ind. Electron. 55, 348–363 (2008) 2. Li, Y., Zhao, W., Pan, J.: Deformable patterned fabric defect detection with fisher criterion-based deep learning. IEEE Trans. Autom. Sci. Eng. 14, 1256–1264 (2016)

Defect Detection Network

781

3. Guo, Q., Zhang, C., Liu, H., Zhang, X.: Defect detection in tire X-Ray images using weighted texture dissimilarity. J. Sens. 2016, 1–12 (2016) 4. Zhang, Y., Li, T., Li, Q.: Defect detection for tire laser shearography image using curvelet transform based edge detector. Opt. Laser Technol. 47, 64–71 (2013) 5. Wang, R., Guo, Q., Lu, S., Zhang, C.: Tire defect detection using fully convolutional network. IEEE Access 7, 43502–43510 (2019) 6. Zhang, C., Li, X., Guo, Q., Yu, X., Zhang, C.: Texture-invariant detection method for tire crack. J. Comput.-Aided Des. Comput. Graph. 25, 809–816 (2013). (in Chinese) 7. Xiang, Y., Zhang, C., Guo, Q.: A dictionary-based method for tire defect detection. In: ICIA, pp. 519–523 (2014) 8. Zhang, Y., Li, T., Li, Q.: Detection of foreign bodies and bubble defects in tire radiography images based on total variation and edge detection. China Phys. Lett. 30 (2013). Article: 084205 9. Cui, X., Liu, Y., Zhang, Y., Wang, C.: Tire defects classification with multi-contrast convolutional neural networks. Int. J. Pattern Recognit. Artif. Intell. 32 (2018). Article: 1850011 10. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision. 60, 91–110 (2004) 11. Lin, T., Doll´ ar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR, pp. 2117–2125 (2017) 12. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C., Berg, A.: SSD: Single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016. ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0 2 13. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 3431–3440 (2015) 14. Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Leibe B., Matas J., Sebe N., Welling M. (eds.) Computer Vision – ECCV 2016. ECCV 2016. LNCS, vol. 9912, pp. 483–499. Springer, Cham (2016). https:// doi.org/10.1007/978-3-319-46484-8 29 15. Ghiasi, G., Fowlkes, C.: Laplacian pyramid reconstruction and refinement for semantic segmentation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, vol. 9907, pp. 519–534. Springer, Cham (2016). https://doi.org/10.1007/978-3-31946487-9 32 16. Zhou, P., Ni, B., Geng, C., Hu, J., Xu, Y.: Scale-transferrable object detection. In: CVPR, pp. 528–537 (2018) 17. Boominathan, L., Kruthiventi, S., Babu, R.: Crowdnet: a deep convolutional network for dense crowd counting. In: ACM MM, pp. 640–644 (2016) 18. Dong, C., Loy, C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds) Computer Vision – ECCV 2016. ECCV 2016. LNCS, vol. 9906, pp. 391–407. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6 25 19. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR, pp. 580–587 (2014) 20. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS, pp. 249–256 (2010) 21. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoderdecoder architecture for image segmentation. IEEE Trans. Pattern Anal. 39, 2481– 2495 (2017)

782

M. Wei et al.

22. Yu, F., Koltun, V., Funkhouser, T.: Dilated residual networks. In: CVPR, pp. 472–480 (2017) 23. Wang, P., et al.: Understanding convolution for semantic segmentation. In: WACV, pp. 1451–1460 (2018) 24. Zheng, Z., Zhang, S., Yu, B., Li, Q., Zhang, Y.: Defect inspection in tire radiographic image using concise semantic segmentation. IEEE Access 8, 112674–112687 (2020)

Guided Filter Bank Hui Yin1(B) , Yuanhao Gong2 , and Guoping Qiu2,3 1

2

School of Intelligent Manufacturing and Equipment, Shenzhen Institute of Information Technology, Shenzhen, China Intelligent Robotics Center, College of Information and Engineering, Guangdong Key Laboratory for Intelligent Information Processing, Shenzhen University, Shenzhen, China 3 School of Computer Science, University of Nottingham, Nottingham NG7 2RD, U.K.

Abstract. Guided filter can perform edge-preserving smoothing by utilizing the structures of a guidance image. However, it is difficult to obtain two images with different contents from the same scenario. Therefore, we focus on the case that the input image and the guidance image are identical. In this case, the direction of the gradient of the output image is the same as the guidance image. Based on this discovery, we change the regularization term of guided filter and develop a more general model which can generate a bank of guided filters. To take examples, we pick up three filters from this bank, where L1 guided filter and L0.5 guided filter are newly proposed filters. Mathematical and experimental analysis are performed to demonstrate that the new filters have totally different properties from the guided filter. L1 guided filter is very suitable for edgepreserving and texture-removing tasks, while L0.5 guided filter can do enhancement automatically. We applied them to a variety of image processing applications, including image smoothing, image denoising, edge detection, image detail enhancement and X-ray image enhancement. The experimental results reveals the effectiveness of the newly proposed filters and their state of the art performance. We also believe that more interesting filters can be developed from our model. Keywords: Edge-preserving

1

· Filter · Guided filter · Smoothing

Introduction

Many applications in the fields of computational photography and image processing need an edge-preserving filter to smooth images. Typical examples include image denoising [16], fusion of differently exposed images [11] and tone mapping of high dynamic range (HDR) images [4]. For decades, various edge-preserving filters were proposed in the literature. Basically, these filtering algorithms can be divided into two categories. One category is global optimization based algorithm. Examples of global optimization algorithms include the total variation (TV) method [17], its extension [20] and c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 783–792, 2022. https://doi.org/10.1007/978-3-030-80119-9_50

784

H. Yin et al.

weighted least squares (WLS) [13] method. The global optimization algorithms solve an optimized problem which is formulated as a combination of a fidelity term and a smooth regularization term. Although the global based filters often generate good results, they are usually very time consuming. The other category is local optimization based algorithm. Examples of local patch based filters include bilateral filter (BF) [19] and its accelerated versions [3,14, 15], guided filter (GF) [7], its modified versions [9, 12] and its iterative version [18, 24]. In general, the local based filters can be calculated in real time, but they always introduce unwanted artifacts, such as gradient reversal artifacts near edges [7]. He [7] proposed a guided filter (GF) that can avoid gradient reversal artifacts. It can perform edge-preserving smoothing while utilizing the structures of a guidance image. Moreover, GF has a fast and non-approximate linear-time algorithm. Because of its low computational complexity and high efficiency, GF has become very popular in a great variety of computer vision applications [2, 9, 12, 21, 23]. However, it is difficult to get two images of the same scenario with different contents. Therefore, research on guided filter when the input image is the same as the guidance image is still meaningful. Through analysis of GF, we find that when the input image and the guidance image are identical, the direction of the gradient of the output image is the same as the guidance image. Based on this insight, we change the regularization term of GF to a general form and introduce a bank of new filters, named the guided filter bank. By setting the index of the regularization term to 1 and 0.5, we get two new filters named as L1 GF and L0.5 GF. They have different properties from GF. L1 GF can better smooth the flat regions with relatively small variation than GF. Surprisingly, unlike traditional filter, L0.5 GF can enhance the details in images automatically. The contributions of this paper are: 1. By changing the regularization term of GF, we develop a group of edgepreserving filter, called guided filter bank. 2. By setting values to the index of the regularization term, we get three filters. We mathematically analyze and compare them and reveal the property for each filter. 3. We show their performances in a variety of image processing applications including image smoothing, image denoising, edge detection, image detail enhancement and X-ray image enhancement.

2

Related Works and Motivations

Various algorithms have been proposed to improve GF. Li [12] proposed a weighted guided filter (WGF). In WGF, an edge aware factor was introduced to the constraint term of the GF. Kou [9] proposed to incorporate an explicit first-order edge-aware constraint into GF and developed the gradient domain guided filter (GGF). Different from these improvement strategy, we only focus on the case when the input image p and the guidance image I are identical.

Guided Filter Bank

(a) GF vs. L1 GF

785

(b) L0.5 GF

Fig. 1. Comparison ak for Filters with h = 2, 1, 12 .

GF assumes that the filter output image q is a linear transform of I in a window ωk centered at the pixel k: qi = ak Ii + bk , ∀i ∈ ωk

(1)

where (ak , bk ) are some linear coefficients assumed to be constant in ωk . This local linear model ensures that q has an edge only if I has an edge, because ∇q = a∇I. This means a describes the relationship between ∇q and ∇I. When p = I, in GF, ak can be calculated as: ak =

σk2 +

σk2

(2)

that is, ak ≥ 0 is always satisfied in this case. This conclusion inspires us to propose a more general framework to generate a bank of guided filters.

3

Guided Filter Bank

Based on the findings that when p = I, a ≥ 0 is always satisfied, we propose the following model: E(ak , bk ) =

 1 1 ( (ak Ii + bk − Ii )2 + ahk ) 2 h i∈ω

(3)

k

which minimizes the difference between q and I. Here  is a regularization parameter and h is the index of ak . h can be an integer or a fraction. The solution to (3) can be calculated by:  ∂E(ak , bk ) = (((ak − 1)Ii + bk )Ii + ah−1 )=0 k ∂ak i∈ω

(4)

 ∂E(ak , bk ) = ((ak − 1)Ii + bk ) = 0 ∂bk i∈ω

(5)

k

k

786

H. Yin et al.

From Eq. (5), we can get bk = (1 − ak )μk

(6)

Substitute Eq. (6) into (4), we can get  σk2

= (1 − ak )a1−h k

(7)

where μk and σk2 are the mean and variance of I in ωk . Given h, we can calculate the expression of ak and bk . To simplify calculation, we set h = 2, 1 and 12 and calculate ak . bk can be easily obtained by Eq. (6). Case 1 : h = 2

3.1

In this case, the regularization term is the L2 norm of ak , which is the same as GF. Therefore, the solution is GF when I = p, that is, = aGF k

σk2 σk2 + 

(8)

From Eq. (8) we can observe that ak is non-negative. Case 2 : h = 1

3.2

In this case, the regularization term is the L1 norm of ak , so we named this new filter as L1 guided filter (L1 GF). This regularization term is well used in T V − L1 model [22]. The solution is  c, if c ≥ 0 L1 ak = (9) 0, if c < 0 where c=1−

 σk2

(10)

Equation (9) maintains that ak is non-negative. 3.3

Case 3 : h =

1 2

In this case, the regularization term is the L0.5 norm of ak , so we named it as L0.5 guided filter (L0.5 GF). The fractional index is used in hyper-Laplacian model [6,10] to fit the distribution of natural images. Instead of solving (7), we solve the following equivalent equation: (1 − ak )2 ak =

2 σk4

(11)

Guided Filter Bank

Fig. 2. Image smoothing (r = 5).

(a) Input

(b) GF

(c) L1 GF

Fig. 3. Image denoising (r = 5,  = 0.005).

787

788

H. Yin et al.

(a) input

(b) RTV

(c) GF

(d) L1 GF

Fig. 4. Edge detection (r = 3,  = 0.05).

(a) input

(b) GF

(c) L0.5 GF

Fig. 5. Image detail enhancement (r = 5,  = 0.01).

(a) input

(b)  = 0.001

(c)  = 0.005

Fig. 6. X-ray image enhancement with L0.5 GF (r = 5).

Guided Filter Bank

789

To simplify the expression, we define c1 and c2 as  σk2 √  1 c2 = (−2 + 27c21 + 3 3 −4c21 + 27c41 ) 3 c1 =

(12) (13)

and ak is calculated as follows 1

0.5 = aL k

4

23 1 c2 [2 + + 1] 3 c2 23

(14)

Analysis of the Filters

Figure 1 shows the changing of ak vs. σ 2 for different filters. Based on this figure, we observe the properties of the three filters. 4.1

Properties of GF and L1 GF

From Fig. 1(a), we can see that 1 – When σ 2 is large (region with edges), aGF and aL k k are approximated to 1, 1 which means both of them can preserve edges. Since aGF > aL k k , GF can better preserve edges than L1 GF. GF 1 – When σ 2 is small (flat region), aL k is much smaller than ak , which means L1 GF can make these regions smoother than GF. – When  is decreased from 0.1 to 0.001, the difference between GF and L1 GF is getting smaller gradually.

4.2

Property of L0.5 GF

From Fig. 1(b), we can see that 0.5 – When σ 2 is large (region with edges), aL is approximated to 1. This reveals k that L0.5 GF can preserve the edges as GF and L1 GF. 0.5 is always bigger than 1, which means L0.5 can enhance the gradient of – aL k the input image. This is consistent with [5], which uses fractional index as a 0.5 becomes global regularization. Especially when σ 2 is small (flat region), aL k very large. This reveals that L0.5 GF can enhance the details in the input image. – When  is increased, the intensity of the enhancement is enlarged.

We summarize the properties of the three filters in Table 1.

790

H. Yin et al. Table 1. Summery of the properties of the filters

5

L1 GF L0.5 GF

Property

GF

Edge-preserving

Better Yes

Yes

Detail-smoothing Yes

Better No

Detail-enhancing No

No

Better

Applications

In this section, we apply the filters to various image processing applications, such as image smoothing, image denoising, edge detection, image detail enhancement and X-ray image enhancement. 5.1

Image Smoothing

Figure 2 shows the image smoothing results of GF and L1 GF.  is increased from 0.001 to 0.01. Comparing Fig. 2(a), (b) and (c), we find that the edges, such as the boundaries of the petals, can be preserved well by both GF and L1 GF. While the textures, such as the wrinkles on the petals and the veins on the leaves, are removed more clearly by L1 GF than GF. This is more clearly shown on the gradient curves, which is calculated on the middle line of the results (only the R channel), below the images. When  is increased to 0.01, both GF and L1 GF blurs edges and L1 GF removes more textures than GF. 5.2

Image Denoising

GF and L1 GF are used to remove noise of low light images [1]. The results are shown in Fig. 3. On one hand both GF and L1 GF can preserve edges in the images. On the other hand, the noisy speckles are removed more clearly by L1 GF. These results demonstrate the perfect properties of L1 GF and are consistent with our analytical conclusions in Table 1. 5.3

Edge Detection

We use canny detector [8] to detect edges from the filtered image. One example is shown in Fig. 4. In the input image, many tiles exist. These are the textures we want to remove and the edges of the food are need to preserved. To better compare the results, we also show the results of relative total variation (RTV) [20], which is a global optimization method and very good at this application. Visually comparing the results, we can see that L1 GF can get a clear edge detection results similar as RTV’s. This reveals that L1 GF can remove textures more thoroughly than GF and preserve the edges as well.

Guided Filter Bank

5.4

791

Image Detail Enhancement

Edge-preserving filters can be used to enhance image details as shown in Fig. 5. The result of GF is achieved by amplifying the detail layer by 5 times. As can be seen, L0.5 GF can enhance the details and preserve edges as better as GF. However, using L0.5 GF to enhance image details is more convenient, because the intensity of the enhancement is controlled by only one parameter (). 5.5

X-Ray Image Enhancement

In general, the details of the X-ray images are hard to be seen, which may impact the diagnosis. In this section, we use L0.5 GF to enhance the details of the X-ray images. Figure 6(a) is the input image. As can be seen, the veins in the lungs are not very clear shown in the input image. After enhancing by L0.5 GF, the details of the veins are more and more clearly shown.

6

Conclusion

In this paper, we focus on the case that the input image and the guidance image are identical and propose a general model to generate a bank of guided filters. Furthermore, by setting values to the index of the regularization term, we obtain two new filters, L1 GF and L0.5 GF. From theoretical analysis and application experiments, we demonstrate that L1 GF can better smooth textures than GF. Thus, it is very suitable to do edge-preserving and texture-removing tasks. Moreover, L0.5 GF can enhance the image details automatically by adjusting one parameter, which is beyond the capacity of traditional filters. Our future work will explore to find more useful filters from our model.

References 1. Anaya, J., Barbu, A.: Renoir - a dataset for real low-light image noise reduction. J. Vis. Commun. Image Rep. 51(2), 144–154 (2018) 2. Dai, L.: Interpreting and extending the guided filter via cyclic coordinate descent. arXiv, 1705.10552v1 3. Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. Graph. 21(3), 257–266 (2002) 4. Farbman, Z., Fattal, R., Lischinshi, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. 27(3), 67:1–67:10 (2008) 5. Gong, Y., Sbalzarini, I.: Local weighted Gaussian curvature for image processing. In: IEEE International Conference on Image Processing (2013) 6. Gong, Y., Sbalzarini, I.: A natural-scene gradient distribution prior and its application in light-microscopy image processing. IEEE J. Sel. Top. Signal Process. 10(1), 99–114 (2016) 7. He, K., Sun, J., Tang, Z.: Guied image filtering. IEEE Trans. Pattern Anal. Mach. Learn. 35(6), 1397–1409 (2013)

792

H. Yin et al.

8. John, C.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679–698 (1986) 9. Kou, F., Chen, W., Wen, C., Li, Z.: Gradient domain guided image filtering. IEEE Trans. Image Process. 24(11), 4528–4539 (2015) 10. Krishnan, D., Fergus, R.: Fast image deconvolution using hypre-laplacian priors. Neural Inf. Process. Syst. (2009) 11. Li, Z., Zheng, J., Rahardja, S.: Detail-enhanced exposure fusion. IEEE Trans. Image Process. 21(11), 4672–4676 (2012) 12. Li, Z., Zheng, J., Zhu, Z., Yao, W., Wu, S.: Weighted guided image filtering. IEEE Trans. Image Process. 24(1), 120–129 (2015) 13. Min, D., Choi, S., Lu, J., Ham, B., Sohn, K., Do, M.: Fast global image smoothing based on weighted least squares. IEEE Trans. Image Process. 23(12), 5638–5653 (2014) 14. Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. In: ECCV, pp. 568–580 (2006) 15. Porikli, F.: Constant time o(1) bilateral filtering. In: CVPR (2008) 16. Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992) 17. Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 60(1–4), 259–268 (1992) 18. Shen, X., Zhou, C., Xu, L., Jia, J.: Mutual-structure for joint filtering. IEEE Int. J. Comput. Vis. 125, 19–33 (2017) 19. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: ICCV, pp. 839–846 (1998) 20. Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. ACM Trans. Graph. 31(6), 139:1–139:10 (2012) 21. Yang, Y., Que, Y., Huang, S., Lin, P.: Multiple visual features measurement with gradient domain guided filtering for multisensor image fusion. IEEE Trans. Instrum. Meas. 66(4), 691–703 (2017) 22. Yin, W., Goldfarb, D., Osher, S.: Image cartoon-texture decomposition and feature selection using the total variation regularized l1 functional. In: International Workshop on Variational, Geometric and Level Set Methods in Computer Vision, pp. 73–84 (2005) 23. Zhang, C., Ge, L., Chen, Z., Qin, R., Li, M., Liu, W.: Guided filtering: toward edge-preserving for optical flow. IEEE Access 26958–26970 (2018) 24. Zhang, Q., Shen, X., Xu, L., Jia, J.: Rolling guidance filter. In: ECCV, pp. 815–830 (2014)

High-Speed Multi-person Tracking Method Using Bipartite Matching Alexander Gilya-Zetinov1(B) , Alexander Bugaev1 , Alexander Khelvas1 , Egor Konyagin2 , and Julien Segre3 1

Moscow Institute of Physics and Technologies, Institutskii per. 9, Dolgoprudnii, Russian Federation [email protected] 2 Skolkovo Institute of Science and Technology, Bolshoy Boulevard 30, bld. 1, Moscow, Russian Federation 3 Stanford University, Stanford, CA 94305, USA http://www.mipt.ru

Abstract. The article proposes a new multi-person pose tracking-bydetection method capable of very fast execution without use of GPU acceleration. Starting from a common online approach to object tracking, based on graph bipartite matching, we further improve the method with several novel additions. First, linear regression is used to estimate the movement of a pose root point, allowing the tracker to better preserve identity recognition despite missing detections. It is combined with a new cost function term based on per-joint distance to improve accuracy in scenes with high densities of individuals. Additionally, an alternative for visual feature matching without the use of neural networks is provided. Experiments on PoseTrack dataset have shown that proposed method outperforms other current tracking-by-detection methods based on graph bipartite matching and achieves tracking quality close to the state-of-the-art in general. Keywords: Pose estimation tracking · Machine vision

1

· Bipartite graph · Multi-person pose

Introduction

The rapid advancement of deep learning in recent years has led to development of many efficient single-frame pose estimation methods [1, 2]. They allow the detection of pose keypoints for each video frame, and treat detections obtained on different frames independently. But for a wide range of practical applications such as person re-identification or human activity monitoring, it is desirable to perform identity matching between detections from different frames. One way to achieve this is to add an additional tracking stage, combining detected poses into tracks (Fig. 1). The task of multiple object tracking (MOT) is a long-standing problem in the field of machine vision [3]. It involves two key challenges: to correctly extract c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 793–809, 2022. https://doi.org/10.1007/978-3-030-80119-9_51

794

A. Gilya-Zetinov et al.

Fig. 1. Example of Several Keypoint Estimation Results and Proposed Tracker Output (Bottom Right). Single-Frame Estimation Results Contain Accidental Detection Misses and Spurious Detections, Making the Tracking Task More Difficult. Different Colors Stand for Different Identities.

objects of interest from background and to determine the identity of selected objects throughout the course of the given video. Such objects extracted are often approximated with simple primitives, such as points or bounding boxes [4]. In the case of person tracking this can lead to the loss of valuable pose information, which may ultimately be needed to handle cases of partial occlusion and to accurately process crowded scenes. The variation of the MOT problem applied to the tracking of individuals has received a name of multi-person pose tracking problem (MPPT). In the recent years, considerable attention has been garnered because of the rapid advancement of machine vision technology and wide range of possible applications [5] to [17]. For the MPPT, the tracked subjects are usually represented by sets of connected

High-Speed Multi-person Tracking Method Using Bipartite Matching

795

keypoints. This information is more rich than the bounding boxes and could be leveraged at tracking stage to reduce errors. Current methods either use keypoints through simple functions based on distance [9, 11, 20] or employ neural networks to process them [13, 17], increasing performance cost significantly and making tracking algorithm to depend on learnable parameters. In this article, we propose a new method for tracking by keypoints data with handling of missing detections by means of linear regression. Additionally, it is capable to take visual features into account by calculating temporal color statistics in the vicinity of detected keypoints.

2

Related Work

To efficiently compare performance between different methods of solving the same task, an openly available benchmark is necessary. For multi-person pose tracking, the most popular benchmarks to date are PoseTrack 2017 and 2018 datasets, originally provided at ICCV’17 and ECCV’18 workshops, respectively [5, 6]. They include 1356 annotated video fragments combined, and provide an evaluation code calculating mean average precision (mAP) and multiple object tracking accuracy (MOTA) metrics for benchmarking purposes. During the workshop challenges and over the last couple years, a number of potential solutions to this problem were proposed in the literature. There are two groups, each differentiated by their pipeline structure. Methods from the first group, being the winner at both competitions at the time, have a pipeline that can be split into two parts - an object detection stage, providing pose estimates for individual input frames, and a tracking stage, combining obtained poses into tracks by assigning identities to them. Methods from the second group perform both detection and tracking simultaneously [7]. A typical two-stage method for multi-person body tracking is presented in [8]. It is built on top of a single frame pose estimatior Mask-RCNN [2]. To convert individual poses into connected tracks, Mask-RCNN employs bipartite matching in a weighted graph connecting previous predictions with new detections. The associated cost of edges is defined as intersection-over-union (IoU) ratio for corresponding bounding boxes, and does not take into account pose similarity, motion model or visual features. These tracker limitations were targeted in the later work [9], through the addition of two new terms in the edge cost function. The first term is responsible for handling the visual features and equals to the cosine distance between feature vectors extracted from a detection region. Feature vectors are generated by additional CNN (a subset of ResNet layers was used). This term reflects the fact that it is expected that an individual’s appearance does not change significantly over the track. The second new term is proportional to the percentage of correct keypoints in relation to head segment length (PCKh) between the new and old pose [15]. It further increases the probability of matching for near pose pairs, together with the IoU term.

796

A. Gilya-Zetinov et al.

A logical continuation to that method is proposed in [11]. Instead of comparing the previous bounding box location to the current values, it performs the prediction of new locations by computing a gradient of the position from several last frames. It was also observed that using Euclidean distance, as opposed to the cosine distance, resulted in better matching outcomes for visual features in that particular task. The current record-holder for the PoseTrack 2017 challenge data set DetTrack [19] uses the same bipartite matching approach for the tracking stage, but employs an OKS (Object Keypoint Similarity) value in the cost function. The OKS value is defined as the normalized total distance between joint coordinates, and is conceptually similar to PCKh. To make a complete picture of the current leading methodology, we have included approaches to this problem completely relying on neural networks for tracking. One such solution, by Ning et al., employs a Graph Convolution Network to match poses in adjacent frames and was shown to outperform more orthodox tracking methods [13]. Another solution by Snower et al. [17] uses a transformer architecture and pose keypoint tokenization, achieving results close to DetTrack. Both solutions rely only on keypoint coordinates to match person identity and consequently do not require heavy computations, compared to neural network based methods using image data.

3

Our Approach

We propose a new tracking stage algorithm for two-stage multi-person body tracking in videos, continuing the ideas previously presented in the literature [9, 11,20]. This method is online, and it updates the current state sequentially processing the video frames. The overall algorithm flow chart is presented at Fig. 2. The first step is to compute pose estimations for individual frames, using an arbitrary single frame multi pose estimator (Sect. 3.1). Then, the obtained pose is matched with the currently active tracks. A track is a data structure corresponding to a previously encountered person with an assigned identity number (track ID), who may potentially appear again. The track stores all relevant information from the previously processed video data, such as previous coordinates of root joint, last detected pose keypoints, visual features representation, etc. We describe this structure and used definitions in the Sect. 3.2. The task of matching detections with tracks may be defined using a weighted bipartite graph G = (T, D, E) (Fig. 3). Nodes in the first set T represent active tracks, nodes in the second set D represent detections returned by the singleframe estimator. Weighted edges E connect every i-th track from T with every j-th detection from D and have an assigned cost value C (i,j) . C (i,j) is calculated from the track and detection data is constrained using prior knowledge about possible object state evolutions in time (taking into account limitations due to trajectory continuousness, visual features, etc.), and is central to the tracking

High-Speed Multi-person Tracking Method Using Bipartite Matching

797

Fig. 2. Proposed algorithm flow chart.

methods based on bipartite matching. We define proposed components of cost function depending on keypoint coordinates and visual features in Sects. 3.3 and 3.4, respectively. The cost value C (i,j) depends on i-th track data and j-th estimated pose. It may be interpreted as negated log likelihood function: − log P(Uj = i|T, D) + A˜ = C (i,j) = C(Ti , Dj )

(1)

where Uj is the assigned track ID for detection j, Ti is the information about ˜ track  i, Dj is the detection data j. A is a normalization constant chosen so ∀j i P(Uj = i|T, D) = 1. Its value does not affect the optimal solution and thus maybe omitted. The maximum likelihood estimate can be found by min(Uj ,j) . This task is known as the assignment problem, and may imizing jC be formulated as finding a matching in the aforementioned graph G, where the summed weights of chosen edges is a maximum. This can be achieved by the means of Hungarian algorithm [18]. Thereby the next step in the algorithm is to compute the cost function for every possible {active track, detected pose} pair. Then the Hungarian algorithm is applied on the obtained value table in order to find an optimal assignment Uj between available tracks, including previously existing and new options, and

798

A. Gilya-Zetinov et al.

Fig. 3. An Example of Correct Track-Detection Matching (Red Edges). Active Tracks T Store Root Joint Trajectory, Used to Calculate Edge Cost C (i,j) to Every Detection in D.

the obtained pose detections. Finally, we update the tracks - assign new IDs if new tracks were used, and remove tracks if they were unused for a given time threshold. This step is described in Sect. 3.5. 3.1

Single Frame Pose Estimation

In our work, the freely available OpenPose framework is used for keypoint estimation [1]. It uses a bottom-up system and introduced a concept of Part Affinity Fields (PAFs). PAFs are 2D vector fields that indicate the most probable directions of joint connections, allowing for more precise matching of keypoints in crowded scenes. Since we are aiming at developing a robust tracking stage independent to detector choice, OpenPose weights have not been specifically tuned to the PoseTrack dataset and instead we used the provided model weights which were learned on the CMU Panoptic Studio dataset. By means of OpenPose a set of keypoint detections is obtained for each person in every frame. We employed the BODY25 model, presented at Fig. 4.

High-Speed Multi-person Tracking Method Using Bipartite Matching

799

Fig. 4. The BODY25 OpenPose model used in the work.

3.2

Tracking Task Description and Definitions

In this section, we will describe values used by the tracking algorithm, and introduce the necessary definitions for the rest of the paper. The values provided by the keypoint estimation algorithm and input data are: 1. 2. 3. 4. 5.

rj,k - k-th keypoint of j-th person in the currently processed frame. cj - root keypoint of j-th person in the current frame. T - time from start of the video, corresponding to the current frame. ti - time from start of the video, corresponding to i-th frame. mx,y,c - color channel c value of pixel with x, y coordinates in the current frame

The intermediate tracker variables that constitute the Ti structure and are stored for every active track: (i)

1. rk - saved last detected position of k-th keypoint for i-th track (i) 2. cj - saved root keypoint of i-th track in the frame j. 3. d(i) - characteristic distnace for i-th track, calculated as running average of maximum bounding box dimension for that track. 4. a (i) , b (i) - current root position linear regression coefficients for i-th track. 5. p (i) - projected root position of i-th track for the current frame. (i) (i) (i) 6. ηj , σj , wj - temporal statistics for the j-th feature of i-th track, as further explained in Sect. 3.4. All vectors are 2D and connect the top-left pixel with a pixel in the frame coordinate space.

800

3.3

A. Gilya-Zetinov et al.

Cost Function

The central innovation of this work is the cost function calculation. We propose two new components with similar roles to IoU or PCKh terms of related works. The cost value for matching the i-th track with j-th predicted pose is as follows: (i,j)

(i,j)

(i,j) (i,j) + Ccenter + Cpose + Cvis C (i,j) = Cnew

(2)

(i,j)

The Cnew term is the penalty for creating a new track instead of continuing an existing one. The penalty equals zero if the track already existed, and 1 if it was just created. It may be viewed as a normalization term for the cost value. (i,j) If the sum of all terms other than Cnew is greater than 1, then a new track would be created instead of continuing an old one, aiding in choosing the correct (i,j) parameter values. A functional dependency of Cnew on the screen coordinates for pose j may be introduced in order to model video zones with higher probability of new track creation, e.g. frame borders or entrances. (i,j) The Ccenter is the first novel term, replacing the function of IoU. It is designed to allow the tracker to preserve identity if there is a discontinuity in the detection of a given individual, assuming that the human velocity has not changed drastically over that period. It is proportional to distance between predicted root point for track i and calculated center point for pose j. The definition of root point may vary with skeleton model utilized - the requirements for it are to have the most linearizable motion possible, and to be derivable from partially occluded detected pose keypoints. Provided M (i) is a set of frame indices where an associated detection is present for track i, we can write the predicted root position p (i) as: p (i) = a (i) ∗ T + b (i)     (i)  (i) (i) wtot k∈M tk ck − tk ck k∈M k∈M a (i) = 2  (i)  wtot k∈M t2k − k∈M tk   (i) (i) k∈M ck − a k∈M tk (i) b = (i) wtot  (i) w(i) (T − tk ) wtot =

(3) (4)

(5) (6)

k∈M

The above is, essentially, a weighted linear regression applied independently for (x, t) and (y, t) and calculated over frames with pose present. w(Δt) is a function specifying weight based on time difference. If w(Δt) has a form of eβΔt , then calculation of p (i) may be simplified to be done at O(1) per frame, (i,j) otherwise, it is achievable in O(|M |) time. Finally, Ccenter may be calculated: (i,j)

Ccenter = αcenter

|p (i) − cj | d(i)

(7)

High-Speed Multi-person Tracking Method Using Bipartite Matching

801

where d(i) is a characteristic distance for track i, calculated as running average of maximum bounding box dimension. (i,j) The second novel term, Cpose , is equal to the average normalized distance between joints present both in last pose in track i and candidate pose j. Its main role is to detail correct selection of track continuation in case if there are several close candidate poses. If J is a set of indices of limbs shared between pose j and last saved pose in track i, and if it is not empty, then it may be calculated as:  (i) |rk − rj,k | αpose  (i,j) (8) min , βpose Cpose = |J| d(i) k∈J One can see that this term is very similar to the PCKh value, the main difference being that it is continuous for nearly matched poses. Furthermore, joints present only in the track or in target pose do not affect the resulting value - this is done to eliminate the likelihood decrease if a person gets partially occluded during motion. 3.4

Visual Feature Matching (i,j)

The Cvis is the term responsible for visual feature matching between tracked persons and new detections. In the related works, the features are usually extracted via a truncated convolutional network, such as VGG or ResNet [8, 9], for object detection. We propose the use of simple color statistics obtained from image pixels in vicinity of detected joints instead. Visual feature term calculation may be split into several parts. First, the pose keypoint data is used to approximate the image region containing the person using simple geometric shapes. Then, a set of feature values is calculated from pixel values for every shape composing this region. Individual temporal statistics for each of these feature values are collected afterwards in order to approximate their distribution and calculate the cost function. Two simple base shapes - ellipses and trapezia - were evaluated for body part approximation, and their performance was found to be nearly identical. Provided a set of image regions representing the i-th person Ri,j , a set of respective numerical features fi,k may be calculated as:  1 mx,y,c S (Ri,j ) (x,y)∈Ri,j

  1  2 = (mx,y,c − fi,6j+c ) S (Ri,j )

fi,6j+c =

fi,6j+3+c

(9)

(10)

(x,y)∈Ri,j

where S (Ri,j ) denotes the area of region Ri,j in pixels. We assume that the input image is a 3-channel RGB image, implying 0 ≤ c ≤ 2. Each j-th region creates 6 numerical features with indices in [6j, 6j + 5]. If some of the regions Ri,j are not present (due to some of keypoints defining

802

A. Gilya-Zetinov et al.

it being omitted by the keypoint detector), then the six respective features are expected to be omitted as well. If a k-th pose was assigned to track i, then for each feature value j present (i) (i) in the pose temporal statistics ηj (estimated mean), σj (estimated mean of squared value) are updated recurrently: (i) (i)

(i) ηj

=

wj ηj

(i) wj

(i)

wj

=

wj σj

(11)

+1

(i) (i)

(i) σj

+ fk,j 2 + fk,j

(i)

wj + 1   (i) = kvis wj + 1

(12) (13)

Primed variables represent variable values in previous frame. If the j-th feature is omitted in the assigned pose, then updates (11), (12) are not applied, (i) (i) and (13) is calculated as wj = kvis wj instead. At track initialization, (i)

ηj

(i)

= σj

(i)

= 0.5, wj

= 1.0 (provided input RGB values are normalized to (i)

be in range 0..1) . Initialization with wj = 0 could lead to an excessively strict distribution approximation during the first iterations. (i) (i) Obtained values of ηj and σj allow us to approximate the standard devi (i) (i) 2 ation of the j-th feature value as σk − ηk . The best results were achieved i,j is calculated as following: if Cvis

(i)

(i)

βvis i,j = Cvis

|fj,k − ηk |   (i) (i) 2 max , σk − ηk    (i) min lj,k , 1



lj,k =

αvis |F |

(14)

(15)

k∈F

where F is a set of indices for features present in detected pose j, and  = 0.0001 is a small value to prevent division by zero or imaginary root values (e.g. due to rounding errors). 3.5

Track Matching and Update

When a new frame is processed, new empty tracks are added until two conditions are met: 1. Number of tracks is larger than or equal to the number of persons. 2. At least one new track has been added.

High-Speed Multi-person Tracking Method Using Bipartite Matching

803

The first requirement ensures that it is possible for every detected person to have an assigned track, as no tracks can be assigned to more than one person. The second requirement allows new tracks to be created, if this leads to higher total likelihood value. After the tracks are added, a Hungarian algorithm is executed, assigning tracks to every detected person in a way minimizing the total  (i(j),j) C . If optimal assignment included all of the newly provided tracks, then j an additional track is created and the process repeats, allowing the creation of more than a single track per frame if necessary. Hungarian algorithm returns an optimal solution to the given assignment problem. Nonetheless, even if the obtained assignment maximizes likelihood function for each two adjacent frames, it is possible that the optimal solution for the whole video will be different. For that reason, an approximate greedy algorithm could obtain slightly better MOTA in some works [11], but in overall the difference is negligible and we limit our research to the Hungarian algorithm only. A track will be removed if it was not used during the previous frames, with time interval Tmax .

4 4.1

Experiments The PoseTrack Dataset

For experiments, we used the validation parts of the PoseTrack 2017 and 2018 datasets, which consist of 50 and 170 short videos respectively [5, 6]. Posetrack 2017 provides annotations of human head bounding boxes and 15 body keypoint locations per labeled person. The annotation format of the 2018 dataset differs slightly and contains two additional “ear” keypoints. Since keypoint definitions do not match exactly any of OpenPose models, we employ the BODY25 OpenPose model for both datasets, while permuting the keypoints. There are three tasks the dataset targets: pose estimation in single frame, pose estimation in video, and video-based pose tracking. The first two tasks are evaluated by per-keypoint Average Precision metrics (mAP) [16], and the last one is evaluated by Multiple Object Tracking Accuracy metrics (MOTA) [14]. To calculate these metrics, we use the evaluation code provided by the authors. MOTA metrics for a keypoint with index i are defined as:  F Nti + F Pti + IDSWti i  (16) M OT A = 1 − t i t GTt where t is the video frame index, F Nti and F Pti the count of false negative and false positive keypoint detections respectively, GTti the count of ground truth annotations, and IDSWti the number of detected identity switches. If it is the tracker performance that is being evaluated and keypoint detections do not vary, then IDSWti is the only component that matters for performance comparison. For that reason, we also present Percentage of Identity Switch values (%IDSW) [17], not originally exposed by the evaluation code:

804

A. Gilya-Zetinov et al.

 i t IDSWt %IDSW =  i t GTt i

(17)

MOTA metrics values commonly found in the literature have a strong dependency on individual keypoint detection performance and, as a result, they can not be compared in order to evaluate the tracker if the keypoint estimation algorithm is different. For that reason, we re-implemented IoU and PCKh based tracking algorithms for evaluation, and applied them on the top of OpenPose predictions, providing us a baseline for algorithm assessment. 4.2

Experiment Details

We used open source implementation of OpenPose based on Caffe framework [1]. All tracking methods evaluated were implemented in C++. The neck joint of BODY25 model was used as the tracker root keypoint, since it occurs the most frequently in the dataset. The best results were achieved with the following parameter values: αcenter = 0.22, αpose = 0.35, αvis = 0.5, βpose = 0.5, βvis = 1.5, kvis = 0.8. Characteristic distance d(i) was calculated as the maximum bounding box dimension multiplied by 0.6. Because video FPS values are not provided in the PoseTrack, maximum track period with missing detections before removal is tied to frame count and equals Tmax = 30 frames. We perform ablation study of this approach by conducting several experi(i,j) (i,j) ments where different C (i,j) are zeroed. First, since terms Ccenter , Cpose are both (i,j) related to trajectory continuity, we performed additional studies with Ccenter = 0 (i,j) and Cpose = 0. In these tests, the relevant parameters were αcenter = 0.0, αpose = 1.4 and αcenter = 0.5, αpose = 0.0, respectively. 4.3

Results

Table 1. PoseTrack validation mAP. Dataset Head Sh.

Elbow Wrist Hip Ankle Total

2017

51.3

73.4 65.7

55.5

60.5 51.1

58.9

2018

50.8

70.9 63.9

54.9

55.8 50.4

57.2

Results of OpenPose mAP evaluation are shown in Table 1. One can note that relative head mAP is lower than average for other methods, which is due to lack of HeadTop equivalent joint. Overall, the keypoint estimation method we use is significantly outperformed by the current state-of-the-art, but it should be noted that OpenPose was not trained on the dataset. The main results of the algorithm evaluation are presented in Table 2 and Table 3, and visualisation of several track occlusion cases in Fig. 5. As can be seen,

High-Speed Multi-person Tracking Method Using Bipartite Matching

805

Fig. 5. Examples of correctly processed person occlusion in video. Different colors stand for different identity Ids.

this tracking approach achieves better MOTA and %IDSW metrics than other methods. If no image-based features are used (Cvis = 0), then the improvement is about 30% (1.9 to 1.3 %IDSW) in comparison with IoU+PCKh. For the version including all the cost function terms, the level of error decreases by approximately 40% (1.9 to 1.1 %IDSW). Elimination of either Ccenter or Cpose increases number of ID switches significantly, indicating that both terms are necessary. In the Table 2 we also present the average runtime of a single frame processing obtained on the PoseTrack 2017 dataset. The runtime includes both cost function calculations and Hungarian algorithm execution, but excludes the keypoint estimation step. If αvis = 0, then per-pixel statistics computation is also included. The evaluation code was run on a single core of Intel(R) Xeon(R) Silver 4108 processor at 1.8 GHz. The obtained results indicate that all methods not involving the pixel value processing are almost equally fast and may be executed up to 5k FPS on the evaluation system. Visual feature matching significantly slows tracking to around 200 FPS on average, but it is still notably faster than any of the current state-of-the-art multi-pose keypoint estimation methods.

806

A. Gilya-Zetinov et al. Table 2. PoseTrack 2017 validation set results. a

Tracking algorithm

MOTA %IDSW Time, ms

IoU-based

50.96

2.1

0.11 ± 0.18

IoU+PCKh-based

51.18

1.9

0.21 ± 0.54

Ours, w/o Ccenter , Cvis 50.39

2.6

0.21 ± 0.49

Ours, w/o Cpose , Cvis

50.91

2.1

0.16 ± 0.43

Ours, w/o Cvis

51.73

1.3

0.23 ± 0.52

Ours 51.89 1.1 5.14 ± 4.67 a Mean and standard deviation of processing time for a single frame. Table 3. PoseTrack 2018 validation set results. Tracking algorithm

MOTA %IDSW

IoU-based

48.27

1.76

IoU+PCKh-based

48.47

1.57

Ours, w/o Ccenter , Cvis 47.78

2.26

Ours, w/o Cpose , Cvis

48.13

1.90

Ours, w/o Cvis

48.90

1.13

Ours

49.00

1.03

To compare accuracy of this tracker with the more computationally intensive state-of-the-art methods, we also evaluated it using annotated ground truth (GT) keypoints instead of OpenPose output. Validating on ground truth keypoints allows to assess the tracking step independently from the keypoint estimation step. These results, together with results of other tracking algorithms applied to GT annotations provided in the paper [17], are presented in Table 4. It can be seen that the proposed tracking method, without visual feature matching, outperforms Graph Convolutional Network from [13]. With the addition of visual feature matching, the method outperforms the Optical Flow-based tracking algorithm [21] as well. The best GT-based tracking result reported is achieved by Pose Entailment method [17], which uses a transformer neural network architecture in order to perform tracking by keypoints.

5 5.1

Discussion Dataset Challenges

After inspecting the videos included in the PoseTrack datasets, we noted several potential challenges affecting the %IDSW value, which may be absent in real world scenarios:

High-Speed Multi-person Tracking Method Using Bipartite Matching

807

Table 4. PoseTrack 2018 validation results on ground truth annotations. Tracking algorithm MOTA %IDSW GCN

98.5

1.5

Ours, w/o Cvis

98.56

1.39

Optical flow

98.7

1.2

Ours

99.07

0.88

Pose entailment

99.3

0.7

Fig. 6. Examples of abrupt single-frame transitions occuring in the PoseTrack17 dataset.

1. Abrupt single-frame camera view changes (Fig. 6). Proposed method was primarily developed to process a continuous video stream from a single fixed camera and usually matches the closest (in frame space) poses before and after the transition, which is treated as wrong identity switch. It is possible to detect these scene changes and e.g. reset all the current track IDs manually, or re-assign tracks through matching visual features only. However, an additional challenge is that, for some videos, all annotated individuals receive different tracking ID before and after view change, even if, it appears, according to human observation, that some individuals are identical. In other cases, the annotation maintains the individuals identity correctly. 2. Camera movement with high acceleration and rotations. In addition to motion blur causing misses in keypoint estimation, rapid unpredictable speed changes (i,j) in the frame coordinate space render the Ccenter term useless. This could be avoided by additional video analysis and compensating for the camera movement using predicted positions.

808

5.2

A. Gilya-Zetinov et al.

Study Limitations and Future Research

We have applied this tracking method to the OpenPose detections and ground truth annotations only, which does not allow to compare results to the public OpenTrack leaderboards. We are planning to combine a keypoint detection stage from one of the state-of-the-art submissions together with the proposed tracking stage to observe if a score improvement may be obtained. Parameter values presented in this paper were found through a manual search in the parameter space. Even though the observed results were robust to small parameter variations, a more correct approach would be to employ a parameter search algorithm to find optimal values. As we have noted in the Sect. 5.1, there is a number of challenges specific to PoseTrack that were not addressed in the tracker design. For example, it is possible to use a simple machine vision algorithm to detect the view changes and (i,j) (i,j) zero location-specific scores (Ccenter and Cpose terms), to encourage matching based on visual features during transitions. Also it is planned to modify the (i,j) Ccenter term to account for the camera motion.

6

Conclusion

In this paper, we have presented a novel pose tracking method for the MultiPerson Pose Tracking problem. It further improves the classic tracking approach based on graph bipartite matching by introduction of a new form of the likelihood function. The proposed method is able to deal with partial and full pose occlusions predicting pose center movement by linear regression. In addition, it includes a simple way to incorporate visual features into the tracking method without relying on supplementary feature extracting neural networks. The experiments have shown that this method outperforms other presented tracking methods based on bipartite matching on the PoseTrack benchmark, while preserving a high execution speed. The achieved result on ground truth annotations is close to the state-of-the-art even among more computationally heavy methods. Acknowledgments. The reported study was funded by Russian Foundation of Basic Researches, project number 19-29-09090.

References 1. Cao, Z., Hidalgo, G., Simon, T., Wei, S.E., Sheikh, Y.: OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. arXiv preprint arXiv:1812.08008 (2008) 2. He, K., Gkioxari, G., Doll´ ar, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017) 3. Luo, W., et al.: Multiple object tracking: a literature review. arXiv preprint arXiv:1409.7618 (2018) 4. Milan, A., Leal-Taix´e, L., Reid, I., Roth, S., Schindler, K.: MOT16: a benchmark for multi-object tracking. arXiv preprint arXiv:1603.00831 (2016)

High-Speed Multi-person Tracking Method Using Bipartite Matching

809

5. Iqbal, U., Milan, A., Gall, J.: PoseTrack: joint multi-person pose estimation and tracking. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, pp. 4654–4663 (2017). https://doi.org/10.1109/CVPR. 2017.495 6. Andriluka, M., et al.: PoseTrack: a benchmark for human pose estimation and tracking. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, pp. 5167–5176 (2018). https://doi.org/10.1109/CVPR. 2018.00542 7. Raaj, Y., Idrees, H., Hidalgo, G., Sheikh, Y.: Efficient online multi-person 2d pose tracking with recurrent spatio-temporal affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4620–4628 (2019) 8. Girdhar, R., Gkioxari, G., Torresani, L., Ramanan, D., Paluri, M., Tran, D.: Simple, efficient and effective keypoint tracking. In: ICCV PoseTrack Workshop (2017) 9. Girdhar, R., Gkioxari, G., Torresani, L., Paluri, M., Tran, D.: Detect-and-track: efficient pose estimation in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 350–359 (2018) 10. Bao, Q., Liu, W., Cheng, Y., Zhou, B., Mei, T.: Pose-guided tracking-by-detection: robust multi-person pose tracking. IEEE Trans. Multimed. (2020) 11. Wen, S. Y., Yen, Y., Chen, A.Y.: Human tracking for facility surveillance. In: Science and Information Conference, pp. 329–338. Springer, Cham (2019) 12. Xiu, Y., Li, J., Wang, H., Fang, Y., Lu, C.: Pose flow: efficient online pose tracking. arXiv preprint arXiv:1802.00977 (2018) 13. Ning, G., Pei, J., Huang, H.: Lighttrack: a generic framework for online top-down human pose tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 1034–1035 (2020) 14. Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the CLEAR MOT metrics. EURASIP J. Image Video Process. 2008, 1–10 (2008) 15. Yang, Y., Ramanan, D.: Articulated human detection with flexible mixtures of parts. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2878–2890 (2012) 16. Pishchulin, L., Insafutdinov, E., Tang, S., Andres, B., Andriluka, M., Gehler, P.V., Schiele, B.: Deepcut: joint subset partition and labeling for multi person pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4929–4937 (2016) 17. Snower, M., Kadav, A., Lai, F., Graf, H.P.: 15 Keypoints is all you need. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6738–6748 (2020) 18. Ramshaw, L., Tarjan, R.E.: On minimum-cost assignments in unbalanced bipartite graphs. HP Labs, Palo Alto, CA, USA, Technical report HPL-2012-40R1 (2021) 19. Wang, M., Tighe, J., Modolo, D.: Combining detection and tracking for human pose estimation in videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11088–11096 (2020) 20. Bochinski, E., Eiselein, V., Sikora, T.: High-speed tracking-by-detection without using image information. In: 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6 (2017) 21. Xiao, B., Wu, H., Wei, Y.: Simple baselines for human pose estimation and tracking. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 466–481 (2018)

Modular Image Synthesizer for Annotated Test Sets on Incremental Parameter Fields Benny Platte(B) , Rico Thomanek, Christian Roschke, Tony Rolletschke, Frank Zimmer, and Marc Ritter Hochschule Mittweida, Technikumplatz 17, 09648 Mittweida, Germany [email protected]

Abstract. There are many ways to improve the quality of indoor localization. A big lever here is the quality and quantity of training data. For the evaluation of algorithms, fully annotated and repeatable accurate training data is essential. In order to reduce the personnel effort for manual annotations, it is advisable to generate these training data automatically. This offers the additional advantage of changing existing scenes in small steps. This enables a finely granular evaluation of algorithms trained with it. For this purpose we developed a M odular S ynthesizer for Annotated T est S ets (MSATS). With MSATS we generate large data sets of completely annotated image data based on publicly available 3D CAD models. The scenes rendered automatically in small-step increments using multidimensional, modular parameter fields. The object annotations are determined from the projected virtual 2D viewer plane using a raycast method and coupled to the 3D data. The composed scenes are presented in orbitals. The resulting sets of images allow us to evaluate object detectors in detail and compare their output with the MSATS output. Keywords: Synthetic ground truth · Image synthesis · Testset generation · Object orbitals · Multidimensional parameter map · Object recognition

1

Introduction, Related Work

Currently the accuracy of indoor localization strongly depends on a large variety of additionally used technology, which must be used in large numbers. It ranges from beacons [6, 16] over electromagnetic field strength distributions of WLAN [13] or Bluetooth [4,8] and last but not least detection of optically active markers [18] or a combination of them [7]. In the context of a larger project we investigate the possibilities of autarkic image-based indoor navigation without the need for external infrastructure. A viable solution can be provided by convolutional neural networks (CNN). Ideally, scene contents are detected and the objects recognized and named. However, purely image-based systems without markers show a stronger dispersion c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 810–822, 2022. https://doi.org/10.1007/978-3-030-80119-9_52

MSATS

811

of the extrapolated position [22]. Die Qualit¨ at eines rein bildbasiertes Systems h¨ angt unmittelbar von der Menge und vor allem der Qualit¨ at der Ground Truth Daten ab. Erst mit hochqualitativen Ground Truth Daten werden fein granulare Performance-Untersuchungen der Algorithmen m¨ oglich. The quality of a purely image-based system depends directly on the amount and, above all, the quality of the ground truth data. Only high-quality ground truth data makes finely granular performance studies of the algorithms possible. A few approaches for furniture datasets can be found in publicly accessible databases including 3D objects and annotations. Examples are IKEA online planer [1], furniture design software [3, 20] or SceneNet [12]. Nevertheless the scope of available data is limited [1, 12], small-step changes are not provided and some are commercially distributed [3, 20]. It is difficult to extract data from these applications that connect 3D data with 2D projections. In addition, there is no influence on technical data of the image sensor. In previous works, Fisher [10] showed a program-controlled placement of furniture combinations. Others, such as Silberman, explored ways of understanding scenes with a Kinect camera. Here, however, no small-step changes were investigated [19] . The data set of SceneNet [14,15] offers many images with annotations including object data. However, a viewport of the camera is missing and design material can not be changed. In consideration of our requirements, publicly available data sets offer little clues. We found many data sets with images of furniture from different angles, but metadata of the viewer corridor projected onto the 2D surface was missing. To overcome these limitations and to create a toolset that is expandable for the future using freely available 3D CAD models, we developed a generic synthesizer pipeline to gather a large amount of object images and associated annotation data. The data pool of generated image data and linked metadata provides universal pixel-perfect ground truth data, as fundamental for image recognition.

2 2.1

Ground Truth Synthesizer Overview

Our goal was to develop a generic system for generating numerous object images with associated metadata. The proposed pipeline consists of the elements as illustrated in Fig. 1: 1. Composing the scene using values stored in a database. This task is processed divided by – Object Composer, – Environment creator and the – Optical Parameteriser 2. Image generation with the set parameters by the Orbital Renderer. 3. Determination of generated metadata (visibility, bounding box) 4. Storage of metadata related to image files

812

B. Platte et al.

Fig. 1. Workflow overview: all data is read from a postgres database. After rendering, pixel-accurate rich metadata are transferred back to the database.

For this we focus on the systematic modification of individual object properties and in addition ambient parameters for office furniture. Think about training data set based on small step shifts in spatial angles, focal lengths, noise performance, light situations, materials. For the rendering we used the software blender [11]. The metadata was stored in a Postgres database [2]. Objects informations based on COCO Dataset [14]. 2.2

System Structure

Figure 1 shows our workflow. The Synthesizer is built around the render module: the szene generator assembles the scene on the basis of the parameters in the individual modules and transfers it to the two staged render engine. The scene is composed of the individual modules Object Composer, Environment Composer and Optical Parameteriser. The base is 3D CAD data of objects. These are equipped with object-related data such as material, reflexivity, color, etc. Possible properties are available in individual tables of the database and can be extended individually. The main focus is on easy extensibility of the individual components. During the run, the individual components are put together in selectable combinations to perform a clean bulk operation. For the light sources blender (possible since v2.8) groups were used. The advantage of these is that all objects and light sources are visible in the user interface for a later user. Users have full visual access and can orient themselves intuitively. Furthermore, users can easily control all light settings and hide certain objects or light settings for individual toolchain runs. This is a comprehensive enhancement of usability. In the next step, this object is integrated into the scene. After correct positioning, scene-related parameters such as light situation and type of lighting are used. The last parameters to be set are the optical properties of the recording

MSATS

813

Fig. 2. Camera fly around the szene (“Ballhaus-Kreisel”). For example 3 elevation angles in different colors are shown (green, red, blue).

system, such as camera distance and focal length. Due to modularization, the system scales with future parameters. The database can also be expanded. 2.3

Rendering and Saving of Metadata

In this state, the scene is passed to the renderer. This renders the image and saves the output. Figure 2 at right shows exemplary output pictures. The reference to the image(s) is stored in the database. The Synthesizer then sends non-recursive virtual rays through each pixel of the rendered image back into the scene for ray–surface intersection tests. Ray casting excludes the possibility of obtaining false results from reflections, refractions or the natural fall of shadows. Each hitt object is saved. Thereafter it is known where the object is located in the finished image after applying all parameters. From this, the bounding box, the object segmentation and the depth information can be determined pixel by pixel. The camera is always moved at the same distance from the selected center of the scene and aligned exactly to it. The viewpoint of the camera is located within the scene. The camera distances can be adjusted as desired. The focal length is adjusted so that the objects are fully visible from any solid angle. Partial visibilities will be the focus of further investigations in the expansion of the system. The pairings of focal length and viewing distance follow the relationship: tan

s gp δ = = 2 2f 2d

−→ d =

f gp s

(1)

where: s = sw = sh = sensor height and width (same on quadratic sensor) f = focal length δ = visual angle, 2δ for both sides of the optical axis d = distance object to lens gp = Size of the projected object in perpendicular plane

814

B. Platte et al.

After a complete Ballhaus-Kreisel the camera moves back to azimuth position 0◦ . Starting from this position, another Ballhaus Kreisel of the camera takes place. At the end of the process the camera captured a complete equidistant orbital around the object and rendered it from all positions. This results in 312 rendered images per orbital, including the associated metadata from the raycast output (see Sect. 2.5). 2.4

Parameter

Light Sources are stored as parameter sets in the database like the other parameters. These light sets allow different illumination of the objects according to the later use in indoor localization situations. Office lighting is usually bright and illuminates the objects evenly. But what happens if the later user of the algorithm has not switched on a light? Or the sun illuminates the scene sharply with partial light and nearly black shadows when viewed at least from the perspective of a mobile image sensor. Or if the electricity has failed: there are many scenarios in which the light is completely different. To cover these cases, we have implemented different lighting parameter fields. These include impractical lighting, such as lighting from below. With these special cases we test the robustness of the recognition algorithms. Object/Background Color and Texture: Proceedings of [21] have shown that when applying detectors to virtual data, the representation of natural backgrounds was not useful. Equally good results were obtained on a white background with uniform materials. [21] explains in [21] that a classifier concentrates on learning the edges of the object shape and is almost indifferent to color and texture. We have therefore decided to make this work without a floor and with a uniform grey background. In the future we will also investigate these interrelationships. The representation without floor has no influence on the view from the upper hemisphere except for the missing shadow. A big advantage here is that the missing floor allows the view of the scene from the lower hemisphere. Mobile Device Specific: For the training of nets on specific environments with high detection rates for objects occurring there, the dimensional parameters can also be varied by activating different transformations such as translation, rotation, zoom, stretch. A major advantage is the possibility of generating deviceappropriate image data by adapting the synthesization to the technical parameters of the acquisition device. Currently, focal length and camera distance are possible. 2.5

Metadata

Before Rendering: First, all relevant scene parameters for each orbital are stored in the database (objects, materials, light setting, sensor size, focal length, image sensor distance). In addition, frame-relevant metadata is stored in a 1:n relationship for each individual frame of each orbital (azimuth, elongation).

MSATS

815

Fig. 3. Comparison of needed rendertimes in the first pipeline run of different objectmaterial-combinations on AMD Radeon Pro 560. It shows the required time in seconds per generated orbital with 312 images each.

After Rendering: The metadata calculation is autonomous, completely bypassing manual annotations. The object annotations are determined from the projected virtual 2D viewer plane using a raycast method. For this a virtual beam emitted by the observer is sent through each pixel of the 2D projection into the 3D scene and its point of impact is determined. The rays of all pixels of the generated 2D image result in a map of the impact points in 3D space. This data is stored as metadata in a database. This data is stored as metadata in a postgres database. The modularization also allows a fast exchange of the database adapters: the output can be adapted as needed. When using the dataset, this ground truth metadata allows a pixel-precise evaluation of the detection results of different object detectors and thus a very accurate statistical estimation of the detection quality. This makes the choice of detection algorithms quantifiable with pixel precision. Even small improvements of a detector become measurable.

3

Dataset Generation First Run

11 chairs and 10 tables as well as 9 further different office furnishings in up to 7 materials each were produced. These combinations were illuminated with 5 light situations each and rendered orbital afterwards. The lighting situations were ranging from global lighting over several light sources to deliberately bad illumination. This run generated about 1035 orbitals with 322920 images. Figure 3 shows the calculation time of the main scenes on a Intel Core i7 with AMD Radeon Pro 560.

816

B. Platte et al.

The annotations for all pictures were stored in the database. Material, light setting, dimensions were stored as well as the screen-projected 2D coordinates from the raycast (see Sect. 2.3). The results of the raycasting shows Fig. 4a exemplarily at the chair from Fig. 2. The polar heatmaps represent a orbital of visibility. Each 15◦ segment of the sphere (312 in total, 156 per hemisphere) shows one value. For the visualization, imagine that this sphere was cut through at the equator and the upper half-sphere was shifted to the right. We now look at the half-spheres from above. The left picture shows the lower half-sphere, into which we look virtually. In the case of Fig. 4a, a value of 1 means that the object seen from this point covers most of the pixels of the rendered image relative to the entire orbital. A value of 0.5 means that from this position 50% of the pixels are visible in relation to the segment with the highest visibility. The object in Fig. 4a shows the highest visibility from the front inclined top (right hemisphere) and from the back inclined bottom. The reason for this is the increase of the visibility of the seat surface on the 2D projection of the camera. When looking from the right or left (90◦ and 270◦ ), the surface normal of the seat and the back of the chair show 90◦ away from the camera. As a result, these positions will show the lowest visibility.

4

Dataset Testing: Running Object Detector on the Generated Dataset

After generating the images, we tested the generated images with an object detector to investigate the influence of different parameter changes. For this first test, we used YOLOv3 [17], which we had already optimized to run at low latency on mobile devices (iPhone 7 in this case). In the future we will evaluate further detectors. We determined the overlap of the pixel-sharp ground truth bounding box of the displayed object with the detection result of the detector. The results are shown as an example Fig. 6. It shows that the ground truth boundingboxes determined by the detector largely correspond to the ground truth boundingboxes (the points are collected at the top of the box). However, a matching bounding box is not necessarily the same as a correct detection result. For example the “table” Class shows high Boundingbox matching, but only in a probability range up to about 0.8. At a higher probability of the detector false positive results are more frequent. Here the detector assumed a wrong class despite a very good boundingbox. This corresponds in this respect to the intuition that a table seen in plan view represents a well delimitable surface, but also cannot necessarily be identified as a table by humans: since the legs are invisible, it could also be a board or a patterned drawing sheet. If all detection results for all yaw and pitch angles are drawn separately, visibility correlations result as shown in Fig. 4b. It shows detection results for a random chair from a random picked material and light combination. Figure 7 shows two light settings in detail, which we had expected contrary:

MSATS

817

Fig. 4. Object visibility and recognition from all discrete spatial angles of the orbital. The position is characterized by the azimuth and elevation of the camera origin.

The most frequent correct detections here are in the upper hemisphere, while the number of false detections is higher in the lower hemisphere. Figure 4c shows the detector results of the typical chair for different light settings. The results are very different, although the underlying situation is the same due to the synthetic generation of the scene except for the lighting. Most detections (true positive plus false positive) are unexpectedly shown by illumination settings “03 - Area low” and “04 - Spot low half”. This lightsettings was included to simulate room lighting where the sun is illuminating objects halfway through a window. Our assumption that this illumination would provide the worst detector results was not confirmed with respect to true positives. On the contrary: this lighting provided the best results in Fig. 8. The expectation was that the most uniform, diffuse illumination from all sides (“global sun”) would provide the best detector results. Surprisingly, this ranked somewhere in the midfield.

818

B. Platte et al. materials

0.7

Cadeira Wood-ok-01 Wood-ok-02 Wood-ok-03 lacquered_steel plastic

0.6

0.5

0.4

0.3

0.2

0.1

0.0 chair

table

desk

Fig. 5. Box-and-whisker plot [5] of detections on all chairs, tables and desks. The distribution of the detection rate of all orbitals of the respective COCO class is shown. Each point represents the detection rate (true positive) of an entire orbital. 1 means 100%. These can be achieved if the object from all space segments of the entire orbital is correctly classified. It can be seen that the detection rate of chairs across all materials is much more reliable than the detection rate of tables.

In general, the detector rates show significantly lower counts when looking at the objects from the lower half-sphere. This behavior corresponds to the expectation of pre-trained nets if they were trained on the Imagenet Dataset [9]. The images contained therein show the objects from the usual photo position and thus seldom from below. The three COCO labels currently most strongly represented in the dataset with some objects per class are chair, table and desk. All generated orbitals of these three COCO labels are shown in Fig. 5. Each dot corresponds to one orbital with 312 images each. This shows that the detection rate of the COCO label chair has a median value of 0.48 and the quartile span is 0.2. Starting from the lower quartile boundary, 75% of all orbitals with the label chair have a detection rate of more than 37%, 50% detection rate related to the orbital reach 47% of all chair orbitals and a quarter of all chair orbitals reach detection rates greater than 57%. At the label table these values were much lower. Here half of all orbitals have a detection rate below 4%. A detection rate of at least 20% is only achieved by a quarter. The situation is somewhat better for desks with the COCO label desk. If there are almost a quarter orbitals in table with detection rates less than 2%, the minimum value in desk is at least 5%.

MSATS

819

Fig. 6. Boundingbox intersection-over-union of first run detections on all chairs, tables, desks and lamps. “TP” means true positive count and “FP” means false Positive count.

Fig. 7. Detail of two light settings of an office desk. On the right a selection of the corresponding render images.

The precision-recall-values in Fig. 8 shows the highest detection rate on structured materials (all woods). The lowest detection rate appears on lacquered material. The picture on the right shows the same object class depending on the illumination. Here the global illumination shows the most uniform progress. A lighting from below with shadowing the upper area of the objects provides higher accuracy values surprisingly.

5

Summary and Outlook

The developed MSATS pipeline for ground truth data works - once parameterized - autonomously and generates large quantities of pixel-precise annotated images. A use of large data sets depends on how user-friendly they are. This was achieved by a generic SQL interface. The annotated metadata are stored in a database. The database design allows an uncomplicated query of all data. Modularisation enables low-threshold integration of future applications.

820

B. Platte et al.

Fig. 8. Precision-recall-curve of the object class “chairs”, based on 119267 detections. The left diagram shows the precision for different materials, the right diagram the precision for different light situations. In both cthe average of all values of the other parameter was calculated.

Results of our first tests with object detectors show that vulnerabilities in pre-trained nets can be detected with the images generated from MSATS. These can then be specifically post-trained. Our short example for detection evaluation test with YOLOv3 for detection shows that complex data survey can be realized easily. The generated objects or their images are trivial at first sight. They unleash their magic when they are created in thousands with tiny changes. This offers an approach for the persistent detection of weak points in algorithms. The procedural generation of large amounts of images together with all the 3D metadata available for each individual pixel of the 2D view has proven to be very advantageous. We are still at the beginning here and have many approaches for further work on the MSATS project. It is conceivable that the development of object detectors with regular, automated mass tests based on specifically generated synthetic datasets could be carried out in order to compare them and track their development. In further progress, more detailed scenes are generated and the partial object occlusion is measured pixel sharp. Another challenge is the calculation of a “recognizability index” of objects. This index should give a numerical value, indicating the theoretical possibility to detect an object from the given perspective. The mass generation of finely grained ground truth data makes it possible to find statistical biases of certain combinations and to fill up under-representations of these parameter combinations with synthetic data. We believe that the generation and use of synthetic data will become even more important. This is especially true when there is not enough real rich annotated data, when the collection of real data is difficult due to data protection laws or when statistically imbalanced ground truth data is available.

MSATS

821

References 1. IKEA Planer & Planungshilfen - IKEA (2020). https://www.ikea.com. https:// www.ikea.com/ms/de DE/campaigns/services/planer und ratgeber.html# 2. PostgreSQL: The world’s most advanced open source database (2020). https:// www.postgresql.org 3. Roomeon - Die erste Interior Design Software. Fotorealistisch und in 3D (2020). http://de.roomeon.com. http://de.roomeon.com 4. Paterna, V.C., Aug´e, A.C., Aspas, J.P., Bullones, M.P.: A bluetooth low energy indoor positioning system with channel diversity, weighted trilateration and Kalman filtering. Sensors 17(12) (2017) 5. Chambers, J.M., Cleveland, W.S., Tukey, P.A., Kleiner, B.: Graphical Methods for Data Analysis, 1st edn. Duxbury Press, Belmont, Calif. Boston (1983) 6. Chang, H.-l., Tian, J.-b., Lai, T.-T., Chu, H.-H., Huang, P.: Spinning beacons for precise indoor localization. In: Proceedings of the 6th ACM Conference on Embedded Network Sensor Systems - SenSys 2008, Raleigh, NC, USA, p. 127. ACM Press (2008) 7. Chen, K., Vadde, K.R.: Design and evaluation of an indoor positioning system framework. Technical Report EECS-2016-16, University of California, April 2016 8. Choi, M.-S., Jang, B.: An accurate fingerprinting based indoor positioning algorithm. Int. J. Appl. Eng. Res. 12(1), 86–90 (2017) 9. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, June 2009 10. Fisher, M., Ritchie, D., Savva, M., Funkhouser, T., Hanrahan, P.: Example-based synthesis of 3D object arrangements 31(6), 1 (2012) 11. Blender Foundation. Blender - Free and Open 3D Creation Software 12. Handa, A., Patraucean, V., Stent, S., Cipolla, R.: SceneNet: understanding real world indoor scenes with synthetic data. arXiv:1511.07041 [cs], November 2015 13. Luo, J., Liang, F.: A smartphone indoor localization algorithm based on WLAN location fingerprinting with feature extraction and clustering. Sensors 17(6), 1339 (2017) 14. Lin, T.-Y., et al.: Microsoft COCO: Common Objects in Context. arXiv:1405.0312 [cs], May 2014 15. McCormac, J., Handa, A., Leutenegger, S.: SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth, December 2016 16. Miekk-oja, V.: Static beacons based indoor positioning method for improving room-level accuracy. PhD thesis, University, School of Electrical Engineering, August 2015 17. Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement. arXiv:1804.02767 [cs], April 2018 18. Shahjalal, M., et al.: An implementation approach and performance analysis of image sensor based multilateral indoor localization and navigation system. Wirel. Commun. Mobile Comput. 2018, 1–13 (2018) 19. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33715-4 54 20. BeLight Software. Live Home 3D (2020). https://www.livehome3d.com. Accessed 28 Feb 2019

822

B. Platte et al.

21. Sun, B., Saenko, k.: from virtual to reality: fast adaptation of virtual object detectors to real domains. In: Proceedings of the British Machine Vision Conference 2014, pp. 82.1–82.12, Nottingham. British Machine Vision Association (2014) 22. Werner, M., Kessel, M., Marouane, C.: Indoor positioning using smartphone camera. In: 2011 International Conference on Indoor Positioning and Indoor Navigation, Guimaraes, Portugal, pp. 1–6. IEEE, September 2011

Learning Action Correlation and Temporal Aggregation for Group Representation Haoting Wang1 , Kan Li1(B) , and Xin Niu2 1

School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China {3120181035,likan}@bit.edu.cn 2 Science and Technology on Parallel and Distributed Laboratory, College of Computer, National University of Defense Technology, Changsha, China Abstract. In this work, we propose a deep graph model for collective activity recognition. Based on person’s visual embedding, we explore action correlation to construct the contextual information through GNN reasoning. Our proposed layer generates local evolution descriptor for each person, which contains action correlation and spatial information. Besides, we design temporal aggregation module to encode them into a meta-action space and then aggregate these descriptors to construct final group representation for collective activity recognition. We conduct experiments on two collective activity recognition datasets (collective activity dataset and volleyball dataset) and achieve 89.6% and 91.3% recognition accuracy respectively, which outperforms the compared state-of-the-art methods. Empirical results on collective recognition demonstrate that the effectiveness of learning action correlation and temporal aggregation for video-level group representation. Keywords: Collective activity recognition Temporal aggregation

1

· Action correlation ·

Introduction

Collective activity recognition receives a significant amount of attention nowadays due to its wide applications in many areas, such as surveillance, sports video analysis, and social behavior understanding. Inspired by the remarkable successes of Convolutional Neural Networks (CNNs) [11, 14] for classification of objects in the still image, deep CNNs have been applied widely in the field of single-person activity recognition [22, 25, 28] in the trimmed video. Different from human action recognition that only concentrates on the spatio-temporal dynamics of one or two persons, collective activity recognition further requires understanding the complicated pairwise interactions among many persons and the scene-level context over global scene. Moreover, various behavior of each person not only depend on its spatio-temporal features but also on the various degrees of interactions propagated by relevant persons. Therefore, the key c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 823–833, 2022. https://doi.org/10.1007/978-3-030-80119-9_53

824

H. Wang et al.

challenge in recognizing collective activity is how to construct spatio-temporal representation with action correlation for collective activity. In order to solve the challenge mentioned above, we introduce action correlation to build contextual information for each person and trainable aggregation to learn video-level activity descriptors with sparse graph-structured data. In a real scene, collective activity intensely relies on relation structures over collective scene. For instance, there are several key participants as ones with “spike” and “block” in the volleyball game, and some separated subgroups attending the same meta-activity like “walking” and “talking” in the collective scene. This phenomenon shows that not all pairs have an important connection and everyone locally influences others to varying degrees. Based on this observation, we argue that action correlation is an important cue to form contextual structure, which contains more discriminative information than raw visual embedding. In our model, we utilize action correlation to generate local contextual descriptors for each person through GNN model. Besides, we design a temporal aggregation to encode them into a meta-action in several frames and then aggregate these meta descriptors into final group representation for collective activity recognition. Our method achieves a good performance compared with the state-of-the-art models on two real world datasets. In summary, the contributions of the paper are given as follows. 1. We design action correlation for modeling the context of person through GNN reasoning. Our proposed graph model generates local evolution descriptors for each person, which contain action correlation and spatial information, and thus can improve the performance of collective activity recognition. 2. We present a temporal aggregation module to learn global video-level activity representation. It encodes local evolution descriptors into the semantic meta-action representation and generate global video-level feature. 3. We conduct experiments to evaluate the performance of two publicly available datasets. The results demonstrate show that the performance of our model is competitive with that of the state-of-the-art methods.

2

Related Work

In the past few years, many research works have been devoted to model the global representation for collective activity recognition. Most recent methods address this challenges with a two-stage recognition pipeline: 1) extract visual features(appearance feature and pose estimation) for each person; 2) aggregate multiple person’s features into an integrated spatial-temporal representation for collective activity. Traditional approaches construct probability graph models [3,5,7,9, 15,16, 19], AND-OR grammar models [2,21] and dynamic Bayesian networks [27] based on hand-crafted spatio-temporal features to achieve activity recognition task. Since the shallow features employed in these approaches could not encode higher-level information, many deep learning architectures [8,12, 17,26] have been proposed to use deep convolutional features for collective

Action Correlation and Temporal Aggregation

825

activity recognition. Recurrent neural network is popularly utilized to explore spatio-temporal evolutions for collective activity with densely sampled frames. For example, Deng et al. [8] utilize a recurrent neural network for message passing that refines estimates of an individual person’s action and performs structure learning via gating functions that determine connections between people in a scene. Ibrahim et al. [12], build a hierarchical structured model that incorporates a deep LSTM framework to capture the temporal dynamics of the whole activity. Qi et al. [17] presented a novel RNN framework with semantic graph and spatio-temporal attention that could explicitly extract spatio-temporal interobject relationships in a dynamic scene. Wang et al. [26] proposed a hierarchical recurrent interactional context encoding framework to model interactional context features. By analyzing the aforementioned RNN-style model, spatiotemporal dynamics and interactions between people are definitely two crucial cues for recognizing collective activity.

3

Method

In this section, we present an overview of our proposed model before going into details. The overview of the architecture is shown in Fig. 1. Given by a sequence of person’s bounding boxes and frame as inputs, we firstly extract visual feature for each person, and then feed them into our proposed graph model with action correlation, called local relation graph. These stacked layers adaptively and jointly determine action correlation among people and construct the semantic graph which contains both visual and contextual information. At last, we design a trainable aggregation module to project the local evolution descriptors into a semantic space and generate more discriminative video-level representations, leading to high-accuracy recognition for collective activity tasks.

Fig. 1. The overview of our model architecture.

826

3.1

H. Wang et al.

Visual Feature Extraction

Before performing graph layer reasoning, visual feature for each person is required to apply hand-craft or deep methods for extraction. For a trimmed video, we decompose it into several frames [I1, I2, · · · , IN ] and each frame contain N bounding boxes to look person’s descriptors. Inspired by the remarkable successes of deep model in object recognition, we chose deep methods for visual feature extraction. This procedure takes three key steps. Firstly, Given by a sequence frame of a video, a deep convolutional neural network is trained to explore visual feature of the whole scene in a still frame. Then inspired by the design of RPN module [18], feature map of each individual is extracted through RoIAlign [10] on the scene-level representation resulting from the backbone network. Lastly, we flatten aligned individual representation at different convolution layers and feed it into a fully connected layer to get a spatial visual feature. 3.2

Action Correlation

In this section, we give detailed descriptions of reasoning over graph model with action correlation. The entire reasoning process can be decomposed into three key components: interaction scope, relation encoding and information fusion. Additionally, every person is a node in our graph layer. Similar to ‘attention’ mechanisms [23] recently used in neural networks, relation encoding and interaction scope tells information fusion module which pair of relation needs more attention or neglect, without relying on understanding the graph structure in advance. In the process of information fusion, each node gather interactive information from neighboring nodes to enrich itself raw representation. Based on this idea, we define a integral formulation for a single graph layer with action correlation as follows: 





eR(hi ,hj ) h )  R(hi ,hk ) j e k∈N (i) j∈N (i)

hi = hi + γ · σ(

(1)

where the input to our layer is a set of node features h = [h1 , h2 , · · · , hN ] and N is the number of nodes in a scene. The output h = [h1 , h2 , · · · , hN ] is generated by a combination of hidden representation of local observations with “shortcut connection” [11]. In this way, the node feature retains original semantic information and aggregates local interactions with a deep residual learning framework. During the aggregation process, R(hi , hj ) is relation encoding function to explore action correlation between two nodes. In addition, the input of first graph layer is initialized with deep visual features introduced in Sect. 3.1. Interaction Scope. Without explicit annotated edge between nodes, it is necessary to define a range of spatial position involved in information aggregation, thereby limiting the aggregation computation within a local area. In this module, we take center of bounding boxing of each person as the position of nodes and define the first-order neighbors of node i as follow:

Action Correlation and Temporal Aggregation

Ni = {j | d(Pi , Pj ) < μ, i = j}

827

(2)

where Pi is the position of node i, d(Pi , Pj ) denotes the euclidean distance between nodes and μ is a hyper-parameter which acts as a distance threshold. In our experiments, we set it to be 1/4 of the image height. This module masks the interactions from distant nodes, which is less significant than interactions between locally near nodes. Relation Encoding. The original GCN layer uses a static and non-adaptive propagation which does not change over the layers. In collective activity recognition task, it is hard to represent the relation between two neighboring nodes through a simple connection. Inspired by [24], we design a relation encoding function R(hi , hj ) to adaptively determine the relevance degree of nodes as follow: (3) R(hi , hj ) = LeakyReLU (W [φi (hi )φj (hj )] + b) Where  is the concatenation operation, W and b are learnable weights that project the concatenated vector to a scalar, followed by a LeakyReLU nonlinearity (with negative input slope α = 0.2). φi is a learnable linear transformation on node i, which refines the low-level spatio-temporal features. To make relation encoding easily comparable across different nodes, we normalize them across all choices of j using the softmax function: αij = sof tmaxj (R(hi , hj )) = 

eR(hi ,hj ) R(hi ,hk ) k∈Ni e

(4)

where αij is the soft-assignment contribution from node j to node i. Thanks to relation encoding, every node engaged in unequal interaction and LRG adaptively captures more discriminative semantic graph. Information Fusion. Contextual information for each node is a crucial cue for collective activity recognition. To construct contextual information, this module computes a combination of semantic representation of neighboring nodes and updates itself with a residual block as follows:  αij hj ) (5) hi = hi + γ · σ( j∈Ni

where σ is an activation function and set to ReLU, γ is initialized as 0 and gradually learns to assignment more weight [29]. The resulting hidden representation for each node is a weighted sum of representations from a local observation and original representation. Therefore, it has local contextual view and selectively aggregates information according to the normalized local relation.

828

3.3

H. Wang et al.

Temporal Aggeration

Through our graph layer with action correlation, we obtain a collection of local evolution descriptors for video. Consider hi,t , a local evolution extracted from spatial local i ∈ {1 · · · N } and frame t ∈ {1 · · · T } of a video. In order to construct final video-level representation, we would like to aggregate these descriptors temporally over the whole video. This is achieved by first dividing the descriptor space into K meta-action cells using a vocabulary [a1 , a2, · · · , aK ]. Each local evolution descriptor hi,t is then assigned to one of the cells. Our proposed aggregation stores the sum of residuals for each meta-action cell, where hi,t − ak recording the residual between local evolution descriptors and meta-actions, as follows: N T   e−βhi,t −ak   −βh −a   (hi,t − ak ) (6) Vk = i,t k k e t=1 i=1 where the first term in the two summing operators is the soft-assignment of descriptor hi,t to ak , β is a tunable hyper-parameter. Our aggregation module combine the first order statistics of residual hi,t − ak in the different parts of the descriptor space weighted by the soft-assignment of descriptor hi,t to k-th metaaction. Finally, we stack the K residual descriptors into a vector to represent the entire video. The two summing operators represent aggregation over time and space, respectively. The output is a matrix [V1 , V2 , · · · , VK ], where Vk represents aggregated descriptor in the k-th cell. 3.4

Inference

To train graph layers, all local evolution descriptors are temporally aggregated together to form activity representation. The local evolution descriptors and activity representation are fed into two classifiers to predict person’s action and collective activity, respectively. We formulate the overall objective function with a regularized crossentropy loss of collective activity and person’s action as follows: Ni C C   1  ygi log yˆgi − λ1 yaj log yˆaj + λ2 W  (7) L=− N i i=1 i=1 j=1 The first and second terms in Eq. 7 are respectively the crossentropy losses for collective activity and person’s action, and the last term regularizes all the parameters of our model. ygi and yaj denote respectively the ground-truth label of collective activity and person’s action. Ni is the number of person detections in i-th frame. λ1 and λ2 balance the l2 regularizer and the losses for collective activity and person’s actions. In the process of training the temporal aggregation, we freeze parameters of other modules, and set λ1 = 0 to learn merely video-level representation.

Action Correlation and Temporal Aggregation

4

829

Experiments

We experiment on two widely-adopted datasets, namely, Volleyball dataset and Collective Activity dataset for the task of collective activity recognition, and introduce the implementation details of our approach. 4.1

Datasets

Collective Dataset. The Collective Activity dataset [6] consists of 44 short video sequences (about 2500 frames) from 5 group activities (waiting, crossing, walking, queueing and talking) and 6 individual actions (NA, waiting, crossing, walking, queueing and talking). The activities that most people participate in to define the group activity label for the frame. In the light of the same evaluation scheme as [17], 1/3 video sequence is tested, and the rest is for training. Volleyball Dataset. The Volleyball dataset [12] consists of 4830 clips collected from 55 volleyball matches, including 3493 training clips and 1337 testing clips. Every clip is noted with 8 group activity categories (i.e., right winpoint, right pass, right spike, right set, left winpoint, left pass, left spike and left set), while the middle frame utilizes 9 individual action labels (i.e., falling, spiking, waiting, setting, blocking, jumping, moving, digging and standing) to label the player’s bounding box and their individual actions. Corresponding to 5 frames before the annotated frame and 4 frames after, our model is trained and tested with 10 frames following [12]. The track data provided by [4] is adopted to obtain the boundary boxes of the ground truth of the uncommented frames. 4.2

Implementation Details

In this section, we show the implementation details of our model. We implement our method based on deep learning framework PyTorch [1]. During training, we randomly sample 3 frames for the video snipets. We adopt resnet as basic network and use RoIAlign to extract 256-d visual feature for each person. The 4 graph layers with action correlation are employed in our graph model. The proposed models are trained using the mini-batch stochastic gradient with ADAM [13] and we fixed the hyper parameters β1 = 0.9, β2 = 0.999,  = 1e−8. In all experiments, we respectively set the learning rate and the momentum to be 1e−4 and 0.9. The learning rate was exponentially decayed over time. We take 16 videos per batch, and train them for 120 epochs. Following [4], we train our model in three stages: we firstly fine-tune the ImageNet pretrained on single frames. Secondly, we freeze the backbone network and train our model with multiple graph layers. Lastly, we train merely temporal aggregation module with K = 16. The balance weight λ1 of person’s action is set to be 1. We train it on a PC with 3.4 GHz CPU, four 1080Ti GPUs, and 128G RAM.

830

H. Wang et al.

Table 1. Comparisons of recognition accuracy (%) on Volleyball dataset and CAD dataset

4.3

Method

Collective activity Volleyball activity

baseline

86.5

baseline+AC

89.6

91.3

baseline+AC+TA 90.3

91.8

89.9

Results and Analysis

Ablation Study. To examine the effectiveness of each component in the proposed method, we conduct ablation studies with the following baseline and variants: baseline: This baseline is resnet model fine-tuned for person action and group activity recognition in a single frame. baseline+AC: This variant freezes the backbone network, and train multiple graph layers with action correlation to generate local evolution descriptor for each person. baseline+AC+TA: Compared with the former, this variant add temporal aggregation to construct the video-level representation to recognize collective activity. Table 1 shows the performance of all the variants on two benchmarks. As shown in the table, our proposed model significantly outperforms the baseline model. And we can observe that taking both action correlation and temporal structure into account is a promising way to improve the performance of collective activity recognition. Table 2. Comparisons of recognition accuracy (%) on Volleyball dataset and CAD dataset. Method

Collective activity Volleyball activity Volleyball action

HDTM [12]

81.5

81.9

–

Deep structure model [8] 80.6

–

–

CERN [20]

87.2

83.3

69.0

stagNet [17]

89.1

89.3

–

SSU-temporal [4]

–

89.9

82.4

ARG [4]

90.1

92.0

83.0

ours

90.3

91.8

83.5

Action Correlation and Temporal Aggregation

831

Results on the Collective Activity Dataset. As shown in Table 2, our accuracy of recognizing activity on Collective activity dataset is 89.6%. The classification performance of our proposed model outperforms that of the compared state-of-the-art methods. This result shows that our model can adaptively capture action correlation and build a more discriminative semantic representation for collective activity. In addition, the confusion matrix of our model on the collective activity dataset is shown in Fig. 2(a). We can see that our model has solved the recognition of ‘queueing’ and ‘taking’ with 100% recognition accuracies, which proves the effectiveness of our mate-action mechanism. However, ‘crossing’ and ‘waiting’ are partly confused with ‘waking’. This is probably because those activities share high similarities of spatio-temporal dynamics and action correlation structures.

(a) Collective Activity Dataset

(b) Volleyball Dataset

Fig. 2. Confusion matrices for the two collective activity datasets.

Results on the Volleyball Dataset. The recognition results on the volleyball dataset of our method and the state-of-the-art ones are also shown in Table 2. It can be seen that the activity recognition and person’s action accuracies are competitive with the compared state-of-the-art methods. It should be noted that classification performance of activity surpasses that of person’s action by about 10%–20% in CERN, SSU-temporal and our model. We draw the confusion matrix for volleyball dataset in Fig. 2(b). Our model achieves promising recognition accuracies on the most activities (i.e., right winpoint, right pass, right spike, right set, winpoint, left, left spike and left set) and there are few failure cases between left activities and right counterpart, which proves that our model can build effective relation structures to describe the scene context. Nevertheless, 11% ‘Lpass’ is mistaken as ‘Lset’, which is probably due to the factor that these two activities have similar appearance and action correlation.

5

Conclusions

We propose a deep graph model with action correlation for collective activity recognition. We explore action correlation to construct the contextual

832

H. Wang et al.

information through GNN reasoning. Our proposed layer generates local evolution descriptor for each person, which contains action correlation and spatial information. Besides, our proposed temporal aggregation capture detailed evolution information temporally over video and construct semantic activity representation. Our end-to-end model achieves state-of-the-art results on two widelyadopted datasets, namely, collective activity dataset and volleyball dataset. The result shows that exploiting action correlation and spatio-temporal structure is a promising way to improve the accuracy of collective activity recognition. Acknowledgments. This research is supported by Beijing Natural Science Foundation (No. L181010 and 4172054), National Key R&D Program of China (No. 2016YFB0801100).

References 1. Pytorch. http://pytorch.org. Accessed 31 May 2020 2. Amer, M.R., Xie, D., Zhao, M., Todorovic, S., Zhu, S.C.: Cost-sensitive topdown/bottom-up inference for multiscale activity recognition. In: European Conference on Computer Vision, pp. 187–200. Springer (2012) 3. Amer, M.R., Lei, P., Todorovic, S.: Hirf: hierarchical random field for collective activity recognition in videos. In: European Conference on Computer Vision, pp. 572–585. Springer (2014) 4. Bagautdinov, T., Alahi, A., Fleuret, F., Fua, P., Savarese, S.: Social scene understanding: end-to-end multi-person action localization and collective activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4315–4324 (2017) 5. Choi, W., Savarese, S.: Understanding collective activities of people from videos. IEEE Trans. Pattern Anal. Mach. Intell. 36(6), 1242–1257 (2013) 6. Choi, W., Shahid, K., Savarese, S.: What are they doing?: collective activity classification using spatio-temporal relationship among people. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pp. 1282–1289. IEEE (2009) 7. Choi, W., Shahid, K., Savarese, S.: Learning context for collective activity recognition. In: CVPR 2011, pp. 3273–3280. IEEE (2011) 8. Deng, Z., Vahdat, A., Hu, H., Mori, G.: Structure inference machines: recurrent neural networks for analyzing relations in group activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4772–4781 (2016) 9. Hajimirsadeghi, H., Yan, W., Vahdat, A., Mori, G.: Visual recognition by counting instances: a multi-instance cardinality potential kernel. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2596–2605 (2015) 10. He, K., Gkioxari, G., Doll´ ar, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017) 11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016) 12. Ibrahim, M.S., Muralidharan, S., Deng, Z., Vahdat, A., Mori, G.: A hierarchical deep temporal model for group activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1971–1980 (2016)

Action Correlation and Temporal Aggregation

833

13. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 14. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012) 15. Lan, T., Sigal, L., Mori, G.: Social roles in hierarchical models for human activity recognition. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1354–1361. IEEE (2012) 16. Lan, T., Wang, Y., Yang, W., Robinovitch, S.N., Mori, G.: Discriminative latent models for recognizing contextual group activities. IEEE Trans. Pattern Anal. Mach. Intell. 34(8), 1549–1562 (2011) 17. Qi, M., Qin, J., Li, A., Wang, Y., Luo, J., Van Gool, L.: stagNet: an attentive semantic RNN for group activity recognition. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 101–117 (2018) 18. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015) 19. Ryoo, M.S., Aggarwal, J.K.: Stochastic representation and recognition of high-level group activities: describing structural uncertainties in human activities. In: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, p. 11. IEEE (2009) 20. Shu, T., Todorovic, S., Zhu, S.C.: CERN: confidence-energy recurrent network for group activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5523–5531 (2017) 21. Shu, T., Xie, D., Rothrock, B., Todorovic, S., Chun Zhu, S.: Joint inference of groups, events and human roles in aerial videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4576–4584 (2015) 22. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in Neural Information Processing Systems, pp. 568– 576 (2014) 23. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017) 24. Veliˇckovi´c, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017) 25. Wang, L., et al.: Temporal segment networks: towards good practices for deep action recognition. In: European Conference on Computer Vision, pp. 20–36. Springer (2016) 26. Wang, M., Ni, B., Yang, X.: Recurrent modeling of interaction context for collective activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3048–3056 (2017) 27. Xiang, T., Gong, S.: Beyond tracking: modelling activity and understanding behaviour. Int. J. Comput. Vis. 67(1), 21–51 (2006) 28. Yue-Hei Ng, J., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., Toderici, G.: Beyond short snippets: deep networks for video classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4694–4702 (2015) 29. Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. arXiv preprint arXiv:1805.08318 (2018)

Self-adaptive Perception Model for Action Segment Detection Jiahe Li1 , Kan Li1(B) , and Xin Niu2 1

Beijing Institute of Technology, 5 South Zhongguancun Street, Haidian District, Beijing, China {3120181000,likan}@bit.edu.cn 2 Science and Technology on Parallel and Distributed Laboratory, College of Computer, National University of Defense Technology, Changsha, China

Abstract. Action segment detection is an important yet challenging problem, since we need to localize the proposals which contain an action instance in a long untrimmed video with arbitrary length and random position. This task requires us not only to find the precise moment of starting and ending of an action instance, but also to detect action instances as many as possible. We propose a new model Self-Adaptive Perception to address this problem. We predict the action boundaries by classifying start and end of each action as separate components, allowing our model to predict the starting and ending boundaries roughly and generate candidate proposals. We evaluate each candidate proposals by a novel and flexible architecture called Discriminator. It can extract enough semantic information and generate precise confidence score of whether a proposal contains an action within its region, which benefit from the self-adaptive architecture. We conduct solid and rich experiments on large dataset Activity-Net, the result shows that our method achieves a competitive performance, outperforming most published stateof-the-art method in the field. And further experiments demonstrate the effect of each module of our model.

Keywords: Deep learning

1

· Video analysis · Action detection

Introduction

Action detection is an important branch of video analysis. This task can be divided into two main tasks: (1) Proposal generation and (2) Classification. Given an untrimmed video, our system needs to detect whether there are any action instances or not. If an action instance is detected, we need to figure out when it happens and ends. The relationship between action recognition and action detection is similar to image recognition and object detection. The precision of action recognition and image classification can both reach up to 90%, however the state-of-the-art mAP of action detection still has a huge performance gap comparing to the task of object detection. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 834–845, 2022. https://doi.org/10.1007/978-3-030-80119-9_54

Self-adaptive Perception Model for Action Segment Detection

835

Recently, deep learning methods improve the performance [1, 3, 6] by learning the RGB and optical flow feature representations automatically from the video data. Inspired by some objection detection methods like Faster R-CNN [13], some work proposes the method using anchor-based scheme to generate flexible proposals. However, according to the variable scale and flexible position of action instance, the utilization of anchor boxes or candidate proposals has some drawbacks. These methods will absolutely cause huge amount of computation. Besides, the scale of action instances is various. Although we can use anchor mechanism to enumerate possible locations, scales and aspect ratios for the search of the objects, their performance and generalization ability are also limited which is caused by the design of anchor mechanism. Considering the flexible scale and the various position of action instances, it’s such a challenging task to get a high recall with fewer candidate proposals. Two-stage style architecture is very classical for action detection task. But this style still has some obstacle while detecting the candidate action proposals. The proposal evaluation procedure receive the candidate proposals and generate the confidence score for us to select the proposal which is more possible to contain a complete action instance. However it’s challenging to summarize a candidate proposal and feed it into CNN model, because the lengths of proposals are variable. Inspired by [8], we adopt a new architecture to adapt variant length of video segment. Besides, we also improve the boundary prediction module to make more precise prediction of boundaries. For details, we inject position embedding which contains the temporal information into the clip-level visual feature. And we also use a perception model to directly find the scene change position using HSV distribution, because a scene change always indicates the starting or ending of an action instance. To this end, we propose a self-adaptive boundary sensitive network to generate proposals with higher quality. We test our model for action proposal generation on Activity-Net dataset, and the result demonstrate that our outperforms most models. In summary, the contributions of this paper are list as follows: 1. We proposed a deep model to address the task of action segment detection in local-to-global fashion. It localize the action instance by detecting boundaries and generating candidate proposals. 2. We proposed a novel and flexible component called Discriminator which can generate more precise confidence score of proposals with arbitrary length, so that we can evaluate the proposals correctly 3. Our proposed method is evaluated on the Activity-Net dataset. We achieve competitive results on action segment detection baselines, and present ablation studies to demonstrate the contributions of each component of our proposed model.

836

2 2.1

J. Li et al.

Related Work Action Analysis

There is a long history of research in video analysis, including action recognition and detection, which is a important branch in computer vision area. Previously, some method like improved Dense Trajectory(iDT) can make handcraft feature. Recent years, Deep neural networks learn representations automatically from raw inputs to recover the compositional hierarchies in many natural signals, and are widely adopted in many works of this area. Typically, there are two mainstream method. One is the 3D convolutional method. These methods [16] adopt 3D convolutional layers to directly capture both appearance and motion features from raw frames volume. The other methods [4, 15, 18] are two-stream network, which learns appearance and motion features based on RGB frame and optical flow field separately. 2.2

Action Detection

Action detection task not only requires networks to figure out what the action is, but also localize the position when it happens. Some previous methods take it as a two-stage job, while some others make take the two-stage jointly. Gao et al. [6] follow the faster R-CNN architecture and transfer it into this area to get a nice performance sequentially. Shou et al. [14] use the sliding-window with various scale to cover all the possible proposals, in order to raise the recall. However, the scheme of anchor boxes is not the optimal way to solve this task. Recently, some novel ways are proposed. Lin et al. [12] use a two-stage deep architecture to localize the precise temporal boundaries and also the likelihood of being actions directly, which brings the “local to global” fashion. Some methods using slidingwindow design also achieve a competitive performance.

3 3.1

Method Problem Formulation

We can denote an untrimmed video as v = {x1 , x2 , · · · , xn } ∈ V where xt denotes the frame at timestep t, and n is the total number of frames in the video. Annotation of video V is composed by a set of action instances: M

s ϕp = {ξn = [ts,n , te,n ]}n=1

where Ms is the number of truth action instances in video V , ts,n and te,n are starting and ending time of action instance ξn separately. Our task is to generate a set of candidate proposals which can cover ϕp with high recall and high temporal overlap.

Self-adaptive Perception Model for Action Segment Detection

3.2

837

Our Proposed Model (Fig. 1)

Fig. 1. Our proposed model. (a) Two-stream network that is used to extract snippetlevel visual feature, and position embedding is also equipped via concatenation. (b) Boundary prediction module will receive the clip-level feature as input, and output the starting and ending possibilities of each location. (c) SPP-Net can transfer a proposal of arbitrary size to a feature vector fixed size, so that we can input the clip-level feature into CNN Model. (d) The evaluation module takes the vector of fixed size as input and generates the confidence score of each candidate proposals.

3.3

Visual Encoding (Fig. 2)

Fig. 2. For feature encoding, we adopt two-stream network [19], where BN-Inception [10] network is used as temporal network and ResNet [9] network is used as spatial network. after visual encoding, we embed the position information to characterize the ordering information to each clip-level feature, which can benefit the action detection task.

838

J. Li et al.

We first extract the visual feature of the video, like many mainstream architecture, we adopt two-stream network [19] to extract clip-level feature of the video to encode the visual information. However, the difference is, our deep model needs not only the visual information and, but also the temporal ordering information to make better detection. Inspired by [7,17], we utilize the position embedding to inject the temporal information into our feature sequence. The temporal information of the n-th frame is embedded into a feature defined as   PE(pos,2i) = sin pos/100002i/dmadt   PE(pos,2i+1) = cos pos/100002i/dmadt where pos is the position and i is the dimension, which means each dimension of the positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000 * 2π. 3.4

Boundary Prediction (Fig. 3)

Fig. 3. An example of action detection of weightlifting. our boundary prediction module consists of two component, transitional state detection and scene changing detection. the scene changing detection help us find the ending moment of the weightlifting action is the ending boundary of scene 2, however it is not able to find the starting moment. with the help of transitional state detection, we can get several other possible boundaries and construct the candidate proposals.

We adopt a deep network architecture Transitional State Detector to predict the possibility of starting, ending of each position. The network can be denoted as: Conv(512; 3; Relu)− > Conv(512; 3; Relu)− > Conv(3; 1; Sigmoid)

Self-adaptive Perception Model for Action Segment Detection

839

where the stride is 1. And the two channels of output indicate the possibilities of starting, ending respectively. Besides, we also adopt another component called Temporal Context Detector, to improve the precision and callback further more. We find that a scene change is more possible to indicate the boundary of starting or ending. When scene change comes out, the adjacent frames are obviously different. Such as the background or the environment which is reflected in the HSV distribution. Therefore, we can detect whtn does the scene change comes out by calculating the difference between two adjacent frames. So we adopt a perception model based on HSV (Hue, Saturation, Value). We can directly get the RGB value from a frame, and then transform them into HSV value. For details of the perception: max = M ax(r, g, b) min = M in(r, g, b) v = max/255  s=

h =

(max − min)/max, max! = 0 0, otherwise

⎧ ⎨

(g − b)/(max − min), r = max 2 + (b − r)/(max − min), g = max ⎩ 4 + (r − g)/(max − min), otherwise h = 180 × h/π

Then we can get HSV value following: ⎧ 0, ⎪ ⎪ ⎪ ⎪ 1, ⎪ ⎪ ⎪ ⎪ 2, ⎪ ⎪ ⎨ 3, H= 4, ⎪ ⎪ ⎪ ⎪ 5, ⎪ ⎪ ⎪ ⎪ 6, ⎪ ⎪ ⎩ 7,

h ∈ [316, 20) h ∈ [20, 40) ⎧ h ∈ [40, 75) ⎨ 0, h ∈ [75, 155) , S = 1, h ∈ [155, 190) ⎩ 2, h ∈ [190, 270) h ∈ [270, 295) h ∈ [295, 316)

⎧ s ∈ [0, 0.2) ⎨ 0, s ∈ [0.2, 0.7) , V = 1, ⎩ s ∈ [0.7, 1) 2,

v ∈ [0, 0.2) v ∈ [0.2, 0.7) v ∈ [0.7, 1)

Then we can calculate the value of L : L = 9H + 3S + V The value of L ranges from 0 to 71, and then we can calculate the similarity of two frames. ai = Ai /

L k=1

Ak

840

J. Li et al.

Similar(A, B) =

L

min (ai , bi )

i=1

Where Ai is the amount of the pixels whose L values Li. If the similarity of two frame is less than m, the scene cut happens, otherwise, nothing to do. 3.5

Discriminator

The length of candidate proposals is variable, but we can only input vectors into CNN with fixed size, so we propose a new design, Discriminator to determine whether a proposal contains an action instance. We design this module in order to receive rich semantic information from candidate proposal and make evaluation more precise. Considering the length of candidate proposals is variable, we adopt SPP-Net [8] for its ability to transfer a proposal of arbitrary size to a fixed size, so that we can input the window into the CNN model. This module can be defined as: Linear(channels)− > Linear(1024)− > Linear(1) where the channel denotes the dimension of the vector generated by the SPP-Net. And the output is the confidence which scales from 0 to 1. 3.6

Learning

For boundary prediction module, the training loss is defined as + Lend LBP = Lstart bp bp We adopt the sum of binary logistic regression loss which can be denoted as : LBP =

length 1 (gi · log (pi ) + (1 − gi ) · log (1 − pi )) length i=1

We denote the starting boundary area as [i − dt, i + dt], if the position is in the area, it will be labeled as 1, otherwise 0. Same for ending boundary labeling. For proposal evaluation module, the training loss is defined as LD =

N 1 2 (pconf,i − giou,i ) N i=1

where the groundtruth giou is defined as the maximum of IoUs with all the action instances in the video, N is the amount of candidate proposals for training.

Self-adaptive Perception Model for Action Segment Detection

4 4.1

841

Experiments Dataset and Setup

Dataset: We evaluate the quality of our model on a large-scale video benchmark for human activity understanding. This dataset consists of more than 648 h of untrimmed videos from a total of about 20K videos. It contains 200 different daily categories activities such as: ‘walking the dog’, ‘long jump’, and ‘vacuuming floor’. The distribution among training, validation, and testing is 50%, 25%, and 25% of the total videos, respectively. We train our model on training dataset and validate the quality on the validation set for which the annotations of testing dataset are not available. Evaluation Metrics: Interpolated Average Precision (AP) is used as the metric for evaluating the results on each activity category. Then, the AP is averaged over all the activity categories (mAP). To determine if a detection is a true positive, we inspect the temporal intersection over union (tIoU) with a ground truth segment, and check whether or not it is greater or equal to a given threshold (e.g. tIoU = 0.5). The official metric used in this task is the average mAP, which is defined as the mean of all mAP values computed with tIoU thresholds between 0.5 and 0.95 (inclusive) with a step size of 0.05. 4.2

Details

Our task is intended to generate high quality action proposals. We completed our experiment on PyTorch 1.0. In the preprocessing stage, the pretrained twostream model TSN is used to extract visual feature. We follow BSN’s experiment to generate the feature sequence of each video to new length L = 100 by linear interpolation, and also position embedding is injected into the feature sequence. In this work, we use sine and cosine functions of different frequencies to embed the position. After generating the possibility sequence, we choose the candidate starting and ending boundaries only if p > 0.9 or p is a peak. Then we construct all candidate proposals, and fetch the corresponding feature sequence. Then we feed the feature sequence of arbitrary size into the Discriminator, and get the vector of fixed size. We set the Discriminator component to four layers, so we get the fixed length of 85. After the Discriminator calculates the confidence score for each candidate proposal, we adapt Soft-NMS, and set the threshold to 0.8 by empirical validation. 4.3

Performance

To evaluate the proposal quality, we adopt different IoU thresholds to calculate the average recall (AR) with average number of proposals (AN). A set of IoU thresholds [0.5:0.05:0.95] is used on ActivityNet-1.3. For ActivityNet-1.3, area under the AR vs. AN curve (AUC) is also used as the evaluation metrics (see Fig. 4 and Table 1).

842

J. Li et al.

Fig. 4. AR-AN on the validation set of ActivityNet-1.3 Table 1. AR-AN and AUC on the validation set of Activity-Net1.3 Metric

SAP’s performance

AR@1

32.92

AR@5

48.40

AR@10

55.39

AR@100 74.74 AUC

4.4

66.60

Ablation Study

SAP-PEM: Considering the Variable length of candidate video proposals, we adopt our new architecture to receive video clip-level feature of arbitrary length, and output the feature vector with fixed length so that we can input them into the CNN network. In addition, the Discriminator can extract the summary from the input feature without losing much information. As comparison with PEM module in BSN, we show the results respectively, and demonstrates the effect of Discriminator. SAP-PE: We discard the position information of the input video feature, and feed the visual feature into temporal evaluation module directly. The result shows the improvement brought by Position Embedding. SAP-C3D: For the test of different backbones, C3D network is compared in our experiments. We test it on the Activity-Net dataset and also the speed of two different backbone network. The result demonstrates that C3D outperforms two-stream in speed test, for the extraction of optical flow costs much more time than C3D network. But two-stream network get a higher mAP in the test.

Self-adaptive Perception Model for Action Segment Detection

843

Table 2. Ablation studies on the validation set of ActivityNet-1.3 in terms of AUC and AR@100. Method

SAP-PEM SAP-PE SAP-C3D SAP

AR @ 100 73.94

73.90

72.56

74.74

AUC

65.58

64.08

66.60

65.90

As shown in Table 2, our full model SAP outperforms all its variants, namely SAP-PEM, SAP-PE, SAP-C3D, which verifies the effectiveness of the Discriminator, Position Embedding module which is contained in our full model. The reason is that the position embedding brings the position information which can benefit our model to localize action instances more precisely, and the Discriminator’s ability to transfer a matrix of arbitrary size into a vector of fixed size, helps our model to get accurate feature of proposals, and get precise confidence score. 4.5

Comparison with Published Methods

We compare our proposed method SAP with the published methods, namely, TCN [2], MSRA [20], Prop-SSAD [11], CTAP [5], and BSN [12], in terms of AUC and AR@100. It can be observed that our method outperforms the mainstream methods. Table 3. Performance comparisons between sap and the other proposal generation methods on the validation set of Activity-Net1.3 Method

TCN MSRA Prop-SSAD CTAP BSN Ours

AR @ 100 – AUC

–

61.56 64.18

73.01

73.17

74.16 74.74

64.80

–

66.26 66.60

As illustrated in Table 3, our proposed SAP model outperforms the mainstream methods in terms of AUC with different IOU thresholds, which demonstrates that our model can achieve a competitive performance of action proposal generation task.

5

Conclusion

We propose a new model for the task of action detection in videos. The position embedding which contains the position information helps to improve the performance of our model. And the Discriminator architecture also provides its ability to transfer a proposal of arbitrary size to a fixed size, which improve the Proposal Evaluation module to generate more precise confidence score of candidate

844

J. Li et al.

proposals of arbitrary length. Finally, we proposed a Action Perception model which achieve a competitive result in Activity-Net dataset. The result shows that the proposed model is a promising way to improve the quality of generated proposals. Acknowledgments. This research is supported by Beijing Natural Science Foundation (No. L181010 and 4172054), National Key R&D Program of China (No. 2016YFB0801100).

References 1. Buch, S., Escorcia, V., Shen, C., Ghanem, B., Niebles, J.C: Sst: single-stream temporal action proposals. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2911–2920 (2017) 2. Dai, X., Singh, B., Zhang, G., Davis, L.S., Chen, Y.Q.: Temporal context network for activity localization in videos. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5793–5802 (2017) 3. Escorcia, V., Caba Heilbron, F., Carlos Niebles, J., Ghanem, B.: Daps: deep action proposals for action understanding. In: European Conference on Computer Vision, pp. 768–784. Springer (2016) 4. Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1933–1941 (2016) 5. Gao, J., Chen, K., Nevatia, R.: Ctap: complementary temporal action proposal generation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 68–83 (2018) 6. Gao, J., Yang, Z., Chen, K., Sun, C., Nevatia, R.: Turn tap: temporal unit regression network for temporal action proposals. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3628–3636 (2017) 7. Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N.: Convolutional sequence to sequence learning. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1243–1252. JMLR. org (2017) 8. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015) 9. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016) 10. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015) 11. Lin, T., Zhao, X., Shou, Z.: Temporal convolution based action proposal: Submission to activitynet (2017). arXiv preprint arXiv:1707.06750 (2017) 12. Lin, T., Zhao, X., Su, H., Wang, C., Yang, M.: Bsn: boundary sensitive network for temporal action proposal generation. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018) 13. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)

Self-adaptive Perception Model for Action Segment Detection

845

14. Shou, Z., Wang, D., Chang, S.F.: Temporal action localization in untrimmed videos via multi-stage cnns. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016 15. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in neural information processing systems, pp. 568– 576 (2014) 16. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4489–4497 (2015) 17. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017) 18. Wang, L., Xiong, Y., Wang, Z., Qiao, Y.: Towards good practices for very deep two-stream convnets. arXiv preprint arXiv:1507.02159 (2015) 19. Wang, L., et al.: Temporal segment networks for action recognition in videos. IEEE Trans. Pattern Anal. Mach. Intell. 41(11), 2740–2755 (2018) 20. Yao, T., et al.: Msr asia msm at activitynet challenge 2017: trimmed action recognition, temporal action proposals and dense captioning events in videos. In: CVPR ActivityNet Challenge Workshop (2017)

Hierarchical Feature Generating Network for Zero-Shot Learning by Knowledge Graph Yi Zhang1 , Kan Li1(B) , and Xin Niu2 1

Beijing Institute of Technology, 5 South Zhongguancun Street, Haidian District, Beijing, China {3120181000,likan}@bit.edu.cn 2 Science and Technology on Parallel and Distributed Laboratory, College of Computer, National University of Defense Technology, Changsha, China

Abstract. Zero-Shot Learning (ZSL) has received much attention and has achieved great success. Most of existing ZSL methods transfer knowledge learned from seen classes to unseen classes by utilizing shared side information, such as, annotated attribute vectors, word embeddings, etc. Recently, the most popular method in ZSL is utilizing generative models to do semantic augmentation for unseen classes. However, these models generate features in one step which will lead to the domain shift problem and don’t leverage rich shared information between seen and unseen classes in the knowledge graph. Thus we construct a hierarchical generative model that synthesizes features for unseen classes layer by layer instead of one-step like previous ZSL work. Experimental results on various datasets show that our method can significantly improve the performance compared with the state-of-the-art ZSL models.

Keywords: Zero-shot learning generative model

1

· Knowledge graph · Hierarchical

Introduction

Conventional visual recognition models are based on supervised learning which require a large number of labeled training samples. However, in real-world situations, it is very difficult or expensive to obtain enough category labeled images for all classes, especially uncommon classes, such as newly discovered species on remote islands and uncommon deep sea animals for training. In these cases, the performance of those data-driven visual models will be greatly reduced. Then zero-shot learning (ZSL) [16] was introduced as a promising solution and has attracted a lot of attention. The task of zero-shot learning is to recognize novel classes, or unseen classes, whose labeled samples are not provided during training. When the test samples are only from unseen classes, the task is traditional ZSL; if test samples are from both seen and unseen classes, we consider the task as generalized ZSL c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 846–856, 2022. https://doi.org/10.1007/978-3-030-80119-9_55

Hierarchical Network for Zero-Shot Learning by Knowledge Graph

847

(GZSL). ZSL models all leverage side information or semantic descriptors, such as attribute vectors or word embeddings, which acts as a symbol of each class to associate visual space with corresponding semantic. Recently, generative models are introduced in ZSL [8] which generate samples for unseen classes to mitigate the domain shift problem discussed in [5, 10]. However, it is obvious that using generative models can not resolve domain shift problem completely. Because the generator only trained by side information of seen classes. Therefore, the generated samples via the trained generator tend to show the patterns of training categories. Inspired by a fact that the extracted features by convolutional neural network are refined layer by layer, we utilize the hierarchy to generate features layer by layer to obtain more and more specific features, which can mitigate the domain shift problem. Meanwhile, we noticed that in the hierarchical structure, although seen and unseen classes are disjoint at the bottom (leaf) layer, they share superclasses (non-leaf) in the top layers. Therefore, the knowledge graph contains much shared information between seen and unseen classes which is helpful for transfer learning. In our work, we propose a novel hierarchical model by pairing taxonomic structure with a generative model to obtain shared information from seen classes and utilize the information to help recognition for unseen classes. Based on most widely known generative adversarial networks we construct a hierarchical GAN by utilizing the knowledge graph. Besides, We are inspired by [25] which generates features instead of images for target classes to simplify the classification process. Concretely, we divide features into two parts, generated features of superclass as the benchmark and current layer’s generated features are considered as the bias. The latter is generated by current layer’s generative network. Besides, because our model can’t get the real feature maps of unseen classes, we have to perform random initialization. Then we construct a transductive model which updates real features of all nodes(except seen classes) by synthetic samples with corresponding pseudo labels, which is from last iteration to obtain more and more accurate results. We highlight our contributions as follows: – We propose a novel ZSL framework called Hierarchical-GAN which exploits knowledge graph and generative models to generate features layer by layer for unseen classes, which makes better use of hierarchical information and is beneficial for mitigating domain shift problem. – We construct the knowledge graph on AwA2 and SUN datasets based on taxonomy, respectively. – Extensive experiments on three datasets, SUN, AwA and CUB demonstrate the performance of our model is over the state-of-the-art methods significantly in both ZSL and GZSL settings.

848

2 2.1

Y. Zhang et al.

Related Work Zero-Shot Learning

In ZSL problem, we assume labeled samples of unseen classes are not available during training and ZSL can further be subdivided into traditional ZSL and generalized ZSL(GZSL). There has been a number of ZSL models. Early works on zero-shot learning are concentrated on mapping visual features and side information which represents categories into a common space to complete image classification tasks, such as DAP [12], DeViSE [4], LATEM [24], SAE [11], ESZSL [19] and ALE [1]. The difference of these methods are the choice of common space, the mapping function and the side information. However, learning an ideal embedding is difficult and the hubness problem [18] is irreversible in a high dimensional space. On the other hand, recent works, such as [26] shows that the performance of these models in GZSL setting is poor. Because of the powerful data generation capability, generative models such as Generative Adversarial Networks (GAN) or Variational Autoencoder (VAE) is another way to deal with the task of ZSL. Using generative models also is a promising way in this setting. A number of researches are appealed to leverage generative models to do semantic augmentation for unseen classes. The authors in [2, 8,20] leveraged generative models to synthesize samples for target classes, [25] generated features instead of samples and SE-GZSL [21] generated and leveraged examples from unseen classes. Besides, few work pays attention to exploit knowledge graph (taxonomic structure) that is effective for learning more information for knowledge transfer. In [14], the HierSE model used WordNet hierarchy to improve word embedding of labels. [13] concentrated on learning transferable visual features. The model of [23] paid attention to learn the visual classifier for each category via knowledge graph and semantic embedding, which will be expensive when the number of classes is large. [9] was expanded on the basic of [23] that adds a weighted dense connection scheme to prevent dilution of knowledge from distant nodes. All of these approaches don’t leverage the structure to divide features into parts and distill shared information between seen classes and unseen classes contained in the knowledge graph. 2.2

Generative Adversarial Network

There has been an increasing amount of work on Generative Adversarial Network (GAN) [6] both in computer vision and natural language processing. A typical GAN model that aims to generate new samples from the same domain with the training data, consists of two parts, a Generator and a Discriminator. A noise vector is drawn from a prior distribution for Generator to synthesis new data, and the role of Discriminator is to distinguish whether the input data is real or synthetic. In ZSL, Conditioned-GAN [15] is a commonly used technology that combines a condition usually class label with noise vector as input of Generator to generate samples for specified categories. We borrow a more stable GAN in our

Hierarchical Network for Zero-Shot Learning by Knowledge Graph

849

work, Wasserstein-GAN(w-GAN) [7] and inspired by [25], we generate features for each class instead of images to save the process and loss of extracting features of synthetic samples.

(a)

(b)

Fig. 1. (a) Shows the hierarchical structure which contains L layers and different colored circles represent different types of nodes; (b) Shows the overview of our proposed model which mainly contains aggregation model and Hierarchical-GAN(HGAN) and the details are elaborate in Sect. 3.

3 3.1

Approach Problem Definition

We introduce the notations used in this paper and define the task we investigate. Without loss of generality, we assume there is a prior hierarchy(knowledge graph), as shown in Fig. 1(a), consisting of L layers. To simplify, we denote L as the class layer(last layer of knowledge graph) and l(l = 1, ...L − 1) represents L − 1 superclasses layers. If L = 1, our model can be considered as f-CLSWGAN [25]. For the level L, there are two types of nodes, seen(source) nodes and unseen(target) nodes respectively. For each node at level l(l = 1, ...L−1), as long as one descendant node belongs to source classes, we use green circle to represent it, or it is represented by a pink circle. Let Ys denotes the set of source classes and Yt denotes the set of target classes. In ZSL, these two sets are disjoint, so we have Ys ∩ Yt = ∅. For each sample from both source and target domain, it has L − 1 Ns superclasses and L labels. A training set is represented like Ds = {(xi , yi )}i=1 , dx where xi ∈ R is the dx −dimensional feature map by utilizing a pre-trained CNN network. yi = {yi1 , ..., yiL }, where yil ∈ Yl (l = 1, ..., L − 1) and yiL ∈ Ys Nt of the i−th sample. For each sample xk from target set Dt = {(xk , yk )}i=1 , the task of our framework is to predict its corresponding label set yk = {yk1 , ..., ykL }, where yil ∈ Yl (l = 1, ..., L − 1) and yiL ∈ Yt .

850

3.2

Y. Zhang et al.

An Overview of Framework

We first introduce the transductive framework of our model shown in Fig. 1(b). Our model mainly consists of three steps in which the first two steps belong to stage 1(see orange arrows in Fig. 1(b)) and last one is stage 2(see blue arrows in Fig. 1(b)). The first step of stage 1 is to obtain real features and attribute vectors for all superclasses in the hierarchy through feature maps of all samples from both source and target domain(test samples with pseudo-labels) and attribute vectors of all classes by utilizing aggregation models(see red box in Fig. 1(b)). In our work, we compare two candidate aggregation methods and choose the mean aggregation based on experimental results. Next, based on the semantic embedding and all real features, we train a hierarchical-GAN(HGAN) and generate features for unseen classes (see purple box in Fig. 1(b)) layer by layer. At last, in stage 2, our model predicts labels for unseen classes and reobtains corresponding real features according to predicted labels. Naturally, we should recompute real features of superclasses and the node features. We iteratively carry out the stage 1 and stage 2 to obtain higher accuracy for target samples. 3.3

Aggregation Model

It is known that in the knowledge graph, we only have image-level features extracted by a pre-trained CNN model, such as ResNet-101, and attribute vectors of corresponding classes (the nodes at last layer of the structure) , therefore, the first thing is to obtain real features frl (l = 1, ..., L − 1) and attribute vectors a(yl )(l = 1, ..., L − 1) of super-classes, as shown the red box of Fig. 1(b). In this work, we examine two candidate aggregation functions. As for the every unseen class i ∈ Yt , we initialize frli by a Gaussian distribution and after the first training round, we get pseudo-labels for all test images and fykL are updated. Mean Aggregation. The first one is to adopt the mean operator simply. For category yil belongs to level l(l = 1, .., L − 1), its one-hop child set Chdyil having ncyil elements is obtained from the hierarchical structure, and the corresponding features frli is obtained as follow: ncyil

frli =



fr(l+1)j , j ∈ Chdyil

(1)

j=1

Deep Neural Network. The second method of obtaining features for superclasses is to train L − 1 deep neural networks SF Ml (l = 1, .., L− 1). We train the model SF Ml consisting of several fully-connected layers for level l on training set by solving a minimum optimization problem: min L(fl , yl )

(2)

Hierarchical Network for Zero-Shot Learning by Knowledge Graph

851

where fl = SF Ml (fl−1 ) and we use cross entropy as loss function to measures the distance of predicted labels and ground truth. Experiments show that performance of the first method, Mean Aggregation is better. In our work, we use the same way to obtain attribute vectors for superclasses in knowledge graph. 3.4

Hierarchical-GAN for Feature Generation

To fully utilize the shared information and mitigate domain shift problem, we propose a Hierarchical-GAN(HGAN) model which uses GAN and hierarchical structure to generate features for categories at each level on the basis of obtained attribute vectors and real features of all nodes in the knowledge graph, as shown in Fig. 1(b) (see the purple box ). At level l, we have a branch network GANl , which mainly consists of a w-GAN. The GANl is learned by optimizing the following objective: minGl maxDl (E[log Dl (frl , a(yl ))] +E[log(1 − Dl (f  gl , a(yl )))] + λ ∗ LGP )

(3)

Referring to the general practice of the network GAN, we take the combination of a noise vector z ∈ Rdz , sampled from Gaussian distribution and the semantic embedding a(yl ) as the input of generator Gl . Dl is the discriminator that is used to discriminate whether the connection of features and class embed is ding a(yl ) is true or not. frl has been stated in detail in Sect. 3.2 and fgl obtained by the following equation:   = (1 − α) ∗ fgl + α ∗ fg(l−1) fgl

(4)

where fgl = Gl (z, a(yl )) acts as the offset and the feature of upper layer  is considered as foundation. α is hyper-parameter that decides the imporfg(l−1) tance of two parts. In this work, when α = 0.75, we can obtain the best results.  = fg1 . The third term in Equation (3) consists of the For the first layer, fg1 gradient penalty LGP and the coefficient λ. Because a number of works have introduced the LGP in detail, here we won’t cover it again and we use λ = 10 as suggested in [7]. 3.5

Zero-Shot Classification and Iteratively Learning

After the training is completed with training dataset, the trained model is used to generate features layer by layer for test categories. For each test sample and its corresponding ancestor label set yk1 , yk2 , ...yk(L−1) , and last label ykL ∈ Yl . There are L ∗ ng synthetic features generated by trained HGAN model, where ng is the number of generated features of per category. According to the experiments, we set the value of ng be 300. Then we train a classifier Cl at level l for synthetic features of each layer by optimizing the following objective: Lclsl = L(sl + sl , yl )

(5)

852

Y. Zhang et al.

where sl = Cl (fgl ) and sl has the same dimension with sl and is extended by sl−1 . For the predicted score of level l − 1, sl−1 = {sl−1,1 , sl−1,2 ...sl−1,nl−1 } and nl−1 represents the number of discrete classes at level l − 1. We believe that if the value of sl−1,j is higher, the corresponding child nodes also should be higher. Therefore, we exploit the score of Cl−1 to help train current Cl . For element sl,i , its value is assigned by score of corresponding parent j, sl−1,j . We choose cross entropy as the loss function to compute the first item. At the last step, for each test image k, we take its real feature frl of level l as the input of classifier Cl to inference its label set yk1 , yk2 , ...ykL . We consider these predicted labels as the pseudo-labels to recompute the real features frl (l = 1, ..., L) of all nodes in the knowledge graph. In particular, it’s obvious that the accuracy of first few training iterations is low and if we accept all test images with the predicted labels, the node features of target classes will be outrageous. Therefore, we set a confidence for each test sample. Specifically, the more labels of the predicted label set yk1 , yk2 , ...ykL belong to same branch in the structure, the higher the confidence is. For simplicity, we just set the confidence as 1 of the test samples whose predicted labels among the sets belong to the same branch and set others to 0. In other words, only the value of confidence is 1, the features of samples can change its ancestors’s image-level features and class-level node features. 3.6

Network Details

We utilize the pre-trained ResNet-101 to extract features for all images. All the discriminators and generators consist of multi-Layer perceptions, and the only difference is that generators have a sigmoid function as the last layer. In addition, classifier at level l Cl is a fully connected layer.

4 4.1

Experiments Datasets

We conducted our experiments on three datasets: Caltech-UCSD-Birds 200–2011 (CUB) [22], Scene Recognition (SUN) [17], and the Animals with Attributes 2 (AwA2) [26]. SUN dataset contains 14,340 scene images belonging to 717 classes, 86 subfamilies, 13 families and the structure is built on the taxonomy structure. CUB dataset is widely used for ZSL task that covers 200 species. The same as [3], we exploit the ornithological systematics to build the structure which contains 122 genera, 37 families, and 13 orders. Finally, AwA2 also is a wellknown dataset for evaluating the performance of ZSL models. AwA2 has 37,322 images and 50 classes totally. We constructed the hierarchical structure with 50 species, 25 genera and 10 orders according to the animal taxonomy. The details of the three datasets are shown in Table 1. We use the zero-shot splits proposed by [26] for AWA, CUB and SUN insuring that none of the training classes are present in ImageNet.

Hierarchical Network for Zero-Shot Learning by Knowledge Graph

853

Table 1. Statistics on datasets, in terms of layers of the structure (Level), Dimension of attribute vectors (Av-d), Number of images (Samples), Train classes (Seen) and Test classes (Unseen). Dataset Level Av-d Samples Seen Unseen AwA2 CUB SUN

4.2

3 4 3

85 312 102

37,322 11,788 14,340

40 150 614

10 50 72

Evaluations

At testing, to evaluate the performance of our model and compare with stateof-the-art models, we use the unified evaluation in [26]. In ZSL, test samples are only from unseen classes, and we compute average value of per-class top-1 accuracy as the result. For GZSL setting, test samples are from both seen and unseen classes. To mitigate the effect of seen class accuracy to the overall results, we use the harmonic mean H that is computed by H = 2 ∗ s ∗ u/(s + u) as final accuracy in GZSL, where u represents the average per-class top-1 accuracy on unseen classes and s is the accuracy on training set. 4.3

Effectiveness of HGAN

In our proposed model, we exploit HGAN to generate features layer by layer. In this section, we compare our proposed models HGAN-DNN and HGAN-MA, which uses deep neural network and mean aggregation model respectively as aggregation method, with start-of-the-art ZSL models on SUN, CUB and AwA2 datasets to evaluate the effectiveness. The ZSL part of Table 2 shows the comparison results. The results of our model mean the accuracy at the last level. Table 2. Comparison with the State-of-the-Art ZSL and GZSL Models. u and s Represents Top-1 Accuracy on Unseen and Seen Classes. H Means Harmonic Mean. Method DEViSE LATEM SAE ESZSL ALE SE-ZSL f-CLSWGAN HGAN-DNN HGAN-MA

ZSL-Top1/GZSL-H (u,s) SUN CUB

AwA2

56.5/20.9(16.9,27.4) 55.3/19.5(14.7,28.8) 40.3/11.8(8.80,18.0) 54.5/15.8(11.0,27.9) 58.1/26.3(21.8,33.1) 63.4/34.9(40.9,30.5) 61.5/41.3(31.1,35.5) 65.7/33.5(36.4,31.0) 67.7/34.8(37.2,32.7)

54.2/27.8(17.1,74.7) 55.1/20.0(11.5,77.3) 54.1/2.20(1.10,82.2) 58.6/11.0(5.90,77.8) 62.5/23.9(14.0,81.8) 69.2/62.8(58.3,68.1) 68.2/47.6(57.2,52.0) 76.9/60.3(53.2,69.6) 78.4/64.3(57.7,72.5)

52.0/32.8(23.8,53.0) 49.3/24.0(15.2,57.3) 33.3/13.6(7.80,54.0) 53.9/21.0(12.6,63.8) 54.9/34.4(23.7,62.8) 59.6/46.7(41.5,53.3) 61.5/40.2(59.3,47.9) 65.1/50.7(40.7,67.2) 66.3/52.5(42.5,68.7)

854

Y. Zhang et al.

As Table 2 shows, we observe the performance of mean aggregation model is better than the deep neural network. Besides, the proposed model HGAN with mean aggregation method outperforms the previous ZSL models and the overall accuracy improvement on SUN is from 63.4% to 67.7%, on CUB from 59.6% to 66.3% and from 69.2% to 78.4% on AwA2. Specifically, compared with the model f-CLSWGAN [25] which uses annotated attribute vectors and only contains one layer W-GAN to generate features for unseen classes, our model achieves better results which demonstrates that our model is very effective. This is because we adapt the strategy of dividing the features into two parts, so that generated features of current layer for unseen classes can obtain shared information from seen classes. We also consider that at level l, the generated features only change minor on the basis of last layer instead of radical change like the previous models that can help to reduce the domain shift problem. Besides, our model utilizes the structure of knowledge graph to obtain more credible predicted labels as pseudo-labels to recompute real features of unseen classes and their ancestors, so that we can obtain higher results gradually. Furthermore, we explore the influence of generated features of upper layer on current layer, which is the value of α in Equation (7). Figure 2 shows the results of HGAN on dataset AwA2, we can find that at first, the higher the value of α is, the higher the results are in ZSL. However, when the value of α reaches 0.75, there is a turning point. Specifically, comparing the purple curve(α = 0) with others in Fig. 2, we can find that on the foundation of upper layer, the model will converge faster and the results are better. When the value of α is from 0.25 to 0.75, the best result on AwA2 is increasing gradually and when α = 0.75, the result can achieve the highest. While we continue to increase the value of α, the accuracy will decrease. The reason is that if the generated features of upper layer have too much impact on current layer, there will be less information of bias part and lack of difference among sibling nodes. So we set α = 0.75 for HGAN model.

Fig. 2. Iterations vs. Unseen class accuracy with different value of α.

Hierarchical Network for Zero-Shot Learning by Knowledge Graph

4.4

855

GZSL Results

In this section, we compare the effect of our proposed model to state-of-the-art methods on SUN, CUB and AwA datasets on GZSL task. For convenience, we just evaluate the results of the last layer directly. The models in Table 2 use attribute vectors or word embeddings as side information. The results are shown as a part of Table 2. In GZSL, to fully exploit our model, we generate features for seen and unseen classes and corresponding superclasses to train the classifiers instead of using real feature maps of training samples. In Table 2, we also observe that our model outperforms on dataset CUB and AwA2 and the results are from 46.7% to 52.5%, 62.8% to 64.3% respectively. While the on SUN the accuracy is 34.8%. We consider the reason is that the knowledge graph is constructed according to the purpose of the scenes instead of taxonomic knowledge.

5

Discussion and Conclusion

In this paper, we train a HGAN model constructed on the knowledge graph to generate features layer by layer so that the domain shift problem can be mitigated and much shared information can be fully used. Our model and CNN have the same tune that they all obtain features from generalized to detailed. Experiments on datasets AwA2, CUB and SUN demonstrate strategy of generating features layer by layer can significantly improve the performance. Besides, our framework can be extended to few-shot learning where a small number of labeled samples of some classes can help to get better initial features to improve the performance of the model. Acknowledgments. This research is supported by Beijing Natural Science Foundation (No. L181010 and 4172054), National Key R&D Program of China (No. 2016YFB0801100).

References 1. Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C.: Label-embedding for image classification. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1425–1438 (2016) 2. Arora, G., Verma, V.K., Mishra, A., Rai, P.: Generalized zero-shot learning via synthesized examples. In: CVPR, pp. 4281–4289 (2017) 3. Tianshui, C., Wu, W., Yuefang, G., Le, D., Xiaonan, L., Liang, L.: Fine-grained representation learning and recognition by exploiting hierarchical semantic embedding, pp. 2023–2031. ACM, MM (2018) 4. Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Mikolov, T.: Devise: a deep visual-semantic embedding model. In: NIPS, pp. 2121–2129 (2013) 5. Fu, Y., Hospedales, T.M., Xiang, T., Fu, Z.Y., Gong, S.: Transductive multi-view embedding for zero-shot recognition and annotation. In: ECCV, pp. 584–599 (2014) 6. Goodfellow, I., et al.: Generative adversarial nets. In: NIPS, pp. 2672–2680 (2014) 7. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. In: NIPS, pp. 5767–5777 (2017)

856

Y. Zhang et al.

8. Guo, Y., Ding, G., Han, J., Yue, G.: Synthesizing samples for zero-shot learning. In: IJCAI, pp. 1774–1780 (2017) 9. Kampffmeyer, M., Chen, Y., Liang, X., Wang, H., Zhang, Y., Xing, E.P.: Rethinking knowledge graph propagation for zero-shot learning. In: CVPR, pp. 11487– 11496 (2019) 10. Kodirov, E., Xiang, T., Fu, Z.Y., Gong, S.: Unsupervised domain adaptation for zero-shot learning. In: ICCV, pp. 2452–2460 (2015) 11. Kodirov, E., Xiang, T., Gong, S.: Semantic autoencoder for zero-shot learning. In: CVPR, pp. 4447–4456 (2017) 12. Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In: CVPR, pp. 951–958 (2009) 13. Li, A., Luo, T., Lu, Z., Xiang, T., Wang, L:. Large-scale few-shot learning: knowledge transfer with class hierarchy. In: CVPR, pp. 7212–7220 (2019) 14. Li, X., Shuai, L., Lan, W., Du, X., Gang, Y.: Zero-shot image tagging by hierarchical semantic embedding. In: CVPR, pp. 879–882 (2015) 15. Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv, 1411.1784 (2014) 16. Palatucci, M., Pomerleau, D., Hinton, G.E., Mitchell, T.M.: Zero-shot learning with semantic output codes. In: NIPS, pp. 1410–1418 (2009) 17. Patterson, G., Chen, X., Hang, S., Hays, J.: The sun attribute database: beyond categories for deeper scene understanding. Int. J. Comput. Vis. 108(1–2), 59–81 (2014) 18. Radovanovic, M., Nanopoulos, A., Ivanovic, M.: Hubs in space: popular nearest neighbors in high-dimensional data. J. Mach. Learn. Res. 11, 2487–2531 (2010) 19. Romera-Paredes, B., Torr, P.H.S.: An embarrassingly simple approach to zero-shot learning. In: ICML, pp. 2152–2161 (2015) 20. Tong, B., et al.: Adversarial zero-shot learning with semantic augmentation. In: AAAI, pp. 2476–2483 (2018) 21. Verma, V.K., Arora, G., Mishra, A., Rai, P.: Generalized zero-shot learning via synthesized examples. In: CVPR, pp. 4281–4289 (2018) 22. Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset. In: California Institute of Technology (2011) 23. Wang, X., Ye, Y., Gupta, A.: Zero-shot recognition via semantic embeddings and knowledge graphs. In: CVPR, pp. 6857–6866 (2018) 24. Xian, Y., Akata, Z., Sharma, G., Nguyen, Q., Hein, M., Schiele, B.: Latent embeddings for zero-shot classification. In: CVPR, pp. 69–77 (2016) 25. Xian, Y., Lorenz, T., Schiele, B., Akata, Z.: Feature generating networks for zeroshot learning. In: CVPR, pp. 5542–5551 (2018) 26. Xian, Y., Schiele, B., Akata, Z.: Zero-shot learning - the good, the bad and the ugly. In: CVPR, pp. 3077–3086 (2017)

Comparing Partitions: Metric Characterizations, Mean Partition, and Distortion Jyrko Correa-Morris1(B) , Abel Urra-Yglesias2 , Estefano Reyes1 , Juan Mart´ınez1 , and Belarmino Gonzalez1 1

2

Miami Dade College, Miami, FL 33135, USA [email protected] Florida International University, Miami, FL 33199, USA

Abstract. Measures for quantifying the distance between two partitions are involved in many current applications, including machine vision, image processing and understanding, image and video segmentation, biology and genetics, among others. This article investigates three fundamental aspects of the comparison of partitions: the characterization of metrics for comparing partitions; their role in average-based consensus of partitions; and how the different metrics distort the spatial organization of the partitions of a finite data set. In particular, we significantly reduce number of properties in the existing characterizations of metrics for comparing partitions such as Variation of Information and Mirkin metric. Moreover, we compile the main results describing the impact of these metrics on the construction of average-based consensus functions, as well as we compute the exact reduction of the search space that the different pruning criteria (quota rules) provide. Finally, we compute the distortion value for different β-entropy metrics. Keywords: Lattice of partitions · Consensus · Consensus function Mean partition · Clustering · Cluster ensemble

1

·

Introduction

The problem of comparing partitions of a finite data set gained widespread attention in the late 1960s and the early 1970s, mainly for the purpose of evaluating the performance or adequacy of classification algorithms. Unquestionably, the social and natural sciences have contributed to advance the techniques and methodology for the comparison of partitions, since the creation of taxonomies is an intrinsic aspect of the practice in these areas. During this period, articles addressing the comparison of partitions were commonly published in journals related to psychology, education, ecology, and statistics [1–5]. Many of the distance measures published in this time came with some theoretical foundation, supported by statistics models while entailed a specific probability distribution of data. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 857–875, 2022. https://doi.org/10.1007/978-3-030-80119-9_56

858

J. Correa-Morris et al.

Within the same statistical framework, during the 80s, some significant outputs emerged, associated to mathematicians such as Fowkles, Mallows [6], Hubert and Arabie [7]. In the same spirit as their predecessors, these works were mainly based on contingency tables. These methods lean on a division of all pairs of data into four disjoint groups by attending to whether the partitions being compared agree to place the two data in the same cluster or not. By somehow combining the cardinalities of these groups of pairs, any measure of this kind quantifies the distance between partitions. This class of measures is commonly used for comparison and assessment of classifications, even nowadays. Simultaneously, in the early 1980’s, several French mathematicians —among them, Bruno Leclerc, Jean-Pierre Barth´elemy and Bernard Monjardet— studied the problem of consensus in different contexts from an algebraic perspective. Particularly, they developed various aspects related to average-based consensus in the ambit of partitions. To some extent, they were also motivated by problems in social sciences. On this side of the story, a point which deserves further highlight is the exploitation of the lattice structure which the space of partitions is endowed with, aspect almost ignored by the statistical approach. The refinement of partitions is a partial order relation which induces a spatial organization/distribution of the partitions encoded in the Hasse diagram —an undirected graph in which the partitions are placed according to the order. A primitive notion of proximity arises naturally in association to the lengths of the paths in this graph. Metrics compatible with this primary notion of proximity were a major interest to these French specialists. Although measures for comparing partitions were mainly used to evaluate the performance of classification algorithms by comparing and contrasting their outputs with certain ground-truth clustering [8], with the rise of consensus methods such measures were promoted to play a major role in cluster analysis [9], with a plausible performance in current applications such as machine vision [10, 11], image processing [12], image and video segmentation [13–15]. The study of measures for comparing partitions are currently focused on two main branches: (1) The characterization of the different measures by determining their distinguishing features [16–18]. In particular, some attention has been dedicated to investigate the existence of metrics with certain desirable properties [19]; and, (2) their impact in the average-based consensus methods [9, 20–24]. However, despite the valuable contributions reported in the literature, some questions are still unanswered. As for the characterizations of measures to compare partitions, there is a series of articles by Marina Meilˇ a devoted to this topic (see, for instance, Meilˇ a [16,17]). Every characterization relies on five properties, some of them encoding very specific characteristic of the measure in question; however, the analogies and structural differences among the sets of properties describing respectively different measures remains unrevealed. This prompts the following question: are there generic structural properties that enable us to describe most of measures of the state-of-the-art? The answer to this question is affirmative as we will show below.

Metric Characterizations, Mean Partition, and Distortion

859

Other inconclusive aspects of the comparison of partitions have to do with the role of these measures in the definition and computation of average-based consensus partitions; especially, with their location within the lattice of partitions and the associated pruning strategies. The more complete characterization of average-based consensus solutions is that provided by Barth´elemy and Leclerc [20] for consensus functions built up from lower valuations and upper valuations, respectively. According to these results, these such consensus solutions satisfy certain refinement relations with the majority rules, which provide a significant reduction of the search space when the partitions forming the ensemble/profile share enough structural information. However, many of the proposals intended to advance the development of consensus methods perform an arbitrary search of the consensus solutions; much of this is potentially due to ignorance of either the rules and properties that describe the location of the corresponding solutions, or the compliance of the underlying measure with the prerequisites and assumptions of the known rules and properties that govern the distribution and position within the lattice of partitions of the consensus solutions corresponding to some specific classes of measures. Another important point of this subject lies in the deformation (i.e., compression and expansion) that the lattice of partitions experience when a measure for computing the distance between partitions is replaced by another measure. The use of a specific measure impose determined topological relationships among the partitions that can notably differ from those established by a different measure. Such discrepancies can affect the performance of any algorithm or method relying on a measure for comparing partitions. This is a completely unexplored question in this area. Motivated by these critical factors, in this article, we introduce several results (including Theorems 5 and 7) characterizing metrics defined from lower valuations and upper valuations, respectively, which in particular enable us to significantly reduce the characterizations provided by Meilˇ a [16, 17]. Furthermore, we use the contribution of Bath´erlemy and Leclerc [20] to provide information about the quality and location of the average-based partitions associated to several well-known measures for quantifying the distance between partitions. Finally, we explore some aspects of the distortion associated to the β-entropy metrics, a special class of metrics to compare partitions that generalize Variation of Information. In addition to Introduction, the paper is organized as follows. Section 2 introduces the basic concepts related to the space of partitions of a finite set. In Sect. 3, complete characterizations of the metrics associated to the lower valuations and upper valuations, respectively, are introduced. Section 4 is concerned with the impact of these metrics in the design of average-based consensus mechanisms, while Sect. 6 provides formulas for the exact reduction in size of the search space corresponding to the rules compiled in Sect. 5. The deformation of the space of partition associated to a change of metric is empirically addressed in Sect. 6. Some concluding remarks are given in Sect. 7.

860

2

J. Correa-Morris et al.

The Space of Partitions

To facilitate the access to the contributions of this article, we shall first describe the construction of the space of partitions of a finite set X = {x1 , x2 , . . . , xn }. The space of all the possible partitions of X is organized into n levels, as many as the elements the set X contains, which are placed one on top of the other. At the lowest level (level zero), only the all-singletons partition {{x} : x ∈ X} appears, denoted throughout this paper by mX . At each subsequent level, all those partitions p that can be obtained by merging exactly two clusters of some partition p lying in the immediately previous level, and only these, appear. In this case, we say that p covers p and we write p  p . This procedure continues until clusters can be merged no more, moment at which we will have all the levels of the space. Notice that, at the highest level ((n − 1)th level), only the onecluster partition {X} appears, which we denote by gX . Henceforth PX stands for the space of all partitions of the finite set X. Often, this space is graphically represented by its Hasse diagram, which is the graph whose set of vertices is PX and two partitions are connected by an edge if and only if one of them covers the other. The refinement order relation then arises in a natural way: p refines p if and only if there exists a sequence of partitions p1 , p2 , . . . , ps such that p = p1  p2  . . .  ps = p . We thus have the following equivalent ways of defining precisely this order relation: R1 For every pair of objects x, x ∈ X, if x and x are in the same cluster of p, then they are also in the same cluster of p . R2 For every pair of objects x, x ∈ X, if x and x are in different clusters of p , then they are also in different clusters of p. R3 Every cluster of p is contained in some cluster of p . R4 Every cluster of p is the union of clusters of p. R5 Every cluster of p splits into clusters of p. The relation “p refines p ” is denoted by p  p , which can be also read as “p is finer than p ” and, equivalently, as “p is coarser than p.” We use, p ≺ p to mean that p refines p , but p and p are not the same partition. Here are some remarks about this spatial organization: 1. mX refines every partition, and every partition refines gX . 2. Partitions placed in the same level of the Hasse diagram have the same number of clusters; however, they are not comparable (i.e. neither of them refines the other). 3. For any two partitions p and p there are partitions that refine both. The coarsest partition that satisfies this property is denoted by p ∧ p and is called the meet of p and p . Two elements x, y ∈ X are placed in the same cluster of p ∧ p if and only if they are simultaneously placed in the same cluster of p and in the same cluster of p . Therefore the clusters of p ∧ p are all the possible non-empty intersections of one cluster of p with one cluster of p .

Metric Characterizations, Mean Partition, and Distortion

861

4. For any two partitions p and p there are partitions that are refined by both. The finest partition that satisfies this property is denoted by p ∨ p and is called the join of p and p . Two elements x, y ∈ X are placed in the same cluster of p ∨ p if and only if there is a sequence x = xi1 , xi2 , . . . , xik = y such that, for all j ∈ {1, 2, . . . , k − 1}, xij and xij+1 are either placed in the same cluster of p or in the same cluster of p . Thus, the cluster of p ∨ p are the subsets of X that can be expressed both as unions of clusters of p and as unions of clusters of p , and are minimal with respect to this property. 5. Every partition is the join of partitions placed in level one (second level). For this reason, the partitions of level one are either called atoms of PX or atomic partitions. From this point, the atomic partition whose only nonsingleton cluster is {x, x } is denoted as pxx . In the case of the set of atoms in PX it is denoted as a and the set of atoms that refine the partition p is denoted as a(p). 6. Every partition is the meet of partitions placed in (n − 2)th level (i.e., the penultimate level). For this reason, the partitions of this level are either called co-atoms of PX or co-atomic partitions. From now on, pM denotes the coatomic partition whose only clusters are M and X − M . c denotes the set of co-atoms in PX and c(p) denotes the set of co-atoms that cover partition p.

3

Characterizing Measures for Comparing Partitions

One of the main questions related to the problem of comparing partitions lies in revealing those characteristic features that completely determine the functioning of a given measure. In this regard, to characterize various of the state-of-the-art measures, Meilˇa [16,17] introduced several structural properties: A1 Symmetry: For any two partitions p and p , d(p, p ) = d(p , p). A2 Additivity w.r.t refinement: Denote by mX and gX the unique partitions having n = #X clusters and one cluster, respectively. For any partition p, d(mX , p) + d(p, gX ) = d(mX , gX ). A3 Additivity w.r.t the meet: For any two partitions p, p d(p, p ) = d(p, p ∧ p ) + d(p ∧ p , p ), where p ∧ p is defined as the partition whose clusters are all the nonempty intersections of a cluster of p and a cluster of p . A4 Convex Additivity: Let p = {C1 , . . . , Ck } be a partition of X and p a refinement of p. Denote by pi the partitioning induced by p on the ith cluster k  ni d(gCi , pi ), where ni denotes the number of Ci of p. Then d(p, p ) = n i=1 elements in the ith cluster Ci of p. Using these properties (and some variants of them), Meilˇa characterized Variation of Information (V I), van Dongen metric V D, and Mirkin metric dΔ (which is directly proportional to the symmetric difference metric).

862

J. Correa-Morris et al.

Theorem 1 (Meilˇ a [16]). Variation of Information, defined for any two partitions p = {C1 , . . . , Ck } and p = {C1 , . . . , Ck  } by 

V I(p, p ) := −

k  k  nij i=1 j=1



n

 log

nij ni



 + log

nij nj

 ,

(1)

where ni = #Ci , nj = #Cj , nij = #(Ci ∩ Cj ), is the unique measure for comparing partitions that satisfies A1, A2, A3, A4 and Scale. Denote by pU k the “uniform” partition, i.e. the partition with k equal-sized exists, then d(gX , pU clusters. If pU k ) = log k. k Theorem 2. Van Dongen metric, defined by 

V D(p, p ) := 2n −

k  i=1



max nij − j

k  j=1

max nij , i

(2)

is the unique measure for comparing partitions that satisfies A1, A3, A4 and A2.D maxi ni

1 1− . d(gX , p) = 2 n Theorem 3 (Meilˇ a [16]). Mirkin metric, defined by ⎛ ⎞ k k k  k    1 n2 + nj − 2 n2ij ⎠ , dΔ (p, p ) = ⎝ n i=1 i j=1 i=1 j=1

(3)

is the unique measure for comparing partitions that satisfies A1, A2, A3, A4.M: Let p = {C1 , . . . , Ck } be a partition of and p a refinement of p. Denote by pi the partition induced by p on the ith cluster Ci of p. Then d(gX , p) =

maxi ni

1 1− , and 2 n

A5.D: d(gX , pU k)=1−

1 . k

In turn, Somovici et al. [25] axiomatically generalized the notion of entropy, which for every positive real number β determine the entropy function Hβ : PX → R≥0 defined for a partition p = {C1 , C2 , . . . , Ck } by:   k   ni  β Hβ (p) = cδβ 1 − , (4) n i=1 where c stands for a positive constant and δβ is equal to 1 if β > 0 and is equal to −1 if β < 1 (ni and n as above). The entropies that correspond to

Metric Characterizations, Mean Partition, and Distortion

863

β = 1 are called non-Shanon entropies, while the corresponding to β = 1 is called Shanon entropy. It was shown in [25] that the entropy metric corresponding to the Shanon entropy d1 is the Variation of Information metric V I. The notion of β-entropy naturally induces a conditional β-entropy, which is defined for two partitions p = {C1 , . . . , Ck } and p = {C1 , . . . , Ck  } by: k   β  nj 



Hβ (p|p ) =

j=1

n

Hβ (p Cj ),

(5)

where p Cj is the partition of Cj induced by p (i.e., p Cj = {Ci ∩ Cj = ∅, Ci ∈ p}). The conditional β-entropy Hβ (p|p ) can be rewritten as: Hβ (p|p ) =

1 21−β − 1

k  k   nij β 

i=1 j=1

n

−

 n β  j

n

.

(6)

The entropy metrics dβ : PX × PX → R are then defined through the analytical expressions: (7) dβ (p, p ) = Hβ (p|p ) + Hβ (p |p). The following theorem compiles the main properties of the entropy metrics given in [25]. Theorem 4 (Simovici and Jaroszewicz [25]). 1. 2. 3. 4. 5.

dβ is a metric. dβ (p, p ) = 2Hβ (p ∧ p ) − Hβ (p) − Hβ (p ). dβ (p, p ) = 2dβ (p ∧ p , gX ) − dβ (p, gX ) − dβ (p , gX ). dβ (p, p ) = dβ (p, mX ) + dβ (p , mX ) − 2dβ (p ∧ p , mX ). The function μβ : PX → R defined by μβ (p) = dβ (p, gX ) is submodular (i.e., for any partitions p, p ∈ PX , μβ (p) + μβ (p) ≥ μβ (p ∧ p )). 6. The function νβ : PX → R defined by νβ (p) = dβ (p, mX ) is supermodular (i.e., for any partitions p, p ∈ PX , νβ (p) + νβ (p) ≤ νβ (p ∧ p )). Another class of measures for comparing partitions that has been formally studied is that of the metrics associated to lower valuations and upper valuations, respectively. A function νβ : PX → R is said to be a lower valuation if ν is an order-preserving (i.e. p  p implies ν(p)  ν(p )) and supermodular function. Analogously, a function μβ : PX → R is said to be an upper valuation if μ is order-preserving and submodular function. The opposite of a β-entropy (−Hβ ) is an example of lower valuation; in particular, −H, where H is the Shannon entropy, is a lower valuation. Another example of lower valuation is the function that maps each partition p into the number #a(p) of atoms that refine it. The examples of upper valuation are less abundant. The typical example is the rank function of the lattice of partitions, r : PX → R, defined by r(p) = n − #p. The function that maps each partition p into the number #c(p) of co-atoms refined by it constitutes another example.

864

J. Correa-Morris et al.

Every lower valuation ν has an associated metric dν , which is defined by: dν (p, p ) = ν(p) + ν(p ) − 2ν(p ∧ p ).

(8)

Analogously, every lower valuation μ has an associated metric dμ , which is defined by: (9) dμ (p, p ) = 2μ(p ∨ p ) − μ(p) − μ(p ). The entropy metrics, in particular Variation of Information, are metrics associated to lower valuations (see Theorem 4 (6)). The metric associated to the function p → #a(p) is known as the symmetric difference metric, which can be rewritten da (p, p ) = a(p)Δa(p ) (here, Δ stands for the symmetric difference as a set operation). The so-called Mirkin metric differ from symmetric difference metric by a multiplicative constant. In turn, the metric associated to the rank function is the lattice metric, while the metric associated to the function p → #c(p) is the dual symmetric difference metric, which can also be expressed as da (p, p ) = c(p)Δc(p ). Both, the metrics associated to lower valuations and upper valuations, respectively, were proven to be shortest-path-length metrics in the neighborhood graph associated to the Hasse diagram of PX [20]. This fact seems to have been overlooked by Meilˇ a when establishing her characterizations [16, 17], which is evidenced in the fact that most of the characterized metrics happen to be metrics associated to either lower valuations or upper valuations. Based on this property of the metrics associated to lower valuations and upper valuations, respectively, we shall introduce below several results (including Theorems 5 and 7) that characterize these metrics in much intrinsic way and significantly reduce the list of structural properties required by the current ones. 3.1

Characterizing Metrics Associated to Lower Valuations

Theorem 5. Let D : PX × PX → R be a dissimilarity measure for partitions such that: LVM1: D is symmetric. LVM2: For any two partitions p and p , D(p, p ) = D(p, p ∧ p ) + D(p ∧ p , p ). LVM3: There is a lower valuation ν such that, for any partitions p, p ∈ PX , with p  p , D(p, p ) = ν(p ) − ν(p). Then, D is the metric dν associated to the lower valuation ν. Proof. Let D be a dissimilarity measure satisfying LV M 1 − 3. Then, for any partitions p, p ∈ PX , we have: D(p, p ) = D(p, p ∧ p ) + D(p ∧ p , p ), by LV M 2. D(p, p ) = D(p, p ∧ p ) + D(p , p ∧ p ), by LV M 1. D(p, p ) = ν(p) − ν(p ∧ p ) + ν(p ) − ν(p ∧ p ), by LV M 3 since p ∧ p  p, p . D(p, p ) = ν(p) + ν(p ) − 2ν(p ∧ p ). D(p, p ) = dν (p, p ).

Metric Characterizations, Mean Partition, and Distortion

865

Theorem 5 enables us to characterize all the entropy metrics, including Variation of Information, and Mirkin metric (i.e. the symmetric difference metric). Proposition 1. The entropy metric dβ is the only dissimilarity measure D for comparing partitions that satisfies LV M 1 − 2, and for every partition p and p , with p  p , β k  k   ni β  nj  D(p, p ) = − . n n i=1 j=1 In particular, Variation of Information is the only dissimilarity measure D that satisfies LV M 1 − 2 and and for every partition p and p , with p  p , 



D(p, p ) =

k  k 

δij

i=1 j=1

nj log n



ni nj

 ,

where δij = 1 if Cj ⊆ Ci , and otherwise 0. Proposition 2. Similarly, the symmetric difference metric is the only dissimilarity measure D that satisfies LV M 1 − 2 and and for every partition p and p , with p  p , ⎛ ⎞ k k   1 D(p, p ) = ⎝ n2 − (n )2 ⎠ . 2 i=1 i j=1 j Notice that the afore-introduced characterization of Variation of Information and Mirkin metric significantly reduce by half those given by Meilˇ a in [16]. Notice that, Meilˇa’s conditions A1 and A3 are conditions LVM1 and LVM2, respectively, while A2 and A4 are encoded within LVM3. Another option that we have which still produces simpler characterizations lies in splitting Property LVM3 into other two properties, LVM3I and LVM3II. The first of these properties establishes that the measure D is additive over the chains in PX , while LVM3II simplifies LVM3 by assuming that the partition p covers the partition p instead of requiring that p is arbitrarily refined by  p. Theorem 6. Let D : PX × PX → R be a dissimilarity measure for partitions that satisfy LV M 1, LV M 2, and: LVM3I: For any three partitions p, p and p , with p  p  p , D(p, p ) = D(p, p ) + D(p , p ). LVM3II: There is a lower valuation ν such that, for any partitions p, p ∈ PX , with p  p , D(p, p ) = ν(p ) − ν(p). Then, D is the metric dν associated to lower valuation ν.

866

J. Correa-Morris et al.

Proof. We know that p refines p if and only if there is sequence of partitions p1 , p2 , . . . , ps such that p = p1  p2  . . .  ps = p . Using LVM3I, we get D(p, p ) = D(p = p1 , p2 ) + D(p2 , p3 ) + . . . + D(ps−1 , ps = p ). as:

(10)

Using now LVM3II, D(pi , pi+1 ) = ν(pi+1 ) − ν(pi ), and we can rewrite (10),

D(p, p ) = ν(p2 ) − ν(p = p1 ) + ν(p3 ) − ν(p2 ) + . . . + ν(ps = p) − ν(ps−1 ) = ν(p ) − ν(p),

which is LVM3. The result now follows from Theorem 5. 3.2

Characterizing Metrics Associated to Upper Valuations

In this section, we shall introduce results analogous to Theorems 5 and 6 for characterizing the metrics associated to upper valuations. Theorem 7. Let D : PX × PX → R be a dissimilarity measure for partitions such that: UVM1: D is symmetric UVM2: For any two partitions p and p , D(p, p ) = D(p, p ∨ p ) + D(p ∨ p , p ). UVM3: There is a upper valuation μ such that, for any partitions p, p ∈ PX , with p  p , D(p, p ) = μ(p ) − μ(p). Then, D is the metric dμ associated to the upper valuation μ. Proof. Let D be a dissimilarity measure satisfying U V M 1 − 3. Then, for any partitions p, p ∈ PX . D(p, p ) = D(p, p ∨ p ) + D(p ∨ p , p ), by U V M 2. D(p, p ) = D(p, p ∨ p ) + D(p , p ∨ p ), by U V M 1. D(p, p ) = μ(p ∨ p ) − μ(p) + μ(p ∨ p ) − μ(p ), by U V M 3, since p, p  p ∨ p . D(p, p ) = 2μ(p ∨ p ) − μ(p) − μ(p ). D(p, p ) = dμ (p, p ). Analogously as we did with LVM3, we can replace UVM3 by two equivalent conditions where the condition “p refines p ” is substituted by the weaker condition “p covers p.” The former condition amounts to saying that there exists a path in the Hasse diagram of PX connecting p and p’, while the later condition establishes that these partitions are connected by an edge in this diagram. Theorem 8. Let D : PX × PX → R be a dissimilarity measure for partitions that satisfy U V M 1, U V M 2, and: UVM3I: For any three partitions p, p and p , with p  p  p , D(p, p ) = D(p, p ) + D(p , p ). UVM3II: There exists an upper valuation μ such that, for any partitions p, p ∈ PX , with p  p , D(p, p ) = μ(p ) − μ(p). Then, D is the metric dμ associated to upper valuation μ.

Metric Characterizations, Mean Partition, and Distortion

4 4.1

867

Consensus of Partitions Mean Partition and q-Quota Rules

Quota rules and dual quota rules are perhaps the most intuitive approach to consensus. Given an ensemble E = {p1 , p2 , . . . pm } of partitions of the set X, and a real number q ∈ [0, 1], the q-quota rule pq is formally defined as the finer partition refined by all the atoms that refine more than mq members of E. Analogously, the dual q-quota rule pdq , is the coarser partition that refines all the co-atoms refined by more than mq members of the ensemble. In the case of the unanimity rules and the majority rules, q = 1 and q = 1/2, respectively. Every quota rule pq (resp. dual quota rule pdq ) has a weak version, pq,w (resp. pdq,w ) whose definition is obtained by replacing the phrase “more than” in the definition of the quota rule pq (resp. dual quota rule pdq ) by the phrase “no less than.” Despite quota rules being themselves intuitive, easily interpretable and trustworthy consensus mechanisms, they have been relegated to the condition of assessors/evaluators of average-based consensus solutions. The reason for this lies in the fact that quota rules are very vulnerable to the presence of factions in the ensemble. Quota rules are based on agreement, which means that they merely count how many votes the members of the ensemble granted to the proposal “x and x should be in the same cluster of the partition” to then take the final decision by performing a simple comparison of such number of votes and a prefixed tolerance threshold qm. In practice, this procedure leads to rather fragmented solutions (i.e., many small clusters), as a consequence of a divided ensemble. As an attempt to solve this issue, consent-based methods were introduced, particularly average-based consensus methods. These have empirically proven to provide more informative solutions, at least in terms of the number and the size of the clusters. Consequently, average-based consensus methods have prevailed against quota rules. The level of non-consent that the ensemble E assigns to the proposal “p is the final solution” relies heavily on a prefixed dissimilarity measure D for comparing partitions, and it is quantified by the consensus function m  D(p, pi ). Naturally, the average-based consensus solutions are the ϕ(p) := i=1

minimizer of ϕ (and vice versa). The main disadvantages of average-based consensus solutions lie firstly in their interpretation, because the inner functioning of the consensus function ϕ is not as clear/intuitive as we would desire; and, secondly, in the minimization process of the consensus function ϕ, which tend to be computationally expensive. As a consequence of this dualism, average-based consensus functions are the responsible of producing consensus solutions, while quota rules are in charge of evaluating the quality of such solutions (see [20–22]). In this regard, we formally say that a consensus function ϕ is in compliance with a quota rule pq (resp. dual quota rule pdq ) if the latter refines (resp. is refined by) all the consensus solutions produced by ϕ. Thus, quota rules (resp. dual quota rules) are used as lower (resp. upper) bounds of the set of average-based solutions produced by ϕ, which also

868

J. Correa-Morris et al.

provides computational advantages because of the consequence reduction of the search space. The main contributions of Barth´elemy and Leclerc [20] are summarized in the following theorems. Theorem 9 (Barth´ elemy and Leclerc [20]). Let dν be the metric associated to the lower valuation ν and let ϕ be the corresponding average-based consensus function for an arbitrary given ensemble E. Then any consensus solution yielded by ϕ refines the dual majority rule pdm . Besides, at least one of these consensus 2 solutions refines the weak dual majority rule pdm ,w . 2

Theorem 10 (Barth´ elemy and Leclerc [20]). Let dμ be the metric associated to the upper valuation μ and let ϕ be the corresponding average-based consensus function for an arbitrary given ensemble E. Then any consensus solution yielded by ϕ is refined by the majority rule p m2 . Moreover, at least one of these consensus solutions is refined the weak majority rule p m2 ,w . 4.2

V I, dK and d∇ in the Ambit of Consensus

The results introduced in this section are trivial consequences of Theorems 9 and 10 that have been overlooked in the applications of average-based consensus methods to the area of machine learning and pattern recognition. Theorem 11. Let ϕ be the consensus solution associated to any of the measures dH and dK and an arbitrary ensemble E. Then, 1. The consensus solutions yielded by ϕ are not always refined by the unanimity rule pm . Consequently, the consensus solutions yielded by ϕ are not always refined by the majority rule p m2 . 2. The consensus solutions yielded by ϕ refine the dual majority rule pdm . 2 (Mirkin metric also falls in this category [20, 22]). Theorem 12. Let ϕ be the consensus solution associated to d∇ and an arbitrary ensemble E. Then: 1. The consensus solutions yielded by ϕ not always refine the dual unanimity rule pdm . Hence, the consensus solutions yielded by ϕ not always refine the dual majority rule pdm . 2 2. The consensus solutions yielded by ϕ are refined by the majority rule p m2 . A counterexample for the compliance of V I with the unanimity rule is given. Let E := {p1 , p2 , p3 , p4 , p5 } be the ensemble: p1 = {{x1 , x3 , x4 , x5 , x8 , x9 }, {x2 , x10 }, {x6 , x7 }, {x11 }}, p2 = {{x1 }, {x2 , x8 , x11 }, {x3 , x4 , x5 , x7 }, {x6 , x9 }, {x10 }}, p3 = {{x1 , x2 , x10 }, {x3 , x4 , x5 , x6 }, {x7 , x8 }, {x9 , x11 }}, p4 = {{x1 , x2 , x4 , x5 , x8 , x11 }, {x3 , x6 , x9 }, {x7 }, {x10 }}, p5 = {{x1 , x2 , x4 , x5 , x6 , x7 }, {x3 }, {x8 , x9 , x10 }, {x11 }}.

Metric Characterizations, Mean Partition, and Distortion

869

pm = {{x1 }, {x2 }, {x3 }, {x4 , x5 }, {x6 }, {x7 }, {x8 }, {x9 }, x10 }, {x11 }} is the corresponding unanimity rule, while the consensus solution that ϕ produces is mX .

5

Reduction of the Search Space

Let X be a data set with n data. The nth Bell number, in notation, Bn , stores the number of partitions in PX . We can use any of the equivalent formulas below to compute the nth Bell number: ∞

1  kn (Dobinski’s formula); e k! k=0 n−1  n − 1 Bk , B0 = 1; Bn = k

(11)

Bn =

Bn =

k=0 n 

(12)

S(n, k).

(13)

k=1

Equation (13) defines the Bell numbers as the sum of the Stirling numbers of the first kind, S(n, k), where for specific values of n and k, S(n, k) gives the number of partitions of n elements into k clusters. Additionally, Equations (11) and (12) together show that the difference between two consecutive Bell numbers is an exponential, which implies that Bn increases voraciously as n increases. Table 1 shows the first ten Bell numbers, which are additionally compared to 2n−1 , as well as the difference between any two consecutive of them, to numerically illustrate the claim above. The visual illustration of the reduction is shown in Fig. 1. Table 1. First ten bell numbers, two consecutive terms difference, and 2n−1 . n

1 2 3 4

Bn

1 2 5 15 52 203 877 4,140 21,147 115,975

Bn − Bn−1 n−1

2

5

6

7

8

9

10

1 3 10 37 151 674 3,273 17,007 1 2 4 8

16

32

64

128

256

94,828 512

870

J. Correa-Morris et al.

gX m 

gX m 

pi

i=1

pi

i=1

pdm

pdm

mX

mX

(a)

(b)

2

2

Fig. 1. Visual illustration of the reduction provided by: (a) dual unanimous rule; (b) dual majority rule. The shadowed region contains the consensus solutions.

The reduction (in number of partitions) of the search space that the dual majority rule (as an upper bound of the search space) provides is given by k  Bni , where Bn (the nth Bell number) stands for the number of all Bn − i=1

possible partitions of any finite set with n elements, k denotes the number of clusters of pdm , and ni is the number of elements in the ith cluster of pdm . The 2 2 tables below (Tables 2 and 3) shows some illustrative numbers. Table 2. Illustrative numbers of the reduction provided by the dual majority rule n

k n1 ; n2 ; . . . ; nk

k 

Bni Bn

Reduction

(%)

i=1

10 2 4; 6 12 3 4; 3; 5 15 3 6; 4; 5

3,045

115,975

3,900

4,213, 597

112,930 97.37 4,209,697 99.9

158,340 1,382,958,545 1,382,800,205 99.98

Analogously, the reduction provided by the majority rule p m2 is Bn − Bk , where k now denotes the number of clusters of p m2 . The chart below illustrates our explanation (Fig. 2):

Metric Characterizations, Mean Partition, and Distortion

871

Table 3. Illustrative numbers of the reduction provided by the majority rule n

k

Bn

Bk

Reduction

10

9

115,975

21,147

12

8

4,213, 597

4,140

(%)

94,828 81.76 4,209,457 99.9

15 12 1,382,958,545 4,213,597 134,082,257 96.95

gX

gX

p m2

p m2

m 

m 

pi

i=1

pi

i=1

mX

mX

(a)

(b)

Fig. 2. Visual illustration of the reduction provided by: (a) unanimous rule; (b) majority rule. The shadowed region contains the consensus solutions.

6

How Does the Space of Partitions Distort When a Metric d is Replaced by Another d ?

In this section, we empirically explore an almost-ignored aspect of the comparison of partitions: how the space of partitions is deformed by the different metrics. We focus on a particular class of metrics: the β-entropy metrics. Our main goal here is to investigate how the β-entropy metrics distort the spatial organization of partitions (e.g., contraction and compression) that the space of partitions of a finite set experience as the parameter β varies. Figure 3 shows the deformation associated to the β-entropy metrics, for β = 2 and β = 6, respectively. Let us suppose that d and d are two arbitrary metrics on the space of partitions PX of a finite set X. The expansion and contraction that space experience when d is replaced by d are defined respectively by: expansion(d → d ) = max 

p,p ∈PX

d (p, p ) . d(p, p )

(14)

872

J. Correa-Morris et al.

Fig. 3. Hasse diagrams with edges weighted by β-Entropy metrics

contraction(d → d ) = max 

p,p ∈PX

d(p, p ) . d (p, p )

(15)

The distortion is defined as the product of the expansion and the contraction: distortion(d → d ) = expansion(d → d ) · contraction(d → d )

(16)

For example, consider the metrics d and d on P{a,b,c} , where the rows and columns of the matrix below correspond to the linear order of the the partitions: {{a, b, c}}, ⎛c}}, {{a}, {b}, {c}}. ⎛ {{a, b}, {c}}, {{a, c}, {b}}, ⎞ ⎞ {{a}, {b, 0 0.19 0.19 0.19 1.08 0 0.82 0.82 0.82 1.77 ⎜ 0.19 0 0.4 0.4 0.90 ⎟ ⎜ 0.82 0 1.63 1.63 0.95 ⎟ ⎜ ⎜ ⎟ ⎟  ⎜ 0.19 0.4 0 0.4 0.90 ⎟ ⎟ 0.82 1.63 0 1.63 0.95 = d=⎜ d ⎜ ⎜ ⎟ ⎟ ⎝ 0.19 0.4 0.4 0 0.90 ⎠ ⎝ 0.82 1.63 1.63 0 0.95 ⎠ 1.08 0.90 0.90 0.90 0 1.77 0.95 0.95 0.95 0 





(p,p ) d(p,p ) Thus , distortion(d → d ) = maxp,p ∈PX dd(p,p  ) · maxp,p ∈PX d (p,p ) = 0.95 · 4.31 = 4.09 Table 4 shows the distortion between the 0.1-entropy metric and β  -entropy metric, with β  ranging from 0.1 to 3.75. It can be noticed that the distortion increases at a non-linear pace as β  increases. This suggests that the larger the difference between the values of the parameter β, the greater the deformation the space of partitions experience when a β-entropy metric is replaced by another of the same kind.

Metric Characterizations, Mean Partition, and Distortion

873

Table 4. β  and distortion size variation for β = 0.1 β 0.1 0.25 0.5 0.75 1.25 1.5 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75

Distortion Size 1.00 1.04598512803 1.13110696477 1.227947217 1.46256751307 1.60361295015 1.76293559432 1.9426071411 2.3723389862 2.3723389862 2.6276625386 2.91391071178 3.2344232943 3.5920785601 3.9933296111

Table 5. β  and distortion size variation for β = 2 β 0.1 0.25 0.5 0.75 1.25 1.5 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75

Distortion Size 1.942607141 1.8572034 1.717438935 1.5819956379 1.3282170729 1.211394021857 1.10191611505 1.00 1.10413791515 1.22121397371 1.3526474205 1.5 1.66499094222 1.8495137199 2.05565475720

A similar experiment, fixing β at the value 2 and varying β  from 0.1 to 3.75, revealed the same behavior. However, in contrast with the previous example, the values of β  gets farther from β = 2 not only by assigning to β  values greater than that fixed for β, but also smaller values (see Table 5). This reaffirms that the magnitude of the distortion becomes larger as the difference between the values of the parameter β augments. In the third experiment, β is fixed at 3.75 while β  varies over the same range as before. Now, β  gets farther from β by taking only smaller values (see Table 6). Once again, the values of the distortion, and consequently the deformation of the space of partitions, is more significant.

874

J. Correa-Morris et al. Table 6. β  and distortion size variation for β = 3.75

β 0.1 0.25 0.5 0.75 1.25 1.5 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75

7

Distorsion Size 3.99332961116 3.8177690142 3.5304615173 3.25203685910 2.73035574458 2.49020788387 2.26515910395 2.05565475720 1.86177354203 1.6832879425 1.51972696355 1.37043650480 1.23463419833 1.11145688458 1.00

Conclusions

This paper introduced characterizations of metrics for comparing partitions that advanced the state-of-the-art of this topic. Particularly, we significantly reduce number of properties in the existing characterizations of metrics for comparing partitions such as Variation of Information and Mirkin metric. The role of such metrics in average-based consensus of partitions was further investigated as well as the exact reduction of the search space that the different pruning criteria (quota rules) provide was computed. Lastly, how the different metrics distort the spatial organization of the partitions of a finite data set was investigated.

References 1. Rand, W.M.: Objetive criteria for the evaluation of clustering methods. J. Am. Stat. Assoc. 66, 846–850 (1971) 2. Mirkin, B.G., Chernyi, L.B.: Measurement of the distance between distinct partitions of a finite set of objects. Autom. Remote. Control. 31, 786–792 (1970) 3. Arabie, P., Boorman, S.A.: Multidimensional scaling of mesures of distance between partitions. J. Math. Psychol. 10, 148–203 (1973) 4. Hubert, L.J.: Nominal scale response agreement as a generalized correlation. Br. J. Math. Stat. Psychol. 30, 98–103 (1977) 5. Hubert, L.J.: Matching models in the analysis of cross-classification. Psychometrika 44, 21–41 (1979) 6. Fowlkes, E.B., Mallows, C.L.: A method for comparing two hierarchical clusterings. J. Am. Stat. Assoc. 78(383), 553–569 (1983) 7. Hubert, L., Arabie, P.: Comparing partitions. J. Classif. 2(1), 193–218 (1985) 8. Amig´ o, E., Gonzalo, J., Artiles, J., Verdejo, F.: A comparison of extrinsic clustering evaluation metrics based on formal constraints. Inf. Retrieval 12(4), 461–486 (2009)

Metric Characterizations, Mean Partition, and Distortion

875

9. Barth´elemy, J.P., Monjardet, B.: The median procedure in cluster analysis and social choice theory. Math. Soc. Sci. 1(3), 235–267 (1981) 10. Xiao, W., Yang, Y., Wang, H., Li, T., Xing, H.: Semi-supervised hierarchical clustering ensemble and its application. Neurocomputing 173(3), 1362–1376 (2016) 11. Sun, N., Yu, H.: A method to determine the number of clusters based on multivalidity index. In: Nguyen, H.S., Ha, Q.-T., Li, T., Przybyla-Kasperek, M. (eds.) IJCRS 2018. LNCS (LNAI), vol. 11103, pp. 427–439. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99368-3 33 12. Zaleshina, M., Zaleshin, A.: Multiscale integration for pattern recognition in neuroimaging. In: Pardalos, P.M., Conca, P., Giuffrida, G., Nicosia, G. (eds.) MOD 2016. LNCS, vol. 10122, pp. 411–418. Springer, Cham (2016). https://doi.org/10. 1007/978-3-319-51469-7 35 13. Vega-Pons, S., Jiang, X., Ruiz-Shulcloper, J.: Segmentation ensemble via kernels. In: ACPR 2011, pp. 686–690 (2011) 14. Huang, D., Lai, J.-H., Wang, C.-D., Yuen, P.C.: Ensembling over-segmentations: from weak evidence to strong segmentation. Neurocomputing 207, 416–427 (2016) 15. Zhang, Y., Wang, H., Zhou, H., Deng, P.: A mixture model for image boundary detection fusion. IEICE Trans. Inf. Syst. E101-D(4), 1159–1166 (2018) 16. Meilˇ a, M.: Comparing clusterings: an axiomatic view. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 577–584 (2005) 17. Meilˇ a, M.: Comparing clusterings-an information based distance. J. Multivar. Anal. 98(5), 873–895 (2002) 18. Meilˇ a, M., Heckerman, D.: An experimental comparison of model-based clustering methods. Mach. Learn. 42(1–2), 9–29 (2001) 19. Correa-Morris, J.: Comparing partitions: shortest path length metrics and submodularity. Int. J. Math. Models Methods Appl. Sci. 13, 45–51 (2019) 20. Barth´elemy, J.-P., Leclerc, B.: The median procedure for partitions. Partitioning Data Sets 19, 3–34 (1993) 21. Vega-Pons, S., Correa-Morris, J., Ruiz-Shulcloper, J.: Weighted partition consensus via kernels. Pattern Recogn. 43(8), 2712–2724 (2010) 22. Vega-Pons, S., Avesani, P.: Clustering ensemble on reduced search spaces. In: COPEM Workshop (2013) 23. Franek, L., Jiang, X.: Ensemble clustering by means of clustering embedding in vector spaces. Pattern Recognit. 47(2), 833–842 (2014) 24. Ilc, N.: Weighted cluster ensemble based on partition relevance analysis with reduction step. IEEE Access 8, 113720–113736 (2020) 25. Simovici, D.: On generalized entropy and entropic metrics. J. Multiple Valued Logic Soft Comput. 13(4/6), 295 (2007)

A Preliminary Comparison Between Compressive Sampling and Anisotropic Mesh-Based Image Representation Xianping Li1(B) and Teresa Wu2 1

Arizona State University, Poly Campus, Mesa, AZ 85212, USA [email protected] 2 Arizona State University, Tempe, AZ 85281, USA [email protected]

Abstract. Compressed sensing (CS) has become a popular field in the last two decades to represent and reconstruct a sparse signal with much fewer samples than the signal itself. Although regular images are not sparse on their own, many can be sparsely represented in wavelet transform domain. Therefore, CS has also been widely applied to represent digital images. However, an alternative approach, adaptive sampling such as mesh-based image representation (MbIR), has not attracted as much attention. MbIR works directly on image pixels and represents the image with fewer points using a triangular mesh. In this paper, we perform a preliminary comparison between the CS and a recently developed MbIR method, AMA representation. The results demonstrate that at the same sample density, AMA representation can provide better reconstruction quality than CS based on the tested algorithms. Further investigation with recent algorithms is needed to perform a thorough comparison. Keywords: Compressive sampling · Mesh-based image representation · AMA representation · PSNR · Structural similarity index measure

1

Introduction

Compressive sampling (CS, also known as compressive sensing, compressed sensing, sparse sampling) has become popular in signal processing for the last two decades, and numerous research has been performed in this field. The literature is superabundant to be listed entirely, and only a few are referred here [1–8]. CS has been applied to computed tomography (CT) [9, 10] and MRI [11–13] reconstructions to improve the acquisition speed. It has also been applied to represent digital images obtained from different sources [14,15], although many images are not sparse on their own. In this paper, we focus on the representation of digital images from different sources, such as digital cameras, remote sensing images, as well as MR images that have already been obtained/reconstructed. In many processing tasks of those images, it is also desirable to represent the images with fewer pixels to c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 876–885, 2022. https://doi.org/10.1007/978-3-030-80119-9_57

Compressive Sampling vs AMA Representation

877

reduce the image size and improve processing efficiency. One example is image segmentation via a variational approach. In those cases, we need to work on the image pixels directly. Adaptive sampling based on image pixels, especially, triangular mesh-based image representation (MbIR), has recently gained much interest, although not as much as compressive sampling that works on transformed domain. Some publications about MbIR are listed here [16–25]. In particular, Li developed the anisotropic mesh adaptation for image representation framework (AMA representation) in [25] that provides better representation quality with reasonable computational cost than other MbIR methods. Although CS is much more popular than MbIR, it is not clear if it performs better than MbIR in terms of reconstruction quality, especially for natural images that are not sparse on their own. This paper aims to perform a preliminary comparison between CS and MbIR for digital images from different sources and provide guidance for further research efforts in the area of image representation. The rest of the paper is organized as follows. Section 2 gives a brief description of compressive sampling (CS) as well as mesh-based image representation. Some CS algorithms and a recently developed MbIR method, anisotropic mesh adaptation for image representation (AMA representation), are introduced. Section 3 describes the comparison metrics and presents the results for images of different styles and resolutions. A short conclusion is given in Sect. 4.

2

Compressive Sampling and Mesh-Based Image Representation

In this section, we briefly introduce the two image representation approaches: compressive sampling and adaptive sampling, in particular, anisotropic mesh adaptation for image representation (AMA representation) that is a recently developed anisotropic mesh-based image representation framework. 2.1

Compressive Sampling

The key idea of compressive sampling (CS) is the sparsity of the signal that needs to be reconstructed. A signal is called sparse if most of its components are zero. It is possible to represent a sparse signal using a small number of measurements and reconstruct it by solving an underdetermined linear system. By reducing the sampling rate, CS can improve the signal acquisition and transmission speed and reduce hardware requirements for sensors. It takes more computational effort during reconstruction on the receiver side. Let x ∈ CN be a sparse vector needs to be reconstructed from the underdetermined measurements xmeas = Ax ∈ Cm with m  N . Then there are two key steps in CS. 1. Design the measurement (also called sensing) matrix A ∈ Cm×N so that as much intrinsic information will contribute to the reconstruction. 2. Solve x from the underdetermined linear system xmeas = Ax with balance of efficiency and accuracy.

878

X. Li and T. Wu

For image representation purpose, we convert the 2D images into a vector x ∈ RN , which serves as the ground truth. Although x may not be sparse on its own, it can be sparse in wavelet or Fourier domain. Thus, m( N ) sample points are randomly chosen from x in wavelet or Fourier domain. Then a random matrix A of the corresponding size is generated according to the sample points. The measurement data is simulated by multiplying A and x as xmeas = Ax ∈ Rm . The ratio m/N is denoted as the sample density of the representation. For the reconstruction step, we consider two methods here. One is to solve the l1 -minimization problem by using the log-barrier algorithm, and the l1 -MAGIC package [26] (written in MATLAB) is used for the computations. The other is the iterative soft thresholding algorithm (ISTA), and the pit package [27] (written in Python) is used for the computations. Minor modifications of the program codes have been made to display the reconstructed image as well as compute the quality of reconstruction. 2.2

AMA Representation

Different from CS, adaptive sampling works directly on image pixels. The idea is to choose the essential pixels of the image and discard non-important ones. Various approaches, especially mesh-based image representation methods, have been developed since the 1990s. Here, we briefly introduce the recently developed AMA representation framework [25] that consists of four key steps. 1. Generate an initial triangular mesh based on the desired sample density. The sample density determines the number of vertices in the mesh. Delaunay triangulation can be used to generate the initial mesh. 2. Assign values to mesh vertices from original image grey values using linear finite element interpolation and compute the user-defined metric tensor M on the mesh. 3. Adapt the mesh to be a quasi-M-uniform mesh that almost fits the provided metric tensor M. 4. Reconstruct the image using the final adaptive mesh with finite element interpolation for triangles. Steps 2 and 3 can be repeated a few times in order to obtain a good adaptive mesh, which is the part that takes most computational resources in this framework. Li also proposed the GPRAMA method in [25] that combines the greedypoint-removal algorithm with AMA representation using mesh patching techniques. GPRAMA can provide better representation quality than AMA but is computationally more expensive due to the greedy-point-removal procedure. As a preliminary investigation, we only consider AMA representation in this paper.

3

Comparison Results

In this section, we present the results obtained for eight images of different styles and resolutions. The parameter values for CS algorithms are taken from

Compressive Sampling vs AMA Representation

879

the packages as default. For AMA representation, the anisotropic metric tensor Maniso (Eq. (8) in [25]) is used. In l1-MAGIC package, the CS tveq() function solves a total variation minimization problem with equality constraints. For the tested images, this function provides better reconstruction quality as well as takes less time than CS tvqc() function that solves a total variation minimization problem with quadratic constraints. Although CS tvqc with quantization runs faster than CS tveq(), the reconstruction quality is worse. Thus, only the results obtained from CS tveq() function are presented in this paper and denoted as TVeq. The results obtained from the Python package pit using the iterative soft thresholding algorithm are denoted as ISTA. And the results obtained from AMA representation are denoted as AMA. 3.1

Comparison Metrics

To compare the quality of the reconstruction from different approaches, we consider two commonly used metrics: peak-signal-to-noise-ratio (PSNR) and structural similarity index measure (SSIM). PSNR estimates the absolute error between the reconstructed image and the original image. It is calculated based on the mean square error (MSE) as follows.   p 2 −1 , (1) PSNR = 20 log10 √ M SE where p is the sample precision. For grayscale images, p = 8. The psnr() function in MATLAB is used for the calculation. Larger PSNR value indicates better representation quality. On the other hand, SSIM considers the inter-dependence of the pixels that carries important information about the structure of the objects in the image. The measure between two images x and y of common size is calculated as follows: SSIM(x, y) =

(2μx μy + C1 )(2σxy + C2 ) , + μ2y + C1 )(σx2 + σy2 + C2 )

(μ2x

(2)

where μi and σi are average and variance of i (i = x, y), respectively, and σxy is the covariance of x and y. The ssim() function in MATLAB is used for the calculation that also provides the local SSIM map. Larger SSIM value indicates better representation quality. 3.2

Comparison Results

Figure 1 shows eight images chosen for this comparison task, all of which are converted into grayscale. The first four images Fig. 1(1–4) are publicly available from USC-SIPI or MATLAB image folder. The high-resolution brain MR image Fig. 1(5) is one of the slices extracted from a 7-Tesla MRI of the ex vivo human brain at 100-µ resolution by Edlow et al. [28]. The last three images Fig. 1(6–8) are provided by the ASU-Mayo Center for Innovative Imaging.

880

X. Li and T. Wu

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

Fig. 1. Original Images: (1) Cameraman, 256 × 256, (2) Lena, 512 × 512, (3) Beach, 730 × 730, (4) bridge, 3000 × 3000, (5) Brain MRI, 1760 × 1760, (6) Heat Bubble, 720 × 720, (7) AD (PET), 960 × 960, and (8) Migraine (MRI), 960 × 960.

The results for sample density at 3% are shown in Table 1, while those for sample density at 10% are shown in Table 2. It is clear that the representation quality improves when sample density increases because more information is embedded in the measurement data. For results in each table, that is, at the same sample density, AMA provides the best representation quality in terms of both PSNR and SSIM. ISTA performs the worst in terms of the reconstruction quality, which might be due to the default threshold values used in the code. Finding optimal values for the parameters could improve the performance. However, these comparison results

Compressive Sampling vs AMA Representation

881

deliver an overall message that AMA representation can provide better quality than CS algorithms. Table 1. Comparison of representation qualities at sample density 3% Image

Resolution

PSNR (dB) SSIM TVeq ISTA AMA TVeq ISTA AMA

Cameraman

256 × 256

30.69 18.90 33.09 0.61

0.54

0.74

Lena

512 × 512

31.98 23.59 35.23 0.74

0.69

0.85

Beach

730 × 730

32.82 21.35 35.04 0.75

0.69

0.83

Bridge

3000 × 3000 34.91 27.31 36.92 0.80

0.74

0.85

Brain MRI

1760 × 1760 35.38 28.24 36.15 0.68

0.65

0.71

Heat bubble∗

720 × 720

30.80 25.23 34.91 0.73

0.77

0.87

AD (PET)∗

960 × 960

40.82 38.53 44.49 0.94

0.95

0.96

Migraine (MRI) 960 × 960 32.08 24.27 34.09 0.69 0.68 (∗ : provided by ASU-Mayo Center for Innovative Imaging.)

0.78

∗

Table 2. Comparison of representation qualities at sample density 10% Image

Resolution

PSNR (dB) SSIM TVeq ISTA AMA TVeq ISTA AMA

Cameraman

256 × 256

32.29 21.21 34.44 0.73

0.66

0.82

Lena

512 × 512

34.51 27.26 38.22 0.84

0.80

0.92

Beach

730 × 730

34.28 23.68 36.59 0.81

0.78

0.90

Bridge

3000 × 3000 37.58 30.86 40.50 0.87

0.82

0.92

Brain MRI

1760 × 1760 36.09 30.05 36.84 0.72

0.70

0.74

Heat bubble∗

720 × 720

34.43 31.34 40.94 0.87

0.89

0.96

AD (PET)∗

960 × 960

47.59 40.23 47.36 0.98

0.96

0.98

Migraine (MRI) 960 × 960 35.41 26.94 36.21 0.84 0.77 (∗ : provided by ASU-Mayo Center for Innovative Imaging.)

0.87

∗

It is interesting to observe that, for the high-resolution brain MR image in Fig. 1(5), TVeq performance is close to that of AMA in terms of both PSNR and SSIM. At the same time, ISTA is also close in terms of SSIM. Similar results are observed for AD (PET) image in Fig. 1(7). The reason might be due to the high sparsity of those two images. Figure 2 shows the reconstruction results for image Lena at sample density 10%, and Fig. 3 shows the reconstruction results for image migraine (MRI) at sample density 3%. The results for other images are similar.

882

X. Li and T. Wu

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 2. Reconstruction of image lena at sample density 10%: (a) TVeq, PSNR = 34.51, (b) ISTA, PSNR = 27.26, (c) AMA, PSNR = 38.22, (d) TVeq SSIM Map, SSIM = 0.84, (e) ISTA SSIM Map, SSIM = 0.80, and (f) AMA SSIM Map, SSIM = 0.92.

(a)

(b)

(c)

(d)

(e)

(f)

Fig. 3. Reconstruction of image migraine (MRI) at sample density 3%: (a) TVeq, PSNR = 32.08, (b) ISTA, PSNR = 24.27, (c) AMA, PSNR = 34.09, (d) TVeq SSIM Map, SSIM = 0.69, (e) ISTA SSIM Map, SSIM = 0.68, and (f) AMA SSIM Map, SSIM = 0.78.

Compressive Sampling vs AMA Representation

(1)

(2)

(3)

(4)

(5)

(6)

(7)

883

(8)

Fig. 4. Anisotropic triangular meshes for AMA representation at sample density %3: (1) cameraman, (2) Lena, (3) Beach, (4) Bridge, (5) Brain MRI, (6) Heat Bubble, (7) AD (PET), and (8) Migraine (MRI).

Figure 4 shows the anisotropic triangular meshes for the original images produced by AMA representation at sample density 3%. As can be seen, the triangular elements concentrate around the edges of the images so that they can represent the key features of the images.

4

Conclusions

In this paper, we have performed a preliminary comparison between compressive sampling and adaptive sampling, specifically, AMA representation. Due to the

884

X. Li and T. Wu

large number of methods developed in the CS field, this comparison is by no means thorough. However, the work does provide some guidance for future efforts in the area of image representation. Our results indicate that AMA representation can provide better representation quality than CS algorithms. CS works well for ideal sparse signals but may not for natural images. Although different measurement matrix can be designed, and various parameter values in different algorithms can be optimized, the improvement of CS performance, especially on natural images, may still be modest. Further comparison with recently developed CS algorithms is needed. On the other hand, AMA representation works directly on image pixels by performing anisotropic mesh adaptation to concentrate triangular elements around the edges of the images. Although some parameters are used for the mesh adaptation procedure, their default values in the program code work well for general problems. The representation procedure is straightforward without the need to specify different parameters other than the desired sample density. Note that the AMA representation quality can still be improved by adopting the greedy-point-removal algorithm (GPRAMA) at the expense of more computational cost. Besides better representation quality than compressive sampling, AMA representation can also be applied to improve the efficiency of partial differential equation based image processing such as image segmentation.

References 1. Candes, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006) 2. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006) 3. Chartrand, R.: Exact reconstructions of sparse signals via nonconvex minimization. IEEE Signal Process. Lett. 14(10), 707–710 (2007) 4. Donoho, D.L., Tsaig, Y.: Fast solution of l1-norm minimization problems when the solution may be sparse. IEEE Trans. Inf. Theory 54(11), 4789–4812 (2008) 5. Blumensath, T., Davies, M.: Iterative thresholding for sparse approximations. J. Fourier Anal. Appl. 14, 629–654 (2008) 6. Dai, W., Milenkovic, O.: Subspace pursuit for compressive sensing signal reconstruction. IEEE Trans. Inf. Theory 55(5), 2230–2249 (2009) 7. Needel, D., Vershynin, R.: Signal recovery from incomplete and inaccurate measurements via regularized orthogonal matching pursuit. IEEE J. Sel. Top. Sig. Process. 4(2), 310–316 (2010) 8. Kovarik, L., Stevens, A., Liyu, A., Browning, N.D.: Implementing an accurate and rapid sparse sampling approach for low-dose atomic resolution STEM imaging. Appl. Phys. Lett. 109(16), 164102 (2016) 9. Tian, Z., Jia, X., Yuan, K., Pan, T., Jiang, S.B.: Low-dose CT reconstruction via edge preserving total variation regularization. Phys. Med. Biol. 56(18), 5949–5967 (2011)

Compressive Sampling vs AMA Representation

885

10. Li, X., Luo, S.: A compressed sensing-based iterative algorithm for CT reconstruction and its possible application to phase contrast imaging. Biomed. Eng. Online 10, 73 (2011) 11. Lustig, M.: Sparse MRI: the application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58(6), 1182–1195 (2007) 12. Lustig, M., Donoho, D.L., Santos, J.M., Pauly, J.M.: Compressed sensing MRI. IEEE Signal Process. Mag. 25(2), 72–82 (2008) 13. Tashan, T., Al-Azawi, M.: Multilevel magnetic resonance imaging compression using compressive sensing. IET Image Proc. 12(12), 2186–2191 (2018) 14. Fei, X., Wei, Z., Xiao, L.: Iterative directional total variation refinement for compressive sensing image reconstruction. IEEE Signal Process. Lett. 20(11), 1070– 1073 (2013) 15. Liu, L., Xie, Z., Feng, J.: Backtracking-based iterative regularization method for image compressive sensing recovery. Algorithms 10, 1–8 (2017) 16. Terzopoulos, D., Vasilescu, M.: Sampling and reconstruction with adaptive meshes. In: Proceedings of the IEEE Computer Vision and Pattern Recognition Conference (CVPR 1991), pp. 70–75 (1991) 17. Davoine, F., Antonini, M., Chassery, J.-M., Barlaud, M.: Fractal image compression based on Delaunay triangulation and vector quantization. IEEE Trans. Image Process. 5(2), 338–346 (1996) 18. Ramponi, G., Carrato, S.: An adaptive irregular sampling algorithm and its application to image coding. Image Vis. Comput. 19, 451–460 (2001) 19. Yang, Y., Wernick, M.N., Brankov, J.G.: A fast approach for accurate contentadaptive mesh generation. IEEE Trans. Image Process. 12(8), 866–881 (2003) 20. Su, D., Willis, P.: Image interpolation by pixel-level data-dependent triangulation. Comput. Graph. Forum 23(2), 189–201 (2004) 21. Brankov, J.G., Yang, Y., Wernick, M.N.: Tomographic image reconstruction based on a content-adaptive mesh model. IEEE Trans. Med. Imaging 23(2), 202–212 (2004) 22. Demaret, L., Iske, A.: Adaptive image approximation by linear splines over locally optimal Delaunay triangulations. IEEE Signal Process. Lett. 13(5), 281–284 (2006) 23. Sarkis, M., Diepold, K.: Content adaptive mesh representation of images using binary space partitions. IEEE Trans. Image Process. 18(5), 1069–1079 (2009) 24. Adams, M.D.: A flexible content-adaptive mesh-generation strategy for image representation. IEEE Trans. Image Process. 20(9), 2414–2427 (2011) 25. Li, X.: Anisotropic mesh adaptation for image representation. EURASIP J. Image Video Process. 2016, 26 (2016) 26. Candes, E., Romberg, J.: L1-MAGIC. https://statweb.stanford.edu/∼candes/ software/l1magic/. Accessed 15 Sept 2020 27. Kliesch, M.: pit – Pictures: iterative thresholding algorithms. https://github.com/ MartKl/CS image recovery demo. Accessed 15 Sept 2020 28. Edlow, B.L., Mareyam, A., Horn, A. et al.: 7 Tesla MRI of the ex vivo human brain at 100 micron resolution. Sci. Data 6, 244 (2019). https://doi.org/10.1038/ s41597-019-0254-8

Self-exciting Point Processes with Image Features as Covariates for Robbery Modeling Mateo Dulce Rubio1(B) , Paula Rodr´ıguez D´ıaz1 , Juan S. Moreno Pab´ on1 , ´ Alvaro J. Riascos1,2 , and Jorge E. Camargo3 1 Quantil, Bogot´ a, Colombia {mateo.dulce,paula.rodriguez,juansebastian.moreno, alvaro.riascos}@quantil.com.co 2 Uniersidad de Los Andes, Bogot´ a, Colombia 3 Uniersidad Nacional de Colombia, Bogot´ a, Colombia [email protected]

Abstract. State-of-the-art crime prediction models exploit the spatiotemporal clustering patterns and the self-exciting nature of criminality to predict vulnerable crime areas. However, omitting spatial covariates correlated with the occurrence of crimes potentially bias the estimated parameters. This research combines self-exciting point processes, generalized additive models and environmental attributes extracted through convolutional neural networks from street-level images to predict robbery hotspots across the locality of Chapinero in Bogota, Colombia. Our model using image features as covariates outperforms a standard self-exciting point process and shed light on the association between crime occurrence and the socioeconomic and environmental conditions of the city.

Keywords: Self-exciting point process images · Environmental attributes

1

· Crime modeling · Street-level

Introduction

Crime prediction models are frequently used by law enforcement agents as an input for the efficient allocation of police resources. In particular, robberies1 have been subject of academic study for decades from various approaches due to their high frequency, in contrast to other types of crimes such as homicides or 1

We consider a robbery as the act of taking something from a person and using force, or the threat of force to do it. Hence, robbery is considered a theft with the use of violence.

Results of the project “Dise˜ no y validaci´ on de modelos de anal´ıtica predictiva de fen´ omenos de seguridad y convivencia para la toma de decisiones en Bogot´ a” funded by Colciencias with resources from the Sistema General de Regal´ıas, BPIN 2016000100036. The opinions expressed are solely those of the authors. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 886–898, 2022. https://doi.org/10.1007/978-3-030-80119-9_58

SEPP with Images for Robbery Modeling

887

terrorist attacks. In addition, robberies can be statistically proven to satisfy the self-exciting property, which is not an evident property for other types of crime. This particular property allows to accurately model robberies as self-exciting point processes or spatio-temporal models that account for the spatio-temporal spreading of crime. In this regard, models exploiting the self-exciting nature of robbery occurrences, such as self-exciting point process models, are the state-of-art in crime modeling. In particular, the self-exciting point process model proposed by [12] has been successfully implemented in several cities around the world to predict vulnerable crime areas with promising results [3,7,13]. Some extensions of this model incorporate spatial attributes correlated with the occurrence of crime events, to avoid biases in the estimated parameters and study potential risk factors of criminal occurrence [14]. In this work we use a generalized additive model to predict robbery occurrences in Bogota using spatial attributes of the city and a self exciting component. The urban environment of the city is extracted from street-level images through convolutional neural networks and Latent Dirichlet Allocation that summarize the information contained in each image in 14 visual topics. Our model using image features as covariates to predict robbery hotspots in Bogota outperforms a standard self-exciting point process and shed light on the association between crime occurrence and the socioeconomic and environmental conditions of the city. The rest of the article is organized as follows. Section 2 presents literature related to crime prediction models that exploit the self-exciting nature of robberies. Section 3 provides a description of the methodology used to predict robberies in Bogota. Section 4 details the implementation and data sources. Section 5 discusses the results. Section 6 describes the current deployment of the model for the allocation of police resources in the city. Finally, Sect. 7 concludes.

2

Related Work

In [2] the authors compared the predictive capacity of different off-the-shelf crime prediction models using historical crime data from Bogot´ a. They found that the self-exciting point process model proposed in [12] has the best performance among five implemented models. To conclude this, the authors used the Predictive Accuracy Index (PAI) as performance metric, which computes the percentage of crimes correctly predicted per unit of area labeled as hotspot. As a baseline, [2] used a point model which divides the city into pre-defined micro areas and count the number of criminal events that occur in each of these cells [18, 19]. Moreover, they implemented a spatial ellipses model that fits ellipses in areas with high concentration of crime [10]. In addition, they employed a non-parametric Kernel Density Estimation (KDE) model which fits a Gaussian distribution for each record in the dataset centered on the geographic location of each crime and adds such distributions to get a smooth map over the city. This improves the previous two models in that it does not assume predetermined

888

M. Dulce-Rubio et al.

forms of criminal aggregation such as cells or ellipses. The KDE model has been used extensively in the related literature [4,6]. Finally, the authors implemented a Poisson point model [12] which identifies crime vulnerable areas by incorporating spatial and temporal information on past crimes in an epidemic-type aftershock sequence (ETAS) model. This approach divides crimes into background and aftershock events and is estimated using stochastic declustering and KDE. The self-exciting point process model proposed by [12] has been successfully implemented in several cities around the world with promising results [3, 7, 13]. Furthermore, several extensions have been proposed to improve its predictive power and robustness. These extensions include the use of kernels with variable bandwidth [12], marked point processes [11], anisotropic kernels [15], and including spatial attributes to estimate the background rate of the process [14]. The first of these extensions proposes the use of variable bandwidths, in which each kernel uses a bandwidth according to the concentration of criminal events where it is centered. This approach avoids over-smoothing the criminal intensity in some regions and under-smoothing it in others [12]. Another extension uses anisotropic kernels to incorporate geographic restrictions by weighting the different spatial directions in which crime spreads. For example, burglaries are physically limited by areas without residential homes (e.g., parks). This type of restrictions can be incorporated through kernels that penalize certain directions or through the estimation of regularized densities using detailed information on the location of the crimes [15]. One of the limitations of the model proposed in [12] is that omitting spatiotemporal attributes correlated with the occurrence of crimes bias the estimated parameters that capture the spreading effect of the crime. In this regard, [14] extends the model to estimate the background component through a log-linear relationship with spatial variables to model criminal events in Pittsburgh, Pennsylvania. In the same direction, [17] proposes an additive generalized model to predict criminal incidents in discrete predefined units in Charlottesville, Virginia. An advantage of this model is that it captures non-linear associations between the included covariates and the occurrence of criminal events. Among the geographic information included in this work is the location of roads, businesses, and schools, as well as demographic information such as population, housing values, racial distribution, among others. Furthermore, [8] include features extracted from urban images of Chicago, together with climatic and sociodemographic variables, to predict crime in the city using a deep learning approach.

3

Methodology

I. The Self-Exciting Point Process Model The self-exciting point process model (SEPP) proposed by [12] is constructed under three assumptions: (i) criminality concentrates in specific areas of the city (ii) there is a higher incidence of crime at certain times of the day and certain days of the week, and (iii) crime spreads spatially like seismic activity.

SEPP with Images for Robbery Modeling

889

With this in mind, robberies are classified between background and aftershock events, the former being those that arise independently given their spatiotemporal location, while the latter occur as triggering of past crimes nearby. Robbery occurrences are modeled as a self-exciting point process in which the past occurrence of robberies increase the probability of new events occurring in the near future. A spatio-temporal point process is uniquely characterized by its conditional intensity λ(x, y, t), which can be defined as the expected number of points falling in an arbitrarily small spatio-temporal region, given the historical events that occurred up to t. For the purpose of robbery prediction and according to the initial assumptions on the behavior of robbery occurrences, it is assumed that the conditional intensity takes the following functional form  g(x − xk , y − yk , t − tk ), (1) λ(x, y, t) = μ(x, y, t) + {k:tk I_y.mean(): I_y[x] = 3 elif I_y[i] > 0.5: I_y[i] = 2 elif I_y[i] > 0.25: I_y[i] = 1 else: I_y[i] = 0 I = I_y.resh(I.sh[0], I.sh [1]) plt.imshow(I, cmap=‘I’) And if the grayscale pixel values vary or if the grayscale discrepancy is high, we’ll get the correct segments. A successful initialization can benefit from unsupervised and layer-wise pre-training with an Auto-Encoder (AE) or Restricted Boltzmann Device. Dropout and data augmentation can be used to reduce school overfitting.

960

U. A. Usmani et al.

Fig. 4. Segmentation of the CamVid Dataset: In this Dataset these are the Segmentation of some Images from the CamVid Dataset. After the Segmentation Process, we get some very Good Results by our Algorithm.

In the function map of each internal sheet, an induced multi-channel picture is named, where each pixel may be interpreted as a particular feature, and the ability to extract features defines the precision of the regression and target classification. It also makes it easier for you to compare our approach to that of those who use motion, shape, and video segments. Through testing our segmentation algorithm on the same dataset as Segnet, we may compare it to other studies. For this larger sample, the CamVid research library will be used. Figure 4 shows the segmentation of the CamVid Dataset. In this dataset these are the segmentation of some images from the CamVid Dataset. After the segmentation process, we get some very good results by our algorithm which implies that our algorithm extract the objects accurately. Figure 5 shows the comparison of our segmentation result with the SEGNET algorithm (Published in IEEE Transactions PAMI). It is clearly visible by comparing both the images that our reinforcement learning method performs much better than the SegNET algorithm. Figure 6 shows the comparison of our segmentation algorithm on RGB indoor scenes. The segmentation results are better when the classes are more accurately precised. Our results have been compared with many current segmentation algorithms such as Seg-NET, DeepLab, FCN, FCN(LearnDeconV), DeconvNet, etc. and our method performs better than all the other methods.

Enhanced Reinforcement Learning Model for Extraction

961

Fig. 5. Comparison of our Segmentation Result with SEGNET (Published in IEEE Transactions PAMI). It is Clearly Visible by Comparing Both the Images that our Reinforcement Learning Method Performs very well than the SegNET Algorithm.

Fig. 6. Comparison of our Segmentation Algorithm on RGB Indoor Scenes. The Segmentation Results are Better when the Classes are more accurately precised. Our results have been Compared with Many Current Segmentation Algorithms such as SegNET, DeepLab, FCN, FCN(LearnDeconV), DeconvNet

4 Conclusions In the article, the following contributions are made: The description of different segmentation methodologies used in digital image processing is briefly described in this image segmentation report analysis.

962

U. A. Usmani et al.

The paper also discusses conclusions from multiple visualization methodologies utilized in this field of study for image segmentation and other research problems. These methods are particularly useful for detecting and recognizing patterns in margins, images, and points. The image segmentation strategies described in this research article are used by many sophisticated machines to discern faces, items, and patterns. In medical science, picture segmentation is used to distinguish cancerous cells from medical images. Picture segmentation, as the universal segmentation algorithm, has a bright yet bleak future ahead of it, and is reportedly under investigation. There is no one solution that is best for all types of photos, and no single technique is equally acceptable for all types of pictures. Image segmentation remains a difficult problem in image processing and computer vision for the reasons mentioned above, and it is still a pending issue in the region. Still picture segmentation is another methodology that can be used in a multitude of settings.

References 1. Zhu, Q., Shao, L., Li, Q., Xie, Y.: Recursive kernel density estimation for modeling the background and segmenting moving objects. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1769–1772 (2013) 2. Kaganami, H.G., Beiji, Z.: Region-based segmentation versus edge detection. In: Fifth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, pp. 1217–1221 (2009) 3. Dubey, R.B., Hanmandlu, M., Gupta, S.K., Gupta, S.K.: The brain MR image segmentation techniques and use of diagnostic packages. Acad. Radiol. 17(5), 658–671 (2010). https://doi. org/10.1016/j.acra.2009.12.017 4. Rastgarpour, M., Shanbehzadeh, J.: Application of ai techniques in medical image segmentation and novel categorization of available methods and in tools. In: Proceedings of the International MultiConference of Engineers and Computer Scientists (2011) 5. Gomez-Moreno, H., Maldonado, S., Gil-Jimenez, P., Lafuente, S.: Goal evaluation of segmentation algorithms for traffic sign recognition. IEEE Trans. Intell. Transp. Syst. 11(4), 917–930 (2010). https://doi.org/10.1109/TITS.2010.2054084 6. Roth, K.A., Almeida, J.S.: Coming into focus: computational pathology as the new big data microscope. Am. J. Pathol. 185(3), 600 (2015) 7. Kang, W.X., Yang, Q.Q., Liang, R.P.: The comparative research on image segmentation algorithms First International Workshop on Education Technology and Computer. First Int. Workshop Educ. Technol. Comput. Sci. 2, 703–707 (2009) 8. Mitrovi´c, D., Hartlieb, S., Zeppelzauer, M., Zaharieva, M.: Scene segmentation in artistic archive documentaries. In: Leitner, G., Hitz, M., Holzinger, A. (eds.) USAB. LNCS, vol. 6389, pp. 400–410. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-166075_27 9. Mcandrew, A.: An Introduction to Digital Image Processing with Matlab Notes for SCM2511 Image Processing 1 Semester 1,” Image Processing (2004) 10. Choong, M.Y., Kow, W.Y., Khong, W.L., Liau, C.F., Teo, K.T.K.: An image segmentation using normalised cuts in multistage approach. Image 5(6), 7 (2012) 11. Senthilnath, J., Omkar, S.N., Mani, V., Tejovanth, N., Diwakar, P.G., Shenoy, A.: Multispectral satellite image classification using glowworm swarm optimization. In: 2011 IEEE International Geoscience and Remote Sensing Symposium, pp. 47–50 (2011)

Enhanced Reinforcement Learning Model for Extraction

963

12. Liu, Y., Xiao, K., Liang, A., Guan, H.: Fuzzy c-means clustering with bilateral filtering for medical image segmentation. In: Corchado, E., Snášel, V., Abraham, A., Wo´zniak, M., Graña, M., Cho, S.-B. (eds.) HAIS. LNCS, vol. 7208, pp. 221–230. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28942-2_20 13. Xiong, X., et al.: VIIRS on-orbit calibration methodology and performance. Journal of Geophysical Research: Atmospheres 119(9), 5065–5078 (2014). https://doi.org/10.1002/2013JD 020423 14. Yang, L., Zhang, Y., Wang, Z., Rao, N.: Image registration of rabbit tooth soft tissue based on B-Spline. In: 4th International Congress on Image and Signal Processing, vol. 2, pp. 1104– 1107 (2011) 15. Nosrati, M.S., Andrews, S., Hamarneh, G.: Bounded labeling function for global segmentation of multi-part objects with geometric constraints. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2032–2039 (2013) 16. Dandin, O., Teomete, U., Osman, O., Tulum, G., Ergin, T., Sabuncuoglu, M.Z.: Automated segmentation of the injured spleen. Int. J. Comput. Assis. Radiol. Surg. 11(3), 351–368 (2015). https://doi.org/10.1007/s11548-015-1288-9 17. Dogantekin, E., Yilmaz, M., Dogantekin, A., Avci, E., Sengur, A.: A robust technique based on invariant moments – ANFIS for recognition of human parasite eggs in microscopic images. Expert Syst. App. 35(3), 728–738 (2008). https://doi.org/10.1016/j.eswa.2007.07.020 18. Lian, X., Wu, Y., Zhao, W., Wang, F., Zhang, Q., Li, M.: Unsupervised SAR image segmentation based on conditional triplet Markov fields. IEEE Geosci. Remote Sens. Lett. 11(7), 1185–1189 (2014) 19. Shi, Z., Lihuang, S., Li, L., Hua, Z.: A modified fuzzy C-means for bias field estimation and segmentation of brain MR image. In: 25th Chinese Control and Decision Conference (CCDC), pp. 2080–2085 (2013) 20. Wan, Q., Rao, S.P., Kaszowska, A., Voronin, V., Panetta, K., Taylor, H.A., Agaian, S.: Face description using anisotropic gradient: thermal infrared to visible face recognition. Mob. Multimed. Image. Process. Secur. App. 10668, 106–680 (2018) 21. Haralick, R.M., Shapiro, L.G.: Image segmentation techniques. Comput. Vis. Graph. Image Process. 29(1), 100–132 (1985). https://doi.org/10.1016/S0734-189X(85)90153-7 22. Karch, P., Zolotova, I.: An experimental comparison of modern methods of segmentation. In: 2010 IEEE 8th International Symposium on Applied Machine Intelligence and Informatics (SAMI), pp. 247–252 (2010) 23. Kettaf, F.Z., Bi, D., Beauville, J.A.D.: A comparison study of image segmentation by clustering techniques. In: Proceedings of Third International Conference on Signal Processing (ICSP 1996), vol. 2, pp. 1280–1283 (1996) 24. Zhang, Y.J.: A survey on evaluation methods for image segmentation. Pattern Recogn. 29(8), 1335–1346 (1996) 25. Zhang, H., Cholleti, S., Goldman, S.A., Fritts, J.E.: Meta-evaluation of image segmentation using machine learning. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 1, pp. 1138–1145 (2006) 26. Sonawane, M.S., Dhawale, C.A.: A brief survey on image segmentation methods. Int. J. Comput. App. 975, 8887 (2015) 27. Jiang, X., Zhang, R., Nie, S.: Image segmentation based on PDEs model: a survey. In: 3rd International Conference on Bioinformatics and Biomedical Engineering, pp. 1–4 (2009) 28. Macqueen, J.: Some methods for classification and analysis of multi- variate observations. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 281–297 (1967) 29. Matta, S.: Various image segmentation techniques. Int. J. Comput. Sci. Inf. Technol. (IJCSIT) 5(6), 7536–7539 (2014)

964

U. A. Usmani et al.

30. Choi, H.W., Qureshi, N.M.F., Shin, D.R.: Comparative analysis of electricity consumption at home through a silhouette-score prospective. In: 2019 21st International Conference on Advanced Communication Technology (ICACT), pp. 589–591 (2019) 31. Zhu, C., Ni, J., Li, Y., Gu, G.: General tendencies in segmentation of medical ultrasound images. In: 2009 Fourth International Conference on Internet Computing for Science and Engineering, pp. 113–117 (2009) 32. Kapse, R.S., Salankar, S.S., Babar, M.: Literature survey on detection of brain tumor from MRI images. IOSR. J. Electron. Commun. Eng. (IOSR-JECE) 10(1), 80–86 (2015) 33. Madabhushi, A., Metaxas, D.N.: Combining low-, high-level and empirical domain knowledge for automated segmentation of ultrasonic breast lesions. IEEE Trans. Med. Imaging 22(2), 155–169 (2003). https://doi.org/10.1109/TMI.2002.808364 34. Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. 79(8), 2554–2558 (1982). https://doi.org/10.1073/pnas.79.8. 2554

Causal BERT: Language Models for Causality Detection Between Events Expressed in Text Vivek Khetan1(B) , Roshni Ramnani1 , Mayuresh Anand2 , Subhashis Sengupta1 , and Andrew E. Fano1 1 Accenture Labs, San Francisco, USA {vivek.a.khetan,roshni.r.ramnani,shubhashis.sengupta, andrew.e.fano}@accenture.com 2 UC Santa Barbara, Santa Barbar, USA [email protected]

Abstract. Causality understanding between events is a critical natural language processing task that is helpful in many areas, including health care, business risk management, and finance. On close examination, one can find a huge amount of textual content both in the form of formal documents or in content arising from social media like Twitter, dedicated to communicating and exploring various types of causality in the real world. Recognizing these “Cause-effect” relationships between natural language events continues to remain a challenge simply because it is often expressed implicitly. Implicit causality is hard to detect through most of the techniques employed in literature and can also, at times be perceived as ambiguous or vague. Also, although well-known datasets do exist for this problem, the examples in them are limited in the range and complexity of the causal relationships they depict especially when related to implicit relationships. Most of the contemporary methods are either based on lexico-semantic pattern matching or are feature-driven supervised algorithms. Therefore, these methods are more geared towards handling explicit causal relationships leading to limited coverage for implicit relationships, and are hard to generalize. In this paper, we investigate the language model’s capabilities for causal association among events expressed in natural language text using sentence context combined with event information, and by leveraging masked event context with in-domain and out-of-domain data distribution. Our proposed methods achieve the state-of-art performance in three different data distributions and can be leveraged for extraction of a causal diagram and/or building a chain of events from unstructured text. Keywords: Causal relations · Cause-effect · Causality extraction Language models · Causality in natural language text

·

M. Anand—This work was performed during an internship at Accenture Labs, SF. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 965–980, 2022. https://doi.org/10.1007/978-3-030-80119-9_64

966

1

V. Khetan et al.

Introduction

Recent advances in machine learning have enabled progress on a range of tasks. Causal and common sense reasoning, however, has not benefited proportionately. Causal reasoning efforts can be grouped into the task of causal discovery [25] and the task of causality understanding for events described in natural language text [15]. Causality detection in natural language is often simplified/described as detection of “Cause-effect” relation between two events, where an event is expressed as a nominal, phrase, or short span of text in the same or different sentences [22]. Understanding causal association between everyday events is very important for common sense language understanding (e.g., “John was late; Bob got angry.”) as well as causal discovery. For our purposes, however, we will focus on commercial industry applications (e.g., “The phone was dropped in a lake; hence the warranty is void”). Understanding the potential “Cause-effect” relationship between events can form the basis of several tasks such as binary causal questionanswering ([15] and [28]), a plausible understating of the relationship between everyday activities ([12] and [22]), adverse effects of drugs, and decision-support tasks [16].

Fig. 1. Causality in natural language

Most of the current methods for causality understanding write linguistic pattern-matching rules or use careful feature engineering to train a supervised machine learning algorithm [9]. They often lack coverage and are hard to scale for an unseen sequence of events. Identifying the relationship often depends on the whole sentence structure, along with its semantic features. Additionally, it is challenging to decompose every event and the rest of the expression completely in their lexical representation [27]. Furthermore, due to the limited availability of comprehensive annotated datasets for “Cause-effect”, only a few deep learningbased approaches have been proposed [3].

Causal-BERT

967

This work focuses on understanding the causality between events expressed in natural language text. The intent is simply to identify possible causal relationships between marked events implied by a given sequence of text. It is not to determine the validity of those relationships nor to provide a highly nuanced representation of those relationships. The events can be a nominal, a phrase, or a span of text in the given event statement. We built network-architectures on top of language models that leverage overall sentence context, event, context, and events’ masked context for causality detection between expressed events. Pretraining on out of domain data-distribution helped our model gain performance, showing that language models can learn implicit structural representation in natural language text that is in line with the findings of Jawahar, Goldberg and Gururangan [[11, 20] and [14]]. We curated the datasets from Semeval 2010 task 8 [17], Semeval 2007 task 4 [1]1 , ADE [13] corpus. All the sentences in the three mentioned datasets were annotated with the provided event description and were given “Cause-effect” or “Other” labels for each pair of event interactions. Our contributions can be summarised as follows: – We curated three different datasets from publicly available tasks for causality detection in natural language text. – We showed that language models can be trained to learn causal interaction between events expressed in natural language text. – We showed that pre-training of language models using out-of-domain text expression improves the performance in low resource causality detection.

2

Event-Structure and Causality in Natural Language Text

Interaction between events expressed in natural language text is not just dependent on how implicit or explicit the events are mentioned in the expression, but also on the sentence’s arguments and the lexical representation of those arguments [27]. Predicting causal interaction between events is a complex task because it depends on interactions between the particular linguistic expression of information, semantic context established in the text, knowledge of the causal relationships for the domain in question, and the communicative goals of the text’s author (e.g. recognition of a well known causal relationship or conveying a possibly new causal relationship). Causality in the text is expressed in arbitrarily complex ways; even when events are in a single sentence, it might be mentioned in a sparse, ambiguous, or implicit manner. Even explicit causal expressions have rich variation across their syntax, and it is challenging to extract complete lexical features and understand causality based on any single grammatical model. Figure 1 shows a few popular classes of explicit causality as discussed by Khoo et al. [21]. For example, in the 1

We use terms Semeval 2010 and Semeval 2007 for Semeval 2010 Task 8 and Semeval 2007 Task 4, respectively

968

V. Khetan et al.

sentence “smoking causes cancer,” it is easy to understand the causal relationship between smoking and cancer. Nevertheless, it gets complicated when the interaction between events is implicit. In the sentence, “I think the greater danger is from the unelected justices than from the elected Congress and the elected president.” Identifying the “Cause-effect” interaction between “greatest danger” and “unelected justices” from the given text is not a trivial task. The task gets complicated when the “Cause-effect” is not explicitly mentioned between events of interest. Furthermore, as the length of the sentence increases or there is more than one pair of casual events in a given sentence, it becomes difficult for any statistical model to capture the sentence’s context unambiguously. Causality in the text can be expressed simply by using the prepositions of the form “A causes B” or “A is caused by B”. This form of expression is intuitive and can often be subjective to the field and also to the researcher’s views. Instead of predicting “A causes B”, in this work, we predict “could A cause B” from a given textual expression, where causal interaction between events can be explicit or implicit with a wide variety of writing styles.

3

Dataset

We used three different datasets to train and evaluate our models; Semeval 2007 and Semeval 2010 is curated using pattern-based web search while ADE is curated from biomedical text. Semeval 2007, Semeval 2010, and ADE datasets are publicly available and have annotated events and “Cause-effect” interaction between them. Unlike Dunietz et al. and Mirza et al. [6, 24], who incorporated the association between temporal and other relations with causality and Prasad et al. [26], who focuses on inter-sentence causality; in this task, we only focused on causality between events in the same sentence. Table 1 provides statistical details about datasets and Table 2 provides a sample example from each of the curated dataset. Further details on data curation and preparation are below: - SemEval2007: SemEval 2007 [10] is an evaluation task designed to provide a framework for comparing different approaches to classifying semantic relations between nominals in a sentence. The task provides a dataset for seven different semantic relations, including “Cause-effect”. For this work, we use part of the SemEval 2007 dataset with the “Cause-effect” relationship. For a given sentence, if the interaction between marked events is causal, we label it as “Cause-effect” else the sentence is labeled as “Other”. - SemEval2010: Similar to the above dataset, we use SemEval 2010 [17] dataset with causal interaction between events labeled as “Cause-effect”, and all the other types of interactions between events in rest of the sentences are labeled as “Other”. - ADE: The adverse drug effect [13] dataset is a collection of biomedical text annotated with drugs and their adverse effects. The first corpus of this dataset, with drugs causing adverse effects, has drugs as well as effects annotated. In the second corpus, where drugs are not causing any side-effect, the drug and its

Causal-BERT

969

effect name are not manually annotated. We curated a list of unique drugs and affect names using the first corpus data and use this set to annotate the drugs and effect names in the second corpus. While we take sentences with two or more drugs/effect mention in them; for simplicity, we do not replicate the sentence in our final corpus. We marked the first two mention of drug/effect mention we find in the sentence. Table 1. Statistics for curated datasets Train dataset

Test datast

Dataset

Max sentence #Total #Cause-effect #Other #Total #Cause-effect #Other length (train, test)

Semeval 2010

(85, 60)

8000

1003

6997

2717

134

2389

Semeval 2007

(82,62)

980

80

900

549

46

503

ADE

(135, 93)

8947

5379

3568

2276

1341

935

Table 2. Example sentences, sentence with event markers, and masked event markers for curated datasets Curated corpus Dataset

Example sentence

Sentence with event marker

Sentence with masked event marker

Semeval 2007

Most of the taste of strong onions comes from the smell

Most of the taste of strong onions comes from the smell

Most of the blank of strong onions comes from the blank .

Semeval 2010

As in the popular movie “Deep Impact”, the action of the Perseid meteor shower is caused by a comet, in this case periodic comet Swift-Tuttle

As in the popular movie “Deep Impact”, the action of the Perseid meteor shower is caused by a comet , in this case periodic comet Swift-Tuttle

As in the popular movie “Deep Impact”, the action of the Perseid blank is caused by a blank , in this case periodic comet Swift-Tuttle.

ADE

Quinine induced coagulopathy –a near fatal experience

Quinine induced coagulopathy –a near fatal experience

blank induced blank –a near fatal experience

Table 3. Experiment setup Optimizer Learning rate Epsilon Dropout rate Train batch Max sequence length Adam

4

1e-05

1e-08

0.4

16

384

Problem Definition and Proposed Methodology

Our causality understanding approach can be simplified as a binary classification of “Cause-effect”/“Other” relationship between events expressed in natural language text. The detailed methodology, problem definition and network architecture are described below:

970

4.1

V. Khetan et al.

Methodology

Our methodology involves: – Fine-tuning Bert based feed forward network for “Cause-effect”/“Other” relationship label between events expressed in natural language text – Combining both the event’s context and BERT’s sentence context to predict “Cause-effect”/“Other” relationship label between events. This methodology is built on the method suggested by Wu et al. [31] and works on the intuition that the interaction between two events is result of the information in the sentence as well as in the events – Combining both the event’s masked context with BERT’s sentence context to predict “Cause-effect”/“Other” relationship label between events. This methodology is built on the model suggested by Soares et al. [29] and works on the intuition that training with event’s masked context can help language model learn task specific implicit sentence structure ([20] and [14]). 4.2

Problem Definition

The overall problem can be defined as follow: for a given sentence and the two marked events e1 , e2 , the goal is to predict the possible casual interaction c between the events. Mathematically, a given sentence can be seen as a sequence of tokens x, and possible interaction c can be either Cause-effect or Other. Event1 and Event2 are a continuous span of text in the given statement. For a given sequence of token x = [x0 , x1 , x2 , · · · , xn ]

(1)

event1 : E1 = [xi , xj ]

(2)

event2 : E2 = [xk , xl ]

(3)

relation : c ∈ [Cause − effect, Other]

(4)

0 robbery - > incident - > shot. This kind of narrative representation did yield better results than the jumble of words without sequence. We believe more robust models can be made continuing on this path of script type knowledge. Additionally, the poor increase in performance is in part due to the fact that clustering coefficient

Solving the Story Cloze Test Using Graph Feature Extraction

1011

was not developed for the SCT and determining how many triplets are present does not necessarily indicate which ending is the common sense one. Methods such as BERT use a large corpus of unlabeled data to first learn generic word representations. Next, they are able to fine tune models using specific tasks, such as NLI. This involves considering two phrases a premise and a hypothesis and whether they entail or are contradictor [10]. This task is very similar to determining if an ending entails or contradicts the narrative. It is probably techniques such as these which rely on large amounts of data will continue to outperform more human understandable approaches such as graphs.

Fig. 5. This graph shows a story of categorized word relationships that are only related to words in the sentence of the story before or after them.

1012

G. Nathaniel

7 Conclusion Story understanding is an extremely challenging task in natural language understanding with a long-running history. The goal behind the SCT is to require systems to perform deep language understanding and commonsense reasoning for successful narrative understanding, which are essential components for General Artificial Intelligence. In this paper, we explore using word embeddings, parts-of-speech tagging, dependency parses, and sentiment analysis to identity nodes and build graphs from stories and their endings. We then extract features from these graphs to be used in MLP classification. Evaluating on the SCTv1.5 validation set, our holdout dataset, showed a small improvement over the initial baselines from 57.67% to 58.78%, but fell short of state-of-the-art systems to achieve narrative understanding. This paper presents a strong foundation for using various sources of information to build graphs representing narratives. It also establishes methods to evaluate graphs of narratives including feature extraction for neural network classifiers. Cluster coefficient proved to be a robust method of evaluating how well graph nodes cluster together in that it could show signal even on a seemingly unrelated task. It appears from our results and analysis that sentiment is a key factor in correct prediction. However, we believe this area has room for improvement by training sentiment models from story data versus blog or tweet type data. It is also clear that sentiment along is not sufficient, and narrative coherence is critical. While our model did attempt to capture this coherence using word embeddings and NLP information to build our graphs, there is certainly room for improvement. Future work on methods includes exploring using n-gram, of word embedding or POS tags, to build feature vectors for n-grams present over a certain threshold. This work could add more coherence information. Another area for future work we did not explore but shown useful in other research is stylistic features, such as word frequency patterns.

References 1. Charniak, E.: Toward a Model of Children’s Story Comprehension. Ph.D. thesis, MIT (1972) 2. Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., Kohli, P., Allen, J.: A corpus and cloze evaluation for deeper understanding of commonsense stories. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies NAACL HLT 2016, pp 839-849 (2016) 3. Mostafazadeh, N., Roth, M., Louis, A., Chambers, N., Allen, J.F.: LSDsem 2017 shared task: the story cloze test. In: Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics (2017) 4. Schwartz, R., Sap, M., Konstas, I., Li, Z., Choi, Y., Smith, N.A.: The effect of different writing tasks on linguistic style: a case study of the ROC story cloze task. In: Proceedings of CoNLL (2017) 5. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint: arXiv:1810.04805 (2018) 6. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training. Technical report, OpenAI (2018)

Solving the Story Cloze Test Using Graph Feature Extraction

1013

7. Speer, R., Chin, J., Havasi, C.: Conceptnet 5.5: an open multilingual graph of general knowledge. In: AAAI Conference on Artificial Intelligence, pp. 4444–4451 (2017) 8. Sharma, R., Allen, J.F., Bakhshandeh, O., Mostafazadeh, N.: Tackling the story ending biases in the story cloze test. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol. 2, Short Papers), pp. 752–757. Association for Computational Linguistics, Melbourne, Australia (2018) 9. Liu, C., Zhang, H., Jiang, S., Yu, D.: DEMN: distilled-exposition enhanced matching network for story comprehension. In: Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation (2019) 10. Li, Z., Ding, X., Liu, T.: Story ending prediction by transferable BERT. In: IJCAI 2019, pp. 1800–1806 (2019) 11. Cai, Z., Tu, L., Gimpel, K.: Pay attention to the ending: strong neural baselines for the ROC story cloze task. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol. 2: Short Papers). Association for Computational Linguistics (2017) 12. Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: EMNLP (2014) 13. Hutto, C.J., Gilbert, E.E.: Vader: a parsimonious rule-based model for sentiment analysis of social media text. In: Eighth International Conference on Weblogs and Social Media (2014) 14. Luce, R.D., Perry, A.D.: A method of matrix analysis of group structure. Psychometrika 14(1), 95–116 (1949) 15. Watts, D.J., Strogatz, S.H.: Collective dynamics of ‘small-world’ networks. Nature 393(6684), 440–442 (1998). https://doi.org/10.1038/30918

MegaLite-2: An Extended Bilingual Comparative Literary Corpus Luis-Gil Moreno-Jim´enez1(B) and Juan-Manuel Torres-Moreno1,2 1

Laboratoire Informatique d’Avignon (LIA)–Avignon Universit´e, 339 Chemin des Meinajari`es, 84911 Avignon, C´edex 9, France [email protected], [email protected] 2 Polytechnique Montr´eal, Qu´ebec, Canada

Abstract. In this paper we introduced an extended bilingual version of the literary corpus MegaLite. This new version contains literary documents in Spanish and French. The motivation is to provide to the community a specialized and free linguistic support for different NLP tasks. The creation of this genre of corpus is very important for designing algorithms of Text Generation, Text Classification and Sentiment Analysis. The corpora contain about 6 500 documents: 1 500 in French (MegaLite-Fr) and near of 5 000 in Spanish (MegaLite-Es), all collected manually into the genres narrative, poetry and plays. A shallow linguistic comparison using the Jensen-Shannon divergences is presented and discussed. The MegaLite-2 corpora will be available to the community as a free resource in several suitable formats. Keywords: Spanish and French Comparative Literary Corpus · Machine Learning Algorithms · Divergence of probability distribution Linguistic resources

1

·

Introduction

For many years, researchers in Artificial Intelligence areas have addressed their efforts to simulate the creative human process under the intuition of creating artistic artefacts in an artificial way. The purpose is to offer an alternative paradigm to generate artistic artefacts respecting the Creative Process explained by Boden in [2]. As a result, a new area of research in Computational Science has been created, the Computational Creativity [11]. For an optimal performance, the Computational Creativity (CC) demands a large amount of data in order to detect and learn different patterns, necessary for the learning process of artificial models. Today we have algorithms to create paintings, music, poetry, etc. In works focused in literary text generation, it was detected the absence of an enough rich corpora, useful for the training and testing process. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1014–1029, 2022. https://doi.org/10.1007/978-3-030-80119-9_67

MegaLite-2: An Extended Bilingual Comparative Literary Corpus

1015

In this work we introduce a new Spanish/French literary corpus to aim the training process of algorithms focused mainly in literary text generation. This paper is structured as follows. In Sect. 2 we show some works related to the development and analysis of corpora. In Sect. 3 we describe and characterize the new MegaLite-2 corpus. Next, in Sect. 4 we explain the performed training of Word2vec model using the proposed corpus. We also explain the parallel versions of our corpora in Sect. 5 as well as some tables of n-grams in Sect. 6. Then, in Sect. 7 we discuss a superficial linguistic comparison between the two languages included in MegaLite-2 corpus. Finally in Sect. 8, we present some conclusions and ideas for future works.

2

Related Works

The linguistic corpora have been always used for NLP researchers in different ways [12], but recently those composed by literary documents have taken place more strongly in the NLP community. In this section we present some works proposing new corpora for literary language analysis. In [13] the authors describe a new corpus, SL¨ aNDa, constituted by Narrative and Dialogue in Swedish. The corpus contains 44 chapters with over 220K tokens annotated manually. The annotation identified 4 733 instances of cited material and 1 143 named speaker–speech mappings. The corpus is useful for developing computational tools for different types of analysis of literary narrative and speech. The RiQuA corpus, composed by quotations for English literary text [10]. RiQuA comprises 11 works of 19th-century. For each quotation, interpersonal information like speakers’ names and addressees are included providing a rich view of dialogue structure. This corpus is publicly available. For Spanish, the LiSSS corpus has been proposed in [7]. It is constituted by literary sentences collected manually from several literary works. The LiSSS corpus is available in two versions: a first one having 500 sentences manually multi-annotated (13 persons); and a second one having 2 000 manually monoannotated sentences. This small corpus has been annotated into five emotions: love, angry, happiness, hope and fear. The manual annotations have been compared with some automatic classification algorithms. The LiSSS corpus is a free available resource. A first version of the MegaLite corpus was proposed in a preliminary study [8]. It is a monolingual corpus containing near of 5 000 literary documents but only in Spanish. It was a first approach to the bilingual corpora presented in this paper. We also found some bilingual corpora. For example, in [3] a word-aligned corpus between German and Italian is introduced, being a first approach for constructing a real parallel corpus at discourse level. It can be noticed that most of literary text corpora are constituted in others languages different from Spanish or French.

1016

3

L.-G. Moreno-Jim´enez and J.-M. Torres-Moreno

The MegaLite-2 Corpora

Our main objective was the creation of free bilingual literary corpora for the development, training and testing of NLP models. In this way we carry out the conception and creation of corpora having millions of literary phrases in French and Spanish. We have chosen Spanish and French languages because both have a great lexical richness, an important expressivity and a high level of morphological complexity. In spite of their common origin as Romance languages and having a certain closeness, the linguistic structures between Spanish and French differ a lot when they are studied at a literary level. The MegaLite-2 corpora is constituted of two individual corpora: the first one is the MegaLite-Es corpus and the second one the MegaLite-Fr. Both corpora contain some thousands of literary works in Spanish and French, respectively. In terms of number of documents, the French corpus is smaller than the Spanish one, but in terms of linguistic characteristics (words and characters) both corpora share significant similarities. Because of copyrights of several works, the MegaLite-2 corpus can not be entirely distributed as-in. Nevertheless, to avoid this inconvenient, we will disseminate our corpora under several alternative formats1 : – A Word2vec model is trained for each MegaLite-2 corpus. The obtained tables of embeddings are exposed in Sect. 4 – Two parallel versions of MegaLite-2 corpus are detailed in Sect. 5, where: 1. The first version consists in replacing the lexical words2 by its corresponding POS tags, and 2. For the second version, the same words are replaced by its corresponding lemmas. – Then, we construct a set of tables containing n-grams frequency information for each document (Sect. 6). To generate these tables, the corpora were treated separately. The n-grams computed are: 1. Unigrams, 2. Bigrams, and 3. SU4-bigrams (Skip-2 bigrams) [1, 4]. 3.1

Pre-processing

The original documents coming from several heterogeneous formats3 were preprocessed to create utf8 encoded documents. Because of the heterogeneous source documents, the corpus contains a large quantity of errors (truncated, concatenated or splitted words, strange symbols and unusual disposition of paragraphs). These errors are usually found in corpora with same dimensions. In 1 2 3

Different linguistic resources of the MegaLite-2 corpus is available on the website: http://juanmanuel.torres.free.fr/corpus/megalite Lexical words are those containing the most important information in the text (Adjectives, nouns, verbs and adverbs). Like as .pdf, .txt, .html, .doc, .docx, .odt, etc.

MegaLite-2: An Extended Bilingual Comparative Literary Corpus

1017

order to minimise this inconvenient, a carefully segmentation process was performed to split the texts into like-as-sentence units, using a tool developed in PERL 5.0 with suitable regular expressions. This pre-processing reduces some undesirable and uninformative sentences and tokens (editorial stamps, remarks, tables of contents, summaries, copyrights, page numbers, etc.) The MegaLite-2 corpus has the advantage of being extensive and suitable for Machine Learning NLP models, automatic text generation and others NLP tasks. They are also suitable for testing the quality and performance of such algorithms. It has, however, the disadvantage that not all sentences are considered as literary. Many of them are support sentences often containing general vocabulary: these sentences give a fluent reading and provide the necessary links between the ideas expressed among the literary sentences. To preserve the coherence, the support sentences were not suppressed. The documents in the MegaLite-Es and MegaLite-Fr corpora have been named with the following standardized convention: authorLastname, authorName-workName(GENRE).txt The label (GENRE) may take the values: – TEATRO: Plays; – POESIA: Poems, or – void: Narrative Finally, we have grouped alphabetically the files by their author’s last name (see Tables 3 and 5). 3.2

The Spanish Corpus: Megalite-Es

This corpus was constituted by approximately 5 075 literary documents (mainly books) in Spanish coming from 1 336 Spanish-speaking authors or using authors translations from languages other than Spanish. Some statistics of this corpus are detailed in Table 1. Table 1. MegaLite-Es Spanish corpus. M = 106 and K = 103 [8]. Docs Sentences Tokens Chars MegaLite-Es

5 075 15 M

Average per document –

3K

212 M

1 265 M

41.8 K

250 K

The genres and author’s distribution are shown in Table 2. The narrative genre is the most popular genre present in the corpus (over 90% of works). In Table 3 we show the works distribution by author’s name. For example, the first row corresponds to the anonymous author’s number works; in the second row, the same information of authors whom last name starts by letter “A”.

1018

L.-G. Moreno-Jim´enez and J.-M. Torres-Moreno Table 2. MegaLite-2 corpus: genres and authors distribution

Corpus

Plays

Poems

Narrative

Authors Works per author

MegaLite-Es 247 (4.9%) 138 (2.7%) 4 690 (92.4%) 1 336

3.8

MegaLite-Fr

4.4

32 (2.0%)

10 (0.7%) 1 538 (97.3%)

358

Table 3. Corpus MegaLite-Es of literary Spanish documents, recovered from [8]. Docs Authors Sentences Tokens Anonymous

3.3

1634101

Chars

33

1

93959

A

352

84

1174273

16619594 100024972

9216905

B

702

137

1568512

21225072 127892399

C

509

135

1476444

20934719 124866495

D

330

53

1032057

12726072

76063834

E

75

27

236646

3164613

18549948

F

94

39

325573

4956622

29488095

G

239

74

649937

9383297

56205711

H

222

61

844050

10941801

65291682

I

18

9

43934

733205

4323428

J

101

24

368267

5023269

29978644

K

161

29

639763

8727518

51278207

L

332

72

930510

13340874

79340787

M

244

107

633093

9165379

54505778

N

49

19

171528

2004784

12079194

O

32

16

98934

1495109

8880273

P

356

81

992518

16719626

97675796

Q

68

10

52451

702085

4133920

R

135

57

353089

5108727

30374508

S

489

150

1412459

T

119

38

456940

6468959

38356381

U

24

3

25761

414029

2356309

V

209

52

760585

11192371

67226725

W

123

34

418423

5867287

35242050

Y

13

5

37540

640867

3821728

Z

46

12

201073

2963965

17842523

20272228 120623680

The French Corpus MegaLite-Fr

The Megalite-Fr corpus is constituted of 1 580 literary documents in French coming from 358 authors. In the most of cases, the documents were written originally in French. An important part of this corpus comes from the website

MegaLite-2: An Extended Bilingual Comparative Literary Corpus

1019

Bibebook4 , and it is available under the public domain licence Creatives Commons BY-SA5 . Some statistics of this French corpus are presented in Table 4. Table 4. MegaLite-Fr french corpus. M = 106 and K = 103 . Docs Sentences Tokens Chars MegaLite-Fr

1 580 5 M

Average per document –

3.1 K

105 M

621 M

66.4 K

393 K

The genres and author’s names works distribution are shown in Tables 2 and 5.

4

Word2vec Training: Embeddings

In NLP tasks, an embedding (or often called word embedding) aims to quantify and categorize semantic similarities between linguistic items based on their distributional properties in large samples of language data. Embeddings are a powerful word representation for NLP algorithms employing Machine/Deep Learning strategies. To generate embeddings from MegaLite-2 corpus, a semantic analysis was performed implementing a Word2vec model [6] using the Gensim Python library6 . The result of this process are two abstract structures containing 420 757 embeddings for the MegaLite-Es corpus and another one having 171 544 embeddings for the MegaLite-Fr corpus. In Table 6 we show the parameters used to train the Word2vec model, where: Iterations (i) refers to the number of training epochs; Minimal count (m) indicates the minimal number of the occurrences that a word must appear in the corpus to be included in the model’s vocabulary; Vector size (s) specifies the dimension of the embeddings vectors; and Window size (w) represents the windows length’s in words, that will be related to the current word, in the model’s training. In Table 7 some examples of embeddings can be observed. These examples were obtained employing three terms (keywords), indicated in bold. The cosine similarity between the embeddings and the keyword retrieved by the Word2vec model are also shown. The embeddings were (approximately) translated in order to make them more understandable for the reader. In the first column, it can be noticed that the keyword blue has produced embeddings like violet, green and some other colors. Considering that the embeddings are numerical vectors, mathematical operations can be computed. So, if we 4 5 6

http://www.bibebook.com. http://creativecommons.org/licenses/by-sa/3.0/fr. The Gensim library is available on the website https://pypi.org/project/gensim.

1020

L.-G. Moreno-Jim´enez and J.-M. Torres-Moreno

Table 5. Corpus MegaLite-Fr of literary French documents. Docs Authors Sentences Tokens Anonymous

Chars

4

1

20536

327705

1907684

A

74

22

223515

4457632

26484995

B

210

45

545462

11414110

67836946

C

94

33

319802

6576283

38680061

D

244

25

1131212

E

20

5

61285

F

57

14

G

90

21

H

56

17

I

10

J

3

23078186 134800184 1362187

8000691

243264

4510311

26662677

292446

6102004

36186582

235322

4266687

25412860

3

26123

499504

2987246

2

3416

52710

313955

K

21

7

77093

1406500

8116154

L

135

33

416048

7816177

46285075

M

139

30

376771

7239940

42773187

N

10

6

25154

449116

2666096

O

23

7

84664

1562811

9348458

P

88

14

126053

3151424

19064828

Q

0

0

0

0

0

R

95

25

320474

6283031

38070255

S

123

28

390770

8292692

48545287

T

8

3

58399

1254796

7413062

U

0

0

0

0

0

V

39

11

92800

1144530

6721612

W

14

5

60281

1011178

6109955

Y

0

0

0

0

0

Z

23

1

147025

2851778

17297584

Table 6. Word2vec configuration parameters. Parameter Iterations (i) Minimal count (m) Vector size (s) Window size (w)

Value 1 3 60 5

MegaLite-2: An Extended Bilingual Comparative Literary Corpus

1021

Table 7. Word embeddings recovered from MegaLite-Es corpus [8]. Keywords Azul (blue)

Cosine similarity

Amor (love)

Cosine similarity

Mujer (woman)

Cosine similarity

Violet

0.934

Girl

0.930

Hope

0.818

Green

0.930

Old woman

0.915

Honey

0.805

Aquamarine 0.923

Little girl

0.912

Delight

0.801

Purple

0.923

Child

0.909

Enjoy

0.794

Yellow

0.918

Young lady

0.900

Soul

0.789

Crimson

0.915

Little girl

0.892

Passion

0.779

Orange

0.908

Maiden

0.884

Suffering

0.778

Red

0.908

Little old lady

0.871

Affection

0.776

Bluish

0.907

Little woman

0.870

Love

0.775

Silvery

0.901

Girl

0.868

Desire

0.774

do the following operation: (([’blue’ ] + [’ocean’ ]) - [’violet’]), the result is going to be a numerical vector, from which calculating a cosine similarity, the closest embeddings can be recovered. The Gensim library has implemented several functions to compute embedding mathematical operations. This kind of operations generates, for our example, the following embeddings: sea, Pacific, Atlantic, atoll, lake, reef, Mediterranean, Arctic, Ocean, Pantanal. This is an interesting point, because with an appropriated analysis, different semantic fields can be configured. In Table 8, we show the computed embeddings from the same set of queries of the Table 7, but now using the MegaLite-Fr corpus. It can be observed that these embeddings are almost the same than those recovered from the MegaLiteEs corpus. However, we notice that the similarity distances are lower than those presented in Table 7. We could explain this, considering that both corpora have different dimensions. Table 8. Word embeddings recovered from MegaLite-Fr corpus. Keywords Bleu (Blue)

Cosine similarity

Femme (Woman)

Cosine similarity

Amour (Love)

Cosine similarity 0.829

Violet

0.915

Girl

0.944

Soul

Dark

0.912

Mistress

0.857

Filial

0.822

Green

0.886

Daughterin-law

0.856

Egoism

0.813

White

0.884

Demoiselle

0.837

Pride

0.812

Glittery

0.884

Nanny

0.830

Ambition

0.802

Silver

0.867

Housekeeper

0.826

Happiness

0.790

Grey

0.865

Lady

0.824

Inconsistency 0.786

Dark-blue

0.861

Maid

0.822

Platonic

0.784

Bluish

0.860

Peasant

0.813

Blindness

0.782

lis´ er´ e

0.849

Companion

0.796

Attachment

0.780

1022

5

L.-G. Moreno-Jim´enez and J.-M. Torres-Moreno

Study on the Parallel Corpora

Two versions for each MegaLite-2 corpus were constituted, employing a linguistic analysis using the morphological analyzer Freeling [9]. In the first version, we replace the words of each document for its corresponding POS tag7 . For the second version, only the lexical words (verbs, nouns and adjectives) were replaced by its lemmas. In total, four parallel versions are proposed: two for the Spanish corpus and two for the French one. 5.1

POS Version

A morph-syntactic analysis was effectuated over all documents of each corpus. As a result, the words on documents were replaced by its corresponding POS tag. Both corpora were treated separately. For example, for the sentence: s=“En la muerte dejar´ an sufrimientos y penas.” 8 , Freeling performs the following analysis (“token” - POS tag label): – – – – – – – –

‘‘En’’ - SP = preposition ‘‘la’’ - DA0FS0 = female singular article, ‘‘muerte’’ - NCFS000 = female singular article, ‘‘dejar´ an’’ - VMIF3P0 = main verb in future tense, ‘‘sufrimientos’’ - NCMP000 = plural male common noun, ‘‘y’’ - CC = connector, ‘‘penas’’ - NCFP000 = plural female common noun, ‘‘.’’ - Fp = punctuation remark.

5.2

Lemmas Version

For this version, the documents were parsed with Freeling to detect the main kind of lexical words: verbs, nouns and adjectives. Then, these elements were replaced by its corresponding lemmas. A lemma represent the canonized form of a word: without any verbal flexion, the nouns in singular form and male genre. The articles, adverbs, prepositions and the others genres out of lexical words were suppressed. For example, for the sentence previously analysed: “En la muerte dejar´ an sufrimientos y penas.” the lemmatized version is: s(lemma)=‘‘MUERTE DEJAR SUFRIMIENTO PENA’’ (‘‘DEAD LEAVE SUFFERING PAIN’’) In Tables 9 and 10 we show some sentences and its parallel versions taken from MegaLite-Es and MegaLite-Fr corpora, respectively. 7

8

A POS tag offers the grammatical information concerning a determined word, a more detailed information about the POS tag used in our work, can be found in: https://freeling-user-manual.readthedocs.io/en/latest/tagsets/. Approximately translation: In death they will leave suffering and pain.

MegaLite-2: An Extended Bilingual Comparative Literary Corpus

6

1023

N -grams Language Model

In order to give more statistical information helpful for Language Model analysis, we also compute the frequency of n-grams (n = 1, 2 and Skip-grams (SU4)). Table 9. Sentences issued from the theater play “Manfredo” of lord byron in different versions of the corpus. Sentence

POS

19 Mi l´ ampara va a apagarse; por m´ as que quiera reanimar su luz moribunda.

19 DP1CSS VMIP3S0 VMIP3S0 SP VMN0000 PP3CN00 Fx SP CC PR0CN00 VMSP3S0 VMN0000 DP3CSN NCFS000 AQ0FS00

Lemmas

20 Si parece que duermo, no es el sueno el que embarga mis sentidos

20 CS VMIP3S0 CS 20 PARECER DORMIR VMIP1S0 Fc RN VSIP3S0 SONAR EMBARGAR DA0MS0 VMIP1S0 SENTIDO DA0MS0 PR0CN00 VMIP3S0 DP1CPS NCMP000

26 He sido generoso y bienhechor, he encontrado la virtud aun entre los hombres

26 VAIP1S0 VSP00SM AQ0MS00 CC AQ0MS00 Fc VAIP1S0 VMP00SM DA0FS0 NCFS000 RG SP DA0MP0 NCMP000

19 LAMPARA IR APAGAR QUERER REANIMAR LUZ MORIBUNDO

26 GENEROSO BIENHECHOR ENCONTRAR VIRTUD HOMBRE

Table 10. Sentences issued from the novel “Le chasseur d’ours” of Charles Buet in different versions of the corpus. Sentence

POS

Lemmas

5 Le visage respire la bont´ e, la franchise, la simplicit´ e, j’oserai mˆeme dire la candeur.

5 DA0MS0 NCMS000 VMIP3S0 DA0FS0 NCFS000 Fc DA0FS0 NCFS000 Fc DA0FS0 NCFS000 Fc VMIP3S0 Frc VMIF1S0 RG VM N0000 DA0FS0 NCFS000 Fp

5 VISAGE RESPIRER ´ FRANCHISE, BONTE, ´ J’ OSER SIMPLICITE, DIRE CANDEUR.

12 Si mes souvenirs ne me trahissent point, la salle ` a manger et la biblioth`eque n’´ etaient point indignes du salon.

12 CS DP1CPS NCMP000 RN PP1CS00 VMIP3P0 RN Fc DA0FS0 NCFS000 SP VMN0000 CC DA0FS0 NCFS000 AQ0MS0 Frc VMIP 3P0 RN AQ0CP00 SP DA0MS0 NCMS000 Fp

12 SOUVENIR TRAHIR, SALLE MANGER ` BIBLIOTHEQUE N’ ´ ETAYER INDIGNE SALON.

19 Quant ` a la biblioth`eque, elle se composait uniquement de livres de voyage et de chasse.

19 SP DA0FS0 NCFS000 Fc PP3FS00 PP3CN00 VMII3S0 RG SP NCCP000 SP NCMS000 CC SP NCFS000 Fp

` 19 BIBLIOTHEQUE, COMPOSER LIVRE VOYAGE CHASSE

1024

L.-G. Moreno-Jim´enez and J.-M. Torres-Moreno

Unigrams and bigrams were computed in a classical form. For Skip-grams we considered the frequency of a pair of words in a window of four words. Indeed, we take a word and then we skip the following n = 1, ..., 4 words, joining the next word. For example, for the sentence: “The kids are playing in the garden”, we can compose the following SU4-bigrams: [The kids, The are, The playing, The in, kids-are, kids-playing, kids in, kids the, are playing, are in, are the, ...], and so on. In Table 11 we show the first 5 n-grams having high occurrences extracted from three works of the MegaLite-Es corpus. Table 11. N -grams frequency from MegaLite-Es corpus. Unigrams Frequency Bigrams

Frequency SU4-bigrams

Frequency

La se˜ norita de Travelez, Carlos Arniches don

613

don gonzalo

362

don gonzalo

391

gonzalo

402

don marcelino

207

don marcelino

215

numeriano

238

don ar´ıstides

27

gonzalo don

76

marcelino

233

gonzalo don

26

don don

72

picavea

196

por dios

14

numeriano don

55

El reloj, Pio Baroja reloj

8

indiferente horas

5

sombr´ıo indiferente

5

noche

8

reloj sombr´ıo

5

reloj indiferente

5

indiferente

5

tictac met´ alico

5

indiferente tristes

5

sombr´ıo

5

horas tristes

5

tictac met´ alico

5

silencio

5

tristes tictac

4

indiferente horas

5

Juvenilia, Miguel Cane colegio

65

miguel can´ e

miguel

48

colegio nacional

47 8

miguel can´ e colegio nacional

47 8

can´ e

47

buenos aires

7

buenos aires

7

juvenilia

46

doctor ag¨ uero

7

doctor ag¨ uero

7

jacques

44

san ignacio

6

colegio tiempo

6

We repeat the same experiment using the French corpus. In Table 12 we show some examples of extracted n-grams.

7

Study Using Divergence of Probability Distribution

In this section we describe the comparative study performed among the MegaLite-Es and MegaLite-Fr corpora by a Jensen-Shannon method implementation. This analysis was effectuated using the POS tag version corpora, explained in Sect. 5, allowing a comparison at a higher abstracted level. The goal is to demonstrate how different are the writing styles between the French-speak and Spanish-speak authors, avoiding the evident differences of their vocabulary. First, it was randomly taken a fraction of corpora MegaLite-Es POS and MegaLite-Fr POS, generating the two sub-corpora Ref-Es and Ref-Fr. We

MegaLite-2: An Extended Bilingual Comparative Literary Corpus

1025

Table 12. N -grams frequency from MegaLite-Fr Corpus. Unigrams Frequency Bigrams

Frequency SU4-bigrams

Frequency

L’Arroseur, Allais Alphonse jeune

43

p` ete sec

17

p` ete sec

monsieur

29

c’´ etait printemps

10

diplomate autrichien

11

femme

29

diplomate autrichien

10

c’´ etait printemps

11

jour

28

jeune femme

10

jeune femme

10

homme

28

pauvre r´ emouleur

9

pauvre r´ emouleur

17

9

Le cur´ e du village, Honor´ e de Balzac cur´ e

403

cur´ e village

224

cur´ e village

228

graslin

336

madame graslin

217

madame graslin

218

madame

317

monsieur bonnet

98

monsieur bonnet

103

v´ eronique

286

monsieur cur´ e

33

monsieur cur´ e

41

monsieur

283

jean fran¸ cois

30

jean fran¸ cois

30

La s` eve immortelle, Laure Conan jean

141

s` eve immortelle

86

s` eve immortelle

86

tilly

106

jean tilly

37

jean tilly

38 35

d’autr´ ee

90

madame d’autr´ ee

35

madame d’autr´ ee

immortelle

89

monsieur tilly

26

mademoiselle d’autr´ ee

32

s` eve

87

colonel d’autr´ ee

19

monsieur tilly

26

established these two sub-corpora as the languages references. Then we take from MegaLite-2 corpus a random number of works from different authors (in both languages). To measure the Jensen-Shannon Divergences, the works from each author were compared with each language reference. The analysed works and languages references are detailed in Table 13. 7.1

Jensen-Shannon Divergence

The Kullback-Leibler divergence DKL [5,14] calculates the divergence between two probability distributions. For two discrete probability distributions P and Q, the divergence DKL of Q in relation to P is: DKL (P ||Q) =

1  Pw Pw log2 2 Qw

(1)

w∈P

Pw is the probability distribution of words w in the document DP and Qw the probability distribution of words w in the document DQ , where DQ ⊂ DP . The following specification will be used for the probability distribution of words w:  Q Cw /|Q| if w ∈ Q P /|P |; Qw = (2) Pw = Cw δ elsewhere P |Q

where Cw is the number of occurrences of the word w in the distribution P or Q; |P | the length of the document DP in words, |Q| the length of the document DQ in words, and δ > 0 a smoothing factor.

1026

L.-G. Moreno-Jim´enez and J.-M. Torres-Moreno

Table 13. Corpora used to calculate Jensen-Shannon divergence DJ S between a reference corpus (P ) and an author corpus (Q), (M = 106 and K = 103 ). Documents Sentences Tokens Chars Sub-corpora Language references, P in Eq. (3) Ref-Es

391

1.36 M

23.0 M 136.0 M

Ref-Fr

314

1.31 M

32.0 M 192.0 M

40.0 K

1.0 M

Spanish-speaking authors, Q in Eq. (3) Gabriel Garc´ıa M´ arquez

31

6.3 M

Jorge Luis Borges

37

10.0 K

200.0 K 1.2 M

Juan Jos´e Ben´ıtez L´ opez

11

123.0 K

1.7 M

Juan-Manuel Torres-Moreno

51

6.4 K

106.0 K 632.0 K

Julio Cort´ azar

42

35.8 K

942.0 K 5.6 M

Miguel de Cervantes Saavedra 24

44.0 K

1.1 M

6.8 M

Rub´en Dar´ıo

9.6 K

91.6 K

547.0 K

15

10.2 M

French-speaking authors, Q in Eq. (3) Charles de Montesquieu

17

11.0 K

323.0 K 1.9 M

Denis Diderot

12

19.0 K

492.0 K 2.9 M

Paul Henri Corentin F´eval

22

126.0 K

2.6 M

15.9 M

Victor Hugo

24

148.0 K

3.0 M

18.5 M

Gustave Aimard

19

70.0 K

1.8 M

11.3 M

Jensen-Shannon divergence (DJ S ) is the symmetrized version of the divergence DKL : 1 1 1 DKL (P ||M ) + DKL (Q||M ); M = (P + Q) 2 2 2 1 2Pw 2Qw = Pw log2 + Qw log2 2 w Pw + Qw Pw + Qw

DJ S (P ||Q) =

(3)

with the same specification as for (2). The normalized divergence values can be used to measure the informative proximity among the sentences of a pair of documents. 7.2

DJ S Results

Figure 1 shows the results of our experiments. The short-lines in the plot represent each work from each author (Q distribution). Authors are placed according to its divergence DJ S calculated from the languages references (P distribution): Ref-Es (blue lines) and Ref-Fr (red lines). The first five authors in the figure are the French-speak ones. We show that their Ref-Es divergences are the highest ones, unlike the Spanish-speak authors, where their most divergences are

MegaLite-2: An Extended Bilingual Comparative Literary Corpus

1027

Fig. 1. Jensen-Shannon divergences.

lower. An opposed comportment is observed with divergences obtained from Ref-Fr analysis: as expected, the Spanish-speak authors have obtained the highest divergences. The continuous line (with the same corresponding colour) shows the averages of calculated divergences. We conclude that it is possible to analyse and classify the writing authors’ style per language without using any lexical information, but only working on a shallow-level of abstraction, computing the grammatical information. In Fig. 2 we show the mean and standard deviation of the divergences. The plot shows an important overlapping between the standard deviations of the divergences coming from five Spanish-Speak authors: Julio Cort´ azar, Jos´e Luis Borges, Rub´en Dar´ıo and Juan-Manuel Torres-Moreno. A verification of the biography of these 5 authors shows that they stayed several years in France or Belgium. Then, we assume that their writing style has been influenced in some manner with the contact with a French environment (culture, language, etc.). This point could explain the overlapped standard deviations obtained.

1028

L.-G. Moreno-Jim´enez and J.-M. Torres-Moreno

Fig. 2. Standard-deviation calculated from Jensen-Shannon divergences.

8

Conclusion and Final Remarks

In this article we have introduced an extended version of MegaLite corpus, now available in two languages: Spanish and French. Several works composing the MegaLite-2 corpus belong to a private collection, they cannot be distributed in their original format. Therefore, we make available some usefully versions suitable for statistical and morphological analysis, Language Model studies and Machine Learning algorithms. A trained model for semantic analysis based on Word2vec is also available. Based on a statistically-entropy POS-tag analysis, we show that it is possible to establish an empirical boundary between two language’s (grammar) style. We were able to establish this classification by breaking down language barriers. As final remarks, we can say that there are advantages and some limitations in our contribution. – Advantages. Our corpora have an enough size to be used in NLP Machine and Deep Learning algorithms, textual assessment, text generation and text classification tasks. The proposed parallel corpora are varied. This allows to perform different studies focused on NLP applications. – Limitations. The documents contain many auxiliary sentences, giving a grammar coherence and better readability to the text. This kind of sentences usually does not contain literary elements. In addition, the size of

MegaLite-2: An Extended Bilingual Comparative Literary Corpus

1029

MegaLite-Fr corpus is lower than the size of MegaLite-Es (see Sect. 3), so an appropriated dimension adjustment is necessary if a parallel analysis or processing between both corpora is envisaged. This fact could also impact in some studies employing, for example, Neural Networks approaches. In the future, new versions will be included with a greater number of authors and documents, particularly belonging to the genres of poetry and theater. Indeed, we will enrich the MegaLite-2 corpus by adding documents in Portuguese and more documents in French, presenting a similar analysis that the effectuated in this paper, but employing a three-lingual protocol. Acknowledgments. This work was funded by the Consejo Nacional de Ciencia y Tecnolog´ıa (Conacyt, M´exico), grant number 661101, and partially by the Universit´e d’Avignon/Laboratoire Informatique d’Avignon (LIA), France.

References 1. ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81. Association for Computational Linguistics, July 2004 2. Boden, M.A.: The Creative Mind: Myths and Mechanisms. Routledge (2004) 3. Bourgonje, P., Grishina, Y., Stede, M.: Toward a bilingual lexical database on connectives: exploiting a German/Italian parallel corpus. In: Italian Conference on Computational Linguistics-CLIC-IT, pp. 53–58 (2017) 4. Cabrera-Diego, L.A., Torres-Moreno, J.-M.: Summtriver: a new trivergent model to evaluate summaries automatically without human references. Data Knowl. Eng. 113, 184–197 (2018) 5. Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Stat. 22(1), 79–86 (1951) 6. Mikolov, T., Yih, W., Zweig, G.: Linguistic regularities in continuous space word representations. In: North American Chapter of the ACL: Human Language Technologies, pp. 746–751 (2013) 7. Jim´enez, L.G.M., Torres Moreno, J.M.: LiSSS: a new corpus of literary spanish sentences for emotions detection. Computaci´ on y Sistemas 24(3), 1139–1147 (2020) 8. Moreno Jim´enez, L.G., Torres Moreno, J.M.: MegaLite: a new Spanish literature corpus for NLP tasks. In: Computing Conference (2021). accepted 9. Padr´ o, L., Stanilovsky, E.: Freeling 3.0: towards wider multilinguality. In: LREC2012 (2012) 10. Papay, S., Pad´ o, R.: RiQuA: a corpus of rich quotation annotation for English literary text. In: 12th Language Resources and Evaluation Conference, pp. 835– 841, Marseille, France. European Language Resources Association (2020) 11. P´erez, R.P.: Creatividad Computacional. Elibro Catedra, Larousse - Grupo Editorial Patria (2015) 12. Sierra, G.: Introducci´ on a los Corpus Ling¨ u´ısticos. UNAM Mexico (2018) ¨ 13. Stymne, S., Ostman, C.: SL¨ aNDa: an annotated corpus of narrative and dialogue in Swedish literary fiction. In: LREC Conference, pp. 826–834, Marseille. European Language Resources Association (2020) 14. Torres-Moreno, J.-M.: Automatic Text Summarization. Wiley, London (2014)

VBSRL: A Semantic Frame-Based Approach for Data Extraction from Unstructured Business Documents Simone Scannapieco1(B) , Andrea Ponza1 , and Claudio Tomazzoli2 1

Real T S.R.L. - R&TD Department, Corso Venezia 7, 37131 Verona, VR, Italy {simone.scannapieco,andrea.ponza}@realt.it 2 CITERA Interdepartmental Centre, Sapienza University of Rome, Piazzale Aldo Moro 5, 00185 Roma, RM, Italy [email protected] https://www.realt.it/ Abstract. The definition of alternative processing techniques as applied to business documents is inevitably at odds with long-standing issues derived by the unstructured nature of most business-related information. In particular, more and more refined methods for automated data extraction have been investigated over the years. The last frontier in this sense is Semantic Role Labeling (SRL), which extracts relevant information purely based on the overall meaning of sentences. This is carried out by mapping specific situations described in the text into more general scenarios (semantic frames). FrameNet originated as a semantic frame repository by applying SRL techniques to large textual corpora, but its adaptation to languages other than English has been proven a difficult task. In this paper, we introduce a new implementation of SRL called Verb-Based SRL (VBSRL) for information extraction. VBSRL relies on a different conceptual theory used in the context of natural language understanding, which is language-independent and dramatically elevates the importance of verbs to abstract from real-life situations. Keywords: Document processing · Frame semantics · Semantic role labeling · Schank analysis · Natural language processing

1

Introduction and Motivations

It is quite accepted in the data analysis community that most business-relevant information comes in an unstructured form and is typically text-heavy. While several quantitative estimates have been given, a general consensus has not been reached for over three decades: from 80% of the whole data available to an organization, as first pointed out in the late 90s [20] and predicted to be the standard by year 2025 [13], to 85% as reported by big data experts such as Teradata.1 Such high forecasts are definitely plausible for a twofold reason. First 1

https://www.tableau.com/reports/gartner.

c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1030–1044, 2022. https://doi.org/10.1007/978-3-030-80119-9_68

Verb-Based Semantic Role Labeling

1031

of all, industrial digitalization has led enterprises to increasingly adopt a cloudbased paradigm for data creation, analysis, and management. Here at least twothirds of the generated data is unstructured text in the form of e-mails, files in portable data format, rich text documents, and plain text documents [12]. The second reason is, while form-based documents are almost always deemed as structured documents, there are cases where semi-structured documents may still be categorized as unstructured, since the inner structure is not beneficial for most data processing tasks. Trivial examples are HTML files, whose markupbased representations are mainly meant for rendering purposes. Thus, the above reported figures are often underestimates of the actual amount of free-form data produced and manipulated by enterprises. As one might expect, the processing of unstructured documents (and business documents in particular) discloses several critical aspects to tackle. This is particularly the case for automated document classification and information extraction. The former refers to inferring the document type based on its content, the latter to detecting and isolating information of interest—or tags—from the document. These two have witnessed a widespread interest in the industrial sector in order to ease (or even replace) human resources in crucial internal business activities like document filtering, retrieval or manual data entry. 1.1

Motivations

Apart from the structuredness of a document, both classification and extraction share an underlying assumption: the document’s content must be somehow interpreted in order to perform the task. In other words, the aim is to find patterns and regularities in data, which are then fed to proper methods to infer the documentary class, or the set of required tags. Such methods often implement well-known techniques inherited from textual data analytics and mining [1] to Natural Language Processing (NLP) [15]. In particular, Semantic Role Labeling (SRL) and Frame Semantics [7] envision the task of data extraction as strictly dependent on the meaning carried out by a sentence. The underlying idea is to denote situations that characterize human experience by schematic representations called semantic frames. A semantic frame SF is denoted by SF = (LU, SR), where (i) LU = {lu1 , . . . , lun } is a set of specific words or syntagms called lexical units, and (ii) a set SR = {sr1 , . . . , srm } is a set of abstract entities called semantic roles or frame elements. Lexical units are meant to trigger (or evoke) the application of a particular schema on a given sentence, so as to map parts-of-speech to the semantic roles prescribed by the schema. For instance, the semantic frame with LU = {citation, cite, cited, notice to appear, summon, summons} and SR = {Authority, Defendant, Charges} may model the legal scenario where an Authority summons a Defendant based on particular Charges, for instance: [The cop]Authority [cited]lu3 [Mrs. Jones]Defendant for [illegal parking]Charges . Online resources such as FrameNet2 are the result of applying semi-automatic frame extraction techniques to real world annotated corpora such as The British 2

https://framenet.icsi.berkeley.edu/fndrupal/.

1032

S. Scannapieco et al.

National Corpus, and illustrate a plethora of schema definitions (among others, a more complete Citing frame, formalizing the scenario reported above). The extreme potential of semantic frame repositories when combined with methods of intelligent data extraction is unquestionable; nevertheless adopting FrameNet for languages other than English is not straightforward. Despite the significant efforts made on creating FrameNets for other languages,3 the results are promising but far from definitive. The Italian language—which was our first target case in the development of data extraction tools—still currently lacks stable or sufficiently semantically annotated resources [2, 11, 16, 21]. Moreover, the very same maintainers of the original FrameNet are asking themselves not only to what extent are the semantic frames developed for English appropriate for other languages, but also under what circumstances may frames be considered as cross-language or, in other words, universal. One possible aspect fostering these questions is that semantic role labeling techniques on top of FrameNet do not strictly follow a distributional semantics approach. This means it does not assume that the statistical distribution of words in context plays a key role in characterizing their semantic behavior, especially on their important role in frame semantics as lexical units [10]. Coping with extremely expressive languages like Italian, a more rigorous method must be devised that accounts for semantic distribution of terms; in particular, we believe verbs should play a preponderant role in the matter, since most of times their semantic content conveys the meaning of the whole sentence [11]. Driven by these considerations, this paper explores an alternative approach to model SRL, that relies on a language-independent, verb-centric strategy to natural language understanding as firstly devised by Schank [19]. This novel approach, called Verb-Based Semantic Role Labeling (VBSRL), is a variant of Prioritized Semantic Role Labeling (PSRL) [18] in the context of Information Retrieval (IR) and Knowledge Discovery (KD). In Sect. 2 the reader can find a brief digression on other solutions to the question at hand, together with a deep dive into VBSRL in Sect. 3. Finally, Sect. 4 sums up the paper’s findings and explores future work.

2

State of the Art

Business documents are oftentimes classified based on structure, format and type, and subsequently subdivided into structured, semi-structured and unstructured [4,9]. Structured documents characteristically have a fixed layout, and some examples include tax forms, questionnaires and balance-sheets. Unstructured documents lie on the other side of the spectrum. They are distinguishable from semi-structured documents by their complete lack of defined features (like a letter, an article or a contract), in contrast with the semi-structured nature of invoices or purchase orders which include a similar type of information despite their lack of an identical layout or design. 3

See https://framenet.icsi.berkeley.edu/fndrupal/framenets in other languages for a complete summary of all undergoing projects.

Verb-Based Semantic Role Labeling

1033

Document processing and data extraction stem directly from such a document classification, and usually involve model-based or rule-based extraction methods. Sarawagi’s Information extraction [17] is the recommended read for an extensive digression on their differences and implementations. An introductory discussion is held to help motivating the choice to introduce VBSRL in the mix. Rule-based methods work especially well on structured and in few simple semistructured documents. They rely, namely, on a set of rules for data extraction, perform very efficiently during document processing, but are heuristic in nature. They suffer from minimal changes in the layout of the documents, which can break the rules and require reconfiguration. In particular, positional systems operate on prior knowledge from the document corpus, asserting some pieces of information will be found around XY coordinates in the page or trigger terms, like in [8] where the authors improve on a fixed-positional model with their proximity-based positional model. Procedures centered on positional information can be tailored such that they perform as if documents were templated. Templatic documents where spatial layout elements like tables and grid formatting are commonplace do not contain much prose, as the authors point out in [14]. Model-based methods adopt Machine Learning (ML), Natural Language Processing (NLP) and other sophisticated techniques to infer information from documents with a certain statistical confidence. Most of the times, documents are encoded to higher level representations that unveil their informative essence and maximize informative entropy (features). These representations are built either starting from a set of application-dependent features (feature engineering), or through general dimensionality reduction techniques such as (Probabilistic) Latent Semantic Analysis [3,6]. Model-based methods require great expertise and time expenditure for a correct and functional craft, particularly to label the dataset or create it from scratch. Despite these initial modeling constraints, they then adapt in a superior way to changes in the document structure. This is why these elegant systems perform best with semi-structured and of course unstructured documents. Endeavors in this direction typically involve Named Entity Recognition (NER), a research area recently surveyed by the authors of [22]. This paper provides the reader with a new mixed method, joining the two main research branches. In fact, VBSRL provides a way to extract information from unstructured and semi-structured documents via a set of rules modeling natural language.

3

Theory of VBSRL

VBSRL inherits the theoretical background of Schank primitives and conceptual inference [19]. Irrespective of the language used, and similarly to frame semantics, Schank analysis aims to map utterances in natural language to structures (called conceptualizations or schemas) that are unambiguous representations of their meaning. Conceptualizations are grounded on two building blocks: – an Actor-ACTION-Object model (“an Actor performs an actual ACTION on an Object”), where the Object of the ACTION may coincide with the Actor

1034

S. Scannapieco et al.

itself. The Schank schema modeling this concept is depicted in Fig. 1 (a). The symbol between Actor and ACTION may be decorated so as to make the , and for past, future and conditional ACTION tense explicit ( tense respectively) or negation of ACTIONs (for instance, stands for the inability of performing ACTION in the past); – An Entity-State model (“an Entity is in a certain State”), where Entity is a term denoting either an Actor or an Object, while State denotes an attribute of the Entity and is described by means of a logical predicate (see Fig. 1 (b)).

Actor

ACTION

O

Object

Entity

State

(a)

(b)

Fig. 1. Schank basic models: (a) Actor-ACTION-object; (b) Entity-state.

Actor-ACTION-Object and Entity-State schemas may be used in combination to describe more complex situations, like the one depicted in Fig. 2 (a).

Actor

p

ACTION

Joe

State (a)

DO fare R

R

Entity

p

chair sedia

broken rotta (b)

Fig. 2. (a) Schank schema; (b) Schema instantiated w.r.t. s.

The intended meaning of the schema is that “an Actor performed an ACTION that causes an Entity to be in a certain State”, expressed through the causal relationship . Causal relationships are usually marked with the bold letter R, r or E whenever a conceptualized state of affairs is the reason, the result, or an enabling condition behind another conceptualization, respectively. Unmarked causal relationships stand for physical cause. For a given sentence s, a Schank conceptualization is said to be instantiated w.r.t. s if Actors, Entities and ACTIONs of the Schank schema are properly substituted by the corresponding syntagms of s. For example, Fig. 2 (b) reports the instantiation of the Schank schema of Fig. 2 (a) w.r.t. s: “Joe broke the chair” (“Joe ha rotto la sedia”, “Joe ruppe la sedia”).4 Notice that the schema must 4

In the following, all references and examples written in Italian shall be reported in italics with the corresponding English translation in regular typeset.

Verb-Based Semantic Role Labeling

1035

properly explicit the actual conceptual relationships, especially the dependency among Actor and ACTION. As a matter of fact, conceptualizing these types ACTION Object” could of sentences with the simpler schema “Actor violate the rule that conceptual actions must correspond to real world actions: the verb “break” does not refer to any action that actually occurred, but rather to the result of the action that actually occurred (which is unknown). On the other hand, the instance in Fig. 2 (b) correctly depicts the situation by semantically unraveling the sentence as “John did something which is the reason behind the chair being broken”. Objects directly affected by ACTIONs (as, for instance, in Fig. 1 (a)) may also be augmented with information about their change of State. Changes may involve the Physical State of the Object (the directive case denoted by a Dlabeled

), or the State of a Relation involving the Object (the relational case

). In the latter case, the schema must also specify denoted by a R-labeled the particular relation involved. The exact syntax and meaning behind changes of State shall be clear by means of examples in the rest of the discussion. Lastly, conceptualizations resulting from the conjunction (∧) or disjunction (∨) of subconceptualizations are allowed. Schank analysis provides for additional syntax that further empowers the expressiveness of conceptualizations; the interested reader should refer to [19] for a complete overview of the notation. As regards ACTIONs by themselves, Schank theory postulates that only fourteen action types—falling into four distinct categories, namely Instrumental, Physical, Mental, and Global—suffice to conceptualize arbitrary complex statements in natural language. In addition, two particular ACTIONs, i.e. DO (“fare”) and BE (“essere, diventare”) are used as placeholders whenever there is no sufficient information about the ACTION performed by an Actor. 3.1

VBSRL by Examples

Consider the following sentence t extracted from a real business document in its original Italian version, along with its English translation:

Suppose that a text analyzer must be instructed to extract all relevant information from sentence t. “Relevant information” roughly refers to all data that makes the document at hand distinctive with respect to all other document

1036

S. Scannapieco et al.

instances (e.g., names, places, dates, and so on). For the sake of clarity, what follows is the description by examples of a simplified version of VBSRL-empowered data extraction system over t; nevertheless we believe it to be complete enough for the reader to capture all the basic working principles. Verb-Based Schema Construction. Following a semantic frame approach, the analyzer should firstly search for particular situations evoked by t, and represent such situations into a more abstract context. For instance, the possession of a flight ticket recalls a physical transaction between Jane Doe and VoloFake Inc. As such, the system should be able to map the situation to a scenario where (i) “A Recipient is in possession of a Good given by a Donor”. In general, real-time situations of this kind convey much more information than that of Recipient, Good and Donor: for instance, that (ii) “The Good may be specified by a unique Identifier” that (iii) “The Good passing from Donor to Recipient may involve an economic transaction for an agreed Amount of Currency”, and that (iv) “An Intermediary may act on behalf of the Recipient by purchasing the Good from the Donor and giving the Good to the Recipient”. The overall conceptualization, named Sc1 , is modeled using Schank semantics as in Fig. 3 (a)), while Fig. 3 (b)) reports Sc1 instantiated w.r.t. t. Being the schema semantically much more abstract than the specific situation evoked by t, it is reasonable that several roles in Sc1 cannot be instantiated when applied to t (the Intermediary, the Amount and Identifier). Sub-schema 1 in Fig. 3 (a) models concepts (i) and (ii) and introduces a Global ACTION called ATRANS, which takes as Objects an abstract binary relation (in this case, OWNERSHIP) that holds between two real-world Entities (in this case, Good and one of either Donor or Recipient). In other words, through ATRANS, Donor acts so that OWNERSHIP(Good, Donor) ceases and OWNERSHIP (Good, Recipient) comes into force. Sub-schema 2 models concepts (iii) and (iv) and stands as the cause why OWNERSHIP relation on Good of sub-schema 1 has changed. There is the conjunction of two state of affairs: an Intermediary acted so that OWNERSHIP(Currency, Intermediary) changed into OWNERSHIP(Currency, Donor) with Amount as an attribute of Currency (i.e., the Intermediary gave Currency quantified as Amount to Donor—sub-schema 3), and the Donor acted so that OWNERSHIP(Good, Donor) changed into OWNERSHIP(Good, Recipient) (i.e., the Intermediary gave the Good to the Recipient—sub-schema 3). Two other conceptualizations should also be used in order to map relevant information, that model: – a scenario where (i) “An Agent starting from an Agent Source uses a Medium so that an Object makes a change of Physical State from a Source to a Destination”, and (ii) “The Medium may be specified by a unique Identifier” (the corresponding schema Sc2 and its instantiation w.r.t. t are reported in Fig. 4 (a) and (b), respectively);

Verb-Based Semantic Role Labeling

1037

Recipient ATRANS

Donor

O

OWNERSHIP: Good

R

Donor

1

Identifier R

Donor Intermediary

ATRANS

O

R

OWNERSHIP: Currency

Intermediary Amount

3

2 Recipient ATRANS

Intermediary

O

OWNERSHIP: Good

R

Intermediary Identifier

4

(a) Jane Doe ATRANS

VoloFake Inc.

O

OWNERSHIP: ticket biglietto

R

VoloFake Inc. PR28H3 R

VoloFake Inc. Intermediary

ATRANS

O

R

OWNERSHIP: Currency

Intermediary Amount

Jane Doe Intermediary

ATRANS

O

OWNERSHIP: ticket biglietto

R

Intermediary PR28H3

(b)

Fig. 3. Abstract transaction scenario: (a) The schank conceptualization we call Sc1 ; (b) The instantiation of Sc1 w.r.t. t.

1038

S. Scannapieco et al.

Medium Agent

PTRANS

O

Object

Identifier

D

Agent_Source

Destination Medium

PROPEL

O

Medium

D

Source Identifier

Identifier

(a)

Jane Doe

PTRANS

O

Jane Doe

D

plane aereo

AB123C

Agent_Source

Berlin Tegel plane aereo

PROPEL

O

plane aereo

D

Milan Malpensa AB123C

AB123C

(b)

Fig. 4. Physical transition scenario: (a) Schank conceptualization Sc2 ; (b) Instantiation of Sc2 w.r.t. t.

– a scenario where (i) “A Causer prevents a Subject to perform an ACTION on an Object”. Moreover, (ii) “The Causer is the responsible of the Cause preventing the Subject to perform the ACTION”, and (iii) “Since the ACTION may have been prevented during its execution, the Object is left in a Resulting State” (see schema Sc3 in Fig. 5 (a)). Notice that Sc3 does not specify the action type performed on the Object; in such cases, the scenario is too generic, and a step back should be taken to reach a trade-off between schema abstraction and flexibility. A possible concretization Sc4 of Sc3 considers cases where (iv) “the ACTION implies a change to the Physical State of the Object from a Source to a Destination” (see Fig. 5 (b)), which suits sentence t well (see the instantiation of Sc4 w.r.t. t in Fig. 5 (c)). The above examples clearly show that Schank schema construction follows the same principles of semantic frames construction: modeling real-life situations by choosing a set of relevant, involved entities (the semantic roles) and the interactions among them. In VBSRL, semantic roles are defined by means of Actors, Objects and States, while the interactions among semantic roles are the result of combining changes of States involving Objects, causal relationships and sub-schemas conjunctions/disjunctions.

Verb-Based Semantic Role Labeling

1039

BE

Cause BE

Cause

R R

DO

Causer DO

Causer

R R

Destination Subject

ACTION

O

Object

PTRANS

Subject

O

Object

D

Source r r

Object

Resulting_State Object

Resulting_State

(a)

(b) BE

hijacking dirottamento R

DO

VoloFake Inc.

R

Berlin Tegel PTRANS

Jane Doe

O

Jane Doe

D

Milan Malpensa

r

Jan Doe

Berlin Schoenfeld

(c)

Fig. 5. Prevented action scenario: (a) Schank conceptualization Sc3 w.r.t. an undefined action type; (a) Schank conceptualization Sc4 w.r.t. an action type involving a change of physical state; (c) Instantiation of Sc4 w.r.t. t.

Undoubtedly, by increasing the number of semantic roles, the schema shall result more and more versatile; this is the case of schema Sc1 , where the original set of Donor, Recipient and Good has been further augmented with Amount, Identifier and Intermediary). As a consequence, the same identical conceptualization is able to model a plethora of completely different utterance templates. For the sentence u “Mrs. Jane Doe bought a flight ticket from VoloFake Inc. [...]”, the system simply evokes schema Sc1 and obtains an extension of Sc1 instantiation w.r.t. t in Fig. 3 (b) where Jane Doe is also the Intermediary; on the same pace, the sentence v “VoloFake Inc. has gifted a flight ticket to Mrs. Jane Doe” recalls the same schema, and the system obtains an extension of the Sc1

1040

S. Scannapieco et al.

instantiation w.r.t. t with Jane Doe as both the Recipient and the Intermediary, and where the Amount is equal to 0. In summation, the text analyzer should access a repository SC = {Sc1 , . . . , Scn } of Schank Conceptualizations that constitutes the system’s capabilities to infer general scenarios based on specific situations. The repository must be built a priori following distinct courses of action: from the isolation of recurrent abstractions from a text corpus of arbitrary size, to a systematic translation of all FrameNet definitions to their corresponding Schank schemas. Either way, an increasing ability to abstract corresponds to a mere augmentation of the knowledge base with no modifications to existing schema definitions. Verb-Based Schema Selection. As mentioned in the previous section, the system must now be instructed on which abstraction(s) to apply on a given sentence during its analysis. This amounts to say that the system knowledge must be augmented with further information about the lexical units that trigger the execution of each Schank conceptualization in the repository. More formally, this new knowledge is given by the set of Schema Triggering n ST = i=1 ({lui1 , . . . , luik }, Sci ) where Sci ∈ SC = {Sc1 , . . . , Scn }, where each lu is a term or a syntagm belonging to a given dictionary and where k depends on i (since a single schema conceptualization shall be triggered by different lus). In this specific case, with SC = {Sc1 , Sc2 , Sc4 }, a reasonable mapping between lexical units and  Schank schemas may be ST = ({possess, possessed, possession, . . .}, Sc1 ) ({fly, flown, flight, depart, departure, arrive, arrival, . . .}, Sc2 ) ({suffer, suffering, hijack, hijacked, hijacking, . . .}, Sc4 ). Notice that lexical units may be standardized by specifying them in canonical representations, that is to say, verbs are made explicit in their infinitive form, substantives and adjectives in their singular male form, and so on. A simple POS tagging of t prior to this phase is sufficient to guarantee a correct Schank schema selection, and is used henceforth as a standard procedure. During the construction of a more organic schema repository, something quite expected emerges: the totality of the lexical units used to trigger any of the Schank conceptualizations was either a verb or a substantivization of a verb (taking synonyms and hypernyms into account). This is perfectly in line with the adoption of a verb-driven method for semantic labeling, and again fosters the intuition that the semantic content of verbs in a sentence is the only responsible for grasping the nature of the involved roles. Trigger Entity Selection and Target Entity Mapping. Consider again the application of Schank schema Sc1 on the example sentence. In this case, the text analyzer is now aware that the sentence could contain high-level entities (henceforth, Root Entities) such as a Donor, a Recipient, a Good, an Intermediary and so on. Nevertheless, the desired behavior is not only to extract the actual syntagms pertaining the Entities, but also to specify (or specialize) them into low-level entities which are tailored for the context at hand.

Verb-Based Semantic Role Labeling

1041

The examined text passage has been extracted from a legal document received by a firm mainly dealing with airline claims, where the people offended in the legal dispute are passengers, and where the object of the dispute essentially regards the reimbursement of flight tickets issued by flight companies. For a document processing based on Schank conceptualizations to be effective over an arbitrary number of such legal documents, the system must be able to automatically infer that the Root Entity Recipient is a Passenger, that the Donor is a Flight Company, or even that the Intermediary is usually a Travel Agency. This clearly involves ontology-based representations of entities [5], where the roots are entirely and solely the Root Entities involved in each Schank conceptualization in SC. This is exactly the way the present framework determines the set of target tags which are later extracted (if found). In the same way action types are the only responsible for framing a text passage into a determined set of conceptualizations, we argue that determined values (Trigger Terms) of a subset of Root Entities involved in each Schank schema (Trigger Entities) are the only ones responsible for specializing the entire set of Root Entities for a particular context. In the case of Sc1 , the Entity Good is sufficient to contextualize all other entities present into every type of trade scenario. Again, data extraction should rely on additional knowledge of two types: (i) a set of Trigger Entity Mappings n TREM = i=1 (Sci , {trei1 , . . . , treik }) where Sci ∈ Sc = {Sc1 , . . . , Scn } that denotes the subset of Trigger Entities for each Schank conceptualization in the schema repository, and (ii) a set of Target Entity Mappings TAEM containing tuples of the type (Sci , trej : trtk ) → {(rei1 → tae1 ), . . . , (reil → tael )} meaning that, if in Schank schema Sci the Trigger Entity trej assumes the value (Trigger Term) trtk , each of the l Root Entities re is mapped to a corresponding Target Entity tae. It is required for Target Entity tael to appear in the ontology representation having the Root Entity reil as the topmost tree node. In other words, tael must appear as leaf in the tree-based ontology representation starting from reil . For example, if (Sc1 , {Good}) ∈ TREM, then Sc1 specified for airline claim disputes is represented via the tuple (Sc1 , Good : flight ticket) → {(Donor → Flight Company), (Recipient → Passenger), (Amount → Ticket Price)} {(Identifier → Ticket ID), (Intermediary → Travel Agency)} in TAEM. Target Entity Extraction. The last phase detects the syntagms in the text that match with each Root Entity. In this regards, the method is very similar to that devised in [18] for logical complement analysis. Schank conceptualizations unveil the inner semantic content of the roles involved in a scenario and, as such, implicitly give syntactic rules to map the role to the corresponding text chunk. For instance, in schema Sc1 , each possible physical transaction of a Good takes place from a Donor to a Recipient. In Italian logical analysis, the Recipient usually appears as the subject (soggetto, henceforth denoted with the label SOGGETTO) of the sentence, the Donor as either a complement regarding possession (complemento di

1042

S. Scannapieco et al.

specificazione, with label C SPECIFICAZIONE) or place (complemento di moto da luogo, with label C MOTO DA LUOGO), while the Good is the object (complemento oggetto, with label C OGGETTO). Other important logical complements are those focusing on the agent performing the main action (complemento di agente, with label C AGENTE) and focusing on the beneficiary of an action (complemento di termine, with label C TERMINE). Each complement for a particular language is uniquely determined by particular syntactic constructions that are easily translated into logical analysis rules. For Italian, the Logical Rule SOGGETTO → (il, lo, la, i, gli, le) + NOUN/ NPR means that “a noun or a named entity (possibly) preceded by either il, lo, la, i, gli, le particle is tagged as a soggetto”. In a similar manner, rules C SPECIFICAZIONE → dello, della, degli, delle + NOUN/NPR, C TERMINE → a, allo, alla, ai, agli, alle + NOUN/NPR and C AGENTE → dallo, dalla, dagli, dalle + NOUN/NPR dictate how the system should recognize a complemento di specificazione, a complemento di termine and a complemento d’agente. Let LR be the set of all Logical Rules specified to the system, and COMP the set of Complement labels denoting the logical complements that appears on the left side of any logical rule in LR. The last step is to determine which kind of logical complement to look for in the sentence that corresponds to a specific Root Entity for a fixed Schank conceptualization. This further knowledge is encoded into tuples of the kind (lui , Scj ) → (rek → cl ), where cl ∈ COMP; the intended meaning is that, whenever the Schank schema Scj is triggered by lexical unit lui , then the Root Entity rek should appear as complement denoted by label cl . All these rules are collected in the set LCM of Logical Complement Mappings. Data extraction by means of sets LR and LCM are extremely flexible in cases where different sentences ascribable in the same conceptualization, appear either in their active form (e.g., “Mrs. Jane Doe purchased a flight ticket from VoloFake Inc.”, “La sig.ra Jane Doe ha acquistato un biglietto aereo dalla VoloFake Inc.”), or in their passive form (e.g., “A flight ticket has been purchased by Mrs. Jane Doe from VoloFake Inc.”, “Un biglietto aereo della VoloFake Inc. `e stato acquistato dalla sig.ra Jane Doe”), or even using a verb or its antonym (e.g. “VoloFake Inc. sold a flight ticket to Mrs. Jane Doe”, “La VoloFake Inc. ha venduto un biglietto aereo alla sig.ra Jane Doe”). Scenario Sc1 should conceptualize all these cases and correctly extract the correct syntagms into the corresponding Target Entity Recipient by setting: – ST (introduced above) in a way that {buy, bought, sell, sold, selling} appear as lexical units for Sc1 in ST; – TREM and TAEM (as introduced above); – LR to contain Logical Rules for SOGGETTO, C AGENTE, and C TERMINE; – LCM = {(comprare, Sc1 ) → (Recipient → SOGGETTO)}  {(essere comprato, Sc1 ) → (Recipient →C AGENTE)}  {(vendere, Sc1 ) → (Recipient →C TERMINE)}.

Verb-Based Semantic Role Labeling

4

1043

Conclusions

Irrespective of their structuredness, most business documents retain relevant information in natural language which usually required human interpretation for a successful extraction. Nowadays, automation efforts have delivered on their promise of successful information extraction with differing degrees of accuracy, precision and overall results. Frame semantics have shown promising outcomes for rule-based systems, all the while model-based systems have had radical and drastic boosts in all the metrics relevant to this task. In this paper, we have presented an alternative approach to Semantic Role Labeling which differs from the standard techniques underlying the generation of online resources like FrameNet. Schank analysis was chosen to model the equivalent of semantic frames (Schank schemas) that are language independent by construction and where verbs unleash all their semantic potential to give meaning to the overall sentence. The theoretical dissertation germinated in [18] and took its time to properly manifest itself into a coherent theory that is applicable to real world document processing. The authors are currently applying the theoretical findings to a brand new document processing architecture, and developing a prototype for classification and data extraction from unstructured legal documents in Italian. This should lead to obtaining meaningful results regarding the factual applicability and validation of such an important theoretical achievement to real world scenarios, which is matter left for future work.

References 1. Aggarwal, C.C.: Data Mining - The Textbook. Springer, Cham (2015). https:// doi.org/10.1007/978-3-319-14142-8 2. Basili, R., Brambilla, S., Croce, D., Tamburini, F.: Developing a large scale FrameNet for Italian: the IFrameNet experience. In: Basili, R., Nissim, M., Satta, G. (eds.) Proceedings of the Fourth Italian Conference on Computational Linguistics CLiC-it, Collana dell’Associazione Italiana di Linguistica Computazionale, Rome, pp. 59–64. Associazione Italiana di Linguistica Computazionale, Accademia University Press, December 2017 3. Cristani, M., Tomazzoli, C.: A multimodal approach to exploit similarity in documents. In: Ali, M., Pan, J.-S., Chen, S.-M., Horng, M.-F. (eds.) IEA/AIE 2014. LNCS (LNAI), vol. 8481, pp. 490–499. Springer, Cham (2014). https://doi.org/10. 1007/978-3-319-07455-9 51 4. Cristani, M., Bertolaso, A., Scannapieco, S., Tomazzoli, C.: Future paradigms of automated processing of business documents. IJIM 40, 67–75 (2018) 5. Cristani, M., Cuel, R.: A survey on ontology creation methodologies. Int. J. Semantic Web Inf. Syst. 1(2), 49–69 (2005) 6. Cristani, M., Tomazzoli, C.: A multimodal approach to relevance and pertinence of documents. In: Fujita, H., Ali, M., Selamat, A., Sasaki, J., Kurematsu, M. (eds.) IEA/AIE 2016. LNCS (LNAI), vol. 9799, pp. 157–168. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-42007-3 14 7. Fillmore, C.J.: Frame Semantics, pp. 111–137. Hanshin Publ. Co., Seoul (1982)

1044

S. Scannapieco et al.

8. Huynh, D.T., Zhou, X.: Exploiting a proximity-based positional model to improve the quality of information extraction by text segmentation. In: Wang, H., Zhang, R (eds.) Proceedings of the Twenty-Fourth Australasian Database Conference. ADC 2013, Adelaide, Australia, vol. 137, pp. 23–31. Australian Computer Society Inc, January 2013 9. Kabak, Y., Dogac, A.: A survey and analysis of electronic business document standards. ACM Comput. Surv. 42(3), 11:1–11:31 (2010) 10. Lenci, A.: Distributional semantics in linguistic and cognitive research. Rivista di Linguistica 20(1), 1–31 (2008) 11. Lenci, A., Johnson, M., Lapesa, G.: Building an Italian FrameNet through semiautomatic corpus analysis. In: Proceedings of International Conference on Language Resources and Evaluation (LREC), Valletta, Malta (2010) 12. Laxmi Lydia, E., Kannan, S., Suman Rajest, S., Satyanarayana, S.: Correlative study and analysis for hidden patterns in text analytics unstructured data using supervised and unsupervised learning techniques. Int. J. Cloud Comput. 9(2/3), 150–162 (2020) 13. MacGillivray, C., Reinsel, D.: Worldwide global DataSphere IoT device and data forecast, 2019–2023. Technical report US45066919, International Data Corporation (IDC), Framingham, MA, USA, May 2019 14. Majumder, B.P., Potti, N., Tata, S., Wendt, J.B., Zhao, Q., Najork, M.: Representation learning for information extraction from form-like documents. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6495–6504. Association for Computational Linguistics, July 2020 15. Manning, C.D., Sch¨ utze, H.: Foundations of Statistical Natural Language Processing. MIT Press, Cambridge (2001) 16. Montemagni, S., et al.: Building the Italian syntactic-semantic treebank. In: Abeille, A. (ed.) Treebanks, pp. 189–210. Springer, Dordrecht (2003). https://doi. org/10.1007/978-94-010-0201-1 11 17. Sarawagi, S.: Information extraction. Found. Trends Databases 1(3), 261–377 (2008) 18. Scannapieco, S., Tomazzoli, C.: Shoo the spectre of ignorance with QA2SPR - an open domain question answering architecture with semantic prioritisation of roles. In: Armano, G., Bozzon, A., Cristani, M., Giuliani, A. (eds.) Proceedings of the 3rd International Workshop on Knowledge Discovery on the WEB. CEUR Workshop Proceedings, vol. 1959, Cagliari, Italy. CEUR-WS.org, September 2017 19. Schank, R.C.: The fourteen primitive actions and their inferences. Technical report, Stanford University, Stanford, CA, USA (1973) 20. Shilakes, C.C., Tylman, J.: Enterprise information portals. Techreport, Merrill Lynch (1998) 21. Tonelli, S., Pighin, D., Giuliano, C., Pianta, E.: Semi-Automatic Development of FrameNet for Italian (2009) 22. Yadav, V., Bethard, S.: A survey on recent advances in named entity recognition from deep learning models. In: Bender, E.M., Derczynski, L., Isabelle, P. (eds.) Proceedings of the 27th International Conference on Computational Linguistics COLING, Santa Fe, New Mexico, USA, pp. 2145–2158. Association for Computational Linguistics, August 2018

Generating Correction Candidates for OCR Errors using BERT Language Model and FastText SubWord Embeddings Mahdi Hajiali, Jorge Ram´ on Fonseca Cacho(B) , and Kazem Taghva Department of Computer Science, University of Nevada, Las Vegas, Las Vegas, NV 89154, USA {Mahdi.Hajiali,Jorge.FonsecaCacho,Kazem.Taghva}@UNLV.edu

Abstract. In this paper, we present a method to identify candidates for Optical Character Recognition (OCR) generated errors during the post processing stage using the BERT language model and FastText subword embeddings. Our results show that given an identified error, the model is able to consistently generate the correct candidate for 70.9% of the errors. Keywords: OCR post processing · Language Modeling FastText · Natural language processing

1

· BERT ·

Introduction

OCR is the process of converting a scanned image of text to a recognizable non-image electronic form of the text, performed by an OCR engine, so that the data can be edited using a word-processing computer program. Although OCR engines are constantly being improved to higher accuracy, many issues still remain when the source image has poor resolution, low contrast, or a noisy or complex background [24]. In other words, OCR is not perfect and even the best OCR engines have errors and require some correction in order to produce satisfactory text from the image; therefore, OCR text needs post-processing to refine the results and achieve improved accuracy. Furthermore, the quality of OCRed texts is crucial for achieving good performance in different Nautural Language Processing (NLP) tasks [27] such as Named-entity recognition (NER), Topic Modelling, Information Retrieval [23], and Sentiment Analysis [11]. For instance, the effects of OCR errors on Information Retrieval (IR) is well studied in [3,22,25]. Generally, errors do not affect average precision due to the robustness of the IR engines. In extreme cases, such as very low accuracy text obtained from historical data or short documents, one can observe retrieval degradation. The IR systems consider an OCR error as a unique word, and c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022  K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1045–1053, 2022. https://doi.org/10.1007/978-3-030-80119-9_69

1046

M. Hajiali et al.

consequently, it will be indexed. This leads to a considerable index size. The post processing helps in reducing the index size. Post-processing OCR approaches basically involve three tasks of detecting the errors, generating the candidates, and correcting the errors. Our focus in this paper is on generating the candidates. In this paper, we propose an approach to generate correction candidates for OCR errors. This method uses pre-trained BERT Masked Language Modeling in generating correction candidates. The remainder of this paper is structured as follows: We first introduce literature review in Sect. 2 to provide some background information. Section 3 is dedicated to the proposed methodology. The Results and Conclusion are described in Sects. 4 and 5, respectively. Finally, we have the acknowledgment.

2

Literature Review

Many approaches have been proposed for OCR post processing over several years. In 2016, Mei et al. [15] proposed a method that can correct 61.5% of the OCR errors in the same dataset that we are using for our model. In their approach, they use Damerau-Levenshtein distance to find the candidates among all the words in the vocabulary. Using universal Levenshtein automaton is another approach used in the work of Schulz et al. [20] for selecting candidate sets from dictionaries. In [16], Levenshtein automata (a deterministic finite state machine) is employed to suggest correction candidates. Furrer et al. [10] proposed a method to find correction candidates by using similar character substitutions and searching shared trigrams. In the work of Taghva and Agarwal [21], Google’s indexed data is utilized and OCRed tokens are sent to the Google search engine as search queries. If the query contains an error, the search engine will suggest replaceable candidates for the errors. In Cappelatti et al.’s [2] work, PyEnchant spell checker is employed to suggest corrections and a modified Needleman–Wunsch algorithm is used to rank those suggestions. In [9] OCRSpell [26] was used to identify the OCR generated errors. Once identified, in order to select possible candidates to correct such errors, the predecessor and successor words to the error are used to search for matching 3-grams in Google Web-1T [5], where the middle token is allowed to be any word. These words are the candidates that are then refined (remove unlikely candidates) by using Levenshtein edit distance [13], along with the frequency of the 3-gram in Google Web 1T to choose the most likely candidate. This context-based approach has the benefit of giving the most likely candidate in a similar way to how a proof reader would choose the most likely word to correct an error based on the surrounding words or context. When it comes to using BERT for OCR error correction, we found Nguyen et al. made use of Neural Machine Translation techniques and character embedding in their correction mechanism by using a pre-trained BERT model for error detection in their model [18].

Generating OCR candidates with BERT

3

1047

Methodology

Our proposed method consists primarily of two parts. In the first part, we employ BERT [4] pre-trained masked language model to generate correction candidates. In the second phase, we train FastText [1] on our data and utilize it in order to produce another set of proper candidates for OCR errors. Ultimately, we combine the results of the two parts. 3.1

BERT

A model is called a language model if it assigns probabilities to a sequence of tokens. In other words, a language model can predict the probability of the appearance of a word, or character, within a sequence based on the words, or characters, already observed in the sequence. BERT [4] is a language model published by Google that is pre-trained on a corpus of books and other media known as BooksCorpus [30]. It is also trained using the English Wikipedia (2,500M words). This model can be used for multiple NLP tasks such as NER, Question Answering, Sequence Prediction, and Sentiment Analysis [12]. BERT employs Transformer architecture [28], an attention mechanism that learns contextual relationships between words or subwords. Figure 1 one shows the architecture of a Transformer model. The transformer includes two procedures: an encoder mechanism that reads the text input, and a decoder mechanism that produces a prediction for the many NLP tasks that can be addressed by this architecture [29]. The difference with BERT is that it is a special form of a transformer, where it uses only the encoder mechanism of the transformer. Because of this, it is known as Bidirectional “Encoder” Representations from Transformers (BERT). Given a sequence of N tokens, t1 , t2 , ..., tN , the forward language model predicts the probability of the sequence as: p(t1 , t2 , ..., tN ) =

N 

p(ti |t1 , t2 , ..., ti−1 )

i=1

The backward language model reverses the sequence and predicts the probability of the sequence as: p(t1 , t2 , ..., tN ) =

N 

p(ti |ti+1 , ti+2 , ..., tN )

i=1

BERT combines the forward language model and the backward language model together; hence, it is a “Bidirectional” Encoder Representation from Transformers (BERT). One can see the architecture of the BERT model in Fig. 2.

1048

M. Hajiali et al.

Fig. 1. Transformer model architecture. Adapted from [28].

BERT is pre-trained on two NLP tasks: Masked Language Modeling (MLM) and Next Sentence Prediction. In the MLM task, 15% of the words randomly in each sequence are replaced with a [MASK] token, and the model has to predict the masked words using the entire masked sequence. We use this feature of BERT to generate candidates for OCR errors. For example, suppose the following sentence is what an OCR engine has generated from a given image: their smaller size, being verj^ pale green, marked with dark reddishbrown blotches, spots, and sometimes lines, and with underlying siennareddish spots; most specimens are principally marked at the larger end.

Generating OCR candidates with BERT

1049

Fig. 2. BERT model architecture. Ti is the Corresponding Output Embedding; Ei is nth Token in the Input Sequence; and Trm is the Transformer Block. Source: Adapted from [4].

The word ‘verj∧ ’ is an OCR generated error. Its correct form is the word ‘very’. In order to feed the sentence to BERT, we need to replace the error with the [MASK] token: their smaller size, being [MASK] pale green, marked with dark reddishbrown blotches, spots, and sometimes lines, and with underlying siennareddish spots; most specimens are principally marked at the larger end.

Given the sentence above, this is the list of candidates the BERT Masked Language Model can generate for the masked token: ‘very’, ‘generally’, ‘a’, ‘mostly’, ‘usually’, ‘mainly’, ‘relatively’, ‘typically’, ‘predominantly’, ‘rather’, ...

We use this list as a list of correction candidates, in which we can hopefully find our correct word. As you see, the first word in the list of BERT suggestions is the correct word, ‘very’. 3.2

FastText

A word embedding is mapping of words into a multi-dimensional space, a vector of real numbers. Word embeddings can place similar words near one another in that space.

1050

M. Hajiali et al.

Word embedding methods such as Word2Vec [17] and GloVe [19] take in the semantics and similarity information of words into embedding. However, they fail to generalize to “Out Of Vocabulary” words (OOV) that were not part of the training set. Unlike Word2Vec and GloVe, FastText [1], which is developed by Facebook, does not take a word as one atomic unit, but rather a union of its character combinations. In other words, FastText, which is an extension of skipgram word2vec [17], computes embeddings for character ngrams, as well as word ngrams. By creating a word vector from subword vectors, FastText makes it possible to exploit the morphological information and to create word embeddings, even for words never seen during the training. In FastText, each word, w, is represented as a bag of character n-grams. Considering the word ‘whose’ and n = 3 as an example, the word will be represented by the character n-grams: . g Let Gw be the set of n-grams appearing in w. FastText associates a vector #» to each n-gram in Gw . Given a training corpus represented as a sequence of words w1 , ..., wT , the objective of the FastText model is to maximize the following log-likelihood: T    [ (s(wt , wc )) + (−s(wt , n))] t=1 c∈Ct

n∈Nt,c

where C is the chosen context position, Wc is the context word, Wt is the word at the position t, Nt,c is a set of negative examples sampled from the vocabulary,  is the logistic loss function log(1 + e−x ), and the scoring function s is:  #» s(w, c) = g · #» c. g∈Gw

where #» g is the vector associated to each n-gram in Gw . The negative sampling algorithm can be calculated in the same form as in the skip-gram [17] model with the aforementioned formula above. In order to use FastText to generate candidates for each error, first we need to train it on our OCRed text data, in which our errors exist. We then need to find similar words to the error by computing the cosine similarity of the word embedding vectors. The cosine similarity between two vectors u = {u1 , u2 , . . . , uN } and v = {v1 , v2 , . . . , vN } is defined as: N u·v i=1 ui vi =  similarity(u, v) = N 2  N 2  |u||v| i=1 ui i=1 vi

Generating OCR candidates with BERT

1051

Similar to the example we had for BERT, this the list of candidates FastText can generate for the word ‘verj∧ ’: ‘ver’, ‘verj’, ‘jerk’, ‘very’, ‘everj’, ‘pre’, ‘ver3’, ‘everjr’, ‘ever3where’, ‘gre3er’, ...

In this step, we can also determine whether a candidate is a correctly spelled English word or not by using a dictionary like Oxford English Dictionary. If the word exists in the dictionary, it is spelled correctly. Otherwise, it is not a correctly spelled word, and we can remove it from the list of candidates.

4

Results

In our work, we used the MiBio dataset [14], which contains fully aligned OCR texts generated by the Tesseract OCR engine, and the ground truth recognition texts of the book “Birds of Great Britain and Ireland”. By applying BERT on the dataset and generating the top 50 candidates, we could find the proper candidate for 49.95% of the errors. By applying FastText, likewise on the same dataset, we could generate the actual candidate among the top 50 candidates suggested by FastText for 53.40% of the OCR errors. By combining the lists of candidates of BERT and FastText, our presented method is able to generate proper candidates for 70.9% of the errors. (See Table 1). Table 1. The accuracy of the proposed method in generating proper candidates for the 2,348 OCR errors that exist in the MiBio dataset. BERT

49.95%

FastText 53.40% Total

5

70.99%

Conclusion and Future Work

One of the key steps in correcting the OCR errors is generating correction candidates for each error. In this paper we proposed an approach for generating plausible candidates for OCR errors, where in 70.9% of the cases we generated the correct candidate for an already identified error. Although generating proper candidates is one of the most important steps in OCR Post Processing, a complete approach also involves detecting the errors and choosing the best suggestion among the list of candidates. In our work, we focus on candidate generation after assuming that the words that contain errors have already been detected as errors. Detecting errors is its own art and many tools, from as simple as Aspell, to OCR focused tools like OCRSpell [26], to Corpus based tools such

1052

M. Hajiali et al.

as Google Web 1T [9], can be used to identify the errors if the goal is to make a comprehensive system. For future iterations of this work, we plan on proposing a new method to choose the best candidate among the list of candidates (like what we presented in [6, 7], where we used SVMs to great success), as well as suggesting more accurate candidates lists for each OCR error. In order to encourage reproducible research [8], the source code and implementation of the model discussed in this paper is freely available on GitHub (https://github.com/Mahdihajiali/). Acknowledgment. This material is based upon work supported by the National Science Foundation under Grant No. 1625677. For more details please see: www.nsf.gov/ awardsearch/showAward?AWD ID=1625677

References 1. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017) 2. Cappelatti, E.: Post-correction of OCR errors using pyenchant spelling suggestions selected through a modified needleman-wunsch algorithm. In: International Conference on Human-Computer Interaction, pp. 3–10. Springer (2018) 3. Croft, W.B., Harding, S.M., Taghva, K., Borsack, J.: An evaluation of information retrieval accuracy with simulated OCR output. In: Symposium on Document Analysis and Information Retrieval, pp. 115–126 (1994) 4. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018) 5. Evert, S.: Google web 1t 5-grams made easy (but not for the computer). In: Proceedings of the NAACL HLT 2010 Sixth Web as Corpus Workshop, pp. 32–40. Association for Computational Linguistics (2010) 6. Cacho, J.R.F.: Improving OCR post processing with machine learning tools. University of Nevada, Las Vegas, Phd diss. (2019) 7. Cacho, J.R.F., Taghva, K.: OCR post processing using support vector machines. In: Science and Information Conference, pp. 694–713. Springer (2020) 8. Cacho, J.R.F., Taghva, K.: The state of reproducible research in computer science. In: 17th International Conference on Information Technology-New Generations (ITNG 2020), pp. 519–524. Springer (2020) 9. Cacho, J.R.F., Taghva, K., Alvarez, D.: Using the google web 1t 5-gram corpus for OCR error correction. In: 16th International Conference on Information Technology-New Generations (ITNG 2019), pp. 505–511. Springer (2019) 10. Furrer, L., Volk, M.: Reducing OCR errors in gothic-script documents (2011) 11. Hajiali, M.: Big data and sentiment analysis: a comprehensive and systematic literature review. Concurrency Comput. Pract. Exper. 32(14), e5671 (2020) 12. Heidari, M., Rafatirad, S.: Using transfer learning approach to implement convolutional neural network model to recommend airline tickets by using online reviews. In: 2020 15th International Workshop on Semantic and Social Media Adaptation and Personalization, SMA, pp. 1–6 (2020) 13. Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet physics doklady, vol. 10, pp. 707–710 (1966)

Generating OCR candidates with BERT

1053

14. Mei, J., Islam, A., Moh’d, A., Wu, Y., Milios, E.E.: Mibio: a dataset for OCR post-processing evaluation. Data Brief 21, 251–255 (2018) 15. Mei, J., Islam, A., Wu, Y., Moh’d, A., Milios, E.E.: Statistical learning for OCR text correction. arXiv preprint arXiv:1611.06950 (2016) 16. Mihov, S., Koeva, S., Ringlstetter, C., Schulz, K.U., Strohmaier, C.: Precise and efficient text correction using levenshtein automata, dynamic web dictionaries and optimized correction models. In: Proceedings of Workshop on International Proofing Tools and Language Technologies (2004) 17. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013) 18. Nguyen, T.T.H., Jatowt, A., Nguyen, N.-V., Coustaty, M., Doucet, A.: Neural machine translation with Bert for post-OCR error detection and correction. In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pp. 333–336 (2020) 19. Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014) 20. Schulz, K., Mihov, S., Mitankin, P.: Fast selection of small and precise candidate sets from dictionaries for text correction tasks. In: Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), vol. 1, pp. 471–475. IEEE (2007) 21. Taghva, K., Agarwal, S.: Utilizing web data in identification and correction of OCR errors. In: Document Recognition and Retrieval XXI, vo. 9021, p. 902109. International Society for Optics and Photonics (2014) 22. Taghva, K., Borsack, J., Condit, A.: Evaluation of model-based retrieval effectiveness with OCR text. ACM Trans. Inf. Syst. (TOIS) 14(1), 64–93 (1996) 23. Taghva, K., Borsack, J., Condit, A.: Information retrieval and OCR. In: Handbook of Character Recognition and Document Image Analysis, pp. 755–777. World Scientific (1997) 24. Taghva, K., Borsack, J., Condit, A., Erva, S.: The effects of noisy data on text retrieval. J. Am. Soc. Inf. Sci. 45(1), 50–58 (1994) 25. Taghva, K., Nartker, T.A., Borsack, J., Condit, A.: UNLV-ISRI document collection for research in OCR and information retrieval. In: Document recognition and retrieval VII, vol. 3967, pp. 157–164. International Society for Optics and Photonics (1999) 26. Taghva, K., Stofsky, E.: Ocrspell: an interactive spelling correction system for OCR errors in text. Int. J. Doc. Anal. Recogn. 3(3), 125–137 (2001) 27. van Strien, D., Beelen, K., Ardanuy, M.C., Hosseini, K., McGillivray, B., Colavizza, G.: Assessing the impact of OCR quality on downstream NLP tasks. In: ICAART (1), pp. 484–496 (2020) 28. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017) 29. Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45 (2020) 30. Zhu, Y., et al.: Aligning books and movies: towards story-like visual explanations by watching movies and reading books. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 19–27 (2015)

Measuring Meaning Ignazio Mauro Mirto(B) Università di Palermo, 90128 Palermo, Italy [email protected]

Abstract. This study will present a tool designed for meaning extraction with monoclausal sentences in Italian. Its main features will be illustrated with instances of the Italian causative clause type featuring the verb fare ‘make’ (e.g. Egli fa piangere il bambino ‘He makes the child cry’), a construction which invariably embeds a clause (e.g. Il bambino piange ‘The child cries’) with which it establishes an entailment relationship. The tool automatically accomplishes the following tasks: it answers relevant wh- questions (e.g. Who cries? Who makes someone cry?) and detects the entailment. Concurrent with this presentation, this study will also encourage a reflection on the research currently being conducted in Computational Linguistics and Natural Language Processing, in which data-driven approaches prevail over rule-based systems. Keywords: Semantic roles · Entailments · Paraphrasing · Question answering

1 Introduction The Italian language features a causative construction with the verb fare ‘to do, to make, to have’ which appears to be an excellent test-bed for parsing tools designed for meaning extraction. This causative clause type, which features two predicates but is monoclausal (see [1]: 11–43), is exemplified below: (1) Max fa piangere Carlo ‘Max makes Carlo cry’

In order to ascertain if one (human or machine) ‘understands’ the meaning conveyed by (1), many questions could be asked. This paper will propose wh- questions such as the following ones: (1a) Who cries? (1b) Who makes someone cry?

It seems realistic enough to claim that a tool capable of automatically providing correct answers to the above questions would ‘understand’ a non-trivial portion of the meaning which (1) carries (any pragmatic, social, or cultural shades of meaning are excluded). Let us now consider the following sentences, which instantiate a type of the pseudocleft construction (see [2]), as answers to questions (1a) and (1b) respectively: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1054–1067, 2022. https://doi.org/10.1007/978-3-030-80119-9_70

Measuring Meaning

1055

(1c) Carlo is he who cries (1d) Max is he who brings about the crying

Together, (1c) and (1d) detect, first, the two participants which sentence (1) brings into linguistic being (i.e. Max and Carlo), and, second, the relationships which (1) establishes between such participants. The aim of this paper is to present a tool, named NLPYTALY , which automatically1 : (i) identifies any instances of the causative construction exemplified in (1); (ii) generates whquestions such as (1a) and (1b); and (iii) answers them with clauses such as (1c) and (1d)2 . Section 2 will be devoted to how the tool represents meaning. Section 3 will explore the entailment relationship that a fare-causative regularly establishes with a simpler clause and show how a novel type of meaning representation permits the tool to capture such relationships between propositions, for example that between (1) and the embedded Carlo piange ‘Carlo cries/is crying’. Section 4 will briefly present a comparison with other systems designed for meaning extraction, whilst Sect. 5 will motivate CSRs and the representation of meaning through Noun-CSR pairings. Section 6 will illustrate how the tool can be employed for Question answering. Section 7 will outline the methodology and Sect. 8 will show how the core meaning of sentences is measured. Finally, Sect. 9 will draw some conclusions.

2 Semantic Representation In order to illustrate the main features of the tool, the fare-causative in (2) below is employed (rfl3 = 3rd person reflexive clitic): (2) L’uomo accompagnato dalla guardia del corpo si è RFL3 is the man accompanied by.the bodyguard fatto dare il denaro dal tizio arrivato ieri made give the money by.the guy arrived yesterday

‘The man (who was) accompanied by the bodyguard had the money given to him [= received the money] by the guy (who) arrived yesterday’ 1 The tool label includes: (a) the acronym NLP, (b) the initial letters of the programming language

deployed, that is Python (3.8), (c) the Italian TAL acronym for NLP (Trattamento Automatico della Lingua), and (d) a revised spelling of the language being analysed. The system does not currently access Python libraries for use with NLP, excluding NLTK, employed for text tokenisation. NLPYTALY makes use of TreeTagger, a probabilistic tagger (the parameter files are those by Helmut Schmid, https://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/). The tagset is by Achim Stein (https://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/data/ita lian-tagset.txt). A repository of the tool, which currently includes the code used for the parsing and the detection of e.g. diathesis and constituency, is available on GitHub at the following link: ignaziomirto2017/nlpytaly. 2 When the infinitive of a fare-causative is an intransitive verb with no transitive counterpart (NLPYTALY contains about 1,000 of such verbs), its subject almost always surfaces as a direct object, and the correct answers (e.g. (1c)) can be easily calculated. With the subject of transitive verbs, however, a few idiosyncratic pairings (i.e. misalignments) may occur, e.g. with aspettare ‘to wait’. Such verbs require an ad hoc treatment.

1056

I. M. Mirto

Structurally, (2) is more complex than (1) in four regards: first, the sentence is reflexive and, as a consequence, the subject is assigned two semantic roles (see [1]: 39– 40). Second, the head of the subject NP is modified by the past participle of a transitive verb (accompagnato), which also licenses the first by-phrase (dalla guardia del corpo). Third, the noun in the second by-phrase (i.e. dal tizio) is modified by the past participle of an intransitive verb (arrivato). Finally, whilst (1) involves two participants (these include [± Animate] nouns), (2) involves four, namely, l’uomo ‘the man’, la guardia del corpo ‘the bodyguard’, il denaro ‘the money’, and il tizio ‘the guy’. Upon typing sentence (2), NLPYTALY returns the following results (the tool’s answers are in Italian only): Table 1. Results returned by NLPYTALY with sentence (2) PARTICIPANTS

A B C D

l’uomo the man la guardia del corpo the bodyguard il denaro the money il tizio the guy

E

l’uomo*

F

il tizio*

G

l’uomo*

RELATIONS BETWEEN EVENT PARTICIPANTS

è is è is è is è is è is è is è is

chi è accompagnato he who is accompanied chi accompagna he who accompanies cosa è dato what is given chi dà he who gives a chi si dà he who is given chi arriva he who arrives chi determina il dare he who brings about the giving

The seven lines in Table 1 are filled with equative sentences, i.e. sentences with the surface structure ‘NP be NP’ (the second NP must be definite), which provide answers to wh- questions of the type shown in (1a) and (1b), e.g. Who accompanies (someone)?, What is given?, Who arrives?. The sentence in line A, for example, answers the whquestion Chi è accompagnato?’Who is accompanied?’ and is to be read as follows: L’uomo è chi è accompagnato ‘The man is he who is accompanied’. Thus the answers which the tool returns are in the form of either canonical or reversed pseudo-clefts (see [2]). The following Section will supply the necessary details on how the tool extracts, generates, and organises the information contained in the above table.

3 Participants and Cognate Semantic Roles Table 1 contains seven lines because the tool detects an equal number of predicateargument relations: two relations (lines A and B) are to be attributed to the past participle of the transitive verb accompagnare ‘accompany’; three relations (lines C, D, and E)

Measuring Meaning

1057

come from the ditransitive dare ‘give’; one depends on the past participle of the intransitive arrivare ‘arrive’ (line F); finally, another one originates from causative fare (line G, see [1]: 17–18). The column termed ‘Participants’, i.e. the one formed by the NPs on the left of the copula, is the by-product of this detection, i.e. the linking of each participant to its licenser(s), which requires the identification of: (i) the sentence diathesis, (ii) the grammatical relations direct object and subject, and (iii) the two by-phrases (the tool functions performing these tasks are available on a GitHub repository at the link ignaziomirto2017/nlpytaly). As pointed out above, sentence (2) yields four participants (in Table 1, repetitions are marked with an asterisk; notice how the participant l’uomo is the argument of three distinct predicates, whilst il tizio is the argument of two). Each participant in the left column is paired with a semantic role, which is expressed with the clausal NPs (see Sect. 4) standing on the right of the copula. For reasons which will be explained shortly, such semantic roles are termed Cognate Semantic Roles (henceforth, CSRs, see [3]). The word cognate is here employed as researchers do either ˇ in diachronic linguistics (e.g. Latin MUSCA and Italian mosca are cognate words) or for the so-called cognate objects, also known as figura etymologica (see [4], which informally introduces CSRs). A sentence establishes a number of relationships between the participants it expresses. CSRs are meant to express such relationships. In Table 1, the clausal NPs on the right of the copula are invariably introduced by the pronouns chi ‘who’ (or a chi ‘to whom’) or cosa ‘what’3 , followed by one of the following verbal forms: (i) the active form of the verb (mostly for notional subjects), (ii) the passive one (in general for notional direct objects) or (iii) an impersonal one, as in line E (mainly for indirect objects). In both Italian and English, the outcome recalls the style of certain proverbs, e.g. He who laughs last laughs loudest, or a frequent sentence pattern in the Bible, e.g. The one who perseveres to the end will be saved (Matthew 24:13). Worthy of note is the fact that, unlike Italian (see the lines A, B, D, F, and G in Table 1), the counterpart of chi in English pseudo-clefts, i.e. who, must be preceded by a pronominal element (e.g. he who, the one who). CSRs depend on predicate-argument relations. They are expressed by using the content morpheme of the verb which is etymologically related to the predicate licensing arguments. Admittedly, CSRs appear as if they are “stating the obvious” ([5]: 139), which is why their raison d’être will be provided below. For the time being, let us have a glance at just one of the advantages they offer, that is entailment detection. To do this, consider that sentence (2) encloses the simple clause in (3) (shown below in a simplified version, that is without past participle modifications): (3) Il tizio diede il denaro all’uomo ‘The guy gave the money to the man’

The representation of meaning outlined above permits the identification of the entailment. This is so because, upon typing (3), the tool will return the same participants and CSRs shown in the lines C, D, and E of Table 1. This partial identity of CSRs allows the tool to ‘understand’ that sentence (2) entails sentence (3) (obviously, the opposite is not true). 3 This choice depends on the [± Animate] value characterizing nouns and noun phrases.

1058

I. M. Mirto

4 Related Work Since NLPYTALY places emphasis on predicate-argument structures, the closest models appear to be FrameNet4 and PropBank5 (we have no knowledge of semantic parsers specifically designed for the Italian causative construction as highlighted above)6 . The data-driven FrameNet and PropBank base their representations of meaning upon Case Grammar roles (see [6]), although with a number of differences (see [7]). Such semantic roles require lexical resources which “provide the training ground for automatic semantic role labellers” ([8]: 929). For example, in proposing the sentences David cooked the rashers and Lorcan heard the door shut, Saeed ([9]: 140) analyzes the subject of the former as an Agent and that of the latter as an Experiencer. This difference in semantic roles depends on the verb and the context, and its automatic identification will require lists of verbs associated with semantic roles (as in the PropBank corpus). Manual annotations are therefore indispensable and the level of accuracy may vary. The representation of meaning with Case Grammar roles can be effectively illustrated with predicate-argument triples, e.g. in Table 2 for the sentence Mary laughed: Table 2. Predicate-argument triple with Mary Laughed A

B

C

PREDICATE ARGUMENT WORDING Laughed

Mary

Agent

Above, A stands for the part of the parsed string functioning as a predicate and assigning a semantic role, whilst B corresponds to the part of the parsed string which (i) functions as an argument of A, and (ii) is assigned a semantic role. Finally, C is the wording required to express the meaning originating from the syntactic relationship between A and B. In each of the three columns in Table 2, the bottom line consists of words. However, whilst the words in A and B come from the string being parsed, and are therefore ‘language in use’, the words to be used in C can be considered as metalanguage. Needless to say, the metalinguistic choice in C is of paramount importance, it being the outcome of a semantic theory and of its syntax-semantics interface. For example, unlike in Case Grammar, the metalanguage of classical logic is often a formula, as Table 3 illustrates (the sentence Fido is a dog and the logical notation in bold come from [5]: 141):

4 See https://framenet.icsi.berkeley.edu/fndrupal/ (last consulted 12/12/2020). 5 See [16] and [17]. 6 Another semantic model based on predicate-argument relations is known as Tectogrammatical

representation. See https://ufal.mff.cuni.cz/pedt1.0/tecto.html and [18].

Measuring Meaning

1059

Table 3. Predicate-argument triple with Fido is a dog A

B

C

PREDICATE ARGUMENT WORDING Dog

Fido

f DOG

The semantic roles introduced above, i.e. CSRs, differ from the models in Tables 2 and 3 only as far as C is concerned, as Table 4 illustrates: Table 4. Predicate-argument triple with Mary laughed A

B

PREDICATE ARGUMENT Laughed

Mary

C WORDING >she who laughs
… she); second, the relative pronoun copies the subject case and the [+Human] feature of the Argument (from Mary/she to > who, see fn. 3 with regard to the feature Animacy); lastly, the verbal form is computed by first using the content morpheme of the Predicate (thus A and C will be co-radical or cognate) and by inflecting it at the third person of the present tense (from laughed to > laughs). With notional direct objects, the passive form is generally needed, as in lines A and C of Table 1. It appears reasonable to claim that the meaning representation illustrated in Table 4 is as granular as possible. The other side of the coin is the proliferation of semantic roles, a fact which imposes a degree of lexical reduction regarding the vocabulary to be employed in C7 . The ‘reward’ –it could be stated– of such granularity is the fully automatic formation of the indispensable metalanguage in column C (this automatic formation is also possible in Table 3). Thus, unlike the well-known semantic roles devised in Case Grammar, e.g. Agent, Patient, etc., CSRs can be obtained in a fully automatic fashion (suppletive forms may be required). For example, with the above sentences David cooked the rashers and Lorcan heard the door shut, David is >he who cooks< whilst Lorcan is >he who hears< . Since CSRs can be extracted automatically, they can be employed with unstructured 7 A promising model of vocabulary reduction could be that of dictionaries destined for use by

non-native learners, in which the number of headwords (paralleled with columns A and B) far exceeds the number of words used in the definitions (paralleled with the metalanguage in column C).

1060

I. M. Mirto

texts. No training is needed and neither manual annotation nor manual disambiguation is necessary. A model exploiting CSRs appears to be in line with Zellig Harris’s view of metalanguage. The metalanguage of natural languages was a main concern of Harris (see [10]: 123–141, [11]: 293–351, 377–391), especially when compared with the metalanguage of other disciplines, particularly mathematics. In a paper published posthumously [12], Harris outlined the steps which led to “the recognition of the metalanguage as being part [a sublanguage] of natural language” ([12]: 1). According to his idea, “the metalanguage of a natural language is a natural language” [10: 274, our emphasis]. Moreover, Harris speaks of metalanguage in terms of statements and sentences: “the metalinguistic statements are themselves sentences of natural language” ([10]: 274, our emphasis). We therefore believe that there is ground to suppose that neither labels nor formulas would be adequate metalanguage in Harris’s theory. As Palmer, Gildea, and Xue write in the introduction of their 2009 book Semantic Role Labeling [7], syntactic parsers “are a long way from representing the full meaning of the sentences that are parsed. In particular, they do not specify ‘Who did What to Whom, and How, When and Where?” ([7]: 1). CSRs can hardly represent the full meaning of the parsed sentence; especially if one has a clear understanding of the complexity hidden in the above usage of the adjective full (see Sect. 7). Nevertheless, CSRs offer a number of advantages and, even though they may at first sight appear to be of little use, due to their self-evident nature, they seem to cope well with the part Who did What to Whom which Palmer, Gildea, and Xue call to mind, as Table 1, in our opinion, demonstrates.

5 The Raison d’être of the Pairings As mentioned earlier, the logical notation introduced in Table 4 and used in Table 1 seems a truism. Below, some cases are provided to illustrate features and advantages related to such a representation of meaning. Let us first consider the pair of sentences in (4) and (5): (4) L’uomo venne puntuale ‘The man came on time’ (5) L’uomo venne arrestato ‘The man was arrested’

In sentence (4), the verb venne expresses movement. It licenses an argument and assigns to it the CSR >he who comeswhat is spentshe who laughsshe who laughs
what is heard
what is drawn
what is eaten
what is neededwhat is wantedshe who takes (something)
what is taken
he who brings about the crying Cost + Risk) [8]. 3.2 Challenge: Balancing Information Technology (IT) Operations Another challenge when transitioning to DevOps is balancing any continuous engineering and/or modernization efforts with the need to maintain continuity and integrity of business operations. In order to do this effectively while considering agile approaches, organizations implementing successful IT operations efforts will need to learn how to balance several competing factors, including: • Strategic (long term) versus tactical (short term) objectives. There is a need to balance ensuring operational performance and safety while enabling modernization of those operational systems.

The Impact of Agile Methods and “DevOps” on Day 2+ Operations

1073

• Operation’s needs versus organizational needs. It is important to not only optimize the flow of operational work, but do so within the context of the larger organizational context. • Standardization versus evolution. In order to reduce the overall cost and risk associated with operations, and to simultaneously make it easier for development teams to test and release changes into production, organizations should consider standardizing as much of their IT infrastructure as possible and managing its safe evolution over time. • Team-based DevOps versus organizational efficiency. The DevOps model works well with individual delivery teams, especially for smaller organizations. However, for organizations with many delivery teams working in parallel causes your costs and risks to quickly increase. These organizations quickly come to the realization that having a flexible operations-infrastructure team and platform in support of delivery teams can help to optimize the overall workflow across the enterprise. An Agile approach also means that typical operational concerns and potential barriers need to be considered in any delivery of capability and subsequent plan for sustainment. For example, the following should be considered: • • • • •

Misaligned expectations between operators and developers. Sacrificing data integrity for rapid application deployment. Lack of preparedness among the users and project teams. Lack of support and training from development teams. Declining productivity for operators if delivered capabilities lack usability.

To fully understand the IT operational needs of a large organization, it is important to consider how this is typically accomplished. Traditionally, operations organizations “execute the business” by adopting IT service management (ITSM) as a set of policies, processes, and procedures for managing the implementation, improvement and support of customer-oriented IT services [9]. In many organizations, Information Technology Infrastructure Library (ITIL) is a defacto framework for ITSM, providing processes and metrics for Service Strategy, Design, Transition, Operation and Continual Service Improvement [10]. As these organizations transform toward Agile and DevOps, it will be important to also transform the way ITSM gets accomplished. As illustrated in Fig. 3, we believe the next generation of IT Service Management will see a tighter integration with Agile methods [18]. ITIL in combination with scaled agile and DevOps has the potential, to lead to better results in digital operating models in several ways. For example, • DevOps could improve collaboration between IT operations and development teams. • Agile frameworks are helpful to facilitate enterprise adoption and are essential for an agile construct like a DevOps enterprise. • The combination of ITIL with other ITSM frameworks and agile methods, has the potential to lead to a dynamic and flexible IT service organization. • Communication between Dev, Ops, and the customer is essential in modern IT departments and can be fostered by agile concepts.

1074

R. Cherinka et al.

Fig. 3. Integrating ITSM, Agile and DevOps [18]

• DevOps combined with ITIL can help to prepare ITSM organizations for future IT challenges by softening borders between processes and roles.

4 Case Study For this presentation, we have selected a generic case study based on our experience working with a public sector, mission critical program to share some lessons learned and recommendations [11]. In supporting that program, we routinely work with the operational user community, the acquisition program offices, as well as the various development organizations that are part of the ecosystem. In addition, we observe their business and operational processes in action including their development pipelines. In Fig. 4, we observe a large mission-critical program that is attempting to modernize legacy capabilities in support of its core mission set. It illustrates this program as a large ecosystem of mission critical, distributed systems requiring 24/7 operations with a large mix of legacy and new software capabilities necessary to support those operations. It also relies on multiple developers operating in parallel to provide these necessary capabilities, including several core development organizations used to both maintain the legacy capability during the transition, as well as to modernize new offerings. However, there are also many other 3rd party developers that build and deploy solutions that the core teams do not provide. This process is very complex for many reasons, and transcends challenges associated with people, process and technology complexities. Based on this case study we have identified several key findings and lessons learned.

The Impact of Agile Methods and “DevOps” on Day 2+ Operations

1075

Fig. 4. Public sector example system of systems

Our first lesson learned is that Day 2+ DevOps needs to consider the effects of operating a range of applications that are creating a complex sustainment challenge. Each application varies widely in user base, complexity of capabilities and non-functional requirements, like performance, reliability, and security. This spectrum of software complexity and require different levels of software engineering rigor to sustain. Our second lesson learned is the need to identify longer-term application ownership. In industry, the typical software enterprise implements a software development and sustainment lifecycle. A critical role within that lifecycle is that of Application Owner, often referred to as the Product Owner or Service Manager. At the macro level of a large enterprise, there is no sole owner of the Applications being developed. This creates uncertainty for both development and sustainment. It is important to ensure there is a strategic owner for each capability being delivered. Our third lesson learned is that DevOps needs to address the lack of an evolutionary enterprise architecture and focus on addressing enterprise cross-cutting enablers. For example, this would include addressing topics such as shared data management, security, scalability, and end-to-end performance. A fourth lesson learned is the need for a Minimum Viable User (MVU) specification. In our case study, we observed the concept of users with limitations on the time-in-station, level of experience with the systems and supporting software products, and the ability to achieve sufficient expertise in such a strictly narrow window of time. The level of qualified learning and feedback on the quality of the product is limited to the quality of the users of that product. To better account for iterations of Minimum Viable Products (MVPs), and generations of products, we believe there is a need for an MVU specification for the user base. Our final lesson learned is related to the challenges with transitioning from traditional to agile acquisition across entire mission/business [12]. This now means organizations must cope with the challenges of such a transformation to include: • How to manage legacy capabilities as they sunset, while maintaining structural integrity of the mission.

1076

R. Cherinka et al.

• How to allocate resources to new areas and optimize current processes business and contracting models. • Changing mindset, structures and strategies for agile adoption. • Developing the skills/upskills of the workforce. These findings and lessons form the basis for the top-level recommendations we discuss in the following section. As of this writing, we are currently in progress working the execution of each of these recommendations with the operational organizations for the case study previously mentioned.

5 Enabling Operations to Take Advantage of Agile and DevOps In order to help operations organizations to prepare for and take advantage of a transition to agile and DevOps, it is important to discuss the need of determining when it is appropriate to be agile and select a suitable agile software development methodology. This is important considering that there are at least 23 distinct agile methodologies for which not every one of them is suitable for all types of software development efforts in the software zoo described in Sect. 2. Consideration should also be given to addressing some of the known challenges with “Agile” before attempting to ramp up to “continuous” development and deployment. Jeff Sutherland is the creator of the most widely adopted agile methodology known as Scrum and he highlights some of the known challenges in his article “A Tale of Two Scrums” [13]. Some of the common ones that we experienced in our case study are discussed below. One challenge is the availability of actual end users. All of the agile software development methodologies encourage (and some even require) the persistent engagement of expert end-users in the actual development process. The idea is that these expert end-users will help to guide the creation of requirements for the software being built, as well as to prioritize the features planned for each future release. In some industries, it is particularly challenging to get expert end-users of systems to engage directly with agile software development efforts. That is particularly true in the public sector environment, where end-users are often only in their roles for short deployments (e.g., 180-day rotations), which makes it challenging for them to become experts at their jobs before they are transitioned to new jobs in different roles. The challenge is to find “Minimally Viable Users” (with sufficient expertise and sufficient availability) to participate in the agile software development process. Another major challenge with agile software development is the presumption of “emergent architecture” and that the overall design of what they are building is going to iteratively emerge with each new release of software that is created. Consider the following: • Software Architecture is comprised of the decisions software engineers make that is expensive to change in the future. • Software Design is comprised of the decisions software engineers make that is (relatively) inexpensive to change.

The Impact of Agile Methods and “DevOps” on Day 2+ Operations

1077

• “Agile” purists argue against “Big Up Front Architecture” and believe that architecture and to some extent, design will “emerge over time”. • In large systems, such as our public sector systems, this “emergent” behavior does not always happen. In fact, these systems typically comprise multiple complex architectures. • To address the challenge of non-emergent architecture, it is paramount to adopt a practice of documenting architecture decisions as they are made, perhaps by using some form of architecture decision report. • To address the challenge of non-emergent design, it is important to engage Software development experts with ample professional experience in order to make agile development effective at fleshing out designs over time. Finally, every agile software development effort accumulates Technical Debt. Technical issues pile up in agile development because of the focus on meeting the time-box (i.e., sprint) deadline. In the world of “agile”, shipping software is always considered to be a higher priority than fixing all the known bugs. In order to succeed, organizations adopting and practicing agile software development and continuous integration should consider implementing code quality inspections and automated testing as an integral part of their continuous development process [14]. With these considerations in mind, and the findings and lessons learned from our case study experience, we can recommend several key areas where an operations organization can prepare to be successful in a DevOps environment. 5.1 Enhance Day2 + Operations and Continuous Sustainment The first recommendation we will discuss is focused on enhancing Day 2+ operations and continuous sustainment by ensuring that 3rd party developers of innovative or enhanced capabilities addressing important mission/business needs are held accountable for providing valuable and timely solutions. One recommended action is the creation of the Operations Architect. This involves establishing an OPs-focused mission, system and security architect with the authority, responsibility, and relevant experience required for establishing a holistic approach to modernization, which includes a coherent vision for a future state technical solution. An Operations Architect would help to define application operational acceptance criteria which would set high expectations with developers. These expectations would include considering impacts of enterprise integration as well as addressing sustainment needs, while holding them accountable for: • Using high-performance teams. • Developing better quality products. • Considering impacts of enterprise integration as well as addressing sustainment needs. Another action is to integrate DevOps into existing ITSM processes and systems for the target operational environments – ideally based on ITIL guiding principles and catalog of best practices. Specifically, the intent is to place more emphasis on the “Ops” aspect of DevOps via continual improvement of service delivery, management, and support

1078

R. Cherinka et al.

– with a primary focus on optimizing value (from an end-to-end mission perspective). Some examples of this integration include: • Establishing an Enterprise Application Dashboard (or Application Catalog). • Implementing an artifact repository of secured, vetted components, executables, and Infrastructure-as-Code scripts to foster re-use, and support enterprise architecture scalability, while incorporating the DevOps practice of Continuous Delivery. • Establishing issue owner responsibility “for the lifetime of an issue” associated with the program office or sustainment organization that will survive the turnover of personnel. • Capturing quality metrics for operations, including basic application telemetry data for new features and Key Performance Indicators (KPIs) to leverage maximum value from the available user base. A third action is establishing a Minimal Viable Team (MVT) that consists of application or product owners, developers and MVUs. The intent is that the team meets on a frequent basis to evaluate application usage, sustainment activities, new application sustainment acceptance, and feature requests. The team persists indefinitely while the members may rotate in/out based on deployment schedules. Given the team’s overlapping deployment schedules, knowledge transfer within the team would need to be preserved. Recommended objectives of an MVU/MVT would include: • Inventorying the breath of applications and assess need and future viability. • Retiring applications that are of little to no use. • Introducing and consistently having end users and developers review applications functions and features. • Requiring developers to hold high Defect Removal Efficiency (DRE) ratings. • Accelerating user orientation to increase the quality of feedback as early as possible. • Enriching feedback loops to include automatically collected telemetry. 5.2 Encourage Community Operations Collaboration A second recommendation is focused on establishing and executing an operations focused user group and/or community of practice. This can help to: • Establish a Baseline of Understanding of the system. • Define and maintain cross-cutting mission workflows and information objects to support those workflows. • Define the MVT, MVU and Product Owners to represent core capability areas that can represent cross-community business needs to 3rd party developers and determine user acceptance criteria and prioritization. • Engage the software development community to look at common code repositories, enterprise application dashboards, and the like.

The Impact of Agile Methods and “DevOps” on Day 2+ Operations

1079

5.3 Empower Operators as “Citizen Coders” A third lesson is to empower the operators as “Citizen Coders” by adopting low-code technology to accelerate digital processes as operators can now take on creation and management of their own applications where it makes sense [15]. We define “Low-Code” as a software development paradigm that centers on professional developers, streamlining and simplifying their work – delivering enterprise-class applications with little or development. The term “citizen coders” commonly refers to business users who can build functional but generally limited apps without having to write a line of code. The intent is to enable operators to become “citizen coders” and to become more effective with: • Evaluating current features, functions, and end users of applications and use low code methods to support rapid development of application functionality. • Transferring Knowledge between users, developers, and product owners by using low-code frameworks to document existing application decisions and functionality through models and workflows, hopefully easing training for the next rotation of operators. • Uplifting skills and knowledge of operators with modern software practices and the use of low-code platforms that enable rapid delivery of business applications with a minimum of hand-coding and minimal upfront investment in setup, training, and deployment. • Fostering relationships and collaboration between the developers, system/platform administrators, and end users using the model-based engineering that low-code technology could provide.

6 Conclusions In this paper, we have attempted share some lessons learned through our work with several public sector, large organizations [11, 16]. In particular, we discussed the “Ops” of DevOps as we discussed key findings and challenges operations organizations face in the era of continuous engineering. Throughout this paper, we have highlighted a number of key points, such as: • Agile, DevOps, and Continuous Engineering is changing the way software is developed and capabilities are delivered. • “Agile” challenges exist and need to be addressed proactively. Agile is not a panacea and it is important to determine when/where to apply it. • Large Organizations are in the middle of this transformation. • Fundamentally, there is a need for a more holistic approach to modernizing mission critical systems while balancing agility with engineering rigor. • The minimum viable user can provide the consistency and experience needed to validate true application needs in the context of tried and proven mission threads. • “Ops” needs to hold “Dev” accountable for delivering value and ensuring that value > cost + risk.

1080

R. Cherinka et al.

Given a transition to Agile and DevOps, we believe that the specific recommendations discussed can be used to empower operators in managing the effective transition and execution of capabilities into operations. For example: • The creation of an Operations Architect can help operators to represent the important mission context for operations to developers, focusing on key cross-cutting enablers required to support those operations. • Integrating DevOps into existing ITSM processes and systems for the target operational environments can facilitate the adoption and ongoing maintenance of delivered capabilities by the operators. • The establishment of MVUs with appropriate operational experience to adequately represent users to the development community can ensure Day 2+ operational concerns are considered by developers. • Establishing community-based operations collaboration across user groups can provide a forum for operators to discuss cross-cutting needs and provide consistent requirements and feedback to developers. • Empowering operators and end users as “Citizen Coders” can enable them to rapidly develop application features and as a way to document requirements and designs used in discussions with full development teams. In future work, we plan to define criteria for the minimum viable user concept discussed in this paper, and evaluate the value and impact in extended case study analysis. This work would also include evaluating the longer term impact agile software development may have on the operations and sustainment of a complex system of systems.

References 1. Andreessen, M.: Why software is eating the world. Wall Street J. (2011). https://www.wsj. com/articles/SB10001424053111903480904576512250915629460 2. McQuade, J.M., Murray, R.M.: DoD, software is never done: refactoring the acquisition code for competitive advantage v3.3. Defense Innovation Board, 12 March 2019 3. Defense Innovation Board Metrics for Software Development Version 0.9. DoD, 9 July 2018 4. Hamilton, J.: Software sustainment. Defense Acquisition University, 7 June 2017 5. Fowler, M.: Is design dead? (2004). https://www.martinfowler.com/articles/designDead.html 6. Kruchten, P.: The Elephants in the agile room. https://philippe.kruchten.com/2011/02/13/theelephants-in-the-agile-room/ 7. Lapham, M.A.: Software sustainment now and future. SEI, January/February 2014. CrossTalk. 8. Continuous deployment. Scaled Agile Inc. 24 August 2018. https://www.scaledagilefram ework.com/continuous-deployment/ 9. Next generation ITSM: making IT Service Management ready for the digital age. Deloitte Consulting Technical Report, March 2018 10. ITIL Wikipedia. https://en.wikipedia.org/wiki/ITIL 11. Cherinka, R., Foote, S., Prezzama, J.: Lessons learned in adopting agile software development at enterprise scale. In: Proceedings of the World Multiconference on Systematics, Cybernetics and Informatics (WMSCI), October 2020 12. Digital transformations challenges in large and complex organizations, Cffo, 21 March 2019

The Impact of Agile Methods and “DevOps” on Day 2+ Operations

1081

13. Sutherland, J.: A tale of two scrums: agile done right and agile done wrong. Defense Science Board, Design and Acquisition of Defense Systems (2018) 14. Vassallo, C., Palomba, F., Bachelli, A., Gall, H.C.: Continuous code quality: are we (really) doing that?. In: ASE 2018: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 790–795(2018). https://doi.org/10.1145/3238147.324 0729 15. Natis, Y.: Magic quadrant for low-code application platforms. Gartner IT Symposium Expo Report, October 2019 16. Brunelle, J.F., Frisk, D., Mayer, B., Randall, P., Sheikh, A.: Measuring the impact of innovation activities in government. Def. Acquisition Res. J. 27(4), 398–435 (2020) 17. Dallosta, P., Simcik, T.: Designing for supportability, driving reliability, availability, and maintainability. Defense AT&L product support issue March-April 2012. https://www.dau. edu/library/defense-atl/DATLFiles/Mar_Apr_2012/dallosta_simcick.pdf 18. Hinchcliffe, D.: Rethinking IT service management in the era of clud, customer experience, and deign thinking. Constelletion Res. Blog. (2017). https://www.constellationr.com/ blog-news/rethinking-it-service-management-era-cloud-customer-experience-and-designthinking

Mapping Risk Assessment Strategy for COVID-19 Mobile Apps’ Vulnerabilities Tanusree Sharma, Hunter A. Dyer, Roy H. Campbell, and Masooda Bashir(B) University of Illinois at Urbana-Champaign, Champaign, IL 61820, USA {tsharma6,hadyer2,rhc,mnb}@illinois.edu

Abstract. Recent innovations in mobile technologies are playing an important and vital role in combating the COVID-19 pandemic. While mobile apps’ functionality plays a crucial role in tackling the COVID-19 spread, it is also raising concerns about the associated privacy risks that users may face. Recent research studies have showed various technological measures on mobile applications that lack consideration of privacy risks in their data practices. For example, security vulnerabilities in COVID-19 apps can be exploited and therefore also pose privacy violations. In this paper, we focus on recent and newly developed COVID-19 apps and consider their threat landscape. Our objective was to identify security vulnerabilities that can lead to user-level privacy risks. We also formalize our approach by measuring the level of risk associated with assets and services that attackers may be targeting to capture during the exploitation. We utilized baseline risk assessment criteria within the scope of three specific security vulnerabilities that often exists in COVID-19 applications namely credential leaks, insecure communication, and HTTP request libraries. We present a proof of concept implementation for risk assessment of COVID-19 apps that can be utilized to evaluate privacy risk by the impact of assets and threat likelihood. Keywords: Mobile apps · Privacy risks · Threat likelihood · COVID-19

1 Introduction In the wave of the recent COVID-19 pandemic, mobile apps have been considered as an important technological medium in combatting the spread of the virus. While information sharing is a key component for the functionality of COVID-19 mobile apps, it is not clear if basic security and privacy protections are being put in place in these apps while we are rushing to deploy them around the world. For example, a recent research study found that the use of Google’s Firebase Analytics service by OpenTrace (which is a version of related close-source app TraceTogether) has potential scopes of privacy risks around (IP-based) location track as well as storage of user phone numbers in the Firebase authentication services [1]. Since one of Google’s primary business is advertising, this type of user data collection creates a potential conflict of interest [1]. Likewise, there exists inherited denial of service vulnerabilities in COVIDSafe version 1.0 and 1.1 from OpenTrace code which allows an attacker within Bluetooth range to crash the © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1082–1096, 2022. https://doi.org/10.1007/978-3-030-80119-9_72

Mapping Risk Assessment Strategy for COVID-19

1083

application, potentially can cause harm to user’s device [2]. Furthermore, smartphones’ heterogeneous nature of having various sensor data, communication logs, and mobility of user device throughout a day, involve connections to various networks that are often not secure. Therefore, sensitive data can be transferred from client applications across unsecured channels and lead user’s information to possible exploitation. In addition, there are studies that show that some android users are not using HTTPS and therefore making users vulnerable to man in the middle attacks [3] and other privacy risks. In this context, users’ personal information on such apps can make organizations economically attractive source to attackers [3]. Hence, traditional risks may reappear with increased impact through COVID-19 apps. Attackers can pose security challenges that includes widened attack vectors through location surveillance in COVID-19 applications. Such threats can make traditional countermeasures ineffective and take away users’ trust from technology [4, 25]. For COVID-19 applications, user trust in technology is one of the important factors to consider. In contact tracing apps, proximity tracing process is supported by a backend server that distributes anonymous exposure information where the server is trusted not to add fake exposure events and remove real exposure events [5]. In this case, the backend server is expected to act solely as a communication platform rather than processing and it is assumed to have users’ privacy intact even if the server is compromised. For the purpose of this study, we illustrate privacy risks through a scenario in which several security vulnerabilities could directly put the user’s data at risk within the context of a COVID-19 contact tracing application. Our aim for this illustration is to demonstrate how seemingly minor vulnerabilities can lead to very undesirable results. While these attacks are not novel, they help illustrate how existing mobile environments can facilitate some very basic attacks that can violate privacy protections. To begin this illustration, in Fig. 1, we examine exposed API keys and how an attacker could obtain the keys by manually inspect the decompiled code. After obtaining keys, network traffic could be observed for requests to be made, and to find the target within the API. This attack can then be taken in multiple directions, but we will assume that in the general case, the API has the ability to submit user information. Keys could then be used to submit a large number of new fraudulent user accounts. This could act as a denial of service attack if the volume of requests and methodology is correct. It could also open the door to further attacks on the infrastructure by attempting to simulate a large number of locations, such that if one of the fake accounts could be marked as positive for COVID-19, it could lead to many legitimate users believing they have encountered someone that tested positive for COVID-19, when in fact they did not. This attack largely leads to the availability and integrity of the application to be put under strain. There may also be capabilities of the API if it was not designed well in which user information could be extracted with only a key, leading to a large, but preventable data breach. Next, we examined insecure communication. There are many well-known issues with HTTP traffic. For example, it can be read by anyone that observed the request in transit. The use of HTTP enables a person-in-the middle scenario in which the attacker would be able to extract all information coming into the server, because HTTP isn’t encrypted. This allows a large amount of sensitive data to be leaked. Finally, the nature of HTTP libraries doesn’t inherently make them an attack vector, but having multiple HTTP

1084

T. Sharma et al.

libraries increases the attack surface by giving more places in which a vulnerability could be found and exploited. So, if our illustrative application had multiple HTTP libraries within it, then as an attacker, there are more libraries to look for possible exploitation or possibly use an already known exploit if it were not patched within the application. As an example, there might be some exploit within one library that allows for a person-inthe-middle attack that could intercept and decrypt HTTPS traffic, which would allow a large leak of personal information that was thought to be secure.

Fig. 1. Possible attack paths when security smells are present.

In COVID-19 application scenario, these exploitations and leakage of a user’s information can also help build a social graph of people with which they had contact since in the system both identifiers and COVID-exposure are computed centrally and therefore, the backend server can associate uploaded ephemeral broadcast identifiers to permanent pseudo-identifiers for individual devices. Thus, the backend server can not only reconstruct a social graph, but it can reconstruct an interaction graph. COVID-19 apps and associated security threats can potentially create privacy risk. If data is manipulated or stolen due to insecure communication, then this technology will not be trustworthy to users which is a risk of authentication at user level. At the same time, if these apps are not working properly and not performing correct exposure notification, there can raise the issue of integrity of this application while credential leaks lead to a potential risk of confidentiality. Keeping previous and current security exploitation in mind and associated privacy risks, our study approach is to assess COVID-19 apps from different security parameters to identify security vulnerabilities in credential leaks, HTTP request libraries, and insecure communications. For this study, our motivation is to map these chosen security exploitations and develop assessment strategies for associated privacy risk, as well as provide suggestions for how these vulnerabilities can be addressed in the development process. We demonstrated step by step how security vulnerabilities can be assessed and measured so that they can be quantified as risk factors. Then, we provided a risk assessment methodology that can be utilized when assessing privacy risk in mobile apps. While there is still lack of formal risk assessment methods for determining risks at an individual level,

Mapping Risk Assessment Strategy for COVID-19

1085

we utilized a targeted risk assessment method to assess user specific assets and threats that is focused on COVID-19 applications. We believe our study contributes towards the development of equations which considers users’ assets and threats associated with the COVID-19 apps, as well as the likelihood of risk. Therefore, our research questions are: RQ1: What are the possible privacy risks and security vulnerabilities that are specific to COVID-19 applications? RQ2: How can we measure or quantify risk based on prior knowledge/likelihood of threats that considers recent privacy risk associated with those applications? The remainder of this paper is organized as follows. Section 2 provides relevant background information. Section 3 describes our research methodology. We present our app analysis results and possible resolutions in Sect. 4. Section 5 presents App’s Risk Assessment Strategy and then we conclude the paper with Sects. 6 and 7 which provides a discussion of the implications, limitation, and future research directions as well as our conclusions.

2 Background Several previous studies have shown the prevalence of smartphone privacy concerns and security threats around android permissions that can take place via privilege escalation attack that allows a malicious application to gain more capabilities leaked from a benign application [6], web API where message passing is also a medium of attack surface while providing availing inter-application collaboration. Adversarial can sniff, modify, steal or replace content with compromising user privacy [7, 19]. Again, malicious application can inject malicious messages/code which can lead to users’ data breaches. Furthermore, there are security concerns around ready-to-use code without proper caution such as [8], poor authentication and authorization, and lack of proper encryption of sensitive data [9] which keeps exposing users’ data to malicious actors. These attack scenarios are not novel to COVID-19 apps. The urgency in testing basic security and privacy vulnerabilities within these apps are ever more vital because all types of organizations and governments around the world are rapidly working on developing and deploying contact tracing apps to track and mitigate the spread of COVID-19. While the developments of these apps are extremely important amid the COVID-19 pandemic, security and privacy researchers must address these basic anticipated vulnerabilities and the risks they may pose to users’ personal information. One such research study analyzed Singapore’s OpenTrace app and its use of Google firebase services to manage user data and deployment of reversible encryption and found that such method can be vulnerable to secret key disclosure [1]. Other researchers are proposing different security and privacy aware frameworks that are better at anticipating associated risk and adversarial attacks. Some of the proposed or implemented designs include decentralized proximity tracing [5]; construction of token in Bluetooth contact tracing [10]; distributed hash table to build a decentralized messaging system for infected users [11]; minimizing re identification risk strategy [12], user-centric privacy design [24] to ensure privacy is well preserved with formal security model mechanisms.

1086

T. Sharma et al.

Generally, organizations and IT sectors implement security standards and controls, as well as risk assessments, to decrease the likelihood of risk for their assets and services from the evolving landscape of exploitation [11]. Qualitative and quantitative research has been done regarding decision making on security measures implementation based on the estimation of attack paths and security costs by attack graphs, tress, and other models [2, 13]. Our approach of assessing various assets and their associated risks as it pertains to COVID-19 apps where we consider users’ personal information as an asset is valuable and unique. Previous research has identified different types of assets like personal data, financial assets, and personal and political reputation based on the privacy risks [14, 23]. With these prior studies as our motivation, we have designed our experiment considering three security vulnerabilities in COVID-19 apps to assess risks and articulate assessment strategies for investigated vulnerabilities based on user level asset and threat likelihood.

3 Method Following previous research papers’ approach in examining privacy violations, we focused our study on specific threats, namely sensitive data exposure, broken authentication, and security misconfiguration. While there are many other security threats that can be evaluated, we believe these three make up the most common security issues and it is supported by previous studies [2, 17, 20]. Therefore, in terms of COVID-19 apps, we focus on Credential leaks as one of the security exploitations since it involves sensitive data including financial, healthcare, and PII to be compromised if there are not enough security measures, such as encryption for the different keys and data storage. Next, we considered insecure communication and HTTP request libraries which can lead to insecure default configurations, incomplete or ad-hoc configurations, misconfigured HTTP headers, and verbose error messages containing sensitive information. Therefore, we believe these three specific security vulnerabilities selected in our assessment of COVID-19 apps constitutes the most common vulnerabilities [2, 17, 21] that can be encountered in those apps. Thus, guarding against these vulnerabilities can ensure timely and secure configuration of COVID-19 apps. App Selection. We selected 7 apps from the recent CVE report on apps having frequent exploitation report [2]. Initially, we assessed apps by automated frameworks for understanding fundamental security requirements. This step is not necessarily a part of this study, but rather was used as our preliminary step towards our initial understanding of those apps. Our goal is to examine security vulnerabilities in terms of privacy violation. Selecting Security Parameters to Check. After having our initial inspection, we emphasize on credential leaks; HTTP Request Libraries and insecure communication based on security exploitations [15, 16] that have been identified by previous studies. In addition, our review of the national security vulnerabilities database [2] suggest that when evaluating security vulnerabilities, we are to place an increased priority on patching the most commonly known vulnerabilities and controls to avoid risks. The purpose of this database is to provide security professionals with an update on the most common and exploited security issues so that they can prioritize patching and place security accordingly. As outlined below, we evaluated each selected COVID-19 app for the presence of

Mapping Risk Assessment Strategy for COVID-19

1087

the three specific vulnerabilities. Next, we recorded how frequently these vulnerabilities occurred in our selected apps. In addition, we recorded potential user privacy violations that might take place from such vulnerabilities. More specifically, we performed the following steps: • We selected 7 COVID-19 apps as described above to examine. • We examined each of the 7 apps for the presences of three types of security vulnerabilities (credential leaks, insecure communication and HTTP request libraries). • If the app had any of the 3 security vulnerabilities, then we analyzed/recorded how frequently those vulnerabilities occurred. For example, an app can have 3 credential leaks such as API key, password, and username leaks. • We also investigated how these 3 security vulnerabilities can lead to privacy violations. Mapping Risk Assessment Strategy for COVID-19 Apps. After analyzing each of the 7 apps for security vulnerabilities as described above, we mapped associated privacy risks for users in respect to assets and services provided by those apps. To conduct this risk assessment, we measured the severity of privacy/security risk based on previous literature [18]. It is important to note that our measurement was not entirely based on performing quantitative evaluations but rather building and proposing an exploratory strategy to assess risk that can be tested/verified in future studies. More specifically, the following steps were performed for this mapping. • Our risk strategy considered 3 security vulnerabilities and a list of associated risks that can impact user’s and industries’ assets. Assets include personal information, health information, location information, financial information that can impact availability, confidentiality, and integrity in any security infrastructure. For this study, we assessed impact on assets that would be compromised by these selected 3 security vulnerabilities. • Then we considered the likelihood of privacy risks associated with those assets listed above. We adopted the Likelihood metrics from previous literature that included measures of a) historical risk data, (b) statistics of the known vulnerabilities, and (c) existing controls to reduce vulnerabilities. For example, in our study, credential leaks are considered a vulnerability. In this case, threat likelihood is calculated on the basis of, a) the likelihood of credential leaks in the COVID-19 smartphone platform, b) the threat incident likelihood from previous incidents, i.e. statistics on threat incidents in the platform or previous incidents experienced by the user, and c) the relevant security control existence/absence. • Therefore, we propose to calculate risk by the combination of impact on user privacy and likelihood of vulnerabilities. We followed this method in our risk calculation that has been proposed in previous risk assessment studies [13, 14].

4 Result and Possible Solutions In this section, we reported our conducted analysis specifically for three security exploits for 7 apps. Three focus areas for security measurements are credential leaks, insecure communication, and HTTP request libraries.

1088

T. Sharma et al.

4.1 Credential Leaks The development of mobile applications often hinges upon using web applications and services through the application’s programming interface, also known as an API, in order to serve dynamic content, collect data, or perform other complex operations that may not be otherwise appropriate or possible on users’ devices through HTTP or HTTPS requests. Most APIs use keys for authentication within the headers of the requests. The consequences of keys being wrongfully obtained might mean private information is leaked, or financial repercussions based on billing agreements for the API which also fits under a broader topic of credential leaks where credentials are revealed to unauthorized entities through unintended channels. This could include emails, passwords, or tokens, among other sensitive information that might be used for authentication purposes. The primary focus of this assessment was to find types of credentials like API keys. One email was found but it seemed to be a remnant of code that was meant for testing an upload of data. Within the apps examined, there were multiple instances in which API keys were hardcoded and easily accessible by decompiling the application, as was done within the study. This practice is often considered insecure partially due to the possibility of wrongfully obtaining keys. Inspection of these applications was done manually; only the code that would have been written by the developers was inspected. This not only made the manual inspection more feasible, but also allowed us to assess the extent of the credential leaks that the developers would have been directly responsible for. Due to the variance in length, possible characters, and forms that API keys may take, we searched for terms that would possibly appear near these credentials (terms like key, token, etc.) and inspected the files in which matches showed up. Results were counted if they were directly labeled as a key or token, which may not fully account for every single key hardcoded based on naming scheme or associated labels. Notably, the parameters for which we counted keys or tokens omitted pieces of data that may be labeled as “ids”. There are some cases in which a value labeled ID would be equivalent to a key, and those scenarios would be excluded from this analysis. Most of the API keys found in each case were found in the resources /res/values/string.xml file after decompiling as can be seen in an excerpt from CoronaSUS in Fig. 2. These keys were mostly related to Google libraries and services.

Fig. 2. Example Snippet of Credential Leak within CoronaSUS

Results in Table 1 denoted with a* were situations in which two of the leaked credentials were related to Google libraries, and had security features available to mitigate or

Mapping Risk Assessment Strategy for COVID-19

1089

prevent potential unauthorized use. These two credentials were uniformly named across the apps that they occurred in and were labeled Google api key and Google crash reporting api key. If we omit these findings, then we are left with three applications that had . credentials leaked, Aarogya Setu, COVIDSafe, and Table 1. Leaked credentials total breakdown by application Security smell: 1 Application

Instances of leaked credentials

Aarogya Setu

3*

ABTraceTogether 0 CoronaSUS

2*

COVIDSafe

1

Protego

2*

TraceTogether

2* 3*

Privacy Risks. With consideration of these occurrences not being disclosed to developers, discussion of further leaked credentials is discussed in aggregate. 4 keys found could be classified broadly as an application API key and 2 keys related to 3rd party services that handle application engagement and location management. Notably, the key that directly relates to the API could have direct consequences to the privacy of other users if data can be exfiltrated from the server through different endpoints. Inversely, it could also act as a way to deny service to users if an excess of requests is made. These two consequences are on opposite sides of the spectrum of things that can be done with keys in related circumstances. Oftentimes attacks are dependent implementations of the server, along with the possible utilities that the API provides. These attacks are also dependent on proper permissions and control of the API key not being implemented. Further, with the location service, there may be a risk that a malicious actor is able to gather user locations in the event where API is not designed well, or could possibly flood the API with excess data, which would be problematic for offering intended service of the mobile application it is used in. API implementations that are unsecured and don’t have proper access control implementations in place, such as a restriction on requesting IP addresses can be subject to serious attacks if the proper care is taken with the key that is used, especially in a time in which decompiling an app is a readily available service with relatively low barrier to use. Proposed Resolution. In terms of solving the specific problem of leaving API keys hardcoded within applications is to make it more difficult to obtain such keys or key could not be obtained solely through decompiling the application. This can mean that if an adversary wanted to obtain the key, they might have to use methodologies for examining the memory of the application or other debugging tools. Alternatively, employing

1090

T. Sharma et al.

methodologies similar to what was seen with Google libraries in which key owners have the ability to strictly limit when and where the keys can be used, but this may open up other possible attacks.

4.2 Insecure Communication With many mobile applications relying on web services in order to provide services, it is very important for the communication channels in which this information is conveyed to be secure. The predominant method for this communication is HTTP and HTTPS. HTTP is transmitted through plaintext methods, whereas HTTPS is transmitted through encrypted methods. The consequence of not using HTTPS is that it severely endangers the privacy for the user of the application of data being stolen or manipulated. Within our analysis of URLs that were hardcoded into the applications, we only examined domains that would have been directly coded by developers of the app. URLs within 3rd party libraries or URLs included as part of the Android application build process were omitted. Notably, this excludes URLs from the schemas.android.com subdomain, as this is used to host base resources for most Android applications. These include domains such as apache.org. URLS that were hard-coded but did not contain the prefix of http or https were counted under undetermined as it would likely be determinant on the HTTP library used to make the request or how the string is later manipulated in code. The counts below are the number of unique URLs to avoid double counting if a URL is hardcoded in more than one place [22]. In Table 2, our inspection result has been recorded. Table 2. Breakdown of URLs found through manual inspection Application

Security smell: 2 #HTTP

Aarogya Setu

#HTTPS Undetermined

0

5

2

ABTraceTogether 0

5

0

CoronaSUS

1

5

2

COVIDSafe

0

5

1

Protego

0

7

2

TraceTogether

0

5

2

0

78

2

Since the hardcoded URLs were inspected manually, it wasn’t feasible at this time to find the exact purpose and behavior of each link through the code. However, insight to the URL’s purpose can be found from its name and general form. The http: http://mob ileapps.saude.gov.br/coronavirus and following the link in a browser only returns text in JSON format, which might indicate this is an API URL, though it is not clear what

Mapping Risk Assessment Strategy for COVID-19

1091

information might be served with requests to this URL, but if it is sensitive information, then it is a threat to user privacy as discussed earlier. Within links that were HTTPS, they could be grouped into two main descriptions. They were either linked to external websites or resources, such as FAQs, privacy agreements, or EULAs, or the URLs were to have had 78 used for API requests. The former is the driving reason for hardcoded HTTPS links. Collectively, there was one definite case in which this code smell was present, while the undetermined URLS might be potential cases. Proposed Resolution. Unfortunately, there aren’t as many flexible fixes to this problem as there are with the API keys. One possible solution is to ensure that HTTP Request libraries used in your application (also discussed in the next section) are capable of making HTTPS requests, and that this functionality is used. The use of HTTPS is all dependent on the server being communicated with having the capability as well. So, in the event that developers are using an API they built and are hosting, they should take the time to ensure that they have properly set up HTTPS capabilities on their server to attempt to protect the inbound and outbound communications.

4.3 HTTP Request Libraries It is common practice to use 3rd party libraries within software development to facilitate and simplify the development cycle. This can certainly prove to be efficient and effective for completing development, it can introduce other concerns in regard to potential credit leaks and cohesiveness of the application. While the implications of this security vulnerability may not be as significant as others, it is still worth considering in its relation to the other two security vulnerabilities. The use of multiple HTTP libraries in each application can also increase the potential attack surface of the app, as it presents opportunities for more vulnerabilities to arise. This can pose a security threat to the application’s infrastructure while also posing a threat to the users of the application. It is worth noting that the consequences of an attack on an HTTP library would be potentially reliant upon which URLs the HTTP library serves. The libraries used were determined by manual inspection of the 3rd party libraries that were included within each application. If the library was self-described within official documentation or webpages as a library for making generic HTTP or API related requests, then it was included within the survey of libraries below. 3rd party services that offered a library specifically for their own owned service were not included as these libraries are often specialized and provide other services. Libraries were surveyed based on the inclusion of library folders in the decompiled application. In some cases, the libraries could not be fully assessed due to code obfuscation. have some obfuscation and files were Two applications, TraceTogether and not named in conventional naming conventions and had names of the form similar to C0000a.java, where the number could be a four-digit number and the last letter could be any letter of the alphabet. Some of the consequences of our assessment could be due to the compiler that we used. Results were gathered from the folders whose naming structure remained intact. The distribution of HTTP libraries is dominated by OKHTTP3 and has some other inclusions. As discussed earlier, there is some risk with using multiple

1092

T. Sharma et al.

libraries, and it seems that this selection of apps abides by this. The use of libraries that are written for a specific product in mind can be advantageous in that it should ideally take the burden of security considerations from the developers and places them onto the companies that have a better working knowledge of their product, and assumedly have more incentive, time, and resources to devote to such considerations compared to the developers using their library. Using HTTP libraries that are meant to flexibly handle different HTTP requests can place this burden back on the developers as it may not directly promote good practices for the purposes of API requests or other services that require key authentication.

5 App’s Privacy Risk Assessment Strategy In this study of COVID-19 app assessment, we have demonstrated associated risks for users’ privacy. In this section, we specify our proposed risk assessment strategy for COVID-19 smartphone apps that is described in Sect. 4. Our risk assessment strategy considers associated risks that can impact users’/industries’ assets and the likelihood of those risks. We have found that our selected security vulnerabilities can lead to personal information leaks, data sharing/storage risks, and insecure communication between server and client application. Therefore, in our risk assessment strategy, we considered credential leaks, HTTP libraries, and insecure communication as attack vectors to do our analysis. The frequency of those attacks and the associated impact on users’ privacy due to those vulnerabilities while using COVID-19 apps are shown in the results section. While our risk assessment strategy is somewhat exploratory, we believe that our proposed approach for evaluating COVID-19 apps is robust, as it not only considers the current privacy impact, but also includes statistics from past exploitations. To assess COVID-19 smartphone apps’ privacy risk, our first criteria was to evaluate its assets. Assets identified were related to COVID-19 apps’ components, for example, personal information, health information, location information, financial information. Based on that, impact of such insecurity scenarios (loss of availability, confidentiality and integrity) can be assessed. For instance, the impact of data type is inferred to their associated data sources. This means that if there are ‘personal’ data types that those apps are dealing with, then the disclosure impact for the data source are: “exposure notification”, “data Processing” and “Data storage/sharing” can be calculated, as follows: Impact(Data Sharing) = max{Impact(personal data), Impact(health data)} I=

n a=0

avg(IPd , Ihd )

Where each vulnerability here, such as, credential leaks, insecure communication and HTTP request library will be present and considered as a single vulnerability vector: Vi = {0, 1} for all i , i = 1, 2, 3, 4, . . . , n So, the overall COVID-19 smartphone apps’ data sharing impact is the max impact related to personal data and health data. Some other attacks can happen to other types of assets, i.e., the device, data, applications, and connectivity.

Mapping Risk Assessment Strategy for COVID-19

1093

The other variable that can be measured is threat likelihood for risk assessment that is assessed on the basis of: (a) experience and applicable statistics, (b) vulnerabilities, and (c) existing controls to reduce vulnerabilities. Each threat, for example, credential leaks is grouped in the appropriate attack vector dimension. Asset refers to the topics targeted by the threat. For example, Credential leaks are considered as the threat. In this case, threat likelihood is valuated on the basis of a) the likelihood of credential leaks in the COVID19 smartphone platform, b) the threat incident likelihood from previous incidents, i.e. statistics on threat incidents in the platform or previous incidents experienced by the user, and c) the relevant security control existence (for mobile apps data management and security). Incident Information from the organization’s historical database recorded in system log files can be used in modeling which can predict threats likelihood. In this case, existing model, such as, attack graph can be used for new incoming threats. Tj = {0, 1} for all j, j = 1, 2, 3, 4, . . . , m Where each threat here, like users’ information leaks, data stolen/manipulations are considered as threat vector. Where Tj represents an individual threat. Value 1 indicates the presence of this threat in the information systems and otherwise 0. So, within vulnerability and threat  to individual level, likelihood of incoming threats can be measured by Lji = Tj , Vi where threat acting over a vulnerability [14]. Combining the impact assessment of assets/services and threat likelihood can be used to derive the risk for a particular security vulnerability in COVID-19 applications. Therefore, we can assess risk based on the current attack landscape and previous attack history assuming there exists controls against those threats Ck = {0, 1} for all k, k = 1, 2, 3, . . . , p. Risk =

n m  

I .Lji

j=1 i=1

6 Discussion The advancement of ICTs and its critical role in society in recent times requires us to develop timely and robust test methods, reference data, proof of concept implementations, and technical analyses. The use of COVID-19 apps throughout the world as one mechanism to combat the pandemic demonstrates the urgent need for better data privacy/security management that accounts for not only known privacy risks but also for unintended privacy risk for both users and organizations. While COVID-19 apps provide a timely vehicle for tackling the spread of the virus, it also provides a unique and global scenario for users’ loss of privacy. Mobile platforms continue to be one of the main sources of personal information exchange and storage and therefore more vulnerable to un/expected attacks that can violate users’ privacy [15]. To be able to protect against these risks, a risk assessment strategy is crucial. In our study, we propose a risk assessment strategy for COVID-19 apps based on our COVID-19 app assessment. Our assessment strategy includes the impact on users’ privacy and the likelihood of those specific privacy vulnerabilities.

1094

T. Sharma et al.

While the proposed assessment strategy is exploratory, we believe that our strategy is the initial and essential step towards building a more comprehensive privacy/security assessment framework for mobile applications. We hope to improve this strategy in subsequent iterations by assessing more mobile apps and validating our approach in order to develop better methods for evaluating and managing privacy risks. Additionally, the complexities and innovative features of the mobile platforms will continue to introduce privacy and security risks that will need to be assessed quickly, cost-effectively, and easily, in order to reduce cyber risks. Therefore, including risk assessments as one part of the system evaluation process can be an effective approach that can facilitate decision making at all risk assessment hierarchy including organizational levels, mission/business process level, and information system level [5]. Furthermore, this type of risk assessment can be made part of the system development life cycle that involves pre-system acquisition and post system acquisition. Perhaps our proposed strategy can be utilized by software developers in their Software Development Life Cycle (SDLC) to make decision on the proper privacy requirements based on the impact and likelihood score of the vulnerabilities. As mentioned above, risk assessment strategy is an iterative development processes and it changes overtime with new threat landscape. Our proposed model provides the flexibility to add new security attack vectors and the associated privacy risk by the impact and likelihood variable. These two risk assessment variables consider both system risk and users’ risk. Further research is needed to verify this initial proposed strategy and to validate our proof of concept framework. While there are various studies related to COVID-19 apps’ privacy and security, our study provides a new approach towards assessment of risk which is not present in the current literature of COVID-19 apps. Our proof of concept approach can be used on Covid19 mobile applications to assess privacy risks that maybe easily overlooked. While we believe our approach is a critical step forward, it is important to note that our security assessment considered only 3 main types of security vulnerabilities which does pose a limitation. However, in future studies additional vulnerabilities can be added to the list to broaden the scope and the privacy risks. Another limitation of this study is that our initial analysis considered only six COVID-19 mobile apps and therefore in order to confirm these findings and extend the scope of our findings additional Covid-19 apps needs to be considered and analyzed. For our future research, we will be examining a larger and diverse set of COVID19 apps for security vulnerabilities to conduct our experiment. In addition, we will be utilizing our risk strategy on a publicly available risk incident data set in order to expand and focus our strategy.

7 Conclusion Protecting user’s privacy and security against adversaries and unauthorized access continues to be a challenging task in the age of technological innovation. Mobile devices and applications make this task even more difficult due to its nature of constant information sharing that includes a wide range of user’s personal information. In our paper, we demonstrate specific security vulnerabilities relate to COVID-19 apps and provide

Mapping Risk Assessment Strategy for COVID-19

1095

proof of concept strategy that can be utilized for privacy and security risk assessment. We believe that our proposed risk assessment strategy can be expanded and developed further into a framework that can provide an efficient and novel approach for assessing security and privacy risks in mobile application. In our future work, we plan to develop this risk assessment framework further by including more training and testing data from public database.

References 1. Leith, D.J., Farrell, S.: Coronavirus contact tracing app privacy: what data is shared by the Singapore OpenTrace App? In: Park, N., Sun, K., Foresti, S., Butler, K., Saxena, N. (eds.) SecureComm 2020. LNICSSITE, vol. 335, pp. 80–96. Springer, Cham (2020). https://doi. org/10.1007/978-3-030-63086-7_6 2. NIST. National vulnerability database, automating vulnerability management, security measurement and compliance checking. http://nvd.nist.gov/home.cfm 3. Wei, X., Wolf, M.: A survey on HTTPS implementation by Android apps: issues and countermeasures. Appl. Comput. Inform. 13(2), 101–117 (2017) 4. Sharma, T., Bashir, M.: Use of apps in the COVID-19 response and the loss of privacy protection. Nat. Med. 26(8), 1165–1167 (2020) 5. Troncoso, C., et al.: Decentralized privacy-preserving proximity tracing. arXiv preprint arXiv: 2005.12273 (2020) 6. Davi, L., Dmitrienko, A., Sadeghi, A.-R., Winandy, M.: Privilege escalation attacks on android. In: Burmester, M., Tsudik, G., Magliveras, S., Ili´c, I. (eds.) ISC 2010. LNCS, vol. 6531, pp. 346–360. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-181788_30 7. Chin, E., Felt, A.P., Greenwood, K., Wagner, D.: Analyzing interapplication communication in Android. In: Proceedings of the 9th International Conference on Mobile Systems, Applications, and Services, pp. 239–252, June 2011 8. Fischer, F., et al.: Stack overflow considered harmful? The impact of copy&paste on android application security. In: 2017 IEEE Symposium on Security and Privacy (SP), p. 121136. IEEE, May 2017 9. Krishnan, M.: Survey on security risks in Android OS and an introduction to Samsung KNOX. Int. J. Comput. Sci. Inf. Technol. 6(4), 3965–3967 (2015) 10. Cho, H., Ippolito, D., Yu, Y.W.: Contact tracing mobile apps for COVID19: privacy considerations and related trade-offs. arXiv preprint arXiv:2003.11511 (2020) 11. Viduto, V., Maple, C., Huang, W., LóPez-PeréZ, D.: A novel risk assessment and optimisation model for a multi-objective network security countermeasure selection problem. Decis. Support Syst. 53(3), 599–610 (2012) 12. Sharma, T., Bambenek, J.C., Bashir, M.: Preserving privacy in cyberphysical-social systems: an anonymity and access control approach (2020) 13. Wang, L., Noel, S., Jajodia, S.: Minimum-cost network hardening using attack graphs. Comput. Commun. 29(18), 3812–3824 (2006) 14. Theoharidou, M., Mylonas, A., Gritzalis, D.: A risk assessment method for smartphones. In: Gritzalis, D., Furnell, S., Theoharidou, M. (eds.) SEC 2012. IAICT, vol. 376, pp. 443–456. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30436-1_36 15. Stevens, R., Gibler, C., Crussell, J., Erickson, J., Chen, H.: Investigating user privacy in android ad libraries. In: Workshop on Mobile Security Technologies (MoST), vol. 10, May 2012. Citeseer

1096

T. Sharma et al.

16. Zhou, Y., Wu, L., Wang, Z., Jiang, X.: Harvesting developer credentials in android apps. In: Proceedings of the 8th ACM Conference on Security & Privacy in Wireless and Mobile Networks, pp. 1–12, June 2015 17. Wichers, D.: Owasp top-10 2013. OWASP Foundation February 2013 18. Hiller, J.S., Russell, R.S.: Privacy in crises: the NIST privacy framework. J. Contingencies Crisis Manag. 25(1), 31–38 (2017) 19. Sharma, T., Bashir, M.: Privacy apps for smartphones: An assessment of users’ preferences and limitations. In: Moallem, A. (ed.) HCII 2020. LNCS, vol. 12210, pp. 533–546. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50309-3_35 20. Sharma, T., Bashir, M.:. Are PETs (Privacy Enhancing Technologies) giving protection for smartphones?--A case study. arXiv preprint arXiv:2007.04444 (2020) 21. Sun, R., Wang, W., Xue, M., Tyson, G., Camptepe, S., Ranasinghe, D.: Vetting security and privacy of global COVID-19 contact tracing applications. arXiv preprint arXiv:2006.10933 (2020) 22. Saha, T.K.., Sharma, T.: A dynamic data placement policy for heterogeneous hadoop cluster (2017) 23. Lewis, T.L., Wyatt, J.C.: mHealth and mobile medical apps: a framework to assess risk and promote safer use. J. Med. Internet Res. 16(9), e210 (2014) 24. Sharma, T., Wang, T., Bashir, M.: Advocating for users’ privacy protections: a case study of COVID-19 apps (2020) 25. Sharma, T., Bashir, M.: An analysis of phishing emails and how the human vulnerabilities are exploited. In: Corradini, I., Nardelli, E., Ahram, T. (eds.) AHFE 2020. AISC, vol. 1219, pp. 49–55. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52581-1_7

Explainable Graph-Based Search for Lessons-Learned Documents in the Semiconductor Industry Hasan Abu-Rasheed1(B) , Christian Weber1 , Johannes Zenkert1 , Roland Krumm2 , and Madjid Fathi1 1 University of Siegen, Siegen, Germany

[email protected] 2 Elmos Semiconductor AG, Dortmund, Germany

Abstract. Industrial processes produce a considerable volume of data and thus information. Whether it is structured sensory data or semi- to unstructured textual data, the knowledge that can be derived from it is critical to the sustainable development of the industrial process. A key challenge of this sustainability is the intelligent management of the generated data, as well as the knowledge extracted from it, in order to utilize this knowledge for improving future procedures. This challenge is a result of the tailored documentation methods and domain-specific requirements, which include the need for quick visibility of the documented knowledge. In this paper, we utilize the expert knowledge documented in chip-design failure reports in supporting user access to information that is relevant to a current chip design. Unstructured, free, textual data in previous failure documentations provides a valuable source of lessons-learned, which expert design-engineers have experienced, solved and documented. To achieve a sustainable utilization of knowledge within the company, not only the inherent knowledge has to be mined from unstructured textual data, but also the relations between the lessons-learned, uncovering potentially unknown links. In this research, a knowledge graph is constructed, in order to represent and use the interconnections between reported design failures. A search engine is developed and applied onto the graph to answer queries. In contrast to mere keyword-based searching, the searchability of the knowledge graph offers enhanced search results beyond direct matches and acts as a mean for generating explainable results and result recommendations. Results are provided to the design engineer through an interactive search interface, in which, the feedback from the user is used to further optimize relations for future iterations of the knowledge graph. Keywords: Knowledge graphs · Graph-based search · Text mining · Explainable search results · Lessons learned

1 Introduction and Research Purpose Sufficient utilization of well-documented lessons-learned for an application in the domain of semiconductor design is a non-trivial task. With a multitude of design cases, © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1097–1106, 2022. https://doi.org/10.1007/978-3-030-80119-9_73

1098

H. Abu-Rasheed et al.

their relevant failures and corresponding solutions, user’s access to relevant past experiences presents a challenge in the industrial environment [1]. Traditional information retrieval, e.g. keyword-based search, is not sufficient to discover and define similarities in semi- and unstructured textual data in chip-design documentations [2]. Moreover, processing free textual input for search queries itself reveals multiple challenges, such as the multilingual nature of documentation, the domain-specific terminology, abbreviations, misspellings, grammatical mistakes and incomplete sentences. Defining the relevancy of a current design case to other failures, which occurred in the past, requires a semantic understanding of the domain-specific description of those failures. Natural language processing (NLP) and text mining pipelines offer the first step in this understanding. However, their strength in handling textual inputs is still limited when multiple representations of information are extracted from different data sources. This can happen when integrating basic textual documentation with structural metadata representations of the chip modules and their corresponding elements. To address this challenge, a semantic definition of interlinked networks of available knowledge is needed to represent the relations between entities, which are extracted from multisource, multitype data [3, 4]. This fusion and semantic definition of knowledge entities, extracted from the lessons-learned information, is accomplished in this research through the construction of a multidimensional knowledge graph. In the graph, not only failure cases are represented in form of lessons-learned, but also their structural information and semantic relations. A graph search functionality is then designed to boost the exploitation of previous lessons-learned for search queries and thus establish an easy access to their documentation. This is then used to provide previous expert-knowledge to new design engineers. Such search functionality is offered through an interface that supports a feedback loop for assessing retrieved results and enabling a sustainable learning and knowledge update. However, providing the feedback is also strongly influenced by user’s understanding of the reasons behind retrieving a certain search result from the knowledge graph. Therefore, this research designs a graph-based explainability approach, to offer the user explanations on the reasons that led the system to consider a certain search result relevant to their search query and design scenario. Explanations are generated from the NLP functionality and the knowledge graph structure. They are provided to the users in textual and visual formats through the interface. In the following sections of this paper, we highlight the literature background of the conducted research in Sect. 2. In Sect. 3, the developed methods, including system and graph structures, graph-based explainability and the interactive interface, are presented and discussed. Section 4 provides an overview on the archived results; and the work is concluded with future highlights in Sect. 5.

2 Background and Related Work Methods that are intended to analyze structured, semi-structured and unstructured textual data utilize text mining pipelines and techniques [5, 6]. Hybridizing multiple techniques is also a method that can be used for further enhancing models in the industrial context [7]. A suitable preparation of the textual data is essential to achieve the goal of text mining.

Explainable Graph-Based Search for Lessons-Learned Documentssss

1099

Each domain-specific corpus of text requires special data pre-processing, based on the methods of documentation, data storage and the domain terminology itself [2]. Therefore, in the context of semantic representation, the utilization of domain-specific features plays a considerable role in relation extraction, and thus in constructing knowledge graphs from the information extracted from the text. An example is the work of Subasic et al. [8], where the solution to adapt the knowledge graph to the domain terminology was to build the graph on two levels: a static level, corresponding to predefined databases, and a dynamic level, that corresponds to the special terminology. Knowledge graphs have emerged as a graphical representation of a knowledge base; and got integrated with multiple technologies and applications [9]. Among the tasks that knowledge graphs have been involved in, enhancing search and matching is a promising field, especially with the advances in search engines and semantic web. Supported by text mining, Tiwari et al. [10] developed a concept to integrate the Resource Description Framework (RDF) with natural language understanding, in order to find a common format that supports human and machine understanding together. This, in turn, supports the representation and retrieval of information from heterogeneous and distributed data sets. Our approach intersects with the previous examples in terms of utilizing domainspecific text mining in constructing the knowledge graph for search and information retrieval. However, we propose an approach to integrate domain information in the graph structure itself, in order to enable: 1. Domain-specific relation extraction alongside the text-mining-based relation extraction. 2. Domain-specific search and information retrieval. 3. An easy integration of new knowledge in the graph, supporting a dynamic process of transferring lessons-learned from the past and present to future design scenarios. 4. An explainable information retrieval process, both verbally and visually, with the utilization of NLP and the knowledge graph structure. In order to achieve those contributions, a range of knowledge extraction, interlinking and management methods have been used, which are discussed in detail in the following section.

3 Search Methods 3.1 Multidimensional Domain-Specific Knowledge Graph In the handled industrial scenario, multiple information sources provide insights into the lessons-learned from previous design failures. In order to harvest the information from those sources and extract the knowledge within them, we design a knowledge graph structure that is capable of reflecting domain-specific features in heterogeneous data sources. Proposed knowledge graph consists of multiple types of nodes, each corresponding to a different data source, and thus, a different dimension of the available information in design processes. Those multi-type nodes are connected through textmining-based semantic relations, in order to model the relationships between them, and enhance the visibility of the lessons-learned documents.

1100

H. Abu-Rasheed et al.

Design cases

Linking Nodes Project Nodes

Fig. 1. Proposed multi-node-type knowledge graph

The nodes in the proposed graph concept, see Fig. 1, include: • Design Case Nodes: This type of nodes corresponds to the lessons that are learned from previous design failures and solutions. The node includes detailed information about the failure description, its reasons and solution. It represents the design document that has been generated when handling the failure. • Project Element Nodes: This type of nodes corresponds to the meta-data that expresses a structural dependency of design projects. It reflects the relationships between project modules and the chip elements. Relations between this type of nodes can uncover the relevance between failure cases that occur in multiple modules, whether within the same project or within multiple projects. • Linking Nodes: This type of nodes is the part of the proposed solution, which tailors the graph structure itself to reflect data mining results and domain-specific features. Linking nodes play two main roles in the graph: 1) representing the text mining techniques that discover similarities between past and future design cases; 2) representing the domain-specific terminology used in the context of chip-design documentation. In this research, we use Term Frequency and Inverse Document Frequency (TFIDF)

Explainable Graph-Based Search for Lessons-Learned Documentssss

1101

to provide a simple mean of ranking the similarities and reflecting an order of relevance [11]. To overcome the limitations of TFIDF, we utilize an N-gram approach and a specialized technical-term classifier, in order to correspond to documentation patterns and domain-specific terminology, such as the writing styles of design engineers, technical vocabulary and abbreviations. An entity recognition approach has been utilized to identify certain categories of vocabulary that are commonly used to describe failures and their solutions. A pre-study revealed a high tendency of design engineers to use certain entities as domain-dependent vocabulary in their documentations. This vocabulary emerges from the specific nature of the technical domain. Therefore, we exploit those entities in the form of highly weighted linking nodes that reflect a relational priority in the search functions within the graph. This, in turn, integrates the domain-specific information effectively in the information retrieval task. In addition to defining graph node types, relation extraction is the following essential task that allows graph structure to represent the application domain, and thus enhance the domain-specific information retrieval. Relation extraction in the proposed approach is based on three levels: 1. Extracting relations between the lessons-leaned documents themselves. 2. Extracting relations to linking nodes and structural nodes in the graph. 3. Adding new relations that reflect newly added lessons-learned. The first type of relations reflects the semantic inter-relations between design-failure descriptions. It inherits a cause and effect logic from the structure of the documented design reports. Relations of this type are weighted based on the TFIDF scores and the expert knowledge. The second type of relations represents the influence of domain specific terminology and project structure. It relates lessons-learned nodes to each other through the domain-specific linking nodes and project structural nodes. This type of relations has a high priority in the search and retrieval task, since domain experts have expressed a particular interest for retrieving results based on domain-specific terminology. The third type of relations represents the dynamic potential of updating the knowledge graph with accumulated design-experience through new lessons-learned. This plays the important role of ensuring a sustainable knowledge update in the organization. Those multiple levels of relations are also designed to: 1) provide a mean of ordering retrieved search results, 2) provide multiple levels of result explanation to the user. Explainable results are meant to support the user’s ability of assessing the relevancy of retrieved documents to their current design scenario. In the proposed approach, we utilize this feature to support a feedback loop, in which the expert user can effectively evaluate the retrieved result, in order to enable future relation analysis and update of the graph structure. These are continuous feedback and enhancement processes during the deployment of the system, especially with new lessons-learned being added dynamically to the graph.

1102

H. Abu-Rasheed et al.

3.2 Explainable Graph Search Functionality The goal of the proposed solution is to support design engineers with quick accessibility to previous chip design cases, including their descriptions and solutions. Therefore, the use-case of the proposed approach takes the form of a search engine, which provides the possibility to retrieve relevant design cases to a current one. The engine allows users to search previous lessons learned with a short description of either the failure itself, the module in which the failure occurs or other failure-related details. Search functionality that is designed for this solution follows the prioritized relations in the graph, to produce an ordered list of relevant, previous, design cases. From the information included in graph’s relations, search functions can retrieve results and also generate a high-level explanation regarding the reasons behind retrieving these results. Graph-based search plays a considerable role in the proposed system for retrieving not only directly related search results, but also relevant results that are not shown to the user with traditional key-word searches. In the proposed search engine, search strategies define direct-hit results, which are retrieved from graph nodes corresponding to a direct match to the search query. The strategy then utilizes graph shortest-path calculation, starting from the direct-hit node, in order to find relevant nodes to the search query. Shortest paths represent a certain length of defined relations between two nodes. Therefore, the well-defined types of graph relations in Sect. 3 perform a key role in finding those relevant results in the constructed graph. Both direct hits and relevant results are then ordered, equipped with a generated textual explanation, and provided to the user through the search interface, as shown in Fig. 2. This allows the expert to navigate to the retrieved result, get the information they are looking for and provide any necessary feedback, if needed. From a diverse spectrum of explanation methods, verbal and visual explanations were selected for this research. Verbal explanations were a direct result of the text analysis in the developed pipeline. It enabled the algorithm of following the feature similarity between two documents. Moreover, the structure of the knowledge graph revealed transitive relations between documents that are relevant to each other due to a third document that intermediates the two. Graph relations are also expressed verbally, alongside the document features in a template-based, human-understandable explanation. This explanation method was also supported by a visual component, which is based on the graph structure itself. It was provided to the user as a partial graph, containing their search input and all retrieved results, as an interlinked network of documents. This way, the user is able to see the relations directly, read their reasons in the graph, and easily differentiate the search direct results from the transitively retrieved ones. 3.3 Search Interface and Feedback Collection Enhancing the visibility of previous lessons-learned depends on the underlying representation methods of those lessons. In the application field of this research, expert design-engineers can utilize a filtered, use-case-driven document accessibility, which enables a goal-oriented retrieval of lessons learned. This is due to the considerable amounts of available information in the chip design scenario. Following that principle, an interactive search interface has been proposed to design engineers, supported by

Explainable Graph-Based Search for Lessons-Learned Documentssss

1103

Search Fields

Color Coded Results Feedback Fields

Result ExplanaƟon

Full DescripƟon of the Retrieved Result

Subset of the Knowledge Graph with the Retrieved Results

Fig. 2. Proposed graph-based search interface

tailored features, such as semi-assisted, dictionary-based writing feature, color-coded results, result explanations and quick feedback fields for a continuous online evaluation. Semi-assisted search fields provide expert users with an intermediate validation of the availability of typed-in search terms. This dictionary-based method is quicker than graph scan. It evaluates the searched terms based on their potential existence in the text corpus of the lessons-learned. When a search term is missing from the corpus, the writing assistant suggests similar vocabulary that is assured to retrieve results once searched within the knowledge graph. Explanations in the interface accompany each retrieved result, in order to enable the user to better assess the suitability and relevance of that result to their current design. Feedback fields are designed to be simple and quick, including the user’s assessment of the result’s relevance and value-added. This feedback can be potentially utilized in enhancing the graph structure and relations for better domain-specific retrieval.

4 Evaluation and Results In order to evaluate the value-added to the design engineer, we define a set of domainspecific key performance indicators (KPIs), which reflect the need for higher visibility of information regarding similar previous design cases. Defined KPIs include quantitative measures, which reflect the change in number of results through the utilization of graph

1104

H. Abu-Rasheed et al.

relations; as well as qualitative measures that have been defined to reflect the satisfaction level of users and the value-added through the explanations. The experiment setup for testing the proposed system included experts in the domain and developers. Domain experts defined multiple combinations of challenging search terms, which are usually documented in different ways depending on the engineer who is writing the report. Developers have generated variations of the search terms to cover the multiple searching methods the interface offers. Search test-combinations were then fed to the system. Retrieved results were validated in terms of the defined KPIs, i.e. their numbers in comparison to key-word search (see Fig. 3), the result’s relevancy to the current design scenario, the ability of the system to find semantically similar documents, as well as the reasoning and ratio of the graph-based transitively retrieved results to the direct-hits ones. Moreover, the feedback loop was evaluated by the experts on two levels: 1) the potential to assess retrieved result based on the generated explanations, and 2) the ability to dynamically modify the graph structure based on the provided feedback. The latter point was evaluated by the time needed to get an updated version of the graph, which reflects the provided feedback.

Fig. 3. Quantitative comparison of search results

Preliminary qualitative results showed that the expert users were able to reach relevant documents that are semantically related to their current design, despite the fact that failure descriptions used different vocabulary. The relation between those failures was discovered from the contextual information of the failure, which is successfully embedded in the knowledge graph. Qualitative evaluation of the system also revealed that the explanations have provided the experts with new insights of the documentation process itself. Experts reported that they were able to identify multiple documentation practices that should be enhanced in the company, in order to enhance the information

Explainable Graph-Based Search for Lessons-Learned Documentssss

1105

retrieval in the future. An example of such practices is the use of similar, yet different, expressions to describe the failure’s location on the chip. Figure 3 represents a preliminary outlook on the quantitative change in the number of retrieved results, through the comparison between a normal key-word search in the database, and the search through extracted relations in the knowledge graph. In this figure, a group of 22 sample searches is used. The numbers show an increase of relevant results, which are retrieved from the graph-database through their semantic relations. It is noticeable also that direct search results themselves are different from the key-word results, since they are improved by the implemented text-mining techniques. On average, a 75.6% increase of result retrieval, and thus visibility to the user, is achieved with the adoption of the proposed solution.

5 Conclusion and Future Perspectives In this research, a domain-specific search engine has been proposed in the field of semiconductor chip design. Proposed system is developed to support design engineers with a quick, filtered and use-case-oriented visibility of lessons-learned from previous design failures. A knowledge graph forms the base of the proposed search engine, which captures the semantic relations between information extracted from multiple documentation sources. Textual semi- and un-structured data, along with project metadata are mined and fused in one knowledge graph. The graph is then equipped with an explainable search functionality, which allows the retrieval of direct and relevant search results based on the graph semantic relations. An interactive search interface has been designed to provide color-coded search results along with their explanations, allowing users to provide their feedback on the retrieved results. Feedback loop is meant to support future enhancements of the graph relation extraction and prediction. The developed system is being tested on-site with end users to collect their quantitative and qualitative feedback. Future steps of the development include the design of rating functions that provide more refined weights of the search results, keeping the domain-specific needs as the main focus of the information retrieval process. Acknowledgments. This work was supported by the EU project iDev40. This project has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 783163. The JU receives support from the European Union’s Horizon 2020 research and innovation program and Austria, Germany, Belgium, Italy, Spain, Romania. The information and results set out in this publication are those of the authors and do not necessarily reflect the opinion of the ECSEL Joint Undertaking.

References 1. Chien, C.-F., Chen, L.-F.: Data mining to improve personnel selection and enhance human capital: A case study in high-technology industry. Expert Syst. Appl. 34(1), 280–290 (2008). https://doi.org/10.1016/j.eswa.2006.09.003

1106

H. Abu-Rasheed et al.

2. Abu-Rasheed, H., Weber, C., Zenkert, J., Czerner, P., Krumm, R., Fathi, M.: A text extraction-based smart knowledge graph composition for integrating lessons learned during the microchip design. In: Arai, K., Kapoor, S., Bhatia, R. (eds.) Intelligent Systems and Applications, IntelliSys 2020. Advances in Intelligent Systems and Computing, vol. 1251. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-55187-2_43 3. Thoma, S., Rettinger, A., Both, F.: Knowledge Fusion via Embeddings from Text, Knowledge Graphs, and Images, ArXiv170406084 Cs Stat (2017). Accessed on 12 Mar 2019. http://arxiv. org/abs/1704.06084 4. Guo, L., Zuo, W., Peng, T., Yue, L.: Text matching and categorization: mining implicit semantic knowledge from tree-shape structures. Math. Probl. Eng. 2015, 1–9 (2015). https://doi.org/ 10.1155/2015/723469 5. Shi, F., Chen, L., Han, J., Childs, P.: A Data-Driven Text Mining and Semantic Network Analysis for Design Information Retrieval. J. Mech. Des. 139(11), 111 (2017). https://doi. org/10.1115/1.4037649 6. Allahyari, M., et al.: A Brief Survey of Text Mining: Classification, Clustering and Extraction Techniques, ArXiv170702919 Cs, Jul. 2017, Accessed on 22 Nov 2018. http://arxiv.org/abs/ 1707.02919 7. Zenkert, J., Weber, C., Klahold, A., Fathi, M., Hahn, K.: Knowledge-Based Production Documentation Analysis: An Integrated Text Mining Architecture, p. 4 (2018) 8. Subasic, P., Yin, H., Lin, X.: Building Knowledge Base through Deep Learning Relation Extraction and Wikidata, p. 8 (2019) 9. Dietz, L., Kotov, A., Meij, E.: Utilizing knowledge graphs for text-centric information retrieval. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval – SIGIR 20ss18, Ann Arbor, MI, USA, 2018, pp. 1387–1390. doi: https://doi.org/10.1145/3209978.3210187 10. Tiwari, A., Rajesh, K., Srujana, P.: Semantically Enriched Knowledge Extraction with Data Mining. Int. J. Comput. Appl. Technol. Res. 4(1), 7 (2015). https://doi.org/10.7753/IJCATR 0401.1002 11. Al-Natsheh, H.: Text Mining Approaches for Semantic Similarity Exploration and Metadata Enrichment of Scientific Digital Libraries, p. 176 (2019)

Improving the Efficiency of Industrial Enterprise Management Based on the Forge Software-analytical Platform Nodirbek Yusupbekov1 , Fakhritdin Abdurasulov2 , Farukh Adilov3 , and Arsen Ivanyan3(B) 1 Tashkent State Technical University, Tashkent, Republic of Uzbekistan 2 “Oltin Yo’l GTL” Company, Tashkent, Republic of Uzbekistan

[email protected]

3 LLC “XIMAVTOMATIKA”, Tashkent, Republic of Uzbekistan

{Farukh.Adilov,Arsen.Ivanyan}@himavtomatika.uz

Abstract. This paper presents a unique development in the field of digitalization of a modern industrial enterprise – the software and analytical platform FORGE from Honeywell company, which allows you to visualize consolidated data from all enterprise information and automation systems that cover or are associated with a particular business process of this enterprise in a single portal window. At the same time, the FORGE platform is not just a software shell for visualizing data obtained by integrating a set of different interfaces, but is a full-fledged computing and analytical platform that works with large amounts of data and is built according to the concept of the industrial Internet of things. Keywords: Industrial Internet of things (IIOT) · Software-analytical forge platform · Digitalization · Process Reliability Advisor · Process Optimization Advisor · UniSim digital twin · Analytics · Assets · Cyber-security · Key performance indicator (KPI)

1 Introduction In the industrial sphere, production process management is subject to strict goals of constantly increasing productivity while maintaining minimum periods of enterprise shutdown for preventive maintenance and without allowing unforeseen failures in the operation of the production facility. Any failure to comply with one of these conditions leads to significant economic losses, while at the same time increasing the level of risk to the reliability and safety of the production facility. The situation is further complicated by the fact that a modern industrial facility is still managed by human personnel and also depends on the factors of organizational interaction between the various services of the enterprise, as well as on the level of qualification, knowledge and experience of employees. These factors are constantly changing and it is extremely difficult to ensure stable management of the object in these conditions. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1107–1113, 2022. https://doi.org/10.1007/978-3-030-80119-9_74

1108

N. Yusupbekov et al.

In addition, in a kind of negative role was played by technological progress in the aspect of development of systems of automation and informatization, which led to the equipping of individual assets of the enterprise in the direction of highly specialized, critical tasks of sustainable functioning and even improve the efficiency of the asset in offline mode, i.e. without considering the linkages of the asset or individual asset item other assets. Contribution of our investigation described in this paper helps to overcome these challenges and provide industrial companies with analytics from real-time data, visual intelligence and actionable recommendations to sustain benefits across their enterprise. It is additional fundamental element into the concept of Industrial Internet of thing (IIOT) technology [1, 2] with new form of automation system architecture that balances the computational and life-cycle benefits of cloud computing with the requisite onpremise, appliance-hosted capabilities necessary to provide safe, secure and long-lasting automation for complex manufacturing systems and processes.

2 Scientific-Technical Solution To solve these problems, it is necessary to implement the concept of a holistic informational-control complex, which considers the entire production facility and its business processes as a whole. Such informational-control complex is built based on software-analytical Forge platform, which includes a set of analytical tools for working with real-time databases (RTDB), tools for visualized intelligence and a generator of recommendations for successful management of the production facility. The concept of the Forge software and analytical platform is to maximize the digitalization of data from all sources, to model all assets of a production facility in projections of various aspects of the purpose of models, and to apply a number of methods for analyzing large amounts of data obtained [3] both from real physical assets and from models of virtual digital twins of these assets [4, 5]. At the same time, a number of effective methods are based on comparing the modes of functioning of a real asset and its virtual model. The philosophy of increasing the efficiency of a production enterprise through digitalization and the use of the Forge software and analytical platform is based on the following areas of optimization of production processes, which respectively form the basis of the software and technical components of the Forge platform: 1. 2. 3. 4. 5.

Increasing the productivity of the technological processes themselves Increasing the productivity of process control Improving asset productivity Increase staff productivity Increase the productivity of the enterprise as a whole.

2.1 Increasing the Productivity of the Technological Processes Improving the performance of technological processes themselves is based on modeling of digital twins of technological processes in the Unisim software environment, which is

Improving the Efficiency of Industrial Enterprise Management

1109

used for the design and optimization of industrial technological processes and is equipped with functionality of subsequent dynamic simulation of the flow of these processes. To identify ways to improve the efficiency of technological processes, Unisim digital twin interacts with the software-analytical modules Process Reliability Advisor and Process Optimization Advisor [6]. The Process Reliability Advisor software-analytical module is based on diagnostics and detection of events that anticipate a certain direction of the further flow of the technological process, which is recommended to change. The module is based on mathematical models of failures in the technological process, developed by the licensor of this technological process, that is, the highest expert party with many years of deep experience and knowledge about this process. The Process Reliability Advisor software analysis module allows you to: • continuously monitor the current performance of the process plant; • understand how close the current process mode is to the performance limits of the process plant; • identify events at an early stage and based on this make decisions about changing the course of the technological process; • generate suggestions for operational and dispatching personnel for further management of the technological process. The software-analytical module Process Optimization Advisor is based on calculation algorithms from the process licensor that calculate changes in the outputs of the target and by-products of the process depending on different scenarios for the further flow of this process, simulated by Unisim digital twin, as well as the resulting economic indicators after applying these changes. 2.2 Increasing the Productivity of Process Control Increasing the productivity of process control mainly gives the maximum effect for complicated high-technology industries, where the final result depends largely on the accuracy and fineness of setting the process control mode. The corresponding Profit Performance Monitor software module is essentially a cloud-based solution for the application of the advanced process control system (APC) [7], embedded into the software and analytical Forge platform. In addition to standard tools for managing of manipulated variables (MV) and controlled variables (CV) associated with specific parameters of process control algorithms, the Profit Performance Monitor software module combines different scenarios of interaction of all algorithms in a complex and calculates the economic component of the process plant operation as a whole. 2.3 Increasing Asset Productivity Improving asset performance in the software-analytical Forge platform is provided by Asset Performance Management software technology, which allows you to collect data from all assets of the enterprise in real time mode and again by means of a simulated digital twin (performed in a different projection of the modeling object) to perform

1110

N. Yusupbekov et al.

subsequent analytical actions with conclusions and recommendations for further more efficient operation of the enterprise assets. The toolkit of Asset Performance Management technology includes the following features: 1. Integration of asset data with process data. 2. Built-in expertise of asset knowledge. This function implies that the software includes libraries of expert knowledge about this asset from the manufacturer of this equipment. The accumulated templates and reference characteristics of equipment operation from the manufacturer serve as a model for comparing real statistical data on the operation of specific units of equipment (assets) with them and for alerting deviations in the operation of this asset. Working templates and reference characteristics are formed in the form of a standard mathematical model of the performance of this asset. 3. Predictive Analytics. The Asset Performance Management component has builtin machine learning models that provide predictive analytics to further improve performance indicators. While the standard asset performance model determines the current parameters of energy consumption and efficiency of this asset, the predictive analytics model based on big data identifies inconsistencies in the operation of the asset, determines the root cause of such behavior, and assesses the likelihood of future consequences. 4. Interactive computing mechanism. All calculations in digital twin models are performed in a software environment linked to a real-time database that continuously collects events from real physical assets and technological processes. This, firstly, allows to save time for calculations with parallel computing for a large number of assets simultaneously, and, secondly, ensures a constant interaction between the real object and digital twin with all benefits of this synchronization and mutual correlation of the behavior of a real object and a digital twin. In addition, the advanced configuration capabilities of the computing engine allow you to embed standard calculations from manufacturers, perform advanced programming (for example, in C# and VB), and run utilities for data analysis and scientific research (Python, R, etc.) directly from the database of the Asset Performance Management component. 5. Detection of events. This function is an integral part of the mechanism for visualization and notification of any detected events, including data about assets and technological processes, results of performance models and analytical models, and results of computing mechanisms. Notification of an event occurs in real time via electronic communication, on the wide interface of various intuitive visualization panels, into the MES and ERP systems [8] that perform further actions on auxiliary business processes (for example, automatic transmission of information to the system of pre-ordering of equipment items that are predicted to fail after a certain period of time). 2.4 Increasing Staff Productivity Improving staff productivity means equipping an employee with a set of personal digital devices that interact in real time with the Forge software and analytical platform. Increased efficiency is provided both by the completeness of the set of all necessary

Improving the Efficiency of Industrial Enterprise Management

1111

devices (which, in fact, complement or integrate into the set of personal protective equipment), and by a wide range of specialized applications installed on these mobile devices. 2.5 Increasing the Productivity of the Enterprise as a Whole Increasing the productivity of the enterprise as a whole is essentially provided by a combination of all the components of the software and analytical Forge platform, but the quintessence of this combination is the software shell for visualization of the entire Forge platform, in which manufacturers paid great attention to creating the most useful, convenient and intuitive interface for faster and correct decision-making on the management of the production enterprise (see Fig. 1).

Fig. 1. Human-machine interface of software-analytical forge platform.

2.6 Cyber-Security Aspect in Forge Platform The Forge software and analytical platform is a clear example of a scientific paradigm shift in the theory of managing production processes and objects [9, 10]. Being closely related to scientific achievements in the field of big data analytics, the concept of the Forge platform implies a change of approach to the theory of creating information systems and control systems. At the same time, the new concept pays special attention to another important component of the platform, which performs its highly specialized task, but at the same time ensures the integrity and security of the platform’s information model. This is a component that provides information security (or, as indicated in many literature sources, cyber security [11, 12]) of the Forge software and analytical platform. The Forge Cybersecurity component performs the following functions to ensure the platform’s cybersecurity:

1112

N. Yusupbekov et al.

1. Monitoring and managing threats and monitoring the compliance of events with information security policies 2. Secure remote access to the platform 3. Secure data transfer between platform components 4. Analysis of integrated assets for cyber security 5. Management of operating system and basic software update systems 6. Managing antivirus software systems

3 Conclusion Our investigation described in this paper is based on previous and related works for creation of single-pane-of-glass technology to consolidate and visualize all data of large enterprise [9, 10, 13]. Forge software-analytical platform is adopted to variety of industrial applications like upstream, midstream and downstream segments of oil and gas industry, chemical and petrochemical industry, pulp and paper industry, mining and metallurgy, as well as many others potential various applications. This is the field of our future works which will be dedicated to development of structural set of solutions and applications for embedding them into universal infrastructure of Forge platform targeting KPIs of every industrial sector and concrete enterprises. Forge software-analytical platform is a landmark scientific-application solution that completely rethinks the approach to an integrated information management system for a production facility. This solution is based on a total digital transformation of the production enterprise and involves a significant transfer of functions and responsibilities from human to machine through the real application of the latest scientific achievements in the field of big data analytics, machine learning and artificial intelligence in a programmable environment.

References 1. McLaughlin, P., McAdam, R.: The Undiscovered Country: The Future of Industrial Automation. https://www.honeywellprocess.com/en-US/online_campaigns/IIOT/Doc uments/The-Undiscovered-Country.pdf (2016) 2. Blanchette, B.: How to implement the Industrial Internet of Things, Control Engineering. https://www.controleng.com/articles/how-to-implement-the-industrial-internet-ofthings (2016) 3. Zhiqiang, G., Zhihuan, S., Steven, X.D., Biao, H.: Data mining and analytics in the process industry: the role of machine learning. IEEE Access 5(12), 20590–20616 (2017) 4. Kaur, M.J., Mishra, V.P., Maheshwari, P.: The convergence of digital twin, IoT, and machine learning: transforming data into action. In: Farsi, M., Daneshkhah, A., Hosseinian-Far, A., Jahankhani, H. (eds.) Digital Twin Technologies and Smart Cities. IT, pp. 3–17. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-18732-3_1 5. Yusupbekov, N., Abdurasulov, F., Adilov, F., Ivanyan, A.: Concepts and methods of “digital twins” models creation in industrial asset performance management systems. In: Kahraman, C., Cevik Onar, S., Oztaysi, B., Sari, I.U., Cebi, S., Tolga, A.C. (eds.) INFUS 2020. AISC, vol. 1197, pp. 1589–1595. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-511562_185

Improving the Efficiency of Industrial Enterprise Management

1113

6. Yusupbekov, N., Abdurasulov, F., Adilov, F., Ivanyan, A.: Application of cloud technologies for optimization of complex processes of industrial enterprises. In: Aliev, R.A., Kacprzyk, J., Pedrycz, W., Janshidi, M., Sadikoglu, F.M. (eds.) 13th International Conference on Theory and Application of Fuzzy Systems and Soft Computing — ICAFS-2018, pp. 852–858. Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-04164-9_112 7. Yusupbekov, N., Abdurasulov, F., Adilov, F., Ivanyan, A.: Application of advanced process control technologies for optimization of polymers production processes. In: Aliev, R.A., Kacprzyk, J., Pedrycz, W., Jamshidi, Mo., Babanli, M.B., Sadikoglu, F.M. (eds.) 10th International Conference on Theory and Application of Soft Computing, Computing with Words and Perceptions – ICSCCW-2019. AISC, vol. 1095, pp. 345–351. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-35249-3_44 8. Matvienko, E., Ivanyan, A.: Intelligent information technologies for integrated management systems of enterprises with a complex scheme of gas-extraction and processing. J. Multimedia Inform. Syst. 2(3), 249–254 (2015) 9. Yusupbekov, N., et al.: Using of Sentience platform for integration of intelligent systems and devices into cloud. In: Chemical Technology. Control and Management, N4-–5 (82–83), WCIS-2018, pp. 67–71 (2018) 10. Yusupbekov, N., et al.: Advantages of analytic solutions of sentience cloud platform. In: Chemical Technology. Control and Management, N4–5 (82–83), WCIS-2018, pp. 9–12 (2018) 11. Ziegler, S.: Internet of things cybersecurity paradigm shift, threat matrix and practical taxonomy. In: Ziegler, S. (ed.) Internet of Things Security and Data Protection. IT, pp. 1–7. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-04984-3_1 12. Brékine, A., et al.: Network threat analysis. In: Ziegler, S. (ed.) Internet of Things Security and Data Protection. IT, pp. 81–92. Springer, Cham (2019). https://doi.org/10.1007/978-3030-04984-3_5 13. Yusupbekov, N.,Sattarov, Sh., Doudin, I., Adilov, F., Ivanyan, A.: Application of solutions of connection of production cluster to analytical cloud in chemical industry. In:Proceedings of the International Conference on Integrated Innovative development of Zarafshan Region: Achievements, Challenges and Prospects, October 26–27, 2017, vol. I, pp. 246–252

Consumer-Oriented Web of Things Solution for Competitive Swimmers Rusho Yonit(B) , Resh Amit, Benaroch Mickael, Bronshtein Nikita, Shahar Lior, and Beni Valotker Shenkar College, Ramat-Gan, Israel [email protected]

Abstract. The current study address the question: can consumer-oriented Web of Things (WoT) leverage personal sports achievements? If properly exploited, accurate data collected via WoT architecture can help to gain valuable insights regarding personal sport achievements. The field of swimming suffers from a lack of budget in some countries. The direct consequences are lack of equipment and tools for swimmers and trainers, mainly for the purpose of recording swimming achievements and tracking them. This study presents a comprehensive solution for the following purposes: (1) provide a reliable swimming recording mechanism using consumer-oriented WoT; (2) provide abilities to produce and consume personal information based on collected data using Cloud Computing and emerging Web technologies. Moreover, putting data accuracy in the limelight, the WoT architecture aims to synchronize data between several wireless sensors and communicate collected data to the Cloud using minimal delay. The description about the interconnecting WoT architecture and the integrated mechanisms for reliable measurement of parameters by smart sensors and transmission of data via the Web is being presented. The main goal of the current research is to use WoT architecture for the design and implementation of a comprehensive solution for personal consumer-oriented WoT to leverage sports achievements. The research contribution is laying the basis to the future of consumer-oriented WoT for the individual level using Web and Computer Engineering. Keywords: Web of Things · WoT architecture · Consumer-oriented WoT · WoT for sport

1 Introduction Can consumer-oriented Web of Things leverage personal sports achievements? Industry 4.0, the fourth industrial revolution, aims to achieve higher level of operational efficiency, productivity and automation [1]. On the one hand, by transferring data between machines, and by embracing Internet of Things (IoT) technologies, Industry 4.0 establishes knowledge management, adaptation of human needs, smart cities, factories and more [2]. On the other hand, consumer-oriented IoT refers to individual-level customization. Meaning, using key technologies in order to improve not only high-level industrial © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1114–1127, 2022. https://doi.org/10.1007/978-3-030-80119-9_75

Consumer-Oriented Web of Things Solution for Competitive Swimmers

1115

eco-systems, but also local, personal and individual needs. The current research integrates Wireless Sensor Networks (WSN) and real-world devices to the cloud, allowing clean synchronization with other virtual and physical resources using Restful resources over HTTP. Hence, this paper addresses a case study of customizing an industrial product in the competitive swimming field, using Web of Things for individual use in local pools. The customized platform enables collecting data, producing private information and predicting achievements based on data analysis. The most important parameter in swimming competitions is time recording and accuracy. In the field of competitive swimming, a millisecond is significant. Swimmers spend most of their time in training in order to improve their achievements, and the absent of appropriate technology is critical. The Web of Things (WoT) is considered a new paradigm, which enhances human lives by combining technologies [3] and communicates on the Web. In the context of sports, the adoption of emerging WoT methods significantly improves sport experience [4]. In addition to improving experience, our goal is to improve sport achievements. Considering the level of accuracy needed for measuring swimmers accomplishments, the use of WoT outside the pool (by wireless sensors attached to diving boards) and inside water (wireless sensors attached to the pool walls) is an appropriate approach, though there are architectural limitations, explained later. A solution is needed in order to record achievements with maximum accuracy, and to enable effective training for both trainers and swimmers. Expensive IoT solutions are not suitable for local swimming associations. Therefore, if an automated solution cannot be used, trainers measure achievements with a stopwatch and manually write records on a physical page. However, this method disables reuse of information for statistical examination, information search, and automatic detection of a downward trend or increase in swimming achievement. In addition, if no digital solution exists, trainers define training routine on a board outside the pool. For example, Fig. 1 illustrates a training routine written manually on a board, when a team has no digital solution.

Fig. 1. A manual training program on a physical board

1116

R. Yonit et al.

Therefore, the first requirement is as following: The comprehensive solution aims to provide trainers with a reliable recording system. Focusing on data accuracy, timestamp should be recorded at the exact millisecond a swimmer jumps to the water or touches the wall inside the pool. The system should record swimming times in training, not merely in competitions. If used during training, swimmers can simulate competitions. There are specific time recording needed: (1) jumping reaction time (2) time when touching the far wall with some part of the body (3) time when touching the wall, at the end. The second functional requirement refers to the management capabilities. The solution aims to provide abilities to define manage and track accomplishments in addition to define training. Meaning, trainers should have the ability to define trainings processes, view swimmers achievement history, set goals for swimmers and receive notifications regarding achievement improvement or weakening. In addition to trainers, the solution should assists swimmers in seeing their achievements trend along a season. The main goal of the current research is to use WoT architecture for the design and implementation of a comprehensive solution for personal consumer-oriented WoT to leverage sports achievements. The contribution of the current research is to lay the basis to the future of consumeroriented WoT for personal and customization use. The next section elaborates on related work.

2 Related Work The previous section detailed two functional requirements: recording swimming achievements and managing personal trainings. Further, this section refers to functional and no-functional requirements based on related work in the contexts of WoT architecture and information production. The first functional requirement relates to the ability to record competitive swimmers achievements. Therefore, time should be recorded using IoT technology, and information production and analyze should be performed on WoT architecture. Indeed, creating networks of “smart things” using RFID, wireless sensors and embedded devices on a large scale has become the goal of Internet of Things researches and solutions. Using popular Web technologies (HTML5, JavaScript, Ajax, React, Node.js, etc.) can be implemented in applications using smart things, and users can leverage well-known Web methods (such as browsing, searching, linking) to interact and communicate with these devices in a Web of Things architecture [5]. Two optional ways enables integration of devices to the Web using REST: a direct integration and a smart gateway-based approach for resource-limited devices, which have no Internet Protocols (IP) [6]. The first approach relates to communication between sensors with IP and the second approach refers to a gateway solution for integrating through API Access. Maximum synchronization between devices for accurate data collection is a challenge when sending and receiving data using REST. In the field of smart healthcare systems, wearable devices and other ubiquitous technologies become inevitable. Using for fitness, medical indexes, wearable devices leverage consumer-oriented services of the IoT technology [7]. IoT based health status measurement are widely used by sport

Consumer-Oriented Web of Things Solution for Competitive Swimmers

1117

companies and intelligent sport equipment [8]. Advancements in Wireless Sensor Networks usage and methods is reported and researches suggest effective implementations in low cost [9]. Nevertheless, if implementing technologies such as cloud computing and Big Data in order to store a large volume of data, collected by variety wireless distributed sensors, the traditional WoT architecture may cause delays or inefficiency. However, time precision is the most important parameter in swimming competition. FINA, the world swimming governing body, used to increase its timing precision by adding thousandths-of-seconds to the recorded time. After repeated studies and discussions, they concluded that hundredths-of-a-second is sufficient for the accuracy of swimming times [10]. Hence, the first non-functional requirement is as following: The WoT architecture aims to synchronize data between several sensors and communicate collected data to the Cloud using minimal delay. Meaning, a solution is needed with maximum accuracy. The second functional requirement relates to the ability to define, manage and track records and training based on collected data. IoT health and fitness data is collected in variety of different heterogeneous sources and in proprietary formats. A Web platform for collecting and analyzing information was developed for public use: sharing, understanding, discovering and reusing the information [11]. Public information refers to goods which are non-excludable and non-rivalrous (meaning, consumption is not regulated by pricing, and one’s use does no reduce the availability to others) [12]. Though, in the case of personal consumer-oriented WoT, due to privacy regulations, the produced information must be private, restricted by authentication. The collected data from the Wireless Sensors for personal training routing is an example for the need of cloud computing and servers. As the second functional requirement relates to the ability to manage training routines, questions arise regarding the process of producing this critical information and regarding the best devices for this goal. Answering the first question, the process of creating information is defined as a five stages circular process: specification, design, implementation, validation and evolution [13, 14]. The specification step includes determine target audience (swimmers who need to see their training compared to their actual records). In addition, this first step includes understanding the goal and constrains of the produced information. For example, records in a different seasons vary. The design step includes planning how the consumers-swimmers will use the produced information. For example, the personal training will be watched on a mobile phone prior to the actual training. The development step is the actual creation of the information. Meaning, creating a personal training and defining goals for the upcoming season. The validation ensures that produced information fulfills consumers’ expectations and needs and that mistakes are minimal. For example, if the goal is to maximize safety for children riding on their kick scooter to school, then through the validation step, the producer should go backwards and verify that the produced track is the safest path for the child, before the child actually uses the track in real-time. The final step is evaluating the produced information. It can be performed by the trainer testing swimming records it in real-time, compared to the plan.

1118

R. Yonit et al.

Having explained the roles of information consumption and production in the context of a private consumer-oriented WoT, and the advantages of mobile phone, the third functional requirement is as following: The solution should allow the trainer to produce private information in advanced, and enable both swimmer and trainer to consume it in during and post swimming sessions. Regarding the second question, mobile phone combined with web-enabled devices and sensors, have the potential to enhance user knowledge, perception and experience [15]. Hence, the second non-functional requirement is: the solution should be mobilebased. Meaning, combine ubiquitous computing via mobile phone and WoT architecture. The name of the integrated solution is SwimTouch. The following sections elaborate on its design and implementation of the solution.

3 Solution Approach With the goals to record swimmers’ achievements when jumping and under water, and manage swimmers’ achievements; the SwimTouch solution embraces WoT methods for communication between devices. In order to answer both main goals, the integrated solution is divided into two subsystems. The first sub-system refers to Swimmer Achievement Record System, which focuses on creating an embedded system for recording swimmers’ achievements. This functionality supports various swimming styles: butterfly stroke, breaststroke, front crawl or freestyle, backstroke and the combined medley. All measures occurs in real time and with maximum precision. Figure 2 shows the Swimmer Achievement Record System in action during training. Two swimmers are jumping. The trainer starts the timer. The swimmers hear the “beep” and jump to the water. The record system analyzes their reaction time, which is crucial in swimming.

Fig. 2. Swimmer achievement record system in action

The second sub-system refers to the Swimmer Achievement Management System, which focuses on (a) providing functionalities for the trainer to produce information regarding training routines via a mobile phone; (b) tracking swimming achievements;

Consumer-Oriented Web of Things Solution for Competitive Swimmers

1119

and (c) visualize statistics and analytics. Figure 3 displays the calendar in the trainer’s interface. Each color represent a training for different groups over the month.

Fig. 3. Calendar in swimmer achievement management system for the coach

SwimTouch has two main roles: swimmer and trainer. Each role can perform specific activities in the system. The swimmer is an end user who uses the system to view the next training, get information about achievements, watch statistics and watch recording achievements during training.

Fig. 4. System activities by roles

1120

R. Yonit et al.

A trainer is an end user who uses the system to produce personal information regarding trainings, obtain information about achievements, view statistics and use the system actively during training. The record module is in charge of recording swimming achievements during training. The data is detected in a sensor in each side of the pool and on the springboard. The data is sent to the broker. The broker component is responsible for coordinating communication between sensors and data base in the cloud repository. The broker transmits data to the trainer and swimmer management sub-system. Figure 4 displays the system activities by roles and sub-systems: management and recording modules. The following section describes the software and hardware architectures and system design of the integrated SwimTouch solution.

4 System Overview The system includes a software and a hardware approaches. From the software point of view, it includes activities that can be performed before, during or post training. Before training, the trainer can produce personal information such as new training: personal/group and daily/monthly/yearly and assimilation in the system. After the training is set up, the system server sends it to the swimmer as a future assignment. Following, the trainer enters the system, chooses the training and launches the inner exercise. A trainer can update or delete each training. If updated- the swimmer receives the update. A trainer can watch swimmer’s history achievements and statistics by filtering via swimmer, date or style of swimming. In addition, a trainer can set and check swimmers goals. This activity includes creating goals for each swimmer in a specific style. Moreover, the trainer receives automatic notifications if one or more swimmers fail to achieve their goals. The second role that can activate the system before training is the swimmer. The swimmer, as a consumer, can view personal information, such as the list of tasks sent by the trainer, records history and personal goals. During training, the trainer launches the swimmers individually or in groups according to pre-set training. Moreover, a trainer can watch and track results in real time on a tablet or a smartphone. The second role that can activate the system during training is the swimmer. The swimmer can view records in real-time. From the hardware perspective, there are two main sensors. The touch sensor is located under water on each wall of the pool. The swimmer touches it when turning and at the end of the round. Each sensor detection signal is transferred to the recording system. It includes the exact touch time and the unique ID of the sensor to know where to associate the data and display it in real time. The timer continues to run continuously until the end of the timer. The second sensor is located on the diving board. It identifies weight on the sensor. Its starting time t0 is 0 s. The weight sensor identifies when the swimmer leaves the diving board. It stops the timer and delivers a package with the exact time with a unique ID of the sensor, in order to associate the data to the specific diving board. When the swimmers are standing on diving boards, the trainer can launch the training exercises from the smartphone. A buzzer is heard, and swimmers jump into the water. Each sensor on each diving board detects the reaction time. The reaction time is sent as

Consumer-Oriented Web of Things Solution for Competitive Swimmers

1121

a signal with time stamp to the record module. When swimmers touch the touch sensor located under water on the far wall of the pool, each sensor sends its signal with time stamp to the record module. By the end of the exercise (after the last touch), the record module sends all the times as an Array to store in the database on the cloud. Figure 5 displays in-training flow.

Fig. 5. During training flow

Post training all information is saved on the Cloud. Once trainers or swimmers enter the system, the management module runs an automatic job that checks each swimmer’s achievement history and compares them. If the system identifies an improvement or a weakness in achievement trend, it sends notification to both swimmer and trainer using Sockets. Additionally, the trainer is able to enter the system and navigate to see goals, and check if a specific achievements of specific swimmers. The post training system flow is detailed in Fig. 6.

Fig. 6. Post training flow

The following section details the configuration of the solution.

1122

R. Yonit et al.

5 Limitations SwimTouch has limitations. There must be an active Wi-Fi connection near the swimming pool, in a radius of 15–20 m. In addition, the Arduino sensor should be configured to work with the existing network system. The configurator should write the Wi-Fi credentials (username and password) into the Arduino C language code. On one side of the pool, every couple of touch-sensor and weight-sensor should be connected to the master Arduino. In the opposite side of the pool, every touch sensor should be connected to the slave Arduino. In addition, the Buzzer should be connected to the master Arduino. Before launching a training, every swimmer has to stand up on the diving board. When a trainer launches a training, he or she has to enter the system via a computer or a smartphone, search and choose the specific training, and then the specific exercises in the training. Next, the coach clicks “Enter TV Mode”. In this way, every device that is connected to the web socket will get real time results from the system. Due to the fact that the touch sensors are located under water, they must be vacuumed. Swimmers must be with no shoes when standing on diving boards.

6 Implementation Details As mentioned, SwimTouch solution includes embedded hardware and programmed software. Regarding the software, the programming languages in use are as following: Clientside is written in Typescript and uses Angular 7 JavaScript framework. Server-side is written in Javascript and uses NodeJS. Broker component is written in JavaScript and uses NodeJS. The next figures display the interfaces. Figure 7 displays statistics page of a particular swimmer. For example, swimmers can see their reaction jump-time statistics.

Fig. 7. Swimmers’ reaction jump-time statistics

Figure 8 displays the launch screen interface, used during training to launch the swimmers. It includes real-time records for reaction time and each wall-touch. Regarding the hardware for the recording system, we used ESP8266 system-onchip (SoC), which integrates a microcontroller with Wi-Fi module. In addition, we used MPR121 capacitive touch sensor module. The jumping springboard was developed to match real use in wet area. A metal tape was placed on a jumping board and connected

Consumer-Oriented Web of Things Solution for Competitive Swimmers

1123

Fig. 8. Launch screen during training

to capacitive touch sensor used as electrode to register a changes in capacitance when jumps. Figure 8 shows the final springboard. The touch sensors are located on the wall in each side of the pool, under water, in each route. Touch boards are used as a switch to register a swimmer touch. Figure 9 shows the final touch surface located under water in the pool. In addition, we used an electronic buzzer for the sound of startup in the beginning of the race. Figure 10 displays the electronic buzzer.

Fig. 9. The springboard

Fig. 10. Touch surface located under water

Figure 11 displays the electronic schema of the entire recording system.

1124

R. Yonit et al.

Fig. 11. Electronic schema of the recording system

The following section elaborates on the evaluation of the integrated solution.

7 Validation and Evaluation The validation process of SwimTouch is divided into two levels: testing the recording module and testing the management module. The recording module was tested under water with a high degree of chlorine and a water pressure. Tests were performed at a local half-Olympic swimming pool and in a laboratory. In order to test the module without installing the system physically under water, we wrote a script in JavaScript, which simulate jump and touch detection and publish a random time from a publisher to all subscribers via web-sockets. The main concern when evaluating the record system was time accuracy. At first, each touch and jump time were transmitted separately to a web server by MQTT protocol- a lightweight messaging protocol for small sensors and mobile devices. Following, timestamp was taken at the server and the delta time between each two timestamps was calculated and saved as the recording time. The first architecture was as seen in Fig. 12. Following series of tests, we came to realize that this approach causes time delays. Therefore, in order to reduce time delay, we changed the architecture. We measured local times of each microcontroller. Hence, one microcontroller was a master and the second a slave. Data was transmitted between them with TCP/IP by Wi-Fi module. When the buzzer sounds, the reactive time is taken at a master microcontroller and each touch of a sensor is transmitted by a slave to a master. This approached improved time delay. Time delay measurements between two sensors after the upgrade of the architecture in stage 2, are displayed in Fig. 13. Testing the management module was performed with Karma tool for JavaScript servers test. Every component was being tested on every build process which must have returned a success key. However, if test fails, the build fails consequently. For client side, we tested all Angular components with Jasmine and Karma.

Consumer-Oriented Web of Things Solution for Competitive Swimmers

Fig. 12. Architecture at stage 1

Fig. 13. Time delay measurements

1125

1126

R. Yonit et al.

8 Future Work The first improvement relates to the use of SwimTouch As-a-Service. In order to serve the swimming sport industry, the entire solution should be wrapped in an API. The sensors should improve their water-resistant feature. Network delay should be reduced. Currently, the delay is about thousand-a-second, depending on Wi-Fi speed.

9 Conclusion The field of swimming suffers from a lack of budget in some countries. The direct consequences are lack of equipment and tools for swimmers and trainers, mainly for the purpose of recording swimming achievements and track them. On the one hand, Industry 4.0 aims to achieve higher level of operational efficiency and productivity. Using IoT technologies, Industry 4.0 leverage smart cities, factory and the entire society. On the other hand, consumer-oriented solution is needed for individual and personal needs. This project integrates WoT architecture with physical sensors, mobile phones and Cloud Computing techniques. Consumer-oriented WoT relates to the use of combining Web technologies and methods in order to improve personal customization. Existing solutions in the field of competitive swimming are divided to solutions which record swimmer’s achievements without the possibility of managing their progress over time, and vise versa. The current comprehensive solution combines the two core functions. Moreover, local swimming teams cannot use these systems to improve their swimmers’ achievements in local pools due to high costs. Hence, the current study aims to offer a recording and managing swimmers achievements solution for the swimming industry, while keeping accuracy as a first-priority engineering requirement at minimal costs. The comprehensive solution integrates an embedded hardware module, using wireless sensors, for the measurement of swimmer’s achievements; and a responsive software solution for the production of personal information, based on the collected data. The study started with the question: Can consumer-oriented Web of Things leverage personal sports achievements? This research lays the ground for the use of WoT architecture to improve experience in the sports field. Putting accuracy in the limelight, this study address the inevitable link between data accuracy and competitive swimming. Further study should establish consumer-oriented WoT for variety fields and personal needs, seeking for solutions for the individual level using Web Engineering.

References 1. Yang, L.: Industry 4.0: a survey on technologies, applications and open research issues. J. Ind. Inform. Integr. 6, 1–10 (2017). https://doi.org/10.1016/j.jii.2017.04.005 2. Meško, M., Suklan, J., Roblek, V.: The Impact of the Internet of Things to Value Added in Knowledge-Intensive Organizations. In: Mohiuddin, M., Norrin Halilem, S.M., Kobir, A., Yuliang, C. (eds.) Knowledge Management Strategies and Applications. InTech (2017). https://doi.org/10.5772/intechopen.70070 3. Ikram, M.A., Alshehri, M.D., Hussain, F.K.: Architecture of an IoT-based system for football supervision (IoT Football). In: IEEE World Forum on Internet of Things, WF-IoT 2015 – Proceedings, pp. 69–74 (2015)

Consumer-Oriented Web of Things Solution for Competitive Swimmers

1127

4. Mainetti, L., Patrono, L., Stefanizzi, M.L.: An Internet of sport architecture based on emerging enabling technologies. In: 2016 International Multidisciplinary Conference on Computer and Energy Science, SpliTech 2016 (2016) 5. Bucherer, E., Uckelmann, D.: Business Models for the Internet of Things. In: Uckelmann, D., Harrison, M., Michahelles, F. (eds.) Architecting the Internet of Things, pp. 253–277. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19157-2_10 6. Guinard, D., Trifa, V.: Towards the Web of Things: Web Mashups for Embedded Devices. In: Workshop on Mashups, Enterprise Mashups and Lightweight Composition on the Web (MEM 2009), pp. 1–8 (2009) 7. Poongodi, T., Krishnamurthi, R., Indrakumari, R., Suresh, P., Balusamy, B.: Wearable Devices and IoT. In: Balas, V.E., Solanki, V.K., Kumar, R., Ahad, M.A.R. (eds.) A Handbook of Internet of Things in Biomedical and Cyber Physical System. ISRL, vol. 165, pp. 245–273. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-23983-1_10 8. Ray, P.P.: Generic Internet of Things architecture for smart sports. In: 2015 International Conference on Control Instrumentation Communication and Computational Technologies, ICCICCT 2015, pp. 405–410 (2016) 9. Nayak, R.C., Agarwal, R.: A model of creativity and innovation in organizations. Int. J. Transform. Bus. Manag. 1, 1–8 (2011) 10. Burke, T.: This Is Why There Are So Many Ties In Swimming (2016). (Online). https://dea dspin.com/this-is-why-there-are-so-many-ties-in-swimming-1785234795 11. Sweeney, L., Abu, A., Winn, J.: Identifying participants in the personal genome project by name. SSRN Electron. J. (2013). https://doi.org/10.2139/ssrn.2257732 12. Raban, D.R., Rafaeli, S.: The effect of source nature and status on the subjective value of information. J. Am. Soc. Inf. Sci. Technol. 57(3), 321–329 (2006) 13. Rusho, Y., Raban, D.R.: Hands on – information experiences as sources of value. J. Assoc. Inf. Sci. Technol. 14. Rusho, Y., Raban, D.R.: The effects of information production process on experience and evaluationes. In: The 19th Annual Conference of the Association of Internet Researchers (AoIR 2018), Transnational Materialities Montreal, Canada, 10–13 Oct. 2018 (2018) 15. Kamilaris, A., Pitsillides, A.: Mobile phone computing and the internet of things: a survey. IEEE Internet Things J. 3(6), 885–898 (2016)

Factors Influencing the Adoption of Online Banking in South Africa: An UTAUT Study Mathias Mujinga(B) School of Computing, University of South Africa, Johannesburg, South Africa [email protected]

Abstract. Online banking has become a mainstream model of conducting banking, especially in the developed economies. Unfortunately, the adoption in the developing countries is still lagging behind due a number of factors. Prior research studies have identified these factors (positive or negative) that affect online banking adoption using a plethora of technology adoption models. One such model widely used in adoption studies is the unified theory of acceptance and use of technology (UTAUT). The research constructs outlined in this study were developed based on the UTAUT2 with additional control variables in South African context. There is limited empirical work that combines adoption factors and usability of the service that might hinder potential users from accepting the technology. This paper explores the factors affecting online banking adoption using an integration of UTAUT2 model and usable security principles to propose a theoretical model to explain customers’ intention to use online banking. To researchers, the paper offer insights to the usefulness of integrating UTAUT and other aspects such as usability in identifying and addressing adoption obstacles of new technologies. The implications of integration of UTAUT and usability into the proposed online banking adoption model are discussed. The paper presents a research model that combines UTAUT and usable security to address online banking adoption and security. Keywords: UTAUT · Online Banking · Usable Security · Information Security · Usability

1 Introduction Online banking (OB) has since revolutionise the way individuals and organisations conduct their banking activities. With the increased worldwide internet penetration, financial institutions are taking advantage and providing their clientele with self-service banking solutions. OB provides benefits to both clients and financial institutions. Banks benefit by improving their service quality in the ever-increasing competitive retail banking industry [1, 2]. While clients benefit through instant access to their financial information at their convenience [3–6]. OB is a key component of ecommerce as increased adoption of OB has direct impact on the whole ecommerce model [7] as customers that might not have access to credit © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1128–1135, 2022. https://doi.org/10.1007/978-3-030-80119-9_76

Factors Influencing the Adoption of Online Banking in South Africa

1129

cards pay online merchants directly through online banking transfers. There has been recent studies that investigated online banking adoption factors in South Africa. Brown [8] found that faster internet access speed is important for internet banking as makes it easier and convenient to conduct banking. Internet speed and costs have since improved since but still online banking adoption is still low in South Africa.

2 Online Banking Online banking affords users the opportunity to access banking activities anywhere and anytime, with services such as pay bills, bank transfers, and view transactional history. Online banking has been widely received in developed economies. In the United States, 90% of the population have internet access [1] of which 67% used digital banking in 2016 – a rise from 51% the previous year [2]. In comparison with 2016 figures, 49% of the total internet audience of European Union countries accessed online banking [3], while the global average in 2012 was 28.7% of the internet audience [4]. Africa and the Middle East had 8.8% of internet users using online banking, with South Africa recording only 8.5% [4]. The online banking penetration rate for South Africa and the African region is far lower than the global average. Although this trend is expected given the rate of internet access in this region, still the A survey by FinMark Trust [5] indicated that 43% of internet users in South Africa considered branch banking to be safer than online banking, with 64% feeling vulnerable when doing financial transactions online. Moreover, just less than half (49%) reported that they believed making payments offline was more reliable than online. Online banking, specifically the use of technological services in banking, has seen banking institutions reduce back-office and front-desk costs, resulting in increased customer satisfaction [6]. Customer adoption of online banking has been on the increase worldwide, driven by the perceived benefits of convenience [7] and reduced costs [8]. Yet banks and online banking users have increasingly been targeted by cybercriminals, alerting banks to stepping up security systems. Users cite the perceived lack of security in online banking as an obstacle to adoption, even if their security fears are not based on actual experience. This can be attributed to the wide media coverage of some notable security and privacy breaches. Among the factors affecting online banking adoption, Poon [9] identified these ten determinants in Malaysia: convenience, accessibility, speed, feature availability, bank management and image, content, security, privacy, design, and fees and charges. The study also revealed that security, privacy, and convenience were significant factors that facilitated users’ acceptance of online banking. In addition, there are other external factors such as infrastructure, income level, and governmental policies that have an effect on online banking adoption [10–12].

3 Theoretical Framework UTAUT unifies constructs from eight prominent technology acceptance models [13, 14]. The theory identifies significant factors, variables, and relationships among items to provide a theoretical overview for the design of secure and usable online InfoSec

1130

M. Mujinga

applications. The synergy of these building blocks is important in delivering online banking that is secure and usable, giving the user an overall satisfactory experience. Therefore, it was critical to consider all aspects of the process, from user attitude and behaviour towards InfoSec systems to design approaches that are user-centred. This study integrates UTAUT and usable security for a comprehensive model of developing secure online banking environment. 3.1 Unified Theory of Acceptance and Use of Technology The main theoretical foundation of this study was UTAUT2, especially with regard to how security and usability aspects affected acceptance and continued use of technology, using online banking as a case study. UTAUT2 unifies eight disparate theories or models of user information technology acceptance and use, thereby streamlining the determinants of information technology adoption and continued use. Hence, UTAUT2 covers a wide range of constructs to model user interaction with information systems. The current version, UTAUT2, incorporates additional constructs that influence behavioural intention and use behaviour. In this study, use behaviour was measured by use frequency. The researcher investigated key constructs, namely, performance expectancy, effort expectancy, social influence, hedonic motivation, facilitating conditions, price value, and habit as direct or indirect determinants of acceptance, adoption, and continued use of online banking. Gender, age, experience, income, education, ethnicity, device, and use frequency were hypothesised to moderate the effect of the constructs on adoption and behavioural intention. 3.2 Development of Hypotheses This section outlines the hypotheses developed and tested through the survey data based on the underlying constructs of the theoretical framework. The constructs of the model are first explained briefly, and the main study hypotheses associated with the constructs are listed. The broader study hypothesis postulated that ‘socio-technical aspects of system design enhance the adoption and continued use of information security applications’. Therefore, the study hypothesised that constructs from UTAUT2, usability, and security had an effect on the adoption and intention to use online banking. The definitions of some of the UTAUT2 constructs were first provided in the original UTAUT and the eight unified models; these definitions are still relevant in the second version of the model. The effect of the constructs on adoption and continued use of online banking was moderated by a number of factors, which included gender, age, experience, education, employment, income, ethnicity, and device. The data analysis determined which of these moderating factors had a statistically significant impact on the constructs under investigation. Performance Expectancy (PE). PE is the degree to which an individual believes that using the system enhances activity performance [14]. PE was first defined in UTAUT and is a derivative of five constructs from the different models unified in UTAUT [CITE]. It is the strongest predictor of intention and remains significant at all points of measurement in both voluntary and mandatory settings [14].

Factors Influencing the Adoption of Online Banking in South Africa

1131

H1: Performance expectancy significantly affects individual behavioural intention to use online banking services. Effort Expectancy (EE). EE is the degree of ease associated with the use of the system and is a derivative of three constructs from the unified models, namely, perceived ease of use, complexity, and ease of use [14]. H2: Effort expectancy significantly affects individual behavioural intention to use online banking services. Social Influence (SI). SI is the degree to which an individual perceives that other people believe he or she should use an information system, as it determines behavioural intention, and is a derivative of subjective norms, social factors, and image [14]. H3: Social influence significantly affects individual behavioural intention to use online banking services. Facilitating Conditions (FC). FC refers to the degree an individual believes that organisational and technical infrastructure exists to support the use of the system [14]. The researcher investigated the effects on intention to use online banking of access to resources such as internet connection to, and assistance in, using the service. H4: Facilitating significantly affect individual behavioural intention to use online banking services. Hedonic Motivation (HM). HM is the enjoyment of using a system, which plays a significant part in influencing technology acceptance and continued use [15]. HM complements UTAUT’s strongest predictor, that is, performance expectancy, which emphasises usefulness [16]. H5: Hedonic motivation significantly affects individual behavioural intention to use online banking services. Price Value (PV). PV is the consumer’s awareness of the trade-off between the perceived benefits of the application and the monetary cost of using the application [17]. The cost of using a system is significant in a consumer context compared to an organisational context where employees do not bear such costs. Hence, the introduction of PV in UTAUT2 has a significant influence on the adoption and continued use of the system from an individual perspective [16]. H6: Price value has a positive and significant impact on behavioural intention to use online banking services. Habit (H). H is the extent to which people tend to perform behaviours spontaneously due to learning [18]. As an additional construct in UTAUT2, habit is related to experience in system use based on the duration from first use [16]. H7: Habit has a positive influence on behavioural intention to use online banking services.

1132

M. Mujinga

Behavioural Intention (BI). BI is an individual’s perceived likelihood to engage in a given behaviour [19]. Venkatesh et al. [14] theorise that behavioural intention has a significant positive influence on technology usage. In addition, the researchers theorised that user intention to use technology depended on the effort involved in using the technology. H8: Behavioural intention has a positive and significant impact on effort expectancy. H9: Behavioural intention significantly affects individual use behaviour (use frequency). 3.3 Moderating Factors UTAUT2 has three moderators, namely, age, gender, and experience. In addition to these, this study postulated six additional moderators: income, education, ethnicity, device, use frequency, and bank. Although data was collected on all these moderating factors, the analysis did not include or mention bank names for ethical reasons to avoid prejudice. The moderating factors were not mentioned explicitly in the hypotheses, but their effects on model constructs were investigated and reported separately and incorporated in the final structural model. Factors that could be recoded into dichotomous groups were analysed for statistical significance using independent samples t-tests.

4 Research Model The study is underpinned by the UTAUT2 model with an extension that includes usable security, which includes InfoSec and usability to explain online banking adoption and continued use. The theory is modified to include information security and usability principles that are postulated to have an impact on users’ intention to adopt and continue using online banking. The constructs are moderated by seven moderators; the original two from UTAUT2 (gender and age) and an additional five namely, education, income, ethnicity, device, and use frequency. Figure 1 shows the research model that includes UTAUT2 constructs, usable security constructs, and moderators that are hypothesised to have an effect on the adoption and continued use of online banking service.

Factors Influencing the Adoption of Online Banking in South Africa

1133

Performance expectancy

Effort expectancy

Security

Social influence

Behavioural intention

Facilitating conditions

Usability

Hedonic motivation Price value Habit Gender, Age, Education, Income, Ethnicity, Device, Use frequency Fig. 1. The research model

5 Research Methodology The study intends to use a quantitative research using survey approach. A quantitative questionnaire instrument will be used to collect data for the constructs in the research model. The questionnaire consisted of the following sections: demographic details of respondents, general online banking questions, online banking acceptance and use construct statements, usability constructs (including a 10-item system usability scale), security and privacy statements, and qualitative overall assessment questions. Current online banking users with any of the South African banks were eligible survey respondents. In terms of sample size, there are significantly different opinions on the appropriate sample size in factor analysis and other advanced statistical analyses, which give rise to varying guidelines and rules of thumb on sample size. Tabachnick and Fidell [20] rule of thumb suggests at least 300 responses, while [21] recommend at least 100 responses. A more detailed guideline is given by [22]: 100 (poor), 200 (fair), 300 (good), 500 (very good),

1134

M. Mujinga

and 1 000 or more (excellent). The study strive to collect more than 500 responses for proper inferential statistical analysis.

6 Data Analysis Data analysis consists of an instrument reliability test using Cronbach’s alpha. In this study, Cronbach’s alpha (α), also known as the alpha coefficient, is used to test the reliability of the overall instrument as well as that of all the constructs. Cronbach’s alpha as extracted from SPSS, will adhere to the 0.7 threshold and any constructs below the threshold are eliminated from the model. Structural equation modelling depicts relationships among constructs, which are dependent and independent variables, involved in the analysis. SEM consists of two models: the measurement model that shows how measured variables represent constructs and the structural model that shows how constructs are related to one another [21].

7 Future Work The paper presents a research model for usable online banking, the revised and final research model based on comprehensive inferential data analysis is beyond the scope of this paper. As part of future work, the research model will be presented as part of the data analysis process that includes structural equation modelling on constructs of the research model.

8 Conclusion The study presents a research model for a framework of secure and usable online banking service. The data collection will validate the research model based on the developed hypotheses. The inferential statistical analysis of latent variables and moderating factors will moderate the model constructs. The study postulates that for successful design of secure online banking, both the technical and social aspects need attention. The study further postulates security and usability affects as aspects that affect adoption. In addition to the conventional concepts identified by a unified model such as UTAUT2, the structural model will display the factors that had a significant impact on online banking adoption and continued use.

References 1. Internet World Stats: Internet World Stats. http://www.internetworldstats.com/stats.htm. Accessed 15 August 2017 2. Gonzalez-Garcia, J.: Online and mobile banking statistics. http://www.creditcards.com/cre dit-card-news/online-mobile-banking.php. Accessed 20 November 2017 3. Statista: Online banking penetration in selected European markets in 2014. https://www. statista.com/statistics/222286/online-banking-penetration-in-leading-european-countries/. Accessed 8 September 2017

Factors Influencing the Adoption of Online Banking in South Africa

1135

4. Statista: Global online banking penetration in April 2012, by region. https://www.statista.com/ statistics/233284/development-of-global-online-banking-penetration/ Accessed 6 November 2017 5. FinMark Trust: FinScope South Africa 2016. http://www.finmark.org.za/wp-content/uploads/ 2017/12/08Nov2016_FinScope_Consumer-survey-South-Africa-2016.pdf. Accessed 5 April 2018 6. Berger, A.N.: The economic effects of technological progress: evidence from the banking industry. J. Money Credit Bank. 35(2), 141–176 (2003) 7. Vatanasombut, B., Igbaria, M., Stylianou, A.C., Rodgers, W.: Information systems continuance intention of web-based applications customers: the case of online banking. Inform. Manage. 45(7), 419–428 (2008) 8. Hernando, I., Nieto, M.J.: Is the internet delivery channel changing banks’ performance? The case of Spanish banks. J. Bank. Finance 31(4), 1083–1099 (2007) 9. Poon, W.: Users’ adoption of e-banking services: the Malaysian perspective. J. Bus. Ind. Market. 23(1), 59–69 (2008) 10. Brown, I., Hoppe, R., Mugera, P., Newman, P., Stander, A.: The impact of national environment on the adoption of internet banking: comparing Singapore and South Africa. J. Global Inform. Manage. 12(2), 1–26 (2004) 11. Mujinga, M., Eloff, M.M., Kroeze, J.H.: A socio-technical approach to information security. In: Proceedings of the 23rd Americas Conference on Information Systems (AMCIS), Boston, MA, 10–12 August, 2017, pp. 1–10 12. Brown, I., Molla, A.: Determinants of internet and cell phone banking adoption in South Africa. J. Internet Bank. Commerce 10(1), 1–9 (2005) 13. Venkatesh, V., Davis, F.D.: A theoretical extension of the technology acceptance model: four longitudinal field studies. Manage. Sci. 46(2), 186–204 (2000) 14. Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User acceptance of information technology: toward a unified view. MIS Q. 27(3), 425–478 (2003) 15. Brown, S.A., Venkatesh, V.: A model of adoption of technology in the household: a baseline model test and extension incorporating household life cycle. MIS Q. 29(3), 399–426 (2005) 16. Venkatesh, V., Thong, J., Xu, X.: Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS Q. 36(1), 157–178 (2012) 17. Dodds, W.B., Monroe, K.B., Grewal, D.: Effects of price, brand, and store information on buyers’ product evaluations. J. Market. Res. 28(3), 307–319 (1991) 18. Limayem, M., Hirt, S.G., Cheung, C.M.K.: How habit limits the predictive power of intention: the case of information systems continuance. MIS Q. 31(4), 705–737 (2007) 19. Ajzen, I.: The theory of planned behavior. Organ. Behav. Hum. Decis. Process. 50(2), 179–211 (1991) 20. Tabachnick, B.G., Fidell, L.S.: Using Multivariate Statistics, 6th ed. Pearson Education, Essex (2014) 21. Hair, Jr., J.F., Black, W.C., Babin, B.J., Anderson, R.E.: Multivariate Data Analysis, 7th ed. Pearson Education, Essex (2014) 22. Comrey, A.L., Lee, H.B.: A First Course in Factor Analysis, 2nd edn. Psychology Press, New York, NY (1992)

Machine Learning Application in LAPIS Agile Software Development Process Tu˘grul Tekbulut, Nurdan Canbaz(B) , and Tu˘gba Öztürk Kaya Logo Yazılım, Kocaeli, Turkey {tugrul.tekbulut,nurdan.canbaz,tugba.ozturkkaya}@logo.com.tr

Abstract. It is necessary to carry out improvement works at every stage of the software processes to adapt to changes in the world of software. For this reason, primarily, the processes should be specific, automatable, and measurable; the opinions of the teams on improvement should be supported both for these processes and their own works. In this study, the contribution of the teams to the improvement works in the retrospective meeting, which is part of the improvement-oriented agile product development process, LAPIS (Logo Agile Process Improvement System) developed by LOGO, by being inspired by the lean manufacturing concept, has been supported by machine learning. The aim has been to expand the opportunities for the works of teams to improve and to be based on data. The effects of the study results on the software process measurement results have been examined, and the effects of the improvements on the company noticed by the teams have been evaluated. Keywords: Agile · Software Process · Continual Improvement · Machine Learning · Sentiment Analysis

1 Introduction In 1985, the concept of VUCA was introduced by two academics to describe the conditions of the post-cold war era [1]. The concept of VUCA, short for volatility, uncertain, complex, and ambiguous, has also been adopted in the business world and has been used in concepts such as changing business manners and leadership models. Today, this concept has started to gain a place in software development processes. Software development is, by definition, abstract and complex. Software development is, by definition, abstract and complex. However, customer needs are changing and the infrastructure of the products is insufficient with the rapid development of technology. The products that cannot adapt to rapid change dissatisfy customer, and unfortunately have to leave their place in the market to other products. For this reason, agile methodologies, which adapt changing customer needs in software development processes and build their strategy accordingly, have been adopted. LAPIS [2] is an improvement-oriented agile product development process, which is developed by Logo, by being inspired by the lean manufacturing concept, embraces constant change and aims to adapt to changes together with team. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1136–1148, 2022. https://doi.org/10.1007/978-3-030-80119-9_77

Machine Learning Application in LAPIS Agile Software

1137

LAPIS adopts the principle of embracing change. It is intended to use this change as a wind effect so that each product in Logo software can realize its strategy and achieve its goals. For this purpose, it has been ensured that the irreversible time is fixed and all stakeholders (customer, support, sales, etc.) act in a common way according to this time scale. Time is fixed at the beginning of the year according to the releases being scheduled and announced. The methodology, which avoids wastage by prioritizing efficiency, aims at the self-improvement of the process by enabling teams to adopt continuous improvement. Within the scope of the study, the retrospective studies done to stop inefficiency, which is one of the most important challenges of adapting to change, has been associated with the sentiment of the teams. The effects on the measurement results in software development processes have been examined.

2 Related Studies There are many studies on measuring software development. With the increasing popularity of data mining and machine learning, these techniques are also applied in software development processes. One of them is a sentiment analysis study conducted by Sin-ha and his colleagues according to the feedback of developers during the code reviews [2]. In the study, 28,466 projects on GitHub have been evaluated, and the codes written by the developers have been grouped as large, medium, and small. According to the analysis results, it has been concluded that negative comments are 10% more than positive comments. With the increasing popularity of agile software development, Grigori Melnik and Frank Maurer conducted a study measuring the satisfaction of software developers with machine learning. The study includes sentiment analyses of software developers engaged in agile software development and non-agile software development. According to the analysis, there is a correlation between agile software development and satisfaction, and it has been concluded that teams developing agile software are more satisfied [3]. Studies have been conducted on the tools used as well as people. The study in which agile project management tools are evaluated by Twitter comments is an example of this. In this study, comments on project management tools have been collected and analyzed using machine learning methods [4]. There are some studies using fuzzy logic to configure the agile methodologies [5]. Rinky Dwivedi and Daya Gupta mentioned that they use fuzzy logic methods in configurations that they use during the implementation of agile methodologies in their studies.

3 LAPIS (Logo Agile Process Improvement System) LAPIS is a product development model influenced by product development management models based on different methodologies and lean manufacturing concept [1]. The general flow of the process is given in Fig. 1.

1138

T. Tekbulut et al.

Fig. 1. LAPIS (Logo Agile Process Improvement System)

LAPIS, influenced by lean manufacturing concept for its main approach, is inspired by Scrum [6, 7], which is one of the agile methods. Unlike Scrum, software development teams are not cross-functional but specialized in different areas. The roles such as product owner, business analyst, system analyst, software developers, test specialists in teams work in accordance with their expertise in the LAPIS process. Especially in companies developing products, their areas of expertise are of capital importance. Sprint length is determined for each product at the beginning of the year according to the position of the product in the market and the need. A minimum 20% time of a sprint is allocated for test weeks, where integration tests are performed. Product Owner prioritizes the issues in Product Backlog before the sprint starts. In the sprint planning meeting, Product Owner and development team determine which prioritized issues will be included in the following release in accordance with the capacity of the development and test team. The agreed issues are announced as Sprint Backlog. These issues are automatically announced to business partners via the support system used in Logo. The development teams work on the bug-fix requests submitted by the test team and start developing the following version in the week of integration tests. Product Owner shares the completed release notes with support and project teams in the sprint review meeting within the same week. A retrospective meeting is held by the process manager in order to evaluate the things done in the sprint and to determine the things to be improved at the end of the sprint. Each step in the LAPIS process is tracked in Atlassian Jira [9] Project and Request Management System. In the schedule prepared at the beginning of the year, the important dates for each release (sprint start/end dates, test start/end dates) are defined and the flow and end times of the works are automatically assigned to the related users. These processes are tracked and controlled by process managers.

4 Retrospective Study In agile methodologies, the retrospective meeting is the meeting in which the sprint is evaluated by the team at the end of the sprint. Although there are many techniques, a method based on Sailboat Technique introduced in Luke Hohmann’s Innovation Game

Machine Learning Application in LAPIS Agile Software

1139

book is applied in Logo. The first goal is to identify the obstacles that the team may encounter in order to achieve the goal and seek solutions [10]. Before the sailboat technique is applied, the team analyzes itself. All team members evaluate the specified metrics in a range of 1–10, from bad to good in order for the team’s feedback to be converted into data on a common platform. As a result, team’s average velocity is determined for each metric and shown on the radar chart. A sample evaluation form and team radar are given in Fig. 2. All information is stored on the Confluence system of Atlassian [11].

Fig. 2. Team radar chart

The retrospective continues with the sailboat technique after the team radar in order to identify the things to be improved. Figure 3 is drawn on the board. Sailboat has an utterly metaphorical meaning and makes the team feel they are on the same ship. The common goal is for the team to reach the island and the effect of the wind is used to achieve the goal. In the first 5 min of the meeting, the issues (winds) that help the team achieve the sprint goals (island) are written by everyone. The goal here is to notice the good rather than focus on the negative. Then, the obstacles (anchors) making it difficult to achieve the sprint goal and slowing the team down are written for 5 min. Finally, things that are not considered obstacles in this sprint, but the risks that the team could encounter in the future, are written. Everyone evaluates the sprint in 15 min. Each issue is read in the remaining time. Both the things going well are shared with the team, and solutions are sought together for each step considered as obstacles and risks.

Fig. 3. Retrospective - sailboat technique

1140

T. Tekbulut et al.

5 Measurement and Improvement Method The LAPIS process is evaluated with a Goal-Question-Metric approach, which aims to associate high-level goals with low-level metrics with the appropriate purpose, based on the ISO 15939 [12] standard [13,14,16]. It is aimed to set achievable goals with the right metrics and to provide end-to-end traceability instead of collecting random data in the process and interpreting them in the evaluation. The measurement goals, questions asked to achieve goals, and metrics are given in Table 1 [8]. Table 1. Measurement goals and metrics Goal

Question

Metric

Customer orientation

Do teams release versions on time?

Release Compliance Rate Compliance to the release dates determined at the beginning of the year

Do teams deliver the promises Release Completion Rate made to customers on time? Completion rate of issues agreed for release (Number of Issues) Completion rate of issues agreed for release (Story Point)

Assuring quality in product development

Efficiency

What is the response time of the software teams to customer requests?

Issue Age of New Features Average issue age of new features in product backlog

What is the response time of the software teams to bug-fix requests?

Issue Age of Defects Average issue age of defects in product backlog

Can the errors be detected before customers?

Error Detection Rate Rate of errors detected by test team to all errors in the year

Do teams comply with the regular test weeks?

Compliance with Test Week Rate of the issues sent to test to planned issues when the error is not corrected in test week

Do teams comply with the capacity planning?

OTEQ (Overall Team Efficiency, Quality) [2] Compliance * Performance * Quality (continued)

Machine Learning Application in LAPIS Agile Software

1141

Table 1. (continued) Goal

Question

Metric

Can capacity be planned efficiently?

Effective Planning Rate Rate of the total number of completed issues to the number of planned issues in the release (excluding internal test errors)

Do teams comply with the strategic goals in the products?

Compliance with Strategic Goals Compliance rate with strategic goals set at the beginning of the year

Cards have been created for each metric, the roles, time, and place to track these metrics have been determined. Here, two approaches have been identified to improve the measured values. The first of these approaches are the retrospective meetings mentioned in Sect. 3. The other approach is the evaluation of metrics with team managers. The metrics shown in Fig. 4 are checked by managers in the product-based periods, and the actions to be taken are determined. The goal is to increase the values representing each point of the radar charts to 100%.

Fig. 4. Product-planning radar chart

The actions taken in the retrospective meetings held with the teams have been evaluated by goals and the effects of the studies on the measurement results have been examined. The measurement table is given in Table 2 and a sample evaluation result is given in Table 3.

1142

T. Tekbulut et al. Table 2. LAPIS measurement table Category Range

Badge

A

90–100 LAPIS Guru

B

80–89

LAPIS Expert

C

60–79

LAPIS Master

D

30–59

LAPIS Explorer

E

0–29

LAPIS Starter

Table 3. LAPIS sprint based evaluation Sprint Point Badge R1

66.75 LAPIS Master

R2

90.93 LAPIS Guru

R3

87.93 LAPIS Expert

R4

87.21 LAPIS Expert

R5

89.40 LAPIS Expert

6 Machine Learning Sentiment Analysis According to the Oxford dictionary, sentiment analysis is the process of defining and classifying opinions expressed in a piece of text to determine whether an author’s attitude towards a particular subject is positive, negative or neutral [15]. Methods used for sentiment analysis: • Manual Processing It is a method in which people evaluate and score the sentences manually. It is the method with the highest accuracy. However, it may not give 100% accurate results. It has become very difficult to use this method, especially in recent years, when the number of data has increased very rapidly. • Keyword Processing Keyword processing algorithms rate a word as positive or negative, then give the article an overall percentage score. For example: Positive words, such as great, like, love, or negative words, such as terrible, dislike. • Natural Language Processing (NLP: Text Analytics, Data Mining, And Machine Learning Linguistics) Algorithms, mostly prepared in Python, can also use grammatical capabilities to make the sentences positive/negative/neutral.

Machine Learning Application in LAPIS Agile Software

1143

The natural language processing method, one of the sentiment analysis methods, has been used in this study.

7 Sentiment Analysis of Retrospective Data In the study, the sentiment analysis of teams has been performed with the R language and R Studio application by using data written by team members in retrospective meetings which are the improvement ritual of the LAPIS process. The aim here is to analyze the sentiments of the members from their feedback and to track the effect of changing negative sentiment into positive on the measurement results. In accordance with this purpose, the study has been progressed through the data mining stages specified in Fig. 5. The data is adapted for analysis at the data preparation stage.

Fig. 5. Work life cycle

The data content is given in Table 4. The dataset includes the positive and negative sentiments of team members in the sprint, which is a period in the product development cycle. A sample of retrospective raw data is shown in Fig. 6. Table 4. Data content The issues that help team to move forward (Wind) The impediments that slow down the team (Anchor) The potential obstacles to affect product development in the long term (Iceberg)

1144

T. Tekbulut et al.

Fig. 6. Retrospective data

7.1 Data Preprocessing The feedback of the team members has been sorted by using natural language processing techniques. The steps are shown in Fig. 7.

Fig. 7. Data preprocessing steps

7.2 Modeling The sentiment analysis techniques have been used as modeling methods. Semantic inference is intended by using a data-based approach. Two methods are commonly used to calculate the term weight. The first is Count Vectorizer, and the second is the TF-IDF approach. The count vectorizer determines how frequently the relevant term is used in the field. Both term frequency (TF) and inverse document frequency (IDF) are performed in the TF-IDF approach. The limit for the most frequently used words is 50, and some of the words found are [‘team’, ‘product’, ‘support’, ‘work’, ‘time’, ‘description’]. Finally, the word cloud, which shows the most frequently used words in retrospective data, is given in Fig. 8. The feedback of the teams in the retrospective meeting is shown in the matrix as in Fig. 9 after sorting the data. Here the lines represent sentences in the retrospective meetings, while the columns represent eight sentiments. The cells show proximity to each sentiment. A bar chart has been used to see the sentiment distribution. The sample chart is shown in Fig. 10. It has been observed that 33% of the team have confidence in, while 22% have expectations.

Machine Learning Application in LAPIS Agile Software

Fig. 8. Word cloud

Fig. 9. Vector data

Fig. 10. Sentiment distribution

1145

1146

T. Tekbulut et al.

7.3 Evaluation In this study, it has been observed how retrospective studies, one of the LAPIS rituals, affect the measurement results. The first product where the study will be conducted has been determined according to the feedback and foresight of the process manager. A retrospective meeting has been held with the team, which the process manager observes the expectations of product management and some communication problems. In the sentiment analysis performed with the data collected as a result of this retrospective meeting, it has been observed that anticipation is dominant in the team in parallel with the observation of the process manager. The results of the first sentiment analysis are given in Fig. 11. When the results are analyzed with the word cloud given in Fig. 7, the product, time constraint, support and communication stand out in the word cloud. One of the team’s challenges is the product management team working in a different location than the software and test teams. As an improvement action, the product management team also attends the following retrospective meeting in order to eliminate this challenge. It has been observed that sentiments of fear and sadness increases in the sentiment analysis performed according to the feedback given by the teams in the retrospective meeting held with the product management team, software team, and test team.

Fig. 11. Software team sentiment analysis result

The results of a retrospective meeting held with the product management team, software team, and test team are shown in Fig. 12. In retrospective meetings, the feedback of team members is written first, and all feedback is read and all participants meet on a common ground for each issue open to improvement. Studies have been conducted for improvement actions determined in the retrospective meeting held with the product management team, software team, and test team for a sprint time; and study results have been followed. The results are shown in Fig. 13.

Machine Learning Application in LAPIS Agile Software

1147

Fig. 12. Sentiment analysis results including product management

As a result of the sentiment analysis, a 22% decrease in feelings of fear, sadness, and expectation has been observed. Due to the decisions taken after the retrospective meetings made with the product management teams, a 75% decrease in the number of “anchors” reported by the team in the following retrospective meetings has been observed.

Fig. 13. Sentiment analysis result indicating improvement at the end of a sprint

1148

T. Tekbulut et al.

8 Results and Future Studies The effects of the study have been tracked for six sprints with metrics measured by LAPIS. Accordingly, based on ISO 15939 [12], it has been observed that the team is firstly promoted from “Lapis Master” to “Lapis Expert”, and then to “Lapis Guru” at the end of the third sprint in the planning group determined according to the gamification logic, in the measurement model created with the Goal-Question-Metric approach. There is a 14% increase in the Compliance to Planning value and a 12% increase in the production speed of the team in the relevant group. The biggest contribution of the study to the process has been to show the improvement points of the process more transparently as a result of the support of the evaluation made by the process manager with data. In the following studies, software internal quality metrics will be evaluated together with the feedback given in the retrospective meeting, and different techniques of machine learning will be applied.

References 1. Warren Bennis, T.: Leaders: The Strategies for Taking Charge Hardcover, May 1, 1985 2. Tekbulut, T., ˙Inal, A., Do˘ganay, B.: LAPIS – (LOGO Agile Process Improvement System, UYMS (2015) 3. Melnik, G., Maurer, F.: Comparative Analysis of Job Satisfaction in Agile and Non-agile Software Development Teams. In: Abrahamsson, P., Marchesi, M., Succi, G. (eds.) XP 2006. LNCS, vol. 4044, pp. 32–42. Springer, Heidelberg (2006). https://doi.org/10.1007/117741 29_4 4. Gupta, D.: Applying machine learning for configuring agile methods. Int. J. Softw. Eng. Appl. 9, 29–40 (2015) 5. Dwived, R.: The agile method engineering: applying fuzzy logic for evaluating and configuring agile methods in practice. Int. J. Comput. Aided Eng. Technol. 9, 408–419 (2017) 6. Schwaber, K., Sutherland, J.: The Scrum Guide (2014) 7. Beck, et al.: Agile Manifesto. http://www.agilemanifesto.org/ (2001) 8. Tekbulut, T., Canbaz, N., Kaya, T.O.: LAPIS Çevik Yazılım Süreci Ölçme Metodu (2018) 9. JIRA: The Leading Product for Project and Issue Tracking. https://www.atlassian.com/sof tware/jira/ (2002) 10. Derby, E.: Agile Retrospectives: Making Good Teams Great Paperback, August 5, 2006 11. CONFLUENCE: The leading product for Document Collaboration. https://www.atlassian. com/software/confluence/ (2003) 12. Standardization, I.O.f. ISO/IEC 15939:2002 Software engineering – Software measurement process. https://www.iso.org/standard/29572.html (2002) 13. Nicolette, D.: Software Development Metrics, 1st edition (2015) 14. Van Solingen, R., et al.: Goal question metric (gqm) approach. In: Encyclopedia of Software Engineering, John Wiley & Sons, Inc. (2002) 15. Oxford English Dictionary: eISBN: 9780191727665, Oxford University Press (2010) 16. Davis, C.W.H.: Agile Metrics in Action: Measuring and Enhancing the Performance of Agile Teams, 1st edition (2015)

Transforming HR and Improving Talent Profiling with Qualitative Analysis Digitalization on Candidates for Career and Team Development Efforts Antonis Vatousios1 and Ari Happonen2(B) 1 Anveto Group OU, Sepapaja 6, 15551 Tallinn, Estonia 2 School of Engineering Science, LUT University, 53850 Lappeenranta, Finland

[email protected]

Abstract. The digital transformation of HR is the new normal in the enterprise business world, in quantitative analysis side, but the decision making will remain qualitative for the long unforeseen future. We present digital qualitative methods for talent profiling in order to understand each team member’s mindset, by utilizing brainstorming type questionnaire data. As a result of the study, a stand-alone solution is being developed which improves HR operations and talent profiling for individuals and organizations. Furthermore, this abductive approach indicates that this open-ended conceptual framework returns a promising qualitative analysis. The research work addresses the issues of the subjective nature of human resources with an open-ended approach which sharpens our decision-making without excluding it. Keywords: HR · Human resources · Talent management · Profiling · Mindset · Digitalization · Digital transformation · Analysis · Brainstorming questionnaire

1 Introduction Digitalization is the new normal in today’s world. The world is changing fast, as technology is getting constantly smaller and sensors, plus computational capacity, in the edge of networks, is taking huge leaps forward. In a similar way, global digital transformation is pushing traditional business areas asset-based companies towards knowledge companies [33–35, 39]. Additionally, artificial intelligence solutions and sustainability-based practices [28–30, 59] are helping companies to transform their business operations through use of digital design process tools [12, 13]. Companies should be able to transform their HR practices to today’s standards, by embracing the potions that digitalization can offer them as digital transformation of HR [1] is the new normal in the enterprise business world. But in the process, these new tools for HR should also help them e.g. find new employees with cultural fit personalities [32], support growth and innovation [33, 38], etc. Basically, the key is to not digitalize just for the fun of it, but instead, the goal should be efficiency with more precise and knowledge-revealing tools. In this © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1149–1166, 2022. https://doi.org/10.1007/978-3-030-80119-9_78

1150

A. Vatousios and A. Happonen

context, this research provides answers to the issues which arise by qualitative analyses in human capital approach. By employing statistical techniques, rich visualizations, and brainstorming questionnaires, we aim to address the subjectivity which is inherent in qualitative approaches and decision making. In the world of recruitment and human resource management, nowadays, it is important to be able to combine traditional recruitment processes with modern team analysis [57] and be able to apply approaches where possible new employees under consideration could show their skills in practice, like in university – industry collaborative hackathon events [41, 61–64]. Furthermore, by employing mindset mapping tools, most suitable candidates get hired, enhancing RDI and development teams’ productivity. The goal is to reveal a new, novel way which adds value in HR people analytics work [43] by employing common data science and survey tools, but with an innovative approach [6]. Initially, two basic research questions are set: (1) Does this brainstorming type questionnaire, combined with demonstrated toolset, work effectively as a mindset mapping and visualization tool to help HR being more precise in their recruitment process? (2) What values the comparison of respondents’ mindsets could provide to HR operations? The main hypothesis is that brainstorming type standardized questionnaires can be used as a map to unfold a candidate’s mindset. The critic about standardized tests, questionnaires and surveys is quite harsh due to its closed structure, because they do not provide the option of a spontaneous answer [10]. However, they can be a powerful tool for research and development through an evolutionary approach [2]. Educators claim that standardized tests and questionnaires do not cultivate creativity, and to some extend that is true [10]. However, talent assessment can be approached by employing brainstorming questionnaires on a variety of topics, when respondents choose the most representative answer which they mostly agree with. Therefore, if the variety of topics is extensive, there is both broad and narrow focus, and the options are not biased which is an inherent characteristic of questionnaires, respondents will unfold their values, inclinations, and their mindset. In a word, the most successful approach for talent profiling is an abductive approach with open-ended clusters as topics. The open-ended approach which sharpens our decision-making without excluding it [5, 8, 9] helps to address the issues of the subjective nature of human resources. Also, in this research case, the abductive approach offers opportunity to indicate the value of open-ended conceptual framework [3, 4, 7]. Generally, in social sciences, it can be said, that it is worth trying to be approximately right and not precisely wrong. Consequently, a map is being built with directions and orientations but not clear-cut borders. The key-point for respondents is to choose freely or not even choose at all.

2 Creating the Questionnaire Set Technically, this questionnaire is created by using the Shinyforms Package of R opensource tools. However, the purpose of such a questionnaire is the respondent’s profiling.

Transforming HR and Improving Talent Profiling

1151

Therefore, its content aims to map the respondent’s mindset. As basis for creating the questionnaire set, the following structural base components were used: • • • •

Gardner’s multiple intelligence theory [12] Management theory [13] Business practitioners to construct a questionnaire for talent profiling Big Five personality traits: OCEAN (Openness, Conscientiouness, Extraversion, Agreeableness, Neurotism) [11, 14].

OCEAN is used by HR practitioners to evaluate how candidates and marketers understand the potential customers of their products. Additionally, this questionnaire is focused on brainstorming, meaning that the research did aim to unfold the respondent’s mindset but not to test them for their knowledge or their capacity. Moreover, this questionnaire contains statements and not necessarily questions, since the goal is to visualize the respondent’s mindset creating his talent fingerprint. Hence, mind mapping was employed through brainstorming by searching mainly for semantics at the respondent’s answers. By utilizing of the previously mentioned content sources, a brainstorming type questionnaire with seven sections (Temperament, Individuality, Interconnection, Culture, Communication, Character and Skills [11, 44, 45]) was created. In the following, these seven sections are presented one by one. Temperament is a term which defines the personal traits of an individual, as well as their values and attitude. However, it is an economic term, too. It is important, both in business and in life, to know yourself - where you stand and where you want to end up. In practice, temperament, by and large, defines the expertise, the profession, and the field in which an individual pursues a career [14]. Individuality is closely connected with Character and Temperament, but it expresses the uniqueness of an individual which makes them different from the group. Therefore, individuality is what makes a person be an independent thinker and doer, even though they might have similar traits and behaviors to other people [15, 16, 21]. Interaction is relevant to Communication, but expression is not at its core. For that reason, the main aspect is how an individual or a team interacts with the world. Interaction is critical in innovation, research, product development, investing and any other activity where micro and macro are mutually related. In a word, perception matters in action and interaction [17]. ‘Culture eats strategy for breakfast’, a famous quote, was credited to management consultant Peter Drucker [50]. Culture is not only the background, experience, and orientation - it is identity too. People behave according to their beliefs and thoughts as much as to their environment [22, 23]. However, some environments allow talents to flourish and others do not [18]. Communication is at the heart of team building. Bringing people together who speak different ‘languages’ is the way to succeed in terms of growth, productivity and innovation [10, 24]. Consequently, great achievements in business and science are not coming by mixing up the ingredients [27]. It is not a simple sum that generates great outcomes. Great products are the result of dynamic relationships. In terms of Character, ‘Hire Character, train Skill’ is a widely known quote credited to former CEO of Porsche, Peter Schutz. Meaning that, quite often, character and cultural

1152

A. Vatousios and A. Happonen

fit are more important than skills, although it is more difficult to get it right due to its subjective nature. The qualities of an individual belong to a long-term strategy which is what really matters if it is combined with competence, strong work ethic and willingness to learn. As it is said, skills can be taught or developed, though it is easier said than done. On the other hand, character cannot be changed as regards its core principles, values, and life view. As the author Zig Ziglar succinctly put it, “You don’t build a business, you build people, and then people build the business”. Skills is a term which is used to describe technical knowledge of different forms and levels, but skills go far beyond that [19]. Day by day, skills become a behavioral aspect of our own personality. That is why talent is a reference not only for work, business, and social life but, also, for sharing ideas, beliefs, and values [49, 50]. Nevertheless, talent is as much common as scarce. It is said among techies and geeks that good engineers can be ten to hundred times better than mediocre ones. However, the question will be ‘Is it teamwork or individuality which matters most in organizations?’ [31, 46] Of course, the answer depends on the context, the nature of skills and the leadership. In each of the aforementioned seven sections of the questionnaire, relevant statements or phrases are used with a few available options. The respondent can choose up to 25 options in total, for each of these sections contains roughly 50 options. Each respondent can use their available options as they wish. The answers are stored (commonly in.csv files) and analyzed per section and collectively. A research milestone materializes with the terms they choose consistently, in a manner which shows preference and familiarity. Subsequently, the sections show the keywords which describe the respondent best, are of high-frequency and, also, are relevant, forming a topic or a cluster.

3 Text Analysis At first, a dataset of single words is created, which consists of the respondent’s answers without numbers, punctuation, grammatical prepositions and a collection of stop-words which provide no value to data analysis of natural language data.

The TM package of R-Project processes the data and remove all the words which can skew the final results. A sample of the code follows:

Transforming HR and Improving Talent Profiling

1153

The final vector output contains all words which represent the candidate’s mindset for each section of the questionnaire. Then R tools for wordclouds, pie charts and heatmaps are used to analyze and visualize each dataset. Wordcloud analysis is employed for the entire dataset without clustering and then a clustered wordcloud analyzes and calculates the natural language data per section.

Table 1 presents an example of a clustered wordcloud output for R. Results show the frequency of each single word, i.e. how many times it is repeated, for each section

1154

A. Vatousios and A. Happonen

of the questionnaire separately. With this dataset, the clustered wordcloud is generated, meaning that each section at the wordcloud is plotted separately. On the other hand, the ‘total wordcloud’ is plotted considering the frequency of each single word for all sections cumulatively. Table 1. Structured sub-dataset Terms

Skills Character Communication Interaction Individuality Temperament Culture

Always

0

1

1

0

0

0

0

Business

3

1

0

0

1

0

3

Create

1

0

1

0

0

0

0

Creative

1

0

0

0

0

0

1

Depends

0

0

1

1

1

0

1

Development 0

0

1

0

0

0

1

Different

0

0

0

2

1

0

0

Environment 0

0

1

0

0

0

1

Field

0

0

0

0

1

0

1

Human

1

0

0

0

0

0

1

Important

1

1

0

0

1

1

0

Individual

0

0

0

1

0

0

1

Intelligence

0

2

2

0

0

2

0

Knowledge

1

0

0

0

0

1

0

Life

1

0

0

1

1

1

1

Love

0

1

0

0

0

1

0

Make

0

0

0

1

1

0

0

Matters

0

1

0

0

0

0

1

Natural

1

0

0

1

1

0

0

Needs

1

0

0

0

0

1

0

Numbers

0

0

2

1

0

0

0

Problem

1

0

1

1

0

0

0

Product

0

0

2

0

0

0

1

Right

2

1

0

0

1

0

1

Sell

0

0

1

0

0

0

1

Simple

0

0

1

1

0

0

0

Skills

2

0

0

0

1

0

0

Strong

0

0

0

1

0

0

1

Values

0

1

0

0

1

0

0

Without

0

0

1

1

1

0

0

Transforming HR and Improving Talent Profiling

1155

4 Data Analysis and Visualizations R and Python are employed to analyze questionnaire’s data using popular IDEs, apps, and packages as R Studio, R Shiny, Shinyproxy and their Server versions. The following visualizations are about two respondents, A and B, as a case study example for a Web Development team context. For this case, authors have applied team members-based team development analysis methodology [60], where in this particular case, the respondent A did belong to a creative subgroup while respondent B was in a developers’ subgroup. Like explained at the previous section, a dataset consists of the respondent’s answers without numbers, punctuation, grammatical prepositions, and a collection of stop-words is analyzed and visualized for interpretation, based on the frequency or the weight of each term. The type of visualization which has been chosen to depict the text analysis is the wordcloud. The word clouds present the significant ones words with enhanced fonts and closer to the center of the vacuolation. While, moving gradually towards the periphery of the cloud, the remaining less significant terms appear in smaller sizes. It is named total wordcloud because it uses no clustering. The left side of Fig. 1 depicts the Total WordCloud with word weighing using TFIDF Technique, which is a common statistical tool for normalization in information retrieval and text mining. Then the right side of Fig. 1 presents a clustered wordcloud of the same dataset, for respondent A, using word weights. Meaning that the wordcloud is determined separately for each section. Therefore, the clustered wordcloud is being created by splitting the data in seven datasets which represent the respondent’s answers at the seven sections of the questionnaire. The coloring shows the frontiers of Temperament, Individuality, Interaction, Communication, Culture, Character and Skills.

Fig. 1. Total wordcloud (left) & respondent a clustered wordcloud (right)

The bar plot at Fig. 2 shows the total wordcloud terms with the biggest weights, which are: Audience, Equally, People, Skills, Different, Science, Believe, Digital,

1156

A. Vatousios and A. Happonen

and Passionate. Therefore, respondent A presents a people-centric profile with interests in science and skills development. Moreover, Fig. 2 shows the bar plot of the clustered wordcloud terms with biggest weights: Handle, Different, Reasoning, Mindset, Digital, Art, Communication, Intelligence, Incentives, and Emotional.

Clustered WordCloud

audience

equally

people

handle

skills

different

science

believe

reasoning

mindset

digital

art

communica on

passionate

intelligence

incen ves

7 6 5 4 3 2 1 0 emo onal

Word Weights

Terms with bigger Weights

Total WordCloud

Fig. 2. Word weights for the total and the clustered wordcloud of respondent A

Figure 2 provides the respondents talent fingerprint for HR, which can be a critical factor in the hiring process and/or team development. In the example, the respondent presents a clear inclination in people-centric terms. On another hand, human mindset and talent are not set in stone, hence managers should approach them as sign of potential. In practice, by comparing these two types of wordclouds, the total and clustered one, we come up with two interpretations of the same dataset. The total wordcloud is more people-centric because of the most weighted words, which are the ones that the respondent preferred the most, focus on Audience, People, Skills, Belief, Passion, Life, Art, and Emotion. While the clustered wordcloud presents, also, a managerial and marketoriented side using terms as Handle, Digital, Reasoning, Mindset, Communication, Intelligence, and Incentives. Therefore, the big picture of their talent fingerprint is people-centric, but with a closer look, there is business thinking, too. In conclusion, as the most preferred terms show, the total wordcloud of respondent B (at the left side) depicts a strong business and product development profile. This is an important piece of information for HR experts and talent hunters [56]. People’s inclinations are the fields and directions in which they would be more productive, cooperative, and creative, both as human beings and professionals. Figure 3 also presents the clustered wordcloud of respondent B (at the right side) with weighting. Consequently, clustering brings at the surface new aspects of respondent’s profile. At this case, respondent B emerges with an engineering side.

Transforming HR and Improving Talent Profiling

1157

Fig. 3. Total wordcloud (left) & respondent B clustered wordcloud (right)

Figure 4 shows the respective bar plot with the most weighted words of the respondent’s dataset. By these means, visualization of their inclinations is important in orientation, development, coaching and management. Their most weighted words from the total wordcloud, as Fig. 4 shows, are: Product, Life, Skills, People, Market, Intelligence, Different, Science, Integrity and Believe. Additionally, their most weighted words from the clustered wordcloud are: Solving, Communication, Skills, Different, Acumen, Language, Programming, Competition and Data.

Clustered WordCloud

product

solving

communica on

life

skills

people

market

intelligence

different

science

integrity

acumen

language

programming

on compe

believe

12 10 8 6 4 2 0 data

Word Weights

Terms with bigger Weights

Total WordCloud

Fig. 4. Word weights for the total and the clustered wordcloud of respondent B

1158

A. Vatousios and A. Happonen

In addition to the total wordcloud, the clustered one depicts that respondent B presents a business profile but with an engineering flair, too. In a word, wordclouds with proper text analysis based on weights provide a pictorial way to present a general view of talent profiling and mindset mapping. The outcome is qualitative, and it depends largely on the questionnaire and the HR specialist.

5 Population in Group Popularity is a measure which represents the number of respondent’s answers which are identical with other participants. All the analysis and visualizations of popularity have been implemented with R opensource tools. It is worthy to note that on popularity analysis there is no text analysis or natural language processing because it focuses solely on identical answers. Figure 5 shows such a visualization for the candidate 14. Their responses are visualized and compared, for each section of the questionnaire, to the respective ones of a group of 14 participants. The answers which are unpopular within a group imply that only a few other participants have made the same choices at this section of the questionnaire. LP stands for answers of low popularity, MP for medium popularity and HP for highly popular answers. LP answers are those which are not preferred by more than 20% of the group, MP answers are those between 20% and 50% and HP answers are those which are chosen by more than 50% of the group participants. Just a look is enough to conclude that the candidate’s answers are quite popular in the group for the Communication and Culture sections. Also, there seem to be polarized answers in the Interaction section, always considering the choices of the group. Consequently, at this type of analysis, we visualize a candidate’s popularity per section in a specific group of people.

Fig. 5. Popularity and polarization

Figure 6 visualizes the answers of all respondents of the group without splitting them into sections. This visualization shows the most popular candidates among the group. Popularity, again, is the number of the identical answers in total i.e. how many answers were identical among participants at the Brainstorming Type Questionnaire.

Transforming HR and Improving Talent Profiling

1159

Candidates 5 and 8 are among the most popular respondents in this group, while candidate 1 was most unpopular which may imply that he is just an outlier. Certainly, candidate 1 is a misfit to that group because picture 13 shows that his Low Popularity (LP) area is almost 50%, while his LP and MP areas combined are almost 80%, implying that his answers are quite unpopular in the group. Furthermore, as the following chapter shows, that claim is also verified with the heatmap analysis, at Fig. 6, because the column of candidate 1 shows low similarity score with all the other candidates. Almost all his similarity pairs are colored by deep blue or green. However, as it has been already noted, a misfit team member may be the one it is needed for team growth and development. Therefore, outliers must be analyzed more so as to reach solid conclusions.

Fig. 6. Popularity in the group

6 Similarity in a Group Similarity is different than Popularity because in this case it is expressed by the Power Predictive Score which is a statistical method. Therefore, this analysis does not focus on identical answers but on similar patterns. Consequently, Similarity includes text analysis but without frequencies or weights. The input is the term patterns for a single candidate which is compared to all other members’ term patterns of the group. Code Section 4 presents a sample of the R code which has been used for the implementation of the following heatmap graph.

1160

A. Vatousios and A. Happonen

The results.txt file is the output of ppscore, a Python implementation of the Power Predictive Score (PPS) [58]. The PPS is asymmetric and data agnostic that may detect linear and/or non-linear relationships between two columns. The PPS ranges from 0, implying no predictive power, to 1 which is perfect predictive power. It is important to note that it may be used as an alternative to the correlation (matrix). Practically, with PPS, column x predicts column y and vice versa. • A score of 0 means that x cannot predict y better than a naïve model. • A score of 1 means that x can perfectly predict y. • A score between 0 and 1 declares the predictive power the given model achieved compared to the baseline model. At this case, PPS employs a decision tree which is a learning algorithm with the following properties: • • • • •

It can detect any non-linear bivariate relationship Satisfying predictive power in a wide variety of use cases It is a robust model which can handle with outliers and overfitting. It can be used for classification and regression. It is faster compared to other algorithms.

Transforming HR and Improving Talent Profiling

1161

Moreover, at this project Correlation cannot be used because it works only for numerical data with linear relationships. However, in this word analysis there are columns with categorical data. Additionally, the PPS detects every relationship that the correlation does. Thus, PPS matrix can detect and analyze linear or nonlinear patterns in categorical data. Lastly, the predictive power score can be used to discover good predictors at target column which is useful information in industrial and commercial applications. Figure 7 depicts a heatmap with 14 individuals as a group. PPS (Power Predictive Score) [45, 46] is being used to match individuals based on their responses at the questionnaire. The color scale from Blue to Yellow is translated as low to high Similarity, respectively, for each pair of candidates. Rows represent features and Columns the predictors, meaning that high scores imply accurate predictions by using features as a reference and vice versa. But as it has been already noted the outcome is not necessarily symmetrical. For example, the pair (2,11) presents high similarity while the pair (3,7) low similarity. Implying that the first pair of candidates, number 2 at x axis as feature, and number 11 at y axis as predictor, have similar mindset. Moreover, the candidates 1 and 3 are outliers in that group, meaning that they do not match with other members in this group.

Fig. 7. Predictive score heatmap

In fact, an important fact to keep in mind in Talent Management [20] is the reality that the indicators have multiple interpretations, but they can still be more than a useful map for Team Leaders, Managers and Directors [53, 57]. People management depends on human judgement and this is not a cliché [54, 55]. However, data science tools provide critical information and neutrality on talent profiling and assessment. Additionally, comparison of profiles returns information for team development. In teams, commonly, there are similar, complementary, and irrelevant profiles. Furthermore, time to time, outliers in teams and companies make a difference.

1162

A. Vatousios and A. Happonen

Being an outlier means nothing for one’s capacity as an individual, but it’s sign that he may play a different role in the group or not, even, participate at all, like an intrapreneurs with special role in organizations. A role where they provide a special input for team growth and development. Also, an outlier in ‘team sense’ might be a highly talented evaluator or progress monitor for the team as he takes different point of views into discussion table, what the team itself in general does. Lastly, non-similarity can be interpreted it as complementarity in some cases, which is a basic characteristic for exceptional teamwork [42, 43].

7 Discussion A profiling tool for talent assessment [25, 51] and management should be approached as utility [47], since performance, which is what really matters, can be improved only by employing the right metrics. Implying that it must add value, in real terms, on management, administration, teams and, above all, individuals [30]. Qualitative analysis tools can be used as a guide map, but they cannot replace human judgement because emotional intelligence, which is expressed through vision and creativity, is a human trait. We use IT tools to gather and analyze information but not to make executive decisions. Practically, individual assessment is data analysis based on the responses of each single candidate. This research has shown that with proper interpretation of respondents’ answers, one can unfold their mindset and values through the brainstorming type questionnaire. By its nature, this approach is not precise but abductive which is what really matters in profiling and orientation. Analytics frame respondents’ preferences, as well as categorize and match them with other respondents. The outcome is the orientation of individuals and/or teams which has been implemented by word weighting of their options and their similarity score in a group. Moreover, group analysis is also important, especially, in team development [26, 48]. Most companies need HR insights for teamwork, evaluation, coaching and personal development [36, 37, 40]. Grouping is all about personality match, similarity, and complementarity [41, 42]. Despite the technical importance of data analysis, talent profiling must not be downgraded to a technicality [34, 35]. Finally, in regard to team development, research must be extended beyond similarity so as to bring more solid results. As it is noted, team members’ profiles can be, also, complementary which is of high importance in business and product development because teams need people with different strengths and mindsets. However, complementarity is a more challenging issue than similarity from a research point of view.

8 Conclusions and Future Research Recommendations This research was focused on individual and group analysis creating a conceptual framework [7] employing a method which visualizes an individual’s mindset using a brainstorming questionnaire and text analysis. The outcome has shown that wordcloud visualizations depict differences in mindsets and career orientation. Moreover, Popularity is a single measure which presents how popular a member is in a group. On the other side, Similarity score unveils the similar mindsets between candidates and/or members

Transforming HR and Improving Talent Profiling

1163

of a group. Should the Power Predictive Score be high, with 1 representing the perfect predictive power, it would imply that there is a pattern at the datasets of two members. However, as it has been noted at the discussion section, Complementarity is also important for group analysis, especially, for team development. Therefore, future research must include similarity and complementarity so as to provide more information for group members. Lastly, the coarse and fine focus is, also, on a future research framework in Qualitative Analysis and HR product development [55]. Certainly, the right balance between course and fine focus returns the best outcome on (TP) True Positives and (FN) False Negatives [52]. However, as it is known, empirically human resources [20] is a long-term game which needs time and consistency to be tested and validated. It is a case study of longterm investment which pays off once the management gets the right balance [49, 50, 56]. Additionally, the relevance of the most frequent or weighted words at wordcloud visualizations is a point of interest. At large or specific datasets, Relevance based clustering returns information for respondents and candidates. HR digital transformation, known as HR 4.0, needs quality and quantity to improve hiring process and productivity. At the moment, this talent assessment tool can be functional for small groups. Therefore, future research could involve the development of KPIs which can rank candidates providing HR specialists with specific metrics and not only visualizations. For instance, a similarity analysis tool could be developed based on word weights for group analysis and profiling. Hence, it would be easier to rank candidates employing specific indicators. Important part of future research is the automation of the process. Meaning that the HR departments would have at their disposal, after the analysis, a shortlist which may be used for interview calls. This automation should be depended on certain KPIs to match candidates with certain positions’ profiles or, even, evaluate unsolicited applications. Those KPIs should focus on finding the best fit for teams and positions. On other side, building a list with candidates for particular roles is, also, important for growing companies. Consequently, this talent profiling tool could contribute to the trend of HR 4.0 with further development towards full digitalization of HR processes. For example, should a candidate present a people-centric profile and, at the same time, appear to have a quite similar profile with a company’s salesmen group, it would imply that they will be suitable for such a role. Hence, building two KPIs on talent profiling and similarity would provide, instantly, HRs with a robust solution for candidate shortlisting at some job categories.

References 1. Strohmeier, S.: Digital human resource management: a conceptual clarification. German J. Hum. Resour. Manage.: Zeitschrift für Personalforschung 34(3), 345–365 (2020) 2. Breslin, D., Jones, C.: The evolution of entrepreneurial learning. Int. J. Organ. Anal. 20(3), 294–308 (2012) 3. Douven, I.: Abduction. In: Zalta, E.N. (eds.) The Stanford Encyclopedia of Philosophy, Summer 2017 Edition. Metaphysics Research Lab, Stanford University (2017). https://plato.sta nford.edu/archives/sum2017/entries/abduction/

1164

A. Vatousios and A. Happonen

4. Jaakkola, E.: Designing conceptual articles: four approaches. AMS Rev. 10(1–2), 18–26 (2020). https://doi.org/10.1007/s13162-020-00161-0 5. Hoon, C., Hack, A., Kellermanns, F.: Advancing knowledge on human resource management in family firms: an introduction and integrative framework. German J. Hum. Resour. Manage.: Zeitschrift für Personalforschung 33(3), 147–166 (2019) 6. Harsch, K., Festing, M.: Managing non-family talent: Evidence from German-speaking regions. German J. Hum. Resour. Manage.: Zeitschrift für Personalforschung 33(3), 249–279 (2019) 7. Adom, D., Hussein, E.K., Agyem, J.A.: Theoretical and conceptual framework: mandatory ingredients of a quality research. IJSR 7(1), 438–441 (2018) 8. Younger, J., Small Wood, N., Ulvich, D.: Developing your organizations brand as a talent developer. Hum. Resour. Plan. 30(2), 21–29 (2007) 9. Reeves, M., Deimler, M.: Adaptability: The New Competitive Advantage. Harvard Bus. Rev. (July–August 2011) 10. Piili, H., et al.: Digital design process and additive manufacturing of a configurable product. J. Adv. Sci. Lett. 19(3), 926–931 (2013) 11. Keller, S., Meaney, M.: Attracting and Retaining the Right Talent. McKinsey & Company, New York, NY, USA (2017) 12. Andrianova, S., Maor, D., Schaninger, B.: Winning With Your Talent-management Strategy. McKinsey & Company, New York, NY, USA (2018) 13. Walsh, B.: When Testing Takes Over. Harvard Graduate School of Education, Cambridge, MA (2017) 14. De Raad, B.: The Big Five Personality Factors: The Psycholexical Approach To Personality. Hogrefe & Huber Publishers, Germany (2000) 15. Gardner, H.: Frames of Mind: The Theory of Multiple Intelligences. Basic Books, New York (2011) 16. Kwok, F.: The evolution of management theories: a literature review. Nang Yan Bus. J. 3(1), 28–40 (2014) 17. Raad, B.D., Perugini, M.: Big five factor assessment: Introduction. In: de Raad, B., Perugini, M. (eds.) Big Five Assessment, pp. 1–18. Hogrefe and Huber Publishers, Göttingen (2002) 18. Widmaier, T., et al.: Digital design and manufacturing process comparison for new custom made product family – a case study of a bathroom faucet. Est. J. Eng. 19(1), 76–89 (2013) 19. Chen, X.: Culture, temperament, and social and psychological adjustment. Dev. Rev. 50(A), 42–53 (2018) 20. Merkin, R.: The relationship between individualism/collectivism: consultation and harmony needs. J. Intercult. Commun. 39, 4 (2015) 21. Cantor, P., Osher, D., Berg, J., Steyer, L., Rose, T.: Malleability, plasticity, and individuality: how children learn and develop in context. Appl. Dev. Sci. 23(4), 307–337 (2019) 22. Douglas, H.E., Bore, M., Munro, D.: Openness and intellect: an analysis of the motivational constructs underlying two aspects of personality. Person. Individ. Differ. 99, 242–253 (2016) 23. Combi, M.: Cultures and technology: an analysis of some of the changes in progress-digital, global and local culture. In: Borowiecki, K., Forbes, N., Fresa, A. (eds.) Cultural Heritage in a Changing World. Springer, Cham (2016) 24. Singhal, A., Gupta, A.K.: Managing human resources for innovation and creativity. Res. Technol. Manag. 36(3), 41–48 (1993) 25. Castro Laszlo, K., Laszlo, A.: Evolving knowledge for development: the role of knowledge management in a changing world. JKM 6(4), 400–412 (2002) 26. McDonnell, A., Collings, D.G., Mellahi, K., Schuler, R.: Talent management: a systematic review and future prospects. Eur. J. Int. Manag. 11(1), 86–128 (2017) 27. Kramar, R.: Beyond strategic human resource management: is sustainable human resource management the next approach? Int. J. Hum. Resour. Manag. 25(8), 1069–1089 (2014)

Transforming HR and Improving Talent Profiling

1165

28. Ghoreishi, M., Happonen, A.: New promises AI brings into circular economy accelerated product design: a review on supporting literature. E3S Web Conf. 158, 06002 (2020). https:// doi.org/10.1051/e3sconf/202015806002 29. Ghoreishi, M., Happonen, A.: Key enablers for deploying artificial intelligence for circular economy embracing sustainable product design: three case studies. AIP Conf. Proc. 2233(1), 1–19 (2020) 30. Ghoreishi, M., Happonen, A., Pynnönen, M.: Exploring industry 4.0 technologies to enhance circularity in textile industry: role of internet of things. 21st International Working Seminar on Production Economics, Innsbruck, Austria, pp. 1–16 (2020) 31. Lewis, R.E., Heckman, R.J.: Talent management: a critical review. Hum. Resour. Manag. Rev. 16, 139–154 (2006) 32. Cusumano, M.: How microsoft makes large teams work like small teams. Sloan Manag. Rev. 39(1), 9–20 (1997) 33. Kortelainen, H., Happonen, A.: From data to decisions – the re-distribution of roles in manufacturing ecosystems. VTT Blog, 1 (2018) 34. Kortelainen, H., Happonen, A., Hanski, J.: From asset provider to knowledge company— transformation in the digital era. In: Mathew, J., Lim, C.W., Ma, L., Sands, D., Cholette, M.E., Borghesani, P. (eds.) Asset Intelligence through Integration and Interoperability and Contemporary Vibration Engineering Technologies. Lecture Notes in Mechanical Engineering LNME, pp. 333–341. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-957111_33 35. Metso, L., Happonen, A., Rissanen, M., Efvengren, K., Ojanen, V., Kärri, T.: Data openness based data sharing concept for future electric car maintenance services. In: Ball, A., Gelman, L., Rao, B.K.N. (eds.) Advances in Asset Management and Condition Monitoring. Smart Innovation, Systems and Technologies SIST, vol. 166, pp. 429–436. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-57745-2_36 36. Salas, E., Cooke, N.J., Rosen, M.A.: On teams, teamwork, and team performance: discoveries and developments. J. Hum. Factors Ergon. Soc. 50(3), 540–547 (2008) 37. Whitzman, C.: ‘Culture eats strategy for breakfast’: the powers and limitations of urban design education. J. Urban Des. 21(5), 574–576 (2016) 38. Santti, U., Eskelinen, T., Rajahonka, M., Villman, R., Happonen, A.: Effects of business model development projects on organizational culture: a multiple case study of SMEs. TIM 7(8), 15–26 (2017) 39. Eskelinen, T., Räsänen, T., Santti, U., Happonen, A., Kajanus, M.: Designing a business model for environmental monitoring services using fast MCDS innovation support tools. TIM 7(11), 36–46 (2017) 40. Amberg, J.J., McGaughey, S.L.: Strategic human resource management and inertia in the corporate entrepreneurship of a multinational enterprise. Int. J. Hum. Resour. Manag. 30(5), 759–793 (2019) 41. Porras, J., et al.: Hackathons in software engineering education – lessons learned from a decade of events. ICSE 2018, 40–47 (2018) 42. Noe, R.A, Hollenbeck, J.R., Gerhart, B., Wright, P.M.: Human resource management: Gaining a competitive advantage. McGraw-Hill/Irwin (2017) 43. Wright, P.M., McMahan, G.C., McWilliams, A.: Human resources and sustained competitive advantage: a resource-based perspective. Int. J. Hum. Resour. Manag. 5(2), 301–326 (1994) 44. Salmela, E., Santos, C., Happonen, A.: Formalization of front end innovation in supply network collaboration. Int. J. Innov. Region. Dev. 5(1), 91–111 (2013) 45. ppscore – a Python Implementation of the Predictive Power Score. www.pypi.org 46. Wetschoreck, F.: RIP Correlation. Introducing the Predictive Power Score (PPS). Towards Data Science, Canada (2020)

1166

A. Vatousios and A. Happonen

47. Belbin Team Roles. Imperial College, London, Online resources 48. Meredith Belbin: Belbin Associates, Belbin (2020) 49. Ewenstein, B., Hancock, B., Komm, A.: Ahead of the Curve: The Future of Performance Management. McKinsey & Company, New York, NY, USA (2016) 50. Baptiste, B.: The Relationship Between the Big Five Personality Traits and Authentic Leadership. Walden Dissertations and Doctoral Studies. Walden University, USA (2018) 51. Personality. American Psychological Association. http://www.apa.org 52. Truss, C., Shantz, A., Soane, E., Alfes, K., Delbridge, R.: Employee engagement, organizational performance and individual well-being: exploring the evidence, developing the theory. Hum. Resour. Manag. J. 24(14), 2657–2669 (2013) 53. Cappelli, P.: Talent Management for the Twenty-First Century. Harv. Bus. Rev. 86(3), 74–81 (2008) 54. de Mol, E.: What Makes a Successful Start Up Team. Harvard Business Review (2019) 55. Tarique, I., Schuler, R.S.: Global talent management: literature review, integrative framework, and suggestions for further research. J. World Bus. 45(2), 122–133 (2010) 56. McDonnell, A., Hickey, C., Gunnigle, P.: Global talent management: exploring talent identification in the multinational enterprise. Eur. J. Int. Manag. 5(2), 174–193 (2011) 57. Collings, D.G., Mellahi, K.: Strategic talent management: a review and research agenda. Hum. Resour. Manag. Rev. 19(4), 304–313 (2009) 58. Scikit-Learn Developers, F1 Score, Scikit-Learn (2019) 59. Ghoreishi, M., Happonen, A.: The case of fabric and textile industry: the emerging role of digitalization, internet of things and industry 4.0 for circularity, Advances in Intelligent Systems and Computing, p. 14 (2021) 60. Vatousios, A., Happonen, A.: Renewed Talent Management: More Productive Development Teams with Digitalization Supported HR Tools, Sage Open, pp. 1–25 (2021, under review) 61. Happonen, A., Minashkina, D., Nolte, A., Medina Angarita, M.A.: Hackathons as a company – university collaboration tool to boost circularity innovations and digitalization enhanced sustainability. AIP Conf. Proc. 2233(1), 1–11 (2020) 62. Porras, J., Knutas, A., Ikonen, J., Happonen, A., Khakurel, J., Herala, A.: Code camps and hackathons in education – literature review and lessons learned. HICSS 2019, 7750–7759 (2019) 63. Happonen, A., Minashkina, D.: Ideas and experiences from university industry collaboration: hackathons, code camps and citizen participation. LUT Scientific and Expertise Publications report 86, 1–21 (2018) 64. Palacin-Silva, M., Khakurel, J., Happonen, A., Hynninen, T., Porras, J.: Infusing design thinking into a software engineering capstone course. In: 30th IEEE Conference on Software Engineering Education and Training (CSEE&T), pp. 212–221 (2017)

Author Index

A Abad, Janine Grace B., 757 Abdurasulov, Fakhritdin, 1107 Abu-Rasheed, Hasan, 1097 Adilov, Farukh, 1107 Ahmed, Amin, 922 Al Mudawi, Naif, 520 Aldawsari, Layla S., 118 Aloudat, Anas, 563 Alrayes, Amal, 563 Al-Shamaileh, Ons, 563 Altman, Tom, 118 Amit, Resh, 1114 Anand, Mayuresh, 965 Ananthanarayanan, Amritha, 657 Archibald, Jacqueline, 697 Ayad, Mustafa, 666 Aziz, Izzatdin Abdul, 946 B Bakaoukas, Anastasios G., 75 Bakaoukas, Nikolaos G., 75 Bansal, Arvind K., 737 Bashir, Masooda, 612, 1082 Beloff, Natalia, 520 Bendicho, Carlos, 471 Benmouhoub, Farah, 363 Bernier, William, 580 Bouhank, Asma, 423 Boukhari, Samah, 154 Brooks, Ian, 612 Bryan, Emily, 626 Bugaev, Alexander, 793 Burgo, J., 1068

C Caligiuri, Luigi Maxmilian, 169 Camargo, Jorge E., 886 Campbell, Roy H., 1082 Campbell, Sean, 911 Canbaz, Nurdan, 1136 Carmichael, Dawn, 697 Carvalho, Anderson, 911 Chen, Jim Q., 494 Cherinka, R., 1068 Codreanu, Valeriu, 102 Correa-Morris, Jyrko, 857 D Daghighi, Amirali, 494 Dahmani, Isma, 154 Daoudi, Mourad, 423 De Leon, Marlene M., 899 De Luca, Pasquale, 232 Diaper, Dan, 383 Diveev, Askhat, 718 Dolalas, Jerome M., 757 Dulce Rubio, Mateo, 886 Dushkin, R. V., 337 Dyer, Hunter A., 1082 E El-Messiry, Haythem, 922 Eoff, Ullysses, 244 F Fano, Andrew E., 965 Fathi, Madjid, 1097 Fonseca Cacho, Jorge Ramón, 1045

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 K. Arai (Ed.): Intelligent Computing, LNNS 283, pp. 1167–1169, 2022. https://doi.org/10.1007/978-3-030-80119-9

1168

Author Index

Foote, S., 1068 Franco, Caleb, 580 Frazelle, Chase G., 657 Fukuda, Shuichi, 401

Krumm, Roland, 1097 Kumar, Rohan, 657 Kyngäs, Jari, 274 Kyngäs, Nico, 274

G Galletti, Ardelio, 232 Garoche, Pierre-Loic, 363 Gheorghe-Pop, Ilie-Daniel, 188 Gilya-Zetinov, Alexander, 634, 793 Giné, F., 591 Gong, Yuanhao, 783 Gonzalez, Belarmino, 857 Guo, Qiang, 771

L Li, Jiahe, 834 Li, Kan, 823, 834, 846 Li, Xianping, 876 Lin, Yu-Yuan, 218 Lior, Shahar, 1114 Luo, Focus, 244

H Hacid, Hakim, 563 Haghbayan, Mohammad-Hashem, 685 Hajewski, Jeff, 133 Hajiali, Mahdi, 1045 Happonen, Ari, 1149 Haroglu, Hasan, 326 Hauswirth, Manfred, 188 Heikkonen, Jukka, 685 Hernández, Virginia, 981 Hifi, Mhand, 154, 410 Hosseini, Zahra, 930 Host, Kristina, 645 Hussian, Faris, 922 Huyck, Chris, 383 I Ikoro, Gibson O., 548 Ivanyan, Arsen, 1107 Ivaschenko, Anton, 539 Ivasic-Kos, Marina, 645 J Jaafar, Jafreezal, 946 Jabonete, Daniel S., 899 Jafari, Fahimeh, 255 K Karami, Amin, 255 Kaya, Tu˘gba Öztürk, 1136 Kethireddy, Sowmya, 657 Khaled, Hany, 922 Khelvas, Alexander, 634, 793 Khetan, Vivek, 965 Ko, Chen-Ho, 657 Kolesnikov, Valeriy, 311 Konyagin, Egor, 634, 793 Koutsonas, Spiridon, 326 Krivosheev, Arkadiy, 539 Krpalkova, Lenka, 911

M Macabebe, Erees Queen B., 757 MacEachen, Claire, 697 Maher, Ahmed, 922 Mahony, Niall O’., 911 Marcellino, Livia, 232 Martel, Matthieu, 363 Martínez, Juan, 857 Mickael, Benaroch, 1114 Mir, Nader F., 458 Mirto, Ignazio Mauro, 1054 Mohamed, Sherif A. S., 685 Mollinga, Joris, 102 Moloney, William, 483, 509 Mondragon, Raul J., 548 Moreno Pabón, Juan S., 886 Moreno-Jiménez, Luis-Gil, 1014 Mujinga, Mathias, 1128 N Nagadevi, 439 Nathaniel, Grabaskas, 1000 Nikita, Bronshtein, 1114 Niu, Xin, 823, 834, 846 Nurmi, Kimmo, 274 O Okkonen, Jussi, 930 Oliveira, Suely, 133 Osemwengie, Lucky, 255 Ossorio-Castillo, J., 292 Othman, Salem, 580 P Parocha, Raymark C., 757 Platte, Benny, 810 Plosila, Juha, 685 Pobar, Miran, 645 Pomorski, Krzysztof, 1 Ponza, Andrea, 1030 Prabhu, Vignesh, 657 Prezzama, J., 1068

Author Index Q Qiu, Guoping, 783 Quintana, Cynthia, 981 R Raja, Kasmir, 439 Ramnani, Roshni, 965 Reyes, Estefano, 857 Riascos, Álvaro J., 886 Riordan, Daniel, 911 Ritter, Marc, 810 Ritter, Tom, 188 Rodríguez Díaz, Paula, 886 Roesch, Phil, 580 Rolletschke, Tony, 810 Romero, Diana G., 757 Roschke, Christian, 810 Roy, Arunava, 946 S Sadeghsa, Shohre, 410 Santhanam, Chandramouli, 458 Scannapieco, Simone, 1030 Segre, Julien, 634, 793 Seigel, Carly, 626 Senbel, Samah, 626 Sengupta, Subhashis, 965 Sharma, Kriti, 458 Sharma, Tanusree, 1082 Singh, Aditi, 737 Sitaraman, Murali, 346 Sitnikov, Pavel, 539 Sivilotti, Paolo A. G., 346 Solé, M., 591 Stolbova, Anastasia, 539 T Taghva, Kazem, 1045 Tatashin, Pavel, 483, 509 Tcholtchev, Nikolay, 188 Tekbulut, Tu˘grul, 1136 Tengler, Szymon, 56 Tenhunen, Hannu, 685 Thomanek, Rico, 810 Tomazzoli, Claudio, 1030 Tornero, José M., 292 Torres-Moreno, Juan-Manuel, 1014 Tzeng, Jeng-Nan, 218

1169 U Urra-Yglesias, Abel, 857 Usero, Belén, 981 Usmani, Usman Ahmad, 946 V Valls, M., 591 Valotker, Beni, 1114 Vasudevan, Bhargav Srinivas, 657 Vatousios, Antonis, 1149 Voyles, Richard, 666 W Walker, Ian D., 657 Walsh, Joseph, 911 Wang, Haoting, 823 Wang, Ren, 771 Wang, Tian, 612 Warwas, Kornel, 56 Watada, Junzo, 946 Weber, Christian, 1097 Wei, Minghua, 771 Weide, Alan, 346 White, Graham, 548 White, Martian, 520 Wu, Teresa, 876 X Xiang, Haihao, 244 Xing, Xiaoyu, 133 Xu, Guangxin, 244 Y Yasin, Jawad N., 685 Yıldırım, Ahmet Artu, 204 Yin, Hui, 783 Yonit, Rusho, 1114 Yusupbekov, Nodirbek, 1107 Z Zenkert, Johannes, 1097 Zhang, Yi, 846 Zhao, Juan, 244 Zhong, Hongcheng, 244 Zimmer, Frank, 810 Zuev, Ilya, 634