Imaging and Sensing for Unmanned Aircraft Systems: Deployment and Applications (Control, Robotics and Sensors) [2] 1785616447, 9781785616440

Table of contents :
Cover
Contents
About the editors
Preface
1 UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0
1.1 Introduction
1.2 Cloud computing
1.3 Collective UAV learning
1.4 Human computation, crowdsourcing and call centres
1.5 Open-source and open-access resources
1.6 Challenges and future directions
1.7 Conclusions
References
2 UAS human factors and human–machine interface design
2.1 Introduction
2.2 UAS HMI functionalities
2.2.1 Reconfigurable displays
2.2.2 Sense and avoid
2.2.3 Mission planning and management
2.2.4 Multi-platform coordination
2.3 GCS HMI elements
2.4 Human factors program
2.4.1 Requirements definition, capture and refinement
2.4.2 Task analysis
2.4.3 Hierarchal task analysis
2.4.4 Cognitive task analysis
2.4.5 Critical task analysis
2.4.6 Operational sequence diagram
2.4.7 Systems design and development
2.4.8 Design evaluation
2.4.9 Verification and validation
2.5 Future work
2.6 Conclusions
References
3 Open-source software (OSS) and hardware (OSH) in UAVs
3.1 Introduction
3.2 Open source software
3.3 Open source UAS
3.4 Universal messaging protocol
3.5 GCS software
3.6 Processing software
3.7 Operator information and communication
3.8 Open source platform
3.9 Future work
3.9.1 OSH challenges
3.9.2 Open data
3.9.3 Cloud data centre
3.9.4 Crowd-sourced data in UAV-CPSs
3.9.5 Control of UAV swarms
3.10 Conclusions
References
4 Image transmission in UAV MIMO UWB-OSTBC system over Rayleigh channel using multiple description coding (MDC)
4.1 Introduction
4.1.1 The efficiency of the flat Rayleigh fading channel
4.2 Multiple description coding
4.3 Multiple input–multiple output
4.4 Diversity
4.5 Simulations results
4.6 Discussion and future trends
4.7 Conclusion
References
5 Image database of low-altitude UAV flights with flight condition-logged for photogrammetry, remote sensing, and computer vision
5.1 Introduction
5.1.1 Image processing system for UAVs
5.2 The aerial image database framework
5.2.1 Database requirements
5.2.2 Database design
5.3 Image capture process
5.4 Results
5.4.1 Images collected
5.5 Use of the image database
5.5.1 Mosaics
5.5.2 Development of CV algorithms
5.6 Conclusion and future works
Acknowledgements
References
6 Communications requirements, video streaming, communications links and networked UAVs
6.1 Introduction
6.2 Flying Ad-hoc Networks
6.3 The FANET protocol
6.4 FANET: streaming and surveillance
6.5 Discussion and future trends
6.5.1 FNs' placement search algorithms
6.5.2 Event detection and video quality selection algorithms
6.5.3 Onboard video management (UAV)
6.5.4 Video-rate adaptation for the fleet platform
6.5.5 FNs coordination
6.5.6 Data collection and presentation
6.5.7 Software-Defined Networking
6.5.8 Network Function Virtualisation
6.5.9 Data Gathering versus Energy Harvesting
6.6 Conclusion
References
7 Multispectral vs hyperspectral imaging for unmanned aerial vehicles: current and prospective state of affairs
7.1 Introduction
7.2 UAV imaging architecture and components
7.2.1 Future scope for UAV
7.3 Multispectral vs. hyperspectral imaging instruments
7.3.1 Multispectral imaging
7.3.1.1 Low-resolution imaging
7.3.1.2 High-resolution imaging
7.3.2 Hyperspectral imaging
7.3.3 Satellite imaging vs UAV imaging
7.4 UAV image processing workflow
7.4.1 Atmospheric correction
7.4.2 Spectral influence mapping
7.4.3 Dimensionality reduction
7.4.4 Computational tasks
7.5 Data processing toolkits for spatial data
7.6 UAV open data sets for research–multispectral and hyperspectral
7.7 Applications of MSI and HSI UAV imaging
7.7.1 Agriculture monitoring
7.7.2 Coastal monitoring
7.7.3 Forestry
7.7.4 Urban planning
7.7.5 Defence applications
7.7.6 Environmental monitoring
7.7.7 Other commercial uses
7.8 Conclusion and future scope
References
8 Aerial imaging and reconstruction of infrastructures by UAVs
8.1 Introduction
8.2 Related Studies
8.3 Visual sensors and mission planner
8.3.1 Image projection
8.3.2 Path planner
8.4 3D reconstruction
8.4.1 Stereo mapping
8.4.2 Monocular mapping
8.5 Data-set collection
8.5.1 Experimental setup
8.5.2 Data set 1
8.5.3 Data set 2
8.6 Experimental results
8.6.1 Indoor scenario
8.6.2 Outdoor Scenario 1
8.6.3 Outdoor Scenario 2
8.6.4 Underground scenario
8.7 Future trends
8.8 Conclusions
References
9 Deep learning as an alternative to super-resolution imaging in UAV systems
9.1 Introduction
9.2 The super-resolution model
9.2.1 Motion estimation
9.2.2 Dehazing
9.2.3 Patch selection
9.2.4 Super-resolution
9.3 Experiments and results
9.3.1 Peak signal-to-noise ratio
9.4 Critical issues in SR deployment in UAV-CPSs
9.4.1 Big data
9.4.2 Cloud computing services
9.4.3 Image acquisition hardware limitations
9.4.4 Video SR
9.4.5 Efficient metrics and other evaluation strategies
9.4.6 Multiple priors
9.4.7 Regularisation
9.4.8 Novel architectures
9.4.9 3D SR
9.4.9.1 Depth map super-resolution
9.4.10 Deep learning and computational intelligence
9.4.11 Network design
9.5 Conclusion
References
10 Quality of experience (QoE) and quality of service (QoS) in UAV systems
10.1 Introduction
10.1.1 Airborne network from a CPS perspective
10.2 Definitions
10.2.1 Parameters that impact QoS/QoE
10.2.2 Impact of cloud distance on QoS/QoE
10.2.3 QoS/QoE monitoring framework in UAV-CPSs
10.2.4 Application-level management
10.2.5 Network-level management
10.2.6 Cloud distance management
10.2.7 QoS/QoE service-level management
10.2.8 QoS/QoE metrics in UAV-CPSs
10.2.9 Mapping of QoS to QoE
10.2.10 Subjective vs objective measurement
10.2.11 Tools to measure QoS/QoE
10.3 Applications
10.3.1 Social networks, gaming and human–machine interfaces
10.3.2 Data centres
10.3.3 Electric power grid and energy systems
10.3.4 Networking systems
10.3.5 Surveillance
10.4 Case studies
10.4.1 Application scenario 1: UAV-CPSs in traffic congestion management
10.4.1.1 QoS management at the FN
10.4.1.2 QoS management at the UAV-CPS UCN
10.4.2 Application scenario 2: congestion and accident avoidance using intelligent vehicle systems
10.5 Future and open challenges
10.5.1 Modelling and design
10.5.2 Collaborative services
10.5.3 Streaming
10.5.4 Security
10.5.5 Flying ad hoc networks
10.5.6 User emotions
10.6 Conclusion
References
11 Conclusions
Index
Back Cover

Citation preview

IET CONTROL, ROBOTICS AND SENSORS SERIES 120

Imaging and Sensing for Unmanned Aircraft Systems

IET International Book Series on Sensing—Call for Authors The use of sensors has increased dramatically in all industries. They are fundamental in a wide range of applications from communication to monitoring, remote operation, process control, precision and safety, and robotics and automation. These developments have brought new challenges such as demands for robustness and reliability in networks, security in the communications interface, and close management of energy consumption. This Book Series covers the research and applications of sensor technologies in the fields of ICTs, security, tracking, detection, monitoring, control and automation, robotics, machine learning, smart technologies, production and manufacturing, photonics, environment, energy, and transport. Book Series Editorial Board ●

Dr. Hartmut Brauer, Technische Universita¨t Ilmenau, Germany

● ●

Prof. Nathan Ida, University of Akron, USA Prof. Edward Sazonov, University of Alabama, USA

●

Prof Desineni “Subbaram” Naidu, University of Minnesota Duluth, USA

●

Prof. Wuqiang Yang, University of Manchester, UK Prof. Sherali Zeadally, University of Kentucky, USA

●

Proposals for coherently integrated international multi-authored edited or co-authored handbooks and research monographs will be considered for this Book Series. Each proposal will be reviewed by the IET Book Series Editorial Board members with additional external reviews from independent reviewers. Please email your book proposal to: [email protected] or [email protected].

Imaging and Sensing for Unmanned Aircraft Systems Volume 2: Deployment and Applications Edited by Vania V. Estrela, Jude Hemanth, Osamu Saotome, George Nikolakopoulos and Roberto Sabatini

The Institution of Engineering and Technology

Published by The Institution of Engineering and Technology, London, United Kingdom The Institution of Engineering and Technology is registered as a Charity in England & Wales (no. 211014) and Scotland (no. SC038698). † The Institution of Engineering and Technology 2020 First published 2020 This publication is copyright under the Berne Convention and the Universal Copyright Convention. All rights reserved. Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may be reproduced, stored or transmitted, in any form or by any means, only with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publisher at the undermentioned address: The Institution of Engineering and Technology Michael Faraday House Six Hills Way, Stevenage Herts, SG1 2AY, United Kingdom www.theiet.org While the authors and publisher believe that the information and guidance given in this work are correct, all parties must rely upon their own skill and judgement when making use of them. Neither the authors nor publisher assumes any liability to anyone for any loss or damage caused by any error or omission in the work, whether such an error or omission is the result of negligence or any other cause. Any and all such liability is disclaimed. The moral rights of the authors to be identified as authors of this work have been asserted by them in accordance with the Copyright, Designs and Patents Act 1988.

British Library Cataloguing in Publication Data A catalogue record for this product is available from the British Library ISBN 978-1-78561-644-0 (Hardback Volume 2) ISBN 978-1-78561-645-7 (PDF Volume 2) ISBN 978-1-78561-642-6 (Hardback Volume 1) ISBN 978-1-78561-643-3 (PDF Volume 1) ISBN 978-1-78561-679-2 (Hardback Volumes 1 and 2)

Typeset in India by MPS Limited Printed in the UK by CPI Group (UK) Ltd, Croydon

Contents

About the editors Preface

1 UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0 Ana Carolina B. Monteiro, Reinaldo P. Franca, Vania V. Estrela, Sandro R. Fernandes, Abdeldjalil Khelassi, R. Jenice Aroma, Kumudha Raimond, Yuzo Iano, and Ali Arshaghi

xi xiii

1

1.1 Introduction 1.2 Cloud computing 1.3 Collective UAV learning 1.4 Human computation, crowdsourcing and call centres 1.5 Open-source and open-access resources 1.6 Challenges and future directions 1.7 Conclusions References

2 5 7 8 8 10 13 13

2 UAS human factors and human–machine interface design Yixiang Lim, Alessandro Gardi and Roberto Sabatini

23

2.1 2.2

2.3 2.4

2.5

Introduction UAS HMI functionalities 2.2.1 Reconfigurable displays 2.2.2 Sense and avoid 2.2.3 Mission planning and management 2.2.4 Multi-platform coordination GCS HMI elements Human factors program 2.4.1 Requirements definition, capture and refinement 2.4.2 Task analysis 2.4.3 Hierarchal task analysis 2.4.4 Cognitive task analysis 2.4.5 Critical task analysis 2.4.6 Operational sequence diagram 2.4.7 Systems design and development 2.4.8 Design evaluation 2.4.9 Verification and validation Future work

23 26 28 28 28 28 29 33 36 38 38 39 39 40 41 42 43 44

vi

3

Imaging and sensing for unmanned aircraft systems, volume 2 2.6 Conclusions References

46 46

Open-source software (OSS) and hardware (OSH) in UAVs Pawel Burdziakowski, Navid Razmjooy, Vania V. Estrela, and Jude Hemanth

49

3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9

49 50 51 55 57 57 59 61 61 61 62 62 63 63 64 64

Introduction Open source software Open source UAS Universal messaging protocol GCS software Processing software Operator information and communication Open source platform Future work 3.9.1 OSH challenges 3.9.2 Open data 3.9.3 Cloud data centre 3.9.4 Crowd-sourced data in UAV-CPSs 3.9.5 Control of UAV swarms 3.10 Conclusions References 4

Image transmission in UAV MIMO UWB-OSTBC system over Rayleigh channel using multiple description coding (MDC) Ali Arshaghi, Navid Razmjooy, Vania V. Estrela, Pawel Burdziakowski, Douglas A. Nascimento, Anand Deshpande, and Prashant P. Patavardhan 4.1

5

67

Introduction 4.1.1 The efficiency of the flat Rayleigh fading channel 4.2 Multiple description coding 4.3 Multiple input–multiple output 4.4 Diversity 4.5 Simulations results 4.6 Discussion and future trends 4.7 Conclusion References

67 72 72 74 76 77 82 84 85

Image database of low-altitude UAV flights with flight condition-logged for photogrammetry, remote sensing, and computer vision Helosman Valente de Figueiredo, Osamu Saotome, Elcio H. Shiguemori, Paulo Silva Filho, and Vania V. Estrela

91

5.1

Introduction 5.1.1 Image processing system for UAVs

92 92

Contents 5.2

The aerial image database framework 5.2.1 Database requirements 5.2.2 Database design 5.3 Image capture process 5.4 Results 5.4.1 Images collected 5.5 Use of the image database 5.5.1 Mosaics 5.5.2 Development of CV algorithms 5.6 Conclusion and future works Acknowledgements References 6 Communications requirements, video streaming, communications links and networked UAVs Hermes J. Loschi, Vania V. Estrela, D. Jude Hemanth, Sandro R. Fernandes, Yuzo Iano, Asif Ali Laghari, Asiya Khan, Hui He and Robert Sroufe 6.1 6.2 6.3 6.4 6.5

Introduction Flying Ad-hoc Networks The FANET protocol FANET: streaming and surveillance Discussion and future trends 6.5.1 FNs’ placement search algorithms 6.5.2 Event detection and video quality selection algorithms 6.5.3 Onboard video management (UAV) 6.5.4 Video-rate adaptation for the fleet platform 6.5.5 FNs coordination 6.5.6 Data collection and presentation 6.5.7 Software-Defined Networking 6.5.8 Network Function Virtualisation 6.5.9 Data Gathering versus Energy Harvesting 6.6 Conclusion References 7 Multispectral vs hyperspectral imaging for unmanned aerial vehicles: current and prospective state of affairs R. Jenice Aroma, Kumudha Raimond, Navid Razmjooy, Vania V. Estrela and Jude Hemanth 7.1 7.2 7.3

Introduction UAV imaging architecture and components 7.2.1 Future scope for UAV Multispectral vs hyperspectral imaging instruments 7.3.1 Multispectral imaging

vii 94 94 94 95 98 98 100 100 103 106 107 107

113

114 114 115 119 121 121 122 123 123 123 124 124 125 126 128 128

133

133 136 138 138 138

viii

8

Imaging and sensing for unmanned aircraft systems, volume 2 7.3.2 Hyperspectral imaging 7.3.3 Satellite imaging vs UAV imaging 7.4 UAV image processing workflow 7.4.1 Atmospheric correction 7.4.2 Spectral influence mapping 7.4.3 Dimensionality reduction 7.4.4 Computational tasks 7.5 Data processing toolkits for spatial data 7.6 UAV open data sets for research – multispectral and hyperspectral 7.7 Applications of MSI and HSI UAV imaging 7.7.1 Agriculture monitoring 7.7.2 Coastal monitoring 7.7.3 Forestry 7.7.4 Urban planning 7.7.5 Defence applications 7.7.6 Environmental monitoring 7.7.7 Other commercial uses 7.8 Conclusion and future scope References

140 140 141 142 142 143 143 144

Aerial imaging and reconstruction of infrastructures by UAVs Christoforos Kanellakis, Sina Sharif Mansouri, Emil Fresk, Dariusz Kominiak and George Nikolakopoulos

157

8.1 8.2 8.3

157 158 160 160 161 162 162 163 164 164 166 167 168 168 168 170 171 171 173 173

Introduction Related Studies Visual sensors and mission planner 8.3.1 Image projection 8.3.2 Path planner 8.4 3D reconstruction 8.4.1 Stereo mapping 8.4.2 Monocular mapping 8.5 Data-set collection 8.5.1 Experimental setup 8.5.2 Data set 1 8.5.3 Data set 2 8.6 Experimental results 8.6.1 Indoor scenario 8.6.2 Outdoor Scenario 1 8.6.3 Outdoor Scenario 2 8.6.4 Underground scenario 8.7 Future trends 8.8 Conclusions References

144 147 147 147 147 148 148 148 148 148 150

Contents 9 Deep learning as an alternative to super-resolution imaging in UAV systems Anand Deshpande, Prashant P. Patavardhan, Vania V. Estrela, Navid Razmjooy and Jude Hemanth 9.1 9.2

Introduction The super-resolution model 9.2.1 Motion estimation 9.2.2 Dehazing 9.2.3 Patch selection 9.2.4 Super-resolution 9.3 Experiments and results 9.3.1 Peak signal-to-noise ratio 9.4 Critical issues in SR deployment in UAV-CPSs 9.4.1 Big data 9.4.2 Cloud computing services 9.4.3 Image acquisition hardware limitations 9.4.4 Video SR 9.4.5 Efficient metrics and other evaluation strategies 9.4.6 Multiple priors 9.4.7 Regularisation 9.4.8 Novel architectures 9.4.9 3D SR 9.4.10 Deep learning and computational intelligence 9.4.11 Network design 9.5 Conclusion References 10 Quality of experience (QoE) and quality of service (QoS) in UAV systems Asif Ali Laghari, Asiya Khan, Hui He, Vania V. Estrela, Navid Razmjooy, Jude Hemanth and Hermes J. Loschi 10.1 Introduction 10.1.1 Airborne network from a CPS perspective 10.2 Definitions 10.2.1 Parameters that impact QoS/QoE 10.2.2 Impact of cloud distance on QoS/QoE 10.2.3 QoS/QoE monitoring framework in UAV-CPSs 10.2.4 Application-level management 10.2.5 Network-level management 10.2.6 Cloud distance management 10.2.7 QoS/QoE service-level management 10.2.8 QoS/QoE metrics in UAV-CPSs 10.2.9 Mapping of QoS to QoE 10.2.10 Subjective vs objective measurement

ix

177

177 178 181 182 183 183 185 186 186 186 188 188 189 190 191 192 193 195 197 198 199 199

215

216 217 218 219 220 220 222 223 223 223 223 224 224

x

Imaging and sensing for unmanned aircraft systems, volume 2 10.2.11 Tools to measure QoS/QoE 10.3 Applications 10.3.1 Social networks, gaming and human–machine interfaces 10.3.2 Data centres 10.3.3 Electric power grid and energy systems 10.3.4 Networking systems 10.3.5 Surveillance 10.4 Case studies 10.4.1 Application scenario 1: UAV-CPSs in traffic congestion management 10.4.2 Application scenario 2: congestion and accident avoidance using intelligent vehicle systems 10.5 Future and open challenges 10.5.1 Modelling and design 10.5.2 Collaborative services 10.5.3 Streaming 10.5.4 Security 10.5.5 Flying ad hoc networks 10.5.6 User emotions 10.6 Conclusion References

225 226 226 227 227 227 227 228 228 231 232 232 233 234 234 235 237 237 238

11 Conclusions Vania V. Estrela, Jude Hemanth, Osamu Saotome, George Nikolakopoulos, and Roberto Sabatini

247

Index

249

About the editors

Vania Estrela is a Faculty/Researcher at Telecommunications Department, Universidade Federal Fluminense (UFF), and a visiting scholar at UNICAMP. Her interests are biomedical engineering, electronic instrumentation, modelling/ simulation, sustainable design, multimedia, artificial intelligence, remote sensing, STEM education, environment, and digital inclusion. Reviewer for IEEE, Elsevier, ACM, IET, Springer Verlag and MDPI. She has extensive experience as a project manager, post-graduate advisor (M.Sc. and D.Sc.) as well as an editor of books and special issues. ORCID 0000-0002-4465-7691 Jude Hemanth is an Associate Professor in the ECE Department of Karunya University (KU), India. He is a member of the IEEE task force on deep learning and serves as an Associate Editor and Editorial Board Member for several international refereed journals. Osamu Saotome is a Professor at the Instituto Tecnolo´gico de Aerona´utica (ITA), Brazil. He has been involved in several international research and cooperation projects with the Brazilian Air Force, INPE, IEAv (France, Sweden, USA and Japan). George Nikolakopoulos is a Professor in robotics and automation at the Department of Computer Science, Electrical and Space Engineering at Lulea˚ University of Technology (LTU), Sweden. He is also a member of the ARTEMIS Scientific Council of the European Commission. He has significant experience in Managing European and National R&D&I projects funded by the EU, ESA, Swedish and the Greek National Ministry of Research. Roberto Sabatini is a Professor of Aerospace Engineering and Aviation at RMIT University (Australia) specialising in Avionics and Intelligent/Autonomous Systems for Aerospace and Defence applications. Currently, he serves as Deputy Director (Aerospace) of the Sir Lawrence Wackett Centre and Chair of the Cyber-Physical Systems Group at RMIT University. Professor Sabatini is a Fellow and Executive Member of the Institution of Engineers Australia, Fellow of the Royal Aeronautical Society, and Fellow the Royal Institute of Navigation. Throughout his career, he led numerous industry and government-funded research projects and he has authored or co-authored over 250 peer-reviewed international publications. In addition to his primary faculty duties, Professor Sabatini serves as Vice-Chair of the IEEE-AESS Avionics Systems Panel and editor for several high-impact international journals.

This page intentionally left blank

Preface

An unmanned aerial vehicle (UAV)—otherwise called a drone, unmanned aircraft system (UAS) or remotely piloted aircraft system (RPAS)—is a flying machine without a human pilot on-board. Its flight can be controlled autonomously by embedded computers on the vehicle or by remote control in semi-autonomous or full manual approaches. UAVs, in general, can help in an enormous number of assignments, for example, observation, disaster alleviation, human services in remote districts. Automatons are remarkably equipped for infiltrating territories, which might be unreasonably perilous for the guided speciality. Making a UAV independent requires tending to issues from different disciplines, as mechanical design, aviation, automatic control, software engineering, sensor advancements and computerised reasoning, to name a few. The UAV cyber-physical system (CPS) includes (i) the subsystems and interfaces, (ii) mechatronic framework (aeronautics) and (iii) the ground control station [1,2]. To accomplish the desired levels of autonomy, the avionics have an optimal design, multi-dimensional sensors’ frameworks and actuation. An altogether self-governing UAV can (i) get data about the Earth, (ii) work for an all-encompassing time frame without human interference, (iii) fly over its operating area without human assistance and (iv) avoid risky circumstances for individuals and their assets. UAVs calls for CPSs, and these resulting UAV-CPSs are undoubtedly multidimensional, particularly considering their full potentials, extensive and multiple operational scales and social implications. The wireless sensor networks (WSNs) and visual sensor and actuator networks (VSANs) are very important for UAVs in CPSs. Preferably, they have to be of low cost, low power, multi-functional, and friendly to all sorts of devices (mobile or not). UAV-CPSs impose many restrictions on the WSNs design, such as scalability, fault tolerance, operating environments, hardware restrictions, production budgets, network topology, and power consumption [3–18]. These challenges have led to demanding research to address the possible collaboration among sensors in information acquisition and processing. Therefore, an essential requirement for sensor and actuator nodes is to operate with a limited source of energy. The network should work successfully and actively, giving enough time to deploy the necessary applications. A wireless multimedia sensor network (WMSN) is a network of wirelessly connected sensor nodes with multimedia components, e.g., that can handle high-dimensional sensor data, still images, video and audio streams. WMSNs support a wide range of prospective applications, which necessitate multimedia data

xiv

Imaging and sensing for unmanned aircraft systems, volume 2

such as reconnaissance dense sensor networks, e-government control systems, cutting-edge healthcare provision, elderly assistance, telemedicine, and disaster remediation as well as control. In these applications, multimedia support has the potential of enhancing the level of evidence collected, expanding the coverage, and permitting multi-resolution views (i.e., in comparison to the measurements of scalar data). WMSNs have certain characteristics and challenges, in addition to those of WSNs, because of the nature of the real-time multimedia information like high bandwidth need, real-time provision, acceptable end-to-end delay, and suitable jitter and frame loss rate. Likewise, WMSNs have countless different resources of restrictions involving power, bandwidth, processing capability, data rate, memory, and buffers’ size as a consequence of the substantially small size of the sensors and the multimedia nature of the applications that usually produces a massive amount of data. Hence, to satisfy the quality of service (QoS) necessities and to administrate the network of hardware and software resources efficiently, the WMSN characteristics along with other concerns such as coverage and safety (Figure P.1). These characteristics should be pondered almost certainly at the different communication protocol layers, while these issues will be presented and discussed in detail over the subsequent chapters. Furthermore, given the relatively high redundancy in the VSAN data, WMSNs have extra requirements like in-node multimedia processing (such as

Security

Hardware and software constraints

Hardware & Testbeds

Application layer Transport layer QoS Routing layer MAC layer Physical layer

WMSNs and VSANs coverage and connectivity

Figure P.1 A UAV cyber-physical system seen from the network point of view

Preface

xv

distributed multimedia coding and data compression), application-specific QoS needs, and multimedia in-network computational modules (e.g., several data fusion possibilities, storage management, off-board processing and aggregation). The physical layer can use three technology groups as far as frequency bands go: narrowband, spread spectrum, and ultra-wideband (UWB). These technologies rely on several modulation structures and bandwidth considerations, with different standard protocols (IEEE 802.15.1 Bluetooth, IEEE 802.15.4 ZigBee, IEEE 802.11 Wi-Fi, 802.15.3a UWB). ZigBee is the most common standard radio protocol employed in WSNs because of its a lightweight standard and low-cost and low-power features. ZigBee allows for data rates up to 250 kbps at 2.4 GHz, coding efficiency of 76.52 %, and a range of 10–100 m. Nonetheless, ZigBee is not appropriate for high data rate usages, such as multimedia streaming over WMSN and for ensuring application QoS. Conversely, other standards like Bluetooth and Wi-Fi have higher data rates and code efficiency, but they require extra power. This volume is organised into the following chapters: Chapter 1: UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0 The first chapter focuses on several issues and current discussions on UAV-CPSs. Several trends and needs are discussed to foster criticism from readers to the upcoming chapters and provide food for thought. Chapter 2: UAS human factors and human–machine interface design Some of the significant UAV-CPSs advantages are their capability to withstand the duties and objectives of human beings while accomplishing chores for them. Simultaneously, they also take some decisions and execute some actions independently. Therefore, people and machines must collaborate. Even though these capabilities offer noteworthy paybacks, there is still a tremendous amount of effort necessary to fully master suitable ways of assisting human-machine interaction. Chapter 3: Open-source software (OSS) and hardware (OSH) in UAVs Using low-cost, open-source components together with multiple sensors and actuators is quite a challenge in terms of effort and cost. Hence, it is desirable to employ Open Source Software (OSS) and Hardware (OSH) in UAV-CPSs. Whenever possible, available OSH and OSS should be used in designs independently of the type of hardware framework and the operating system. Chapter 4: Image transmission in UAV MIMO UWB-OSTBC system over Rayleigh channel using multiple description coding (MDC) The performance of an orthogonal frequency division multiplexing (OFDM) UAVCPS can be enhanced by adding channel coding (i.e., error correction code) to identify and correct the errors that happen throughout data transmission. The massive bandwidth of UWB channels can originate new effects compared to conventional wireless channel modelling in several MAV applications. UWB technologies based on IEEE 802.15.4a have various uses when there is a need for highly accurate

xvi

Imaging and sensing for unmanned aircraft systems, volume 2

localisation with various sensors for stabilisation and navigation that are still absolute indoor positioning challenges for the UAVs. Chapter 5: Image database of low-altitude UAV flights with flight conditionlogged for photogrammetry, remote sensing, and computer vision Flying a UAV in unstructured settings with changeable conditions is challenging. To support the development of better algorithms, a multipurpose data set for low-altitude UAV flights in a given Brazilian environment is proposed as a benchmark for positioning and other avionics tasks, so that computer vision procedures are assessed in terms of robustness and generalisation with a baseline for depth estimation with and without landmarks. This stage of development can help the advancement of future integration with remote sensing (RS) modules that will bring in more spectral information to the analysis. Chapter 6: Communications requirements, video streaming, communications links and networked UAVs UAV-CPSs involve a great deal of knowledge on networking—more specifically— on flying ad-hoc networks (FANETs). The fact that the traffic of high-dimensional multimedia data streams through UAV-CPSs tends to grow exponentially raises several issues towards future research directions. Chapter 7: Multispectral vs hyperspectral imaging for unmanned aerial vehicles: current and prospective state of affairs The texture is an important feature to recognise objects or regions of interest (ROIs) in an image, and it has been commonly used for image classification from satellite images, for instance. UAV imagery takes advantage of ultra-high spatial resolution [13,14], which shows that the texture is also a paramount source of knowledge. Nevertheless, texture in the UAV imagery was seldom used in surveillance. Moreover, merging ground hyperspectral data could compensate for the limited bands of UAV sensors and increase the estimation precision of analyses. Consequently, the goals of this chapter are (i) to explore UAV-based multispectral imagery and (ii) to improve several types of estimation accuracy through hyperspectral information. Chapter 8: Aerial imaging and reconstruction of infrastructures by UAVs This chapter will present the application of camera-equipped UAVs for aerial visual inspection of the infrastructure, such as constructions, buildings, bridges, forest reservations and other types of infrastructure arrangements that need frequent cycles of aerial visual inspection. These UAV-CPSs can frequently examine all sorts of sites, monitor the work in progress and generate detailed 3-D reconstructions of these environments for further utilisation. Chapter 9: Deep learning as an alternative to super-resolution imaging in UAV systems Regardless of the sophistication of the on-board sensors, the Cloud, RS, computational intelligence and communication advances, super-resolution (SR) will be in

Preface

xvii

demand for quite a long time. This will continue to happen in contexts where acquiring imageries is expensive and troublesome like in healthcare, astronomy and disaster relief. Chapter 10: Quality of experience (QoE) and quality of service (QoS) in UAV systems As seen in Figure P.1, QoS and QoE (besides other qualitative performance metrics) will take a pivotal role to impulse further improvements in all stages of a UAV-CPS. This book intends to be a reference for current and forthcoming applications of UAV-CPSs. It will display fundamental aspects, ongoing research efforts, accomplishments and challenges faced when it comes to the deployment of imaging capabilities and sensor integration in UAVs. Vania V. Estrela Jude Hemanth Osamu Saotome George Nikolakopoulos Roberto Sabatini

References [1] Estrela, V.V., Saotome, O., Loschi, H.J., et al. 2018. Emergency response cyber-physical framework for landslide avoidance with sustainable electronics. Technologies. 6:42. doi:10.3390/technologies6020042 [2] Estrela, V.V., Monteiro, A.C.B., Franc¸a, R.P., Iano, Y., Khelassi, A., and Razmjooy, N. 2019. Health 4.0: applications, management, technologies and review. Med. Technol. J. 2(4):262–276. doi: 10.26415/2572-004xvol2iss1p262-27 [3] Ezequiel, C.A.F., Cua, M., Libatique, N.C., et al. UAV aerial imaging applications for post-disaster assessment, environmental management and infrastructure development. In Proceedings of the 2014 IEEE International Conference on Unmanned Aircraft Systems (ICUAS), Orlando, FL, USA, 2014; pp. 274–283. [4] Lewkowicz D.J., and Ghazanfar A.A. 2009. The emergence of multisensory systems through perceptual narrowing. Trends Cogn. Sci. 13(11):470–478. doi:10.1016/j.tics.2009.08.004 [5] Zmigrod, S., and Hommel, B. 2010. Temporal dynamics of unimodal and multimodal feature binding. Atten. Percept. Psychophys. 72(1):142–152. doi:10.3758/APP.72.1.142 [6] Nitti, D.O., Bovenga, F., Chiaradia, M.T., Greco, M., and Pinelli, G. 2015. Feasibility of using synthetic aperture radar to aid UAV navigation. Sensors. 15:18334–18359. [7] Park, C., Cho, N., Lee, K., and Kim, Y. 2015. Formation flight of multiple UAVs via onboard sensor information sharing. Sensors. 15:17397–17419.

xviii

Imaging and sensing for unmanned aircraft systems, volume 2

[8] Blaauw, F.J., Schenk, H.M., Jeronimus, B.F., et al. 2016. Let’s get Physiqual – an intuitive and generic method to combine sensor technology with ecological momentary assessments. J. Biomed. Inform. 63:141–149. [9] Kang, W., Yu, S., Ko, S., and Paik, J. 2015. Multisensor super resolution using directionally-adaptive regularization for UAV images. Sensors. 15:12053–12079. [10] Karpenko, S., Konovalenko, I., Miller, A., Miller, B., and Nikolaev, D. 2015. UAV control on the basis of 3D landmark bearing-only observations. Sensors. 15:29802–29820. [11] Yoon, I., Jeong, S., Jeong, J., Seo, D., and Paik, J. 2015. Wavelengthadaptive dehazing using histogram merging-based classification for UAV images. Sensors. 15, 6633–6651. [12] Wen, M., and Kang, S. 2014. Augmented reality and unmanned aerial vehicle assist in construction management. Comput. Civil. Building. Eng. 1570–1577. doi: 10.1061/9780784413616.195 [13] Li, H., Zhang, A., and Hu, S.A 2015. Multispectral image creating method for a new airborne four-camera system with different bandpass filters. Sensors. 15:17453–17469. [14] Aroma, R.J., and Raimond, K. 2017. A novel two-tier paradigm for labeling water bodies in supervised satellite image classification. Proc. IEEE 2017 International Conference on Signal Processing and Communication (ICSPC), Coimbatore, India, 384–388. [15] Chiang, K.-W., Tsai, M.-L., Naser, E.-S., Habib, A., and Chu, C.-H. 2015. New calibration method using low cost MEM IMUs to verify the performance of UAV-borne MMS payloads. Sensors. 15:6560–6585. [16] Roldan, J.J., Joossen, G., Sanz, D., del Cerro, J., and Barrientos, A. 2015. Mini-UAV based sensory system for measuring environmental variables in greenhouses. Sensors. 15:3334–3350. [17] Gonzalez, L.F., Montes, G.A., Puig, E., Johnson, S., Mengersen, K., and Gaston, K.J. 2016. Unmanned aerial vehicles (UAVs) and artificial intelligence revolutionizing wildlife monitoring and conservation. Sensors. 16:97. [18] Razmjooy N., Ramezani M., and Estrela V.V. (2019) A solution for Dubins path problem with uncertainties using world cup optimization and Chebyshev polynomials. In: Iano Y., Arthur R., Saotome O., Vieira Estrela V., and Loschi H. (eds) Proc. BTSym 2018. Smart innovation, systems and technologies, vol 140. Springer, Cham. doi: 10.1007/978-3-030-16053-1_5

Chapter 1

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0 Ana Carolina B. Monteiro1, Reinaldo P. Franca1, Vania V. Estrela2, Sandro R. Fernandes3, Abdeldjalil Khelassi4, R. Jenice Aroma5, Kumudha Raimond5, Yuzo Iano1, and Ali Arshaghi6

The extensive Cloud pool of resources and infrastructure can deliver significant improvements to Unmanned Aerial Vehicle (UAV) Cyber-Physical Systems (UAV-CPSs) relying on data or code from a network to operate, but not all sensors, actuators, computation modules and memory depots from a single fixed structure. This chapter is organised around the potential benefits of the Cloud: (1) Big Data (BD) access to visual libraries containing representations/descriptive data, images, video, maps and flight paths, (2) Cloud Computing (CC) functionalities for Grid Computing (GC) on demand for statistical analysis, Machine Learning (ML) algorithms, Computational Intelligence (CI) applications and flight planning, (3) Collective UAV Learning (CUL) where UAVs share their trajectories, control guidelines and mission outcomes and (4) human–machine collaboration through crowdsourcing for analysing high-dimensional high-resolution (HDHR) images/ video, classification of scenes/objects/entities, learning and error correction/ concealment. The Cloud can also expand UAV-CPSs by offering (a) data sets, models, all sorts of literature, HDHR benchmarks and software/hardware simulators, (b) open competitions for UAV-CPS designs with Open Source Hardware (OSH) and (c) Open-Source Software (OSS). This chapter talks about some open challenges and new trends in UAV-CPSs.

1

Laboratory of Computer Vision (LCV), FEEC, UNICAMP, Campinas, SP, Brazil Telecommunications Department, Federal Fluminense University (UFF), RJ, Brazil 3 Instituto Federal de Educacao, Ciencia e Tecnologia do Sudeste de Minas Gerais, Juiz de Fora, MG, Brazil 4 Department of Informatics, Faculty of Sciences, Tlemcen University, Tlemcen, Algeria 5 Karunya Institute of Technology and Sciences, Coimbatore, India 6 Department of Electrical Engineering, Central Tehran Branch, Islamic Azad University, Tehran, Iran 2

2

Imaging and sensing for unmanned aircraft systems, volume 2

1.1 Introduction Cloud Computing (CC) can expand the range and potential of applications of Unmanned Aerial Vehicles (UAVs) by modelling them as parts of a CyberPhysical System (CPS) [1]. This approach is helpful because users can concentrate on their goals instead of being disturbed about maintenance, power outages, software/hardware updates, the presence of crowds, and weather conditions. The Cloud also allows for different processing scales and eases information sharing across applications and users. Most UAV-CPSs still work using on-board processing only, memory, and software. Evolving advances and the growing disposal of networking in the cloud demands new tactics where computation happens with remote access to dynamic and effective global services, means and data sets to give support and resources’ scalability for a variety of functions and operational scenarios [2]. CC enables both current and upcoming UAV-CPSs to use wireless networking services, Big Data (BD), Open-Source Software (OSS), Open Source Hardware (OSH), Grid Computing (GC), Statistical Machine Learning (ML), Computational Intelligence (CI), Crowdsourcing, and other collective resources to upsurge performance in a wide assortment of uses, for example, assembly lines, inspection, driving, package transport, warehouse logistics, caregiving to the incapacitated and elderly, disaster prevention/mitigation, astronomy, and medical applications. UAV-CPSs can rely largely on CC with robots, operators, end-users, Ground Control Stations (GCSs), and Remote Sensing (RS) equipment connected through wired and wireless. Due to network latency, flexible Quality of Service (QoS), and downtime, Cloud UAV-CPSs habitually possess some local processing capacity for low-latency responses as well as throughout periods when network access is absent or unreliable. This chapter attacks the subsequent potential CC benefits [3–12]: 1. 2. 3. 4.

BD access to remote libraries of images, maps, trajectories, and object data CC provision of parallel GC on demand for visual statistical analysis, learning, in addition to flight planning Collective UAV Learning (CUL) due to UAVs sharing control policies, trajectories, and outcomes Human inferences using crowdsourcing access to remote human expertise for analysing images, classification, learning, and error recovery.

This chapter also cites instances where the Cloud can boost UAV-CPSs via facilitated access to (a) simulation tools, data sets, models, benchmarks, and publications, (b) open competitions for designs and systems, and (c) OSS like Operating Systems (OSs) [13–19]. The concept of a Cloud-based UAV-CPS permits several other new initiatives such as the Robot Operating System (ROS) [14] and the Internet of Things (IoT) to talk effortlessly. For instance, an RFID module and inexpensive FPGA processors can play several roles in a vast array of UAVs, GCSs, and physical objects from a medical facility to permit communication, control, and information exchange.

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0

3

The Cloud can afford UAV-CPSs access to vast resources of data as well as information repositories that are inappropriate to on-board memory. BD describes data that surpasses the processing capacity of conventional database systems including images, video, RS data, maps, real-time network and commercial transactions, and vast arrays of networks of sensors. Data can be collected from many sources in a Cloud-based UAV-CPS that combines streaming data from distributed processing and multiple sensors, UAVs, maps, and RS repositories taking into thought the BD and CUL aspects of CC as portrayed in Figure 1.1. Some BD algorithms’ opportunities and challenges can provide reasonable estimates to queries on large data sets to keep running times manageable, but these approximations can be harshly affected by degraded data. A Cloud-based UAV-CPS can use Global Potioning Systems (GPSs) from mobile devices to collect transportation data, process them, and dispense commands while amassing and sharing information about several types of environmental parameters, for example, noise and pollution levels. Large data sets can facilitate ML, as has been demonstrated in the Computer Vision (CV) context. Large-scale image data sets have been used for object and scene recognition while reducing the necessity for manually labelled training data. Augmented Reality (AR) applications can be rendered more efficiently, due to Cloud processing using open visual data collections. Combining Internet imageries with visual semantic querying from a local human worker can afford more robust CV learning algorithms [20–27]. Deep Learning (DL) relies on many layered neural networks that can profit from BD and has been employed for CV [28,29].

Cloud computing Network description Proxy ID server

Filter server

Internal cluster Multi-cloud without changes

Data server

Auxiliary server

API Web

User

Auto-scaling, Benchmarking, Compliance, Reporting

Public

Traffic report server

Traffic estimation server

Computation provider

Figure 1.1 A Cloud-based UAV-CPS

Cloud computing

4

Imaging and sensing for unmanned aircraft systems, volume 2

Navigation is a persistent UAV-CPSs’ challenge that can benefit from Cloud resources relying on incremental learning by matching sensor information from 2D image structures, 3D features, 3D point clouds [30,31], in addition to demonstrations against 3D or 4D models in online databases. Google Maps can enhance a Cloud-based UAV-CPS [32], as shown later. The UAV captures an image and sends it via the network to the Object Recognition Server (ORS) that returns data for a set of candidate items, each with pre-computed assessment options. The UAV compares the returned CAD models with the detected point cloud to refine identification, estimate pose, and select an appropriate course of action. After navigation, for instance, data on the current path and state are used to update models in the Cloud for future references [33,34]. Future designs can use BD, CC, and CUL aspects of Cloud-supported automation. A UAV-CPS store data related to any types of objects/entities and maps for uses like objective recognition, mobile navigation, grasping, and manipulation (please refer to Figures 1.2 and 1.3) [36,37]. There exist several online data sets to appraise different algorithms’ aspects. One research challenge is defining cross-platform formats for representing data. Sensor-acquired data (e.g., images, videos, and point clouds) have a few widely used formats. Even relatively simple data, for instance, trajectories, have no universal standards yet despite ongoing research. Sparse representations for efficient transmission of data, for example, algorithms for sparse motion planning [38–41] are another caveat. Large data sets gathered from distributed sources regularly enclose corrupted information with erroneous, repeated, or degraded data [42,43] such as 3D position data collected during UAV calibration [44]. New approaches must be robust to untreated raw data [45–49].

Figure 1.2 Google’s object recognition system combines a large visual data set and textual labels (metadata) with ML to simplify object recognition in the Cloud [35]

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0 Cloud Object recognition engine

UAVs

Execution results from navigation commands Cloud storage

Object label

3D Model

Images

Cameras

5

3D Sensors

Point clouds

Pose estimation

Candidate paths

Feasible path with highest probability

Figure 1.3 System for a Cloud-based Object Recognition Server (ORS)

1.2 Cloud computing Massively parallel computation on demand is widely available [50] from commercial sources such as Amazon’s Elastic Compute Cloud, Microsoft’s Azure [51], and Google’s Compute Engine. These systems provide access to a multitude of remote computers for short-term processing tasks. These services were at first used primarily by web application developers but are becoming popular in scientific and technical High-Performance Computing (HPC) applications [51–56]. Uncertainty in sensing, models, as well as control is a central matter in UAVCPSs, and they can be modelled as perturbations in position, shape, orientation, and control. CC is ideal for sample-based Monte Carlo analysis [57]. For example, parallel CC can be used to compute the outcomes of the cross-product of many possible perturbations in object/environment pose, shapes, and UAV response to sensors and commands. Medicine and particle physics explore this idea. Cloud-based sampling aids in computing robust trajectories when shape uncertainty exists. For example, a flight planning algorithm can accept as input a polygonal outline following a Gaussian distribution around each vertex and the centre of mass and uses parallel sampling to compute quality metrics for several tasks. CC has the potential to expedite many computationally intensive UAV-CPS tasks such as navigation via SLAM in the Cloud as exemplified in Figure 1.4, next view planning aimed at object recognition and UAVs formation control. For optimal sampling-based path planning methods [36,59], CC is useful to generate the graphs; it is also important to recognise that these graphs can proliferate. Consequently, algorithms for graph reduction are necessary to simplify data transfer, as illustrated in Figure 1.3. The Cloud also expedites visual data analysis [60,61], mapping [62] (Figures 1.2 and 1.3), and assisted living technology [63,64]. Figure 1.4 demonstrates how computer-intensive bundle adjustment for navigation using Simultaneous Localisation and Mapping (SLAM) performed in the Cloud [58,65].

6

Imaging and sensing for unmanned aircraft systems, volume 2

Servers and databases Results

Surveillance

Cloud

Control station 1

Control station 2

Control station N

Figure 1.4 UAV navigation using cooperative tracking and mapping within a Cloud (C2TAM) [58]

GS: Ground station y x

Goal

Goal z x

GS

y

GS

Figure 1.5 Distributed sampling-based flight planning Figure 1.5 illustrates a 2D motion planning map in a High-Dimensional HighResolution (HDHR) space to save computational and networking resources. Effective UAV motion planning can make operation more flexible, lighter, and the computation can be split between the UAV and the cloud only when needed. It is central to acknowledge that the Cloud is prone to fluctuating network latency and QoS. Some usages are not time-sensitive like decluttering or

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0

7

pre-computing navigation or offline optimisation of machine scheduling. On the other hand, many missions comprise real-time requirements, which is a research area up and about. There is still enough room to propose open standards for selfreporting sensing and actuation devices for all sorts of UAV-CPSs. Cloud-assisted storage of data from networks of sensors and actuators can enable collaborative sharing of information for transportation routing and other applications with the CUL aspect in mind [66,67].

1.3 Collective UAV learning The Cloud eases UAVs’ data sharing and learning by gathering evidence from several instances of real-world trials and environments. For example, UAV-CPSs subsystems can share initial and looked-for conditions, accompanying control policies and trajectories, and vital data on the performance and outcomes. Figure 1.6 suggests a framework for CUL by indexing trajectories from many UAVs over a myriad of tasks employing CC for parallel planning and trajectory adjustment [68]. The UAV-CPS can learn from pre-computed flight plans stored in the Cloud. The planner attempts to discover a new plan as well as discern another existing plan from a problem similar to the current one. Whichever finishes first is chosen, and this framework can use the BD, CC, and CUL aspects of Cloud Robotics and Automation. Such systems can also be expanded to global networks to assist shared path planning, including traffic routing, according to Figures 1.5 and 1.6 [33,34].

Cloud

Flight info

Path library

Library control

Retrieve path

Repair path

UAV flight trajectory Path optimiser

Flight info analyser

Plan new path

Figure 1.6 Schematic architecture for path planning

8

Imaging and sensing for unmanned aircraft systems, volume 2

Sharing data through CUL can also advance the capabilities of UAVs with restricted computational resources. Social network with UAVs can profit the same way as people from socialising, collaborating, and sharing. Moreover, UAVs can benefit from sharing their sensor material, giving insight on their current state perspective. The UAV-CPSs’ databases must be updated with new evidence from connected UAVs among other sources to learn from public Internet resources, computer simulations, and real-life expeditions.

1.4 Human computation, crowdsourcing and call centres Human skill, experience, and intuition are being tapped to solve several problems such as image labelling for CV, learning associations between classes’ labels and locations, and gathering data. Shortly, UAVs will be assigned and dispatched on-demand on crowdsourcing basis where human workers who experience redistribution can perform tasks that exceed the capabilities of computers. Contrary to automatic reservation systems, a future scenario will take into account errors and exceptions perceived by UAVs and distributed computer systems, which then contact humans or advanced artificial intelligence units at remote call centres for guidance. Research projects investigate how this can be used for path planning, to determine depth layers, normal and symmetry from images, besides, to refine image segmentation. Researchers are working to understand pricing models and to crowdsource navigation with knowledge-based solutions. Networked computing has allowed UAVs to be remotely teleoperated, and the expanded Cloud resources foster new studies into remote human operation. Figure 1.7 depicts an interface for operators to control tasks in a UAV-CPS using a set of different strategies. The results indicate that humans can select better and more robust strategies. Crowdsourcing object recognition aids UAV navigation via a Cloud UAV-CPS that integrates semantic knowledge about the world with subjective decisions. The DARPA Robotics Challenge (DRC) used an open-source Cloud-based simulation platform for UAV-CPS performance testing on a variety of disaster management tasks. The Cloud allows running collaborating, real-time parallel simulation of activities (e.g., predicting and assessing performance, corroborating design decisions, optimising designs, training users, etc.).

1.5 Open-source and open-access resources The Cloud supports the ongoing evolution of Cloud-supported UAV-CPSs by smoothing human access to (a) data sets, publications, models, benchmarks, and simulation tools; (b) OSH design challenges; and (c) OSS. The success of OSS is widely accepted, such as the ROS [13,14], which provides libraries as well as tools for software developers craft UAV applications.

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0

9

UAV autonomy

Autonomous execution

Applications

Task planner

Task list: ... ... . . . . .

Assisted execution

Assisted execution

Teleoperated execution

High-level task primitives

Low-level task primitives

Joint angles/ pose control

Operator assistance

Operator assistance

Teleoperation

Figure 1.7 Human assistance with Cloud-based teleoperation

ROS also works with Android while being commonly used by almost all UAV developers in both research and industry. Additionally, several UAV-CPS simulation libraries are now OSS, which allows individuals to rapidly set up and adapt themselves to new systems and share the resulting software. There exist many open-source simulation libraries, video games’ simulators, simulation environments geared specifically towards UAVCPSs, a motion-planning library, and other tasks. These OSS and OSH libraries allow modifications to suit applications and incompatible with their original design. Figure 1.8 displays a quadcopter built of recycled material which can make a difference in impoverished countries where UAVs can rely on surplus or re-manufactured parts [1,69]. Another exciting trend is in OSH, where CAD models and the technical minutiae of devices can be freely available. The Arduino project is a widely used OSH platform with multiple types of sensors and actuators available. Special care must be given to open-architecture medical UAVs developed. Recent advances in 3D printing are poised to have a significant impact on countless fields, including OSH development in designs that help humanitarian engineering [70]. The Cloud enables open design challenges that can attract a diverse and geographically scattered population of innovators. OSH can promote Cloud usage for simulation and UAV-CPS performance testing on a variety of disaster missions. The Cloud supports interactive, real-time parallel simulations for prediction and assessment of performance, design validations, optimisations, and training users.

10

Imaging and sensing for unmanned aircraft systems, volume 2

Figure 1.8 Ecoquadcopter: A UAV made up of electronic garbage and remanufactured parts [69]

1.6 Challenges and future directions The Cloud inherent connectivity raises a range of new privacy and security concerns besides challenges. These concerns encompass data from Cloud-connected UAVs, sensors, and actuators, notably as they may contain HDHR multimedia data [7,71–73]. Cloud automation services also bring together the potential of UAVCPSs to suffer hacker attacks that disrupt functionalities or cause damage. For example, UAVs can be troubled by inexpensive GPS spoofing schemes. These concerns call attention to new regulatory, accountability, and legal matters related to control, safety, and transparency. New algorithms and methods are needed on the technical front to cope with timevarying network latency and QoS. More rapid data networks, both wired and wireless Internet connection standards such as Long-Term Evolution (LTE) [74], reduce latency. Nevertheless, algorithms must degrade gracefully when the Cloud assets are prolonged, noisy, or unavailable. For example, anytime load-balancing algorithms for mobile devices send information to the Cloud for analysis and at the same time process, it internally uses the best results after a reasonable delay. New algorithms are also needed which scale to the size of BD and which often contain untreated raw data that require cleaning or sampling adequately. It is vital that algorithms oversample to take into account that some remote processors may fail or experience extended delays to turn in results in distributed environments. When using the semantic computation, algorithms need to filter unreliable input while balancing the costs of human intervention with the cost of UAV failure. Deploying UAV-CPS algorithms into the Cloud requires a transition process simplification [75–81]. The possible levels at which a Cloud framework can be

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0

11

implemented are threefold. The lowest level is Infrastructure as a Service (IaaS), where bare OSs working on (perhaps virtualised) machines are provided. The next level, Platform as a Service (PaaS), offers more structure, comprising application frameworks with database access while curbing the choice of programming languages, architectures, and database models that can be employed. The highest structural level is known as Software as a Service (SaaS), and it is typified by the difference between Google Docs, which is a Cloud-based, and Microsoft Word older versions that needed downloading followed by local installation. The UAV can communicate with a CC stage, which follows the PaaS paradigm for transferring computation from UAVs to the Cloud. It also connects to the knowledge repositories to take care of the BD part. This PaaS rationale can be extended to the SaaS paradigm, which offers many potential advantages for UAVCPSs. With SaaS, an interface allows data to be sent to a server that processes it and returns outputs, which does not need data/hardware/software maintenance and allows companies to control proprietary software. This approach can be called Robotics and Automation as a Service (RAaaS). To illustrate the concept, consider two scenarios for someone setting up a UAV work cell with, for instance, a Microsoft Kinect RGBD sensor. The purpose of the work cell is to pick up and examine assembly line parts, requiring object recognition/localisation, and flight planning, among others. ROS has a well-known open-source robotics library, which provides access to innumerable OSS packages, and it runs locally [14]. Many stable ROS packages can simplify deployment, but some software is only offered as a source distribution, which entails the download and installation of dependencies. The person must set up a new machine with, for example, Ubuntu and resolve all library dependencies, including conflict with other packages. In contrast, RAaaS the analysis and planning software runs in the Cloud with users visiting a website to enter input data and commands for UAVs, sensors, actuators, and models. Then, they can select the desired recognition and localisation algorithms, flight planning, and other intelligent algorithms, and utilise convenient human-machine interfaces to execute these processes into a pipeline. Their UAVs can send data up as point clouds from platforms such as the Kinect. The UAV receives and executes motion plans and other tasks, reporting outcomes to the Cloudbased UAV-CPSs to gather feedback to improve the Cloud-based services over time. Several Multimedia Cross-Layer Optimisation (MCLO) methodologies exploiting features of Multimedia and High-Dimensional Data (MHDD) have been suggested for transmission rate regulation, energy conservation, error recovery/ concealment, congestion control, and multipath choice [82–87]. Nevertheless, the multimedia streaming problem in resource-constrained Visual Sensor Networks (VSNs) [88–95] is quite challenging in UAV-CPSs, not entirely solved, and new complications are still arising. An essential consideration to lead future MHDD research handling in UAVCPSs is the manner representation techniques and multimedia coding advance. In VSNs, new codecs come all the time, with different results in data compression, computational sophistication, and error resilience. Long-standing standards have

12

Imaging and sensing for unmanned aircraft systems, volume 2

been further improved, while new procedures as in-networking computation and distributed multimedia coding have been employed for VSNs [45–49]. Regarding the evolution of MHDD, it is worth wondering about what data are relevant for an individual application. For example, video compression with addressevent representation can attain frame-difference coding with a low computational cost. Frame-difference information can indicate motion of a dislocating target in VSNs. In some applications, target moving behaviour knowledge is more significant than the visual facts. In doing so, the target identity may be preserved, but the bystander can still be able to comprehend the target behaviour because very lean codes can characterise frame-difference data even when compared with grey-scale imageries. The MHDD evolution can use cross-layer performance assessment via Compressed Sensing (CS) and DL streaming. CS-represented images display an intrinsic resiliency to link inaccuracies, contrasting with for instance JPEG images, owing to lack of structure and adequacy in image representation. The evolvement of the computational, coding and sensing models will stimulate the MCLO deployment in VSNs. However, other matters such as a tracking system for wireless VSNs need appropriate consideration since they involve multiple mobile sinks. There are issues potentially altering the path choice protocols, congestion regulation, error remediation, in addition to the coding schemes for multimedia streaming. Many researchers think that sensors with an Ultra-Wideband (UWB) transceiver may noticeably augment the costs. This physical layer technology permits transmission rates much higher than the ZigBee [42,82,83,96,97]. Some claim UWB as the ultimate solution for Wireless Multimedia VSNs (WMVSNs), MCLO will have to cogitate both the benefits and streaming challenges of ad hoc VSNs where UWB-enabled links interconnect intermediate nodes [37,57,59,98–102]. The embraced MAC protocols may also impact the Multimedia-based CrossLayer (MCL) design. For instance, the IEEE 802.11 standard can be more adequate for the link layer, but the IEEE 802.15.4 is still favoured. MAC protocols may be less used in future practical VSNS applications, mostly due to power limitations of the sensor nodes. Still, there are some energy-aware MAC protocols for WMVSNs, such as T-MAC and ET-MAC. The type of wireless links may also influence the cross-layer structural design, but differently. Overall, packet dropping in VSNs is a consequence of a network bottleneck or bit errors. Sending packets through wireless links may incur in bit errors happen during the transmission. Linear error probability over a single bit is unrealistic, and bit errors take place in bursts with large packets having a higher probability of being discarded than small packets. As reduced packets imply on an extra protocol header overhead, investigators should be concerned about the ideal dimension of packets transporting multimedia-encoded information. A motivating approach is the usage of MCLO in packet size. However, plenty of investigations will fix pending concerns about the extent of transmitted packets and how to handle them more efficiently. Obvious and practical reasons to regulate the choice of the packet size are missing. As power constraints persist as a captain concern for the upcoming years, MCL design in VSNs has to economise power while accomplishing the desired optimisation, or at least not incur in additional power consumption [103–110]. Many literary works

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0

13

propose MCLO without promoting energy awareness. Future studies could contribute further by adding power-saving mechanisms while achieving feasible solutions for VSNs. New challenges related to transmission rate correction, energy conservation, congestion control, error recovery/concealment, multipath broadcast, and in-network compression should still arise, necessitating additional investigation in MCLO. Some of the possible challenges come from strict requirements involving specific surveillance practices, regarding, for example, multi-tier designs or independent sources. Other problems like source nodes mobility and coverage protection also demand from the academic community innovative studies, directly influencing the employed coding strategy and the implemented cross-layer deployment. To put it briefly, priority-oriented planning established on appropriate CV techniques, such as foreground differentiating, should also be deliberated in future work.

1.7 Conclusions The current broad Cloud resources and foundation can improve altogether UAVCPSs depending on learning or code from a system to work since not all sensor/ actuator parts, calculation modules, and memory warehouses have a place with the equivalent fixed physical framework [20–27]. This part discusses the forthcoming Cloud benefits: (1) BD access to visual libraries, benchmark video, distinct records, maps, and directions, (2) CC functionalities for parallel GC on interest for measurable examination, ML calculations, CI applications, and flight arranging, (3) CUL, where UAVs trade information about their directions, control approaches, and results, and (4) Crowdsourcing to research HDHR pictures/video, arrangement, learning, and blunder rectification and camouflage. The Cloud can likewise grow UAV-CPSs by offering (a) data sets, writing, models, test systems, and a wide range of benchmarks; (b) open rivalries for frameworks’ plans utilizing OSH, and (c) OSS. Further developments will take into consideration some known difficulties and new patterns in UAV-CPSs [111–114] MCLO uses the inward features of multimedia coding employing system conventions to accomplish higher proficiency in VSNs’ design. The best in class research tending to such explicit issues, offering appropriate commitments in the fields of the transmission rate change, power management/distribution, congestion control, error recuperation, multipath decision, and in-network compression will drive new innovation utilities and reveal new requests. Last, future research horizons have been talked about, showing promising examination regions concerning this issue. Crisp and progressively proficient VSNs improve MCLO.

References [1] Estrela V. V., Saotome O., Loschi H. J., et al. “Emergency Response CyberPhysical Framework for Landslide Avoidance with Sustainable Electronics,” Technologies, vol. 6, no. 42, 2018. doi:10.3390/technologies6020042

14

Imaging and sensing for unmanned aircraft systems, volume 2

[2] Kehoe B., Patil S., Abbeel P., and Goldberg K. “A Survey of Research on Cloud Robotics and Automation,” IEEE Transactions on Automation Science and Engineering, 2014;12(2):398–409. [3] Aguero C., Koenig N., Chen I., et al. “Inside the Virtual Robotics Challenge: Simulating Real-time Robotic Disaster Response,” IEEE Transactions on Automation Science and Engineering (T-ASE): SI on Cloud Robotics and Automation, vol. 12, no. 2, 2015, pp. 494–506. [4] Bekris K., Shome R., Krontiris A., and Dobson A. “Cloud Automation: Precomputing Roadmaps for Flexible Manipulation,” IEEE Robotics & Automation Magazine: SI on Emerging Advances and Applications in Automation, 2014;22(2):41–50. [5] Hu G., Tay W., and Wen Y. “Cloud Robotics: Architecture, Challenges and Applications,” IEEE Network, vol. 26, no. 3, pp. 21–28, 2012. [6] Hunter T., Moldovan T., Zaharia M., et al. “Scaling the Mobile Millennium System in the Cloud,” in Proc. 2nd ACM Symposium on Cloud Comp (SOCC), Cascais Portugal, 2011, p. 28. [7] Jangid N. K. “Real Time Cloud Computing,” in Data Management & Security, 2011. [8] Mehrotra P., Djomehri J., Heistand S., et al. “Performance Evaluation of Amazon EC2 for NASA HPC Applications,” in Proc. 3rd ACM Workshop on Scientific Cloud Computing Data-Science Cloud ’12, 2012, New York, NY, USA, pp. 41–50. [9] Michael N., Mellinger D., Lindsey Q., and Kumar V. “The GRASP Multiple Micro UAV Testbed,” IEEE Robotics & Automation Magazine, vol. 17, no. 3, pp. 56–65, 2010. [10] Narita M., Okabe S., Kato Y., Murakwa Y., Okabayashi K., and Kanda S. “Reliable Cloud-Based Robot Services,” in Proc. Conference on the IEEE Industrial Electronics Society. IEEE, 2013, Vienna, Austria, pp. 8317–8322. [11] Nurmi D., Wolski R., Grzegorczyk C., et al. “The Eucalyptus Open-Source Cloud Computing System,” in Proc. IEEE/ACM International Symposium on Cluster Computing and the Grid, Shanghai, China, 2009, pp. 124–131. [12] Proia A. A., Simshaw D., and Hauser K. “Consumer Cloud Robotics and the Fair Information Practice Principles: Recognizing the Challenges and Opportunities Ahead,” Minnesota Journal of Law, Science & Technology, 2015;16(1):1–70. [13] Quigley M., Gerkey B., Conley K., et al. “ROS: An Open-Source Robot Operating System,” in Proc. IEEE ICRA Workshop on Open Source Software, Kobe, Japan, 2009. [14] Estrela V. V., Hemanth, J., Nascimento, D. A., Loschi, H. J., and Razmjooy, N. “Computer Vision and Data Storage in UAVs,” in Advances in UAV Avionics for Imaging and Sensing, vol. 1. IET, 2019. [15] Plaku E., Bekris K. E., and Kavraki L. E. “OOPS for Motion Planning: An Online, Open-Source, Programming System,” in Proc. IEEE International Conference on Robotics and Automation (ICRA), Rome, Italy, 2007, pp. 3711–3716.

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0

15

[16] Nurmi D., Wolski R., Grzegorczyk C., et al. “The Eucalyptus Open-Source Cloud Computing System,” in Proc. IEEE IEEE/ACM International Symposium on Cluster Computing and the Grid, Shanghai, China, 2009, pp. 124–131. [17] Hannaford B., Rosen J., Friedman D., et al. “Raven II: Open Platform for Surgical Robotics Research,” IEEE Transactions on Biomedical Engineering, 60, pp. 954–959, 2012. [18] Davidson S. “Open-Source Hardware,” IEEE Design and Test of Computers, vol. 21, no. 5, pp. 456–456, 2004. [19] Dabbish L., Stuart C., Tsay J., and Herbsleb J. “Social Coding in GitHub: Transparency and Collaboration in an Open Software Repository,” in Proc. IEEE ACM 2012 Conference on Computer Supported Cooperative Work, Bellevue, Washington, USA, 2012, p. 1277. [20] Gouveia B., Portugal D., Silva D., and Marques L. “Computation Sharing in Distributed Robotic Systems: A Case Study on SLAM,” IEEE Transactions on Automation Science and Engineering (T-ASE): SI on Cloud Robotics and Automation, vol. 12, no. 2, 2015 pp. 10–22. [21] Hidago-Pena E., Marin-Urias L. F., Montes-Gonzalez F., Marin-Hernandez A., and Rios-Figueroa H. V., “Learning from the Web: Recognition Method Based on Object Appearance from Internet Images,” in Proc. ACM/IEEE International Conference on Human–Robot Interaction. IEEE, Tokyo, Japan, 2013, pp. 139–140. [22] Krizhevsky A., Sutskever I., and Hinton G. E. “ImageNet Classification with Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems 25, Pereira F., Burges C., Bottou L., and Weinberger K., Eds. Curran Associates, Inc., 2012, pp. 1097–1105. [23] Johnson-Roberson M., Bohg J., Skantze G., et al. “Enhanced Visual Scene Understanding through Human–Robot Dialog,” in Proc. IEEE International Conference on Intelligent Robots and Systems (IROS), San Francisco, CA, USA, 2011, pp. 3342–3348. [24] Le Q., Ranzato M., Monga R., et al. “Building High-Level Features Using Large Scale Unsupervised Learning,” in Proc. International Conference in Machine Learning, ICML 2012, Edinburgh, Scotland, 2012. [25] Samadi M., Kollar T., and Veloso M. “Using the Web to Interactively Learn to Find Objects,” in Proc. IEEE AAAI Conference on Artificial Intelligence, 2012, Toronto, Ontario, Canada, pp. 2074–2080. [26] Sevior M., Fifield T., and Katayama N., “Belle Monte-Carlo Production on the Amazon EC2 Cloud,” Journal of Physics: Conference Series, vol. 219, no. 1, 2010. [27] Chandrasekaran V., and Jordan M. I., “Computational and Statistical Tradeoffs Via Convex Relaxation,” Proc. National Academy of Sciences of the United States of America, vol. 110, no. 13, pp. E1181–E1190, 2013. [28] Hemanth D. J., and Estrela V. V. “Deep Learning for Image Processing Applications,” Advances in Parallel Computers. IOS Press, 2017. ISBN9781-61499-821-1 (print) 978-1-61499-822-8 (online). [29] Dean J., Corrado G., Monga R., et al. “Large Scale Distributed Deep Networks,” in Advances in Neural Information Processing Systems 25,

16

[30]

[31]

[32]

[33]

[34]

[35] [36]

[37]

[38]

[39]

[40]

[41]

Imaging and sensing for unmanned aircraft systems, volume 2 Pereira F., Burges C., Bottou L., and Weinberger K., Eds. Curran Associates, Inc., 2012, pp. 1223–1231. Beksi W., and Papanikolopoulos N., “Point Cloud Culling for Robot Vision Tasks under Communication Constraints,” in Proc. IEEE International Conference on Intelligent Robots and Systems (IROS), Chicago, Illinois, USA, 2014. Lai K., and Fox D. “Object Recognition in 3D Point Clouds Using Web Data and Domain Adaptation,” International Journal of Robotics Research (IJRR), vol. 29, no. 8, pp. 1019–1037, 2010. Kasper A., Xue Z., and Dillmann R. “The KIT Object Models Database: An Object Model Database for Object Recognition, Localization and Manipulation in Service Robotics,” International Journal of Robotics Research (IJRR), vol. 31, no. 8, pp. 927–934, 2012. Higuera J., Xu A., Shkurti F., and Dudek G. “Socially-Driven Collective Path Planning for Robot Missions,” Proc. IEEE Conference on Computer and Robot Vision, pp. 417–424, Toronto, ON, Canada, 2012. Berenson D., Abbeel P., and Goldberg K. “A Robot Path Planning Framework that Learns from Experience,” in Proc. IEEE International Conference on Robotics and Automation (ICRA), Minneapolis, MN, USA, 2012, pp. 3671–3678. Kuffner J. “Cloud-Enabled Robots,” in Proc. IEEE-RAS International Conference on Humanoid Robots, Nashville, TN, USA, 2010. Marinho C. E. V., Estrela, V.V., Loschi, H.J. et al. “A Model for Medical Staff Idleness Minimization,” in Proceedings of the 4th Brazilian Technology Symposium (BTSym’18). BTSym 2018. Smart Innovation, Systems and Technologies, vol. 140, Iano Y., Arthur R., Saotome O., Estrela V. V., and Loschi H., Eds. Springer, Cham, Campinas, SP, Brazil, 2019. Padilha R., Iano Y., Monteiro A. C. B., Arthur R., and Estrela V. V. “Betterment Proposal to Multipath Fading Channels Potential to MIMO Systems,” in Proc. BTSym 2018. Smart Innovation, Systems and Technologies, vol. 140, Iano Y., Arthur R., Saotome O., Estrela V. V., and Loschi H., Eds. Springer, Cham, Campinas, SP, Brazil, 2019. Chilamkurti N., Zeadally S., Soni R., and Giambene G. “Wireless Multimedia Delivery over 802.11e with Cross-Layer Optimization Techniques,” Multimedia Tools and Applications, vol. 47, pp. 189–2051, 2010. Li Z., O’Brien L., Zhang H., and Cai R, “On the Conceptualization of Performance Evaluation of IaaS Services,” IEEE Transactions on Services Computing, vol. X, no. X, pp. 1–1, 2014. Wu Y., Lim J., and Yang M. “Object Tracking Benchmark.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, pp. 1834–1848, 2015. Marins H. R., and Estrela V. V. “On the Use of Motion Vectors for 2D and 3D Error Concealment in H.264/AVC Video. Feature Detectors and Motion Detection in Video Processing,” IGI Global, Hershey, PA, USA, 1, pp. 164–186, 2017. doi:10.4018/978-1-5225-1025-3.ch008

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0

17

[42] Almalkawi I., Zapata M., Al-Karaki J., and Morillo-Pozo J. “Wireless Multimedia Sensor Networks: Current Trends and Future Directions,” Sensors vol. 10, pp. 6662–6717, 2010. [43] Wang L., Liu M., and Meng M. Q.-H, “Real-Time Multi-Sensor Data Retrieval for Cloud Robotic Systems,” IEEE Transactions on Automation Science and Engineering (T-ASE): Special Issue on Cloud Robotics and Automation, vol. 12, no. 2, 2015, pp. 507–518. [44] Mahler J., Krishnan S., Laskey M., et al. “Learning Accurate Kinematic Control of Cable-Driven Surgical Robots Using Data Cleaning and Gaussian Process Regression,” in Proc. IEEE International Conference on Automation Science and Engineering (CASE), New Taipei, Taiwan, 2014. [45] Ciocarlie M., Pantofaru C., Hsiao K., Bradski G., Brook P., and Dreyfuss E. “A Side of Data With My Robot,” IEEE Robotics & Automation Magazine, vol. 18, no. 2, pp. 44–57, 2011. [46] Du Z., Yang W., Chen Y., Sun X., Wang X., and Xu C. “Design of a Robot Cloud Center,” in Proc. International Symposium on Autonomous Decentralized Systems, Tokyo & Hiroshima, Japan, 2011, pp. 269–275. [47] de Croon G., Gerke P., and Sprinkhuizen-Kuyper I. “Crowdsourcing as a Methodology to Obtain Large and Varied Robotic Data Sets,” in Proc. IEEE International Conference on Intelligent Robots and Systems (IROS), Chicago, Illinois, USA, 2014. [48] Deng J., Dong W., Socher R., Li L.-J., Li K., and Fei-Fei L. “Imagenet: A LargeScale Hierarchical Image Database,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 2009, pp. 248–255. [49] Wang J., Krishnan S., Franklin M. J., Goldberg K., Kraska T., and Milo T. “A Sample-and-Clean Framework for Fast and Accurate Query Processing on Dirty Data,” in Proc. ACM SIGMOD International Conference on Management of Data, Snowbird, Utah, USA, 2014. [50] Armbrust M., Stoica I., Zaharia M., et al. “A View of Cloud Computing,” Communications of the ACM, vol. 53, no. 4, p. 50, 2010. [51] Tudoran R., Costan A., Antoniu G., and Bouge L. “A Performance Evaluation of Azure and Nimbus Clouds for Scientific Applications,” in Proc. International Workshop on Cloud Computing Platforms – CloudCP ’12. ACM Press, Bern, Switzerland, 2012, pp. 1–6. [52] Tenorth M., and Beetz M. “KnowRob: A Knowledge Processing Infrastructure for Cognition-Enabled Robots,” International Journal of Robotics Research (IJRR), vol. 32, no. 5, pp. 566–590, 2013. [53] Tenorth M., Clifford Perzylo A., Lafrenz R., and Beetz M. “The RoboEarth Language: Representing and Exchanging Knowledge About Actions, Objects, and Environments,” in Proc. International Conference on Robotics and Automation (ICRA), Minneapolis, MN, USA, 2012, pp. 1284–1289. [54] Tenorth M., Perzylo A. C., Lafrenz R., and Beetz M. “Representation and Exchange of Knowledge about Actions, Objects, and Environments in the RoboEarth Framework,” IEEE Trans Aut Science and Eng (T-ASE), vol. 10, no. 3, pp. 643–651, 2013.

18

Imaging and sensing for unmanned aircraft systems, volume 2

[55]

Torralba A., Fergus R., and Freeman W. T. “80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 11, pp. 1958–1970, 2008. Turnbull L., and Samanta B. “Cloud Robotics: Formation Control of a Multi Robot System Utilizing Cloud Infrastructure,” in Proceedings of IEEE Southeastcon. IEEE, 2013, Jacksonville, FL, USA, pp. 1–4. Wang H., Ma Y., Pratx G., and Xing L. “Toward Real-time Monte Carlo Simulation Using a Commercial Cloud Computing Infrastructure,” Physics in Medicine and Biology, vol. 56, no. 17, pp. N175–N181, 2011. Riazuelo L, Civera J., and Montiel J. “C2TAM: A Cloud Framework for Cooperative Tracking and Mapping,” Robotics and Autonomous Systems, vol. 62, no. 4, pp. 401–413, 2013. van den Berg J., Abbeel P., and Goldberg K. “LQG-MP: Optimized Path Planning for Robots with Motion Uncertainty and Imperfect State Information,” International Journal on Robotics Research (IJRR), vol. 30, no. 7, pp. 895–913, 2011. Salmeron-Garcia J., Diaz-del Rio F., Inigo-Blasco P., and Cagigas D. “A Trade-off Analysis of a Cloud-based Robot Navigation Assistant using Stereo Image Processing,” IEEE Transactions on Automation Science and Engineering (T-ASE): SI on Cloud Robotics and Automation, vol. 12, no. 2, 2015, pp. 444–454. Nister D., and Stewenius H. “Scalable Recognition with a Vocabulary Tree,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, 2006, New York, NY, USA, pp. 2161–2168. Prestes E., Carbonera J. L., Fiorini S. R., et al. “Towards a Core Ontology for Robotics and Automation,” Robotics and Autonomous Systems, vol. 61, no. 11, pp. 1193–1204, 2013. Khamra Y. E., Kim H., Jha S., and Parashar M. “Exploring the performance fluctuations of HPC workloads on clouds.” Proc. 2010 IEEE Second International Conference on Cloud Computing Technology and Science, Indianapolis, IN, USA, 2010, pp. 383–387. Bhargava B., Angin P., and Duan L. “A Mobile-Cloud Pedestrian Crossing Guide for the Blind,” in Proc. International Conference on Advances in Computing & Communication, Kochi, Kerala, India, 2011. Riazuelo L., Tenorth M., Marco D., et al. “RoboEarth Semantic Mapping: A Cloud Enabled Knowledge-Based Approach,” IEEE Transactions on Automation Science and Engineering (T-ASE): Special Issue on Cloud Robotics and Automation, vol. 12, no. 2, 2015, pp. 432–443. Mohanarajah G., Hunziker D., Waibel M., and D’Andrea R. “Rapyuta: A Cloud Robotics Platform,” IEEE Transactions on Automation Science and Engineering (T-ASE), 12, pp. 1–13, 2014. Mohanarajah G., Usenko V., Singh M., Waibel M., and D’Andrea R. “Cloud-based Collaborative 3D Mapping in Real-Time with Low-Cost Robots,” IEEE Transactions on Automation Science and Engineering

[56]

[57]

[58]

[59]

[60]

[61]

[62]

[63]

[64]

[65]

[66]

[67]

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0

[68]

[69] [70] [71] [72] [73] [74]

[75]

[76]

[77]

[78]

[79]

[80]

[81] [82]

[83]

19

(T-ASE): Special Issue on Cloud Robotics and Automation, vol. 12, no. 2, 2015, pp. 423–431. Kumar V., and Michael N. “Opportunities and Challenges with Autonomous Micro Aerial Vehicles,” International Journal of Robotics Research (IJRR), vol. 31, no. 11, pp. 1279–1291, 2012. https://www.instructables.com/id/Build-Your-Own-Eco-Quad-Copter-fromReused-and-Rec/ Lipson H., and Kurman M. Fabricated: The New World of 3D Printing. Wiley, 2013. Winkler T., and Rinner B. “Security and Privacy Protection in Visual Sensor Networks: A Survey,” ACM Computing Surveys, vol. 47, no. 2, pp. 1–2, 42, 2014. Ren K., Wang C., Wang Q., et al. “Security Challenges for the Public Cloud,” IEEE Internet Computing, vol. 16, no. 1, pp. 69–73, 2012. Lin P., Abney K., and Bekey G. A., Robot Ethics: The Ethical and Social Implications of Robotics. The MIT Press, 2011. Astely D., Dahlman E., Furuskar A., Jading Y., Lindstrom M., and Parkvall S. “LTE: The Evolution of Mobile Broadband,” IEEE Communications Magazine, vol. 47, no. 4, pp. 44–51, 2009. Arumugam R., Enti V., Bingbing L., et al. “DAvinCi: A Cloud Computing Framework for Service Robots,” in Proc. International Conference on Robotics and Automation (ICRA), Anchorage, AK, USA, 2010, pp. 3084–3089. Li Z., Zhang H., O’Brien L., Cai R., and Flint S. “On Evaluating Commercial Cloud Services: A Systematic Review,” Journal of Systems and Software, vol. 86, no. 9, pp. 2371–2393, 2013. Gammeter S., Gassmann A., Bossard L., Quack T., and Van Gool L. “Server-Side Object Recognition and Client-Side Object Tracking for Mobile Augmented Reality,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2010, San Francisco, CA, USA, pp. 1–8. Chow H. Y., Lui K.-S., and Lam E. Y. “Efficient On-Demand Image Transmission in Visual Sensor Networks,” EURASIP Journal on Advances in Signal Processing, vol. 2007, pp. 1–11, 2007. Liu B., Chen Y., Blasch E., and Pham K. “A Holistic Cloud-Enabled Robotics System for Real-Time Video Tracking Application,” in Future Information Technology, 2014, pp. 455–468. Juve G., Deelman E., Berriman G. B., Berman B. P., and Maechling P. “An Evaluation of the Cost and Performance of Scientific Workflows on Amazon EC2,” Journal of Grid Computing, vol. 10, no. 1, pp. 5–21, 2012. Kamei K., Nishio S., Hagita N., and Sato M. “Cloud Networked Robotics,” IEEE Network, vol. 26, no. 3, pp. 28–34, 2012. Campelli L., Akyildiz I. F., Fratta L., and Cesana M. “A Cross-Layer Solution for Ultrawideband Based Wireless Video Sensor Networks,” in Proceedings of the IEEE Global Telecommunications Conference. New Orleans, LA, 2008, pp. 1–6. Khattak H. A., Ameer Z., Din I. U., and Khan M. K. “Cross-Layer Design and Optimization Techniques in Wireless Multimedia Sensor Networks

20

[84]

[85]

[86]

[87]

[88]

[89]

[90]

[91]

[92]

[93]

[94]

[95]

[96]

Imaging and sensing for unmanned aircraft systems, volume 2 for Smart Cities,” Computer Science and Information Systems, 16, 1–17, 2019. Chen M., Leung V. C., Mao S., and Li M. “Cross-Layer and Path Priority Scheduling Based Real-time Video Communications over Wireless Sensor Networks,” in Proc. IEEE Vehicular Technology Conference, Calgary, Canada, 2008, pp. 2873–2877. Lin C.-H., Shieh C.-K., Ke C. H., and Chilamkurti N. “An Adaptive CrossLayer Mapping Algorithm for MPEG-4 Video Transmission Over IEEE 802.11e WLAN,” Telecommunication Systems, vol. 42, pp. 223–234, 2009. Shu L., Zhang Y., Yu Z., Yang L. T., Hauswirth M., and Xiong N. “ContextAware Cross-Layer Optimized Video Streaming in Wireless Multimedia Sensor Networks,” The Journal of Supercomputing, vol. 54, pp. 94–121, 2010. Wan J., Xu X., Feng R., and Wu Y. “Cross-Layer Active Predictive Congestion Control Protocol for Wireless Sensor Networks,” Sensors, vol. 9, pp. 8278–8310, 2009. Costa D. G., and Guedes L. A. “The Coverage Problem in Video-Based Wireless Sensor Networks: A Survey,” Sensors, vol. 10, pp. 8215–8247, 2010. Chew L. W., Ang L.-M., and Seng K. P. “A Survey of Image Compression Algorithms in Wireless Sensor Networks,” in Proc. International Symposium on Information Technology, 2008, Kuala Lumpur, Malaysia, pp. 1–9. Zhang L., Hauswirth M., Shu L., Zhou Z., Reynolds V., and Han G. “MultiPriority Multi-Path Selection for Video Streaming in Wireless Multimedia Sensor Networks,” Lecture Notes in Computer Science, vol. 5061, pp. 439– 452, 2008. Lee J.-H. and Jun I.-B. “Adaptive-Compression Based Congestion Control Technique for Wireless Sensor Networks,” Sensors, vol. 10, pp. 2919–2945, 2010. Wang H., Peng D., Wang W., and Sharif H. “Optimal Rate-based Image Transmissions via Multiple Paths in Wireless Sensor Network,” in Proc. IEEE Conference Multimedia and Expo, Beijing, China, 2007, pp. 2146–2149. Nasri M., Helali A., Sghaier H., and Maaref H. “Adaptive Image Transfer for Wireless Sensor Networks (WSNs),” in Proc. International Conference on Design and Technology of Integrated Systems in Nanoscale, Hammamet, Tunisia, 2010, pp. 1–7. Qaisar S., and Radha H. “Multipath Multi-Stream Distributed Reliable Video Delivery in Wireless Sensor Networks,” in Proc. 43rd Annual Conference on Information Sciences and Systems (CISS 2009), 2009, pp. 207–212. Pudlewski S., and Melodia T. “On the Performance of Compressive Video Streaming for Wireless Multimedia Sensor Networks,” in Proc IEEE Ad-hoc, Sensor and Mesh Networking Symposium, Shuzou, China, 2010, pp. 1–5. Wu F., Gui Y., Wang Z., Gao X., and Chen G. “A Survey on Barrier Coverage with Sensors,” Frontiers of Computer Science, vol. 10, pp. 968– 984, 2016.

UAV-CPSs as a test bed for new technologies and a primer to Industry 5.0 [97]

[98]

[99]

[100] [101]

[102]

[103]

[104]

[105]

[106]

[107]

[108]

[109]

21

Akyildiz I., Melodia T., and Chowdhury K. “A Survey on Wireless Multimedia Sensor Networks,” Computer Networks, vol. 51, pp. 921–960, 2007. Razmjooy N., Ramezani M., and Estrela V. V. “A Solution for Dubins Path Problem with Uncertainties Using World Cup Optimization and Chebyshev Polynomials,” in Proc. BTSym 2018. Smart Innovation, Systems and Technologies, vol 140, Iano Y. et al. Eds. Springer, Cham, Campinas, SP, Brazil, 2019. doi: 10.1007/978-3-030-16053-1_5 Turnbull L., and Samanta B. “Cloud Robotics: Formation Control of a Multi Robot System Utilizing Cloud Infrastructure,” in Proceedings of IEEE Southeastcon. IEEE, 2013, Jacksonville, FL, USA, pp. 1–4. Waibel M., Beetz M., Civera J., et al. “RoboEarth,” IEEE Robotics & Automation Magazine, vol. 18, no. 2, pp. 69–82, 2011. Razmjooy N., Ramezani M., Estrela V. V., Loschi H. J., and do Nascimento D. A. “Stability Analysis of the Interval Systems Based on Linear Matrix Inequalities,” in Proc. BTSym 2018. Smart Innovation, Systems and Technologies, vol. 140, Iano Y. et al. Eds. Springer, Cham, Campinas, SP, Brazil, 2019. Razmjooy N., Razmjooy, N., Estrela, V. V., et al. “Image Transmission in MIMO-OSTBC System over Rayleigh Channel Using Multiple Description Coding (MDC) with QPSK Modulation,” in Advances in UAV Avionics for Imaging and Sensing, vol. 2, IET, 2019. Mekonnen T., Porambage P., Harjula E., et al. “Energy Consumption Analysis of High Quality Multi-Tier Wireless Multimedia Sensor Network,” IEEE Access, vol. 5, pp. 15848–15858, 2017. Yu W., Sahinoglu Z., and Vetro A. “Energy Efficient JPEG 2000 Image Transmission over Wireless Sensor Networks,” in Proceedings of the IEEE Globecom, Dallas, Texas, USA, 2004, pp. 2738–2743. Lecuire V., Duran-Faundez C., and Krommenacker N. “Energy-Efficient Image Transmission in Sensor Networks,” International Journal of Sensor Networks, vol. 4, pp. 37–47, 2008. Wang W., Peng D., Wang H., Sharif H., and Chen H.-H. “EnergyConstrained Distortion Reduction Optimization for Wavelet-Based Coded Image Transmission in Wireless Sensor Networks,” IEEE Transactions on Multimedia, vol. 10, pp. 1169–1180, 2008. Wu H., and Abouzeid A. “Energy-Efficient Distributed Image Compression In Resource-Constrained Multihop Wireless Networks,” Computer Communications, vol. 28, pp. 1658–1668, 2005. Aghdasi H. S., Abbaspour M., Moghadam M. E., and Samei Y. “An Energy-Efficient and High-Quality Video Transmission Architecture in Wireless Video-Based Sensor Networks,” Sensors, vol. 8, pp. 4529–4559, 2008. Ferrigno L., Marano S., Pacielo V., and Pietrosanto A. “Balancing Computational and Transmission Power Consumption in Wireless Image Sensor Networks,” in Proc. 2005 IEEE Conference on Virtual

22

[110]

[111]

[112]

[113]

[114]

Imaging and sensing for unmanned aircraft systems, volume 2 Environments, Human–Computer Interfaces and Measurement Systems (VECIMS 2005), Messina, Italy, 2005, pp. 61–66. Politis I., Tsagkaropoulos M., Dagiuklas T., and Kotsopoulos S. “Power Efficient Video Multipath Transmission Over Wireless Multimedia Sensor Networks,” Mobile Networks and Applications, vol. 13, pp. 274–284, 2008. Franc¸a R. P., Peluso M., Monteiro A. C. B., Iano Y., Arthur R., and Estrela V. V. “Development of a Kernel: A Deeper Look at the Architecture of an Operating System,” in Proc. BTSym 2018. Smart Innovation, Systems and Technologies, vol 140, Iano Y. et al. Eds. Springer, Cham, Campinas, SP, Brazil, 2019. Estrela V. V., Khelassi A., Monteiro A. C. B., et al. “Why software-defined radio (SDR) matters in healthcare?”. Medical Technologies Journal, 2019;3 (3):421–429, doi:10.26415/2572-004X-vol3iss3p421-429. Estrela V. V., Hemanth J., Saotome O., Grata E. G. H., and Izario D. R. F. “Emergency response cyber-physical system for flood prevention with sustainable electronics.” In: Iano Y. et al. (eds), Proceedings of the 3rd Brazilian Technology Symposium. BTSym 2017. Springer, Cham, Campinas, SP, Brazil, 2019. Stenmark M., Malec J., Nilsson K., and Robertsson A. “On Distributed Knowledge Bases for Small-Batch Assembly,” IEEE Transactions on Automation Science and Engineering (T-ASE): SI on Cloud Robotics and Automation, vol. 12, no. 2, 2015.

Chapter 2

UAS human factors and human–machine interface design Yixiang Lim1, Alessandro Gardi1 and Roberto Sabatini1

The human–machine interface (HMI) is a crucial yet often overlooked aspect in the design of Unmanned Aircraft Systems (UASs). A properly designed HMI enhances situational awareness and reduces the workload of the ground pilot, thereby contributing to improving the overall mission performance. Typically, a Human Factors Engineering (HFE) program provides a methodological process to support good design. The program comprises three iterative stages: requirements analysis and capture, design and evaluation. A number of approaches can be adopted in the HFE program but given the wide range of applications and missions that are being undertaken by different types of UAS, it is advantageous to adopt a functional approach towards HMI design, where the HMI is designed around specific functions to be performed by either the human user or the system. The typical UAS functions include mission planning, sensor operation, data analysis and sense-andavoid (SAA), and can also extend to multi-platform coordination and collaborative decision-making. The human factors considerations and the associated HMI elements supporting these functionalities are discussed.

2.1 Introduction Unmanned Aircraft Systems (UASs) possess increased automation and autonomy to compensate for the limitations associated with the pilot being physically displaced from the flight deck. UASs aim at higher levels of independence and decision-making ability (e.g., in Guidance, Navigation and Control (GNC) modules [1]) when compared with manned counterparts. Unmanned Aerial Vehicle (UAV) command and control can vary from more tactical modes, such as manual piloting, to a more strategic nature, requiring the management and coordination of multiple manned and unmanned platforms. Therefore, the design of UAS Human–Machine Interfaces and Interactions (HMI2) faces different challenges , especially in more complex operations involving collaboration between

1

RMIT University – School of Engineering, Bundoora, Australia

24

Imaging and sensing for unmanned aircraft systems, volume 2

human operators and autonomous agents. Some Human Factors Engineering (HFE) considerations include [2–5]: ●

●

●

●

●

●

●

●

Certain sensory (vestibular, haptic, auditory) cues that are present within the flight deck are not readily available in a remote piloting station. The latency and performance of data links can restrict the ability of the ground pilot to provide tactical or immediate commands to the UAV. The previous limitations might hinder detection of abnormal operating situations and subsequent recovery from such scenarios. Missions involving long periods of low activity (e.g., when performing highly automated tasks), leading to ‘out-of-the-loop’ effects characterised by increased boredom, decreased attention and loss of situational awareness. Transfer of UAV platform control or sharing of mission tasks between different pilots/stations, particularly for piloting teams in different geographical locations. Management of multiple UAV by a single ground operator, requiring low-level piloting tasks to be automated to allow the ground operator to focus on the higher-level decision making and coordination tasks. Establishing trusted autonomy with systems that provide integrity monitoring and performance assurance capabilities. Human–machine collaboration requiring human operators to cede partial control or decision authority to automated or autonomous systems.

The UAS Ground Control (GC) segment contains the system elements supporting the control and coordination of one or multiple UAV platforms by remote crewmembers. UAS GC segments vary in size, from small mobile units to large centres housing many human operators. Three levels of increasing size (unit, station, centre) are provided to illustrate the different scales of operations. A GC Unit (GCU) is handled by a single remote pilot and is designed to be highly portable, containing all the necessary functionalities and HMI2 for the remote pilot to execute the mission, often with minimal external dependencies. As illustrated in Figure 2.1, a typical GCU is a laptop or handheld unit providing the ground pilot with tactical control of UAVs in the Visual Line of Sight (VLOS). A GC Station (GCS) is usually a deployable structure comprising multiple integrated hardware and software components. The Heterogeneous Airborne Reconnaissance Team (HART) GCS (Figure 2.2) is a GCS example that supports ground forces, integrating intelligence, surveillance and reconnaissance information from multiple manned and unmanned platforms in near real time. The HART GCS communicates with a command and control centre, which coordinates with other networked GCSs to prioritise and allocate tasks and requests. Several human operators are typically involved in manning a GCS, and as described in Peschel and Murphy [6], generic roles assumed by human operators include mission organisation and planning, UAV piloting and control, sensor operation, as well as data processing, exploitation and dissemination. While current GCSs allow the command and control of individual UAV platforms, GCS design is evolving to support multi-platform operations, where multiple platforms can be coordinated by multiple operators [7], thereby allowing for the execution of more complex missions.

UAS human factors and human–machine interface design

25

Figure 2.1 An example of a handheld GCU

Figure 2.2 An example of a GCS, the HART system, courtesy of Northrop Grumman Corporation

A GC Centre (GCC), like NASA’s Global Hawk Operations Centre (Figure 2.3), is a fixed structure housing numerous computers, displays and controls. Centres integrate elements of GCUs and GCSs within a centralised facility, allowing complex missions to be carried out and can also support the coordination of multiple UAV platforms. GCCs are characterised by the need to process and disseminate significant amounts of data, strategic mission planning and high-level decision-making, as well as the coordination of plans and objectives across multiple entities (such as Air Traffic Management (ATM) as well as other centres, stations and units).

26

Imaging and sensing for unmanned aircraft systems, volume 2

Figure 2.3 An example of a GCC – the NASA Global Hawk Operations Centre, image courtesy of NASA Depending on mission and operational needs, reconfigurable and customisable GC displays can support human operators in the traditional ‘aviate, navigate, communicate and manage’ tasks which are also performed in manned aircraft. GC interfaces can include display and control elements related to the following categories: ● ● ● ● ● ● ●

Primary flight status Navigation and traffic information Trajectory planning and mission management System status information and vehicle health management Sensor feed and payload control Mission objectives and task progress Communication with other human operators, via either text or voice.

2.2 UAS HMI functionalities Increasing automation has allowed manual flight tasks to be taken over by automated functions, allowing remote pilots to focus on managing and monitoring the flight and mission systems. Table 2.1 provides a comparison of UAS crew tasks [6,8–10]. As seen in the table, there is an emphasis on the types of management tasks, which are divided into three categories – the systems, data and mission management. Data and mission management are tasks unique to UAS operations and depend mostly on mission and operational requirements. Data management is related to the processing and utilisation of sensor data, whereas mission management is related to the tasking and allocation of UAS resources. Although civil flight decks also include some aspects of data and mission management, they are not considered one of the primary tasks of the flight crew.

Table 2.1 UAS ‘aviate, navigate, communicate and manage’ tasks Tasks

Nehme et al. [8]

Ashdown et al. [9]

Peschel and Murphy [6]

Ramos [10]

Aviate and navigate

– Supervision and optimisation of UAV position Negotiating with and notifying relevant stakeholders Monitoring UAV health and status Monitoring network communications Monitoring payload status

UAV operation –

UAV teleoperation Navigation

Aircraft command and control

Communications

Communications with other teams

Vehicle systems management Payload systems management

Monitor the vehicle’s health

Voice communications Dissemination of payload products Aircraft systems monitoring

Analysing sensor data

Payload data management Data processing Target detection

Communicate Manage – systems

Manage – data

Manage – mission

Monitoring for sensor activity Positive target identification Tracking target Allocation and scheduling of resources Path planning supervision

Monitoring data link status Payload delivery

Tasking of assets Scheduling Route planning Sensor coverage planning

Payload command and control

Sensor operation Visual inspection and tactical direction

Exploitation of payload products

Strategy, planning and coordination

Mission planning and replanning

28

Imaging and sensing for unmanned aircraft systems, volume 2

2.2.1

Reconfigurable displays

The design of UAS interfaces have evolved from the multifunctional display concept used on modern flight decks. Elements of modern Primary Flight Displays, Navigation Displays, Flight Management Systems and Crew Alerting Systems can be seen in many GC displays. Additional features have been introduced, allowing for monitoring of the data-link quality, as well as supporting mission planning and sensor management. Reconfigurable displays allow switching between different display modes and can fulfil multiple functions, providing greater flexibility and efficiency in the use of display screens.

2.2.2

Sense and avoid

SAA is an essential requirement for UAS to operate alongside human-crewed aircraft in unsegregated airspace. Several concepts (such as JADEM [11], MuSICA [12] and ACAS Xu [13]) have emerged in the literature as extensions of existing aircraft collision avoidance systems. While such concepts provide a pathway towards autonomous self-separation in the near future, from the HFE perspective, HMI design should account for the need for human intervention (e.g., in the event of degraded systems performance or off-nominal scenarios), provide appropriate cautions or warnings, along with the necessary information to achieve timely deconfliction.

2.2.3

Mission planning and management

Mission planning and management accounts for the different variables needed to achieve the mission objectives. Typically, the mission plan contains information on the routes, targets and payloads as well as communications and data-link quality. The GC functionalities supporting pre-flight mission planning and management can include: ●

●

● ●

Capability to optimise the mission plan according to different mission objectives (e.g., minimising fuel, maximising sensor coverage, etc.) based on inputs from various sources (e.g., airspace and terrain databases, as well as weather and traffic forecast information) Capability to conduct a risk analysis of the mission, modelling potential damage to personnel and property as a result of forced landings, as well as the effects of any loss or degradation of navigation or communication performance on the overall mission Generation of contingency plans Performing pre-flight rehearsal and simulations.

2.2.4

Multi-platform coordination

Multi-UAV coordination and control allows a single remote pilot to assume control of multiple aircraft, requiring an even greater shift of the pilot’s role towards systems management and mission coordination, with routine tasks being delegated to automation. Different multi-platform coordination paradigms exist in the literature, including: ●

Swarming, where multiple autonomous platforms with the capability to selforganise are commanded as several homogeneous entities

UAS human factors and human–machine interface design ●

●

29

Distinct management of heterogeneous vehicles, typically with different performance limits, equipped with different payloads and having different goals Multi-operator, multi-platform management, with human operators distributed over possibly vast geographical distances coordinating tasks and actions.

Table 2.2 provides further elaboration on these operational concepts. Compared with single pilot operations, multi-pilot operations require greater interactions between different entities, in turn reflecting increasing complexity. The top and bottom rows in the table describe different concepts of operations, with multiplatform operations implicitly requiring UAV platforms to possess greater automation. While the interfaces of single platform operations are designed to support manual piloting tasks, the interfaces designed for multi-platform operations need to assist remote operators in supervising and managing multiple UAV platforms, and draw inspiration from elements of ATC, ATM or UAS Traffic Management (UTM) displays in which the human operator assumes the role of a coordinator [14].

2.3 GCS HMI elements HFE in systems design encompasses the design of suitable HMI2 in line with system requirements. Interface design relates to defining the content of information presented to the operator as well as specifying the format in which the information is presented, while the design of HMIs entails defining the behaviour of the system in response to the user and environmental inputs in both nominal and off-nominal conditions. Good interface design leads to HMI allowing operators to accomplish a given task effectively, efficiently and consistently, with minimal training for use, and that can be easily maintained. Interface design can refer to physical design elements (e.g., of the workstation environment) or digital elements (e.g., of specific displays, alerts or control devices). Workstation: The operating environment for the user is designed such that the placement of consoles and work surfaces in the workstation allow the operator to perform his/her task effectively and comfortably. Design elements include consideration of user anthropometry, environmental stressors as well as safety. Displays/alerts: The visual, audial or haptic devices is designed to present information in an efficient and intuitive manner to the operator. Presented information can take the form of readable messages, graphs, charts, indicators and symbols or voice/sound/motion alerts. Design elements can be classified as physical characteristics (e.g., display size, resolution, placement, etc.); information characteristics (e.g., display formats, alerting conditions, information prioritisation, etc.) and information content (e.g., accuracy, update rate, clutter, level, etc.). Table 2.3 lists key design considerations for these three elements. Controls: The inputs, devices and surfaces which are used to provide inputs and to control the system; they might include buttons, knobs, levers and switches, as well as keyboards, mouse, joystick, touchpad or voice-activated systems. The design aspects of a control interface are presented in Table 2.4 and include the design of control’s physical characteristics, placement, function and user interaction.

Table 2.2 UAS operational concepts and HMI2

Single Platform Ops

Multiple Platform Ops

Single remote operator

Multiple remote operators

Concept of operations (1)

Concept of operations

ü Manual operations of platforms possessing low automation (e.g., low SWAP-C micro and very small UAS) ü Parallels can be drawn with the flight decks of single-pilot general aviation aircraft Concept of operations (2) ü Management of platforms possessing relatively higher automation (e.g., MALE, HALE) ü HMI comprises multiple displays, allowing ground pilots to monitor different subsystems ü HMI can be multifunctional, providing some degree of adaptiveness ü Dedicated displays can be used to perform different functions ü Parallels can be drawn with single-pilot military cockpits Concept of operations (1) ü Swarming, where multiple platforms are commanded either as a single entity or as multiple homogeneous entities ü Swarm behaviour can be modified in several different ways [15]:
Modifying specific swarm characteristics, tactics or algorithms (e.g., through playbooks)
If more precision is required, swarm leaders may be directly controlled Concept of operations (2) ü Management of multiple platforms, where each platform is treated as an individual entity ü Platforms could be either homogeneous or heterogeneous ü Parallels can be drawn with air traffic control stations

ü Control of more complex UAV possessing low automation ü Each operator assumes a different functional role and the HMI are tailored to the different operator roles ü Parallels can be drawn with multi-crew flight decks

Concept of operations ü Coordination of autonomous UAV platforms ü Tasks can be dynamically reallocated between different operators Introduces elements of network-centric operations, requiring sharing of information and collaborative decision-making

UAS human factors and human–machine interface design

31

Table 2.3 Design aspects and key considerations for displays and alerts (from [16]) Design aspects Physical characteristics

Information characteristics

Information content

Key considerations ü Display parameters (e.g., screen resolution, luminance, dimming and contrast) should be adjustable to allow readability in operational conditions. ü The display should be large enough to contain all information in a visible format. ü The screen or information refresh rate should be sufficiently high to prevent flickering or jitter effects. ü The layout of the displays should account for any possible obstructions affecting the user’s visibility. ü Displays should be grouped logically. ü Visual/aural alerts should be sufficiently visible/audible to the operator. ü Information should be consistently ordered under nominal conditions. ü The primary field of view should contain critical or frequently used displays, while the secondary field of view (requiring the user to turn his/her head) can contain ancillary and advisory information. ü Pop-up text or windows should not interfere with the use of the display. However, time-critical warnings should appear in the operator’s primary field of view. ü Information display philosophy (colour coding, symbology, formats, placement, etc.) should be consistent across the flight deck to promote familiarisation and to minimise user workload and error. ü Information elements should be distinct and permit users to recognise the source of the information elements when multiple sources are providing the same information. ü Similar information elements should be grouped and arranged in a logical manner to allow users to easily identify the required information. ü Alerts should be prioritised according to urgency and criticality, and triggered with sufficient time for the user to react. ü Frequent false alarms or missed detections should be avoided as they increase workload and reduce user trust in the alerting system. ü Information specification (from the requirements capture) should be used as a guideline when designing the interface. This can include integrity, precision and refresh rate requirements. ü Displayed/annunciated information should have sufficient latency and be sufficiently detailed to allow users to accomplish his/her task at hand. ü Caution, warning or failure flags should be presented in the location of the information they refer or replace. ü The meaning of symbols should be sufficiently clear to avoid any misinterpretation. ü Graphical elements should be presented at an appropriate clutter level to prevent any confusion to the user and to perform the task at hand. ü A consistent design philosophy should call attention to conditions, presentation, urgency and prioritisation.

32

Imaging and sensing for unmanned aircraft systems, volume 2

Table 2.4 Design aspects and key considerations for control interfaces (summarised from [16]) Design aspects Physical characteristics

Placement

Function

User interaction

Key considerations ü Should be designed to prevent jamming, chafing, any form of interference or significant wear-and-tear. ü The physical appearance of control devices should be made differentiable through form, colour, location and labelling (e.g., pushbuttons should be differentiated from rotational knobs). ü Controls should be grouped and arranged logically and consistently. ü Frequently used and/or critical controls should be easily accessible. ü The number of actions required to actuate frequently used controls should be minimised. ü Controls that serve multiple functions should indicate the current function or allow the current function to be easily queried by users. Use of multi-function controls should not result in unacceptable levels of workload, error rates, speed and accuracy. Hidden functions or excessively modes complex should be avoided. ü Where multiple controls are required to actuate a function, the different control devices should be designated and clearly labelled, and should not result in confusion or any inadvertent operation during use. ü Should provide to users any necessary information at a resolution, latency and priority appropriate to the task at hand. ü Should allow users to provide input at an appropriate speed, precision and accuracy based on the task at hand. ü Data entry should allow users to recover from input errors easily. ü Users should be able to override the control authority of automated functions. ü Appropriate feedback notifying users of any changes in the control mode should be provided (e.g., in noisy conditions, aural notifications should be accompanied by other forms of feedback). ü In multi-crew situations, feedback make all crew members aware of control changes. ü Feedback should not result in false alarms or missed detections. ü Should not result in operator confusion due to any possible inconsistencies within the interface or with other interfaces.

When designing HMIs, the task and functional allocation between the human operator(s) and the automated system(s) is specified, along with the system’s behaviour and its impact on the user’s cognitive processes. These cognitive processes include the user’s workload, situational awareness and vigilance. HMIs need to be carefully designed in scenarios where control and decision authority is shared between human users and an automated system. The interactions must ensure that the task can be completed efficiently and safely at the specified levels of

UAS human factors and human–machine interface design

33

Table 2.5 Levels of automation [17] Levels of automation

1. Manual control 2. Action support 3. Batch processing 4. Shared control 5. Decision support 6. Blended decision-making 7. Rigid system 8. Automated decision-making 9. Supervisory control 10. Full automation

Roles Monitoring

Generating

Selecting

Implementing

Human Human/ computer Human/ computer Human/ computer Human/ computer Human/ computer Human/ computer Human/ computer Human/ computer Computer

Human Human

Human Human

Human

Human

Human Human/ computer Computer

Human/ computer Human/ computer Human/ computer Computer

Human

Human/ computer Human

Computer

Human/ computer Computer

Computer

Computer

Computer

Computer

Computer

Computer

Computer

Human

Human/ computer Computer Computer

automation. Automation can be made adaptive by accounting for the system’s or the user’s state, such as decreasing the level of automation and allowing for user intervention during periods of degraded system performance, or increasing the level of automation to provide additional assistance during increased workload or task complexity. The 10 levels of automation presented in Table 2.5 are based on Endsley and Kaber’s taxonomy [17] and comprise four functions: 1. 2. 3. 4.

Monitoring of system status through visual, audial or other means Generating options or strategies for achieving goals Selecting a strategy Implementing the selected strategy.

2.4 Human factors program HFE is defined as the application of knowledge about human capabilities and limitations to the design/development of systems and/or equipment to achieve efficient, effective and safe system performance at minimum cost and manpower, skill and training demands. HFE provides assurance that the working environment, facilities and equipment, as well as the system and user tasks are designed to be compatible with the physical, perceptual and cognitive attributes of personnel who are responsible for supporting, maintaining and operating them. A HFE program is established to support the design, test and integration of particular hardware, software, systems or

34

Imaging and sensing for unmanned aircraft systems, volume 2

work environments to meet human–machine performance requirements, and can be accomplished by: 1. 2. 3.

4.

5.

6.

Defining the required performance requirements to accomplish operational or mission objectives Establishing workflow procedures based on analysing the required task and functional allocation Designing suitable interfaces, which include the hardware, software and system components as well as the work environment, to enable human users to execute their tasks successfully Identifying critical tasks and design elements that can potentially compromise the safety, reliability, efficiency or effectiveness of the total human–machine performance Assessing the user and system performance through test and evaluation, as well as obtaining feedback on the usability of the designed product. The assessment is used in subsequent redesign of the system interfaces or work procedures. Evaluating the system life cycle cost that includes the operational, maintenance, support and training costs.

FAA/HF-STD-004 provides additional details on common HFE requirements and evaluation procedures [18]. The HFE program, as illustrated in Figure 2.4, comprises three main stages of requirements: definition, design and evaluation. These three stages are usually iterated over multiple cycles, with progressive iterations providing more detail and refinement to the requirements, procedures and system design. The definition, design and evaluation activities should be planned out such that they can be performed concurrently, while also allowing flexibility for successive evaluations to

Top-level requirements

Detailed requirements capture

Design and development

Test, evaluation and validation

Functional/task requirements

Mission/operational requirements

Preliminary subsystem design

Procedure development

Computer simulation and CAD analysis

Proposed design, procedural, equipment or training changes

Operational, maintenance and training documentation

Hardware/ software development

Mock-ups

Failure and error analysis; performance analysis; cost analysis

Human factors testing

Figure 2.4 HFE process

UAS human factors and human–machine interface design

35

influence the future stages. In detail, an HFE program comprises the following activities: 1.

2.

Requirements definition: The requirements provide a framework for identifying specific design elements and also support the assessment of these design elements against specific objectives, thereby driving the design and evaluation process. Initial requirements can be later refined based on the assessment of previous design stages. 1.1. Development of top-level requirements: The top-level requirements are obtained from the customer and specify the product’s purpose (the hardware, software or system being designed). As the top-level requirements form the initial technical baseline for the design and development phases, they should be captured with sufficient detail. 1.2. Definition of mission and operational requirements: These requirements define the mission environment, objectives, operational constraints, as well as performance and manpower requirements. An operating scenario would be a representative case for evaluating the system performance and success – the scenario can be modified and refined based on design and evaluation needs. 1.3. Capture of functional and task requirements: These requirements define the functions and tasks needed to accomplish the proposed mission. The tasks are described in detail along with the conditions in which they are executed. Similar to the mission and operational requirements, these are refined over the design process. Design: Product design encompasses designing and developing procedures, software, systems and equipment meeting specifications established in the requirements definition stage. An initial conceptual design provides the framework for subsequent refinements. Based on the requirements, the initial design should specify the information-flow, task sequencing, system layout, as well as the allocation of tasks and functions. Subsequently, more specific requirements and design objectives can be used to assess the product design, allowing for more detailed refinements to be made. To ensure an efficient development cycle, human factors principles should be applied throughout the entire design process, and just not at the final stages of detailed design. 2.1. Procedural development: It includes defining the procedures and guidelines for staffing, operations, maintenance and training, along with accompanying documentation. 2.2. System design: It includes defining the system architecture, information flow and functional allocation between the system/subsystem’s various modules and components. The various subsystems and components are designed and tested individually in the early design stages so that they can later be integrated into the main system. 2.3. Interface design: Interface design entails developing the HMI2 components of the system. The individual software and hardware components can also be designed and assessed independently of each other and integrated in later design stages.

36 3.

Imaging and sensing for unmanned aircraft systems, volume 2 Evaluation: The process assesses the effectiveness of design in accomplishing the mission objectives against specific performance requirements. Critical areas are identified where the total performance and safety of the operator/ system might be compromised (e.g., tasks leading overloading or underloading of the user). Based on the evaluation, recommendations are made to improve specific design elements and are fed back into the requirements and design of future iterations. 3.1. Analytical evaluation: Heuristic methods are used to determine whether the design meets requirements. This can involve the use of block diagrams, checklists and matrices. 3.2. Mock-up studies: Product mock-ups and prototypes are used early in the evaluation process as a relatively quick method of assessing design concepts. Environment mock-ups can be built from cardboard or foam, and interface displays can be sketched on a paper. These mock-ups provide users with a feel of the product during usability testing, supporting the evaluation. 3.3. Computer-aided evaluation: Computer-Aided Design (CAD) tools are employed to evaluate the human environmental factors by simulating virtual operators in a virtual work environment. At more mature evaluation phases, simulators are used to evaluate the performance of a real operator in a virtual work environment. Software validation and verification and system-level testing tools are also used for computer-aided evaluation. 3.4. Human test and evaluation: Human-in-the-loop evaluations are conducted to evaluate the system performance in a representative mission environment. The evaluation is conducted in stages and typically ranges incrementally from testing individual components to evaluating an integrated system to full-field evaluation. Subjective user (workload/ usability) ratings are employed to assess human performance aspects such as workload, situational awareness and usability. 3.5. Trade studies: They are used to decide between particular design features and are usually evaluated against metrics such as implementation and life cycle cost, effectiveness meeting mission goals, usability, as well as operator safety and risk.

2.4.1

Requirements definition, capture and refinement

Requirements are refined, modified and used throughout the entirety of software development process. As such, it is essential for proper management, documentation and maintenance of requirements throughout the development life cycle. Various types of requirements serving different functions are listed and described in Table 2.6. Requirements should satisfy the following attributes: Specific: The requirement must be specific in that only one aspect of the system design or performance must be addressed. The requirement must be expressed in

UAS human factors and human–machine interface design

37

Table 2.6 Different types of requirements Types of requirements

Description of requirements

Customer requirements

Factual statements or assumptions about the customer’s expectations of the system in terms of mission objectives, operating environment and constraints, as well as measures of effectiveness and suitability The necessary task, action or activity that must be accomplished, subsequently used as the top-level functions when performing functional analysis The level to which a mission, function or task must be executed; generally measured in terms of quantity, quality, coverage, timeliness or readiness Requirements expressed in technical data packages and technical manuals specifying what is needed in the building/coding/purchasing of a product and the processes involved in using the product Requirements implied or extracted from higher-level requirements

Functional requirements Performance requirements Design requirements Derived requirements Allocated requirements

A requirement created when a higher-level requirement is divided or allocated into multiple lower-level requirements

terms of need (what is needed, why it is needed, how well it needs to be done) instead of specifying how the need can be achieved. Measurable: The requirement must have performance metrics that can be measured and must be stated such that it can be verified in an objective and preferably quantitative manner. Achievable: The requirement must reflect a need or objective where a technically achievable solution can be found at relatively affordable costs. Relevant: The requirement must have the appropriate detail for the level of system hierarchy for which it is specified. It should not be excessively detailed to constrain solutions for the current level of design (e.g., a system-level specification would not contain detailed component requirements). Traceable: The lower-level requirements must flow from and support higherlevel requirements. Requirements that cannot be traced to a parent need to be assessed for the necessity of inclusion. Complete: The requirement must be able to clearly describe the customer’s needs (e.g., the concept of operations, utilisation environments and constraints, mission profiles and maintenance needs). Consistent: There must not be conflicting requirements and conflicts must be resolved to ensure consistency. The requirements engineering process is illustrated in Figure 2.5 and comprises an iterative cycle of requirements gathering, analysis, specification, validation and documentation, as well as modelling. Requirements are captured during the gathering phase through interviews, focus groups or team meetings. The requirements are analysed to identify the domain constraints, and any gaps and ambiguity in the requirements are corrected to ensure consistency. They can then

38

Imaging and sensing for unmanned aircraft systems, volume 2 • Find out what problems need to be solved and identify the system boundaries • Use questionnaires and surveys, interviews analysis of existing documentation • Group elicitation techniques • Others

Systems requirements

Requirements gathering and analysis

Requirements specification • Planning and gathering requirements • Mathematical modelling • Functional modelling • Partitioning • Prototyping Modelling • Others

Requirements validation

• Formal technical reviews (users) • Others Requirements documentation

Figure 2.5 Requirements engineering process

be sorted, prioritised and formally expressed in the specification phase (e.g., as a graphical or mathematical model). The validation phase reviews the specifications to ensure that they are consistent, correct, understandable, realistic, testable and traceable. When they are satisfactory, the requirements and models are appropriately documented.

2.4.2

Task analysis

To support user-centred design and development of systems and equipment, where HMI2 are designed around the user’s needs and requirements, task and functional analyses are used to translate top-level requirements into working procedures and specific design criteria. Functions are discrete processes necessary to achieve specific objectives, and tasks are the sequential steps required to achieve a specific function; they are performed through the use of facilities, equipment, software, personnel or a combination. Generally, an analysis is performed first by collecting information on the function and task through interviews, surveys or from a review of existing standards/literature. In the allocation process, functions are partitioned based on a logical sequence to minimise user workload and control interfaces. The result of the design process is a functional architecture, where the system is described in terms of its constituent functions. Some tools for the task and functional analysis are discussed later.

2.4.3

Hierarchal task analysis

Hierarchal task analysis organises the tasks and functions in a hierarchal manner using block diagrams, which represent the top–down relationship between objectives and tasks. Top-level objectives are broken into lower-level sub-objectives, which can themselves be divided into lower-level tasks and sub-tasks. A complex

UAS human factors and human–machine interface design

39

task can thus be represented by a hierarchy of simple actions, which, when undertaken sequentially, allow the parent objective to be achieved. The granularity of detail (or the number of sub-levels) in each task depends on several factors, such as the task complexity or criticality, risk of human error or the complexity of a particular task. Hierarchal task analysis allows different aspects of the task to be described with varying levels of detail and serves as an effective method of organising and representing activities. The hierarchy of tasks allows a preliminary analysis to be conducted to determine the human user’s workload, supporting subsequent allocation of system functions.

2.4.4 Cognitive task analysis To identify and diagnose particular errors and failure modes of tasks with more complex HMIs, a cognitive task analysis can be performed to understand the human user’s underlying mental processes. The cognitive task analysis starts with a list of activities (such as one obtained from a gross task analysis [19]) describing the tasks, sub-tasks and user control actions. The list is then expanded and updated to model the operator’s mental processes by analysing the cognitive cues, information flow requirements and decision-making strategies in particular scenarios. As described in Salmon et al. [20] and McIlroy and Stanton [21], a five-phase analysis process is used to analyse the work domain, control task, strategies, social organization and Cooperation, as well as Worker Competencies. Similar to the hierarchal task analysis, various methodologies are employed in each phase for information acquisition and representation. With sufficient detail, the cognitive work analysis can eventually be used as a tool to produce formal procedural guides and requirements specifications. The decision ladder is one of the results of the cognitive task analysis [22] and describes the information-processing activities at each stage of a user’s decisionmaking process when performing a task. The set of paths represents a decisionmaking strategy through the decision ladder, which is available to the operator and can involve varying levels of situation analysis/diagnosis and action planning. For example, the operator can take a path that bypasses certain processes, based on either heuristics, pre-defined operational procedures or due to time constraints. The cognitive task analysis provides information on the cognitive processes required from the operator, identifies available cognitive strategies and requirements and is ultimately used to drive the systems-level definition of the related human factors requirements. These requirements include information and interface requirements, system performance requirements, as well as training prerequisites.

2.4.5 Critical task analysis A critical task analysis (CTA) is used to evaluate the human factors involved in safety- and mission-critical tasks. The process is similar to cognitive task analysis, consisting of acquiring data from critical tasks, using tables and/or diagrams to represent the tasks, as well as identifying failure modes and factors that influence performance in all operational scenarios, including degraded and emergency situations. The CTA can be used in system design to minimise risk and improve

40

Imaging and sensing for unmanned aircraft systems, volume 2

Table 2.7 List of content requirements for a CTAR Information category Task hierarchical relationships Information flow

Cognitive processes

Physical processes

Workspace environment

Information description ü Required actions to accomplish the task ü Probability and severity of the human error ü The potential for error recovery ü Information required and available to the operator/maintainer. ü Feedback informing operator/maintainer of the adequacy of action(s) taken ü Communications required, including the type of communication ü Operator/maintainer interaction ü Decision evaluation process ü The decision reached after evaluation ü The number of personnel required; their specialities/ experience ü Performance limits of personnel ü Actions are taken for accomplishing the task ü Time available for completing the task ü Body movement required by action taken ü Frequency and tolerance of action ü A number of personnel required; their specialties/experience ü Performance limits of personnel ü Workspace envelope required/available to take action. ü Location and condition of the work environment ü Tools and equipment required ü Job aids, training or references required ü Hazards involved ü Operational limits of hard/software

performance. The format and content requirements for a CTA Report (CTAR) are specified in FAA-HF-004A [23] and contain the items described in Table 2.7.

2.4.6

Operational sequence diagram

An operational sequence diagram is a process flow chart describing the workflow of a particular activity. The tasks and functions, arranged sequentially in the order that they are carried out, are divided between multiple operators and automated functions into different channels in the diagram. The flow of information between the operators and the system is indicated by unique symbology representing different functions or actions. The operational sequence diagram supplies information on decision action, communication links, inter-relationships, task frequency, task time and workload, providing a common reference by the human factors engineer, systems designer and software engineer of the tasks and functions required for a particular activity. Based on the identified functions and information flows in the diagram, the respective sub-systems, interfaces and interactions can then be designed and developed.

UAS human factors and human–machine interface design

41

2.4.7 Systems design and development Software design translates requirements into four classes of design details or activities, providing a software design specification which is used to describe the data, architecture, interfaces and components. Data design is the transformation of the information model into data structures, allowing software to retrieve and store data efficiently, and organising the program into modules to reduce the overall complexity of the software. The types of data structures, the links between data structures and the integrity rules of the data are specified. Architectural design involves identifying the modules within the software and defining the relationships between these modules. The three levels of detail in architecture design are conceptual architecture, which identifies the different software components and allocates responsibilities to each logical architecture that focuses on the design of component interfaces, interactions, connection mechanism and protocols, and execution architecture that focuses on the communication, coordination and resource allocation of software components during runtime. Interface design involves designing the interfaces between the different system components as well as between the program and the end user. It describes how the software communicates internally, with the systems that interoperate with it, and with the human user through data and control flow elements. Component-level design transforms the structural elements of the software architecture into a procedural description of the software components. Typical languages used in the development of avionics and ATM systems include C, Cþþ and Ada. While C is a procedural language, Cþþ also supports an object-oriented programming paradigm. Ada also supports object-oriented programming and is adopted in a variety of safety-critical applications. Ada possesses many characteristics to allow the development of dependable programs and also enhances code safety and maintainability. In recent years, Java and Python have also been adopted for the development of embedded avionics and ATM systems. Enterprise architectures describe the structure and operation of an organisation and allow the organisation to function cohesively and strategically. Architecture frameworks provide a standard basis for creating, using and analysing architectures. The Open Group Architecture Framework (TOGAF) is a widely used framework for designing, planning, implementing and maintaining enterprise information technology (IT) architectures. TOGAF supports the design of enterprise architectures in four interrelated domains. These domains comprise the business architecture that addresses the needs of the users, planners and business management; application architecture that addresses the needs of system and software engineers; data architecture that address the needs of database designers, administrators and system engineers and technical architectures that address the needs of hardware/software acquirers, operators and administrators. Other frameworks include the British Ministry of Defence Architecture Framework (MODAF), Department of Defence Architecture Framework (DODAF)

42

Imaging and sensing for unmanned aircraft systems, volume 2

and NATO Architecture Framework (NAF). These architecture frameworks are used in the defence context to develop architectures for managing complex systemof-systems to ensure interoperability within the network infrastructure and to enable effective decision-making between diverse stakeholder groups. Model-based Systems/Software Engineering is the application of modelling throughout different phases of the software life cycle. Model-based systems/software engineering makes use of industry standard modelling languages such as the Unified Modeling Language (UML), Systems Modeling Language (SysML), DODAF or MODAF to support the specification, analysis, design, verification and validation (V&V) of software and systems. Integrated Development Environments (IDE) such as IBM’s Rational Rhapsody provide a modelling environment for teams to work collaboratively on the development of complex and embedded applications.

2.4.8

Design evaluation

In the evaluation phase, system outputs are assessed against design requirements to determine if the product will function according to user expectations, in terms of metrics such as feasibility, usability, integrity, reliability and impact on operator performance. The evaluation phase also identifies potential safety hazards, failure modes or unintended interactions during operations. Evaluation can take place at varying levels of realism, either on the isolated component or as part of a more extensive integrated system. The results of the evaluation can be then used to refine the requirements and address any potential risks and problems in the design, as well as to propose changes to areas of the design where risks and problems have been identified. Different evaluation techniques are employed depending on the level of maturity in the design cycle and the time and cost requirements. Engineering analysis and evaluation provide a preliminary assessment of the design output, the task workflow, functionality, as well as physical and cognitive ergonomics aspects of the design. Common techniques include functionality and identification testing, CTA, human error, reliability analysis and risk assessment (HRA) or failure modes and effects analysis (FMEA). Design aspects are measured and evaluated against design requirements and/or a human factors checklist. Changes can then be recommended based on the feedback obtained. Critical tasks and functions are identified with detailed requirements for further design. Key evaluation metrics are identified, which can be measured in subsequent evaluations to determine if performance requirements are fulfilled. The preliminary assessment is relatively flexible as it can be adapted to varying levels of detail and has relatively low requirements on cost and time. It should be used early in the design process to ensure that good practices are followed throughout and serves as an initial planning activity for more detailed assessment methods. As the evaluation is carried out on a conceptual level, assumptions are made about the product, operating environment and user, and these will have to be refined in later design stages. The part-task evaluation involves a limited assessment of the design in representative conditions with the end user typically providing feedback. Common techniques include interviews, walkthrough analyses and usability testing. A combination

UAS human factors and human–machine interface design

43

of interfaces, hardware and simulated systems are used to evaluate the performance of the human user when executing specific sets of tasks. The assessed human factors might include the interface visibility, interactions with different automation modes, availability/accuracy of presented information, operator situation awareness and workload, as well as task performance metrics such as error rate, accuracy and execution time. The part-task evaluation assesses isolated components or tasks, allowing parallel evaluation or fault/error isolation. Additionally, as the evaluation is typically built upon prior task analyses, a task evaluation procedure can be refined progressively as the product matures, allowing for iterative improvements in the design process. Compared with a full-system evaluation, the evaluation of an isolated task is more contained and can be relatively more efficient. However, part-task evaluations might not capture the behaviour of larger, more complex systems and unanticipated vulnerabilities might arise from the interactions of different tasks and systems. Additionally, if a partially working system needs to be evaluated, the assessment can only occur at a more mature stage of the development process. Simulation and mock-ups allow the design output to be assessed under representative conditions within a laboratory environment. The realism in the assessment depends on the level of product integration into the host platform, the level of detail in the scenario design, the complexity of the simulated environment, as well as the extent that human operators are involved. As the level of realism can be varied in simulations, they can be utilised at many stages in the design process and also provide much flexibility, as they allow for the evaluation of quick prototypes to detailed models and systems. Complex dependencies and interactions that are typically not captured in isolated task evaluations might surface in realistic simulations. Furthermore, the controlled environment within which the assessment is carried out allows for more comprehensive data collection, repeatable testing and reduced risk as opposed to field tests. However, elaborate mock-ups and simulations can be costly and time-consuming to develop, and there are high costs associated with maintaining a simulator/laboratory. Field tests provide an assessment of the product in a highly realistic operational scenario in an external (field) setting. The assessment is typically conducted at a mature stage of the development process, where the operational effectiveness/ suitability of a finalised design is evaluated, or in the operational phase, where the performance of the product is continually monitored over its life cycle. Field tests provide a highly realistic evaluation of the product robustness/suitability in operational conditions but are typically more time-consuming and have a greater cost relative to lab evaluations. Additionally, testing in an external environment means that there is low experimental control, and safety and risk become more critical considerations.

2.4.9 Verification and validation V&V of software comprises (i) verification of the product to safeguard its intended function devoid of errors or unintended behaviour, and robust enough to respond properly to anomalous inputs and conditions and (ii) product validation to

44

Imaging and sensing for unmanned aircraft systems, volume 2

guarantee the customer’s satisfaction of the high-level requirements, provided they have been suitably translated into lower-level requirements, developed software architecture and source code. V&V provides an evaluation of the system’s functionality against system requirements in its intended operational environment and is also used to discover any system defects. RTCA DO-178 has traditionally been the point-of-reference for conducting V&V, and as of the writing of this chapter DO178B has been extended to include avionics-based software as well as ground-based and air traffic management software in DO-178C and DO-278A, respectively. Further discussion of the RTCA documents appears in [24]. Both static and dynamic methods are used in V&V. In static verification, the software is inspected by document and code analysis tools to detect errors. Software inspections are used to verify that the software conforms to a given specification but cannot verify non-functional characteristics such as performance and usability. Tools, such as static analysers, can be used to supplement the inspection, and work by parsing the program text to try to discover potentially erroneous conditions. The stages of static analysis are as follows: Control flow analysis: checks for loops with multiple exits or entry points, finds unreachable code, etc. Data use analysis: detects uninitialised variables, variables written twice without an intervening assignment, variables that are declared but never used, etc. Interface analysis: checks the consistency of routine and procedure declarations. Information flow analysis: identifies the dependencies of output variables. Does not detect anomalies itself but highlights information for code inspection or review. Path analysis: identifies paths through the program and sets out the statements executed in that path. Dynamic V&V is conducted through software testing, where test data is used to run the system so that its operational behaviour can be observed. Different types of tests are conducted, such as defect testing for discovering system defects, as well as statistical testing for determining the frequency of user inputs and for estimating program reliability. The tests can reveal the presence of errors but not their absence, and a test is considered successful if it discovers one or more errors. If defects or errors are discovered, the program undergoes debugging to locate and repair the errors. Table 2.8 outlines the main system safety assessment components used in the development cycle of aircraft software and systems. These components comprise the Functional Hazard Assessment (FHA), Preliminary System Safety Assessment (PSSA), System Safety Assessment (SSA), and Common Cause Analysis (CCA) and support the systematic definition, evaluation and assurance of system safety requirements.

2.5 Future work With the evolution of Industry 4.0, new developments will have to be taken into account. Research suggests that Industry 5.0 will push the envelope towards (i)

UAS human factors and human–machine interface design

45

Table 2.8 Components of the System Safety Assessment [25] SSA components

Description

FHA

FHA is conducted at the start of the development cycle to identify the failure conditions and the severity classification associated with each function. The FHA allows safety objectives to be assigned for the particular failure conditions and the subsequent assignment of a quantitative probability requirement. Based on the failure conditions established by the FHA, the PSSA conducts an examination of the system architecture to formulate the safety requirements which ensure that the safety objectives are met. PSSA involves performing a Fault Tree Analysis (FTA) on the functions derived from the system functional architecture and calculating the probabilities of failure. The SSA provides a more comprehensive evaluation of the system design using techniques similar to those of the PSSA activities. However, whereas the PSSA is conducted to identify the system safety requirements, the SSA verifies that the requirements specified during the FHA and PSSA analyses are satisfied by the proposed design. SSA involves the calculation of reliability figures and determination of the level of redundancy needed to attain a specified safety target. The CCA assesses the failure modes from integrated systems or functions to ensure that the individual systems’ failure modes are either independent of each other, or that the risks associated with any existing dependence are acceptable. The CCA comprises three types of analysis: zonal safety analysis (ZSA), particular risk analysis (PRA) and common mode analysis (CMA).

PSSA

SSA

CCA

man–machine cooperation, (ii) collaborative UAVs and (iii) greater use of artificial intelligence and machine learning. The sensory substitution arena will permit to construct devices for individuals with sensory losses, which are aesthetically pleasing, miniaturised, low cost and extensively available. Moreover, as the investigator controls the sensory replacement process, novel tools can be introduced to explore relationships and correlations among the perceptual, bodily and brain functionalities with the intricate cognitive process. Research has revealed trustworthy methods relying on sensory substitution to re-establish absent sensory functionalities. There are three chief fronts to be handled in HMI designs for UAV-CPSs: 1. 2. 3.

To devise robust and relatively low-priced technology deployments accessible to a wider public with sensory losses To extend human sensibilities, for example, enabling night vision without interfering with normal vision To expedite noninvasive safe studies employing human subjects to improve the knowledge about brain plasticity in addition to cognitive processes.

46

Imaging and sensing for unmanned aircraft systems, volume 2

2.6 Conclusions The HMI is a pivotal nevertheless regularly neglected angle in the UASs development. An appropriately planned HMI upgrades situational awareness and lessens the outstanding task at hand of the ground pilot, subsequently improving the general mission completion. Regularly, HFE employs methodological procedures and knowledge to assist HMI design. The iterative stages are threefold: (i) prerequisites identification and specification, (ii) framework design and (iii) assessment. Various methodologies can contribute to the anthropogenic, ergonomic and operational engineering aspects of the HMI elements. Given the wide scope of uses and missions comprising various sorts of UASs, user-centered development practices can contribute towards a well-defined development process, where the HMI is designed from a analysis of the requirements from both the human user and the operational framework. The everyday UAS capacities comprise mission organising, sensing, evidence exploration and SAA that can stretch out to multi-platform coordination and collaborative management. The contemplations of human elements and the related HMI components supporting these functionalities are key areas that entail considerable research and will advance via emerging technologies and modelling prototypes.

References [1] F. Kendoul, “Towards a Unified Framework for UAS Autonomy and Technology Readiness Assessment (ATRA),” in Autonomous Control Systems and Vehicles. Intelligent Systems, Control and Automation: Science and Engineering, vol. 65, K. Nonami, M. Kartidjo, K. J. Yoon, and A. Budiyono, Eds., Tokyo: Springer, 2013, pp. 55–71. [2] K. W. Williams, “Human Factors Implications of Unmanned Aircraft Accidents: Flight Control Problems,” DTIC Document 2006. [3] G. L. Calhoun, M. A. Goodrich, J. R. Dougherty, and J. A. Adams, “Human Autonomy Collaboration and Coordination Toward Multi-RPA Missions,” in Remotely Piloted Aircraft Systems: A Human Systems Integration Perspective, N. J. Cooke, L. Rowe, W. Bennett Jr, and D. Q. Joralmon, Eds., UK: John Wiley & Sons Ltd, 2017. [4] N. J. Cooke and H. K. Pedersen, “Chapter 18. Unmanned Aerial Vehicles,” in Handbook of Aviation Human Factors, J. A. Wise, V. D. Hopkin, and D. J. Garland, Eds., Boca Raton: Taylor and Francis Group, 2009. [5] R. Hopcroft, E. Burchat, and J. Vince, “Unmanned Aerial Vehicles for Maritime Patrol: Human Factors Issues,” Defence Science and Technology Organisation, Edinburgh (Australia) Air Operations Div 2006. [6] J. M. Peschel and R. R. Murphy, “Human Interfaces in Micro and Small Unmanned Aerial Systems,” in Handbook of Unmanned Aerial Vehicles, Springer, 2015, pp. 2389–2403.

UAS human factors and human–machine interface design

47

[7] T. Porat, T. Oron-Gilad, M. Rottem-Hovev, and J. Silbiger, “Supervising and Controlling Unmanned Systems: A Multi-Phase Study with Subject Matter Experts,” Frontiers in Psychology, vol. 7(568), 2016, pp. 1–17. [8] C. E. Nehme, J. W. Crandall, and M. Cummings, “An Operator Function Taxonomy for Unmanned Aerial Vehicle Missions,” in 12th International Command and Control Research and Technology Symposium, Washington, DC, 2007. [9] I. Ashdown, H. Blackford, N. Colford, and F. Else, “Common HMI for UxVs: Design Philosophy and Design Concept,” Human Factors Integration, Defence Technology Centre Report, BAE Systems, vol. 17, p. 18, 2010. [10] F. J. Ramos, “Overview of UAS Control Stations,” in Encyclopedia of Aerospace Engineering, West Sussex: John Wiley & Sons, Ltd, 2016. [11] R. C. Rorie and L. Fern, “The Impact of Integrated Maneuver Guidance Information on UAS Pilots Performing the Detect and Avoid Task,” in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2015, pp. 55–59. [12] R. H. Chen, A. Gevorkian, A. Fung, W.-Z. Chen, and V. Raska, “MultiSensor Data Integration for Autonomous Sense and Avoid,” in AIAA Infotech@ Aerospace Technical Conference, 2011. [13] G. Manfredi and Y. Jestin, “An Introduction to ACAS Xu and the Challenges Ahead,” in Digital Avionics Systems Conference (DASC), 2016 IEEE/AIAA 35th, 2016, pp. 1–9. [14] A. Fisher, R. Clothier, P. MacTavish, and J. Caton, “Next-Generation RPAS Ground Control Systems: Remote Pilot or Air Traffic Controller?,” in 17th Australian International Aerospace Congress (AIAC 2017), Melbourne, Australia, 2017. [15] A. Kolling, P. Walker, N. Chakraborty, K. Sycara, and M. Lewis, “Human Interaction with Robot Swarms: A Survey” IEEE Transactions on Human– Machine Systems, vol. 46, pp. 9–26, 2016. [16] M. Yeh, J. J. Young, C. Donovan, and S. Gabree, “FAA/TC-13/44: Human Factors Considerations in the Design and Evaluation of Flight Deck Displays and Controls,” Federal Aviation Administration, Washington DC, USA, 2013. [17] M. R. Endsley and D. Kaber, “Level of Automation Effects on Performance, Situation Awareness and Workload in a Dynamic Control Task,” Ergonomics, vol. 42, pp. 462–492, 1999. [18] FAA, “HF-STD-004: Requirements for a Human Factors Program,” Federal Aviation Administration HF-STD-004, 2009. [19] R. B. Miller, “A Method for Man–Machine Task Analysis,” DTIC Document 1953. [20] P. Salmon, D. Jenkins, N. Stanton, and G. Walker, “Hierarchical Task Analysis vs. Cognitive Work Analysis: Comparison of Theory, Methodology and Contribution to System Design,” Theoretical Issues in Ergonomics Science, vol. 11, pp. 504–531, 2010. [21] R. C. McIlroy and N. A. Stanton, “Getting Past First Base: Going All the Way with Cognitive Work Analysis,” Applied Ergonomics, vol. 42, pp. 358–370, 2011.

48

Imaging and sensing for unmanned aircraft systems, volume 2

[22]

J. Rasmussen, “Outlines of a Hybrid Model of the Process Plant Operator,” in Monitoring Behavior and Supervisory Control, Springer, 1976, pp. 371–383. FAA, “FAA-HF-004A: Critical Task Analysis Report,” Federal Aviation Administration FAA-HF-004A, 2009. S. A. Jacklin, “Certification of Safety-Critical Software Under DO-178C and DO278A,” in AIAA Infotech@ Aerospace Conference, Garden Grove, CA, 2012. SAE, “ARP4761: Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne Systems and Equipment,” Pennsylvania, USA, S-18, 1996.

[23] [24] [25]

Chapter 3

Open-source software (OSS) and hardware (OSH) in UAVs Pawel Burdziakowski1, Navid Razmjooy2, Vania V. Estrela3, and Jude Hemanth4

The popularity of the Open Source Tool (OST) has expanded significantly. This is the case for Unmanned Aerial Vehicles (UAVs) based on open-source hardware (OSH) as well. Open-source software (OSS) and OSH can be applied in a wide range of applications and can improve several technologies. The chapter begins with an introduction to OSS depicting its rationale, description of fundamental differences between OSS and proprietary software (PS), what benefits OSSs provide to overall UAV community, the motives leading people to pick up an OSS instead of a PS, which helps the academic and research community. This chapter also covers some OSSs used within the UAV community to support all aspects of UAV technology and the Remote Sensing (RS) and photogrammetry data post-processing chain. It is possible to build fully autonomous and operational UAV based only on OSH and OSS. The chapter describes the state of the art for OSS widely used in UAV technology, the software used in all aspects of UAV technology such as ARDUPILOT-based Autopilot firmware, MISSION PLANNER-based ground station, OPENTX transmitter software, MINIM On-Screen Data (OSD) software, Open Drone Map photogrammetry data processing suite, Web drone data-processing suite WebODM. This chapter describes several concepts and characteristics of open software/hardware, built-in functions, and particular features as well as platform requirements. A typical UAV photogrammetry workflow for drone construction with flight planning/execution and OSS data processing is provided.

3.1 Introduction Micro Aerial Vehicles (MAVs) are Unmanned Aerial Vehicle (UAV) class that possesses a 5-kg Maximum Takeoff Weight (MTOW), approximately 1-hour 1

Faculty of Civil and Environmental Engineering, Gdansk University of Technology, Poland Department of Electrical Engineering Tafresh University, Tafresh 3 Telecommunications Department, Departamento de Engenharia de Telecomunicac¸o˜es, Universidade Federal Fluminense (UFF), RJ, Brazil 4 ECE Department, Karunya University, Coimbatore, India 2

50

Imaging and sensing for unmanned aircraft systems, volume 2

endurance and an operational range nearby 10 km. MAVs are widespread and widely applied in photogrammetry and Remote Sensing (RS). Potential MAVs’ use in classical photogrammetry demands low-cost alternatives. There are two main ideas, for MAV construction and software development: the open-source hardware (OSH) and the open-source software (OSS). Amateurs and researchers mainly make MAVs built with OSH and OSS. The most fruitful open-source ventures have academical roots, and now its participation embraces a wide variety of academic initiatives. This chapter highlights open source projects’ cons and pros. Purely commercial MAV solutions where the software and hardware are developed, provided, and maintained by private companies may limit the UAV development, especially in the aftermath of a world’s crisis although there are options with high quality and accuracy.

3.2 Open source software First, it is paramount to explain the open-source concept [1], which denotes adjustable and shareable software or hardware. Its design is overtly accessible and associated with a broader set-off (also known as the open-source way). OSS, and OSH projects, products, and initiatives cover open exchange, cooperation, transparency, rapid prototyping, meritocracy, and community-oriented improvement. OSS is a software with source code that anyone can scrutinise, alter, and enhance. In computing science, the source code encompasses a collection of code with annotations, written with a human-friendly programming language, usually as plain text [2]. The program source code is specially designed to smooth programming of the computer-performed tasks done mostly by the written source code. A compiler often transforms the source code into binary code understood by the target machine (PC, Arduino Board, Android, etc.). Most application software is disseminated as executable files only. If the source code was included, then it would be helpful to anybody who might want to study or revise the program. Since programmers can add features to the source code or modify parts that do not always work correctly, the program can be improved by anyone who can do it. Mission Planner, Ardupilot, OpenTX, OpenDroneMap, Web ODM are examples of MAV and photogrammetry OSS. On the opposite side, there is a so-called closed source or commercial or proprietary software (PS). In that case source code is that only the individual, team, or organisation who created it, maintains, controls, and modifies it. In that case, only the original authors of closed software can copy, examine, and rework that software. At this time, almost all commercial MAVs rely on private software. The companies such as DJI (Dajiang) Innovations, Parrot SA, Yuneec offers commercial products; MAVs are running closed software, and the other programmers cannot modify its source code. In general, the OSS promotes collaboration and sharing, which permit other persons to adapt the source code and integrate those modifications into their projects. A very vivid example, within the UAV public, is the ArduPilot Community and organisation. At this time, Ardupilot is the most advanced, full-featured, and

Open-source software (OSS) and hardware (OSH) in UAVs

51

UAV C2

NM

FCM

DAM GCS C2

UAS

Figure 3.1 UAS (UAV-CPS) system basic modules [4] trustworthy existing open-source autopilot software [3,5,6]. It has been developed over the years by qualified engineers, computer scientists, and communities. The autopilot software can regulate any robotic vehicle (e.g. conventional aeroplanes, multi-rotors, helicopters, boats, and submarines). The concept of open-source code implies that it has rapid development always at the cutting edge of technology. Users profit from an all-encompassing ecosystem of sensors, companion computers in addition to communication systems due to many peripheral suppliers and interfaces. In conclusion, the open-source code can be appraised to safeguard compliance with security and confidentiality. As stated in [1], people prefer OSS to PS for some reasons, including: ●

●

●

●

Control. More control over software favours the choice of an OSS. The code can be tested to make sure that if the UAV is not doing anything, then the control should stay dormant. Training. The open-source code can help people to become better programmers. Students, investigators, or anybody else can efficiently study the publicly accessible code to make better software, share their work with others, and invite comment and critique, as development progresses. When some mistakes are discovered in the program’s source code, it can be shared with others to circumvent the repetition of the same mistakes. Security. Some people prefer OSS due to its satisfactory security and stability when compared with PS. Someone might identify and fix errors or omissions not seen by the program’s original authors due to the fact that anyone can assess and modify OSS. Programmers can correct, update, and upgrade OSS more quickly than PS. Stability. The OSS is desired in essential and long-standing projects as the source code is publicity distributed, users can rely on that software for critical activities, and without the fear that their tools will disappear or degrade if the original developers discontinue the project. OSS also tends to incorporate as well as a function according to open standards (Figure 3.1).

3.3 Open source UAS UAV is a part of a Cyber-Physical System (CPS) named UAV-CPS or UAS, which entails a drone, at least a Ground Control Station (GCS) besides a Communication and Control (C2) link between the GCS and UAV [4]. If the complete UAV-CPS

52

Imaging and sensing for unmanned aircraft systems, volume 2

uses OSH and OSS it can be named Open Source Unmanned Aerial System (OSUAS), which merely means that all main components and software used within UAS employs open-source idea. Critical UAV building blocks are onboard UAV platforms such as the Navigation Module (NM), Flight Control Module (FCM), and mechanical servos. Depending on the UAV primary purpose and associated tasks, the payload may differ. For photogrammetry and RS, the payload can be a Data Acquisition Module (DAM) [5–9]. The NM is the most critical module onboard, a UAV that repeatedly provides the aircraft’s location, speed, and altitude to the FCM. Hence, the NM feeds the FCM with critical UAV guiding data. The NM contains Navigation Systems (NSs) to fix the position of a platform (usually via a GNSS) and orientation system with accelerometers (motion sensors) and gyroscopes (rotation sensors) to continuously estimate the orientation, direction, and speed of the Inertial Measurement Unit (IMU) stage. Autopilots encompass NM (NSþOS), and FCM integrated into one module. The FCM is a device commanding a flight, which means leading a UAV to the elected position while setting the correct orientation and speed. The FCM has two parts: (i) the data analysis stage that receives commands from the system operator and (ii) the current flight parameters control module from the NM to analyse and send commands for actually correcting the flight parameters with mechanical servos and an electronic engine speed controller that moves all control surfaces and regulates the engines’ speeds. At this time, most known and considered as a state of art of the OSS is an Ardupilot. Ardupliot software can operate onboard open hardware and closed hardware. The supported open hardware boards are Pixhawk, The Cube (also known as a Pixhawk 2), Pixracer, Pixhack, F4BY, Erle-Brain, OpenPilot Revolution, Beagle Bone Blue, PXFmini RPi Zero Shield, TauLabs Sparky2. When considering OSH, it is worth to remember that it consists of physical technology artefacts designed and made available by the open design effort. OSH means that hardware evidence is easily identified so that others can tie it to the maker movement. Hardware design such as mechanical drawings, diagrams, bills of material, PCB layout documents, HDL source code, integrated circuit layout documents, and the software driving the hardware can be all released under free terms and made accessible for the community. In the case of open hardware autopilots, a manufacturer shares all necessary data to build the same hardware depending on available data. If one connects it with OSS, it makes an extensive and open UAV autopilot receptive to betterments. The GCS comprises stationary or moveable devices to observe, command in addition to controlling the UAV. The GCS can operate from the ground, aquatic, or air, working as a bridge, a connection among a machine and an operator. The UAV GCS design has specific functional requirements. Basic functionalities are: 1. 2.

UAV control – a capability to successfully control and fly the drone through mission Payload control – the ability to operate sensors from the ground.

Open-source software (OSS) and hardware (OSH) in UAVs 3.

4. 5. 6. 7.

53

Mission planning – functionality that aids UAVs operators to plan the mission while providing the required knowledge inputs concerning the UAV capabilities and limitations Payload evidence analysis and broadcasting – the capacity to disseminate data from the payload to eventual users System/air vehicle diagnostics – automatic test procedures for UAV while keeping the GCS effective maintenance and deployment Operator training – the facility to train the drone controller as well as practising mission plans Emergency procedures, post-flight analysis – the capability to amass both flight and payload data to analyse it after the flight.

The most popular GCS OSS supporting the autopilot hardware and software pieces stated above are APM Planner 2.0, Mission Planner, MAVProxy, QGroudControl, Tower, and AndroPilot (Figure 3.2). The DAM module includes optical RS devices, and airborne image acquisition systems with frequencies from the visible band to the Near Infrared (NIR), the Thermal Infrared (TIR), microwave systems, active and passive ranging instruments.

Flight controller Pixhawk Telemetry data GPS & compass module 1 433 MHz radio modem Ublox M8N GPS module 2 Ublox M8N Platform Frame S550 Gimbal and camera Tarot 3D Gimbal, GoPro3 Telemetry data 433 MHz radio modem

RC antenna 2,4 GHz AFHSS Video TX antennas Flight battery 5,8 GHz 10000 mAh lipo Video RX antennas 5,8 GHz (Diversity receiver) GCS Mission planner

Video display with OSD 5,8 GHz (Diversity receiver)

RC transmitter Hitec aurora 9

Figure 3.2 OSH and software-based UAS [5]

54

Imaging and sensing for unmanned aircraft systems, volume 2

Going deeper into the MAV system internal design, Figure 3.3 presents an advanced system architecture relying purely on the open hardware design. The system is initially designated for real-time visual navigation and photogrammetry tasks [6–9, 12]. In this particular design, for real-time UAV applications, a reasonable commuting power is needed. The NVIDIA TX2 processor has been chosen to run all the required calculations in the shortest possible time. Jetson TX2 combines a 256-core NVIDIA Pascal GPU with a hex-core ARMv8 64-byte CPU platform, and an 8-Gb memory (LPDDR4 with a 128-byte interface). The CPU platform has a dual-core NVIDIA Denver 2 together with a quad-core ARM Cortex-A57. The Jetson TX2 building block has a small size, light, low power, a size of 50 mm  87 mm, 85 g, and 7.5-watt power consumption. These features make this credit-card-sized processor very suitable for MAV applications. Jetson TX2 is an outstanding class of GPU-enabled boards marketed today for autonomous frameworks. The central computing power comes from NVidia processor that is only assigned for image calculation or other demanding commuting [7–9]. An application running in this processor should not impact flight control process. The flight control process is controlled by a flight controller (autopilot) ruining different processor. In that particular design, commands come from the visual system (NVidia processor) are passed to the flight controller, only simple commands, ESCs Rotor

Rotor

Rotor

Pixhawk 2.1

Rotor WiFi 5, 8 GHz

NVIDIA TX2 DEV

GPU HMP dual denver 2/2 MB L2+ Quad ARM A57/2 MB L2

Hardware interfaces

Hardware interfaces

GPU NVIDIA pascal, 256 CUDA cores

ACC/GYRO MPU9250

MAG HMC5983

BARO MS5611

ACC/GYRO MPU9250

MAG HMC5983

BARO MS5611

ACC/MAG LSM303D GYRO L3GD20H

FMU STM32F727

Hardware interfaces

IO STM32F100

Hardware interfaces Stereo camera ZED 3D stereo camera

Mono camera 5 MP MIPI SCI camera

Hardware interfaces

Visual sensors

Flight control and state estimation

WiFi 5, 8 GHz

2, 4 GHz + Telemetry link GPS here +

Ground control computer 1 linux

Ground control computer 2 windows

Safety switch

RC control OPEN TX Sensors

Processors

Effector

Ground station

Figure 3.3 Open hardware MAV advanced system design [6]

Interfaces

Open-source software (OSS) and hardware (OSH) in UAVs

55

similar to the remote user command, or some waypoint coordinates. The flight controller performs all the flight calculations. This communication between components happens via a standard protocol, fast and straightforward.

3.4 Universal messaging protocol As it can be noticed, the open-source autopilot software can operate with many GCS software and components. It would not be possible without a universal, opensource, interoperable communication protocol. The most popular, it can be considered at this time as a standard protocol for MAVs, is a protocol called MAVlink, which has an extremely lightweight messaging protocol intended for onboard MAVs’ components relying on a recent hybrid publish-subscribe with a point-to-point design pattern. MAVlink sends (publishes) data streams as topics while configuration subprotocols set on a mission or parameter protocol-employing point-to-point with retransmission. This concept is very analogous to the Robot Operating System (ROS) [10], which is another prominent and influential open-source framework. ROS is a flexible framework with a group of libraries, tools, and conventions to make straightforward the task of creating sophisticated and robust automatic behaviour across a wide diversity of robotic platforms. MAVLink messages adopt XML files where each of them delineates the message group kept up by a specific MAVLink system (aka dialect). Most GCSs and autopilots implement this reference message set in common.xml (so that most dialects use this scheme). MAVs and GCSs in addition to other MAVLink systems employ these generated libraries to communicate [11]. The MAVLink protocol is very efficient and its first version (MAVLink 1) has an 8-byte overhead per packet, together with the starting sign and packet drop recognition. The subsequent version (MAVLink 2) has 14 bytes of overhead and is much safer and more extensible. MAVLink is well-suited for applications with limited communication bandwidth because it does not require any additional framing. Ever since 2009, MAVLink has been allowing communication among many different vehicles, GCSs, and other nodes as well over diverse and challenging channels (with high latency/noise). It is a very reliable protocol because it arranges for methods to probe drops, corruption, and need for authentication of packets. This technology runs on many microcontrollers and OSs (e.g. ATMega, dsPic, STM32, ARM7, Windows, MacOS, Linux, Android, and iOS) while supporting many programming languages. With a maximum number of 255 concurrent subsystems on the network (such as UAVs and GCSs to name a few), it facilitates both offboard as well as onboard communications (e.g. amid a GCS and a MAV, and between autopilot and a surveillance camera MAVLink). The example of advanced software architecture (Figure 3.4) presents highlevel structures of the system, software elements, and relations among them purely based on open software [6]. The specific structural options from any available possibilities have been chosen to enable real-time computing, compatible autopilot interaction possibilities, and any custom payload integration.

56

Imaging and sensing for unmanned aircraft systems, volume 2 Onboard user applications ROS or linux modules

Scripts

Web applications

Offboard user applications Web, mobile applications

Drone OS

Obstacle avoidance Object tracking

APls C++, python, ROS

Authentication and security

Network manager

APls REST, web socket

Package manager Web server SLAM Swarm

Camera manager Payload manager Console Autopilot manager

Neural network

Data logger

Simulator

ROS linux Path planning

Figure 3.4 Advanced open source MAV system architecture [6] On the presented drone software architecture, flight control is on FC open software (responsible for flight control and stand-alone flight parameters computation) and the embedded computer software. The flight controller software uses Ardupilot FC firmware. Embedding commuting (NVidia TX2) is responsible for highly demanding sensors data calculation. NVidia TX2 is working on the Ubuntu Operating System (OS), also open source. A simple MavLink command is sent to FC due to the calculations. Ubuntu OS can work with the ROS for sensor data calculation. The ROS is a flexible framework for creating robot software consisting of an assortment of tools, libraries, in addition to conventions to simplify the creation of multifaceted and robust drone behaviour. On top of the mentioned software, the Flyt OS integrates ROS, MAVlink, and user applications and interface. This OS lets integrate external ROS/Linux libraries with custom data helping the execution of onboard in addition to offboard applications. The custom applications can be run on any OS and hardware (smartphones, PC, MAC, wearables, portable devices) to build an interface between the drone and human operator.

Open-source software (OSS) and hardware (OSH) in UAVs

57

3.5 GCS software The GCS software should be able to operate with the desired autopilot and be able to run and control all necessary functionalities. This onboard software operates specific OSs, including Windows, Linux, Android, or iOS. The main functions entail: 1. 2. 3.

4. 5.

Loading the built-in software (the firmware) into the autopilot controlling a specific vehicle Optimum performance via setup, configuring, and tuning of the drone operation Planning, saving, and loading sovereign undertakings employing the autopilot together with a simple and intuitive point-and-click GUI established on Google or other supported mapping software platform Downloading and analysing mission logs Interfaces with a PC flight simulator to build a full hardware-in-the-loop UAV simulator [5].

With additional hardware, software should enable monitoring the drone’s status in action with backup copies of the telemetry logs, which enclose much more information about the onboard autopilot logs and operating the drone in First-Person View (FPV) mode. The software must support all autopilot functions as well as full mission planning abilities. Each facet of the autonomous or automatic mission may be programmed and controlled via the application (Figure 3.5). Currently, OSS like Mission Planner, APM Planner, MAVProxy, QGroudControl, Tower, and AndroPilot are the most common software. The applications are designed for any OS and several different software architectures (Table 3.1). The common future, which allows operating with Ardupilot firmware, will probably be an open-source communication protocol – MAVlink. GCS acts as an interface between human (operator), translates user inputs to MAVlink command, understandable for UAV’s FC.

3.6 Processing software There are many photogrammetry software on the market, which is designed especially for UAV photogrammetry and RS applications. The most popular commercial software, despite its expensiveness, provides a wide range of image processing Flight data

Flight plan

Tuning graph Detailed graph HUD display System status Quick data Data and system inf. Map window Actual map and vehicle position

Mission plan Detailed mission plan map WPs and commands list All planned WPs and commands

Figure 3.5 The Mission Planner user interface [5]

58

Imaging and sensing for unmanned aircraft systems, volume 2

Table 3.1 GCS Software platforms (own elaboration)

Platform

Source code language License

Mission Planner

APM Planner 2.0

MAVProxy

QGroundControl

Tower

Andropilot

Windows, Mac OS X (using Mono) C#.net

Windows, Mac OS X, Linux

Linux

Qt

Python

Windows, Mac OS X, Linux, Android, and iOS Qt QML

Android phones and tablets Android Studio

Android phones and tablets Scala (Java)

Open source (GPLv3)

Open source (GPLv3)

Open source (GPLv3)

Open source (GPLv3)

Open source (GPLv3)

Open source (GPLv3)

workflows, supporting almost all cameras and MAVs on the market [7,12–15]. Companies guarantee full customer support, global service, training, and distribution. Software is relatively simple to use and is regularly updated. There are significant pros, which have to be mentioned here. There is no equivalent open-source image processing software for MAV, which can cover all functionalities and simplicity of operation. Today, only one open source project can deliver some UAV photogrammetry product such as point clouds, digital surface models (DSMs), orthorectified imagery, and so on [12–20]. The Open Drone Map (ODM) ecosystem is such a case, with its current subprojects WebODM and Node-ODM. The ODM is an OSS for processing mainly aerial drone imagery. It has to be highlighted that the current ODM version number is 0.3.1 bet, and has started in 2011. This software handles micro air vehicle imagery taken by a non-metric camera and is able to produce orthorectified imagery, DSM, digital terrain models (DTM), textured 3D models, and classified point clouds. ODM does not provide a graphical user interface (GUI) to interact with the user, which imposes basic knowledge as for the Linux OS terminal commands. The prepared input images are stored in the specified folder [9,16,17]. Terminal commands start processing the images and save the results in the desired folders structure. ODM does not provide any additional viewing result interface, and due to that fact an additional software for viewing, editing, or evaluating results is necessary. All processes are computed using a Central Processing Unit (CPU). This software does not support calculation by Graphical Processing Unit (GPU). GPU calculations, especially for processing images using NVIDIA CUDA technology [18–20], can be accelerated up to eight times if we compare it with CPU computation. As this project is named ecosystem, additional subprojects support ODM. Based on the ODM engine, the web GUI, to promote interaction with the user, was added to this ecosystem and is called WebODM. The WebODM is designed to be a user-friendly application for drone image processing. This ecosystem also delivers an application programming interface (API) for software developers. Since the WebODM is a kind of web-based user interface for ODM, it generates the same outputs as ODM (Figure 3.6).

Open-source software (OSS) and hardware (OSH) in UAVs

59

Figure 3.6 WebODM user interface: point cloud model

3.7 Operator information and communication During the UAV mission, operators should have actual information about the actual platform status and flight parameters so that there are few ways to handle data. Visually, in case of Visual Line of Sight (VLOS) operator can observe the flying platform and control UAV behaviour. This is a primary method, but must be trained by the operator to get the UAV under control in any emergencies. In this method, only the operator can have the relative position of the platform, relative to other elements of the environment and actual platforms orientation. GCS flight parameters display (telemetry) help during either VLOS or Beyond VLOS (BVLOS) operation. The GCS offers the most comprehensive information about the UAV status and actual information. Telemetry data are transferred to the GCS and the GCS screen. Nevertheless, not all data are useful for the operator, and even some can disrupt the operator’s attention. Usually, the UAV data dispelled on the GCS are monitored by a second operator, in the two-person UAS configuration. In that case, when there is only one operator, the third method can be used. Flight data overlaid on the operator’s video stream and On-Screen Display (OSD). An OSD is a piece of information (image) superimposed on a screen picture. The vital information for the piloting operator is superimposed on the video stream. His method permits the OS to simultaneously monitor the UAV status and flight parameters and control video operators stream, transmitted by the UAV. MinimOSD and its modification such as MinimOSD-Extra are the most popular OSS OS. This software runs on the cheap and open hardware board relying on the MAX7456 processor, which is a single-channel monochrome OSD generator that does not need an external video driver, EEPROM, sync separator, and video

60

Imaging and sensing for unmanned aircraft systems, volume 2

switch. The MAX7456 attends all national and global markets with 256 userprogrammable characters (in NTSC and PAL standards). The Minim OSD Extra receives information from the flight controller via the MAVlink protocol on four configurable screens configured by the operator depending on actual needs. On the presented example, the operator’s screen (Figure 3.7) displays the most critical information, such as the number of GPS satellites fixed (here 0), comas rose and actual heeding (240 ) Home Altitude (HA – 1 m) (from top left). In the middle, there is an artificial horizon line and the exact pitch and roll angle. From the bottom left, the figure displays battery status (voltage, current and used capacity), warnings section (here No GPS fix warning), and actual flight mode. To manually control or supervise the UAV, a radio link is required: a radio transmitter, on the ground, and relevant radio receiver. Radio Control (RC) transmitters are changing the operator’s input to the desired radio signal and finally on the mechanical or electrical response on the UAV. Nowadays, the state of art of the open-source transmitter software is the OpenTX. Precisely, OpenTX is an open-source firmware for RC radio transmitters. This firmware gained popularity due to its high configurability and many more features than the ones found in traditional proprietary radios. According to the main OSS idea, the frequent feedback from the users around the world ensures the continued stability and quality of the firmware. The OpenTX firmware and its hardware base are renowned players in the RC industry since this open-source radio targets the mainstream market. The OpenTX project improves both the hardware and software sides of a product, and they are of low cost and devoid of the customary marketing-driven limitations that most manufacturers place on their offerings. OpenTX offers features that match and even exceed those of the highest end radios in the industry [19]. The most significant difference, when compared with the commercial approach, is the open-source community-driven firmware, so unlike with major manufacturers, if there is a need to implement a particular function or users have good improvement suggestions realised a few days later, and published.

Figure 3.7 The MinimOSD display

Open-source software (OSS) and hardware (OSH) in UAVs

61

3.8 Open source platform Since OSS is shared and written by its community, for use and diffusion, it requires some platforms to share, collaborate, manage changes and versions. Many opensource projects are available on the GitHub platform that hosts code for version control and collaboration and allows people to work together on projects from anywhere [20]. This platform consists of user repositories and delivers particular functionalities. A repository is the essential GitHub element, and it contains a project’s folder containing all its associated files with documentation and historic of revisions of each file. Regardless if they are public or private, repositories allow for multiple collaborators. The GitHub platform hosts mostly code, and additionally, it supports the subsequent formats and essential features such as: ●

●

●

● ● ●

● ●

Issues’ tracking (including feature requests) with labels, milestones, assignees, and a search engine to suggested improvements, tasks or questions related to the repository. For public repositories, anyone can create them, and repository members moderate them. Each issue encompasses its discussion forum, labels, and user assignments. Wiki – a subsite on the repository on which users collaboratively modify content and structure directly from the web browser. In a typical wiki, the text is written using a simplified markup language and often edited with the help of a rich-text editor [21], Pull requests with code review, and comments contain proposed changes to a repository submitted by a user with the corresponding acceptances or rejections by a repository’s collaborators. Each pull request has a discussion forum. Commitment history. Documentation, including automatically rendered README files. Visualisation of repository data such as contributors, commitments, code frequency, punch card, network, and members. GitHub Pages are small websites that can host public repositories on GitHub. Visualisation of additional data alike to 3D render files, geospatial data, Photoshop’s native PSD and PDF documents.

All software mentioned earlier has its repository and full history on the GitHub platform.

3.9 Future work Most of the UAV’s challenges gravitate through the dichotomy computational tasks performed onboard and offboard to decrease energy consumption. Furthermore, the tendency is to have UAVs using while also providing IoT Devices and IoT Services through dependable Wireless Communication Networks [22].

3.9.1 OSH challenges The OSH concept relates to any physical technology, that is, it goes beyond electronics. Depending on how one interprets this notion, it can be considered even

62

Imaging and sensing for unmanned aircraft systems, volume 2

much more popular than OSS because people have been sharing recipes for quite a long time [22]. One problem is the technical expertise and the need to have tools and instruments. Programming requires a computer, reading, and typing even if the people involved are highly capable. One situation where OSH can truly liftoff is in the usage of FPGAs because designing with them is as unpretentious as conniving software. However, OSH has problems more difficult to describe and solve than the development languages available, for example, VHDL, Verilog, and LabVIEW FPGA, because they require thinking about challenges differently. There are timing and parallelisation problems that cannot be learned like in serial programming. Moreover, the hardware is not as easily replicated as software. If one wants a piece of OSS, it is just a matter of downloading it. Hardware replication requires a little money to acquire and transport it. OSS software can be distributed for free, but OSH (in its more accessible form) involves the VHDL/Verilog/FPGA code with some way to produce the physical system. As much as integrated circuits and CAD projects go, it is entirely impractical to produce open-source alternatives given today’s limitations. A hardware description can be copyright, but not the physical realisation itself. The real physical implementation can only be secured by patent, which demands time and money. OSS typically can benefit from copyright because the software can be attached to its license differently from the hardware. Hence, one cannot enforce the OSH philosophy in the courts [22].

3.9.2

Open data

A UAV-CPS cater to a variety of applications that demand different types of databases for the sake of benchmarking all its stages. However, handling Open Source Data (OSD) poses some challenges. The Open4Citizens project [22] has strategies to promote open data usage and to support the foundation of communities and practices. It addresses design complexity over multilayered interventions going from the organisation of extensive processes to the definition of a network. It is important to stress that there is a tendency to decoupling problems so that the workflow of people from different backgrounds is not disturbed.

3.9.3

Cloud data centre

The development of UAV-CPSs needs to build upon cloud computing infrastructure delivered by a combination of tools resembling the Amazon Web Services (AWS). Hence, the analyses and classification procedures, which embrace the segmentation, feature computation, and labelling steps, executed in computer clusters with different virtual machines’ sizes. One cluster node must be reserved for the Hadoop JobTracker (aka the master node) and is only responsible for task scheduling and managing. The Apache Hadoop software library is an arrangement that permits large data sets’ distributed processing across computer clusters via simple programming

Open-source software (OSS) and hardware (OSH) in UAVs

63

models. It scales up from a single server to a myriad of machines, each with local computation and storage. Instead of relying on high-availability high-performance hardware, the library detects and handles failures working at the application layer. Hence, highly available service is delivered on top of a computer cluster where each computer and each cluster may be prone to failures. As big data augments and becomes indispensable, open-source implementations for working with it will undoubtedly remain rising as well: 1. 2. 3. 4.

The Apache Software Foundation (ASF) supports many of these big data projects. Elasticsearch is an enterprise search engine built on Lucene. Its Elastic stack produces comprehensions from structured and unstructured records. Cruise Control by LinkedIn runs Apache Kafka clusters at a massive scale. TensorFlow has proliferated since Google open-sourced it. Thus, it will increase the availability of computational intelligence tools and popularising machine learning as a result of its ease-of-use concept.

3.9.4 Crowd-sourced data in UAV-CPSs UAVs can perform crowd surveillance [23] and use information mined from social networks to facilitate image comprehension. To assess the use cases, video data can be offloaded to be processed by a Mobile Edge Computing (MEC) node compared with the local video data treatment onboard UAVs. MEC-based offloading can help to save the limited energy of UAVs. Cloud computing, big data, software-defined networks/network functions virtualisation, cloud security, sensor networking, distributed machine learning, internet of things software, 3D scanning software, and so on will all grow faster than normal and change in the next 5 years.

3.9.5 Control of UAV swarms Nowadays, cloud services handle a very high number of loosely coupled UAV microservices. The paradigm shift towards microservices takes into consideration that lots of the assumptions about cloud-based UAV-CPSs are continually changing, as well as current opportunities and challenges when optimising Quality of Service (QoS) and cloud utilisation especially when UAV swarms are involved [24–32]. Open source benchmarks built with microservices must allow simulation and verification of comprehensive end-to-end services, in a modular and extensible fashion. To incorporate suites that can permit the combination of swarm control and coordination with other more human-dependent applications such as social network, entertainment/media service, e-commerce, e-banking, and different types of IoT applications still require an OSS and OSH development. This necessity arises from the impact UAV swarms may have in networking, security, and OS, among other issues. They challenge cluster management and other trade-offs regarding application design/deployment and programming frameworks. Scale impacts microservices in real frameworks due to the extremely high number of users, and poses increased pressure on the prediction of required parameters such as performance.

64

Imaging and sensing for unmanned aircraft systems, volume 2

3.10 Conclusions In this chapter, the conception of Open Source Tools (OSTs) has been studied. This panorama extends to this is the situation for MAVs in light of OSH too. OSS and OSH can suit a wide scope of applications and improve technologies. This text starts with a prologue to OSS portraying its method of reasoning, establishing contrasts among OSS and PS. OSS can benefit a large part of the UAV community with the intentions of driving individuals to use OSS rather than a PS, which helps the academic, amateur, and research communities. Afterwards, the chapter covers some OSSs utilised inside the UAV people to help all parts of UAV innovation and the RS/photogrammetric information post-preparing chain. It is conceivable to manufacture completely selfgoverning and operational UAVs dependent on OSH and OSS. This chapter highlights the multidisciplinary nature of issues related to the OSS and the OSH communities. It outlines different perspectives of the communities dedicated to their development [33]: virtual structures, collective knowledge, stimulus, innovation, shared intelligence, community organisation, learning, and achievement. OSH requires physical equipment, while OSS involves virtual entities. Hence, the OSS product can be interminably replicated at almost no price tag and shared straightforwardly. OSH necessitates actual constituents and expenses to reproduce each unit or instance. Moreover, designing hardware has additional necessary degrees of freedom, which are not simple replacements of one another in contrast to the design of software where it is all code.

References [1] Blokdyk, G. “Red Hat Ansible A Complete Guide,” 5STARCooks, Plano, Texas, USA, 2019. [2] P. Iora, M. Taher, P. Chiesa, and N. Brandon, “A one dimensional solid oxide electrolyzer-fuel cell stack model and its application to the analysis of a high efficiency system for oxygen production,” Chemical Engineering Science, vol. 80, pp. 293–305, 2012. [3] A. D. Team, “ArduPilot.” Available at: www.ardupilot.org, vol. 2, p. 12, 2016. [4] P. Burdziakowski and J. Szulwic, “A commercial of the shelf components for a unmanned air vehicle photogrammetry,” 16th International Multidisciplinary Scientific GeoConference SGEM 2016, vol. 2, 2016. [5] P. Burdziakowski, “Low cost hexacopter autonomous platform for testing and developing photogrammetry technologies and intelligent navigation systems,” 2017. [6] P. Burdziakowski, “UAV design and construction for real time photogrammetry and visual navigation,” 2018 Baltic Geodetic Congress (BGC Geomatics), 2018, pp. 368–372. [7] A. M. Coelho and V. V. Estrela, “Data-driven motion estimation with spatial adaptation,” International Journal of Image Processing (IJIP), vol. 6, p. 54, 2012.

Open-source software (OSS) and hardware (OSH) in UAVs

65

[8] D. J. Hemanth and V. V. Estrela, Deep Learning for Image Processing Applications, vol. 31: IOS Press, 2017. [9] B. S. Mousavi, P. Sargolzaei, N. Razmjooy, V. Hosseinabadi, and F. Soleymani, “Digital image segmentation using rule-base classifier,” American Journal of Scientific Research ISSN, 2011. [10] M. Quigley, K. Conley, B. Gerkey, et al., “ROS: an open-source Robot Operating System,” in ICRA Workshop on Open Source Software, 2009, p. 5. [11] D. Mendes, N. Ivaki, and H. Madeira, “Effects of GPS spoofing on unmanned aerial vehicles,” in 2018 IEEE 23rd Pacific Rim International Symposium on Dependable Computing (PRDC), 2018, pp. 155–160. [12] V. V. Estrela and A. E. Herrmann, “Content-based image retrieval (CBIR) in remote clinical diagnosis and healthcare,” in Encyclopedia of E-Health and Telemedicine, IGI Global, ed. 2016, pp. 495–520. [13] P. Moallem and N. Razmjooy, “Optimal threshold computing in automatic image thresholding using adaptive particle swarm optimization,” Journal of Applied Research and Technology, vol. 10, pp. 703–712, 2012. [14] B. S. Mousavi and F. Soleymani, “Semantic image classification by genetic algorithm using optimised fuzzy system based on Zernike moments,” Signal, Image and Video Processing, vol. 8, pp. 831–842, 2014. [15] N. Razmjooy, B. S. Mousavi, M. Khalilpour, and H. Hosseini, “Automatic selection and fusion of color spaces for image thresholding,” Signal, Image and Video Processing, vol. 8, pp. 603–614, 2014. [16] V. Estrela, L. A. Rivera, P. C. Beggio, and R. T. Lopes, “Regularized pelrecursive motion estimation using generalized cross-validation and spatial adaptation,” in 16th Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI 2003), 2003, pp. 331–338. [17] N. Razmjooy, B. S. Mousavi, B. Sadeghi, and M. Khalilpour, “Image thresholding optimization based on imperialist competitive algorithm,” in 3rd Iranian Conference on Electrical and Electronics Engineering (ICEEE2011), 2011. [18] W. Błaszczak-Ba˛k, A. Janowski, and P. Srokosz, “High performance filtering for big datasets from Airborne Laser Scanning with CUDA technology,” Survey Review, vol. 50, pp. 262–269, 2018. [19] T. K. Jespersen, “Software Defined Radio,” 2015. [20] GitHub. (2019). GitHub Help. Available at: https://help.github.com/ [21] F. H. Knight, “Capital and interest,” Encyclopedia Britannica, vol. 4, pp. 779–801, 1946. [22] N. H. Motlagh, T. Taleb, and O. Arouk, “Low-altitude unmanned aerial vehicles-based internet of things services: comprehensive survey and future perspectives,” IEEE Internet of Things Journal, vol. 3, pp. 899–922, 2016. [23] M. Ghazal, Y. AlKhalil, A. Mhanna, and F. Dehbozorgi, “Mobile panoramic video maps over MEC networks,” in 2016 IEEE Wireless Communications and Networking Conference, 2016, pp. 1–6. [24] J. Gray. “Design and implementation of a unified command and control architecture for multiple cooperative unmanned vehicles utilizing

66

[25]

[26]

[27]

[28]

[29]

[30]

[31]

[32]

[33]

Imaging and sensing for unmanned aircraft systems, volume 2 commercial off the shelf components.” Technical Report, Air Force Institute of Technology Wright-Patterson AFB United States, 2015. R. Mur-Artal, and J. D. Tardo´s. “ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras.” IEEE Transactions on Robotics, 33, 2017, 1255–1262. A. Bingler and K. Mohseni. “Dual-radio configuration for flexible communication in flocking micro/miniature aerial vehicles.” IEEE Systems Journal, 13, 3, 2019, pp. 2408–2419. L. He, P. Bai, X. Liang, J. Zhang, and W. Wang. “Feedback formation control of UAV swarm with multiple implicit leaders.” Aerospace Science and Technology, 72, 2018, pp. 327–334 T. Larrabee, H. Chao, M. B. Rhudy, Y. Gu, and M. R. Napolitano. “Wind field estimation in UAV formation flight,” in Proc. 2014 American Control Conference, Portland, Oregon, USA, 2014, pp. 5408–5413. V. Stepanyan, K. K. Kumar, and C. Ippolito. “Coordinated turn trajectory generation and tracking control for multirotors operating in urban environment.” In Proc. AIAA Scitech 2019 Forum, San Diego, California, USA, 2019. https://doi.org/10.2514/6.2019-0957 R. Ahmadi, N. Hili, L. Jweda, N. Das, S. Ganesan, and J. Dingel. “Run-time monitoring of a rover: MDE research with open source software and lowcost hardware.” In Proc. of the 12th Educators Symposium (EduSymp/ OSS4MDE@MoDELS), Saint-Malo, France, 2016, pp. 37–44. R. Mies, J.-F. Boujut, and R. Stark. “What is the “source” of open source hardware?” Journal of Open Hardware, 1(1), 2017, p. 5. https://doi.org/ 10.5334/joh.7. Y. Zhang, Y. Gan, and C. Delimitrou. “Accurate and scalable simulation for interactive microservices.” In Proc. 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Madison, WI, USA, 2019. https://doi.org/10.1109/ISPASS.2019.00034. M. R. Martı´nez-Torres and M. d. C. Diaz-Fernandez, “Current issues and research trends on open-source software communities,” Technology Analysis & Strategic Management, vol. 26, pp. 55–68, 2014.

Chapter 4

Image transmission in UAV MIMO UWB-OSTBC system over Rayleigh channel using multiple description coding (MDC) Ali Arshaghi1, Navid Razmjooy2, Vania V. Estrela3, Pawel Burdziakowski4, Douglas A. Nascimento5, Anand Deshpande6, and Prashant P. Patavardhan7

Orthogonal Space–Time Block Codes (OSTBC) and multiple-input–multiple-output (MIMO) communication system are new techniques with high performance that have many applications in wireless telecommunications. This chapter presents an image transfer technique for the unmanned aerial vehicle (UAV) in a UWB system using a hybrid structure of the MIMO-OSTBC wireless environment in multiple description coding (MDC) deals. MDC technique for image transmission is a new approach in which there is no record of it so far. This ensures that in the packet loss scenario due to channel errors, images with acceptable quality with no need for retransmission can be reconstructed. The proposed system is implemented using a different number of transmitter and receiver antennas UAV. Assuming a Rayleigh model for the communication channels, the MDC image transmission performance is compared with single description coding (SDC). Experimental results confirm that the proposed hybrid method has better performance than the SDC.

4.1 Introduction The unmanned aerial vehicle (UAV) technology has a lot of potential such that they are remarkably increasing especially in recent years. Some of these applications 1

Department of Electrical Engineering, Central Tehran Branch, Islamic Azad University, Tehran, Iran Department of Electrical Engineering, Tafresh University, Tafresh, Iran 3 Telecommunications Department, Federal Fluminense University (UFF), Rio de Janeiro, Brazil 4 Faculty of Civil and Environmental Engineering, Gdansk University of Technology, Poland 5 Department of Electrical and Computer Engineering, Texas A&M University at Qatar, Doha, Qatar 6 Department of Electronics and Communication Engineering, Angadi Institute of Technology and Management Belagavi, Karnataka, India 7 Department of Electronics and Communication Engineering, Gogte Institute of Technology, Belagavi, Karnataka, India 2

68

Imaging and sensing for unmanned aircraft systems, volume 2

include filming, entertainment, accident relief, building management agriculture, and commercial UAV cargo carrying. The application of this technology is expected to increase from 80,000 units in 2015 to 2.7 million units in 2025, and the big companies are looking to use UAVs in rural areas [1]. UAV can also be used to deliver a broadband wireless connection to hotspot areas in transit times during emergencies and disasters such as earthquakes when the communications infrastructure may be damaged [2]. This interest in UAVs leads individuals towards studying aeroplane gaming (AG) with augmented reality (AR), UAV channels for the link budget analysis, and the UAV cyber-physical systems’ (CPSs) design [3–5]. In the end, UWB signals are allowed to receive multipath components (MPCs) with an outstanding temporary resolution, which may be a fascinating technology for the development of a broadband release model (UWB) [6–8]. The high bandwidth of the UWB can also facilitate high data rates, better penetration through materials, and coexistence with narrowband networks for air-to-ground (AG) UAV communications. The UAV comprises a radio control and self-control program [9]. These UAVs usually use photovoltaic energy to transmit the information (like video and multimodal imagery) into a ground control station, called High-Definition Image Transmission System (HDIT), which plays an essential role in HD image processing and computer vision (CV) applications, including encoding, transferring, receiving, and decoding. In particular, the application that transmits HD image and video data to a control office is a critical factor in controlling and directing the flight, so it requires a real-time performance. In the field of optical measurements, aerial surveys, and air photography, high-resolution images are increasingly used in UAVs to improve the accuracy of observation to make the amount of the data extremely high. The 1080p HD video (HDTV) standard produces approximately 500 Mb of data per second at 3 frames per second with 8 byte YCbCr. The 2K, the 4K video format, is also used imperceptible as the next-generation HD format, and it generates a large amount of data. In most applications scenarios, the UAV requires aerial imagery components to be checked, and aerial photography can be real-time transmitted to the ground station. It is necessary to compress the images to control the data rate so that the bandwidth is low in the current UAV areas. In addition, the UAV requires photovoltaic components to provide fast-moving characteristics. It is also very rigid in the power dissipation, volume, and weight of the image transmission system [9,10]. Most traditional military aircraft uses data-linking technology to create a wireless communication system with studies suggesting it as a new scheme for sending a small UAV image. The application areas of this chapter are extended from civilian airlift and aerial photography, resource exploration, power cable discovery, oil pipe monitoring, consumer vehicle, and video transmission of UAVs.

Image transmission in UAV MIMO UWB-OSTBC system

69

Compact video plays a real role in the implementation and operation of the system. Due to the growth of the techniques and the complexity of the system from the construction aspect, the technical implementation capability of this research carries the image transfer to the UAV with UWB and faces all the requirements including power dissipation, volume, and weight. H.264/AVC is still very used but the HEVC is the standard video encoder for the streaming. That coding framework and key technologies have been thoroughly analysed. Often, the research and use of UAV-CPSs are limited to external experiments using a global navigation satellite system (GNSS) receivers for guidance [11]. A visual observation control has recently been inspired by extensive aerial robotic research that has been inspired by the complexity and the precision of the aerial manoeuvres [12]. The low-cost and high-speed progression make the UAVs to be useful in the internal applications in the emergencies [13]. The advancement in UWB communication offers a high-positioning accuracy that provides a new range of applications [14]. The concept of ultra-wide bandwidth spread-spectrum impulse radio has been discussed in [15] as an alternative for communication of short suffering. The ability to solve the multi-path playback makes it a viable candidate for internal locating [16]. The micro UAV technology (MAV) has a lot of potential usages in recent years. Moreover, UAV air photography can also be used in 3D mapping [17], airborne surveying [18], agricultural precision [19], as well as search and rescue [20], and other interesting applications [21]. The method of work for the UWB centralisation system appears in Figure 4.1. In [22], UWB radios are used to estimate the location and the speed of a quadcopter. The UWB module on the quadcopter for distance measurement is to send ranging requests to anchor nodes. In [23], low-cost high-accuracy UWB radios for internal localisation are proposed, which allows tracking drones in and out a warehouse for possible autonomous inventory taking. The designed method is an ultra-wideband (UWB) solution that uses anchor nodes structure. So, it does not need any wired backbones, and the battery will stay strong. We developed a UWB MAC multi-technology protocol for localisation that does not need to be built-in, such as Wi-Fi or Ethernet. In Figure 4.2, the top-level system diagram of the proposed solution is drawn. Image transmission owns a large volume of data and various value information bytes than data transmission [24,25]. Wireless communication highdimensional image and video transmission have much application such as biomedical wireless sensor network, mobile networks, satellite networks, wireless sensor networks, and contingency [25–29]. Since multimedia wireless communication has played a paramount role in the communication field in recent years, the wireless image transmission is widely used. Many technologies have

70

Imaging and sensing for unmanned aircraft systems, volume 2

UWB anchor

Raw distances to anchors Start trilateration

Range calibration UAV EKF Outlier detection

Record timestamp and update register

Prediction

Correction

Position and velocity estimation

Figure 4.1 UWB-based localisation workflow been proposed to improve the performance of image transmission system to get better image quality [30–32]. One of the systems for image transmission is the UWB, which is suitable for the transmission of high-dimensional high-resolution multimodal imageries. UWB communication system is a new technique with high performance that has low power consumption and many applications of wireless telecommunications for ultra-data rates. UWB affords an appropriate solution for telecommunication wireless high speed with short-range. The radio technology can use 25% of the bandwidth of the centre frequency. Power UWB system is low in-band frequency; because it can play in a wide frequency band. This system is robust towards the distortion and the interference and has high contrast and high time resolution in the receiver. Low complexity and low cost are some crucial advantages of the technology which expedite its deployment in mobile applications. In these systems, instead of sinusoidal carriers, very narrow pulses are transmitted, and system bandwidth will appear into many GHz [33]. The time-hopping UWB (TH-UWB) technique and the direct sequence UWB (DS-UWB) methods are the most popular UWB approaches. So far, time-hopping (TH) methods relying on the Pulse Position Modulation (PPM) known as TH-UWB and employing direct sequence (DS) method by

Image transmission in UAV MIMO UWB-OSTBC system

71

Sleeping Anchors z

Selected anchors

A

A

A

(ii) UWB two-way ranging

z A

(iii) long range sub-GHz reporting

(i) sub-GHz activation and scheduling of anchor nodes

A

z A

A BR

UWB

Sub-GHz

RTLS

Figure 4.2 High-level of the UWB localisation system with battery-powered anchor nodes antipodal modulation (DS-UWB) have been built for image transmission of UWB systems [30,34]. There are several types of researches about comparing in the AWGN channel; the performance of the modulation UWB in these systems is different. Researchers showed that the DS-UWB systems with antipodal modulation have better performance and lower complexity towards PPM [35,36]. Orthogonal STBCs represent reputable and straight transmission patterns, which can tackle the same diversity order as the classical maximal-ratio combining. Since their original inventions by Alamouti in [37] and the generalisation by Tarokh and Calderbank in [38], they used it in the multiple-input–multiple-output (MIMO) communication systems. STBCs do the Maximum-Likelihood (ML) decoding and can be performed at the receiver, but the MIMO channel can be transformed into an equivalent scalar Gaussian channel with a response equal to the Frobenius norm of the channel matrix [39,40]. Some theoretical modelling and simulations [41,42] though have explained that the Nakagami-m distribution [43] and other situation are Rayleigh/Ricean distributions. The majority of related studies used the Rayleigh or Ricean fading statistics model [44,45]. A MIMO system has more bandwidth application for the broadband and achieves more data rates by using the MIMO antennas. To achieve more data rates of 1 Gb/s in wireless telecommunications, one can use the system combination of the UWB and MIMO technologies [46].

72

4.1.1

Imaging and sensing for unmanned aircraft systems, volume 2

The efficiency of the flat Rayleigh fading channel

The symbol error probability for a frequency-non-selective Rayleigh fading channel in QPSK (M ¼ 4) can be presented as follows [47]: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi! 3 1 Es =N0 Es =N0 QPSK 1 :cot (4.1) ¼   ps 2 þ Es =N0 4 p 2 þ Es =N0 The multiple description coding (MDC) technique is suitable for wireless environments by assuming the risk of losing data. These techniques are used to improve the protection of image transmission and to resist the corruption of data transmitted over wireless channels [48]. This chapter investigates the transmission of MDC images on OSTBC MIMO channels with QPSK modulation. Transmission of MDC images in the MIMO system is new research that with attention to a study conducted already has not been performed. This chapter evaluates the overall success rate of the proposed wireless Rayleigh channel to compare the achieved results with the results of [49]. The influence of each block is studied by changing the simulated blocks and their valid parameters. The achieved results of the method of MDC and single description coding (SDC) are compared with each other. The chapter is structured as follows. Section 4.2 goes over the image MDC, along with its implementation. Then, the subsequent section (Section 4.3) describes the MIMO system. Sections 4.4 and 4.5 investigate diversity and ST coding, respectively. Section 4.6 presents the results of several simulations with different sets of t parameters changed to influence the implementation of the ideas this chapter proposes. To close, Section 4.7 presents and discusses the conclusions and ideas for future works.

4.2 Multiple description coding MDC is a technique based on source code that protects the image quality against the errors of the channel. The main idea in MDC is to subsample the original image into four sub-images. Figure 4.3 illustrates this procedure where each subsampled image has its pixels representation by a given colour and shape as follows: red circle pixels correspond to Image Version (IV) 1, blue square pixels form IV 2, the yellow diamond-shaped pixels form IV 3, and green stars-shaped pixels form IV 4. For the sake of testing and validating the proposed system, the 256  256 Cameraman image is subsampled, thus resulting in 128  128 low-resolution pictures that will feed the MDC simulation like in Figure 4.4. Each version is divided into 4  4 blocks. After that, the DCT transform of each block is calculated with the DC coefficient, and the next two AC coefficients of the blocks can be mapped to a vector. Hence, this vector feeds procedures corresponding to quantisation, source coding, channel coding, spreading, and modulation to send signals through the channel. In the UAV receiver part, the operation will be done reversely. Figure 4.5 depicts a block diagram of the overall four-stage

Image transmission in UAV MIMO UWB-OSTBC system

73

Figure 4.3 Polyphase subsampling in the space domain image

(a)

(b)

(c)

(d)

Figure 4.4 Sub-images created in the transmitter framework. According to Figure 4.4, the original input image is divided into two or more versions of data (sub-images) [50]. Each version has a satisfactory quality of the original image. If the UAV receiver has received all copies, then the decoder reconstructed image data in high quality. Otherwise, the channel decoder will have low quality. Decoders show the obtained sub-sampled image copies. Decoder 1234, for example, indicates that all four versions 1, 2, 3, 4 have been received, and the rest

74

Imaging and sensing for unmanned aircraft systems, volume 2 DCT/ Quantisation/ Zig-Zag Scan/ Huffman coding DCT/ Quantisation/ Zig-Zag Scan/ Huffman coding

Decoder 1234 Channel 1

Decoder 234 Decoder 123 Decoder 134

Channel 2

Decoder 124 Decoder 12 Decoder 13

Source Decoder 14 DCT/ Quantisation/ Zig-Zag Scan/ Huffman coding

Decoder 23 Decoder 24 Channel 3 Decoder 34

DCT/ Quantisation/ Zig-Zag Scan/ Huffman coding

Decoder 1 Channel 4 Decoder 2 Decoder 3 Decoder 4

Figure 4.5 The structure of four versions of the coding system based on the DCT of decoders presents the version numbers. Decode a string of input symbols: fXk gNk¼1 is transmitted through the four channels, fXik gNk¼1 is the string corresponding to the ith decoder. In this system, a total of 15 different scenarios are possible depending on the IV versions available at the UAV receiver for image reconstruction. Here, Decoder 1234 indicates that all four IVs have been received; Decoder 124 points to the reception of the IVs 1, 2 and 4; Decoder 34 signposts that IVs 3 and 4 have been received; and, finally, Decoder1 stands for the reception of only IV1 [50].

4.3 Multiple input–multiple output Using techniques like MIMO antennas and OFDM helps to bring the most out of the 50 Mb/s data rates. As recommended in the IEEE802.11n, to reach the target of

Image transmission in UAV MIMO UWB-OSTBC system

75

1 Gb/s, more advanced techniques should be used. The UWB technology combined with MIMO might provide a solution. If NT ¼ NR ¼ 1, it is called a single-input, single-output (SISO) system and the corresponding channel is a SISO channel. The output of a frequency-selective SISO channel can be described by: y½k ¼

L t 1 X

h½K; k:x½K  k þ n½K

(4.2)

k¼0

MIMO system is NI signals xm[k], 1  m  NI, that form the input at each time instant k and NO output signals yn [k], 1  n  NO, that pair (m, n) of inputs and outputs is adjoined by a channel impulse response hn[.], m[k, k] [47]. Figure 4.6 illustrates the structure of the MIMO system. yv ½k ¼

N1 L t 1 X X

hv;k ½K; k:xm ½K  k þ nv ½K

(4.3)

m¼1 k¼0

The subsequent discrete-time 2 3 2 y1 h11    6 7 6 6  7 6   6 7 6 6 7 6  6  7¼6  6 7 6 6  7 6   4 5 4 ym

hm1

 



model representing this system is: 32 3 2 3 h1n x1 N1 76 7 6 7  76  7 6  7 76 7 6 7 76 7 6 7  76  7 þ 6  7 76 7 6 7 6 7 6 7  7 54  5 4  5 hmn

xn

(4.4)

Nm

  y ¼ H xþN

(4.5)

 is the where x is the n-dimensional vector with the transmitted symbols, N m-dimensional Additive White Gaussian Noise (AWGN) vector, H is a matrix that contains zero-mean complex circular Gaussian random variables, and hij represent the channel gains from the transmitting antenna j to the receiving antenna i [51]. Figure 4.7 shows the general structure of the MIMO channel.

h11 y1

x1

y2

x2

xn

hmn

Figure 4.6 MIMO system

ym

76

Imaging and sensing for unmanned aircraft systems, volume 2 h1,1[k, κ] h2,1[k, κ]

x1[k]

n1[k] n2[k]

y1[K] y2[k]

x2[k] h2,N1[k, κ] xNI[k]

hNo,N1[k, κ]

nNo[k] yNo[k]

Figure 4.7 The general structure of a frequency-selective MIMO channel

4.4 Diversity MIMO systems afford diversity. Since there is a fading coefficient error in the mean value of the signal-to-noise ratio (SNR), AWGN channels have been improved a lot. The use of spatial diversity embraces methods to deal with the destructive effects of the channel. Composition spatial diversity with a type of diversity such as time or frequency can dramatically improve system performance. Time-space codes simultaneously make the degree of diversity in time and space. The spatial diversity systems use the multiple receive and transmit antenna arrays. The diversity gain represents the system reliability where the diversity gain states in the below equation reproduce the error rate evolution against the SNR [52]. logðpe ðSNRÞÞ SNR!1 logðSNRÞ

d ¼  lim

(4.6)

where pe(SNR) stands for the measured error rate at a fixed SNR value. One must use the NT transmitting antennas, and NR receiving antenna to acquire a maximum diversity gain of NTNR in a MIMO system. In the MIMO systems, the Orthogonal Space–Time Block Code (OSTBC) encoder has been used for encoding an input symbol sequence. OSTBC mixes the input signal (from all of the receiving antennas), and the channel estimates the signal to extract the soft information about the OSTBC-encoded symbols. The MSE and the Peak SNR (PSNR) are metrics to appraise the quality of the reconstructed image (after processing) given as a result of the following equations: MSE ¼

1 SM;N ½I1 ðm; nÞ  I2 ðm; nÞ2 M N

PSNR ¼ 10log10

2552 MSE

(4.7) (4.8)

where the previous expression assumes 256 grey levels; M and N correspond to a maximum number of rows and columns in the image, respectively; MSE presents the cumulative mean squared error between the reconstructed (I2) and the original (I1) images. The less value of the MSE, the less error on the image while the PSNR is increased.

Image transmission in UAV MIMO UWB-OSTBC system

77

4.5 Simulations results This simulation illustrates all the proposed system rationale with the images the UAV sends and receives, along with the steps for sending and receiving information on the basic communications systems in this UAV structure that are also used to transfer images, run tasks sequentially for the channel coding modulation steps, and its opposite steps are taken to get the final image. The block diagram for the proposed OSTBC-MIMO system for a UAV appears in Figure 4.8. The MDC block acquires the 256  256 Cameraman frame and then 128  128 subsampling process is applied to it. DCT is then used to the subsampled versions to quantise them into the next block to use an arithmetic code that is a source coding and at the next block using the convolutional channel coding. Using QPSK modulation, the channel is assumed to follow a Rayleigh distribution MIMO and gets the number of 2  2 and 2  1 antennas with 15 dB SNR. In the UAV receiver, the reverse of the performed operations corresponding to decoding happens. In the next step of the UAV receiver, demodulation and channel decoding and source decoding following by IDCT are applied to the signals. After combining received sub-images with 128  128 dimensions, they are restored with 256  256 dimensions. The final received versions of the image in the UAV receiver appear in Figure 4.9. The next section illustrates how the mutual antenna coupling affects the performance of an OSTBC transmission over a MIMO channel. The simulation of the QPSK-modulated Alamouti code for each SNR value with and without antenna coupling is shown in Figure 4.10. A realisation of the Alamouti code appears through the MIMO channel in each iteration. The 128  128 sub-sampled images are then combined to reconstruct the 256  256 original images with 15 dB SNR (Figure 4.11). Average PSNR values for the different number of lost versions using MDC with DCT transforms are given in Table 4.1. In cases where a prescription is lost, approximations copies of the missing parts can be recovered by averaging the matching pixels.

Read image

DCT

Quantisation

Source coding Arithmetic code

Channel coding Convolutional code

Spreading DS-UWB

Modulation QPSK

OSTBC coder

AWGN CHANNEL MIMO-UWB

Restoration image

IDCT

Source decoding

Channel decoding

Despreading

Demodulation

Figure 4.8 Block diagram of the proposed MIMO system

OSTBC combiner

78

Imaging and sensing for unmanned aircraft systems, volume 2

(a)

(b)

(c)

(d)

Figure 4.9 Sub-images obtained in the receiver

Alamouti-coded 2x2 System - High Coupling, High Correlation

100

Channel without coupling Channel with coupling

BER

10–1

10–2

10–3 0

1

2

3

4

5 6 SNR (dB)

7

8

9

10

Figure 4.10 The BER versus SNR curves plotted under different correlation and coupling scenarios

Image transmission in UAV MIMO UWB-OSTBC system

79

Figure 4.11 Image after the composition of the sub-image via QPSK

Table 4.1 PSNR versus different lost images scenarios Number of copies lost

PSNRSNR ¼ 15 dB

0 1 2 3

24.2543 23.0673 23.6530 22.2139

Table 4.2 PSNR DC coefficient images Number of copies lost

PSNRSNR ¼ 15 dB

0 1 2 3

20.5103 20.4975 20.4538 20.3125

In Table 4.2, the PSNR of the received image for one state is illustrated only for DC coefficients of the original version. The PSNR of the reconstructed image based on the number of lost copies have been analysed in this case. Looking at the PSNR, one can attest that the image with three coefficients of DC, and AC copies have better quality towards the version that has just DC coefficient images that PSNR three coefficients of DC and AC copies; because of using the three coefficients of the original image. Figure 4.12 shows the PSNR values for the number of lost copies. Figure 4.13 and Table 4.3 show a comparison of the system with and without MDC (SDC). In this method, there is a possibility for image loss. MDC method gives high quality of the image against the loss of versions for the SDC method.

80

Imaging and sensing for unmanned aircraft systems, volume 2 Average PSNR for our method 26 snr15 coefficient dc, ac snr1 coefficient dc, ac coefficient dc

25

PSNR

24 23 23 21 20 19

0

0.5

1

2.5 1.5 2 Number of description

3.5

3

4

Figure 4.12 Mean values of PSNR (dB) for different versions of the lost

Figure 4.13 Image obtained without MDC Table 4.3 PSNR image without MDC Image

PSNR

Image cameraman without MDC

28.6540

Modulation is QPSK, fading channel, SNR ¼ 15 dB, MIMO 2  2. PSNR method without using MDC is rarely high. Figure 4.14 illustrates the SNR-PSNR rate of the MDC for the four versions. The result of transmitting DC and next two AC coefficients of the original versions

Image transmission in UAV MIMO UWB-OSTBC system

81

RATE PSNR TO SNR 26 mimo2*2 mimo2*1

24

PSNR

22 20 18 16 14 12

0

2

4

6

8

10 snr

12

14

16

18

20

Figure 4.14 PSNR than SNR of MDC method with DCT

Table 4.4 PSNR image MIMO 2*1 Image

PSNR

Image cameraman

23.8765

is simulated. In this state, the number of the transmitter antenna is 2; the number of receive antenna is 2 (MIMO 2  2), and the number of the transmitter antenna is 2, the number of receive antenna is 1 (MIMO 2  1) that simulate is shown in Figure 4.14. Table 4.4 presents the PSNR in the image OSTBC MIMO system using the MDC with four received copies to the dimensions of 128  128, fading channel, and SNR ¼ 5, 10, 15 dB and MIMO 2  1 and Spread signals. In this case, the DC has one coefficient and AC side has two coefficients of original copies, which have been received with a combination of the received four copies obtained by an image in the size of 256  256. Figure 4.15 illustrates that the SNR fading channel versus the BER (error rate) of 12 signals (3 coefficients of 4 copies that are 12) obtained before modulate and after demodulate is drawn. In Figure 4.16, the error rate before channel coding and after it have been shown for 12 signals which are set to zero. Therefore, the channel code is used to correct all errors.

82

Imaging and sensing for unmanned aircraft systems, volume 2 BER AFTER demodulate over AWGN Channel

10–1

Empirical BER 10–2

BER

10–3

10–4

10–5

10–6

4

6

8

10 12 SNR (dB)

14

16

18

Figure 4.15 Error rates after demodulate BER AFTER DECODE CHANNEL 1 Empirical BER

0.8 0.6 0.4

BER

0.2 0 –0.2 –0.4 –0.6 –0.8 –1 4

6

8

10

12

14

16

18

SNR (dB)

Figure 4.16 The error rates after the channel decoder

4.6 Discussion and future trends UWB facilitates the smooth usage of UAVs in large-scale UAV-CPS applications by extending UWB localisation with monocular SLAM. Hence, UAV indoor

Image transmission in UAV MIMO UWB-OSTBC system

83

navigation becomes viable in zones without wireless localisation. Furthermore, UWB has scalability repercussions regarding performance. It may help cooperative SLAM involving multiple UAVs, lessening the computational load. UWB can constantly refine the positioning in the presence of sparse reference nodes with a given inertial sensors’ structure and particularly tailored for quality assessment of UWB signals [53,54]. UWB along with a micro Radar system with an Inertial Navigation System (INS) aids the navigation of small UAV in GNSS-denied situations. UWB and Radar are complementarians where the usage of Radar delivers enough evidence on the present direction while a stand-alone UWB subsystem may entail a considerable infrastructure to produce adequate location assessments throughout the flight. Velocity from the Radar and positions from the UWB systems can be integrated with an INS, through methodologies like the Extended Kalman Filter (EKF). An Inertial Measurement Unit (IMU) may exhibit a colossal drift owing to the added errors since it can reach several thousands of metres, which calls for an IMU subsystem like GPS, LIDAR, Radar, or UWB to curb inaccuracies. Furthermore, UWB and Radar can be integrated to augment the navigation performance to empower navigation in GNSS-denied settings at a lower cost in terms of infrastructure [55–59]. In communications encompassing compressed video streams is a serious candidate to channel errors degradation. Several approaches exist to mitigate these channel errors. Motion vectors (MVs) interpolation with the support of MV extrapolation (MVE) may conceal corrupted H.264/AVC frames. In UAV-CPSs relying on UWB, the MVs of lost blocks can be recovered or subject to error concealment [60–67] after interpolation with the extrapolated MVs [60–62,68]. Currently, handling video within UAV-CPSs with the H.264/AVC standard [60] is challenging. Some exciting research such as [68] tries to stretch this even more by adopting an H.265 encoder with an adaptive neuro-fuzzy inference system (ANFIS) controller that includes transmission modules, a control portion, and receiver stage with excellent ER and MSE values. Wireless visual area networks (WVAN) appear everywhere, are stress-free to install, and are more accessible than various technologies. A WPAN can fit lowspeed or high-speed applications. As technology heads towards high-definition videos and other high-end images and video communications, WVAN struggles to deliver quality output. Hence, to unravel this shortcoming, UWB technology is merged with WVAN to shape the current loopholes, and it results in great quality output with outstanding speed and straightforward access. UWB distinctive features, for example, high-speed data transmission, low power consumption, inexpensiveness, and short-range, make it a choice for delivering high-end imageries and videos in WVAN topology. UWB can be used along with in-loop filtering method to get rid of the blocking artefacts in the output. If computational intelligence is combined with in-loop filtering for classification purposes, then more robustness will be likely achieved with a lesser computational load [61–67,69–71]. One of the ultimate challenges for UAVs is a safe landing. From now, visual deployed approaches will become increasingly popular because they are appropriate

84

Imaging and sensing for unmanned aircraft systems, volume 2

for landing within the GNSS-denied settings, especially when employing ground computing resources for the deployed guidance subsystem and feedbacks the UAV real-time localisation to its autopilot. A distinct long-baseline stereo framework can have an extendable baseline with a wide-angle field of view (FOV) instead of the orthodox fixed-baseline structures under such challenging circumstances. Moreover, theoretical modelling, in addition to computational analysis, assist architectural accuracy evaluations. The UWB positioning network, together with CV-based navigation, needs further discussions by the community. Passive UWB anchors along the runway may listen to the UWB signals from UAVs. The ground system can determine the target position contingent to the UWB anchors’ geometry and, then, send it back to the UAV. All forms of wireless channels employed in UAV-CPSs, for example, RF, optical, acoustical, among others, convey finite resources for remote sensing (RS) with Radar and data communications. Frequently, these two purposes are conflicting, and they contend for common or incompatible resources. WVAN applications are escalating fast, which calls for RF convergence as a prerequisite for all stakeholders to evolve. The ample solution space connected to this multifaceted impasse comprehends collaboration or code signing of CPSs with sensing as well as communications capabilities together. By bearing in mind the UAV-CPSs’ needs in unison throughout the design stage, rather than propagating the mutual interference notion, all system’s components can have performance enhancements. The UAV-CPS community needs a proper departure design procedure for future enterprises that take into account the applications, topologies, levels of integration, the state of the art while outlining the upcoming data-centric structures [72,73]. Applications like physical activities, sports, and entertainment demand further studies of UWB moving tags in terms of ranging accuracy and precision. These investigations can confirm some previously obtained results. There is evidence that antenna orientation impacts little the precision of ranging in UWB, while obstacles influence precision more. Careful UWB parameter tuning and additional new experimental results can impact the precision of the measurements [74].

4.7 Conclusion First, this chapter introduces ultra-broadband systems. Next, a hybrid structure based on OSTBC MIMO systems for UAV-CPS image transmission using UWB with QPSK modulation throughout Rayleigh channels is depicted. Multiple descriptions coding of the image, DCT block of the original image, and how to get the pictures are presented. The MIMO channels examined were the 2  2 and 2  1 with their respective results compared. Image transmission relying on a number of selected prescription coefficients found by states’ spreading with SNR difference, MDC, and SDC. The outcomes showed that at low SNR, the PSNR does not change. Several scenarios for MDC decoding are examined, and it was shown that the OSTBC-MIMO system for UAV image transmission is robust. The PSNR is presented with and without MDC. MIMO channel with antenna selection and

Image transmission in UAV MIMO UWB-OSTBC system

85

Alamouti coding. The BER versus SNR curves Alamouti system are simulated. Future work in channel modelling can take into account the increasing phenomenon of fading with high accuracy.

References [1] Tractica, “Commercial drone shipments to surpass 2.6 million units annually by 2025,” July 2015, pp. [2] A. K. A. M. amd A. Tuncer, and I. Guvenc, “Unmanned aerial base stations for public safety communications,” IEEE Vehic. Technol. Mag. 2016. [3] H. J. Loschi, V. V. Estrela, O. Saotome, et al., “Emergency response cyberphysical framework for landslide avoidance with sustainable electronics,” Technologies 2018, 6, p. 42. doi:10.3390/technologies6020042. [4] A. C. B. Monteiro, R. P. Franc¸a, V. V. Estrela, et al., “Health 4.0: applications, management, technologies and review,” Medical Technologies Journal, 2019, 2(4), pp. 262–276. [5] T. O. Edoh, “Smart medicine transportation and medication monitoring system in EPharmacyNet,” 2017 Int. Rural Elderly Health Inf. Conf. (IREHI), Lome, Togo, 2017, pp. 1–9. doi: 10.1109/IREEHI.2017.8350381. [6] F. C. Commission, “First report and order 02-48,” Apr. 2002. [7] S. G. I. Guvenc, and Z. Sahinoglu, “Ultra-wideband range estimation: theoretical limits and practical algorithms,” in Proc. IEEE Int. Conf. Ultra-Wideband (ICUWB) 2008, 3, pp. 93–96. [8] S. G. I. Guvenc, Z. Sahinoglu, and U. C. Kozat, “Reliable communications for short-range wireless systems,” 2011. [9] J. Zhao, F. Gao, G. Ding, T. Zhang, W. Jia, and A, Nallanathan, “Integrating communications and control for UAV Systems: Opportunities and Challenges.” IEEE Access, 6, 67519-67527, 2018. [10] V. V. Estrela, and A. M. Coelho. “State-of-the-Art motion estimation in the context of 3D TV.” In: Multimedia Networking and Coding. IGI Global, Hershey, PA, USA, 2013. 148–173. doi:10.4018/978-1-4666-2660-7.ch006. [11] S. R. N. Goddemeier, and C. Wietfeld, “Experimental validation of RSS driven UAV mobility behaviors in IEEE 802.11s networks,” in 2012 IEEE Globecom Workshops (GC Wkshps), Anaheim, CA, USA, Dec. 2012, pp. 1550–1555. [12] M. H. S. Lupashin, M. W. Mueller, A. P. Schoellig, M. Sherback, and R. D’Andrea, “A platform for aerial robotics research and demonstration,” Flying Machine Arena. Mechatronics, 24, 1, 2014. [13] D. H. S. J. Ingram, and M. Quinlan, “Ultrawideband indoor positioning systems and their use in emergencies,” in Proc. Position Location Navigation Symp. PLANS 2004, Monterey, CA, USA, Apr. 2004, pp. 706–715. [14] L. W. M. Vossiek, P. Gulden, J. Weighardt, and C. Hoffmann, “Wireless local positioning – concepts, solutions, applications,” in Radio Wireless Conf., 2003. RAWCON ’03. Proc. Boston, MA, USA, Aug. 2003, pp. 219–224.

86

Imaging and sensing for unmanned aircraft systems, volume 2

[15]

M. Z. W. a. R. A. Scholtz, “Impulse radio: how it works,” Commun. Lett., IEEE, Feb. 1998, 2, pp. 36–38. R. A. S. J. M. Cramer, and M. Z. Win, “Spatio-temporal diversity in ultrawideband radio,” in Wireless Commun. Network. Conf. WCNC. 1999 IEEE 1999, 2, pp. 888–892. F. N. a. F. Remondino, “UAV for 3D mapping applications: a review,” Appl. Geomatics 2014, 6, pp. 1–15. H. K. Amit Shukla, “Application of robotics in onshore oil and gas industry a review part i,” Robot. Auton. Syst. 2016, 75, pp. 490–507. C. Z. a. J. M. Kovacs, “The application of small unmanned aerial systems for precision agriculture: a review,” Precision Agric. 2012, 13, pp. 693–712. D. S. J. Qi, H. Shang, N. Wang, et al., “Search and rescue rotary-wing UAV and its application to the Lushan Ms 7.0 earthquake,” J Field Robot. 2016, 33, pp. 290–321. Y. K. a. I. Nielsen, “A system of uav application in indoor environment,” Prod. Manuf. Res. 2016, 4, pp. 2–22. Z. Q. K. Guo, C. Miao, A. H. Zaini, et al., “Ultra-wideband-based localization for quadcopter navigation,” 2016, 4, pp. 23–34. J. B. N. Macoir, B. Jooris, B. V. Herbruggen, et al., “UWB localization with battery-powered wireless backbone for drone-based inventory management,” Sensors, 19(3), 2019, 467. V. V. Estrela, A. Khelassi, A. C. B. Monteiro, et al., “Why software-defined radio (SDR) matters in healthcare?”. Medical Technologies Journal, 3(3), 2019, pp. 421–9, doi:10.26415/2572-004X-vol3iss3p421–429. A. Herrmann, V. V. Estrela, and H. J. Loschi, “Some thoughts on transmedia communication,” OJCST, 11(4), 2018. doi: 10.13005/ojcst11.04.01. N. Razmjooy, B. S. Mousavi, F. Soleymani, and M. H. Khotbesara, “A computer-aided diagnosis system for malignant melanomas,” Neural Comput. Appl. 2013, 23, pp. 2059–2071. P. Moallem, N. Razmjooy, and M. Ashourian, “Computer vision-based potato defect detection using neural networks and support vector machine,” Int. J. Robot. Autom. 2013, 28, pp. 137–145. P. Moallem, and N. Razmjooy, “Optimal threshold computing in automatic image thresholding using adaptive particle swarm optimization,” J. Appl. Res. Tech. 2012, 10, pp. 703–712. N. Razmjooy, B. S. Mousavi, M. Khalilpour, and H. Hosseini, “Automatic selection and fusion of color spaces for image thresholding,” Signal Image Video Process. 2014, 8, pp. 603–614. H. Z. T. Lv, X. Wang, and X. r. Cui, “A selective approach to improve the performance of uwb image transmission system over indoor fading channels,” in Proc. 6th Int. Conf. Wireless Commun. Networking Mobile Comput. (WiCOM), Shenzhen, China, 2010, pp. 1–4. D. A. do Nascimento, Y. Iano, H. J. Loschi, et al., “Sustainable adoption of connected vehicles in the Brazilian landscape: policies, technical specifications

[16]

[17] [18] [19]

[20]

[21] [22] [23]

[24]

[25] [26]

[27]

[28]

[29]

[30]

[31]

Image transmission in UAV MIMO UWB-OSTBC system

[32]

[33] [34]

[35]

[36]

[37]

[38] [39]

[40] [41]

[42]

[43] [44]

[45] [46] [47]

87

and challenges,” Trans. Env. Elect. Eng. 2019, 3(1), pp. 44–62. doi: 10.22149/ teee.v3i1.130. N. Razmjooy, and M. Ramezani, “Using quantum gates to design a PID controller for nano robots,” Int. Res. J. Appl. Basic Sci. 2014, 8, pp. 2354–2359. I. Oppermann, and A. M. Hamalainen, “UWB theory and applications,” Wiley, New Jersey, USA, 2004, pp. 1–7. M. Z. W. a. R. A. Scholtz, “Ultra-wide bandwidth time-hopping spreadspectrum impulse radio for wireless multiple-access communications,” IEEE Trans. Commun. Apr. 2000, 48, pp. 679–690. K. K. Z. Bai, “Performance analysis of multiuser DS-PAM and TH-PPM UWB systems in data and image transmission,” In Proc. 2005 International Conference on Wireless Communications, Networking and Mobile Computing, Wuhan, China, 2005. W. Z. Z. Bai, S. Xu, W. Liu, and K. Kwak, “On the performance of multiple access DS-BPAM UWB systems in data and image transmission,” Proc. IEEE International Symposium on Communications and Information Technology, Beijing, China, 2005, ISCIT 2005, 2, pp. 851–854. S. M. Alamouti, “A simple transmit diversity technique for wireless communications,” IEEE J. Select. Areas Commun. Oct. 1998, 16, pp. 1451–1467. H. J. V. Tarokh, and A. R. Calderbank, “Space–time block codes from orthogonal designs,” IEEE Trans. Inf. Theory July 1999, 45, pp. 1456–1467. T. L. X. Li, G. Yue, and C. Yin, “A squaring method to simplify the decoding of orthogonal space–time block codes,” IEEE Trans. Commun. Oct. 2001, 49, pp. 1700–1703. H. S. a. J. H. Lee, “Performance analysis of space–time block codes over keyhole Nakagami-,” IEEE Trans. Veh. Technol. Mar. 2004, 53, pp. 351–362. A. H. H. Shah, and A. Nosratinia, “Performance of concatenated channel codes and orthogonal space–time block codes,” IEEE Trans. Wireless Commun. June 2006, 5, pp. 1406–1414. A. M. a. S. Aıssa, “Capacity of space–time block codes in MIMO Rayleigh fading channels with adaptive transmission and estimation errors,” IEEE Trans. Wireless Commun. Sept. 2005, 4, pp. 2568–2578. H. Suzuki, “A statistical model for urban radio propagation,” IEEE Trans. Commun. July 1997, 25, pp. 673–679. D. I. L. M. Matthaiou, and J. S. Thompson, “A MIMO channel model based on the Nakagami-faded spatial eigenmodes,” IEEE Trans. Ant. Propag. May 2008, 56, pp. 1494–1497. W. C. Hoffman, Statistical Methods in Radio Wave Propagation, Pergamon Press, Oxford, UK, 1960, pp. 3–36. F. Z. T. Kaiser, and E. Dimitrov, “An overview of ultra-wide-band systems with MIMO,” in Proc. IEEE Feb. 2009, 97, pp. 285–312. V. Kuhn, “Wireless communications over MIMO channels,” Universitat Rostock, Germany 2006, pp. 17.

88

Imaging and sensing for unmanned aircraft systems, volume 2

[48]

A. R. R. Y. Wang, and S. Lin, “Multiple description coding for video delivery,” Proc. IEEE Jan. 2005, 93, pp. 57–70. V. S. Somayazulu, “Multiple access performance in UWB systems using time hopping vs. direct sequence spreading,” in Proc. IEEE Wireless Communications and Networking Conf Mar. 2002, 2, pp. 522–525. M. N. A. Arshaghi, and M. Ashourian, “Image transmission in MIMO-UWB systems using multiple description coding (MDC) over AWGN and fading channels with DS-PAM modulation,” World Essays J. 2017, 5, pp. 12–24. A. Goldsmith, “Wireless communications,” Cambridge University Press, MA, USA, pp. 315–320. H. Jafarkhani, “Space–time coding theory and practice,” Cambridge University Press, MA, USA, ISBN: 978–0–511–11562–2. J. Tiemann, A. Ramsey, and C. Wietfeld, “Enhanced UAV indoor navigation through SLAM-augmented UWB localization,” in Proc. IEEE Int. Conf. Commun. Workshops (ICC Workshops), 2018, Kansas City, MO, USA, pp. 1–6. J. Tiemann, F. Eckermann, and C. Wietfeld, “ATLAS – an open-source TDOA-based ultra-wideband localization system,” in 2016 Int. Conf. Indoor Positioning Indoor Navigation (IPIN), Alcala de Henares, Madrid, Spain, Oct. 2016. S. Zahran, M, Mostafa-Sami, A. Masiero, A. Moussa, A. Vettore, and N. El-Sheimy, “Micro-radar and UWB aided UAV navigation in GNSS denied environment,” In Proceedings of the 2018 ISPRS TC I Mid-term Symposium: Innovative Sensing – From Sensors to Methods and Applications, Hannover, Germany, 2018; pp. 469–476. C. Cadena, L. Carlone, H. Carrillo, et al., “Past, present, and future of simultaneous localization and mapping: toward the robust perception age,” IEEE Trans. Robot. 2016, 32(6), pp. 1309–1332. K. Kauffman, J. Raquet, Y. Morton, and D. Garmatyuk, “Real-time UWBOFDM radar-based navigation in unknown terrain,” IEEE Trans. Aerospace Electron. Syst. 2013, 49(3), pp. 1453–1466. E. Kim, and D. Choi, “A UWB positioning network enabling unmanned aircraft systems auto land,” Aerospace Sci. Technol. 2016, 58, pp. 418–426. G. A. Kumar, A. K. Patil, R. Patil, S. S. Park, and Y. H. Chai, “A LiDAR and IMU integrated indoor navigation system for UAVs and its application in real-time pipeline classification,” Sensors, MDPI, 17(6), 1268, 2017. H. Marins, and V. V. Estrela, “On the use of motion vectors for 2D and 3D error concealment in H.264/AVC video,” in Feature Detectors and Motion Detection in Video Processing, ed. N. Dey, A. Ashour, and P. Kr. Patra, IGI Global, Hershey, PA, USA, 2017, pp. 164–186. doi: 10.4018/978-1-52251025-3.ch008. J. Zhou, B. Yan, and H. Gharavi, “Efficient motion vector interpolation for error concealment of H.264/AVC,” IEEE Trans. Broadcasting 2011, 57, pp. 75–80.

[49]

[50]

[51] [52] [53]

[54]

[55]

[56]

[57]

[58] [59]

[60]

[61]

Image transmission in UAV MIMO UWB-OSTBC system

89

[62] M. A. de Jesus, and V. V. Estrela, “Optical flow estimation using total least squares variants,” Orient. J. Comp. Sci. Technol. 2017, 10(3), pp. 563–579. doi: 10.13005/ojcst/10.03.03. [63] Y. Li, and R. Chen, “Motion vector recovery for video error concealment based on the plane fitting,” Multimedia Tools Appl. 2017, 76, pp. 14993– 15006. [64] T. L. Lin, C. J. Wang, T. L. Ding, et al., “Recovery of lost color and depth frames in multiview videos,” IEEE Trans. Image Process. 2018, 27, pp. 5449–5463. [65] M. Usman, X. He, K. M. Lam, et al., “Frame interpolation for cloudbased mobile video streaming,” IEEE Trans. Multimedia 2016, 18, pp. 831–839. [66] D. Chatzopoulos, C. Bermejo, Z. Huang, and P. Hui, “Mobile augmented reality survey: from where we are to where we go,” IEEE Access 2017, 5, pp. 6917–6950. [67] Y. L. Chen, H. Wang, Y. Hu, and A. Malik, “Intra-frame error concealment scheme using 3D reversible data hiding in mobile cloud environment,” IEEE Access 2018, 6, pp. 77004–77013. [68] S. M. A. Salehizadeh, “Motion and noise artifact detection and vital signal reconstruction in ECG/PPG based wearable devices.” Ph.D. Thesis, University of Connecticut, Storrs, CT, USA, 2015. [69] D. J. Hemanth, and V. V. Estrela, “Deep learning for image processing applications,” Advances in Parallel Computing Series, IOS Press, Netherlands, 2017, Vol. 31. ISBN 978-1-61499-821-1 (print), ISBN 978-161499-822-8 (online). [70] N. Razmjooy, V. V. Estrela, H. J. Loschi, and W. S. Farfan, “A comprehensive survey of new meta-heuristic algorithms,” in Recent Advances in Hybrid Metaheuristics for Data Clustering, ed. S. De, S. Dey, and S. Bhattacharyya, Wiley, New Jersey, USA, 2019. [71] N. Razmjooy, and V. V. Estrela, “Applications of image processing and soft computing systems in agriculture.” IGI Global, Hershey, PA, USA, 2019, pp. 1–337. doi:10.4018/978-1-5225-8027-0. [72] B. Paul, A. R. Chiriyath, and D. W. Bliss, “Survey of RF communications and sensing convergence research,” IEEE Access 2017, 5, pp. 252–270. [73] S. C. Surender, R. M. Narayanan, and C. R. Das, “Performance analysis of communications & radar coexistence in a covert UWB OSA system,” in Proc. IEEE Global Telecommun. Conf. (GLOBECOM), Miami, FL, USA, 2010. pp. 1–5. [74] T. Risset, C. Goursaud, X. Brun, K. Marquet, and F. Meyer, “UWB ranging for rapid movements,” in Proc. International Conference on Indoor Positioning and Indoor Navigation (IPIN) Nantes, France, 2018, pp. 1–8. [75] W. Kong, T. Hu, D. Zhang, L. Shen, and J. Zhang, “Localization framework for real-time UAV autonomous landing: an on-ground deployed visual approach,” Sensors, 17(6), 1437, 2017.

90

Imaging and sensing for unmanned aircraft systems, volume 2

[76]

D. Garmatyuk, Y. J. Morton, and X. Mao, “On co-existence of in-band UWB-OFDM and GPS signals: tracking performance analysis,” in Proc. IEEE/ION Position, Location Navigat. Symp., Monterey, California, USA, May 2008, pp. 196–202. C. B. Barneto, L. Anttila, M. Fleischer, and M. Valkama, “OFDM radar with LTE waveform: Processing and performance,” In 2019 IEEE Radio and Wireless Symposium, RWS 2019 [8714410] (IEEE Radio and Wireless Symposium, RWS). IEEE Computer Society Press. https://doi.org/10.1109/ RWS.2019.8714410

[77]

Chapter 5

Image database of low-altitude UAV flights with flight condition-logged for photogrammetry, remote sensing, and computer vision Helosman Valente de Figueiredo1, Osamu Saotome1, Elcio H. Shiguemori2, Paulo Silva Filho2, and Vania V. Estrela3

The growth in the number of aerial images available is stimulating research and development of computational tools capable of extracting information from these image databases. However, developing a new computer vision (CV) software is complicated because many factors influence the extraction of information from aerial images, such as lighting, flight altitude, and optical sensors. The CV has been incorporated in most modern machines such as autonomous vehicles and industrial robots. The aim is to produce a high-quality image database of low-altitude Unmanned Aerial Vehicle (UAV) flights with flight condition-logged for photogrammetry, remote sensing, and CV. This work resulted in a collection of aerial images in the visible and thermal spectrum, and this set of images was captured in different schedules of the day, altitudes of flight, times of the year. The cameras are synchronised with the UAVs autopilot, and they were spatially and spectrally characterised in the laboratory. This research makes available low altitude aerial images of a region in Brazil to all community, with the precise flight and capture information, as well as additional features such as ground truth and georeferenced mosaic. Examples of the use of the database are shown for mosaic generation and development of CV algorithms for autonomous navigation of UAVs [1,2]. Furthermore, this database will serve as a benchmark for the development of the CV algorithms suited for autonomous navigation by images.

1 Divisao de Engenharia Eletronica, Instituto Tecnologico Aeronautica – ITA, Sao Jose dos Campos, SP, Brazil 2 Instituto de Estudos Avanc¸ados, Departamento de Ciencia e Tecnologia Aeroespacial, Forca Aerea Brasileira, Sao Jose dos Campos, SP, Brazil 3 Departamento de Engenharia de Telecomunicacoes (TET), Universidade Federal Fluminense (UFF), Rio de Janeiro, Brazil

92

Imaging and sensing for unmanned aircraft systems, volume 2

5.1 Introduction With the increasing number of images available in the geographic databases, the need for tools that efficiently perform information extraction, manipulation, and storage upsurge [3,4]. The development of tools for computer vision (CV) in Unmanned Aerial Vehicles (UAVs) and remote sensing (RS) applications requires the employment of a proper image database to develop and evaluate the proposed improvements [5,6]. Autonomous image navigation systems are used in regions where the Global Navigation Satellite System (GNSS) pre-sent unacceptable errors. Recent developments in photogrammetry and CV have produced significant advances in mobile mapping systems [7,8]. These advances enable UAVs to perform missions without relying on the availability of GNSS in the region to exploit visual-controlled flights in same way manned aircraft pilots do meet regulations, known as Visual Flight Rules (VFR). When using VFR, the pilot should be able to look out of the cockpit and perform attitude and navigation control based on references on the ground. Aviation schools instil this type of navigation for manned aircraft pilots because this expertise enables pilots to land in safety, in the case of failures in attitude and navigation systems [9]. Usually, the UAVs have a GNSS-based navigation scheme, so when this system fails, the aircraft has its autonomy reduced. For an Unmanned Aerial System (UAS), CV should compensate for the lack of information due to GNSS unavailability. Visual odometry, photogrammetry, and georeferencing software should be improved to implement a similar visual flight system [10]. Some researchers propose navigation systems with the aid of satellite images and Simultaneous Localisation and Mapping (SLAM), as seen in the works of [11] and [12]. Recurrent image navigation challenges are selecting landmarks amount of data to be processed [13], alterations in land occupations, climate modifications, changes in natural lighting [14]. A sizeable specific database containing the relevant features of the target environment is required to develop the autonomous flight algorithms because this problem should be split into parts such as selection, detection, tracking of the landmarks, estimation of the relative position of the aircraft, and the new calculation and a new corrected trajectory. A crucial development issue in embedded CV systems is the creation of appropriate benchmark databases to help devise algorithms, and tests, which requires a description of the optical sensors in use, the illumination at collection time, and accurate geo-position must be considered [15]. This work describes the characteristics of new aerial image database called AERIALTOUR CAMPUS UNIVAP that differs from others mainly because (i) it treats a specific region with particular geographic features present in Brazil and (ii) it has a specific time scale to evaluate the influence of light in the images.

5.1.1

Image processing system for UAVs

The use of image processing systems has been increased in the current UAVs and can be noted in target tracking systems, as well as auxiliary systems for correction of inertial sensor errors [16].

Image database of low-altitude UAV flights

93

The work shown in [17] among others discusses the autonomy of image processing methods in UAVs, for navigation, surveillance, RS, and systems that handle simultaneously mapping and positioning. In other works, such as in [1,18–22], the authors enlist and analyse characteristics for detection of landmarks and autonomous landing. For the development and test of applications of CV-oriented UAVs, videos under real flight conditions are necessary. Depending on the region of operation, the image-based navigation system must recognise different characteristics, as is the case of overflights, forests, oceans, deserts, and cities according to [23]. The works mentioned earlier show that image processing will be an essential part of the automation of UAVs. Furthermore, there is a growing interest from military and civilian organisations for researchers in this area, especially when GNSS information is denied or impossible to be obtained for some situations or environments [23,24]. Recently, the growth of the use of digital image processing techniques has generated a significant amount of database for testing and development of these technologies. Some authors have compiled collections in an image database [25,26]. In his work, Fisher organised a collection of image processing techniques and some images database. Price’s research presents a website with diverse information on CV. Both collections try to organise repositories divided into sub-areas of interest, such as image transformations and filters, geometric image features methods, sensor fusion, registration and planning methods, among others. For RS, people have large databases maintained by governments, such as the US government database kept by the U.S. Geological Survey [27]. There is also a database for specific studies such as that maintained by the University of Arkansas, assembled to tackle specific problems such as archaeological site characterisation [28]. However, the authors did not find specific image database related to low altitude images in Brazil along with the corresponding flight log information of the aircraft, such as latitude, longitude, altitude, and attitude. This knowledge is vital for the development of CV algorithms to access the accuracy of the positioning and attitude estimation algorithms through images, besides enabling specific photogrammetric corrections. Some works on databases treat specific characteristics of Brazil, for example [29], which studies the coffee region of Minas Gerais. This image database came from the orbital sensors, with 64  64 pixel images, thus making it challenging to recognise landmarks accurately. The images obtained in Brazil were acquired by amateurs and air videos aimed at marketing and dissemination of businesses, without the necessary characteristics for developing CV algorithms [30–32]. None of the cases met the needs of georeferencing and in diverse flight conditions, which are mandatory for development and testing of the algorithms. Most of the image databases found in Brazil consisted of orbital images (collected at 822 km) or high altitude aerial photogrammetry (starting at 600 m). Until the end of this work, there was no image database captured with UAV at low altitude (up to 120 m). This chapter proposes a database of low altitude aerial images captured by a UAV, called AERIALTOUR CAMPUS UNIVAP where the camera is

94

Imaging and sensing for unmanned aircraft systems, volume 2

synchronised with the UAV autopilot. The cameras employed have precise information because they were characterised spatially and spectral in the laboratory [33,34]. The system allows for closed-loop flights, flights under different illuminations conditions, as well as RGB and thermal image sensors according to different seasons of the year. This research makes available low altitude aerial images of a region in Brazil to all community, with the respective information of the flight realised for capture, as well as additional features as ground truth and georeferenced mosaic [34,35]. As an example of the use of the database are presented four works, two works related to mosaic generation and two related to the development of CV algorithms for autonomous navigation of UAVs. This chapter is organised as follows. Section 5.2 presents the image database proposal. Image capture processes to build the database are presented in Section 5.3. Section 5.4 presents the results and the uses of a UAV image database targeting Brazilian geography. Uses of the Image Database are examined in Section 5.5. Conclusions and future works are presented in Section 5.6.

5.2 The aerial image database framework The recommended image database is intended for developing and testing CV algorithms for localisation, mapping, and navigation. Thus, it must have unique characteristics such as flight altitude of 30 to 120 m, acquisition routes in closed circuits, different soil coverings, acquisition at different times of the day with different illuminations, georeferencing of images, and flight data log. The following sections add more details to its design and time label.

5.2.1

Database requirements

A versatile database must be created to systematically evaluate the accuracy and performance of CV algorithms targeting Brazilian landscapes. The following requirements must be kept in mind: inclusion of different landmarks for aerial systems with diverse colours, sizes and shapes; different illumination levels should be considered; operation of flights under various climatic conditions; image capture in different altitudes; image capture with different sensors; image acquisition considering regions of Brazil.

5.2.2

Database design

The chosen areas for image capture are part of the UNIVAP Urbanova Campus. These areas have been selected because they have different classes, among them, forests, lakes, paved roads, unpaved roads, road intersections, and constructions. The extension of the routes was defined, based on the flight characteristics of the aircraft. Figure 5.1 displays the selected route. The collected data is organised into folders and sub-folders for acquisitions in the visible and thermal spectra, where each folder has a sub-folder for each acquisition made.

Image database of low-altitude UAV flights

95

Figure 5.1 Route 1, green pins indicate the crossing points are numbered in ascending order to indicate the direction of the route

5.3 Image capture process For the first route shown in Figure 5.1, the aircraft chosen to capture the images was a quadcopter type multi-rotor. This aircraft was implemented with the Ardupilot autopilot, which is an opensource platform for small aircraft. At first, the aircraft was configured with the camera for the acquisition of RGB images, and then the aircraft was modified to accommodate the thermal camera and its parts. Two cameras were used to capture images: one for RGB images and another to capture thermal images. The reasons for choosing these cameras will be described in more detail. The Canon Power-Shot S110 camera was chosen for its optical characteristics: 12.1 megapixels, and shutter speeds interval of 1-1/2,000 s. This camera allows integration with Ardupilot and the Canon Hack Development Kit (CHDK), which is an open-source project. CHDK aims to increase the capabilities of the Canon cameras so that the autopilot can synchronise the camera shots. This is essential to add autopilot information to the captured images and to synchronise multiple cameras. With the use of this firmware, one can use the camera USB input to control the shots. This integration lets the camera data to be synchronised with the autopilot data, as well as to allow interfacing with CV applications, for example, applications to acquire images of specific targets only. This camera corresponds to the one used in [33], who evaluated it in three segments. The first one determined the sensitivity of the camera for each wavelength of incident electromagnetic radiation (spectral characterisation). The second segment performed radiometric characterisation to determine the ratio between a radiance of a radiation source (a sphere integrator) as measured by spectroscopy

96

Imaging and sensing for unmanned aircraft systems, volume 2

and a coloured signal, by the Canon S110 camera, of the same radiation source. The spatial characterisation, the third segment, occurred by the calculation of the Contrast Transfer Function (CTF) linked to the level of target perception, according to Johnson’s criterion. The spectral response function determined the spectral characterisation for each of the three Canon camera bands (R, G, B), that is the response for each wavelength of the incident radiation. Similarly, in the radiometric characterisation, for each of the three bands, the corresponding digital number was assigned as the radiation incident on the Canon camera. In the spatial characterisation, the standard USAF 1951 was used and was established in the CTF as threshold frequencies to detect, recognise, and identify a target [33]. The camera used to capture thermal images during the flight was the OptrisR PI450 G7. The main reason for choosing this camera was its market availability. Partners of the National Institute of Space Research indicated interests concerning the AERIALTOUR CAMPUS UNIVAP granted this camera was used. It has a resolution of 382  288 pixels, a maximum frame rate of 80 Hz (which can be reduced up to 27 Hz) with dimensions of 46 mm  56 mm  90 mm and weight of 320 g [36]. This camera must be connected to a recorder to be used in stand-alone solutions. The camera manufacturer itself provides a solution called PI Lightweight Kit, which is specific to applications that will be shipped in small UAVs [37,38]. These characteristics allow the incorporation of cameras to the quadcopter and acquisition of thermal images. The target region to start the acquisition of images for the database has climatic characteristics that made it difficult to work during the early hours of the morning. Since this area usually has a significant amount of fog, the data acquisition schedule was set up for 1:00 p.m., 4:00 p.m., and 6:00 p.m. local time. Acquisition at different times is necessary to evaluate how illumination influences algorithms robustness under real circumstances. Figure 5.2 shows one of these examples of variations. This image database has also ground truth masks manually annotated that can be used for the development of automatic classification algorithms. Figure 5.3 shows an example mask from this database. This mask is an image that can be placed on the georeferenced mosaic to evaluate the performance of the classification software. In Figure 5.3, the mask is represented by the red colour, which represents clay tiles. These masks were generated for objects larger than 9 m2 in area. To improve the precision of this image database, distinct markings in the ground were performed (control points) to guarantee benchmarks accuracy, which is in the order of centimetres. For the construction of the mosaics of the region of interest, several support points were chosen. This procedure involves much more than the minimum required for assembling the tiles with accuracy. This was done because this area would serve as a study region for the development of autonomous flight algorithms. With this amount of collected control points, we can separate this region and several microregions while maintaining the accuracy of the macro-region. These points

Image database of low-altitude UAV flights

(a)

1:00 p.m.

(b)

4:00 p.m.

(c)

5:00 p.m.

(d)

6:00 p.m.

Figure 5.2 Examples of variation of illumination

Figure 5.3 Clay tiles mask

97

98

Imaging and sensing for unmanned aircraft systems, volume 2 Support points Control points

BASE

Figure 5.4 Support points measured in soil with a differential GPS were measured with a differential correction GPS system. Data on these control points are also available in the image database. Figure 5.4 shows the distribution of the support points.

5.4 Results The main result of this work is a set of images with the characteristics necessary for the development and testing of computational vision algorithms for photogrammetry, RS, and autonomous image navigation. The main test cases treated in the example are variations in the conditions of natural light, different landmarks, variations in flight altitude. Section 5.4.1 presents some samples of this set of images and some of these test cases.

5.4.1

Images collected

Figure 5.5 illustrates the results obtained with this image database project by presenting some image samples. These images were collected in UAV flights performed by the same route at different schedules, time of the year, altitude, and climatic conditions. AERIALTOUR CAMPUS UNIVAP can be used for software development in the areas of photogrammetry, automatic classification, pattern recognition, environmental monitoring, and autonomous navigation. The samples presented in Figure 5.5 show the variation of the natural light, one of the requirements to evaluate the robustness of the CV algorithms. Another critical effect provided by natural lighting is the variation of the shadows. Figure 5.6 presents this case of the variance of the shadows. Another critical feature for this database is the altitude variation because the information that can be extracted from the landmarks is different for each altitude;

Image database of low-altitude UAV flights

(a)

Sample 1

(b)

Sample 2

(c)

Sample 3

(d)

Sample 4

99

Figure 5.5 Examples of collected images

(a)

Sample 1

(b)

Sample 2

(c)

Sample 3

Figure 5.6 Examples of shadow variation the higher the altitude of the UAV, the smaller the amount of details in the landmark. Figure 5.7 shows an example of this feature. In order to characterise and compare the optical sensors, standard targets (USAF1951) must be used. This database shows some flights at different altitudes on standardised targets. An example of this type of flight appears in Figure 5.8.

100

Imaging and sensing for unmanned aircraft systems, volume 2 Table 5.1 Examples of flights for image capture Date

Schedule (p.m.)

Altitude (m)

LatLong

N images

17 June 2015 25 June 2015 30 July 2015 31 July 2015 31 July 2015

15:59 14:38 16:49 16:52 17:38

20 20 20 30 40

YES YES YES YES YES

173 166 162 160 166

Table 5.1 presents some examples of flights for image capture. It is intended to continue to perform flights at different times and situations to increase the number of cases that the image bank addresses. To illustrate more easily the presentation of the collected images, georeferenced mosaics were set up in conjunction with the partner institutions, presented in Section 5.5.1.

5.5 Use of the image database Section 5.5.1 shows the corresponding visible spectrum and thermal images as well as mosaics. Section 5.5.2 presents two works that use the developed image database: one consisting of an algorithm for automatic detection of landmarks, and another to develop a visual navigation system based on landmark recognition to estimate the UAV position (latitude and longitude).

5.5.1

Mosaics

The mosaics were assembled using the AERIALTOUR CAMPUS UNIVAP to help visualise the data acquired for this work. To evaluate the accuracy of the assembled mosaic, another mosaic was assembled using only the GPS data information from the UAV and then compared with the mosaic mounted with the ground support points collected with the differential correction GPS. Figure 5.9 illustrates this comparison where the black mark represents the measured point in soil with the GPS using differential correction; the green colour signifies the same coordinates in the mosaic mounted with the ground support points, and the red mark denotes results using only the GPS data from the UAV. Further details on the development of the mosaics can be obtained as in [34,35]. The addition of control points improves the accuracy of the resulting mosaic 17 times as shown in Figure 5.9, dividing the error indicated by the red mark by the error indicated by the green mark (mosaic with support points). These support points measured on the ground also guarantee an ideal point for the position estimation algorithms presented in Section 5.5.2. The mosaic assembly process starts with data from the support points and the georeferenced acquired images using the software Pix4D. Figure 5.10 presents the mosaicking results.

Image database of low-altitude UAV flights

(a)

Sample 1 altitude 30 m

(b)

101

Sample 2 altitude 40 m

Figure 5.7 Flights at different altitudes

(a)

Flight over target

(b)

Target on the ground

Figure 5.8 Flights over standard targets

5,2 cm 20

90,4 cm

Figure 5.9 The error between the coordinates in ground collected by GPS with differential correction (black mark), mosaic mounted with support points in soil (green mark), and mosaic mounted only with GPS data from UAV (red mark)

102

Imaging and sensing for unmanned aircraft systems, volume 2

Figure 5.10 Visible spectrum mosaic. Image from [35]

401250E

N

7432750

N

7432700

401175E

10

Temperature 45 °C

Figure 5.11 Thermal spectrum mosaic Due to the flight limitations of the aircraft, the area of the thermal mosaic was reduced. The software packages used to mount the mosaic were the Pix4D and the Qgis. The results appear in Figure 5.11.

Image database of low-altitude UAV flights

103

With the mosaics shown in Figures 5.10 and 5.11, studies can be done to analyse the region via multi-spectral imaging, which facilitates, for example, the identification of roofs that absorb heat because their contours are more clearly defined in the thermal image.

5.5.2 Development of CV algorithms Several works have already used the proposed aerial image database to test and validate their experiments. [39,40] used AERIALTOUR CAMPUS UNIVAP for the development of an automatic landmark selection algorithm and visual navigation system, respectively. [39] suggested selecting the best landmarks automatically on the flight route. Figure 5.12 displays the result for part of the route where the markings represent targets automatically selected by the developed software. A second work that had been developed and used the image database was [41] work. The main goal of this work was to develop a visual navigation system based on landmark recognition to estimate the UAV position (latitude and longitude). For that, the same image data set of images were used in two different aspects of the work. The first part was to test the algorithm using flights at different times of the day. Figure 5.13 shows an example of the images used for the test. The data consisted of two sequences of photos taken at the same frequency of three photos per second, with images 4,000  3,000 pixel resolution. Each image sequence was obtained in flights at different times flight deployed on the same route. The flight corresponding to the first image sequence occurred on 31 July 2015 at 16:30, and the average flight height was 30 m. The flight for acquiring the second sequence of images was performed on 13 August 2015 at 17:40, which resulted in darker images, because of the sunset in winter. The flight average height was 40 m [40]. The pattern images were as in Figure 5.14. The data set was enough to show promising results for the landmark recognition algorithm developed even in different light conditions besides showing the necessary improvements. One of the conclusions was the need for implementing a new brightness enhancement algorithm that would not degrade the images. Figure 5.15 explains why the resulting brightness enhancement algorithm was not enough.

(a)

(b)

Figure 5.12 Landmark selection for the aerial mosaic using ORB. Image from [40]

104

Imaging and sensing for unmanned aircraft systems, volume 2

(a)

(b)

Figure 5.13 Examples of images from the AERIALTOUR CAMPUS UNIVAP: (a) an image from the 31 July 2015 flight and (b) an image from the 13 August 2015 flight

(a)

(b)

(c)

Figure 5.14 Landmarks chosen from the pattern image flight (31 July 2015): (a) a roundabout landmark, (b) the roof of the campus engineering building, and (c) the roof of a house Figure 5.14 shows resulting brightness patterns obtained with an algorithm containing problems due to shadow situations (Figure 5.14(a)), the landmarks in the structure (Figure 5.14(b)), and when the natural sunlight reaches only one side of the roof (Figure 5.14(c)). The second aspect was the auto-location tests of the algorithm. Since the data set has the UAV flight data synchronised with the images, it was possible to estimate the UAV position and compare it with the GPS/INS position information for navigation. The data (both test images and pattern images) used were from the flight done on 31 July 2015, and the same three landmarks were chosen. Figure 5.16 shows the results obtained.

Image database of low-altitude UAV flights

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

105

Figure 5.15 Candidate analysis for landmark recognition in flights at different times of the day

(a)

(b)

(c)

Figure 5.16 Auto-localisation position comparison between GPS, Landmark Recognition Estimation, related to the real UAV position in the frames. The red dot is the position estimated by the Landmark Recognition System; the green dot is the GPS estimation; and the yellow dot is the real position. Recognition situations: (a) the roundabout landmark, (b) the recognition of the roof of the engineering building, and (c) the recognition of the roof of a house

106

Imaging and sensing for unmanned aircraft systems, volume 2

5.6 Conclusion and future works In this work, images were collected that aided the development of CV algorithms. This database addresses real conditions of use, consider different conditions of natural lighting, and different landmarks. The image database must be expanded to cover as many flight conditions as possible. The implemented image database is already in use by the partner institutions. The acquired images, along with some preliminary algorithms results, have already been tested in the areas of autonomous navigation of aircraft by image, identification and automatic selection of landmarks, photogrammetry, thermal and multispectral aerial analysis. Some of these works were discussed in Section 5.5.2 [1]. The AERIALTOUR CAMPUS UNIVAP permits the development of further research. This image database can be expanded to include a higher number of real cases, and these extensions can increase flight areas, help to perform night flights, use other sensors to capture images, and incorporate different flight altitudes [2,21,22,42–45]. Currently, this database is available to the public, and some partner institutions servers host it. For access to the data collected so far, please contact one of the authors. Super-resolution (SR) reconstruction can help to overcome sensing, hardware, software, and social constraints when it comes to obtaining high-resolution images [46–48]. In particular, a sequence of outcomes can be enriched, combining novel imageries with existing knowledge on a site undergoing scrutiny if the restrictions offer far less suitable data as the amplification factor grows. It is well-known that the usage of a smoothness prior may relief somewhat the tasks. However, for large enough enlargement factors, any smoothness prior exaggerates the results’ smoothing. Consequently, algorithms that can learn priors depicting recognition for particular classes of environments and scenes will probably give far better SR outcomes. Object tracking has been a significant and vigorous research CV area. A large number of tracking algorithms have been suggested recently with some success. Still, the collections of video sequences used for assessment are often not enough or is occasionally biased for certain kinds of algorithms. Many photogrammetric and RS data sets do not have official ground-truth object positions or extents, and this makes valuations among the described quantitative results problematic. Furthermore, the initial settings and parameters of the appraised tracking procedures are not equal, and as a consequence, the quantitative effects can be incomparable or sometimes contradictory. To talk about these issues, extensive evaluation of the contemporary online object-tracking procedures with several evaluation criteria to comprehend how these approaches go within the same framework need to be carried out. It is paramount first to bring together the video sequence aspects for the performance examination. Next, most of the openly available trackers can be combined into one code library with uniform input and output formats to simplify large-scale performance evaluation. Third, extensive performance assessment of algorithms on videos with different initialisation surroundings has to be done. By analysing the quantitative fallouts, one can detect

Image database of low-altitude UAV flights

107

effective methodologies for robust tracking and arrange for potential improvements in prospect research directions. Managers and privacy supporters increasingly demand that confidentiality be shielded through the technical design of products and services, as well as through organisational procedures and policies [49–55]. Privacy research produces insights and methods that empower a new professional in the technology sector—the privacy engineer. Despite great enthusiasm for this approach, there has been little effort to explore if and how this new direction in privacy protection is influencing the design of products. Understanding how design is being used to protect privacy requires analysis of socio-technical systems, not decontextualised technical artefacts. Privacy concerns in public policy debates about UAVs are raised and addressed in concept videos developed by Amazon which depict fictional scenarios involving its future automated drone package delivery service. Unnecessary to say that understanding the social and geographical aspects of a given region can help to monitor and assisting it without causing too deep interventions in the modus vivendi and operandi of the locale. Drawing on the project and communications techniques, one may eventually discover that the concept videos disclose bigger attention to privacy concerns. Researchers need more evidence that privacy concerns can influence product and service designs. Illustrations and simulations about the services will help to shape consumer expectations about how it addresses privacy concerns. While the videos may not characterise a present product, the shifting role such concept videos may reflect on the whole chain related to UAV delivery as well as other services. As representations of product functionality aiming consumers appear, concept videos, like other public announcements, if distorted, could serve as a foundation for statements claims. Last of all, concept videos can be valuable instruments for attracting regulators to reflect on research and other stakeholders in contextually specific deliberations of when and how to conduct product and UAVCPS design to safeguard privacy.

Acknowledgements This work was supported by the Instituto Tecnologico de Aeronautica (ITA), Instituto de Estudos Avancados (IEAv), Instituto Nacional de Pesquisas Espaciais (INPE), Universidade do Vale do Paraıba (UNIVAP), and Faculdade de Tecnologia (FATEC).

References [1] Zeng Y, and Zhang R. Energy-Efficient UAV Communication with Trajectory Optimization. IEEE Transactions on Wireless Communications, 2017;16:3747–3760. [2] Aroma RJ, and Raimond K. A Novel Two-Tier Paradigm for Labeling Water Bodies in Supervised Satellite Image Classification. In: Proc. 2017 International Conference on Signal Processing and Communication (ICSPC), Coimbatore, India, 2017, pp. 384–388.

108

Imaging and sensing for unmanned aircraft systems, volume 2

[3] Li H, Tao C, Wu Z, et al. RSI-CB: A Large Scale Remote Sensing Image Classification Benchmark via Crowdsource Data. 2017. Available from: http://arxiv.org/abs/1705.10450. [4] Yao B, Yao B, and Yang X. Introduction to a Large-Scale General Purpose Ground Truth: Methodology, Annotation Tool and Benchmarks. In: Yuille A.L., Zhu SC., Cremers D., and Wang Y. (eds) Proc. Energy Minimization Methods in Computer Vision and Pattern Recognition. EMMCVPR 2007. Lecture Notes in Computer Science, vol 4679, 2007. Springer, Berlin, Heidelberg. Available from: http://www.springerlink.com/index10.1007/ 978-3-540-74198-5. [5] Li S, and Yeung DY. Visual Object Tracking for Unmanned Aerial Vehicles: A Benchmark and New Motion Models. In: AAAI; 2017, pp. 4140– 4146. [6] Manjunath BS. Aerial Photographs. Image (Rochester, NY), 1998;49(7): 633–648. [7] Han J, and Lo C. Adaptive Time-Variant Adjustment for the Positioning Errors of a Mobile Mapping Platform in GNSS-Hostile Areas. Survey Review, 2017;49(352):9–14. Available from: http://dx.doi.org/10.1080/ 00396265. 2015.1104091. [8] Toth C, Ozguner U, and Brzezinska D. Moving Toward Real-Time Mobile Mapping: Autonomous Vehicle Navigation. Proc. The 5th International Symposium on Mobile Mapping Technology, Padua, Italy, 2007, p. 6. [9] Ray EL. 2015. Available from: https://www.faa.gov/documentlibrary/media/ order/atc.pdf. [10] Conte G, and Doherty P. A Visual Navigation System for UAS Based on GeoReferenced Imagery. ISPRS – International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2011;XXXVIII1/ (September):1–6. Available from: http://www.isprs.org/proceedings/ XXXVIII/1-C22/papers/conte.pdf. [11] Kim J, and Sukkarieh S. Real-Time Implementation of Airborne InertialSLAM. Robotics and Autonomous Systems, 2007;55(1):62–71. [12] Karlsson R, Schon TB, Tornqvist D, et al. Utilizing Model Structure for Efficient Simultaneous Localization and Mapping for a UAV Application. IEEE Aerospace Conference Proceedings, 2008, pp. 1–10. [13] Silva Filho P, Rodrigues M, Saotome O, et al. In: Bebis G, Boyle R, Parvin B, et al., editors. Fuzzy-Based Automatic Landmark Recognition in Aerial Images Using ORB for Aerial Auto-localization. Springer International Publishing, Cham, 2014, pp. 467–476. Available from: http://dx.doi.org/ 10.1007/978-3-319-14249-444. [14] Janschek K, and Dyblenko S. Satellite Autonomous Navigation Based on Image Motion Analysis. In: Proc. 15th IFAC Symposium on Automatic Control in Aerospace, Bologna Italy, 2001. [15] Silva Filho PF, Thomaz LA, Carvalho G, et al. An Annotated Video Database for Abandoned-Object Detection in a Cluttered Environment. Sao Paulo, Brazil, 2014 International Telecommunications Symposium (ITS), 2014.

Image database of low-altitude UAV flights

109

[16] Choi H, Brisbane T, Geeves M, Alsalam B, and Gonzalez F. Open Source Computer-Vision Based Guidance System for UAVs On-Board Decision Making. In Proc. 2016 IEEE Aerospace Conference, Big Sky, MT, USA, 2016, pp. 1–5. [17] Liu YC, and Dai QH. Vision Aided Unmanned Aerial Vehicle Autonomy: An Overview. Image and Signal Processing (CISP), 2010 3rd International Congress on 2010;1:417–421. [18] Saripalli S, Montgomery JF, and Sukhatme G. Vision-Based Autonomous Landing of an Unmanned Aerial Vehicle. Robotics and Automation, 2002 Proceedings ICRA ’02 IEEE International Conference on, 2002;3(May): 2799–2804. [19] Yang K, and Sukkarieh S. Real-Time Continuous Curvature Path Planning of UAVS in Cluttered Environments. In: Proc. ISMA 2008. 5th International Symposium on Mechatronics and Its Applications, Amman, Jordan; 2008, pp. 1–6. [20] Fu C, Carrio A, Olivares-Mendez MA, and Campoy P. Online LearningBased Robust Visual Tracking for Autonomous Landing of Unmanned Aerial Vehicles. In: Proc. 2014 International Conference on Unmanned Aircraft Systems (ICUAS), Orlando, FL, USA,; 2014, pp. 649–655. [21] Razmjooy N, Ramezani M, and Estrela VV. A Solution for Dubins Path Problem with Uncertainties Using World Cup Optimization and Chebyshev Polynomials. In: Iano Y, Arthur R, Saotome O, Estrela VV, and Loschi H, editors, Proc. BTSym 2018. Smart Innovation, Systems and Technologies, vol. 140. Springer, Cham, Campinas, SP, Brazil, 2019. doi: 10.1007/978-3-030-16053-1_5. [22] Razmjooy N, Ramezani M, Estrela VV, Loschi HJ, and do Nascimento DA. Stability Analysis of the Interval Systems Based on Linear Matrix Inequalities. In: Iano Y, Arthur R, Saotome O, Estrela VV, and Loschi H, editors, Proc. BTSym 2018. Smart Innovation, Systems and Technologies, vol 140. Springer, Cham, Campinas, SP, Brazil, 2019. doi: 10.1007/978-3030-16053-1_36. [23] Conte G, and Doherty P. An Integrated UAV Navigation System Based on Aerial Image Matching. IEEE Aerospace Conference Proceedings, Big Sky, MT, USA, 2008, pp. 1–10. [24] Sales D, Shinzato P, Pessin G, et al. Vision-Based Autonomous Navigation System Using ANN and FSM Control. In: Proceedings – 2010 Latin American Robotics Symposium and Intelligent Robotics Meeting, LARS 2010. 2010, pp. 85–90. [25] Fisher R. CVonline: The Evolving, Distributed, Non-Proprietary, On-Line Compendium of Computer Vision; 2016. Available from: http://homepages. inf.ed.ac.uk/rbf/CVonline/CVentry.htm. [26] Price K. VisionBib.Com Computer Vision Information Pages; 2015. Available from: http://www.visionbib.com/. [27] U S Department of the Interior. U.S. Geological Survey; 2017. Available from: https://www.usgs.gov/. [28] University of Arkansas. Geospatial Modeling & Visualization, 2017. Available from: http://gmv.cast.uark.edu/home/about-the-gmv/.

110 [29]

[30] [31] [32] [33]

[34] [35]

[36] [37] [38] [39]

[40]

[41]

[42]

[43]

[44]

Imaging and sensing for unmanned aircraft systems, volume 2 Penatti AB, Nogueira K, and Santos JA. Do Deep Features Generalize from Everyday Objects to Remote Sensing and Aerial Scenes Domains? In Proc. 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA, 2015;44–51. DroneImages. Fotos Aereas Drone Images; 2013. Available from: http:// www.droneimages.com.br/fotos-aereas-drone-images. DRONEFILMAGEMAEREA. FOTOGALLERY; 2014. Available from: https://www.dronefilmagemaerea.com/foto-gallery/. AltasIMAGENS. Imagens do Alto; 2015. Available from: http://www.altasimagens.com.br/imagens/. Almeida RCFd. Avaliacao de Sistemas Eletro-Opticos Imageadores para Missoes de Inteligencia de Imagens na Faixa do Visıvel. Instituto Tecnologico de Aeronautica (ITA). Sao Jose dos Campos-SP; 2016. Oliveira LT. Avaliacao do uso de sensor termal a bordo de VANT atraves de analises radiometricas, espectrais, espaciais e posicionais [Master]; 2017. Nogueira FdC, Roberto L, Korting TS, et al. Accuracy Analysis of Orthomosaic and DSM Produced from Sensors aboard UAV. Santos, SP: INPE; 2017, p. 8. Optris GmbH. Pi 450 g7; 2017. Optris GmbH. PI LightWeight Radiometric Aerial Thermography Optris R PI LightWeight; 2017. Optris GmbH. Operator’s Manual Infrared camera. 2017, pp. 1–79. Melo AdS, Silva Filho P, and Shiguemori EH. Automatic Landmark Selection for UAV Autonomous Navigation. In: Cappabianco FAM, Faria FA, Almeida J, et al., editors, Electronic Proceedings of the 29th Conference on Graphics, Patterns and Images (SIBGRAPI’16). Sao Jose dos Campos, SP, Brazil; 2016. Available from: http://gibis.unifesp.br/sibgrapi16. Silva Filho PF, Melo A, and Shiguemori E. Automatic Landmark Selection for UAV Autonomous Navigation. In: de Campos Velho HF, editor. Proceedings of the 4th Conference of Computational Interdisciplinary Science (CCIS 2016). Sao Jose dos Campos-SP; 2016. Silva Filho PF. Automatic Landmark Recognition in Aerial Images for the Autonomous Navigation System of Unmanned Aerial Vehicles. Instituto Tecnologico de Aeronautica (ITA). D.Sc. dissertation, Sao Jose dos Campos-SP; 2016. Aroma RJ, and Raimond K. A Review on Availability of Remote Sensing Data. In: Proc. 2015 IEEE Technological Innovation in ICT for Agriculture and Rural Development (TIAR), Chennai, India, 2015, pp. 150–155. d’Andrimont R, Marlier C, and Defourny P. Hyperspatial and Multi-Source Water Body Mapping: A Framework to Handle Heterogeneities from Observations and Targets over Large Areas. Remote Sensing, 2017;9:211. Hemanth DJ, and Estrela VV. Deep Learning for Image Processing Applications. Adv. Par. Comp. IOS Press. ISBN978-1-61499-821-1 (print) 978-1-61499-822-8 (online) 2017.

Image database of low-altitude UAV flights

111

[45] Wu Y, Lim J, and Yang M. Object Tracking Benchmark. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015;37:1834–1848. [46] Jesus de MA, Estrela VV, Saotome O, and Stutz D. Super-Resolution via Particle Swarm Optimization Variants. In: Hemanth J. and Balas V. (Eds), Biologically Rationalized Computing Techniques for Image Processing Applications, Springer, Zurich, Switzerland, pp. 317–337, 2018. [47] Deshpande A, and Patavardhan P. Super Resolution and Recognition of Long Range Captured Multi-Frame Iris Images. IET Biometrics, 2017;6:360–368. [48] Deshpande A, and Patavardhan P. Gaussian Process Regression based Iris Polar Image Super Resolution. In: 2016 2nd International Conference on Applied and Theoretical Computing and Communication Technology (iCATccT), Bangalore, India, 2016, pp. 692–696. [49] Wong RY, and Mulligan DK. These aren’t the Autonomous Drones You’re Looking for: Investigating Privacy Concerns Through Concept Videos, Journal of Human-Robot Interaction, 2016;5(3):26–54, DOI 10.5898/ JHRI.5.3.Wong. [50] Estrela VV, Rivera LA, Beggio PC, and Lopes RT. Regularized PelRecursive Motion Estimation Using Generalized Cross-Validation and Spatial Adaptation. In Proc. SIBGRAPI, Sa˜o Carlos, SP, Brazil, 2003. [51] Estrela VV, Rivera LA, and Bassani MH. Pel-Recursive Motion Estimation Using the Expectation-Maximization Technique and Spatial Adaptation. In Proc. 12th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision, WSCG 2004; Pilzen, Czech Republic, 2004, pp. 47–54. [52] Hickey S. Living at Extremes: A Spatio-Temporal Approach to Understand Environmental Drivers for Mangrove Ecosystems, and Their Carbon Potential, D.Sc. thesis, The University of Western Australia, Australia, 2018. [53] Cissell JR, Delgado AM, Sweetman BM, and Steinberg MK. Monitoring Mangrove Forest Dynamics in Campeche, Mexico, Using Landsat Satellite Data, Remote Sensing Applications: Society and Environment, Volume 9, 2018, pp. 60–68. [54] Mueller M, Smith N, and Ghanem B. A Benchmark and Simulator for UAV Tracking. In: Leibe B, Matas J, Sebe N, and Welling M. (Eds), Proc. Computer Vision – ECCV 2016. Lecture Notes in Computer Science, 9905. Amsterdam, Netherlands, Springer, Cham, 2016 [55] Martell A, Lauterbach HA, Schilling K, and Nu¨chter A. Benchmarking Structure from Motion Algorithms of Urban Environments with Applications to Reconnaissance in Search and Rescue Scenarios. 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Philadelphia, PA, USA, 2018, pp. 1–7.

This page intentionally left blank

Chapter 6

Communications requirements, video streaming, communications links and networked UAVs Hermes J. Loschi1,2, Vania V. Estrela3, D. Jude Hemanth4, Sandro R. Fernandes5, Yuzo Iano2, Asif Ali Laghari6, Asiya Khan7, Hui He6 and Robert Sroufe8

Unmanned Aerial Vehicles (UAVs) within Cyber-Physical Systems (CPSs) depend on Flying Ad-Hoc Networks (FANETs) as well as on Computer Vision (CV). The Flying Nodes (FNs) play a paramount role in UAV CPSs because relying on imagery poses austere and diverse restrictions in UAV communications. Nowadays, UAV technology is switching from a single UAV to coordinated UAVs swarms that undertake advanced objectives. This scenario calls for innovative networking paradigms to handle two or more FNs that exchange data (i) straightly (without intermediates within their communication range) or (ii) indirectly via relay nodes like UAVs. Designing a UAVs’ ad-hoc network is intricate because FANET’s prerequisites differ from Mobile Ad-hoc Networks (MANETs) as well as Vehicular Ad-hoc Networks (VANETs). FANETs have particular specificities about FN mobility, FN connectivity, data routing, cloud interaction, Quality of Service (QoS), type of application, and Quality of Experience (QoE), among other issues. This chapter goes through the UAVs’ challenges when functioning as ad-hoc nodes and expounds UAVs’ network models. It also typifies FANETs’ emergent prospects, impact, and how they fit in multimedia applications.

1

Institute of Electrical Engineering, University of Zielona Go´ra, Zielona Go´ra, Poland School of Electrical and Computer Engineering (FEEC), University of Campinas – UNICAMP, Campinas – SP, Brazil 3 Telecommunications Department, Federal Fluminense University (UFF), Rio de Janeiro, Brazil 4 Electrical and Computer Engineering Department, Karunya University, Coimbatore, India 5 Sandro R. Fernandes Instituto Federal de Educacao, Ciencia e Tecnologia do Sudeste de Minas Gerais, Juiz de Fora, MG, Brazil 6 School of Computer Science & Technology, Harbin Institute of Technology, Harbin, China 7 School of Engineering, University of Plymouth, Plymouth, UK 8 John F. Donahue School of Business, Duquesne University, Pittsburgh, USA 2

114

Imaging and sensing for unmanned aircraft systems, volume 2

6.1 Introduction Currently, it is paramount to use UAV swarms to increase the communication range and information acquisition ability of Flying Nodes (FNs) in a Cyber-Physical System (CPS) [1]. Building a reliable besides robust network of teams and ground stations (GSs) become graver when communication infrastructure is lacking (as happens with hard to reach territories or regions undergoing natural or human-made disasters). The primary Flying Ad-Hoc Networks (FANETs) Network Layer (NL) task is to serve as an intermediary while performing the customary routing on its level with UAV swarms working as a Mobile Ad-Hoc Network (MANETs). FANETs’ routing protocols depend largely on the kind of communication that is twofold: (i) UAV-to-UAV (U2U) and (ii) UAV-to-Infrastructure (U2I). In U2U communication, UAVs communicate to fulfil the prerequisites of different applications, such as cooperative path planning and target tracking. UAVs can communicate either straightly with each other or via a multi-hop path over other UAVs. These interconnected UAVs can deal with short or long-range, where the range selection is contingent on the necessary data transfer rate. U2I connections encompass data exchanges between UAVs and fixed infrastructures, for example, satellites, and GSs, to deliver services through networks. Furthermore, broadcasting over UAVs and U2I has caveats. Communication installation and preservation of the networked infrastructure between UAVs and GSs emerge as significant concerns to unravel, principally in the presence of heterogeneous FNs with different payload capacities, sensors, imaging devices, avionics, communication ranges, and flight endurance (Figure 6.1).

6.2 Flying Ad-hoc Networks Recently, multi-UAVs following an ad-hoc style demanded more from the networking layer since the FNs of a UAV-CPSs are highly mobile. Accordingly, this layer is mainly a VANET subclass placed in a sub-layer of an Aerial NL. By an NL,

OGS PGS

Network

Figure 6.1 UAVs working as network gateways. OGS, overloaded GS; PGS, problematic GS

Communications requirements, video streaming, communications links

115

one means the part principally in charge of the routing, which picks the best path to connect the sender and receiver nodes through different routers/nodes. The FANET mainly acts as an intermediary between these layers while undertaking the customary routing on its level [2,3]. A FANET brings in the following advantages [2]: ●

●

●

●

●

Decreases the mission conclusion time: Missions such as reconnaissance, surveillance, crowd protection, search, and rescue can be carried out faster. Decreases total maintenance cost: Several small UAVs cost less and are easier to purchase and maintain than a big, costly UAV. Increases scalability: It augments the operation theatre area by inserting new FNs effortlessly. The UAV-CPS dynamically restructures FNs’ routing tables through an allowance for newly added UAVs. Increases survivability: Multi-UAV CPSs are more resilient to hardware/ sensors faults. When sensors/actuators fail, or an FN misses control, the mission can go on with the remaining UAVs. Lowers detectability (short radar cross-section): Mini-UAVs have low radar cross-sections, small infrared signatures, as well as low acoustic signatures due to their sizes and composite structures. Hence, radars (particularly when compared with aeroplanes and large UAVs) cannot easily detect them.

Multi-UAV CPSs have significant advantages, but the communication of two far away FNs remains a burdensome subject bearing in mind their dynamic network topology. Collected writing works have suggested some FANETs’ routing protocols relying on the FNs’ communication nature. In a FANET, there are predominantly two different kinds of communication: U2U and U2I communications [2–4]. In U2U communication, each UAV exchanges information with others to meet the needs of different application areas, for example, cooperative path planning and object tracking. Either two UAVs can directly converse with each other, or a multihop path can be created over other UAVs. Drones can have a short- or long-range communication between them. The range choice also depends on the expected data transfer rate [2–4]. In U2I communication, UAVs converse with fixed infrastructure, for example, nearby GSs, satellites, or land vehicles to deliver services for other operators in the global networks. Communication between UAVs and U2I is also a problematic issue (Figure 6.2). Other classes of antennas besides sensors can be employed to boost the data transfer rate together with the UAV-CPS performance. Communication links can be effectively established in FANETs [2–4] with GPS receivers and directed antennas (similar to phased-array antennas).

6.3 The FANET protocol The FANET’s link quality fluctuates because of the high FN mobility and the continuously changing distance between the FNs. FANET MAC designs face new tests due to link quality fluctuations and failures. Latency can also complicate

116

Imaging and sensing for unmanned aircraft systems, volume 2 UAV to UAV (U2U) 5 GHz band

UAV

UAV to Ground (U2U) 2.4 GHz band

End Devices Ground

Figure 6.2 Communication between UAVs and U2I [5] matters. A directional antenna can help to increase the communication range in different scenarios, spatial reuse, and enhancing security [6]. The investigation in [7] suggested an adaptive MAC protocol with (a) an omnidirectional antenna for packets’ transfer control and (b) a directional antenna for data packets. End-to-end delay, throughput, and Bit Error Rate (BER) were improved employing this approach. A token-based approach appears in [8] to update target data, to overcome glitches in ordinary contention-based protocols in addition to handle link failures due to excessive mobility. Full-duplex radios with Multi-Packet Reception (MPR) can boost the MAC performance in multi-UAV CPS networks. The full-duplex systems diminish delays because each node can send and receive simultaneously, and Multi-Packet Reception capabilities intensify the throughput in multi-UAV CPSs. The modified Optimized Link State Routing (OLSR) protocol from [9] relies on a directional antenna. In OLSR, the critical step is selecting a Multipoint Relay (MPR). Lessening the quantity of MPRs will result in fewer control packets transferred. In [9], the destination information is employed to transfer the packets, and if the destination distance from the source is less than half of the directional antenna maximum, then DOLSR is used. Otherwise, OLSR accomplishes the routing. This work also proposed decreasing the number of MPR that reduces control overhead, which lessens the delay and enhances the overall throughput. A time-slotted reservation arrangement is used along with AODV to decrease collisions. The research effort in [10] suggests a hybrid methodology to minimise intermediate node communication. This time reservation tactic is similar to the Slotted ALOHA where each node receives a time slot to transmit information to a master node or cluster head with communication privileges over other nodes in this specific time slot. This approach contributes to reducing collisions and improving the packet delivery ratio. A geographic-based routing protocol such as the one suggested in [11] can discover the next best available hop to successfully cut the influence of intermittent connectivity instigated by the extreme dynamic mobility of the UAV-CPS. Initially, a Gauss–Markov mobility model facilitated FN position prediction to drop

Communications requirements, video streaming, communications links

117

routing failure. Furthermore, the mobility relationship helped to select the next hop for routing more accurately. This scheme enhances the stability of the cluster as well as cluster heads. Current clustering algorithms for UAV networking were inappropriate because of high mobility in addition to frequent cluster updates. The work in [12] unravels such problems via a new mobility prediction relying on a cluster weighted model using UAV attributes. It estimates the network topology using a tree as a data structure adequate to dictionary prediction and with support for mobility models for regulating the link expiration time. It supports more stable cluster structures and improved network performance because of the cluster head-electing procedure and on-demand cluster maintenance scheme. FANET’s distinguishing features enforce distinctive design concerns as described as follows. 1.

2.

Adaptability [4]: (a) Several FANET parameters can vary during the multi-UAV CPS operation. FNs are highly mobile and always modify their location. The UAVs’ routes may differ, and the inter-drone distances cannot be constant because of the operating requirements. (b) FNs may fail during the mission because of a technical mishap or an attack against the multi-UAV system. While UAV failures lessen the number of UAVs, alternative strategies may be required to maintain the multi-UAV CPS operational while changing the FANET parameters. (c) Meteorological conditions can also disturb the FANET because if the weather changes suddenly, FANET data links may not persist. The FANET should be designed to continue to manoeuvre in a highly dynamic atmosphere. (d) The assignments may be updated during the multi-UAV CPS operation. New mission information may result in a flight plan update. For instance, during a multi-UAV CPS search and rescue operation, changes may happen after the arrival of a new intelligence report so that the mission concentrates on a specific area, and thus affecting the flight plan along with FANET parameters. (e) FANET design should adjust itself against any vicissitudes or failures. The FANET physical layer should adjust itself to the node density, the internode distances, and environmental changes. It can scan the parameters and select the best physical layer alternative. The highly dynamic nature characteristic of a FANET environment also impacts NL protocols. Ad-hoc network route maintenance is closely related to the topology changes. Consequently, the UAV-CPS performance is contingent on the adaptability of the routing and link changes protocols. The transport layer should also comply with the status of FANET. Scalability [4,13]: (a) Collaborative UAVs’ work can augment the entire UAV-CPS performance in contrast to a single-UAV CPS, which is the primary motivation to employ multi-UAV CPSs. In many applications, performance grows

118

3.

4.

Imaging and sensing for unmanned aircraft systems, volume 2 with the total number of UAVs. For example, more UAVs can accomplish a search and rescue operation faster. FANET protocols and algorithms must allow any number of FNs functioning together with the least possible performance degradation. (b) The surveillance server works directly with the operator to set a mission and offer the necessary information about it. The human user coordinates the mission by defining the surveillance region and the quantity of FNs to be deployed. When the operator inputs the investigated region and how many FNs should be deployed, the server estimates the individual FN placement using an FN placement algorithm. The server accomplishes helpful but critical tasks for the UAV swarm. First, the short flight time limitation can be overcome if a server controls the UAVs’ rotations. When an FN has a low battery, then the server reads its power status and considers the FN faulty. All faulty drones are removed from the fleet of UAVs and exchanged by another module in case any other is accessible. Otherwise, the placement algorithm relocates FNs in a high-priority first fashion. Placement algorithms run on the GS because a GS has a much higher computational ability owing to the lack of weight and size limitations. Latency [14,15]: (a) Latency is a vital design concern for FANETs and latency requirements depend entirely on the application scenario. Particularly for real-time FANET applications, such as monitoring, the information packets must be provided within a specific delay bound. Collision avoidance multi-UAV CPSs also need low latency. (b) A one-hop packet delay investigation was conducted for IEEE 802.11 relying on FANETs where each FN was modelled as an M/M/1 queue with the mean packet delay obtained analytically. A simulation analysis corroborated the rationale with outstanding numerical results relying on the simulation analysis records, where the packet delay followed a Gamma distribution. Zhai et al. [16] studied packet delay performance for IEEE 802.11 traditional wireless LANs applying exponentially distributed random variables to approximate the MAC layer packet service time. It also demonstrates that the packet delay behaviours differ for MANETs and FANETs. The MANET protocols may not fulfil the FANET’s latency requirements so that delay-sensitive UAV swarms entail new FANET protocols and algorithms. UAV platform constraints [17]: (a) UAV weight impacts FANET communication hardware performance. Lighter hardware means a lighter payload, extending the UAV endurance and it allows the installation of additional sensors on the UAV. If the total payload is constant and the communication hardware is light, then more cutting-edge sensors, actuators, and other peripherals can be carried on. (b) Space limitation is another constraint for FANET designs since it affects the communication hardware installable into the UAV platform.

Communications requirements, video streaming, communications links 5.

119

Bandwidth requirement [18]: (a) Most FANET applications aim to collect environment data and to send the acquired information to a ground base in monitoring, surveillance, or rescue operations. The visual data corresponding to the target region must be relayed between the UAV and the command control centre using a strict delay bound, which necessitates high bandwidth. Moreover, it is possible to obtain high-resolution visual information with advanced sensors, which makes the bandwidth requirements much higher. The coordination of several FNs also calls for extra bandwidth. (b) Instead, there are various bandwidth usage constraints such as the communication channel capacity, UAVs’ speeds, the error-prone structure of the wireless links, and broadcast security deficiency. A FANET protocol must fulfil the bandwidth capacity prerequisite so that it can relay a very high-resolution real-time image or video under various constraints.

6.4 FANET: streaming and surveillance Transmission of video streams in real-time poses stringent bandwidth, delay, and loss requirements to guarantee continuous video playback [19,20]. Due to the high encoding rate of the videos, continuous video playback can be guaranteed if the required throughput remains under the available network capacity. A high packet loss or large delays in the video packet reception can cause distortion or frequent pause in the video playback. We now discuss existing examples of video streaming using drones for various applications [21]. A real-time aerial traffic surveillance system using camera-mounted drones is studied in [22]. A relay-based network architecture for streaming a video from a drone is used. The drone streams video to a GS (e.g. laptop) using the analogue transmission, and the GS relays the video stream over the internet using the mobile broadband cellular network, assuming its availability nearby (Figure 6.3). The end users can then stream videos posted over the internet in near real-time. The problem, however, is the network congestion and wireless link fluctuation that does not guarantee a dedicated amount of bandwidth allowing only a low-quality video stream. Field experiments suggest that without the use of a storage server a smooth video could only be streamed at an encoding rate of 45 kbyte/s with a frame rate of 15 frames/s and a screen resolution of 160
120. While a storage server is used, this streams and also stores the video from the internet, and an increase in the encoding rate to 109 kbyte/s is affordable [21]. Video streaming using AR Drones over an 802.11 ad hoc network is studied with the motivation of monitoring an agricultural area [23] that may require enough time and manpower, otherwise. The designed control software allows estimating the communication range and video transfer rate. The tests conducted to stream a video when the distance is varied between 40 and 74 m results on average a video stream of 700 byte/s while in a two-hop scenario (up to 200 m) the average throughput is of 612 byte/s. The authors suggest that an improvement in the video

120

Imaging and sensing for unmanned aircraft systems, volume 2

Figure 6.3 UAVs working as relay nodes quality can be possible using a cross-layer solution and routing protocol with QoS support in a multi-hop scenario [21]. Wilderness Search and Rescue (WiSAR) [24] uses small-size drones having a wingspan of 42–50 in. for a visual-based aerial search. A 900 MHz transceiver is used for data communication while an analogue 2.4 GHz transmitter is used for video streaming. To search for the missing person, the signs including the point last seen and the direction of travel are considered. The video image is digitized at 640
480 resolution. It is found that the detection of unusual colours of clothing and human-made objects is possible if the operational height of the drone remains between 60 and 40 m. The field tests indicate a requirement in the improvement of the video quality since distractive jittery motions, disorienting rotations, noisy and distorted images add difficulty in the search and detection process. Computer Vision (CV) algorithms are used to enhance the stability and temporal locality of video features [21]. Disaster situations such as fire, flood, hurricane, earthquake require postdisaster assessment, planning, and response [25]. In such situations, drones can help provide safe, timely, and critical information to disaster managers for planning relief operations. To transfer imagery data 65 kbyte/s downlink is required [25], while streaming an H.264 or MPEG-2 video over an Ad hoc WLAN requires an encoding rate of about 2 Mbyte/s [26]. Although not many real-world examples for video streaming from drones is available, however, based on the existing examples, a good-quality video stream requires an encoding rate of 2 Mbyte/s. Apart from the video stream requirement, control and telemetry communication data vary from an application to others. Considering the existing technologies, it can be stated that the current unlicensed technologies may not cope with the requirements of diverse video streaming applications in situations where large communication range is required and broadband license-based technology is not available [21].

Communications requirements, video streaming, communications links

121

6.5 Discussion and future trends 6.5.1 FNs’ placement search algorithms A more imperative matter is the physical coupling of a target region and an FN. If a zone is completely decoupled with the allocated FN, then the UAV may have to fly a long distance for each frame, thus resulting in more surveillance downtime. The algorithm design supports a level of physical coupling between a target surveillance region and the drone assigned to it. Due to the FNs’ priority scores and the UAV IDs, the UAV allotted to a target zone will stay apportioned to the same object region, even if it moves. This strategy safeguards the physical coupling of a target region and the FN except when there is a swarm membership alteration or the order of priorities change. The deployment of the FNs equipped with cameras plays a critical part in surveillance schemes. An automatic placement algorithm is much more superior to manual deployment for a mobile surveillance infrastructure with too many FNs, which requires human labour. Arranging as many FNs as possible will continuously make surveillance more effective, but when the number of UAVs is limited, efficient FN placement is fundamental to attain maximum efficacy. Hence, novel algorithms can be devised to automatically deploy FNs to form a surveillance framework based on the evidence given by the administrator. The placement algorithm [27–30] requires score or metrics, and it comprises some steps as follows: (i)

(ii)

(iii)

Each area has an importance degree, and varying the region importance requires a different level of attention. The area status can be expressed by a score involving several issues such as the camera’s quality, which is proportional to the image resolution. Finding the best 3D camera placement for FNs requires excessive computations, which immensely increases with high-resolution imagery. Each place needs to be evaluated with the sum of scores (suitable metrics) of all intersecting cameras and focusing regions. Placement methods with less computational effort can be obtained through Computational Intelligence (CI) [31,32]. New camera placement entails searching for the location, area to be covered, and the UAV size. Iterative UAVs’ placement algorithms do not guarantee a globally optimal placement of FNs. A globally optimal solution needs a more wide-ranging search, which leads to impracticality. A better sub-optimal solution can result from several iteration rounds over a fixed number of FNs with some suitable metrics or via FNs’ sorting based on the performance and focusing on high-priority areas whenever possible. If the number of FNs is not sufficient, then there is no idle time for any of target area. Each FN camera is stretched to fit the size of the uncovered area after an optimisation round of placement iterations. Moreover, if many cameras are monitoring the same area, then the region is distributed to each of the cameras to maximise the scores (metrics) and the effectiveness of surveillance.

122

Imaging and sensing for unmanned aircraft systems, volume 2

6.5.2

Event detection and video quality selection algorithms

Airborne Surveillance Systems (ASSs) characteristics need improvements when the native implementation is scarce, or the performances of the commercial merchandises are not potent enough to keep all the obligatory real-time functionalities concurrently (Figure 6.4). Optimisations arise in CV and multimedia algorithms due to video coding/compression/selection, usage of FNs placement, network control, semantic processing, in addition to other procedures. As visual data are the primary origin of network traffic, one can apply a compression algorithm first to minimise the required bandwidth, then utilise a weight-based algorithm to apportion the FANET resource to the most demanding aerial video stream. The tasks such as compression and Region of Interest (ROI) selection can be carried out via an event detection algorithm or change-detection procedure based on an Optical Flow (OF) technique [33,34]. The OF use may be affected by the UAV movement. Nevertheless, the average movement vector is subtracted from the frame. The UAV’s motion does not disturb the necessary computation events. Due to the UAV-restricted memory, keeping the video with maximum quality during the operation is inefficient. Moreover, a typical ASS overwrites the oldest data without caring for their contents when the memory becomes full. Still, most of the captured imageries may contain an event, so intelligent information storing can prominently improve the memory usage relying on event detection in the scene and accumulate the data using the event metrics. Bearing in mind the limited onboard computational capacity, the event detection procedure may use OF [33,34], which is computationally cheap. Throughout the pre-processing phase, the procedure can normalise each feature vector by deducting the mean value vector from all vectors, to compensate for the FN displacement since the horizontal displacement, tilt, or span is going to force all pixels to assume the same directional vector. Vertical

Ground control station

Base station

Cloud

Object recognition Data storage and retrieval

Pre-processing

Acquisition

Context-aware hardware

Context understanding

Power, control, UAV health, etc. UAV MODULES

Figure 6.4 Simplified Airborne Surveillance Systems

Communications requirements, video streaming, communications links

123

displacements are uncommon unless the FN receives another target zone as a mission with different size. Subsequently, the module verifies the OF value to realise if any event happens. If a predefined number of OF values are greater than an established threshold, the element considers the frame as corresponding to a captured occurrence. According to the event detection outcome [35–38], the algorithm calculates some metrics to assess the visual quality index of the frames and for decision-making. When there are no detected events in recent frames, the video quality index value is gradually diminished. If an event is detected, then the quality index value is augmented in a much greater degree to fast event response.

6.5.3 Onboard video management (UAV) Onboard video management is indispensable because of the limited resources: onboard memory, computing power, and wireless network bandwidth. When there is enough space, it is fine to store a video. Still, with narrow resources, one must remove less essential videos when more space is looked-for. First, a small part of the storage needs to be cleared every time the storage reaches a percentage of its maximum capacity. If the video’s event score is greater than the average score of video candidates’ elimination or another threshold, the video is marked as significant and intact. Otherwise, the videos can be deleted. If the maximum storage is reached, then the UAV returns to a base. Since the onboard computing power is also restricted, inter-frame encoding frameworks may not be feasible. However, the UAV surveillance capacity entails real-time video streaming. Hence, one has used a frame-level compression scheme, a web-based framework with motion JPEG (MJPEG) [39–41]. Each frame can be compressed via the event detection scores. As the video acquisition can be only done with one process at a time when employing an MJPEG server, capture sub-streams are created. These sub-streams can be transmitted to the operators accessing the stream or its saved version [42,43].

6.5.4 Video-rate adaptation for the fleet platform The operators or the administrators should be able to examine the region even when an abnormal situation is not detected by the UAV-CPS [44,45]. Hence, the UAVs should be able to provide users or administrators with a video stream on request. Nevertheless, since the video byte rate is quite high, one can employ a video rate adaptation to utilise the network totally. The ASS reports the video stream data to GS whenever a video stream request service occurs. Consequently, one can get a full network topology including the video stream route in real-time. One can estimate each link capacity from a distance [27] based on the data and the positions of the UAVs.

6.5.5 FNs coordination The GS performs the coordination of a UAV swarm by sending a global position request to each FN consistently [46–48]. A request–reply technique replaces simple continuous updates to allow both the FN and the GS to identify a broken

124

Imaging and sensing for unmanned aircraft systems, volume 2

Figure 6.5 Data collection with UAVs connection. The main target of FNs’ management is to distribute FNs through convenient positions. Using the placement algorithms, the GS directs each UAV where it has to be. The FNs’ identification can be conducted with an index and an IP. The UAV index resembles the index of an array list element that changes every time the membership varies. An index approach is valuable when the job priority is highly significant, and the high priority tasks must be undisrupted. The FN’s IP remains unaffected even under FANET failure. A UAV returns to base whenever there is a broken connection while trying to join the GS. IP-based identification turns the FN back to the preceding operation area (Figure 6.5).

6.5.6

Data collection and presentation

Each FN has a web server to facilitate external access to its data. This web server handles video streaming, and image queries so that the image views can be updated frequently. The server gathers the images/videos and sends them to its web server [48–51]. A private network connects the UAVs so that the GS downloads the visual data and shows them from the local UAVs’ storage to grant access to any GS operator look at the images. The server can show the positions of the UAVs and their images/videos with the web page.

6.5.7

Software-Defined Networking

Software-Defined Networking (SDN) is a network architecture that offers a separation between control and data with centralised network programmability (Figure 6.6). SDN advances performance due to its controller where uniform SDN switches replace the original network switches [52–56]. Hence, SDN facilitates the installation and supervision of new applications and services. Moreover, the FANET management, programmability, as well as reconfiguration can be significantly simplified. FANET can employ SDN to address its environment’s shortcomings and performance issues, for example, dynamic and prompt topology

Communications requirements, video streaming, communications links

125

Real resources Application cases

Virtualised functions Network slices

SDN-enabled NFV control

Figure 6.6 Software-defined networking

modifications, links between FNs when a drone goes out of service. SDN can also facilitate the handling of network management complexity. OpenFlow is an SDN protocol, which separates the network control and data functionalities that support FANETs. The SDN control can be (i) centralised, (ii) decentralised (with the SDN controller dispersed through all FNs), or (iii) a hybrid with processing control of the sent packets performed locally on each FN and control traffic also subsists between the centralised SDN controller and all further SDN components. The controller gathers network statistics from OpenFlow switches and information on the latest FNs’ network topology. Consequently, it is imperative to maintain the SDN controller connectivity with the UAV nodes. The centralised SDN controller has a global network view. SDN platforms’ switches encompass software flow tables in addition to protocols to support communication between the FANET and the SDN control. Then, the controller defines the path and sends it to the FANET’s elements where to transmit the packets.

6.5.8 Network Function Virtualisation Network Function Virtualisation (NFV) is an original networking architecture idea that enables the virtualisation of the networking infrastructure [52–56] (Figure 6.6). More explicitly, there is an improvement in the network functions provisioning methodology via NFV, by leveraging current IT virtualisation technologies. So, virtual machines can replace network devices, hardware and basic functions. NFV can deliver programming capabilities in FANETs and lessens the network management complexity. An FN-cell management structure with UAVs acting as aerial base stations using a multi-tier FN-cell network can balance the land network utilising the NFV and SDN models. A Video Monitoring Platform as a Service (VMPaaS) can be used with a UAV swarm to form a FANET. This platform

126

Imaging and sensing for unmanned aircraft systems, volume 2

employs the recent NFV and SDN paradigms. Due to the NFV complexity of the interconnections and controls, SDN can be combined with NFV.

6.5.9

Data Gathering versus Energy Harvesting

FANETs can assist Energy Harvesting (EH) in wireless networks, due to their advantages compared with fixed land base stations [53]. An FN can be a Flying Base Station (FBS) with on-demand itinerant network services access and also arranges for wireless recharge for ground devices. An FBS can acquire data and recharge depleted ground IoT devices. UAVs with EH modules and wireless recharging features can outspread the network lifetime. Nonetheless, an FN can be a prospective energy source for IoT devices in need. Conversely, the FBS can acquire data from IoT devices provided they possess sufficient energy to broadcast their packets while the FN concurrently accesses the EH RF signals sent to/by IoT devices. The UAV station embraces a restricted coverage time due to its energy constraints. Hence, some investigation on an efficient compromise between the times for ground recharging and information gathering must happen. A control and optimisation scheme to finish the UAV trajectory within a minimum interval, while minimising the energy disbursed and/or increasing the network lifespan, must also be thought. Extensive numerical outcomes and simulations confirm how the system acts under different scenarios and with several metrics can be investigated. The IoT paradigm allows to collect, store, transmit, and process data while linking the real and virtual worlds. Furthermore, the brutal demand for higher data rates with superior QoS entails continuous development to emerge. A strategic feature is to let devices employ their energy resourcefully while boosting up their energy. The energy constraint makes effective self-organisation critical, which is necessary to allow satisfactory QoS communications. Consequently, it is indispensable to handle the way and extent IoT devices effectively consume energy, thus their battery lifetime. Prevailing designs for continuous or on-demand energy supply are minimal, requiring work, and the matter is still open. An important question is by what means to recharge IoT devices both wirelessly and remotely. Hence, FBSs can be used for EH on the downlink to terrestrial IoT devices while keeping their accessibility and basic processes. UAVs can be paramount in interconnecting IoT ecosystem devices via accurate models that capture the objects’ behaviours within the ROI. Still, UAVs have abundant new and flexible features such as high mobility, pervasive network access, and coverage for GS operators and IoT devices. Their reprogrammability throughout the runtime, the way they capture details and obtain new measurements with a plug-and-play deployment are extremely important. Furthermore, the IoT devices energy restrictions need translation into acceptable thresholds. Lessening the energy absorption and prolonging the network lifetime are very troublesome subjects that impact immensely Ultra-Dense IoT Networks having massive and spatially distributed nodes where batteries are the chief energy sources. As a consequence, regularly recharging or periodically exchanging batteries for large nodes

Communications requirements, video streaming, communications links

127

can be pricey and inconvenient. A UAV station for both wireless EH and networking can solve this caveat. Significant efforts to extend the autonomy of FNs and bring in wireless communication coverage everywhere and at any time suggest the control of energy usage throughout uplink information gathering from UAVs to ground IoT devices. The mutual optimisation of sensors’ wake-up schedule and FN trajectory to complete the mission (with reliable and energy-efficient data acquisition) can better organise the UAV-CPS energy-wise. The optimal organisation of UAV SmallCells’ (USCs) networks can be done according to a sub-modular game standpoint. The main goal is to meet a reasonable network availability with adequate encounter rate while expanding the energy efficiency. Moreover, energy efficiency and throughput can employ a large-scale antenna framework as an auspicious form to cut the carbon footprint from a base station. A smart answer for the 3D FN placement problem is to lessen the cost by minimising the number of Mobile Base Stations (MBSs) to be positioned, while each GS must stay inside at least one MBS communication range. When assuming only Line-of-Sight (LoS) can take place in the air-to-land link, then a path loss may happen to achieve the optimal dimensioning of FN base station with optimal network performance. Besides, there is the possibility to combine a low-altitude platform with the USCs’ downlink coverage performance on behalf of efficient deployment of USCs. Utilising FNs as base stations can help to maximise the coverage probability, which hinges on the UAV altitude and transmission power. The security of FNs against CPS attacks must be reinforced and balanced while delivering services over UAV-CPSs predisposed to malicious attacks. So far the models offer jointly network access and wireless energy transfer, where the EH capability is not enough to cover/solve the FN-IoT area. For instance, if the FN transfers energy to a piece of equipment via RF subject to limited energy and not directly supported by a substantial Multiple-Input Multiple-Output (MIMO) data centre, then several safety procedures will follow. When an FN is responsible for wireless power transfer, considering its mobility, one can discover the optimal path to maximise the total energy transported to ground receivers. IoT demands EH solutions for autonomous self-powered systems to produce energy from external environment energy sources, for example, solar, thermal, and wind energy. In radio-based networks, it is better to harvest energy via RF sources, owing to its easy implementation and low EH gain. Stations with EH units can transform the received RF signal into energy to revitalise out-of-battery equipment. RF EH can extend the lifetime of sensors and actuators while decreasing the battery replacement cost. A problem that also needs attention is the information asymmetry among Data Access Points (DAPs) gathering data from IoT gadgets and Energy Access Points (EAPs) that convey energy to devices. An optimal information collection performance can rely on caching with a threshold regulation procedure to cut the number of requests transmitted to a labelled sensor. Nevertheless, in some frameworks, EH in D2D networks presumed an underlay amid Machine Type Communication (MTC) and cellular networks.

128

Imaging and sensing for unmanned aircraft systems, volume 2

Likewise, the optimum time that D2D users spend with MTC traffic energy and consuming their available cellular spectra must be conciliated. Additionally, security is an exciting topic in EH applications. A learning structure for RF EH permits Hybrid Access Point (HAP) to determine adequate power consumption when in the presence of adversarial learning problems can be implemented. Without a doubt, offload computation can help to detect and to recognise distrustful individuals in a crowd via an FN-friendly IoT platform to ameliorate the UAV-CPS reactivity.

6.6 Conclusion Fluid topology with rapid links alterations characterises FANETs where each UAV is a FN with 3D mobility and rapid link quality change between nodes. Hence, different design challenges concerning the communications protocols appear, which call for innovative forms to achieve the communication prerequisites. FANET’s bottlenecks are (i) wireless communications between UAV-GS, UAV-satellite, and U2U; (ii) each UAV must have routing, coordination, cooperation, and communication protocols when frequent connection interruptions occur; (iii) allowing for high-performance fluid network topologies; and (iv) limited UAVs power resources. Consequently, a FANET entails some specialised hardware with novel networking paradigms. The UAV power constraints limit the computational capacity, communication, and endurance of UAVs. Energy-aware UAV fleet deployment with a Low power and Lossy Network (LLT) approach is an option to handle this caveat. SDN and NFV are auspicious methods to tackle sophisticated resources management, given FANETs’ distinctive features. FANETs will also interact with wireless visual sensor and actuation networks that tend to grow exponentially. Hence, protocols and strategies must be carefully established to allow this expansion [54–57].

References [1] V. V. Estrela, O. Saotome, H. J. Loschi, et al., “Emergency Response CyberPhysical Framework for Landslide Avoidance with Sustainable Electronics,” Technologies, vol. 6, p. 42, 2018. doi:10.3390/technologies6020042. [2] O. K. Sahingoz, “Networking Models in Flying Ad-Hoc Networks (FANETs): Concepts and Challenges,” J. Intell. Robot. Syst. Theory Appl., vol. 74, no. 1–2, pp. 513–527, 2014. [3] O. K. Sahingoz, “Mobile Networking with UAVs: Opportunities and Challenges,” 2013 Int. Conf. Unmanned Aircr. Syst. ICUAS 2013 – Conf. Proc., Atlanta, GA, USA, pp. 933–941, 2013. [4] I. Bekmezci, O. K. Sahingoz, and S¸. Temel, “Flying Ad-Hoc Networks (FANETs): A Survey,” Ad Hoc Networks, vol. 11, no. 3, pp. 1254–1270, 2013. [5] A. Guillen-Perez and M.-D. Cano, “Flying Ad Hoc Networks: A New Domain for Network Communications,” Sensors, vol. 18, p. 3571, 2018.

Communications requirements, video streaming, communications links

129

[6] K. Palan and P. Sharma, “FANET Communication Protocols: A Survey,” Ijcsc, vol. 7, no. 1, pp. 219–223, 2015. [7] A. I. Alshabtat and L. Dong, “Adaptive MAC Protocol for UAV Communication Networks Using Directional Antennas,” 2010 Int. Conf. Networking, Sens. Control, Chicago, IL, USA, 2010, pp. 598–603. [8] Y. Cai, F. R. R. Yu, J. Li, Y. Zhou, and L. Lamont, “MAC Performance Improvement in UAV Ad-Hoc Networks with Full-Duplex Radios and Multi-Packet Reception Capability,” Ottawa, ON, Canada, pp. 523–527, 2012. [9] A. Alshabtat, L. Dong, J. Li, and F. Yang, “Low Latency Routing Algorithm for Unmanned Aerial Vehicles Ad-Hoc Networks,” Int. J. Electr. Comput. Energ. Electron. Commun. Eng., vol. 6, no. 1, pp. 48–54, 2010. [10] S. Defense, S. Symposium, J. H. Forsmann, R. E. Hiromoto, and J. Svoboda, “A TimeSlotted On- Demand Routing Protocol for Mobile Ad Hoc Unmanned Vehicle Systems A Time-Slotted On-Demand Routing Protocol for Mobile Ad Hoc Unmanned Vehicle Systems,” 2007. [11] R. Bilal and B. M. Khan, “Analysis of Mobility Models and Routing Schemes for Flying Adhoc Networks (FANETS),” Int. J. Appl. Eng. Res., vol. 12, no. 12, pp. 3263–3269, 2017. [12] C. Zang and S. Zang, “Mobility Prediction Clustering Algorithm for UAV Networking,” 2011 IEEE GLOBECOM Work. GC Wkshps, Houston, TX, USA, 2011, no. 3, pp. 1158–1161, 2011. [13] E. Yanmaz, C. Costanzo, C. Bettstetter, and W. Elmenreich, “A Discrete Stochastic Process for Coverage Analysis of Autonomous UAV Networks,” 2010 IEEE Globecom Work. GC’10, pp. 1777–1782, 2010. [14] E. W. Frew and T. X. Brown, “Networking Issues for Small Unmanned Aircraft Systems,” J. Intell. Robot. Syst., vol. 54, no. 1–3, pp. 21–37, 2009. [15] H. Zhai, Y. Kwon, and Y. Fang, “Performance Analysis of IEEE 802.11 MAC Protocols in Wireless LANs,” Wirel. Commun. Mob. Comput., vol. 4, no. 8, pp. 917–931, 2004. [16] N. Walravens and P. Ballon, “Platform Business Models for Smart Cities: From Control and Value to Governance and Public Value,” Commun. Mag. IEEE, no. June, pp. 72–79, 2013. [17] A. Purohit, F. Mokaya, and P. Zhang, “Collaborative Indoor Sensing with the Sensorfly Aerial Sensor Network,” Proc. 11th Int. Conf. Inf. Process. Sens. Networks – IPSN ’12, p. 145, 2012. [18] M. Quaritsch, K. Kruggl, D. Wischounig-Strucl, et al., “Networked UAVs as Aerial Sensor Network for Disaster Management Applications,” Elektrotechnik und Informationstechnik, vol. 127, no. 3, pp. 56–63, 2010. [19] D. Wu, Y. T. Hou, W. Zhu, Y.-Q. Zhang, and J. M. Peha, “Streaming Video over the Internet: Approaches and Directions,” IEEE Trans. Circuits Syst. Video Technol., vol. 11, no. 3, pp. 282–300, 2001. [20] E. Setton, T. Yoo, X. Zhu, A. Goldsmith, and B. Girod, “Cross-Layer Design of Ad Hoc Networks for Real-Time Video Streaming,” IEEE Wirel. Commun., vol. 12, no. 4, pp. 59–65, 2005.

130 [21] [22]

[23]

[24]

[25]

[26]

[27] [28]

[29]

[30]

[31]

[32]

[33]

[34]

Imaging and sensing for unmanned aircraft systems, volume 2 R. Muzaffar, “Routing and Video Streaming in Drone Networks,” D.Sc. dissertation, Queen Mary University of London, London, 2017. Y. M. Chen, L. Dong, and J. S. Oh, “Real-Time Video Relay for UAV Traffic Surveillance Systems through Available Communication Networks,” 2007 IEEE Wireless Communications and Networking Conference, Kowloon, China, pp. 2608–2612, 2007. DOI: 10.1109/WCNC.2007.485. C. C. Baseca, J. R. Diaz, and J. Lloret, “Communication Ad Hoc Protocol for Intelligent Video Sensing Using AR Drones,” Proc. – IEEE 9th Int. Conf. Mob. Ad-Hoc Sens. Networks, MSN 2013, Dalian, China, pp. 449–453, 2013. J. Montgomery, S. I. Roumeliotis, A. Johnson, and L. Matthies, “The Jet Propulsion Laboratory Autonomous Helicopter Testbed: A Platform for Planetary Exploration Technology Research and Development,” J. F. Robot., vol. 23, no. 1, pp. 245–267, 2006. M. Erdelj, E. Natalizio, K. R. Chowdhury, and I. F. Akyildiz, “Help from the Sky: Leveraging UAVs for Disaster Management.” IEEE Pervasive Computing, 2017, 16, 24–32. A. Detti, P. Loreti, N. Blefari-Melazzi, and F. Fedi, “Streaming H. 264 Scalable Video over Data Distribution Service in a Wireless Environment,” Proc. IEEE WoWMoM, Montreal, QC, Canada, 2010. J. Jung, S. Yoo, W. G. La, D. R. Lee, M. Bae, and H. Kim, “AVSS: Airborne Video Surveillance System,” Sensors, 18(6): 1939, 2018. T. Ahn, J. Seok, I. Lee, and J. Han, “Reliable Flying IoT Networks for UAV Disaster Rescue Operations.” Mobile Information Systems, 2018, 2572460:12572460:12. https://doi.org/10.1155/2018/2572460. R. Fan, J. Cui, S. Jin, K. Yang, and J. An, “Optimal Node Placement and Resource Allocation for UAV Relaying Network,” IEEE Communications Letters, vol. 22, pp. 808–811, 2018. C. C. Lai, C.-T, Chen, and L.-C. Wang, “On-Demand Density-Aware UAV Base Station 3D Placement for Arbitrarily Distributed Users With Guaranteed Data Rates,” IEEE Wireless Communications Letters, vol. 8, pp. 913–916, 2019. D. J. Hemanth and V. V. Estrela, “Deep Learning for Image Processing Applications,” Adv. Par. Comp., IOS Press, Netherlands. ISBN978-1-61499821-1 (print) 978-1-61499-822-8 (online) 2017. N. Razmjooy and V. V. Estrela. “Applications of Image Processing and Soft Computing Systems in Agriculture,” IGI Global, Hershey, PA, USA, pp. 1–337, 2019. doi:10.4018/978-1-5225-8027-0. H. R. Marins and V. V. Estrela, “On the Use of Motion Vectors for 2D and 3D Error Concealment in H.264/AVC Video,” Feature Detectors and Motion Detection in Video Processing. IGI Global, Hershey, PA, USA, pp. 164–186, 2017. doi:10.4018/978-1-5225-1025-3.ch008. V. V. Estrela and A. M. Coelho. “State-of-the-Art Motion Estimation in the Context of 3D TV.” In Multimedia Networking and Coding, ed. R. A. Farrugia and C. J. Debono, IGI Global, Hershey, PA, USA, pp. 148–173, 2013. doi:10.4018/978-1-4666-2660-7.ch006.

Communications requirements, video streaming, communications links

131

[35] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly Detection: A Survey,” ACM Comput. Surv., 41: 15:1–15:58, 2009. [36] A. Farzindar and W. Khreich, “A Survey of Techniques for Event Detection in Twitter,” Computational Intelligence, vol. 31, pp. 132–164, 2015. [37] R. Schuster, R. Mo¨rzinger, W. Haas, H. Grabner, and L. Van Gool. “Real-Time Detection of Unusual Regions in Image Streams,” Proc. MM’10: Proceedings of the 18th ACM international conference on Multimedia 2010, Seoul, Korea, 1, pp. 1307–1310, 2010. [38] K. Zhang, T. Manning, S. Wu, et al. “Capturing the Signature of Severe Weather Events in Australia Using GPS Measurements,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 8, pp. 1839–1847, 2015. [39] T. Richter, A. Artusi, and T. Ebrahimi, “JPEG XT: A New Family of JPEG Backward-Compatible Standards,” IEEE MultiMedia, vol. 23, pp. 80–88, 2016. [40] J. Theytaz, L. Yuan, D. Mcnally, and T. Ebrahimi, “Towards an Animated JPEG,” Proceedings SPIE Optical Engineering þ Applications, Volume 9971, Applications of Digital Image Processing XXXIX; 99711X, San Diego, California, USA, 2016. DOI: 10.1117/12.2240283. [41] S. Mahmoud and Nader Mohamed. “Collaborative UAVs Cloud,” 2014 International Conference on Unmanned Aircraft Systems (ICUAS), Orlando, FL, USA, pp. 365–373, 2014. [42] M. Bae and K. Hwangnam, “Authentication and Delegation for Operating a Multi-Drone System,” Sensors, 19(9):2066, 2019. [43] M. Bae, S. Yoo, J. Jung, et al., “Devising Mobile Sensing and Actuation Infrastructure with Drones,” Sensors, 2018. [44] X. Wang, X. A. Chowdhery, and M. Chiang, “SkyEyes: Adaptive Video Streaming from UAVs,” Proc. 3rd Workshop on Hot Topics in Wireless HotWireless ’16, HotWireless@MobiCom, New York, NY, USA, pp. 2–6, 2016. DOI: 10.1145/2980115.2980119 [45] A. Rozantsev, M. Salzmann, and P. Fua, “Beyond Sharing Weights for Deep Domain Adaptation,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 41, pp. 801–814, 2018. [46] M. Chen, F. Dai, H. Wang, and L. Lei, “DFM: A Distributed Flocking Model for UAV Swarm Networks,” IEEE Access, vol. 6, pp. 69141–69150, 2018. [47] D. Albani, T. Manoni, D. Nardi, and V. Trianni, “Dynamic UAV Swarm Deployment for Non-Uniform Coverage,” Proc. 2018 International Conference on Autonomous Agents and Multiagent Systems, AAMAS, Stockholm, Sweden, 2018. [48] J. Zhang, Y. Zeng, and R. Zhang, “UAV-Enabled Radio Access Network: Multi-Mode Communication and Trajectory Design,” IEEE Transactions on Signal Processing, vol. 66, pp. 5269–5284, 2018. [49] J. Sun, B. Li, Y. Jiang, and C.-Y. Wen. “A Camera-Based Target Detection and Positioning UAV System for Search and Rescue (SAR) Purposes,” Sensors, 16(11), 1778, 2016.

132 [50]

[51]

[52]

[53]

[54]

[55]

[56]

[57]

Imaging and sensing for unmanned aircraft systems, volume 2 Z. Yang, R. Wang, Q. Yu, et al., “Analytical Characterization of Licklider Transmission Protocol (LTP) in Cislunar Communications,” IEEE Transactions on Aerospace and Electronic Systems, vol. 50, no. 3, pp. 2019– 2031, 2014. L. Gupta, R. Jain, and G. Vaszkun, “Survey of Important Issues in UAV Communication Networks,” IEEE Communications Surveys & Tutorials, vol. 18, no. 2, pp. 1123–1152, 2016. H. Shakhatreh, A. H. Sawalmeh, A. Al-Fuqaha, et al. “Unmanned Aerial Vehicles (UAVs): A Survey on Civil Applications and Key Research Challenges,” IEEE Access, vol. 7, pp. 48572–48634, 2019. doi: 10.1109/ACCESS.2019.2909530. S. Arabi, E. Sabir, H. Elbiaze, and M. Sadik, “Data Gathering and Energy Transfer Dilemma in UAV-Assisted Flying Access Network for IoT,” Sensors, vol. 18, pp. 1519, 2018. doi:10.3390/s18051519. V. V. Estrela, A. C. B. Monteiro, R. P. Franc¸a, Y. Iano, A. Khelassi, and N. Razmjooy, “Health 4.0 as an Application of Industry 4.0 in Healthcare Services and Management,” Medical Technologies Journal, vol. 2, no. 4, pp. 262–276, 2019. doi: 10.26415/2572-004X-vol2iss1p262-276. V. V. Estrela, A. Khelassi, A. C. B. Monteiro, et al. “Why software-defined radio (SDR) matters in healthcare?”. Medical Technologies Journal, vol. 3, no. 3, Oct. 2019, pp. 421–9, DOI:10.26415/2572-004X-vol3iss3p421–429. D. Kreutz, F. M. V. Ramos, P. Verissimo, et al. “Software-Defined networking: A comprehensive survey,” Proceedings of the IEEE, vol. 103, pp. 14–76, 2014. S. Sudevalayam, and P. Kulkarni, “Energy harvesting sensor nodes: Survey and implications.” IEEE Communications Surveys & Tutorials, 13, pp. 443–461.

Chapter 7

Multispectral vs hyperspectral imaging for unmanned aerial vehicles: current and prospective state of affairs R. Jenice Aroma1, Kumudha Raimond1, Navid Razmjooy2, Vania V. Estrela3 and Jude Hemanth1

Over the past few decades of imaging, these sensing instruments are now more advanced with multiple missions such as surveillance, monitoring, tracking and destruction of spatial objects. Nowadays, unmanned aerial vehicles (UAVs) are much prevalent, which could acquire a comprehensive view and could perform actions even to the lowest target levels at the ground. The UAV can be developed with minimal cost than other remote mission. Hence, it is much cost-effective. This chapter aims at detailing the critical aspects of two different variants of remotesensing (RS) technologies in UAVs: (a) multispectral imaging (MSI) and (b) hyperspectral imaging, which accounts for the spatial and spectral signatures of the observed underlying natural phenomena.

7.1 Introduction The rapid advancement of scientific technologies has led to the automation of huge industrial machinery, which even encompasses the automobile industry. The automated vehicles on the road is not a distant dream, a few leading companies such as Google, Uber and Tesla Motors are working towards achieving that dream to come true on every road line [1,2]. The use of RS such as satellite imaging technology has tremendous benefits in observing the scenic regions with a higher resolution since that adds more detail to both spectral and spatial aspects. On adopting these instruments for imaging in UAV, the spatial object localisation and region of interest (ROI) changes of any region using images can be improved further. 1 Electronic and Computer Engineering Department, Karunya Institute of Technology and Sciences, Coimbatore, India 2 Department of Electrical Engineering, Tafresh University, Tafresh Iran 3 Telecommunications Department, Universidade Federal Fluminense (UFF), RJ, Brazil

134

Imaging and sensing for unmanned aircraft systems, volume 2

The MSI sensors assist the analyses of the scenic regions, with the spectral resolution limited to few bands of the visual and infrared (IR) regions, where the hyperspectral images (HSIs) comprising hundreds of spectral channels can be observed even within that same region. In the case of HSI sensors, the push-broom and snapshot are two configuration modes that differ in covering the spatial region either simultaneously or one region at a time through a conveyor belt being scanned over that scenic region. The benefit of adopting these imaging instruments lies with observing the highly discriminant spectral signatures of complex spatial objects to be observed. The whole UAV system, being a cyber-physical system (CPS) (UAVCPS), may get an advantage over instantaneous image recognition applications in case of specific life rescuing operations on defence and surveillance, where these spectral-based higher levels of discriminant power can bring a high impact. This paves the way for adopting MSI and HSI sensors for UAV imaging. In the case of satellite imaging, the spatial resolution decides the better clarity of spatial object exploitation. The images captured through these sensing instruments are just the reflectance of spatial objects, which got recorded as digital numbers, which would be more complicated to view in greater heights. However, upon the growth of imaging and sensing instruments, croplands, individual tree heights and vegetation health monitoring are more accessible nowadays. In case of any natural devastations, the real-time changes and devastations of an environment can be instantly acquired and well estimated through launching the UAV excavation in that region [3,4]. Through the mode of imaging with ground-based and satellite sensors, it is highly impossible to get more specific details of information about the underlying spatial objects. Hence, the use of UAV imaging has become a massive explosion in recent aerial monitoring or land use/change detection applications. This UAV imaging has brought down the rate of manual labour through increased human comfort, right from navigation communication, traffic monitoring and many other surveillance applications [5]. The unmanned aerial platforms are turning as the emerging choice for fatal and non-fatal weapons in practice, and it led to a huge transformation in carrying out prosecute defence operations. The highest advantage of UAVs is the auto-pilot invasions in battlefields that could prevent the loss of human lives. UAVs can be used for aerial investigations and such evacuation operations. However, completely automated UAVs are still a high-end cost-consuming tool for a native user. In general, the images acquired from the sensing instruments comprehend different spectral bands of detail corresponding to their respective wavelengths, as revealed in Figure 7.1. The reflection of the spectral region can be analysed for mapping healthy and unhealthy vegetation through sensing, as shown in Figure 7.2. Typically, the green and near-infrared (NIR) reflectance are dominant in plants; that is why such band combination has been applied for deriving vegetation indices. If the NIR region has been reflected higher, the leaf is healthy, otherwise (stressed/diseased leaf) the reflection decreases, which exemplifies the spectral importance looking at different

Multispectral vs. hyperspectral imaging for UAVs

Penetrates earth atmosphere?

Y

N

Y

135

N

Wavelength (meters) 103

10–2

10–5

.5 × 10–6

10–8

10–10

10–12

Pinpoint

Protozoans

Molecules

Atoms

Atomic nuclei

About the size of...

Buildings Frequency (Hz)

104

Temperature of bodies emitting the wavelength (K)

Humans

Honey bee

108

1012

1K

100 K

1015

1016

10,000 K

1018

1020

10 Million K

Figure 7.1 Electromagnetic spectrum [6]

Cuticle

Xylem

Epidermis

Phloem Palisade mesophyll

Red Green Blue

Spongy mesophyll

NIR Oxygen Carbon dioxide

Stoma Veins

Figure 7.2 Vegetation mapping based on spectral analyses

136

Imaging and sensing for unmanned aircraft systems, volume 2

frequency bands when imaging a target scenario. This chapter details the following concepts: 1. 2. 3. 4.

5.

UAV components and working that portrays the limited onboard capability and positioning for MSI and HSI instruments [7,8]; Importance of the spectral vs spatial resolution of spatial data acquired using UAV; The prominence of satellite imaging instruments concerning MSI and HSI using UAVs; UAV (airborne acquisitions) and near distance (ground-based acquisitions) for various research applications (even human movements on the road) and similar trafficking can be monitored [9,10]; and The image dimensionality and data reduction techniques applied for MSI and HSI [11,12].

7.2 UAV imaging architecture and components In general, a UAV is piloted by an automated embedded system called the flight control system. There are a wide variety of sensors such as accelerometers, gyros, Global Positioning System (GPS), pressure sensors accommodated within the system to carry out the UAV mission in the proposed flight plan. Every UAV is composed of five modules to perform the task of aerial surveying [13,14]. Figure 7.3 illustrates the generalised architecture of UAV: (i) flight computer, (ii) mission and payload control, (iii) communication sub-system, (iv) digital camera and sensor instruments and (v) gyro-stabilised observation platform.

Communication subsystem

Flight computer Mission and payload control

Digital camera and sensor instruments

Gyro-stabilized observation platform

Figure 7.3 UAV general architecture

Multispectral vs. hyperspectral imaging for UAVs

137

The external components of a UAV are simple, lightweight and stable with limited space to house the sensing instruments and any sophisticated technological devices. (a)

The flight computer: This component is the central control unit of the UAV. It is a system which has been designed to collect the aerodynamic data through a set of sensors such as accelerometers, magnetometers, GPSs and so on, for automating the flight plane of an aircraft. (b) The mission payload controller: A mission payload is a set of data acquisition sensors such as RGB cameras, sensing instruments such as IR, thermal, MSI and HSI sensors and so forth for gathering the information that can be partially processed instantaneously or applied with post-processing. Moreover, a mission controller is responsible for controlling the sensors in the payload based on the flight plan. (c) The communication infrastructure: It is the backbone of the entire system since all the modules are held in common control with synchronisation through a mixture of communication mechanisms, which would assure a reliable link between the UAV and the base station. Even though the UAV systems are highly automated, still a moderate level of human control from the ground station is mandatory. The flight control computer can perform based on the flight plan and cannot act/respond beyond the schedule in self-handling. Similarly, the payload is mostly operated remotely with semi-automation support. Figure 7.4 details the flow of information to execute the flight campaign for surveying using the UAV. Initially, the ground station issues the command for the

Human operator Commands

Status / data

Ground station Flight control

Controller

Auto-pilot

Unmanned vehicle

Figure 7.4 UAV workflow diagram

138

Imaging and sensing for unmanned aircraft systems, volume 2

flight with a pre-flight schedule for surveying. The controller acts as a central unit for controlling and coordinating the imaging and flight. In general, the existing auto-pilot functionalities of UAV are highly advanced nowadays, in case of low battery or any such hardware failure, the safe return mode will return the UAV to the source safely, even instantaneous image processing is possible with the help of normalised difference vegetation index (NDVI) processing cameras. The standard RGB cameras capture only the visible light, whereas such modified cameras for NDVI processing capture the NIR region also.

7.2.1

Future scope for UAV

The success of UAV in multi-divergent fields has led to numerous research efforts globally, and this eye on the sky has no limit for exploitation. The following areas have wide scope [15–17]: ● ● ● ● ● ● ● ●

Complex aerial object detection UAV transition in land, air and water Automatic positioning in all transition medium Noise reduction in UAV’s flight Solid and waterproof shell to adapt under-water mobility Automated navigation using GPS for coastal search operations Intelligent self-learning capability in object recognition Increasing load stress capability for facilitating the purpose of carrying sensors and instruments for both underwater or aerial tasks.

The UAV-CPS face severe restrictions in terms of energy and payload, which can be partially alleviated with cloud computing, intelligent use of the FANET nodes and better image representations. As semantics and metadata come into play, the amount of onboard memory will be reduced [18–20].

7.3 Multispectral vs. hyperspectral imaging instruments The rate of launched remote sensors is going higher every year for various studies such as atmosphere, land cover, natural vegetation and coastal areas since the launch of the first RS satellite (TIROS-1) in 1960 [21]. These remote sensors produce various types of imagery with different accuracy levels due to different spatial resolution. It is the measure of surface area covered within a digitised pixel of an image [22]. For UAV systems, these MSI and HSI sensors could aid in achieving useful mapping of land cover and agricultural health, as displayed in Figure 7.5.

7.3.1

Multispectral imaging

The MSI sensors perceive the spectrum at different wavelengths where the altitude of the satellite’s orbit is the main factor for their spatial resolution accuracy. The MSI consists of about 3–4 wider bands, along with a single band of panchromatic data. These bands include the region which is not only into the visible spectrum (400–700 nm) that

139

Cr

op

lan

dR

GB

m Cro ul p tis la pe nd ct ra l

Multispectral vs. hyperspectral imaging for UAVs

Figure 7.5 Multispectral imaging

human eyes could sense, but it also extends further to various IR regions such as NIR, mid-infrared, thermal infrared and other ultraviolet layers [23].

7.3.1.1 Low-resolution imaging Low-resolution MSI that lies in a range between 30 and 1000 m helps in agricultural monitoring and other energy applications. Here, the spatial resolution depends on the pixel count, i.e., the more extensive the pixel count, the higher is the image clarity [24]. Low-resolution MSI is vastly used for extensive land cover analysis where image fusion or registration techniques along with super-resolution can improve the MSI imagery quality.

7.3.1.2 High-resolution imaging The high-resolution MSI that ranges from 4 m to a few centimetres above the ground level for spatial resolution is used mainly for surveillance purposes [25]. Higher resolution imagery can be used to identify and map even small patches of land cover, logistics planning, urban development, another habitation mapping, etc. [26]. The typical limitations of MSIs are the coarse resolution and the least discriminant power of spatial objects. Differentiation of complex spatial objects through the use of a simple feature set such as colour, texture and shape, among others, could bring only less accurate results [27].

140

7.3.2

Imaging and sensing for unmanned aircraft systems, volume 2

Hyperspectral imaging

The HSI sensing instruments contain thousands of spectral bands, which are much narrower (10–20 nm), i.e., each digitised pixel holds the light intensity of numerous contiguous spectral bands [28]. Thus, every digitised pixel of an image has a continuous spectrum of detail, which can characterise the spatial objects with more accurate information. The Hyperion imaging spectrometer is a notable HSI sensor that records 220 spectral bands (0.4–2.5 mm) using a 30-m spatial resolution with high radial spectrometry [29]. Figure 7.6 shows the MSI and HSI variations with respect to their spatial accuracy. The detailed observations resulting from HSI afford more evidence than the existing MSI thematic mapping instruments for land cover classification studies. The HSI is widely used in real-world applications such as mineral exploration, surveillance [31], land cover studies using AVHRR [32], plant species monitoring [33] and landslides detection [34]. Few other notable HSI instruments are PROBA, PRISMA, IMS-1 and HySI. Moreover, the significant challenges in HSI data processing are the redundant information due to numerous spectral channel details within a particular wavelength, huge data volume and the limitations on image calibration. Table 7.1 lists the widely used research HSI data sets with instrument specifications.

7.3.3

Satellite imaging vs UAV imaging

The satellite imaging instruments are the widely used mode of aerial imaging since the advent of RS. However, the spatial clarity is not much improved with both the MSI and HSI, though few high spatial resolution images are available from commercial satellites. The UAV images are more precise and subject specific. It can be

Multispectral vs hyperspectral image resolution

N

EO-1 ALI

Landsat 7 ETM+

EO-1 atmospheric corrector

EO-1 hyperion (7.7 Km) (37 Km)

(185 Km)

Figure 7.6 Spatial resolutions: MSI vs. HSI [30]

Multispectral vs. hyperspectral imaging for UAVs

141

Table 7.1 Hyperspectral data sets S. No.

Satellite

Study area

Sensor instruments

Spectral bands

1. 2. 3.

Indian Pines University of Pavia Salinas

AVIRIS sensor ROSIS sensor AVIRIS sensor

224 103 224

4.

Botswana

NASA EO-1

242

5.

Kennedy Space Center

North Western Indiana Pavia, North Italy Salinas Valley, California Okavango Delta, Botswana Florida

NASA AVIRIS

176

Table 7.2 Satellite imaging vs. UAV imaging Factors of influence

UAV imaging

Satellite imaging

Spatial resolution

Sub-centimetre level ( threshold

HR images CNN

Figure 9.2 The architecture of the proposed approach

Deep learning as an alternative to super-resolution imaging in UAV systems

181

9.2.1 Motion estimation There are possibilities of relative shift between input LR images. After identifying the best quality UAV images, it is better to align all other input LR frames globally before the start of the SR process. By aligning LR images globally, the possibility of erroneous alignment in the local patch domain can be minimised, which increases the success rate of local alignment. The selected best frame is used as a template image during the alignment process. To find the relative shift between the template image and LR images, modified diamond search algorithm proposed in [40] is used and discussed as follows. The proposed modified diamond search is an improvement over the cross diamond search [41] to fit small centre-biased characteristics of the videos. The images are divided into 16  16 pixel size of the macroblock. Four search patterns (SPs; up, down, left and right) in the macroblock are as shown in Figure 9.3.

Algorithm -

Step 1: The large diamond search pattern is at the centre of the search window. The sum of absolute errors (SAEs) for the 9 points is calculated, and the minimum SAE point is determined. The SAE is calculated using the below equation. SAE ¼

Q1 P1 X X   Cij  Rij ;

(9.2)

i¼0 j¼0

where P  Q is the size of the macroblock, and Cij and Rij are the pixels being compared in the current macroblock, and reference macroblock, respectively. The block that minimises the SAE will become the motion vector for the block at the position.

–

It represents the search location in a macro-block.

(a)

(c)

Up search pattern

Down search pattern

(b)

Left search pattern

(d)

Right search pattern

Figure 9.3 Search pattern in macroblock: (a) up search pattern, (b) left-side search pattern, (c) down search pattern and (d) right-side search pattern

182

Imaging and sensing for unmanned aircraft systems, volume 2

- Step 2: If the minimum point is not in origin, then update the origin and perform a search. If the minimum SAE occurs at the centre, then search stops otherwise update centre and proceed further. - Step 3: Based on the minimum point obtained from the step 2, the corresponding pattern from Figure 9.3 is selected. Here the number of search point reduces to 6. If the minimum point in the previous step occurs at the upper vertex of the SP, the new SP is (a). ● If the minimum point at the previous step occurs at the lower vertex of SP, the new SP is (b). ● If the minimum point in the previous step occurs at the left vertex of SP, the new SP is (c). ● If the minimum point in the previous step occurs at the right vertex of SP, the new SP is (d). ● If the minimum SAE occurs at the centre of the SP, then search stops, otherwise update the centre and proceed further. The minimum SAE point coincides with the required motion vector. These motion vectors tell how many pixels should be shifted in the image for it to align with another image. In the alignment stage, each input LR image is shifted with exactly the translational shift concerning the template UAV image in the opposite direction to align all images to the template image. Instead of using a global SR algorithm, local patch-based SR approach is used to super-resolve the UAV. In this method, the whole image is divided into subsets of square areas. The local deformation can be avoided using local patches to perform SR on the UAV image in the polar domain. After aligning the frames, each frame is divided into patches of size 10  10. Image patches of different images, which belong to the same location, are stored in the same ‘local set’. The image patches are rearranged and clustered according to their locations. Add image patch in the local set with the first-order derivative of the patch, which is generated by Sobel filter, to get an edgeenhanced patch. The next step is to fuse the pixel information from multiple patches. In this proposed work, an image fusion technique based on variance calculated in Discrete Cosine Transform (DCT) domain [41] is used to fuse the UAV images. The DCT coefficients are calculated for image blocks of size 8  8 pixels. The variance of each block decides the quality of the block. The best quality block is selected to fuse with another block in an image. In this way, pixel intensity information from multiple local patches is fused and generated one final patch which is supposed to be vivid, clean and sharp. This patch is sent to the proposed SR system.

9.2.2

Dehazing

The image captured by the UAV may contain haze, which is responsible for the unclear image. The dehazing block reduces the haze amount in the captured image [42,43]. The haze is reduced using a dark channel prior (DCP) method [44,45]. This method is usually used to produce a natural haze-free image. This method is based on the statistics of outdoor haze-free images. Haze image S ðxÞ is represented as: S ðxÞ ¼ Z ðxÞtðxÞ þ A½1  tðxÞ;

(9.3)

Deep learning as an alternative to super-resolution imaging in UAV systems

183

where A is the atmospheric light, Z ðxÞ is the foreground intensity and tðxÞ is the percentage of residual energy. The proposed method uses a DCP [27], represented as:   x c (9.4) D ¼ minyeWðxÞ min D ðyÞ ; cefr;g;bg

where Dc ðyÞ is a colour channel of D and WðxÞ is a local patch centred at x, and min is a minimum filter. The medium transmission is estimated by: y aWðxÞ   Dc ðyÞ ; (9.5) ť ¼ 1  minyeWðxÞ W min cefr;g;bg Ac where Ac is atmospheric light, W-little quantity of haze in the image to distinguish the depth of the image. Scene radiance J ðxÞ is recovered using the atmospheric light and the transmission medium. J ðxÞ can be calculated as: J ðx Þ ¼

S ðx Þ  A c þ Ac ; maxðtðxÞ; thÞ

(9.6)

where th is a threshold to avoid a low value of the denominator.

9.2.3 Patch selection Instead of using a global SR algorithm, local patch-based SR approach is used to super-resolve the UAV images. In this method, the whole image is divided into subsets of square areas. The contents of UAV image patches are analysed in this block by calculating the variance of each patch. If the variance of the patch is more than the pre-defined value, then the patch is sent to CNN block for SR. If the variance is less than the pre-defined value, then the patch is sent to up-sampling block. If CNN processes the patch which contains less amount of information, then the overall processing time will increase. To avoid this, the patch which contains less amount of information is processed by up-sampling. This patch selection method increases the speed of the SR process.

9.2.4 Super-resolution The image is super-resolved using a convolutional neural network as discussed in [35]. The overview of the CNN is as shown in Figure 9.4. The SR of UAV image is carried out in the following steps: 1.

First layer: This layer is called as patch Extraction and representation layer. The popular strategy for image reconstruction is by extracting patches and representing them by a set of pre-trained bases. This approach is similar to convolving the image by a set of filters, each of which is in this layer, overlapped patches of images are extracted and represented by a set of pre-trained discrete cosine transform bases. Let y be the LR UAV image. It is up-sampled to the required size using bicubic interpolation approach to get Y . The goal is

184

Imaging and sensing for unmanned aircraft systems, volume 2 Low-resolution input UAV image ×

Patch extraction and representation

feature maps of low-resolution image 1×1

feature maps of high-resolution image Non-linear mapping

×

Reconstruction

High-resolution image

Figure 9.4 Super-resolution using a convolutional neural network to recover from Y and image F ðY Þ, which is as similar as possible to the ground truth HR image Y . This layer is represented as: F1 ðY Þ ¼ maxð0; W1  Y þ B1 Þ;

2.

where W1 is a convolution filter having a size of m  f1  f1  n1 in which f1 is the spatial size of the filter, n1 is the number of filters, m is the number of channels and B1 is a bias. The output is composed of n1 feature maps. B1 is an n1 -dimensional vector, whose each element is associated with a filter. Second layer: This layer is called the non-linear mapping layer. n1 -dimensional feature vectors are mapped onto n2 -dimensional feature vectors. This is achieved by: F2 ðY Þ ¼ maxð0; W2  F1 ðY Þ þ B2 Þ:

3.

(9.7)

(9.8)

The size of W2 is n1  1  1  n2 , B2 is n2 dimensional vector and n2 is the number of feature maps. Third layer: This layer is called the reconstruction layer. All predicted HR patches are averaged to produce the final HR image. This is achieved in this manner: F ðY Þ ¼ W3  F 2 ðY Þ þ B 3 : The size of W3 is n2  f3  f3  k and B3 is an l-dimensional vector.

(9.9)

Deep learning as an alternative to super-resolution imaging in UAV systems

185

In the training process, network parameters P ¼ ðW1 ; W2 ; W3 ; B1 ; B2 ; B3 Þ have to be determined to learn the end-to-end mapping function F. These network parameters are obtained by reducing the loss between HR image X and the reconstructed images F ðY ; PÞ. Let Yi be the LR image and Yi be the corresponding HR image. The loss function is calculated as: LðPÞ ¼

n 1X jjF ðYi ; PÞ  Xi jj2 : n i¼1

(9.10)

The stochastic gradient descent with the standard backpropagation [46] is used to minimise the loss.

9.3 Experiments and results The experimental results of the proposed methodology are demonstrated by analysing the performance of the proposed SR system. The algorithms are implemented using MATLAB2012 on Intel Core i5 machine with RAM size of 8 GB. The effectiveness of the proposed technique is validated by performing the experiments on UAV images. In the training process, 220 training images are used. These images are subdivided into 64  64 sub-images to get approximately 28,000 images. The size of filters are set in these experiments as f1 ¼ 9 and f3 ¼ 5. The feature maps n1 ¼ 64 and n2 ¼ 32. The size of the input images is 640  480, and the up-sampling factor is 2. Figure 9.5 shows sample input images. Image quality assessment (IQA) metrics help assess the performance of the proposed system [47–52] such as SSIM, VIFP domain, PSNR and computational time. The SSIM is a measure for the assessment of quality, motivated by the information that the human visual system is sensitive to distortions of the structural components. This measure evaluates the changes in structural components occurred between two images. The SSIM between two images is computed as follows. Divide the original image and the super-resolved image into some patches. The similarity S ðx; yÞ can be calculated as: ! !  2mx my þ K1 2sx sy þ K2 2sxy þ K3 ; (9.11) S ðx; yÞ ¼ m2x þ m2y þ K1 s2x þ s2y þ K2 sx sy þ K3

(a) Industry

(b) Resident

(c) Parking

(d) Field

Figure 9.5 Input images: (a) industry, (b) resident, (c) parking, (d) field

186

Imaging and sensing for unmanned aircraft systems, volume 2

where mx and my represent the mean of patches x and y. sx and sy indicate standard deviations of patches x and y. K1 , K2 and K3 are positive constants to prevent the statistical unsteadiness that may take place when division with denominators. sxy is the cross-correlation of x and y. sxy ¼

T 1 X ðxi  mx Þðyi  my Þ; 跖  1 i¼1

(9.12)

where xi and yi are the pixel intensities in image patches x and y, respectively, and T is the number of pixels in each of the patches x and y. The SSIM score of the image is obtained by taking the average value of SSIM of the patches in the image. The range of SSIM value is between 0 and 1, where for the better quality image, the SSIM is specified by a higher value. The VIFP is the statistics between first and the last stage of the visual channel at a time of no distortion, and common data among the input of distortion chunk and the output of the visual system chunk. The outcome of this is a fidelity measure.

9.3.1

Peak signal-to-noise ratio

The PSNR is the ratio between the highest possible pixel value in the image and the noise power. It is expressed in terms of the logarithmic decibel (dB) and is given by: PSNRdB ¼ 10 log10

ð255Þ2 ; MSE

(9.13)

with the mean square error (MSE) is expressed as: MSE ¼

1 X N 1  X 2 1 M Cij  Rij ; M赊N i¼0 j¼0

(9.14)

where M and N are the height and width of the image respectively, Cij is the original image and Rij is a distorted image. According to Figure 9.6, the proposed method gives a better image quality than the state-of-the-art algorithms. To further analyse the proposed method, the upsampling factor is set to 4 and 6. The performances of the proposed approach for up-sampling factors 4 and 6 are as depicted in Table 9.1.

9.4 Critical issues in SR deployment in UAV-CPSs A few auspicious directions for an upcoming investigation are discussed in the following sections.

9.4.1

Big data

The focal challenges for employing SR to remotely sensed (RS) images are to outdo the scene alterations instigated by temporal differences while adapting the present procedures to colossal amounts of daily observations aka big data (BD).

Deep learning as an alternative to super-resolution imaging in UAV systems

187

0.9 35

0.85

33

Zeyed

32

Haris

SSIM

PSNR

34

1

2 3 Image No.

Zeyed Haris

0.7

Proposed

31

0.8 0.75

Proposed

0.65

4

1

2 3 Image No.

(a)

4

(b) 0.95

VIFP

0.9 0.85 Zeyed 0.8

Haris

0.75

Proposed

0.7 1

2 3 Image No.

4

(c)

Figure 9.6 Performance analysis of the proposed method for an up-sampling factor of 2 for UAV images. The average (a) PSNR, (b) SSIM and (c) VIFP Table 9.1 Performance analysis for the up-sampling factors 2 and 4 L

4

6

Image no.

1 2 3 4 1 2 3 4

Zeyed

Haris

Proposed

PSNR, dB

SSIM

VIFP

PSNR, dB

SSIM

VIFP

PSNR, dB

SSIM

VIFP

31.88 32.34 31.17 30.85 30.44 31.26 29.07 28.93

0.759 0.767 0.696 0.704 0.712 0.705 0.649 0.673

0.764 0.783 0.721 0.743 0.732 0.755 0.675 0.691

32.29 32.86 31.63 31.05 31.54 31.73 30.15 29.43

0.787 0.779 0.742 0.722 0.729 0.726 0.669 0.692

0.798 0.801 0.768 0.751 0.755 0.782 0.695 0.711

33.47 33.5 31.88 31.29 31.83 32.27 30.87 30.19

0.808 0.811 0.783 0.791 0.793 0.756 0.711 0.721

0.849 0.86 0.812 0.79 0.799 0.805 0.773 0.759

Lots of BD repositories will be accessible openly, especially in RS. With the vast amounts of global data, the bottleneck is to convert the raw data into useful knowledge and representations, e.g., metadata, geoinformation, semantic content, geometric facts and so on, to better handle them with almost entirely automatic

188

Imaging and sensing for unmanned aircraft systems, volume 2

treatment to collaboratively observe and connect independent processes that are interrelated. Machine learning (ML) has unlocked a new means of treating and exploiting BD. Particularly for uses involving multi-temporal data, recurrent neural networks (RNNs) can handle dynamic actions and time series explicitly. This is extremely helpful for filling gaps in gathered data, owing to, for instance, clouds or incomplete historical coverage. Furthermore, it facilitates the forecast of events, e.g., weather observation and crop yields. The effective estimation of parameters can impact considerably, and positively the SR usage in UAV-CPSs.

9.4.2

Cloud computing services

The SR requires solving ill-posed inverse problems that demand an extremely cumbersome computational load. In practical applications as is the case of UAVCPSs, this expensive use of computational resources will have to be lessened while leaving room for adaptive filtering and real-time strategies using cloud/edge/ extreme computing together with computational intelligence techniques to curb the number of operations needed in optimisation. Another issue is the fact that several imaging modalities must be fused to provide a better image detailing and description, which calls for cloud/fog/extreme computing and intensive use of communication on-board and off-board. The ability of single users to process massive data sets is limited. Recently, big international technology companies provide users and scientists tools that support the scientific analysis of petabyte-scale geospatial data. Google Earth Engine is an online platform that stores and organises satellite imagery and also provides convenient tools to browse and search data sets. With these tools, global-scale data mining is promoted, and algorithms can be globally applied to the vast amount of available data, without any need to download or store data at the user’s side. All the computations are done in the cloud, and the results are almost instantly presented to the user. Moreover, the data collection contains all the Sentinel 2 and Landsat collections, many MODIS products, precipitation data and elevation data, among others. The Google Earth Engine will soon also be able to run modern DL models after the TensorFlow environment has been connected with the platform. TensorFlow is a computer library that provides efficient tools for DL. Except for Google, also Amazon provides a cloud computing service, Amazon Web Services. This platform similarly contains open geospatial data, including the Landsat 8 and Sentinel 2 archives. Various plugins and services use this platform. In general, beyond these platforms built for geospatial data analysis and visualisation, cloud services enable large scale computations, including DL. Cloud computing is gaining ground and can be expected to lead to novel applications yet to be discovered eventually.

9.4.3

Image acquisition hardware limitations

Whenever possible, the error due to erroneous image registration should be seen as part of the model and relying on non-Gaussian additive noise [53]. More work on the colour channel interactions and the way they capture reflectivity changes can improve image registration.

Deep learning as an alternative to super-resolution imaging in UAV systems

189

For the case when one or more LR imageries are missing, there will be an impact on the associated optimisation problem effort. Sensor failures and shortcomings can be represented into the models to tolerate some control and fault-tolerance mechanisms related to the HR image. If the blur is introduced as is the case study previously addressed, better blind methodologies can be devised to treat the blur sources while generating the HR image [54]. Additional information about the SNR, illumination variations, occlusion, and shadowing from the acquired LR images can help balance the computational effort involved in obtaining an SR image. The use of hyperspectral images (HSIs) is increasing. This challenges SR HSI and also depends on the particular sensors’ configurations with panchromatic images (i.e., RGB images with three bands). Still, it may be worthwhile to add SR in the design step and adapt all the UAV-CPS characteristics conveniently. Research towards an optimised sensor arrangements will permit the signal nature exploitation by merging all the bands, to assure a better outcome [55]. The HSI cameras can catch and perform computational tasks on images in real-time, leading to extra information from highly redundant 4D cubes to allow the dynamic investigation of the complete temporal series. The multi-temporal unmixing [c4] process can possibly benefit from the knowledge of MFSR techniques. The fast, low-latency HSI analyses encourage using inexpensive sensors on small smart devices. The SR of an LR HSI with the usual HR RGB smartphone camera can provide real-time analysis. Lossy GPU-native compression formats such as BCn, ETC and ASTC have been widely used to reduce the video memory footprint of textures. A downloadable GPU application like a mobile 3D game should select the most suitable compression format and resolution per texture while taking a variety of the target device’s capabilities and download volume into consideration [55–58]. The HSIs afford abundant spectral features and help a variety of vision tasks [59–62]. Nevertheless, due to hardware limitations, not only collecting high-quality HSIs is much more difficult than collecting PANs, but also the resolution of collected HSIs is much lower. Thus, SR is introduced into this field, and researchers tend to combine HR PANs and LR HSIs to predict HR HSIs. Among them, Huang et al. [63] present a sparse denoising autoencoder to learn LR-to-HR mappings with PANs and transfer it to HSIs. Masi et al. [64] employ the SRCNN [65] and incorporate several maps of non-linear radiometric indices for boosting performance. Wei et al. [66] propose a much deeper DRPNN based on residual learning [67] and achieve higher spatial-spectral unified accuracy. Recently, Qu et al. [68] jointly train two encoder– decoder networks to perform SR on PANs and HSIs, respectively, and transfer the SR knowledge in the PAN domain to the HSI domain by sharing the decoder and applying constraints such as angle similarity loss and reconstruction loss.

9.4.4 Video SR In terms of video SR, MFs provide much more scene information, and there are not only intra-frame spatial dependency but also inter-frame temporal dependency

190

Imaging and sensing for unmanned aircraft systems, volume 2

(e.g., motions, brightness and colour changes). Thus, the existing works mainly focus on making better use of the spatiotemporal dependency, including explicit motion compensation (e.g., optical flow algorithms [69–71], learning-based methods) and recurrent methods, etc. Among the methods based on optical flow algorithms, Liao et al. [72] employ various optical flow methods to generate HR candidates and ensemble them by CNNs. VSRnet [73] and CVSRnet [74] implement motion compensation by Druleas algorithm [75] and use CNNs to take successive frames as input and predict HR frames. Liu et al. [76,77] perform rectified optical flow alignment and propose a temporal adaptive net to generate HR frames in various temporal scales and aggregate them adaptively. Besides, others also try to learn motion compensation directly. The VESPCN [78] utilises a trainable spatial transformer [79] to learn motion compensation based on adjacent frames and enters MFs into a spatiotemporal ESPCN for end-to-end prediction. Moreover, Tao et al. [80] root from accurate LR imaging model and propose a sub-pixel-like module to achieve motion compensation and SR simultaneously, and thus fuse the aligned frames more effectively. Another trend is to use recurrent methods to capture the spatial-temporal dependency without explicit motion compensation. Specifically, the BRCN [81] employs a bidirectional framework and uses CNN, RNN and conditional CNN to model the spatial, temporal and spatial-temporal dependence, respectively. Similarly, STCN [82] uses a deep CNN and a bidirectional LSTM [83] to extract spatial and temporal information. Furthermore, FRVSR [84] uses previously inferred HR estimates to reconstruct the subsequent HR frame by two deep CNNs in a recurrent manner. In addition to the above works, the FAST [84,85] exploits the compact description of the structure and pixel correlations extracted by compression algorithms, transfers the SR result from one frame to adjacent frames and accelerates the state-of-the-art SR algorithms by 15 times with little performance loss (0.2 dB). Moreover, Jo et al. [86] generate dynamic up-sampling filters and the HR residual image based on the local spatio-temporal neighbourhood of each pixel and also avoid explicit motion compensation.

9.4.5

Efficient metrics and other evaluation strategies

Evaluation metrics are one of the most fundamental components for ML. If the metrics cannot accurately measure model performance, researchers will have great difficulty verifying improvements [87]. Metrics for SR face such challenges and need more exploration. More accurate metrics: The most widely used metrics for SR are PSNR and SSIM. However, the PSNR tends to result in excessive smoothing, and the results often vary wildly between almost indistinguishable images. The SSIM evaluates images in terms of brightness, contrast and structure, but still cannot measure perceptual image quality accurately. Besides, the mean-opinion-score (MOS) is closest to human visual response, but takes a lot of workforce and effort and is nonreproducible. Thus, more accurate metrics for evaluating reconstruction quality are urgently needed.

Deep learning as an alternative to super-resolution imaging in UAV systems

191

Blind IQA methods: Today, most metrics used for SR are all-reference methods, i.e., assuming that LR-HR images have been paired with perfect quality. Nevertheless, since it is difficult to obtain such data sets, the commonly used data sets for evaluation are often conducted by manual degradation. In this case, the evaluation task is the inverse process of pre-defined degradation. Therefore, developing blind IQA methods also has great demands. CNN-based methods have recently achieved great success for image SR. However, most deep CNN-based SR models attempt to improve distortion measures (e.g., PSNR, SSIM, IFC, VIF) while resulting in poor quantified perceptual quality (e.g., human opinion score, no-reference quality measures such as NIQE). Few works have attempted to improve the perceptual quality at the cost of performance reduction in distortion measures. A very recent study has revealed that distortion and perceptual quality are at odds with each other and there is always a trade-off between the two. Often the restoration algorithms that are superior in terms of perceptual quality are inferior in terms of distortion measures. The work analyses the trade-off between distortion and perceptual quality for the problem of SISR called enhanced deep SR (EDSR) network and adapted it to achieve better perceptual quality for a specific range of the distortion measure. While the original network of EDSR was trained to minimise the error defined based on per-pixel accuracy alone, the network was trained using a generative adversarial network (GAN) framework [88,89] with EDSR as the generator module [90]. The work in [87] is called enhanced perceptual SR (EPSR) network and it is trained with a combination of MSE loss, perceptual loss and adversarial loss. Experiments reveal that EPSR achieves the state-of-theart trade-off between distortion and perceptual quality while the existing methods perform well in either of these measures alone.

9.4.6 Multiple priors Blurring is usually modelled by some known distribution, which is far from reality. Ideally, the type of blurring needs to be estimated along with the other model parameters. Blur identification can improve the SR reconstruction considerably and allows for simultaneous image restoration. Mixture prior models can simplify matters by transforming the big non-linear feature space associated with LR images into a collection of linear sub spaces in the training phase. First, the image is partitioned into image patches employing a novel discriminating patch treatment scheme relying on the different curvature of LR patches. Next, the mixture prior models corresponding to each set are learned. Furthermore, since different prior distributions may play various effectiveness in SR, some distributions can be chosen and combined like the student-t prior to providing more robust performance than the accustomed Gaussian prior. The learned compound mixture prior models help to map the contributing input LR features into the proper subspace during the testing phase and lastly reconstruct the equivalent HR picture in a fresh mixed equivalent way. Experimental outcomes indicate that this methodology is both quantitatively as well as qualitatively better than some contemporary SR methods [91].

192

9.4.7

Imaging and sensing for unmanned aircraft systems, volume 2

Regularisation

The SR problem is extremely ill-posed due to an insufficient number of the observed LR frames. A classical methodology to address it is called regularisation, and it uses different kinds of priors [92]. A classical and, perhaps, popular prior is based on the Tikhonov model [9,46,92–94]. This prior introduces into the restoration problem a smoothness constraint that removes extraneous noise from an image. The weakness of the Tikhonov prior is that it tends to destroy edges—an effect that degrades images. Therefore, the prior has captured the interest of many researchers to develop models that simultaneously suppress noise and preserve critical image features: Huber Markov random field (Huber-RMF) [95,96], edge-adaptive RMF [97,98], sparse directional [99,100] and total variation (TV) [5,75,93,101–103]. Of the aforementioned priors, TV has attracted more attention of researchers as it generates results with a pleasing objective and subjective qualities. The major weaknesses of the TV model are blocking and staircasing effects, false-edge generation near edges and non-differentiability property at zero—a situation that makes the numerical implementation rather challenging. The TV model was initially applied in image denoising [104]. Later, the model was adapted to other applications: SR, MRI medical image reconstruction [105], inpainting [106,107] and deblurring [108]. This work explores some classical TV-based approaches to address the SR problem [53,101,105]. In 2008, Marquina and Osher proposed a convolutional model for an SR problem based on the constrained TV framework [101]. In their work, the authors introduced the Bregman algorithm as an iterative refinement step to enhance spatial resolution [109–114]. The use of the Bregman divergence allows for a more encompassing model [115]. The results demonstrate that the method generates detailed and sharper images, but blocking and staircasing artefacts are still evident. In [103], Farsiu and colleagues proposed a bilateral total variation (BTV) prior, which is based on the L1 norm minimisation and the bilateral filter regularisation functional, for an MF SR problem [116–118]. Their method is computationally reasonable and robust against errors caused by motion and blur estimations and produces convincingly sharper images. Still, the method’s bilateral filters introduce artefacts like staircasing and gradient reversal. Moreover, the BTV imperfectly treats the partial image smoothness [119]. Furthermore, implementing the L1 norm through numerical methods is troublesome since they can conceal the SR data term. The TV prior has addressed the following issues in the SR video reconstruction: noise, blurring, missing areas, compression artefacts and errors in motion estimation [70,71,120] with efficacy in several types of motions and scene degradations with experimental outcomes beating other classical SR approaches. Ren et al. proposed an SR method, which is based on the fractional-order TV regularisation, with a focus to handle fine details, such as textures, in the image [121]. Results show that their approach addresses to some extent the weaknesses of the traditional TV. In [21], Li et al. attempted to address the drawbacks of the global TV by proposing two, namely locally adaptive TV and consistency of gradients, to ensure that

Deep learning as an alternative to super-resolution imaging in UAV systems

193

edges are sharper and flat regions are smoother. The method heavily depends on the gradient details of an image, a feature that may produce pseudo-edges in noisy homogeneous regions. Note that both noise and edges are image features with high-gradient (or high-intensity) values. As Li’s method is gradient dependent, it may equally treat both noise and edges, and this may generate unwanted artefacts. Yuan et al. proposed a spatially weighted TV model for the MFSR problem [122]. Their model incorporates a spatial information indicator (difference curvature) that locally identifies the spatial properties of the individual pixels, thus providing the necessary level of regularisation. The authors employed the majorisation-minimisation algorithm to optimise their formulation. Results show that the Yuan et al. method overcomes some challenges of the original TV model (discourages piecewise constant solutions) and is less sensitive.

9.4.8 Novel architectures The SISR has obtained unprecedented breakthrough with the development of CNNs [123,124]. A majority of these methods try to increase the depth of the network to obtain a larger receptive field. However, this work found that blindly stacking feature maps and the simple cascading structure cannot achieve a high rate of utilisation in SR reconstruction with a polished fully CNN for SISR. Based on the assumption that feature maps from different depths or the same depth but different channels have different contributions during image reconstruction, the squeeze and excitation network evaluated the importance of different feature maps while building the network [125]. Besides, densely connection operation is also conducted in the framework for better use of the contextual information and feature maps. Extensive experiments demonstrate that the proposed method can enhance the restoration performance and achieve the state-of-the-art results in the SR task. Sparse coding has been widely applied to learning-based SISR and has obtained promising performance by jointly learning effective representations for LR and HR image patch pairs. However, the resulting HR images often suffer from ringing, jaggy and blurring artefacts due to the strong, yet ad hoc assumptions that the LR image patch representation is equal to is linear, and it lies on a manifold similar to or has the same support set as the corresponding HR image patch representation. Motivated by the success of DL, a data-driven model named coupled deep autoencoder (CDA) for SISR has been proposed [126]. CDA is based on a new deep architecture and has a high representational capability. CDA simultaneously learns the intrinsic representations of LR and HR image patches and a big-data-driven function that precisely maps these LR representations to their corresponding HR representations. Extensive experimentation demonstrates the superior effectiveness and efficiency of CDA for SISR compared to other modern methods on Set5 and Set14 data sets. The paper in [127] addresses the problem of learning robust cross-domain representations for sketch-based image retrieval (SBIR). While most SBIR approaches focus on extracting low- and mid-level descriptors for direct feature matching, recent works have shown the benefit of learning coupled feature representations to describe data from two related sources. However, cross-domain representation

194

Imaging and sensing for unmanned aircraft systems, volume 2

learning methods are typically cast into non-convex minimisation problems that are difficult to optimise, leading to unsatisfactory performance. Inspired by self-paced learning, a learning methodology designed to overcome convergence issues related to local optima by exploiting the samples in a meaningful order (i.e., from easy to hard), the cross-paced partial curriculum learning (CPPCL) framework has been proposed. Compared with existing self-paced learning methods which only consider a single modality and cannot deal with prior knowledge, CPPCL is specifically designed to assess the learning pace by jointly handling data from dual sources and modality-specific prior information provided in the form of partial curricula. Additionally, thanks to the learned dictionaries, the CPPCL embeds robust coupled representations for SBIR. This approach is extensively evaluated on four publicly available data sets showing superior performance over competing for SBIR methods. OctNet is a representation for DL with sparse 3D data [128]. In contrast to existing models, this representation enables 3D CNNs, which are both deep and HR. Towards this goal, the sparsity in the input data to hierarchically partition the space using a set of unbalanced octrees has been done where each leaf node stores a pooled feature representation. This allows to focus memory allocation and computation to the relevant dense regions and enables deeper networks without compromising resolution. The utility of the OctNet representation is done by analysing the impact of resolution on several 3D tasks including 3D object classification, orientation estimation and point cloud labelling [129]. Curriculum learning [130] refers to starting from an easier subtask and gradually increasing the task difficulty. Since SR is essentially an ill-posed problem and some adverse conditions such as large scaling factors, noise or blurring further increase the learning difficulty, the curriculum training strategy helps a lot on this problem. Considering that performing SR with large factors in one step is a very difficult task, Wang et al. [47] and Bei et al. [131] propose ProSR and ADRSR, respectively, which are progressive not only on architectures (Section 3.1.3) but also on training procedure. The training starts with the 2 up-sampling portions, and after finishing training current portions, the portions with 4 or larger scaling factors are gradually mounted and blended with the previous portions. Specifically, the ProSR blends two portions by linearly combining the output of this level and the up-sampled output of previous levels following [132], while the ADRSR concatenates them and attaches another convolutional layer. In contrast, Park et al. [71] divide the 8 SR problem to three subproblems (i.e., 1 to 2 SR, 2 to 4 SR, 4 to 8 SR) and train an individual network for each problem. Then two of them is concatenated and fine-tuned jointly, and then with the other one. Besides, they also decompose the 4 SR under difficult conditions into three sub-problems (i.e., denoising/deblurring, 1 to 2 SR, 2 to 4 SR) and adopt a similar training strategy. Compared to common training procedure, this curriculum learning strategy not only greatly reduces the training difficulty and improves the performance with all scaling factors, especially for large factors, but also significantly shortens the total training time. Despite the breakthroughs in accuracy and speed of SISR using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how to recover the finer texture details using SR at large upscaling factors? The choice of the

Deep learning as an alternative to super-resolution imaging in UAV systems

195

objective function principally drives the behaviour of optimisation-based SR procedures. Recent work has primarily focused on minimising the mean squared reconstruction error. The resulting estimates have high PSNRs, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. SRGAN is a GAN for SR capable of inferring photo-realistic natural images for 4 upscaling factors. It uses a perceptual loss function, which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the SR images and original photo-realistic images. In addition, it uses a content loss motivated by perceptual similarity instead of similarity in pixel space. The deep residual network is able to recover photo-realistic textures from heavily down-sampled images on public benchmarks. An extensive MOS test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original HR images than to those obtained with any contemporary method [133]. Deep CNNs have been widely used and achieved state-of-the-art performance in many images or video processing and analysis tasks. In particular, for image SR processing, previous CNN-based methods have led to significant improvements, when compared with shallow learning-based methods. However, previous CNNbased algorithms with simple direct or skip connections are of poor performance when applied to RS satellite images SR. In this study, a simple but effective CNN framework, namely deep distillation recursive network (DDRN) [134], is presented for video satellite image SR. DDRN includes a group of ultra-dense residual blocks (UDBs), a multi-scale purification unit (MSPU) and a reconstruction module. In particular, through the addition of rich interactive links in and between multiple-path units in each UDB, features extracted from multiple parallel convolution layers can be shared effectively. Compared with classical dense-connection-based models, DDRN possesses the following main properties: 1. 2.

3.

DDRN contains more linking nodes with the same convolution layers. A distillation and compensation mechanism, which performs feature distillation and compensation in different stages of the network, is also constructed. In particular, the high-frequency components lost during information propagation can be compensated in MSPU. The final SR image can benefit from the feature maps extracted from UDB and the compensated components obtained from MSPU. Experiments on Kaggle open source data set and Jilin-1 video satellite images illustrate that DDRN outperforms the conventional CNN-based baselines and some state-of-the-art feature extraction approaches.

9.4.9 3D SR 9.4.9.1 Depth map super-resolution Depth maps record the distance between the viewpoint and the objects in the scene, and the depth information plays important roles in many tasks such as pose estimation [89,123,124], semantic segmentation [135,136], etc. However, due to

196

Imaging and sensing for unmanned aircraft systems, volume 2

productive and economic limitations, the depth maps produced by depth sensors are often LR and suffer degeneration effects such as noise, quantisation, missing values, etc. Thus, SR is introduced for increasing the spatial resolution of depth maps. Today one of the most popular practices for depth map SR is to use another economical RGB camera to obtain HR images of the same scenes for guiding super-resolving the LR depth maps. Specifically, Song et al. [137] exploit the depth field statistics and the local correlation between depth maps and RGB images to constrain the global statistics and local structure. Hui et al. [138] utilise two CNNs to simultaneously up-sample LR depth maps and down-sample HR RGB images, then use RGB features as the guidance of the up-sampling process at the same resolution. Similarly, Ni et al. [139] and Zhou et al. [114] use HR RGB images as guidance by extracting HR edge map and predicting missing high-frequency components, respectively. While Xiao et al. [140] use the pyramid network to enlarge the receptive field, extract features from LR depth maps and HR RGB images, respectively and fuse these features to predict HR depth maps. And Haefner et al. [141] fully exploit the colour information to guide SR by resorting to the shape-from-shading technique. In contrast to the above works, Riegler et al. [142] combine CNNs with an energy minimisation model in the form of a powerful variational model to recover HR depth maps without other reference images. The problem of estimating the latent HR image of a 3D scene from a set of nonuniformly motion-blurred LR images captured in the burst mode using a hand-held camera appears in [i1, i2, h5]. Existing blind SR techniques that account for motion blur are restricted to front-parallel planar scenes. Initially, an SR motion blur model was conceived to explain the image formation process in 3D scenes. Later, this model has been applied to identify the three unknowns: the camera trajectories, the depth map and the latent HR image. The next step is to recover the global HR camera motion corresponding to each LR observation from patches lying on a reference depth layer in the input images. Using the estimated trajectories, the latent HR image is obtained and the underlying depth map iteratively using an alternating minimisation framework. Experiments on synthetic and real data reveal that the proposed method outperforms the state-of-the-art techniques by a significant margin. Single-objective selective-plane illumination microscopy (soSPIM) is achieved with micro-mirrored cavities combined with a laser beam–steering unit installed on a standard inverted microscope. The illumination and detection are done through the same objective. soSPIM can be used with standard sample preparations and features high background rejection and efficient photon collection, allowing for 3D single-molecule-based SR imaging of whole cells or cell aggregates. Larger mirrors enabled to extend the system capabilities [143]. The 3D reconstruction of thick samples using SR fluorescence microscopy (FM) remains challenging due to the high level of background noise and fast photobleaching of fluorescence probe [144]. SR FM can reconstruct 3D structures of thick samples with both high localisation accuracy and no photobleaching problem. The background noise is reduced by optically sectioning the sample using line-scan confocal microscopy and the photobleaching problem is overcome using the DNA-PAINT (Point Accumulation for Imaging in Nanoscale Topography).

Deep learning as an alternative to super-resolution imaging in UAV systems

197

9.4.10 Deep learning and computational intelligence Although deep networks have shown exceptional performance on the SR task, there remain several open research questions. Some of these future research directions are enlisted as follows. Incorporation of priors: Current deep networks for SR are data-driven models that are learned in an end-to-end fashion. While this approach has shown excellent results in general, it proves to be sub-optimal when a particular class of degradation occurs for which a large amount of training data is non-existent (e.g., in medical imaging). In such cases, if the information about the sensor, imaged object/scene and acquisition conditions is known, useful priors can be designed to obtain HR images. Recent works focusing on this direction have proposed both deep network [130] and sparse coding [131] based priors for better SR. Objective functions and metrics: Existing SR approaches predominantly use pixel-level error measures, e.g., l1, l2 distances, another norm or a combination of metrics. Since these measures only encapsulate local pixel-level information, the resulting images do not always provide perceptually sound results. As an example, it has been shown that images with high PSNR and SSIM values give overly smooth images with low perceptual quality [132]. To counter this issue, several perceptual loss measures have been proposed in the literature. The conventional perceptual metrics were fixed, e.g., SSIM [145], multi-scale SSIM [146], while more recent ones are learned to model human perception of images, e.g., LPIPS [147] and PieAPP [148]. Each of these measures has its own failure cases. As a result, there is no universal perceptual metric that optimally works in all conditions and perfectly quantifies the image quality. Therefore, the development of new objective functions is an open research problem. Need for unified solutions: Two or more degradations often happen simultaneously in real-life situations. An important consideration in such cases is how to jointly recover images with higher resolution, low noise and enhanced details. Current models developed for SR are generally restricted to only one case and suffer in the presence of other degradations. Furthermore, problem-specific models differ in their architectures, loss functions and training details. It is a challenge to design unified models that perform well for several low-level vision tasks, simultaneously [124,135]. Unsupervised image SR: Models discussed in this survey generally consider LR-HR image pairs to learn an SR mapping function. One interesting direction is to explore how SR can be performed for cases where corresponding HR images are not available. One solution to this problem is zero-shot SR which learns the SR model on a further down-sampled version of a given image. However, when an input image is already of poor resolution, this solution cannot work. The unsupervised image SR aims to solve this problem by learning a function from unpaired LR-HR image sets. Such a capability is very useful for real-life settings since it is not trivial to obtain matched HR images in several cases [149]. Higher SR rates: Current SR models generally do not tackle extreme SR, which can be useful for cases such as augmenting the resolution of faces in crowd scenes. Very few works target SR rates higher than 8 (e.g., 16 and 32) [j51].

198

Imaging and sensing for unmanned aircraft systems, volume 2

In such extreme up-sampling conditions, it becomes challenging to preserve accurate local details in the image. Further, an open question is how to preserve high perceptual quality in these super-resolved images. Arbitrary SR rates: Impractical scenarios, it is often not known which up-sampling factor is the optimal one for a given input. When the down-sampling factor is not known for all the images in the data set, it becomes a significant challenge during training since it becomes hard for a single model to encapsulate several levels of details. In such cases, it is important first to characterise the level of degradation before training and performing inference through a specified SR model. Real vs artificial degradation: Existing SR works mostly use a bicubic interpolation to generate LR images. Actual LR images that are encountered in real-world scenarios have a different distribution compared to the ones generated synthetically using bicubic interpolation. As a result, SR networks trained on artificially created degradations do not generalise well to actual LR images in practical scenarios. One recent effort towards the solution of this problem first learns a GAN to model the real-world degradation [149].

9.4.11 Network design Proper network design not only determines a hypothesis space with excellent performance upper bound but also supports the data learning efficiently along with data representations without excessive spatial and computational redundancy. Below, some promising directions for network improvements are discussed. Combining local and global information: The large receptive field provides more contextual information and helps generate more realistic HR images. It is promising to combine local and global information for providing contextual information on different scales for SR [150–155]. Combining low- and high-level information: Shallow layers in deep CNNs tend to extract low-level features such as colours and edges, while deeper layers extract higher-level representations like the object identities. Thus combining low-level details with high-level abstract semantics can be of great help for HR reconstruction. Context-specific attention: Different contexts focus on different information for SR. For example, the grass area may be more concerned with colours and textures, while the animal body area may focus more on the hair details. Therefore, incorporating the attention mechanism to exploit contextual information to enhance the attention to key features facilitates the generation of realistic details. Lightweight architectures: Existing SR modes tend to pursue ultimate performance while ignoring the model size and inference speed. For example, the EDSR [136] takes 20s for 4 SR per image of DIV2K [137] on a Titan GTX [114], and DBPN [138] takes 35s for 8 SR [139]. Such a long prediction time is unacceptable in practical applications. Thus, lightweight architectures are imperative. How to reduce model sizes and speed up prediction while maintaining performance remains a problem. Up-sampling layers: Although up-sampling operations play a very important role for SR, existing methods have more or fewer disadvantages: the interpolation-based methods result in expensive computation and cannot be end-to-end learned, the

Deep learning as an alternative to super-resolution imaging in UAV systems

199

transposed convolution produces checkerboard artefacts and the sub-pixel layer brings the uneven distribution of receptive fields. Hence, how to perform effective and efficient up-sampling still needs to be studied, especially with high scaling factors. Nevertheless, the exhaustive usage of SR is still problematic. A particular bottleneck appears in the MF SR frameworks that lead researchers to seek SI example-based SR intensely. Consequently, the performance of single-image SR algorithms hinges on the trustworthiness of the outer database. More adaptive, progressive and speedier techniques with far-reaching applicability are still in need. The rapid hardware progress will also challenge new applications of the SR structure. For example, the Google Skybox project will deliver real-time HR using RS SR imaging. SR can also be expanded FM and multi-baseline tomographic synthetic aperture radar imaging. Nonetheless, the feasibility of SR methods in UAV-CPS is still somewhat limited due to the comparatively poor computational performance and the associated time consumption, as well as the necessary acceleration tactics required by large-scale applications.

9.5 Conclusion The goal of this work is to super-resolve the images captured by UAV. This chapter proposed a CNN-based SR framework for LR images founded on a very extensive bibliographic survey [156–164]. From the analyses, it can be seen that the proposed CNN-based framework can super-resolve the UAV images effectively and gives better performance compared to state-of-art algorithms in term of MSE, PSNR, SSIM and VIFP. The SR imaging can circumvent or compensate the intrinsic hardware and communication restrictions of the imaging framework for affording a clearer image with a better-off and helpful content [165–188]. SR can also assist the front-end pre-processing phase in expediting various computer vision applications to expand their performance. Other metaheuristics can help to preprocess data before deep architectures to improve speed and reduce the computational load of deep learning frameworks [189–192].

References [1] S. K. Seelan, S. Laguette, G. M. Casady, and G. A. Seielstad, Remote sensing applications for precision agriculture: a learning community approach, Remote Sensing and Environment, 88(1), 157–169, 2003. [2] J. Everaerts, The use of unmanned aerial vehicles (UAVs) for remote sensing and mapping, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 37(2), 1187–4, 2008. [3] G. Grenzdo¨rffer, A. Engel, and B. Teichert, The photogrammetric potential of low-cost UAVs in forestry and agriculture, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 31(3), 1207–1213, 2008.

200

Imaging and sensing for unmanned aircraft systems, volume 2

[4] V. V. Estrela, O. Saotome, H. J. Loschi, et al., Emergency response cyberphysical framework for landslide avoidance with sustainable electronics, Technologies, 6, 42, 2018. doi: 10.3390/technologies6020042. [5] M. A. de Jesus, V. V. Estrela, and O. Saotome, Super-resolution in a nutshell, Proceedings BTSym 2016, Campinas, SP, Brazil. [6] Y. Tang, P. Yan, Y. Yuan, and X. Li, Single-image super-resolution via local learning, International Journal of Machine Learning and Cybernetics, 2, 15–23, 2011. [7] W. Wu, Z. Liu, X. He, and W. Gueaieb, Single-image super-resolution based on Markov random field and contourlet transform, Journal of Electronic Imaging, 20(2), 023005–023005, 2011. [8] K. I. Kim, and Y. Kwon, Single-image super-resolution using sparse regression and natural image prior, IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(6), 1127–1133, 2010. [9] A. M. Coelho, and V. V. Estrela, A study on the effect of regularization matrices in motion estimation, International Journal of Computer Applications, 51(19), 17–24, 2012. [10] W. T. Freeman, T. R. Jones, and E. C. Pasztor, Example-based superresolution, IEEE Computer Graphics and Applications, 22(2), 56–65, 2002. [11] K. Zhang, X. Gao, X. Li, and D. Tao, Partially supervised neighbor embedding for example-based image super-resolution, IEEE Journal of Selected Topics in Signal Processing, 5(2), 230–239, 2011. [12] C. Kim, K. Choi, and J. B. Ra, Example-based super-resolution via structure analysis of patches, IEEE Signal Processing Letters, 20(4), 407– 410, 2013. [13] Z. Xiong, D. Xu, X. Sun, and F. Wu, Example-based super-resolution with soft information and decision, IEEE Transactions on Multimedia, 15(6), 1458–1465, 2013. [14] C. Y. Yang, J. B. Huang, and M. H. Yang, Exploiting self-similarities for single frame super-resolution, in Proc Computer Vision–ACCV 2010. (Springer Queenstown, New Zealand, 2011), pp. 497–510. [15] O. Mac Aodha, N. D. Campbell, A. Nair, and G. J. Brostow, Patch based synthesis for single depth image superresolution, in Proc. Computer Vision– ECCV (Springer Florence, Italy, 2012), pp. 71–84. [16] X. Gao, K. Zhang, D. Tao, and X. Li, Joint learning for single-image superresolution via a coupled constraint, IEEE Transactions on Image Processing, 21(2), 469–480, 2012. [17] J. Li, and X. Peng, Single-frame image super-resolution through gradient learning, in Proc 2012 International Conference on Information Science and Technology (ICIST) (IEEE Hubei, 2012), Wuhan, China pp. 810–815. [18] K. Zhang, X. Gao, D. Tao, and X. Li, Single image super-resolution with non-local means and steering kernel regression, IEEE Transactions on Image Processing, 21(11), 4544–4556, 2012. [19] M. Yang, and Y. Wang, A self-learning approach to single image superresolution, IEEE Transactions on Multimedia, 15, 498–508, 2013.

Deep learning as an alternative to super-resolution imaging in UAV systems

201

[20] K. Zhang, X. Gao, D. Tao, and X. Li, Single image super-resolution with multiscale similarity learning, IEEE Transactions on Neural Networks and Learning Systems, 24(10), 1648–1659, 2013. [21] X. Li, Y. Hu, X. Gao, D. Tao, and B. Ning, A multi-frame image superresolution method, Signal Processing, 90(2), 405–414, 2010. [22] B. J. Maiseli, Q. Liu, O. A. Elisha, and H. Gao, Adaptive Charbonnier superresolution method with robust edge preservation capabilities, Journal of Electronic Imaging, 22(4), 043027, 2013. [23] B. Maiseli, O. Elisha, J. Mei, and H. Gao, Edge preservation image enlargement and enhancement method based on the adaptive Perona–Malik nonlinear diffusion model, IET Image Processing, 8, 753–760, 2014. [24] X. D. Zhao, Z. F. Zhou, J. Z. Cao, et al., Multi-frame super-resolution reconstruction algorithm based on diffusion tensor regularization term, Applied Mechanics and Materials, 543, 2828–2832, 2014. [25] J. Yang, and T. Huang, Image super resolution: historical overview and future challenges. In Super-Resolution Imaging, pp. 1–24. CRC Press, Boca Raton, FL, USA, www.ifp.illinois.edu/~jyang29/papers/chap1.pdf – United States. [26] J. Sun, J. Sun, Z. Xu, and H. Y. Shum, Image Super-Resolution Using Gradient Profile Prior, IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 2008. [27] W. S. Tam, C. W. Kok, and W. C. Siu, Modified edge-directed interpolation for images, Journal of Electronic Imaging, 19(1), 1–20, 2010. [28] L. Wang, S. Xiang, G. Meng, H. Wu, and C. Pan, Edge-directed singleimage super-resolution via adaptive gradient magnitude self interpolation, IEEE Transactions on Circuits and Systems for Video Technology, 23, 8, 1289–1299, 2013. [29] M. Bertero, and P. Boccacci. Introduction to Inverse Problems in Imaging, CRC Press, Boca Raton, Florida, USA, 1998. [30] M. V. Joshi, S. Chaudhuri, and R. Panuganti, A learning-based method for image super-resolution from zoomed observations, IEEE Transactions on Systems, Man, and Cybernetics—Part B: Cybernetics, 35(3), 527–537, 2005. [31] M. V. Joshi, S. Chaudhuri, and P. Rajkiran, Super-resolution imaging: use of zoom as a cue, Image and Vision Computing, 14(22), 1185–14, 2004. [32] K. Zhang, X. Gao, D. Tao, and X. Li, Single image super-resolution with nonlocal means and steering kernel regression, IEEE Transaction on Image Processing, 21, 4544–4556, 2012. [33] X. Gao, Q. Wang, X. Li, D. Tao, and K. Zhang, Zernike-moment-based image super-resolution, IEEE Transaction on Image Processing, 20, 2738–2747, 2011. [34] M. Irani, and S. Peleg, Improving resolution by image registration, CVGIP Graphical Models and Image Processing, 53, 231–237, 1991. [35] C. Dong, and C. C. Loy, Image super-resolution using deep convolutional networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295–307, 2016. [36] J. Chen, and X. He, Single image super-resolution based on deep learning and gradient transformation, IEEE 13th International Conference on Signal Processing (ICSP), Chengdu, China, 2016.

202 [37]

[38]

[39]

[40] [41]

[42] [43] [44] [45] [46]

[47]

[48]

[49]

[50]

[51] [52] [53]

Imaging and sensing for unmanned aircraft systems, volume 2 Y. Yoon, and H. G. Jeon, Learning a deep convolutional network for lightfield image super-resolution, IEEE International Conference on Computer Vision Workshop, Santiago, Chile, 2015. Z. Wang, and D. Liu, Deep networks for image super-resolution with sparse prior, IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 2015. Y. Kato, and S. Ohtani, Image super-resolution with multi-channel convolutional neural networks, IEEE International New Circuits and Systems Conference (NEWCAS), Vancouver, BC, Canada, 2016. A. Deshpande, and P. Patavardhan, Multiframe super-resolution for long range captured iris polar image, IET Biometrics, 6(2), 108, 2017. M. Haghighat, H. Seyedarabi, and A. Aghagolzadeh, Multi-focus image fusion for visual sensor networks in DCT domain, Computers and Electrical Engineering, 37(5), 789–797, 2011. R. Fattal, Single Image Dehazing, Proc. ACM SIGGRAPH ’08, Los Angeles, CA, USA, 2008. R. Tan, Visibility in Bad Weather from a Single Image, Proc. IEEE Conf. Computer Vision and Pattern Recognition, Anchorage, Alaska, USA, 2008. J.-B. Wang, H. Ning, Z. Lu-Lu, and L. Ke, Single image dehazing with a physical model and dark channel prior, Neurocomputing, 149, 718–728, 2015. Z. Li, Zhengguo, and J. Zheng, Edge-preserving decomposition-based single image haze removal, IEEE Transactions on Image Processing, 24, 5432–5441, 2015. N. Nguyen, P. Milanfar, and G. Golub, A computationally efficient superresolution image reconstruction algorithm, IEEE Transactions on Image Processing, 10(4), 573–583, 2001. Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing, 13(4), 600–612, 2004. S. Bovik, and G. Veciana, An information fidelity criterion for image quality assessment using natural scene statistics, IEEE Transaction on Image Processing, 14(4), 2117–2128, 2014. T. Lukes, K. Fliegel, and M. Klima, Performance Evaluation of Image Quality Metrics with Respect to their Use for super-resolution Enhancement, Fifth International Workshop on Qualcomm Multimedia Experience, 2013. X. Zhou, and B. Bhanu, Evaluating the Quality of Super-Resolved Images for Face Recognition, IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008. Z. Wang, and A. C. Bovik, Mean squared error: love it or leave it? A new look at signal fidelity measures, IEEE Signal Processing Magazine, 26, 98–117, 2009. A. Tanchenko, Visual-PSNR measure of image quality, Journal of Visual Communication and Image Representation, 25(5), 874–878, 2014. A. M. Coelho, V. V. Estrela, F. P. Carmo, and S. R. Fernandes, Error Concealment by Means of Motion Refinement and Regularized Bregman Divergence, Proceedings of the 13th International Conference on Intelligent Data Engineering and Automated Learning, Natal, Brazil, 2012.

Deep learning as an alternative to super-resolution imaging in UAV systems

203

[54] A Punnappurath, T. M. Nimisha, and A. N. Rajagopalan, Multi-image blind super-resolution of 3D scenes, IEEE Transactions on Image Processing, 26, 5337–5352, 2017. [55] S. Takemura, Optimize deep super-resolution and denoising for compressed textures. SIGGRAPH Asia Posters, Tokyo, Japan, 2018. [56] G. Kim, and N. Baek, A Height-Map Based Terrain Rendering with Tessellation Hardware. 2014 International Conference on IT Convergence and Security (ICITCS), Beijing, China, 2014, pp. 1–4. [57] S. R. Fernandes, V. V. Estrela, Hermes A. Magalhaes, and O. Saotome, On Improving Sub-Pixel Accuracy by Means of B-Spline, Proceedings of the 2014 IEEE International Conference on Imaging Systems and Techniques (IST 2014), Santorini, Greece, pp. 68–72, ISBN: 9781479952199. [58] F. D. Silva, V. V. Estrela, and L. J. Matos, Hyperspectral Analysis of Remotely Sensed Images, Ed.: Carolina Biblio, Oliver Hensel, Jeferson Francisco Selbach (Org.). Sustainable Water Management in the Tropics and Subtropics-and Case Studies in Brazil. 1ed., Kassel, Germany: University of Kassel, 2011, vol. 2, pp. 398–423, ISBN: 9788563337214. [59] M. Fauvel, Y. Tarabalka, J. A. Benediktsson, J. Chanussot, and J. C. Tilton, Advances in the spectral-spatial classification of hyperspectral images, Proceedings of the IEEE, 101, 652–675, 2013. [60] Y. Fu, Y. Zheng, I. Sato, and Y. Sato, Exploiting spectral-spatial correlation for coded hyperspectral image restoration, in Proc. 2016 Conference on Computer Vision and Pattern Recognition (CVPR 2016), Las Vegas, NV, USA, 2016. [61] B. Uzkent, M. J. Hoffman, and A. Vodacek, Real-time vehicle tracking in aerial video using hyperspectral features, in Proc. 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW2016), Las Vegas, NV, USA, 2016. [62] B. Uzkent, A. Rangnekar, and M. J. Hoffman, Aerial vehicle tracking by adaptive fusion of hyperspectral likelihood maps, in Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW2017), Honolulu, HI, USA. [63] W. Huang, L. Xiao, Z. Wei, H. Liu, and S. Tang, A new pansharpening method with deep neural networks, in IEEE Geoscience and Remote Sensing Letters, vol. 12, 5, 1037–1041, 2015. [64] G. Masi, D. Cozzolino, L. Verdoliva, and G. Scarpa, Pansharpening by convolutional neural networks, Remote Sensing, 8, 594, 2016. [65] C. Dong, C. C. Loy, K. He, and X. Tang, Learning a deep convolutional network for image super-resolution, in Proc. 13th European Conference on Computer Vision (ECCV 2014), Zurich, Switzerland, 2014. [66] Y. Wei, Q. Yuan, H. Shen, and L. Zhang, Boosting the accuracy of multispectral image pansharpening by learning a deep residual network, IEEE Geoscience and Remote Sensing Letters, 14, 10, 1795–1799, 2017. [67] K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proc. 2016 Conference on Computer Vision and Pattern Recognition (CVPR 2016), Las Vegas, NV, USA, 2016.

204 [68]

[69]

[70]

[71]

[72] [73]

[74]

[75]

[76]

[77]

[78]

[79]

[80]

[81]

Imaging and sensing for unmanned aircraft systems, volume 2 Y. Qu, H. Qi, and C. Kwan, Unsupervised sparse Dirichlet-net for hyperspectral image super-resolution, in Proc. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, 2018. M. A. de Jesus, and V. V. Estrela, Optical flow estimation using total least squares variants, Oriental Journal of Computer Science and Technology, 10, 563–579, 2017. doi: 10.13005/ojcst/10.03.03. A. M. Coelho, and V. V. Estrela, State-of-the-art motion estimation in the context of 3D TV, Ed.: Reuben A. Farrugia, Carl J. Debono. (Ed.). Multimedia Networking and Coding. 1ed., IGI Global, Hershey, PA, USA, 2013, pp. 148–173, ISBN: 1466626607. V. V. Estrela, M. H, da Silva Bassani, and L. A. Rivera, ExpectationMaximization Technique and Spatial-Adaptation Applied to Pel-Recursive Motion Estimation, Proceedings of the WSCG 2004, Pilzen, Czech Republic, 2004. R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia, Video super-resolution via deep draft-ensemble learning, in Proc ICCV, Santiago, Chile, 2015. A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos, Video superresolution with convolutional neural networks, IEEE Transactions on Computational Imaging, 2, 2, 109–122, 2016. A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos, Super-resolution of compressed videos using convolutional neural networks, in Proc ICIP 2016, Phoenix, Arizona, USA, 2016. V. V. Estrela, H. A. Magalhaes, and O. Saotome, Total variation applications in computer vision. In N. K. Kamila (Ed) Handbook of Research on Emerging Perspectives in Intelligent Pattern Recognition, Analysis, and Image Processing, pp. 41–64. IGI Global, Hershey, PA, USA, 2016. DOI: 10.4018/978-1-4666-8654-0.ch002. D. Liu, Z. Wang, Y. Fan, et al., Robust video super-resolution with learned temporal dynamics, in Proc 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017. D. Liu, Z. Wang, Y. Fan, et al., Learning temporal dynamics for video superresolution: A deep learning approach, IEEE Transactions on Image Processing, 27(7), 3432–3445, 2018. J. Caballero, C. Ledig, A. P. Aitken, et al., Real-time video super-resolution with spatiotemporal networks and motion compensation, in Proc 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, Hawaii, USA, 2017. M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, Spatial transformer networks, in Proc Twenty-ninth Conference on Neural Information Processing Systems (NIPS 2015), Montreal, Canada, 2015. X. Tao, H. Gao, R. Liao, J. Wang, and J. Jia, Detail-revealing deep video super-resolution, in Proc. 2017 International Conference on Computer Vision (ICCV 2017), Venice, Italy, 2017. Y. Huang, W. Wang, and L. Wang Video super-resolution via bidirectional recurrent convolutional networks, TPAMI, 40, 4, 1015–1028, 2018.

Deep learning as an alternative to super-resolution imaging in UAV systems

205

[82] J. Guo, and H. Chao, Building an end-to-end spatial-temporal convolutional network for video super-resolution, in Proc Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), San Francisco, California, USA, 2017. [83] A. Graves, S. Fernandez, and J. Schmidhuber, Bidirectional LSTM networks for improved phoneme classification and recognition, in Proc International Conference on Artificial Neural Networks 2005 (ICANN 2005), Warsaw, Poland, 2005. [84] M. S. Sajjadi, R. Vemulapalli, and M. Brown, Frame-recurrent video superresolution, in Proc. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, 2018. [85] Z. Zhang, and V. Sze, Fast: a framework to accelerate superresolution processing on compressed videos, in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW 2017), Honolulu, Hawaii, USA, 2017. [86] Y. Jo, S. W. Oh, J. Kang, and S. J. Kim, Deep video superresolution network using dynamic upsampling filters without explicit motion compensation, in 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, 2018. [87] S Vasu, T. M. Nimisha, and N. A. Rajagopalan, Analyzing perceptiondistortion tradeoff using enhanced perceptual super-resolution network. Proc, European Conference on Computer Vision (ECCV) workshops, Munich, Germany, 2018, 114–131. [88] Y. Yuan, S. Liu, J. Zhang, Y. Zhang, C. Dong, and L. Lin, Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks, in 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW 2018), Salt Lake City, UT, USA, 2018. [89] A. Bulat, J. Yang, and G. Tzimiropoulos, To learn image superresolution, use a gan to learn how to do image degradation first, in Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 2018, pp. 185–200. [90] C. Ledig, L. Theis, F. Huszar, et al., Photo-realistic single image superresolution using a generative adversarial network. In Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, USA, 2017, pp. 105–114. [91] Y. Huang, J. Li, X. Gao, L. He, and W. Lu, Single image super-resolution via multiple mixture prior models, IEEE Transactions on Image Processing, 27, 5904–5917, 2018. [92] A. M. Coelho, and V. V. Estrela, Data-driven motion estimation with spatial adaptation. International Journal of Image Processing (IJIP), 6(1), 54, 2012. [93] V. V. Estrela, H. A. Magalhaes, and O. Saotome, Total variation applications in computer vision, Ed.: Narendra Kumar Kamila, Handbook of Research on Emerging Perspectives in Intelligent Pattern Recognition, Analysis, and Image Processing. IGI Global, Hershey, PA, USA, 2016. doi:10.4018/978-14666-8654-0.ch002, http://www.igiglobal.com/chapter/total-variation-applications-in-computer-vision/141626

206 [94]

[95]

[96]

[97]

[98]

[99] [100]

[101]

[102]

[103]

[104] [105]

[106]

[107]

[108]

Imaging and sensing for unmanned aircraft systems, volume 2 M. Elad, and A. Feuer, Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Transactions on Image Processing, 6(12), 1646–1658, 1997. D. Rajan, and S. Chaudhuri, An MRF-based approach to generation of super-resolution images from blurred observations, Journal of Mathematical Imaging and Vision, 16, 5–15, 2002. A. Kanemura, S.-i. Maeda, and S. Ishii, Superresolution with compound Markov random fields via the variational EM algorithm. Neural Networks, 22(7), 1025–1034, 2009. K. V. Suresh, G. M. Kumar, and A. Rajagopalan, Superresolution of license plates in real traffic videos, IEEE Transactions on Intelligent Transportation Systems, 8(2), 321–331, 2007. W. Zeng, and X. Lu, A generalized DAMRF image modeling for superresolution of license plates. IEEE Transactions on Intelligent Transportation Systems, 13(2), 828–837, 2012. S. Mallat, and G. Yu, Super-resolution with sparse mixing estimators. IEEE Transactions on Image Processing, 19(11), 2889–2900, 2010. W. Dong, D. Zhang, G. Shi, and X. Wu, Image deblurring and superresolution by adaptive sparse domain selection and adaptive regularization. IEEE Transactions on Image Processing, 20(7), 1838–1857, 2011. A. Marquina, and S. J. Osher, Image super-resolution by TV-regularization and Bregman iteration, Journal of Scientific Computing, 37(3), 367–382, 2008. M. K. Ng, H. Shen, E. Y. Lam, and L. Zhang, A total variation regularization based super-resolution reconstruction algorithm for digital video, EURASIP Journal on Advances in Signal Processing, 2007. doi: 10.1155/2007/74585. S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, Fast and robust multiframe super resolution. IEEE Transactions on Image Processing, 13(10), 1327–1344, 2004. L. I. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena, 60, 259–268, 1992. F. Knoll, K. Bredies, T. Pock, and R. Stollberger, Second order total generalized variation (TGV) for MRI, Magnetic Resonance in Medicine, 65(2), 480–491, 2011. 38. P. Getreuer, Total variation inpainting using split Bregman, Image Processing On Line, 2, 147–157, 2012. A. Deshpande, and P. P. Patavardhan, super-resolution of Long Range Captured Iris Image Using Deep Convolutional Network, IOS Series: Advances in Parallel Computing. Volume 31, Published 2017, Netherlands. V. V. Estrela, O. Saotome, M. A. de Jesus, and D. Stutz, Super-resolution via particle swarm optimization variants, biologically rationalized computing techniques for image processing applications, Eds.: Valentina E. Balas, Jude Hemanth, Berlin: Springer, 2017, pp. 317–337, (book chapter). doi: 10.1007/ 978-3-319-61316-1_14. A. Bini, and M. Bhat, A nonlinear level set model for image deblurring and denoising, The Visual Computer, 30(3), 311–325, 2014.

Deep learning as an alternative to super-resolution imaging in UAV systems [109] [110]

[111]

[112]

[113]

[114]

[115]

[116]

[117]

[118]

[119] [120]

[121] [122]

207

D. Cai, K. Chen, Y. Qian, and J.-K. Kamarainen, Convolutional lowresolution fine-grained classification, Pattern Recognition Letters, 2017. J. Li, X. Liang, Y. Wei, T. Xu, J. Feng, and S. Yan, Perceptual generative adversarial networks for small object detection, in In Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, USA, 2017. W. Tan, B. Yan, and B. Bare, Feature super-resolution: make machine see more clearly, in Proc. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, 2018. N. Huang, Y. Yang, J. Liu, X. Gu, and H. Cai, Single-image superresolution for remote sensing data using deep residual-learning neural network, in Proc. 24th International Conference on Neural Information Processing (ICONIP 2017), Guangzhou, China, 2017. D. S. Jeon, S.-H. Baek, I. Choi, and M. H. Kim, Enhancing the spatial resolution of stereo images using a parallax prior, in Proc. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, 2018. R. Timofte, E. Agustsson, L. Van Gool, et al., Ntire 2017 challenge on single image super-resolution: methods and results, in Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW 2017), Salt Lake City, UT, USA, Honolulu, Hawaii, USA, 2017. B. J. Maiseli, O. A. Elisha, and H. Gao, A multi-frame super-resolution method based on the variable-exponent nonlinear diffusion regularizer, EURASIP Journal on Image and Video Processing, 2015:22, 2015. doi: 10.1186/s13640-015-0077-2. S. Durand, J. Fadili, and M. Nikolova, Multiplicative noise removal using L1 fidelity on frame coefficients, Journal of Mathematical Imaging and Vision, 36(3), 201–226, 2010. S. Yin, X. Li, H. Gao, and O. Kaynak, Data-based techniques focused on modern industry: an overview, IEEE Transactions on Industrial Electronics, 62, 657–667, 2015. S. Yin, S. X. Ding, X. Xie, and H. Luo, A review on basic data-driven approaches for industrial process monitoring, IEEE Transactions on Industrial Electronics, 61(11), 6418–6428, 2014. W. Zeng, X. Lu, and S. Fei, Image super-resolution employing a spatial adaptive prior model, Neurocomputing, 162, 218–233, 2015. A. M. Coelho, and V. V. Estrela, EM-based mixture models applied to video event detection. Ed.: Parinya Sanguansat, In Principal Component Analysis – Engineering Applications, IntechOpen, London, UK, 2012, 101–124. Z. Ren, C. He, and Q. Zhang, Fractional order total variation regularization for image super-resolution, Signal Processing, 93(9), 2408–2421, 2013. Q. Yuan, L. Zhang, and H. Shen, Multiframe super-resolution employing a spatially weighted total variation model, IEEE Transactions on Circuits and Systems for Video Technology, 22(3), 379–392, 2012.

208

Imaging and sensing for unmanned aircraft systems, volume 2

[123]

A. Shocher, N. Cohen, and M. Irani, Zero-shot super-resolution using deep internal learning, in Proc. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, 2018. K. Zhang, W. Zuo, and L. Zhang, Learning a single convolutional superresolution network for multiple degradations, in Proc CVPR, 2018. T. Jiang, Y. Zhang, X. Wu, Y. Rao, and M. Zhou, Single Image superresolution via Squeeze and Excitation Network. In Proc. 2018 British Machine Vision Conference (BMVC 2018), Newcastle, UK, 2018. K. Zeng, J. Yu, R. Wang, C. Li, and D. Tao, Coupled deep autoencoder for single image super-resolution, IEEE Transactions on Cybernetics, 47, 27–37, 2017. D. Xu, X. Alameda-Pineda, J. Song, E. Ricci, and N. Sebe, Cross-Paced Representation Learning with Partial Curricula for Sketch-based Image Retrieval , arXiv 2018. G. Riegler, A. O. Ulusoy, and A. Geiger. OctNet: Learning Deep 3D Representations at High Resolutions. In Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, Honolulu, Hawaii, USA, pp. 6620–6629. P. Lindstrom, and J. D. Cohen, On-the-fly decompression and rendering of multiresolution terrain. In Proc. 2010 Symposium on Interactive 3D Graphics (SI3D 2010), Washington, DC, USA, 2010. D. Ulyanov, A. Vedaldi, and V. Lempitsky, Deep image prior, in in Proc. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, 2018. W. Dong, Z. Yan, X. Li, and G. Shi, Learning hybrid sparsity prior for image restoration: Where deep learning meets sparse coding, arXiv preprint arXiv:1807.06920, 2018. Y. Blau, R. Mechrez, R. Timofte, T. Michaeli, and L. Zelnik-Manor, The 2018 pirm challenge on perceptual image super-resolution, in Proc. European Conference on Computer Vision (ECCV 2018), Munich, Germany, 2018. I. Kassamakov, A. Nolvi, and E. Haggstrom, 3D super-resolution label-free imaging. 2016 Conference on Lasers and Electro-Optics (CLEO), SPIE, San Jose, CA, USA, 2016, pp. 1–2. K. Jiang, Z. Wang, P. Yi, J. Jiang, J. Xiao, and Y. Yao, Deep distillation recursive network for remote sensing imagery super-resolution. Remote Sensing, 10, 1700, 2018. K. I. Kim, and Y. Kwon, Single-image super-resolution using sparse regression and natural image prior, TPAMI, 32, 6, 1127–1133, 2010. B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee, Enhanced deep residual networks for single image super-resolution, In Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW 2017), Honolulu, Hawaii, USA, 2017. E. Agustsson, and R. Timofte, Ntire 2017 challenge on single image superresolution: dataset and study, In Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW 2017), Honolulu, Hawaii, USA, 2017.

[124] [125]

[126] [127]

[128]

[129]

[130]

[131]

[132]

[133]

[134]

[135] [136]

[137]

Deep learning as an alternative to super-resolution imaging in UAV systems [138]

[139]

[140]

[141]

[142]

[143]

[144]

[145]

[146]

[147]

[148]

[149] [150]

209

M. Haris, G. Shakhnarovich, and N. Ukita, Deep back-projection networks for super-resolution, in Proc. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, 2018. C. Ancuti, C. O. Ancuti, R. Timofte, et al., Ntire 2018 challenge on image dehazing: methods and results, in Proc. 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW 2018), Salt Lake City, UT, USA, 2018. H. R. Marins, and V. V. Estrela, On the Use of Motion Vectors for 2D and 3D Error Concealment in H.264/AVC Video. Feature Detectors and Motion Detection in Video Processing. IGI Global, Hershey, PA, USA, 2017, pp. 164–186. doi:10.4018/978-1-5225-1025-3.ch008. E. Hidago-Pena, L. F. Marin-Urias, F. Montes-Gonzalez, A. MarinHernandez, and H. V. Rios-Figueroa, Learning from the Web: recognition method based on object appearance from Internet images, in Proc. 2013 ACM/ IEEE Int’l Conference on Human-Robot Interaction. IEEE, Tokyo, Japan, 2013, pp. 139–140. J. Wang, S. Krishnan, M. J. Franklin, K. Goldberg, T. Kraska, and T. Milo, A Sample-and-Clean Framework for Fast and Accurate Query Processing on Dirty Data, in Proc. ACM SIGMOD Int’l Conf Management of Data, Snowbird, UT, USA, 2014. R. Galland, G. Grenci, A. Aravind, V. Viasnoff, V. Studer, and J. Sibarita, 3D high- and super-resolution imaging using single-objective SPIM, Nature Methods, 12, 641–644, 2015. S. Park, W. Kang, Y. H. Kwon, et al., Superresolution fluorescence microscopy for 3D reconstruction of thick samples, Molecular Brain, 11(1), 17, 2018. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, Image quality assessment: from error visibility to structural similarity, TIP, 13, 4, 600– 612, 2004. Z. Wang, E. P. Simoncelli, and A. C. Bovik, Multiscale structural similarity for image quality assessment, in Proc. The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers 2003 (2003 ASILOMAR), Pacific Grove, CA, USA, 2003. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, The unreasonable effectiveness of deep features as a perceptual metric, in Proc. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, 2018. E. Prashnani, H. Cai, Y. Mostofi, and P. Sen, PieAPP: perceptual imageerror assessment through pairwise preference, in Proc. 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Salt Lake City, UT, USA, 2018. S. Anwar, S. Khan, and N. Barnes, A deep journey into super-resolution: a survey, CoRR, 2019, abs/1904.07523. E. A. Ogada, Z. Guo, and B. Wu, An alternative variational framework for image denoising, in Abstract and Applied Analysis, Volume 2014 (Hindawi

210

[151]

[152]

[153]

[154] [155]

[156]

[157]

[158]

[159]

[160]

[161]

[162]

[163]

[164]

Imaging and sensing for unmanned aircraft systems, volume 2 Publishing Corporation 410 Park Avenue 15th Floor, #287 pmb New York, NY 10022 USA, 2014). P. Perona, and J. Malik, Scale-space and edge detection using anisotropic diffusion, IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(7), 629–639, 1990. Z. Guo, J. Sun, D. Zhang, and B. Wu, Adaptive Perona–Malik model based on the variable exponent for image denoising, IEEE Transactions on Image Processing, 21(3), 958–967, 2012. S. Levine, Y. Chen, and J. Stanich, Image restoration via nonstandard diffusion. Duquesne University, Department of Mathematics and Computer Science Technical Report:04–01, 2004. J. Weickert, Anisotropic diffusion in image processing, Volume 1. (Teubner Stuttgart, 1998). R. Courant, K. Friedrichs, and H. Lewy, On the partial difference equations of mathematical physics, IBM Journal of Research and Development, 11(2), 215–234, 1967. D. Giordan, Use of unmanned aerial vehicles in monitoring application and management of natural hazards, Journal Geomatics, Natural Hazards and Risk, 8, 2017. K. Anderson, and K. J. Gaston, Lightweight unmanned aerial vehicles will revolutionize spatial ecology, Frontiers in Ecology and the Environment, 11(3), 138, 2013. S. Getzin, R. S. Nuske, and K. Wiegand, Using unmanned aerial vehicles (UAS) to quantify spatial gap patterns in forests, Remote Sensing, 6(8), 6988–7004, 2014. D. Giordan, A. Manconi, F. Remondino, and F. Nex, Use of unmanned aerial vehicles in monitoring application and management of natural hazards, Geomatics, Natural Hazards and Risk, 8, 1–4, 2017. J. Yang, J. Wright, T. S. Huang, and Y. Ma, Image super-resolution via sparse representation, IEEE Transactions on Image Processing, 19(11), 2861–2873, 2010. R. Zeyde, M. Elad, and M. Protter, On single image scale-up using sparserepresentations, in Curves Surfaces. Berlin, Germany: Springer, 2012, pp. 711–719. M. Haris, K. Sawase, M. R. Widyanto, and H. Nobuhara, An efficient super resolution based on image dimensionality reduction using accumulative intensity gradient, Journal of Advanced Computational Intelligence and Intelligent Informatics, 18(4), 518–528, 2014. S. C. Park, M. KyuPark, and M. G. Kang, super-resolution Image Reconstruction: A Technical Overview, IEEE Signal Processing Magazine, 20, 3, 21– 6, May 2003. H. Jia, A new cross diamond search algorithm for block motion estimation, In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, Montreal, Quebec, Canada, 2004.

Deep learning as an alternative to super-resolution imaging in UAV systems [165]

[166]

[167]

[168]

[169]

[170]

[171] [172] [173]

[174]

[175]

[176]

[177]

[178]

211

Y. Huang, W. Wang, and L. Wang, Bidirectional recurrent convolutional networks for multi-frame super-resolution, in Proc NIPS, Montreal, Quebec, Canada, 2015. W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, Deep Laplacian pyramid networks for fast and accurate superresolution, in Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018), Honolulu, Hawaii, USA, 2017. M. Ciocarlie, C. Pantofaru, K. Hsiao, G. Bradski, P. Brook, and E. Dreyfuss, A side of data with my robot, IEEE Robotics and Automation Magazine, 18(2), 44–57, 2011. H. Y. Chow, K. S. Lui, and E.Y. Lam, Efficient on-demand image transmission in visual sensor networks, EURASIP Journal on Advances in Signal Processing, 1, 1–11, 2007. B. Liu, Y. Chen, E. Blasch, and K. Pham, A Holistic Cloud-Enabled Robotics System for Real-Time Video Tracking Application, in Future Information Technology, 2014, pp. 455–468. A. A. Proia, D. Simshaw, and K. Hauser, Consumer cloud robotics and the fair information practice principles: Recognizing the challenges and opportunities ahead, Minnesota Journal of Law, Science & Technology, 2014. T. Winkler, and B. Rinner, Security and privacy protection in visual sensor networks: a survey, ACM Computing Surveys, 47(1):1–42, 2014. K. Ren, C. Wang, and Q. Wang, Security challenges for the public cloud, IEEE Internet Computing, 16(1), 69–73, 2012. W. Beksi, and N. Papanikolopoulos, Point cloud culling for robot vision tasks under communication constraints, in International Conference on Intelligent Robots and Systems (IROS), Chicago, IL, USA, 2014. K. Lai, and D. Fox, Object recognition in 3D point clouds using web data and domain adaptation, International Journal of Robotics Research (IJRR), 29(8), 1019–1037, 2010. C. Aguero, N. Koenig, I. Chen, and G. A. Pratt, Inside the Virtual Robotics Challenge: Simulating Real-time Robotic Disaster Response, IEEE Transactions on Automation Science and Engineering (T-ASE): SI on Cloud Robotics and Automation, 12(2), 2015, 494–506. K. E. Bekris, R. Shome, A. Krontiris, and A. Dobson, Cloud Automation: Precomputing Roadmaps for Flexible Manipulation, IEEE Robotics & Automation Magazine: SI on Emerging Advances and Applications in Automation, 2015;22:41–50. L. Wang, M. Liu, and M. Q.-H Meng, Real-time multi-sensor data retrieval for cloud robotic systems, IEEE Transactions on Automation Science and Engineering (T-ASE): Special Issue on Cloud Robotics and Automation, 12(2), 2015, 507–518. S. Jorda´n, T. Haidegger, L. Kova´cs, I. Felde, and I. J. Rudas, The rising prospects of cloud robotic applications. 2013 IEEE 9th International Conference on Computational Cybernetics (ICCC), Tihany, Hungary, 2013;327–332.

212

Imaging and sensing for unmanned aircraft systems, volume 2

[179]

A. Kasper, Z. Xue, and R. Dillmann, The KIT object models database: an object model database for object recognition, localization and manipulation in service robotics, International Journal of Robotics Research (IJRR), 31(8), 927–934, 2012. J. Mahler, S. Krishnan, M. Laskey, et al., Learning accurate kinematic control of cable-driven surgical robots using data cleaning and Gaussian process regression, in IEEE Int’l Conference on Automation Science and Engineering (CASE), Taipei, Taiwan, 2014. G. Mohanarajah, D. Hunziker, M. Waibel, and R. D’Andrea, Rapyuta: a cloud robotics platform, IEEE Transactions on Automation Science and Engineering (T-ASE), 12, 1–13, 2014. G. Mohanarajah, V. Usenko, M. Singh, M. Waibel, and R. D’Andrea, Cloud-based collaborative 3D mapping in real-time with low-cost robots, IEEE Transactions on Automation Science and Engineering (T-ASE): Special Issue on Cloud Robotics and Automation, 12(2), 2015, 423–431. L. Riazuelo, J. Civera, and J. Montiel. C2TAM: A Cloud Framework for Cooperative Tracking and Mapping, Robotics and Autonomous Systems, 62(4), 401–413, 2013. L. Riazuelo, M. Tenorth, D. Marco, et al., RoboEarth semantic mapping: a cloud enabled knowledge-based approach, IEEE Transactions on Automation Science and Engineering (T-ASE): Special Issue on Cloud Robotics and Automation, 12(2), 2015, 432–443. J. Salmeron-Garcia, F. Diaz-del Rio, P. Inigo-Blasco, and D. Cagigas, A trade-off analysis of a cloud-based robot navigation assistant using stereo image processing, IEEE Trans. Automation Science and Engineering (T-ASE): SI on Cloud Robotics and Automation, 12(2), 444–454, 2015. B. Kehoe, S. Patil, P. Abbeel, and K. Goldberg, A Survey of Research on Cloud Robotics and Automation, IEEE Transactions on Automation Science and Engineering, 12(2), 398–409, 2015. V. V. Estrela, A. C. B. Monteiro, R. P. Franc¸a, Y. Iano, A. Khelassi, and N. Razmjooy, Health 4.0: Applications, Management, Technologies and Review. Medical Technologies Journal, 2019;2(4):262–276, DOI: 10.26415/2572-004X-vol2iss1p262-276. 262. V. V. Estrela, A. Khelassi, A. C. B. Monteiro, et al., Why software-defined radio (SDR) matters in healthcare?. Medical Technologies Journal, vol. 3, no. 3, Oct. 2019, pp. 421–9, doi:10.26415/2572-004X-vol3iss3p421-429. J. Hemanth, and V. V. Estrela, Deep learning for image processing applications. Adv. Par. Comp. IOS Press, Netherlands. ISBN978-1-61499-821-1 (print) 978-1-61499822-8 (online), 2017. N. Razmjooy, and V. V. Estrela, Applications of image processing and soft computing systems in agriculture (pp. 1–337). Hershey, PA: IGI Global, 2019. doi:10.4018/978-1-5225-8027-0. N. Razmjooy, V. V. Estrela, and H. J. Loschi, A study on metaheuristicbased neural networks for image segmentation purposes, data science theory, analysis and applications, Taylor and Francis, Abingdon, UK, 2019.

[180]

[181]

[182]

[183]

[184]

[185]

[186]

[187]

[188]

[189]

[190]

[191]

Deep learning as an alternative to super-resolution imaging in UAV systems [192]

213

N. Razmjooy, M. Ramezani, and V. V. Estrela, A solution for Dubins path problem with uncertainties using world cup optimization and Chebyshev polynomials. In: Iano Y., Arthur R., Saotome O., Vieira Estrela V., and Loschi H. (Eds) Proceedings of the 4th Brazilian Technology Symposium (BTSym’18). BTSym 2018, Campinas, SP, Brazil, 2019. Smart Innovation, Systems and Technologies, vol 140. Springer, Cham, Zurich, Switzerland.

This page intentionally left blank

Chapter 10

Quality of experience (QoE) and quality of service (QoS) in UAV systems Asif Ali Laghari1, Asiya Khan2, Hui He1, Vania V. Estrela3, Navid Razmjooy4, Jude Hemanth5, and Hermes J. Loschi6,7

This chapter studies both the quality of service (QoS) and quality of experience (QoE) of unmanned aerial vehicle cyber-physical systems (UAV-CPSs). These parameters help to gather data about the connectivity options in networks containing flying nodes (FNs), ground stations (GSs) and other associated devices. They also support to solve complications related to the number of choices of subnetworks complicates designs, capacity, spectrum efficiency, network coverage and reliability among other issues from a flying ad hoc network (FANET) especially when there is streaming. QoS and QoE permit the discovery of the best conceivable network configurations, and costs for user applications autonomously. Existing lines of attack are listed by function. Restrictions and strong points are emphasised to arrange for initial investigations as well as further studies in this area. If the UAV-CPS network has low QoS, then real-time data will not be accurate and subject to data losses causing inaccuracies. Information loss or delay (due to packet loss, rearrangement and delay) may lessen the satisfaction level of the UAV-CPS user (operator). Network QoS degradation affects the real-time video monitoring and must always be taken into account with on-board needs. This effect leads to data loss and image quality reduction, thus decreasing the QoE.

1

School of Computer Science & Technology, Harbin Institute of Technology, Harbin, China School of Engineering, University of Plymouth, Plymouth, United Kingdom 3 Telecommunications Department, Federal Fluminense University (UFF), RJ, Brazil 4 Department of Electrical Engineering, Tafresh University, Tafresh, Iran 5 Electrical and Computer Engineering Department, Karunya University, Coimbatore, India 6 Institute of Electrical Engineering, University of Zielona Go´ra, Zielona Go´ra, Poland 7 School of Electrical and Computer Engineering (FEEC), University of Campinas – UNICAMP, Campinas – SP, Brazil 2

216

Imaging and sensing for unmanned aircraft systems, volume 2

10.1 Introduction As the computational components are inherently distributed, they have to cope with the uncertainty of the sensor input and need to produce real-time responses. A UAV with all the ground and remote sensing equipment can be perceived as a CPS because it involves several computational and physical elements [1–4] often interacting over different types of networks. Such a system will be treated in this chapter as UAV-CPS. UAVs can also work as an FN belonging to FANET. Technical limitations in the CPSs introduce some shortcomings. UAV-CPS initiatives must address issues such as intelligent and autonomic automobiles, ambient intelligence, self-organisation, plant control and reparation, self-optimisation, smart power grids, and so forth. UAV-CPSs call for a rethinking of the tools for the analysis, requirements, verification, validation, standards, safety, simulation procedures and regulations for certification, among other issues. Novel solutions extendable to CPSs for unwanted coupling in the physical domain and cross-disciplinary interfaces have to be established to avoid interference. UAV-CPSs operate in dynamic contexts involving uncertainty due to abnormal comportment, sporadic activities, openness or structural changes. These unknown dependencies can mess up the independence assumptions and call for adaptation. Moreover, interfaces and scaling can complicate matters. Hence, a UAV-CPS must be robust to mischievous attacks from either the cyber or physical domains with mechanisms for error-prone operation. Moreover, physical constituents differ qualitatively from object-oriented software components. Hence, standard abstractions relying on method calls and threads are not adequate. A typical UAV-CPS architecture appears in Figure 10.1. QoS refers to the ability of the UAV-CPS achieving the best deterministic behaviour in terms of the transmission thru networks with the lowest packet loss, maximum bandwidth, minimum delay, best image/video handling (e.g., capturing, streaming and recording) [5,6]. QoS builds on the objective parameters to enhance and quantify the performance of a UAV-CPS. QoS is a significant UAV-CPS concern because of real-time monitoring, capturing and targeting of different subsystems involving the recording of high-definition (HD) and high-quality images and video to be broadcast via networks to the central or a GS. Quality of experience (QoE) represents the end user’s judgment by measuring the level of the user’s satisfaction, enjoyment and expectations, which is a subjective

Communication infrastructure UAV

Ground station

Figure 10.1 Overview of a UAV-CPS

Quality of experience (QoE) and quality of service (QoS) in UAV systems

217

metric [7]. It entails issues such as perception, the experience of/with the applications, beliefs and network performance. QoE can be obtained from QoS. Vendors use QoE to adjust user necessities and demands thru interviews, web-based surveys and questionnaires to obtain subjective evidence from users about products or services [3,8–13]. However, UAV-CPSs necessitate holistic and continuous QoE measures to allow subsystems to acclimatise to the necessities without too much interaction with users [2]. The growing network traffic, in addition to end-users expectations at a low cost, increases the importance of monitoring high-quality QoE delivery for multimedia thru UAV-CPSs in real-time. The escalating use of application-layer encryption, virtualised networks deployment for high-quality services and the latest perception adjustments in the user privacy prompt QoE inspection solutions to adapt to fast-changing sceneries presented such as UAV-CPSs.

10.1.1 Airborne network from a CPS perspective A UAV-CPS contains an airborne network (AN) with intense interaction between its physical (flight paths, manoeuvring and multimode resources like GSs, ground nodes and control stations) and cyber (computation, communication and networking) modules (Figure 10.2). The ultimate challenge for ANs is to promote a trusted synergistic interaction among cyber and physical components, with FNs, network topology reconfiguration and secure aerial multimedia data sharing significantly increase UAVs’ situational awareness, and improve the UAV-CPS safety.

Ground station level2

Operator Commands

High level control

Data status

Multi-UAV control and coordination

Modem

Ground station

Acknowledgement

Commands

n n Controllers

Radio-controlled network

n UAVs Modem

Location attitude

Commands n Autopilots Commands

Control Sensor data

Actuators

Mechanics

Sensors

n Sensors and Actuators Level 1: UAV

(a)

(b)

Figure 10.2 General UAV-CPS network architecture: (a) complete system, where each block bounded by dashed lines can be physically at a different location and consuming different distributed resources; (b) communication between the GS and a UAV [2]

218

Imaging and sensing for unmanned aircraft systems, volume 2

Figure 10.3 An airborne network entailing terrestrial, satellite and RF links to the GS control and other UAVs [14] Figure 10.3 exemplifies a network consisting of few aerial and ground nodes. Node altitudes and distances between the FNs are expected to be highly variable. The AN nodes may be static or moving, leading to extremely dynamic AN topological changes. The project of high-performance ANs must embrace factors such as integrity and timeliness of shared information, high bandwidth provision and scalability [14]. This chapter is organised as follows: Definitions are given in Section 10.2. Applications of UAV-CPSs appear Section 10.3. Case studies targeting UAV-CPSs are presented in Section 10.4. Finally, future trends, discussions and conclusions are presented in Sections 10.5 and 10.6.

10.2 Definitions QoS helps managing packet losses, delays and jitter on the network infrastructure [5,15]. Service providers (SPs) use differentiated services as well as integrated services to deliver QoS to the user besides assessing and ensuring quality, but this does not ratify the service-level agreement (SLA) between the SPs and customers [6,16] though both methods flexibilise access to bandwidth quality and delay well. QoS is a significant concern in the UAV-CPS communications with central data centre because they are real-time environments to monitor flood, conflicts, fire and object tracking in remote locations to name a few. These hard conditions affect broadcast over wireless access networks (WANs). The QoE of multimedia streaming relies on network impairments and the way the UAV-CPS handles video content. Figure 10.4 displays the end-to-end multimedia QoE

Quality of experience (QoE) and quality of service (QoS) in UAV systems

219

UMTS access network

Sender

Receiver IP core network Depacketizer

Encoder Packetizer

Decoder

Raw video

Degraded video Content classifier WLAN access network

Figure 10.4 End-to-end multimedia QoE

QoS parameters

Encoder related

Content type

SBR, FR

CT

Access network related PER/BLER, MBL

LBW

Figure 10.5 QoS parameters in a UAV-CPS

rationale over WANs [17]. Multimedia data are encoded and packetised at the sender side, whereas the receiver depacketises and decodes data. The QoS of a UAV-CPS amounts to the overall performance of the network both from the access and the core Internet protocol (IP) network. QoE relies on the measured QoS of the UAV-CPS, e.g., in a real-time video broadcast of the application scenario as observed by the user. The QoS of the video contents is established on different elements such as video format (size), entities or information within the video sequence, frame rate and bitrate.

10.2.1 Parameters that impact QoS/QoE Figure 10.5 summarises the parameters impacting the QoS of UAV-CPSs. The QoS parameters are application level, access network level in addition to the content type. The access to network-level parameters comprises packet loss, router waiting and queue delays, multipath routing packet rearrangement, link bandwidth and mean burst length among others [18–20]. The application-level parameters encompass the video codec, bitrate, frame rate and data rate to deliver quality information to users. Moreover, the UAV-CPS video content also impacts the QoS depending on (i) the camera used and (ii) the access network for transmission. Most FNs have excellent quality HD cameras to

220

Imaging and sensing for unmanned aircraft systems, volume 2

capture multimedia. HD multimedia transmission entails high bandwidth network and low traffic, but if network resources are not available, delays and multimedia data losses (packets losses) may happen.

10.2.2 Impact of cloud distance on QoS/QoE Cloud distance is also a paramount issue for the QoS/QoE of the UAV-CPS due to the long distance between FNs and cloud data centres add extra network delays in data broadcast [21]. In short distance cloud and UAV, communication has less delay because several routers and connecting interfaces are low compared to the long distances between the cloud and the FNs where a large number of routers and interfaces of different SPs add extra delays [16]. Organisations favour UAV high-quality video and images, but due to narrow network bandwidths and long distances between cloud components and FNs, they may not have high-quality video, which degrades the QoE [22]. Accessing a UAV-CPS directly like the serverclient system is different, but if the UAV-CPS is cloud controlled, then the impact is different because request goes to cloud management software then it will be distributed in internal racks and clusters, which also add internal cloud delays in data reception and command forwarding to FNs. Increasing network distances between the UAV-CPS and the cloud services disturb startup delays and waiting times until the service rearranges to provide the sought QoE [23,24].

10.2.3 QoS/QoE monitoring framework in UAV-CPSs The UAV-CPSs’ QoS/QoE monitoring frameworks are based on objective QoE, which comprises (i) objective technical factors to infer QoE from available QoS data and (ii) objective human factors related to the human physiological and cognitive system [6]. QoS/QoE monitoring frameworks employ objective QoE based on the technical QoS data to appraise the QoS performance and better manage operations. Figure 10.6(a) and (b) illustrates the end-to-end framework of evaluating the QoS and QoE of multimedia from a UAV-CPS non-intrusively. The video is digitised, encoded/compressed and sent over to the access and core IP networks. The video is depacketized, decompressed/decoded and reconstructed at the receiver. The QoS is obtained either objectively from metrics like VQM, PSNR, etc., or subjectively via the mean opinion score (MOS) of the received video as perceived by a human. These data assist in developing nonintrusive multimedia models to measure QoE. The MOS ratings appear in Table 10.1. Objective QoE monitoring relies on agent technology. Applied functionalities delivered by the simple network management protocol (SNMP) [22,25] help to recover the QoS data. This protocol helps to collect information from and to configure network devices, such as servers, hubs, switches, printers and routers on an IP network. The SNMP employs agents to retrieve QoS network data like routing knowledge from the cloud to the UAV-CPS, the number of packets in and out in addition to the number of network interfaces. SIGAR [26] is used for low-level system info, e.g., total used memory actual free memory, CPU utilisation and specific facts, in other words, memory and CPU spent by a process [27]. Figure 10.7 presents a proposed framework

Quality of experience (QoE) and quality of service (QoS) in UAV systems

Video sequences

Set of objective measures

Subjective quality assessment (MOS)

(for training of the model)

Network

Subjective (MOS)

RMSE Predicted (MOS) Video quality prediction model

(a) MOS-objective Reference video

Encoder

Packetizer

Video quality measurement (PSNR/MOS)

Access and core network

Degraded video

Depacketizer

Subjective (MOS)

Decoder

Raw video

Degraded video

QoE-based prediction model

Content classifier

(b)

Figure 10.6 (a) Block diagram of the video quality prediction model and (b) end-to-end multimedia QoE/QoS framework

Table 10.1 Mean opinion score MOS

Quality

Perception

5 4 3 2 1

Excellent Good Fair Poor Bad

Imperceptible Perceptible Slightly annoying Annoying Very annoying

221

222

Imaging and sensing for unmanned aircraft systems, volume 2 Satellite Satellite dish UAV system

Internet

Cloud DB QoS/QoE agents Ground control station

Figure 10.7 QoS/QoE monitoring structure in UAV-CPSs to monitor application-level QoS (AQoS) and network-level QoS (NQoS) parameters to appraise QoE from them. For a UAV-CPS, the QoS/QoE supervision software monitors the cloud environment for vacant resources like computation, storage and the state of the internal cloud network. QoS data monitoring from the cloud to the UAVCPS contains the following elements: 1. 2. 3. 4. 5. 6. 7. 8.

distance from cloud to the user; number of routers between them; specific delay on network traffic from the router; network bandwidth; type of network; UAV system capability; overall system usage; memory usage (CPU and memory usage has an enormous influence on cloud usage performance); particular delay on router queue; and information for the administration to comprehend the deficiencies in QoS within UAV-CPSs.

10.2.4 Application-level management The QoS/QoE check-in at the application level aids to analyse the performance of applications while handling them properly. A UAV QoS/QoE monitoring structure also encloses the application-level management, which routinely monitors the network traffic for applications which built on protocols such as applications file transfer protocol, hypertext transmission protocol, remote desktop protocol, real-time protocol,

Quality of experience (QoE) and quality of service (QoS) in UAV systems

223

common Internet file system (CIFS) or SQL and exchange. This unit also monitors the hardware resources utilisation as well as free resources and variety of task the UAVCPS performs now, such as image/video recording and streaming.

10.2.5 Network-level management The network-level control of the QoS/QoE framework for a UAV-CPS monitors QoS network knowledge for both sending and receiving data between the cloud data centre and UAV-CPS parts. Agents run from a control centre to the UAV-CPS and cloud storage to collect data about the link capacity between the UAV-CPS and cloud. Agents also measure the overall network traffic, amount of data sent and gathered by the UAV-CPS as well as on cloud interfaces. These collected data are stored and used for the analysis of the network to find and analyse parameters like the error rate, packet loss, delays and reordering, which help the proper network management.

10.2.6 Cloud distance management The cloud databases connected to the UAV-CPSs can store real-time video and imageries. The cloud distance impacts the QoS/QoE of a UAV-CPS since the long cloud distance inserts an extra delay to send and get data. Internal cloud communication between the racks and clusters for data storage and retrieval also introduces delays due to the internal network congested traffic [28–31]. It is preferable to avoid these issues to have a nearby cloud storage location in QoS/QoE structures of UAV-CPSs. The best choice would be a cloud data centre near to the GS, where the UAV can be controlled or used with the cloud for information retrieval to improve the service of the UAV-CPS.

10.2.7 QoS/QoE service-level management The QoS/QoE service-level management monitors and manages the QoS of an entity via key performance indicators. Traditionally, the service-level management utilises traditional surveillance tools like Microsoft SMS, but UAV-CPS networks suffer because of compositional, dynamic and flexible and services remotely accessed. QoS/QoE monitoring framework of the UAV-CPS uses agents to evaluate resources such as network bandwidth, peak-level utilisation at different hours and error rate. Service-level management will compare the actual performance with pre-defined expectations, to define appropriate actions and yield meaningful reports [26].

10.2.8 QoS/QoE metrics in UAV-CPSs The subsequent QoS/QoE metrics have been defined for UAV-CPS communication so that the QoS/QoE framework can measure and report by the UAV operator about the activation of the UAV parts automatically. ●

Throughput: The broad-range measure of the amount of data per flow in a network. It tells how many bits of data get in a network system in a given time, how a CPU treats the volume of data transferred via memory, the operating

224

●

●

Imaging and sensing for unmanned aircraft systems, volume 2 system performance and ad hoc or mobile networks as capacity grows with the network size or decreases with limitations of radio spectrum. Network packet loss and delay: These are network traffic parameters disturbing the transfer of data from the cloud to the UAV-CPS when a packet loss happens (i.e., when a packet is destroyed, then it will never be recovered). If packets with input action information are lost, then the operator will lose control of the FN, thus affecting its performance. Likewise, information packets that arrive after a delay to the UAV-CPS result in input packets sent late to the cloud, which cause unconventional monitoring. Resources metrics: These metrics enclose logs and reports from the UAV-CPS hardware resources such as overall system capability, memory usage and CPU utilisation for each task, available resources and administration of resources. Checking resources will help to manage and maintain the performance of UAV-CPSs.

10.2.9 Mapping of QoS to QoE The QoS to QoE mapping function appears on the monitoring system, AQoS and NQoS, and it employs the cloud distance. In the QoS/QoE UAV monitoring framework, QoE relies on the agent’s captured technical QoS data, which encompasses knowledge about the video, multimodal image, network and cloud distance as well as data storage for visual retrieval and analysis. If the results of collected data possess long network delays or missing packet in video streaming, insufficient onboard UAV memory or low processing power, then it takes time to process instructions to/from the operator, which decreases the QoS/QoE. These phenomena lessen the QoE of the UAV-CPS, degrade performance for video/image recording and streaming to the control centre. The QoS parameters can be mapped either objectively or subjectively to the QoE.

10.2.10 Subjective vs objective measurement Subjective QoE metrics utilise questionnaires, interviews, web surveys and complaint boxes [32] and refer to measures about what people say and experience. Furthermore, they are expensive, time-wasting and less accurate than objective QoE metrics [33]. Occasionally, the subject is blind or unconscious, thus debilitating the exact evaluation of the experience of service and giving the wrong information. The greedy nature of users to get more favour from SP cited in SLA also incites negative feedback, which is also a subjective QoE assessment caveat [34]. Objective QoE metrics rely on the technical QoS data and physiological tests of people on how subjects complete a task, irrespective of what they feel while doing them [35]. Objective QoE assessment provides more accurate data than the subjective ones because objective methods acquire data via agent-based software, and log reports without the real-time participation of subjects to obtain their feedback [36]. There is no consensus about the usage of objective or subjective performance metrics to decide whether something is knowingly perceived. Subjective methods are prone to criterion fluctuations. For example, someone may be much more

Quality of experience (QoE) and quality of service (QoS) in UAV systems

225

inclined to respond; they did not notice anything depending on the person’s surroundings. This can be partially solved, employing signal detection theory to resolve the number of hits and false alarms. However, much of it also hinges on the question formulation, and how the respondent is stimulated to react. A yes/no type of question seems to naturally tap more into a subjective experience, whereas a twoalternative question forces the choice of more objective performance. Nevertheless, formulating both types of questions in a better way may influence the results.

10.2.11 Tools to measure QoS/QoE Many crowdsourcing frameworks are used to collect online QoS/QoE tools given by industry and reference models provided by the researchers to assess the QoS/ QoE. Simple manual methods are used for subjective QoE such as interviews, questionnaire and complaint boxes to measure QoE of users and collected data analysed using MS Excel and Gephi tools [37,38]. Advanced automatic crowdsourcing tools are also developed to capture QoS/QoE in the runtime environment and data analysis. Crowdsourcing is an emerging technique that can be employed to measure the QoE at the end-user but in an uncontrolled environment. Crowdsourcing frameworks provided to collect QoS/QoE of image and video by Sajjad et al. [39] that measures the QoE of online video streaming, as perceived by end-users. The tool also measures important QoS network parameters in real-time (packet loss, delay, jitter and throughput), retrieve system information (memory, processing power, etc.), and other properties of the end user’s system. Related crowdsourcing frameworks also provided by [40–44] for image and video QoS/QoE assessment. Objective QoE tools are given by vendors to measure the QoS/QoE of multimedia streaming by capturing technical QoS data [45,46]. Casas et al. [47] provided an objective QoS/QoE measuring model based on the machine learning, which is capable of predicting the QoE experienced by the end-user of popular smartphone apps (e.g., YouTube and Facebook), using as input the passive in-device measurements. Objective QoS/QoE tools are also included in crowdsourcing frameworks to capture QoS technical data [6,33,41] automatically. The reference models are used by researchers to measure the QoS/QoE in realtime environments such as no-reference model [21,26], reduced-reference model [48] and full-reference model [49]. Many crowdsourcing tools are used to collect online QoS/QoE set by the industry in addition to reference models provided by the investigators to assess the QoS/QoE. Simple manual ways measure subjective QoE of users and acquired data can be analysed with MS Excel and Gephi tools [37,38]. Crowdsourcing is an emergent way to infer the QoE of online video streaming at the end-user in an uncontrolled scenario. This tool also assesses important QoS network parameters (e.g., delays, packet loss, jitter and throughput) in real-time, retrieves UAV-CPS information (such as memory and computational power), as well as other properties of the end user’s location [40–44]. Advanced automatic crowdsourcing tools can also capture QoS/QoE in the runtime environment and while performing real-time data analysis.

226

Imaging and sensing for unmanned aircraft systems, volume 2 Transmission Uncompressed video

channel Encoder

Decoder

normalisation

Compressed video

Objective metrics

Video quality estimation

Figure 10.8 Nonintrusive (aka no-reference) video quality measurement Sellers provide objective QoE tools for multimedia streaming with technical QoS data [45,46]. Objective QoS/QoE assessing models may rely on machine learning to predict the QoE apparent to the end-users of mobile apps (e.g., YouTube, Twitter and Facebook) when using them as inputs for the passive in-device measurements [47]. Objective QoS/QoE tools built-in in crowdsourcing contexts are used to capture QoS technical data [6,33,41] automatically. Researchers employ the reference models to measure the QoS/QoE in realtime environments using such as no-reference [21,26], reduced-reference [48] and full-reference models [49]. 1.

2. 3.

The no-reference model does not know the original stream or source file while tries to predict the QoE by checking several QoS parameters in real-time. Figure 10.8 exhibits the no-reference video quality measurement rationale. The reduced-reference model employs limited knowledge about the original stream and attempts to combine this with real-time readings to get QoE predictions. The full-reference model presumes full access to the reference video, perhaps combined with real-time environment measurements.

10.3 Applications 10.3.1 Social networks, gaming and human–machine interfaces The integration of UAV-CPSs, social networks (SNs) and entertainment can produce groundbreaking impact. For example, UAV-CPS video games augment the visual perception of the cyber world with more inputs from different types of sensors that improve users’ participation with SN information sharing. Although there are privacy concerns, allowing the users to adjust their privacy settings can be accomplished. The research work from [4,50–52] discusses distributed gaming in a 3D tele-immersive environment using a conceptual framework for modelling nontechnical impacts on user experience. Social interaction affects the QoE of graphical interface operators in the UAV-CPS environment and may use metrics like delay and visual quality, along with nontechnical factors such as age.

Quality of experience (QoE) and quality of service (QoS) in UAV systems

227

10.3.2 Data centres A data centre (DC) can be modelled as a CPS. The cyber portion is the online applications and services in charge of communication, image/video retrieval and computation. The physical mechanisms allow correct and nonstop operation. The interaction, balancing and time-varying nature of cyber and physical modules complicate power management. Rao et al. [53] present methods for managing data centre power consumption by exploiting ON/OFF and dynamic voltage and frequency scaling server features. DCs must maximise the benefits from the provided QoS from computational resources while keeping the energy costs for computation and cooling minimal [54,55]. DC models can associate data usage with the system architecture to investigate interactions among the cyber and physical subsystems, the flow and distribution of computational tasks, and how the physical network handles energy.

10.3.3 Electric power grid and energy systems The UAV-CPSs require new degrees of freedom to enable reliable energy-efficient solutions where power delivery follows the smart grid paradigm. The demand subsystems must use green WANs for multimedia streaming in reaction to the severe growing demand in multimedia service in WANs. Power control intended for streaming multiple variable bit rate videos in wireless networks can function both in a centralised and as a low-complexity distributed fashion to optimally schedule the broadcast power for the bitstream to deliver VBRs to mobile users and GSs. Modules must be inserted consistently with the network constraints and devoid of buffer underflow or overflow subjected to wireless channel uncertainty while keeping the adequate QoE requirements [56–59]. A flocking-theory model for communication routing strategies in wide-area monitoring systems appears in [60–63]. For the case of denial-of-service attacks on communication infrastructure, a model provides effective routing strategies to stabilise the transient of faulty power subsystems [55,64,65].

10.3.4 Networking systems In numerous UAV-CPSs, cameras capture, send and receive high-resolution real-time still images and videos requiring ubiquitous broadband network access. Xing et al. [66–68] present a network architecture for supporting video communication between smartphones and the Internet hosts in CPSs, which is useful for several UAV-CPSs that need video-based data acquisition and sharing. The work from [69–71] presents a technique that can help to minimise the networks’ energy consumption for real-time applications in UAV-CPSs by exploring the opportunities to save energy while complying with the timing and precedence limitations.

10.3.5 Surveillance The design of an Internet-connected alarm for a UAV-CPS appears in [72–74] using a mobile communication network with qualitative and quantitative descriptions

228

Imaging and sensing for unmanned aircraft systems, volume 2

on the UAV-CPS resilience. The alarm interconnects interface terminals and operators thru equipment that senses the real world to transmit these data to real-world control and management. Resilience denotes a 3S-oriented design as follows: 1.

2. 3.

Stability means the UAV-CPS can attain a stable sensing-actuation performance with closed-loop control notwithstanding the contamination by noise or disturbance by attacks of the sensors’ inputs. Security entails the ability of the system to overcome/recover from attacks. Systematicness involves the existence of a seamless combination of sensors and actuators.

10.4 Case studies 10.4.1 Application scenario 1: UAV-CPSs in traffic congestion management The UAV-CPSs must also handle traffic congestion supervision via real-time traffic monitoring [75,76]. The Federal Aviation Administration (FAA) and the National Aeronautics and Space Administration (NASA) help the betterment of unmanned aircraft system traffic management (UTM), which controls aerial traffic congestion and avoids collision of the UAV-CPS with FNs. The cloud-based UTM structure will support the operator in air traffic collision avoidance for all UAVs operated beyond the low-altitude visual line of sight [77]. NASA has many partners that arrange for UAVs and other subsystems to test UTM technologies. Meanwhile, NASA is liable for airworthiness, range and flight safety with its Memorandum of Agreement to conduct UTM tests with its partners. The FAA tests operations in certain kinds of remote airspace [78]. The UTM handles (i) information on wind and congestion to UAV operators and the database to avoid collision with objects and edifices, (ii) evidence about all objects and buildings in the surroundings, (iii) suggestions about safe locations to fly safely and (iv) critical situation alerts on whether to fly over restricted areas such as airports and heavy traffic routes or during other critical operations. These arrangements will assist the user to choose adequate paths, as Figure 10.9 depicts. The UTM has four builds for different risk-based situations: the first is for safely flying and landing in unpopulated regions. The second build controls the safety of flights or operations in low-populated zones. The third form provides limited contact with human-crewed aircraft, while the fourth build is for missions in urban situations. Each build facilitates certain kinds of missions, offers specific services and supports the missions or services from the previous build. There are numerous time-critical problems with sensor-actuator networks within multiprotocol environments, e.g., real-time monitoring, real-time communication and decision-making. Multi-protocols can have different latencies. Jitter variations can disrupt the real-time functionalities, especially during an incident detection in a communication operation. An open challenge that entails good thought is cross-platform sensor-actuator communication. Sensor failures influence

Quality of experience (QoE) and quality of service (QoS) in UAV systems

229

Figure 10.9 UTM

the entire UAV-CPS significantly because of its effects on other sensors and actuators. They call for sensor strategies to keep redundancy and robustness. A massive number of sensors employing different protocol standards for robustness or security can decrease energy efficiency significantly [3,79]. Since different standards show dissimilar energy efficiencies, then energy-efficient models in a multiprotocol setting are paramount. Guaranteeing energy efficiency with scalability is another distinctive challenge since different standards have heterogeneous communication ranges with a different number of intelligent connections. Therefore, the overall framework scalability is hard to find. Moreover, security, trust and information privacy are other emerging issues, which require the careful investigation of the security malfunctions through different protocols in addition to interprotocol communication. A unified framework for a UAV-CPS needs endto-end QoS for all the three kinds of components, i.e., cyber, physical and communication components. Nonorthogonal QoS protocols may lead to pathological interactions sometimes [80] thanks to the cyber and physical entities’ interfaces. Consequently, the physical and cyber objects demand novel QoS protocols equally. A web-of-things (WoT) structure for UAV-CPSs [1] can afford end-to-end QoS for all UAV-CPS, cyber, communication and physical elements with five layers: WoT API, WoT context, WoT overlay, WoT kernel together with WoT device. These layers help to build applications that effortlessly connect cyber and physical elements with the necessary QoS [1]. An FN interconnects various physical devices over the WoT device layer. A UAV handles three WoT layers: overlay, kernel and device. The WoT kernel layer handles QoS requests at the FN, using an intelligent scheduler for prioritising communication, event handling besides tasks.

230

Imaging and sensing for unmanned aircraft systems, volume 2

The UAV-CPS UCN (i) facilitates communication amongst FNs, (ii) rules cyber components’ QoSs, (iii) extracts passive context from event streams, (iv) performs smart control employing the active context and (v) blends in with existing decision-making units and other computational functions. The UAV-CPS UCN implements three WoT layers: API, context and overlay.

10.4.1.1

QoS management at the FN

The kernel scheduler can allocate computational resources to threads (events) with divergent priorities to fulfil end-to-end QoS necessities. The scalable pre-emptive scheduling will distribute any incomplete jobs among computing units via a code mobility policy [8] for dynamically altering the bindings between code fragments and their execution places. The WoT kernel can also detect and classify freshly connected/disconnected physical mechanisms along with their related resources, which is fundamental for self-configuration and plug-and-play. The WoT kernel will disclose to the WoT overlay layer and the corresponding cyber objects the resource specifications.

10.4.1.2

QoS management at the UAV-CPS UCN

The UAV-CPS UCN and the WoT overlay layer offer an application-driven and a network-aware logical abstraction upon the Internet infrastructure in progress to cope with network instability in data, latency, jitter and bandwidth, among others. Opting for superior and predictable performance by the network resources guarantees these functions. The UAV-CPS UCN manages QoS demands at the application (cyber) and communication levels. As a UAV-CPS requires QoS for each of its components, novel QoS functionalities for physical and cyber objects will bring about a replacement or adjustments of present TCP/IP at big scales, which is unrealistic in today’s pervasive Internet. Hence, the WoT overlay layer forms an applicationaware virtual network on top of the existent IP network given the QoS settings. The QoS dealings between the WoT overlay (both cyber and communication subsystems) and WoT kernel (physical systems) need an interface mapping from applicationspecific QoS necessities to network QoS parameters maintained by WoT overlay. Previous research about overlay networks targeted smarter best effort routing [81] with data loss warranty and recovery [82], the WoT overlay sets to (i) deliver dynamic models decouple the virtual path end-to-end delay from the fluctuations in queuing delays on each underlying IP router and (ii) arrange for multi-routing, admission regulation, with on-network caching models. The UAV-CPS UCN deploys QoS with the subsequent constituents: ●

●

●

●

Overlay router: It utilises multiple path delivery to guarantee on-time distribution for time-critical RESTful messages. Delay analyser: It employs admission control to safeguard that end-to-end delays satisfy the CPS QoS requirements. Net connect: It packs/unpacks RESTful messages in primary networks while providing network-specific interfaces’ implementations stated by the QoS monitor. QoS monitor: It obtains updated conditions for the underlying network via probes.

Quality of experience (QoE) and quality of service (QoS) in UAV systems ●

●

231

QoS config: It assigns priority label tags to RESTful messages in line with application-specific requirements. QoS-based scheduler: It plans incoming RESTful messages over a priority queue with priority tags.

10.4.2 Application scenario 2: congestion and accident avoidance using intelligent vehicle systems This section describes a case study handling QoS and QoE in multi-UAV environments. A UAV-CPS encompasses many in-vehicle UAV nodes (IVUAVNs) [83]. Furthermore, a UAV-CPS network (UCN) can govern numerous sensors and actuators over a WoT device layer [38,84]. The IVUAVN can also connect to a private vehicular controller area network to facilitate the controls of various physical structures in the FN straightforwardly, such as cameras, navigation and other critical modules. The UCN provides a WoT overlay atop of the primary network to make available QoS, context handling and an interface between the operator, UAVs and UCN. The QoS requirements, which include both computational and hardware QoSs, exceed communication in intelligent UAV-CPSs. If any CPS component fails to provide QoS, the whole system may experience problems. For example, if a UAV unexpectedly becomes faulty, the operator would only have a concise time to stop it. If the data sensor captures this event quickly, but the UAV slow down an action sequence, then some wrong manoeuvre may happen. A WoT framework can perform event communications through the UAV-CPS intelligently, bearing in mind inner FN sensor information and external data from other UAV-CPS sources. For example, if an operator wants to move a UAV, the UCN makes decisions by analysing readings directly from the local UAV sensors and other external nodes. The WAVE short message protocol can help to transmit high-priority external events quickly to other FNs [85]. The UAV-CPS will handle the FN event messages by assigning a high priority over the UAV-CPS UCN, followed by a quick decision by the computing unit since it has a high-priority event that may interrupt other components if its priority is more significant than for other tasks. The UAV-CPS will perform an immediate action (e.g., an acoustic warning) if the outcomes show danger to the FN based on the sensor data. When it is safe, the operator can finish a manoeuvre without any guidance from the UAV-CPS. The UAV has to identify and prioritise all events to minimise disruption to the operator. The successful prioritisation of events is vital as it disturbs the QoS handling of all other UAV-CPS modules. An intelligent FN mitigates internal disruptions produced by FNs’ sensors, actuators and other onboard intelligent subsystems. For instance, the on-board sensors will uninterruptedly transmit FN data to the UAV-CPS UCN. This data are prioritised in the corresponding UAV-CPS node and processes with this priority will be queued at the UAV-CPS UCN along with the event importance level. If the FN is operated erratically and with slow response time, the UAV-CPS will infer that the operator is unable to operate the UAV safely. The operator’s behaviour alteration will intensify the event priority, consequently elevating the QoS for the event messaging,

232

Imaging and sensing for unmanned aircraft systems, volume 2

computing and the action performed by the UAV-CPS. In this example, the UAV will query the navigation structure about the nearest safe location to stop and suggest some rest to the operator. The UAV kernel scheduler handles the event information from the sensor nodes so that critical events are processed first. The operator will see an essential event that may provoke an accident for instantaneous action. The UAV will assist the operator if it is possible whenever applicable. As a UAV in operation will get many event notifications, presenting all data to the operator can be confusing. Hence, the operator will receive event messages cleverly to sense high-priority events that can cause accidents and automatically mitigate the event. A solution can be a framework with a scheduler for normal events (safe situations) and another for crucial events that can lead to accidents. High-priority events from in-vehicle sensors will be treated first at the CPS kernel layer, leaving other regular event processes on hold. Then, these high-priority events are sent to the UAV-CPS UCN for additional processing. Its scheduler will interrupt normal processes when it gathers a high-priority event from external sources. The UAV-CPS UCN will also consider user actions to improve decisionmaking further. Once the UAV-CPS UCN concluded, then it will command the FNs via WoT overlay to manage the condition, manually (by notifying the operator of immediate danger), automatically (by decelerating the UAV) or semi-automatically (where the UAV-CPS UCN informs the operator about the danger besides helping to mitigate it). If the UAV cannot escape a collision, the FN sensors will detect the collision point and reduce the UAV speed. This high-priority event will be treated by the sensor nodes, stopping all other processes. The UAV-CPS UCN will enter highpriority mode depending on the collision severity, upsurge the CPS QoS capacity to relief this situation and send commands to other FNs to safeguard the drones and people in its surroundings while asking for assistance.

10.5 Future and open challenges Even with the advances in UAV-CPSs, when it comes to intensive and wide-scale UAV-CPSs’ use, some challenges persist as follows.

10.5.1 Modelling and design These are challenging in UAV-CPSs. A hardware description language or programming language is not enough to model their behaviour. The expected performance of the computational parts for UAV-CPSs has to be stated regarding their effects on the physical environment. Hence, they require a unifying modelling approach with easy interfacing and consistency. Also, real-world data sets are essential for testing and validating novel research ideas. Furthermore, time-based and event-based programming languages must work together to enable the effective modelling of the asynchronous dynamics at different temporal and spatial scales [49].

Quality of experience (QoE) and quality of service (QoS) in UAV systems

233

Concurrency is natural in real-world UAV-CPSs’ applications where communications with sensors and actuators are not fully covered by current programming languages [18,19,31] and need proper abstractions for intuitive real-world modelling. Functionality enables seamless integration of new modules with novel design methods/characteristics to interfere minimally with existing modules while extending the UAV-CPSs’ functionalities. This will create novel interactions between the suppliers and consumers. An extensible infrastructure will support a great variety of sensor and actuator types with easy access to an abundance of potential users. Verification and validation require novel ways for verifying and authenticating UAV-CPS hardware and software modules for high-degree dependability and reconfigurability [18,19,31,86]. Performance and time sensitiveness mean that short execution time with correctness results in high performance. Accomplishing the demands of timesensitive UAV-CPSs entail highly effective abstractions and tools for analysis and synthesis tasks. Security of the mutual coordination and interdependence of the cyber and physical modules increase the susceptibility to faults and attacks. Mission-critical UAV-CPSs, such as medical systems and disaster remediation need real-time, highly safe, trustworthy operation, self-healing frameworks to enable security-state checking using adequate security performance metrics to judge the UAV-CPSs’ security. Mobility introduces further implementation concerns like speed and synchronisation in UAV-CPSs. Distributed computation and network control in UAV-CPSs must be adaptive, robust to failures and preserve an overall situation awareness when there are incomplete distributed information and local actions. Scalability facilitates UAV-CPS deployment in a variety of circumstances. Power optimisation impacts the cost-effectiveness of UAV-CPSs and is a key design constraint in future on-board tasks because increasing computation can influence energy consumption.

10.5.2 Collaborative services The two types of attributes for UAV-CPS services are functional and nonfunctional [48]. As UAV-CPS services grow, more competitive services with comparable functionality appear every day, and nonfunctional characteristics become a critical factor in service selection since they can discriminate the performance of competitor UAV-CPS services with the same or comparable functionality. QoS becomes an effervescent area in current service and distributed computing research [10]. In the real world, the values of some user-dependent QoS/QoE properties (e.g., throughput and response time among others) may differ mostly due to different physical positions, users, network conditions (like 5G and Wi-Fi, for instance) and other objective issues [11]. It is unrealistic for a single operator to try all candidate UAV-CPS services to attain sufficient QoS data for evaluation. Thus, it is

234

Imaging and sensing for unmanned aircraft systems, volume 2

paramount to predict user-dependent QoS values for operational UAV-CPS services recommendation. Collaborative filtering (CF) QoS prediction is widely used, especially neighbourhood CF [7,12–14,28,29,80–82,85,87–91]. The neighbourhood CF-based method characteristically consists of two phases: (i) discovery of similar operators (or services) and analysing their similarity and (ii) figuring unknown QoS values from historical data produced by similar users (or services). Still, there are two main complications in present tactics: 1.

2.

The data are habitually very sparse when computing the similarity of users (or services). The QoS values from users may be very scarce or even null when the QoS matrix is too sparse. In this case, it is challenging to estimate user (or service) similarity precisely, and data sparseness will decrease the accuracy of predicted QoS values; and Not all users are honest, and they will provide some unreliable QoS data. Some people may give random or fixed answers, while others (e.g., SPs) may claim decent QoS figures for their services and offer poor figures to their competitors [7,82].

Most methodologies do not detect and pre-process unreliable QoS figures before performing QoS prediction. Robust UAV-CPS service recommenders require dealing with data from untrusted users. Hence, they demand alleviation of the data sparseness impact, dismissal of the cold start problem [15] and high estimate accuracy.

10.5.3 Streaming The UAVs are rising in popularity, which calls for addressing challenges associated with the FN movement, limited resources as well as elevated error rates, evidencing the need for an adaptive tool to robustify video broadcasts [92]. Adaptive forward error correction (FEC) procedures can enhance the QoE of video sent over networks predisposed to errors and with high freedom of movement. An adaptive video-aware FEC mechanism using motion vectors [87,88,90,91] minutiae to better real-time UAV video broadcasts, delivering both more exceptional user experience and improved usage of resources. The paybacks and disadvantages of such a mechanism can be analysed and tested via simulations and QoE metrics. Real-time video streaming for FANETs can use a cooperative communication manner for real-time streaming where FNs are congregated into cooperative clusters. An LTE system transfers the video data over links to a designated cluster head employing IEEE 802.11p to multicast the received video inside the cluster. Error concealment methods, along with efficient distribution strategies for resources, ameliorate the received video quality significantly improving QoE and QoS in contrast to the noncooperative FN networking situations.

10.5.4 Security The UAV-CPS networks face several threats and can benefit from solutions for wireless systems containing intelligent and connected FNs [93,94]. The security

Quality of experience (QoE) and quality of service (QoS) in UAV systems

235

worries may become more complicated when there is cooperation among the FNs in the presence of malicious behaviour in cooperating nodes. The adjustments between security and QoS in FANETs need studies to optimise cooperative communication. Likewise, a prevention-based security structure may offer integrity protection and authentication for both hop by hop and end to end. An alternative is to employ both security and QoS provisioning similar to the VANETs’ output capacity, bit error rate and effective throughput. A secure incentive system for reliable cooperative downloading and cooperative forwarding can motivate the UAV operators to work together securely downloading-and-forwarding packets [15]. The cooperative downloading can utilise virtual checks connected with the designated verifier’s signature to assure safe and fair cooperation. Further, a reputation system can motivate cooperation, penalise the malicious nodes, encourage packet forwarding and attain reliability. Throughout the cooperative forwarding stage, an aggregating Camenisch– Lysyanskaya signature is utilised to ensure the incentive mechanism security. A UAV and infrastructure-centric metrics can measure FN safety awareness with awareness calculation for critical neighbours, which are a possible accident threat, nearby an FN neighbourhood [27]. The infrastructure nodes can also include the position error of each neighbouring FN while calculating the awareness. The metrics rely on several received cooperative awareness messages (CAMs) and their safety, significance, accuracy and UAV direction. Before each CAM transmission, an elliptic curve digital signature algorithm can be added to the security procedure. The CAM resides on the security FIFO receiver for verification when its turn arrives.

10.5.5 Flying ad hoc networks FANETs address scenarios necessitating quick network deployment without proper infrastructure. FNs can fly arbitrarily with unpredictable network topology modifications. This behaviour calls for the constant route adaptation and reconfiguration to guarantee internode communication, more extensive coverage region, and FANETs self-configuration and self-organisation [95–99]. The mobility and spatial arrangement of FNs are also essential in routing, so these routes are habitually replanned to permit continuous FN interconnection. Hence, routing must be dynamic with valid and straightforward protocols to augment UAVs autonomy and cut the data delivery delays between a sender FN and a destination FN. The FNs’ locations influence their intercommunication largely. A UAV relay node unites all the acquired information and relays the data to a control centre. Hence, its position is strategic to warrant excellent network performance. During UAVs flights, sensors amass information more intensely than in traditional sensor networks. Therefore, they must intercommunicate more efficiently and with fewer hindrances in their line of sight. Lessening the number of FNs covering a specific region is a sure way to fulfil this requirement, although weather conditions can worsen communication (e.g., wind and rain). The computational load of FANETs calls for a higher information transmission capacity when

236

Imaging and sensing for unmanned aircraft systems, volume 2

transmitting real-time visual data. Until now, no specific routing protocols exist for FANETs. Since FNs possess a high mobility rate, updating the location of all FNs is a critical feature. FANET elements can employ a metaheuristic model for a better result and a superior performance. Future needs of FANET scenario may appear by refining the communication structure and attain acceptable and more precise results. Other metaheuristic optimisers require tests to validate the practice and ratify the best execution time algorithm for the mandatory performance like a genetic algorithm [100, 101, 102], particle swarm [100, 101, 102], and Cuckoo search [103] and World Cup [104], among others.. The main challenge of FANET is to execute cooperative detection with multiple UAVs covering a region [96–98]. This feature leads to an overhead network where its FNs exchange data. Thus, reliable and stable communication helps to maintain proper levels of QoS and QoE. Most FANETs bring together data from the surroundings and transmit them to a GS. FANET topology has several FNs and a UAV relay to receive material from the other FNs and transmitting it to a GS. The UAV relay position with reference to the other FNs is paramount to guarantee the adequate network performance accessible straightforwardly or indirectly (i.e., communication through intermediary devices) to all the other network elements, which demands more studies. Accurate analytical models reduce signal interference among the FNs and the base station. These algorithms do uniform frequency hopping inside the channel bandwidth and reach QoS gains in overload or interrupt cells with some computational intelligence [89,99] technique to find an optimised new UAV relay position [98]. The use of several backbones augments the coverage of the monitored region, but more investigations about the proper position of the elements are required to ensure that the network has the best performance. The minimisation of the number of Mobile Base Stations (MBSs) for adequate wireless coverage by a set of distributed ground terminals (GTs) warrants that a single GT is in the communication range of at least one MBS. An efficient algorithm helps to manoeuvre UAVs in a sophisticated, adaptable and flexible manner. FANET task planning [105] with connectivity constraint comprises numerous parameters as well as the interaction of dynamic variables. There exist also algorithms for distributed intelligent agent frameworks where agents autonomously organise, collaborate, negotiate, decide and trigger actions to execute a particular assignment. The connectivity problem is NP-hard. Therefore, a polynomial-time heuristic can help to solve it while maintaining high-performance data transfer by adequately positioning UAVs. The FNs must communicate with each other as routers. One of the core FANET’s performance problems is to position a UAV relay among other relays to expand communication. Hence, telecommunications and computational intelligence can help to find a better location in real-time using the FN Global Positioning System (GPS; which, on average, transmits location data every second) and the FN IMU (which sends location data at a shorter interval than a GPS) at any time. Consequently, the arrangement of the devices thru the covered area challenges this kind of network, and it has a direct impact on its performance and mobility,

Quality of experience (QoE) and quality of service (QoS) in UAV systems

237

either improving or worsening them. Very high sampling rates can circumvent this trade-off because the discrete controllers perform better than continuous controllers despite some caveats. QoS exploration investigates compromises between cyber resource apportionment and the UAV-CPS performance (comprising controllers) for various discrete service intervals in the CPS. In general, allocating more cyber resources expands performance. Traditionally, network control systems try to identify the conditions leading to stability and high performance in a network involving sensing, control and actuation. Optimisation strategies can identify communication and control inputs simultaneously while handling packet loss. Cyber resource allocation can be maximised by event-triggered control while keeping control performance. A few mechanisms exploiting optimal control can create paths and sampling instants, employing a time-varying sampling rate controller and yield successful results with augmented computational complexity.

10.5.6 User emotions As the number of users accessing online video services grows, streaming services tend to dominate all sorts of mobile traffic. Since even small enhancements in the operator’s viewing experience will augment profitability substantially, content providers and distributors, network personnel and service suppliers will have to adjust videos obtained through FANETs. Although there are ongoing efforts to improve recently a user’s video QoE by the better utilising big data to analyse users’ watching behaviours relying on large-scale, video-viewing history benchmark datasets, it is not possible to accurately scrutinise users’ secret intents and emotional states while they are viewing online videos [106]. Hence, in the future, the user’s emotional reactions will be accounted for to obtain a better QoE assessment. In such a framework, the user’s mood will be detected in a real-time via emotion detection environments [107]. Then, a frame of mind matching procedure will be performed to evaluate the similarity of the operator’s intent and the multimedia content in terms of emotion design to characterise the relationship between QoE and several aspects, including average bitrate, buffer ratio and the operator’s emotions.

10.6 Conclusion This chapter examines QoS and the QoE of UAV-CPSs applications. Leveraging the cumulative assortment of connectivity options in networks involving FNs, GSs and other devices helps to solve complications such as capacity, spectrum efficiency, network coverage and reliability. Anytime, matters such as selection, scheduling, handover or routing demand performance metrics, energy efficiency and low-cost access. The rising number of choices in networks complicates designs. QoS and QoE allow finding the best possible network configurations and price for user applications autonomously. Existing lines of attack are listed by function. Restrictions and fortes are highlighted to provide an initial point for further studies in this area. Since the UAV-CPS network design, FN routing, and FN allocation

238

Imaging and sensing for unmanned aircraft systems, volume 2

problems can impact the end-to-end users’ QoS/QoE, system output metrics must be constantly inferred and optimization goals, as well as tasks, will be reevaluated. Additionally, due to the characteristics of these challenges, it is important to take into consideration how metaheuristics methodologies and their hybridisations can be employed to lengthen the power of metaheuristics while unravelling stochastic and dynamic optimisation problems. If the UAV-CPS network has low QoS, then real-time data will not be accurate and subject to information losses resulting in inaccuracies. Information loss or delay (due to packet loss, rearrangement and delay) may decrease the satisfaction level of the UAV-CPS user (operator). Network QoS degradation affects the realtime video monitoring. This effect leads to data loss and image quality reduction, thus decreasing the QoE.

References [1] Estrela V.V., Saotome O., Loschi H.J., et al. Emergency Response CyberPhysical Framework for Landslide Avoidance with Sustainable Electronics, Technologies, 6, 42, 2018. doi: 10.3390/technologies6020042. [2] Kuang L., Yu L., Huang L., et al. A Personalized QoS Prediction Approach for CPS Service Recommendation Based on Reputation and LocationAware Collaborative Filtering, Sensors, 18, 1556, 2018. doi: 10.3390/ s18051556. [3] Benmir A., Korichi A., and Bourouis Alreshoodi M. Survey on QoE/QoS Correlation Models for Video Streaming over Vehicular Ad-hoc Networks, 26(4), 267–287, 2018. doi: 10.20532/cit.2018.1004278. [4] Khaitan S.K., and McCalley J.D. Design Techniques and Applications of Cyber Physical Systems: A Survey, IEEE Systems Journal, 9, 2015, 350–365. [5] http://docwiki.cisco.com/wiki/Quality_of_Service_Networking [6] Recommendation, I. T. U. T. E. 800: Terms and definitions related to quality of service and network performance including dependability. ITU-T August 1994, 1994. [7] ITU-T Report 2007. Definition of Quality of Experience (QoE), International Telecommunication Union, Liaison Statement, Ref.: TD 109rev2 (PLEN/12), Jan. 2007. [8] Fuggetta A., and Picco G.P. Understanding code mobility, IEEE Transactions on Software Engineering, 24(5), 342–361, 1998. [9] Andersen D., Balakrishnan H., Kaashoek F., and Morris R. Resilient Overlay Networks, ACM SIGOPS Operating Systems Review, 35(5), 145, 2001. [10] Subramanian L., Stoica I., Balakrishnan H., and Katz R. OverQoS: an overlay based architecture for enhancing internet QoS, in Proc. 1st Symp. Networked Syst Des and Implem, USA, 2004. [11] David G. Big goals get big results, in Proc. 2006 World Business Council for Sustainable Development (WBCSD) Liaison Delegates Meeting, China, 2006.

Quality of experience (QoE) and quality of service (QoS) in UAV systems

239

[12] Intelligent Transportation Systems Committee, IEEE trial-use standard for wireless access in vehicular environments (wave)-networking services, IEEE Std 1609.3-2007, vol. 2007, pp. c1-87. [13] Uzca´tegui R.A., and Acosta-Marum G. WAVE: A Tutorial, IEEE Communications Magazine, 47(5), 126–133, 2009. [14] Namuduri K., Wan Y., Gomathisankaran M., and Pendse R. Airborne network: a cyber-physical system perspective, in Proc. 1st ACM MobiHoc Workshop on Airborne Networks and Communications (Airborne’12), Hilton Head South Carolina USA, 2012, pp. 55–60. [15] Laghari A.A., and Rehman K.U. On quality of experience (QoE) for multimedia services in communication ecosystem. PhD diss., Institut National des Te´le´communications, Paris, France, 2012. [16] Hoang D.D., Paik H.-Y., and Kim C.-K. Service-oriented middleware architectures for cyber-physical systems, International Journal of Computer Science and Network Security, 12(1), 79–87, 2012. [17] http://insideunmannedsystems.com/nasa-working-to-develop-unmannedaerial-system-traffic-management-or-utm/ [18] Johnson T., and Mitra S. Parametrized verification of distributed cyberphysical systems: an aircraft landing protocol case study, in Proc. 2012 IEEE/ACM Third International Conference on Cyber-Physical Systems (2012 ICCPS), Beijing, China, 2012, pp. 161–170. [19] Lee E. Cyber physical systems: design challenges, in Proc. 2008 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (2008 ISORC), Orlando, FL, USA, 2008, pp. 363–369. [20] Baheti R., and Gill H. Cyber-physical systems, The Impact of Control Technology, 2011, pp. 161–166. [21] Laghari A.A., He H., Shafiq M., and Khan A. Impact of storage of mobile on quality of experience (QoE) at user level accessing cloud, in Proc. 2017 IEEE 9th Int’l Conf. on Comm. Soft. and Networks (ICCSN), GuangZhou, China, 2017. [22] Laghari, K.U.R., and Connelly, K. Toward total quality of experience: A QoE modelling in a communication ecosystem. IEEE Commun. Mag. 50(4), 58–65, 2012. [23] Ferrer A.J., Marques J.M., and Jorba J. Towards the Decentralised Cloud: Survey on Approaches and Challenges for Mobile, Ad hoc, and Edge Computing, ACM Computing Surveys, 51, 111:1–111:36, 2019. [24] Ratasich D., Khalid F., Geissler F., Grosu R., Shafique M., and Bartocci E. A Roadmap Toward the Resilient Internet of Things for Cyber-Physical Systems. IEEE Access, 7, 13260–13283, 2019. [25] Laghari A.A., He H., Shafiq M., and Khan A. Assessing effect of cloud distance on end user’s quality of experience (QoE), In Proc. 2016 2nd IEEE Int’l Conf. Computer and Comm. (ICCC), Chengdu, China, 2016, pp. 500–505. [26] https://support.hyperic.com/display/SIGAR/Home [27] Alhamazani K., Ranjan R., Jayaraman P.P., Mitra K., Wang M., and Huang Z.G. Real-time QoS monitoring for cloud-based big data analytics applications

240

[28]

[29]

[30]

[31]

[32]

[33]

[34]

[35]

[36]

[37]

[38]

[39]

Imaging and sensing for unmanned aircraft systems, volume 2 in mobile environments, in Proc. 2014 IEEE 15th Int’l Conf’ Mobile Data Management, Brisbane, QLD, Sydney, NSW, Australia, 2014, pp. 337–340. Estrela V.V., Khelassi, A., Monteiro A.C.B., et al. Why softwaredefined radio (SDR) matters in healthcare?. Medical Technologies Journal, vol. 3, no. 3, Oct. 2019, pp. 421–9, doi:10.26415/2572-004X-vol3iss3 p421-429 Motlagh N.H., Taleb T., and Osama O., Low-altitude unmanned aerial vehicles-based internet of things services: Comprehensive survey and future perspectives. IEEE Int Things J, 3, p. 899–922, 2016. Estrela V.V., Monteiro A.C.B., Franc¸a R.P., Iano Y., Khelassi A., and Razmjooy N. Health 4.0: Applications, Management, Technologies and Review, Medical Technologies Journal, 2(4), 262–276, 2019. doi: 10.26415/ 2572-004X-vol2iss4p262-276. Cruz B.F., de Assis J.T., Estrela V.V., and Khelassi A. A Compact SIFTBased Strategy for Visual Information Retrieval in Large Image Databases, Medical Technologies Journal, 3(2), 402–412, 2019. doi: 10.26415/2572004X-vol3iss2p402-412. Wang G., Wang C., Chen A., et al. Service level management using QoS monitoring, diagnostics, and adaptation for networked enterprise systems, In Proc. 2005 9th IEEE Int’l EDOC Enterprise Computing Conference, Enschede, Netherlands, 2005, pp. 239–248. Laghari A.A., He H., Zardari S., and Shafiq M. Systematic Analysis of Quality of Experience (QoE) Frameworks for Multimedia Services, International Journal of Computer Science and Network Security, 17(5), 121, 2017. Laghari A.A., Channa M.I., Laghari K.R., Aman M., and Memon M. EQOM: Enhanced Quality of Experience (QoE) Framework for Multimedia Services, UACEE International Journal of Computer Science and its Applications, 3(1), 85–89, 2013. Laghari A.A., He H., Ibrahim M., and Shaikh S. Automatic Network Policy Change on the Basis of Quality of Experience (QoE), Procedia Computer Science, 107, 657–659, 2017. Laghari A.A., Rehman K.U., and Connelly K. Toward Total Quality of Experience: A QoE Model in a Communication Ecosystem, IEEE Communications Magazine, 50(4), 2012. Karim S., Zhang Y., Laghari A.A., and Asif M.R. Real time image processing based drone for detecting and controlling street crimes, in Proc. 17th IEEE Int’l Conf. Comm. Technology, China, 2017. Khan A., Sun L., and Ifeachor E. QoE Prediction Model and its Application in Video Quality Adaptation over UMTS networks, IEEE Transactions on Multimedia, 14(2), 431–442, 2012. Khan A., Sun L., and Ifeachor E.C. Content classification based on objective video quality evaluation for MPEG4 video streaming over wireless networks, Journal of Multimedia, vol. 4, 2009, pp. 228–239.

Quality of experience (QoE) and quality of service (QoS) in UAV systems

241

[40] Mushtaq M.S., Augustin B., and Mellouk A. Crowd-sourcing framework to assess QoE, In Proc. 2014 IEEE Int’l Conf. on Communications (ICC), Sydney, Australia, 2014, pp. 1705–1710. [41] Ribeiro F., Florencio D., Zhang C., and Seltzer M. Crowdmos: an approach for crowdsourcing mean opinion score studies, In Proc. 2011 IEEE Int’l Conf. Acoustics, Speech and Signal Proc. (ICASSP), Prague, Czech Republic, 2011, pp. 2416–2419. [42] Chen K.-T., Wu C.-C., Chang Y.-C., and Lei C-L. A crowdsourceable QoE evaluation framework for multimedia content, In Proc. 17th ACM Int’l Conf. on Multimedia, ACM, New York, NY, USA, 2009, pp. 491–500. [43] Keimel C., Habigt J., Horch C., and Diepold K. Qualitycrowd—a framework for crowd-based quality evaluation, In Proc. Picture Coding Symposium (PCS), 2012, IEEE, Nagoya, Japan, 2012, 2012, pp. 245–248. [44] Rainer B., Waltl M., and Timmerer C. A web based subjective evaluation platform, In Proc. 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), IEEE, Klagenfurt am Worthersee, Germany, 2013, pp. 24–25. [45] Gardlo B., Egger S., Michael Seufert M., and Schatz R. Crowdsourcing 2.0: enhancing execution speed and reliability of web-based QoE testing, In Proc. 2014 IEEE Int’l Conf. Comm. (ICC), Sydney, Australia, 2014, pp. 1070–1075. [46] www.witbe.net [47] www.qoesystems.com [48] Han J. Cyber-Physical Systems with Multi-Unmanned Aerial Vehicle-Based Cooperative Source Seeking and Contour Mapping. Ph.D. thesis, Utah State University, 2014. [49] Horvath I., and Gerritsen B. Cyber-physical systems: Concepts, technologies and implementation principles, in Tools and Methods of Competitive Engineering Symposium (TMCE), 2012, pp. 19–36. [50] Desai K., Raghuraman S., Jin R., and Prabhakaran B. QoE Studies on Interactive 3D Tele-Immersion, 2017 IEEE International Symposium on Multimedia (ISM), Taichung, Taiwan, 2017, pp. 130–137. [51] Ning Z., Hu X., Chen Z., et al. A Cooperative Quality-Aware Service Access System for Social Internet of Vehicles, IEEE Internet of Things Journal, 5, 2506–2517, 2018. [52] Chen H., Xiong P., Gavrilovska A., Schwan K., and Xu C. A cyber physical integrated system for application performance and energy management in data centers, IGCC, 2012, pp. 1–10. [53] Rao L., Liu X., Xie L., and Liu W. Minimizing electricity cost: optimization of distributed internet data centers in a multi-electricity-market environment, Proc. 2010 IEEE INFOCOM, San Diego, CA, USA, 2010, pp. 1–9. [54] Deylamsalehi A., Cui Y.H., Afsharlar P., and Vokkarane V. Minimizing Electricity Cost and Emissions in Optical Data Center Networks, IEEE/OSA Journal of Optical Communications and Networking, 9, 257–274, 2017.

242 [55]

[56]

[57]

[58]

[59]

[60]

[61]

[62]

[63]

[64] [65] [66]

[67]

[68]

[69]

Imaging and sensing for unmanned aircraft systems, volume 2 Parolini L., Tolia N.H., Sinopoli B., and Krogh B.H. A cyber-physical systems approach to energy management in data centers, in Proc. 2010 ICCPS, Stockholm, Sweden, 2010, pp. 168–177. Akkaya I., Liu Y., and Gorton I. Modeling and analysis of middleware design for streaming power grid applications, in Proc. Industrial Track of the 13th ACM/IFIP/USENIX Int’l Middleware Conf., Montreal, Quebec, Canada, 2012. Cao J. and Li H. Energy-efficient structuralized clustering for sensor based cyber physical systems, in Proc. UIC-ATC. IEEE, Brisbane, QLD, Australia, 2009, pp. 234–239. Li H., Lai L., and Poor H.V. Multicast Routing for Decentralized Control of Cyber Physical Systems With An Application in Smart Grid, IEEE Journal on Selected Areas in Communications, 30, 1097–1107, 2012. Lu A., and Yang G. Input-to-State Stabilizing Control for Cyber-Physical Systems With Multiple Transmission Channels Under Denial of Service. IEEE Transactions on Automatic Control, 63, 1813–1820, 2018. Wei J., and Kundur D. GOAliE: Goal-Seeking Obstacle and Collision Evasion for Resilient Multicast Routing in Smart Grid, IEEE Transactions on Smart Grid, 7, 567–579, 2016. Kateb R., Akaber P., Tushar M.H.K., Albarakati A., Debbabi M., and Assi C.M. Enhancing WAMS Communication Network Against Delay Attacks, IEEE Transactions on Smart Grid, 10, 2738–2751, 2019. Wei J., and Kundur D. A flocking-based model for DoS-resilient communication routing in smart grid, Proc. 2012 IEEE Global Communications Conference (GLOBECOM), Anaheim, CA, USA, 2012, pp. 3519–3524. Huseinovic A., Mrdovic S., Bicakci K., and Uludag S. A taxonomy of the emerging denial-of-service attacks in the smart grid and countermeasures, In Proc. 2018 26th Telecommunications Forum (TELFOR), Belgrade, Serbia, 2018, pp. 1–4. Sridhar S., Hahn A., and Manimaran G. Cyber–Physical System Security for the Electric Power Grid, Proceedings of the IEEE, 100(1), 210–224, 2012. Hong J., Liu C., and Govindarasu M. Integrated Anomaly Detection for Cyber Security of the Substations, IEEE Transactions on Smart Grid, 2014. Guoliang X., Jia W., Du Y., Tso P., Sha M., and Liu X. Toward ubiquitous video-based cyber-physical systems, in Proc. IEEE Intl. Conf. on Systems, Man and Cybernetics, Suntec, Singapore, 2008, pp. 48–53. Leitao P., Colombo A.W., and Karnouskos S. Industrial Automation Based on Cyber-Physical Systems Technologies: Prototype Implementations and Challenges, Computers in Industry, 81, 11–25, 2016. ISSN 0166-3615. Zhang Y., Qiu M., Tsai C., Hassan M.M., and Alamri A. Health-CPS: Healthcare Cyber-Physical System Assisted by Cloud and Big Data, IEEE Systems Journal, 11, 88–95, 2017. Ahmadi H., Abdelzaher T.F., and Gupta I. Congestion control for spatiotemporal data in cyber-physical systems, in Proc. 2010 ICCPS, Stockholm, Sweden, 2010, pp. 89–98.

Quality of experience (QoE) and quality of service (QoS) in UAV systems

243

[70] Xue C.J., Xing G., Yuan Z., Shao Z., and Sha E.H. Joint sleep scheduling and mode assignment in wireless cyber-physical systems, in ICDCS Workshops, Montreal, Quebec, Canada, 2009, pp. 1–6. [71] Scheuermann C. A Metamodel for Cyber-Physical Systems, Technischen Universita¨t Mu¨nchen, DSc dissertation, Munich, Germany, 2017. [72] Ma L., Yuan T., Xia F., Xu M., Yao J.P., and Shao M. A high-confidence cyber-physical alarm system: design and implementation, in Proc. Intl. Conf. on Green Comp and Comm and Intl. Conf. on Cyber, Physical and Social Computing, 2010, pp. 516–520. [73] Hu F., Lu Y., Vasilakos A.V., et al. Robust Cyber-Physical Systems: Concept, models, and implementation, Future Generation Computer Systems, 56, 449–475, 2016. [74] Li W., Meng W., Sakurai K., and Kwok L.F. Towards False Alarm Reduction Using Fuzzy If-Then Rules for Medical Cyber Physical Systems. IEEE Access, 6, 6530–6539, 2018. [75] Laghari A.A., He H., Karim S., Shah H.A., and Karn N.K. Quality of Experience Assessment of Video Quality in Social Clouds, Wireless Communications and Mobile Computing, 2017, 2017. [76] Liu Y., Cheng S., and Hsueh Y. eNB Selection for Machine Type Communications Using Reinforcement Learning Based Markov Decision Process, IEEE Transactions on Vehicular Technology, 66, 11330–11338, 2017. [77] http://www.interdrone.com/news/researchers-to-test-a-drone-traffic-managementsystem [78] https://www.faa.gov/uas/research/utm/ [79] Alnasser A., Sun H., Jiang, J. Cyber Security Challenges and Solutions for V2X Communications: A Survey, Computer Networks, 151, 52–67, 2019. [80] Healthcare Global, 2013, Remote health monitoring. [Online]. Available: http://www.healthcareglobal.com/sectors/medical-devices-products/3g-remotehealth-monitoring-facilitated-sprint-and-bl-healthcare [81] Le Callet P., Mo¨ller S., and Perkis A. Qualinet white paper on definitions of quality of experience. European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003) 3, 2012. [82] Laghari A.A., Sadhayo I.H., and Channa M.I. Enhanced Autonomic Networking Management Architecture (ENAMA), Engineering, Science and Technology, 9, 2015, 1–5. [83] Dillon T.S., Zhuge H., Wu C., Singh J., and Chang E. Web-of-things Framework for Cyber–Physical Systems, Concurrency and Computation: Practice and Experience, 2010. doi: 10.1002/cpe.1629. [84] Singh J. Multicast QoS routing using collaborative path exploration, in Proc. 24th IEEE Int’l Conf. Advanced Information Networking and Applications (AINA), Australia, 2010, pp. 120–125. [85] Laghari A.A., Rehman K.U., and Connelly K. Toward Total Quality of Experience: A QoE Model in a Communication Ecosystem, IEEE Communications Magazine, 50(4), 58–65, 2012.

244 [86]

[87]

[88]

[89]

[90]

[91]

[92]

[93]

[94]

[95]

[96]

[97]

Imaging and sensing for unmanned aircraft systems, volume 2 Rehman Z.U., Altaf S., and Iqbal S. Survey of authentication schemes for health monitoring: a subset of cyber physical system, In Proc.16th Int’l Bhurban Conf. Applied Sc. and Techn. (IBCAST), Islamabad, Pakistan, 2019, pp. 653–660. Coelho A.M., Estrela V.V., Carmo F.P., and Fernandes S.R. Error concealment by means of motion refinement and regularized Bregman divergence, in Proc. 13th Int’l Conf. Intelligent Data Eng. and Automated Learning, Natal, RN, Brazil, 2012. Fernandes S.R., Estrela V.V., Magalhaes H.A., and Saotome O. On improving sub-pixel accuracy by means of B-Spline, in Proc. 2014 IEEE Int’l Conf. Imaging Systems and Techniques (IST 2014), Santorini, Greece, pp. 68–72, ISBN: 9781479952199. Hemanth J., and Estrela V.V. Deep learning for image processing applications. Adv. Par. Comp. IOS Press, Netherlands. ISBN978-1-61499-821-1 (print) 978-1-61499-822-8 (online) 2017. de Jesus M.A., and Estrela V.V. Optical Flow Estimation Using Total Least Squares Variants, Oriental Journal of Computer Science and Technology, 10, 563–579, 2017. doi: 10.13005/ojcst/10.03.03. Coelho A.M., and Estrela V.V. State-of-the-Art Motion Estimation in the Context of 3D TV, Ed.: Reuben A. Farrugia, Carl J. Debono. (Ed.). Multimedia Networking and Coding. 1ed. Malta: IGI Global, Hershey, PA, USA, 2013, pp. 148–173, ISBN: 1466626607. Ahmed E., and Gharavi H. Secure Cooperative Communication, IEEE Transactions on Intelligent Transportation Systems, 19(3), 996–1014, 2018. doi:10.1109/TITS.2018.2795381. Javed M.A., and Hamida E.B. On the Interrelation of Security, QoS, and Safety in Cooperative ITS, IEEE Transactions on Intelligent Transportation Systems, 18(7), 1943–1957, 2017. Yaacoub E., Filali F., and Abu-Dayya A. QoE Enhancement of SVC Video Streaming Over Vehicular Networks Using Cooperative LTE/802.11p Communications, IEEE Journal of Selected Topics in Signal Processing, 9(1), 37–49, 2015. Jailton J., Carvalho T., Arau´jo J., and Frances R. Relay Positioning Strategy for Traffic Data Collection of Multiple UAVs Using Hybrid Optimization Systems: A FANET-Based Case Study, Wireless Communications and Mobile Computing, 2017. doi: 10.1155/2017/2865482. Sahingoz, O.K. Networking Models in Flying Ad-hoc Networks (FANETs): Concepts and Challenges, Journal of Intelligent & Robotic Systems, 74(1–2), 513–527, 2014. Marconato E.A., Maxa J.A., Pigatto D.F., Pinto A.S.R., Larrieu N., and Branco K. R. L. J. C. IEEE 802.11n vs. IEEE 802.15.4: a study on communication QoS to provide safe FANETs, in Proc. 46th IEEE/IFIP Int’l Conf. on Dependable Systems and Networks, DSN-W, Toulouse, France, 2016, pp. 184–191, 2016.

Quality of experience (QoE) and quality of service (QoS) in UAV systems [98]

[99] [100]

[101]

[102]

[103]

[104]

[105]

[106]

[107] [108]

245

Rohde S., Putzke M., and Wietfeld C., Ad hoc Self-healing of OFDMA Networks Using UAV-based Relays, Ad Hoc Networks, 11(7), 1893–1906, 2013. Tong W., Hussain A., Bo W.X., and Maharjan S. Artificial Intelligence for Vehicle-to-Everything: A Survey, IEEE Access, 7, 10823–10843, 2019. de Jesus M.A., Estrela V.V., Saotome O., and Stutz D. Super-resolution via particle swarm optimization variants. In: Hemanth J., and Balas V. (Eds) Biologically Rationalized Computing Techniques For Image Processing Applications. Lecture Notes in Computational Vision and Biomechanics, vol 25. Springer, Cham, Zurich, Switzerland, 2018. DOI: 10.1007/978-3319-61316-1_14 Razmjooy N, and Estrela V.V. Applications of Image Processing and Soft Computing Systems in Agriculture, IGI Global, Hershey, PA, USA, 2019. DOI: 10.4018/978-1-5225-8027-0. Razmjooy N., Estrela V.V., and Loschi H.J. A Study on MetaheuristicBased Neural Networks for Image Segmentation Purposes, Data Science Theory, Analysis and Applications, Taylor and Francis, Abingdon, UK, 2019. Agrawal J., Singhal A., and Yadav R.N. Multipath routing in mobile Adhoc network using meta-heuristic approach. In Proc 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Karnataka, India, 1399–140, 2017. Razmjooy N., Ramezani M., and Estrela V.V. A solution for Dubins path problem with uncertainties using world cup optimization and Chebyshev polynomials, in: Y. Iano et al. (Eds), Proc. 4th Brazilian Techn. Symp. (BTSym’18). BTSym 2018. Smart Innovation, Systems and Technologies, Campinas, SP, Brazil, vol 140. Springer, Cham, Zurich, Switzerland, 2019. DOI: 10.1007/978-3-030-16053-1_5 Gaggero M., Paola D.D., Petitti A., and Caviglione L. When Time Matters: Predictive Mission Planning in Cyber-Physical Scenarios, IEEE Access, 7, 11246–11257, 2019. Atat R., Liu L., Wu J., Li G., Ye C., and Yang Y. Big Data Meet CyberPhysical Systems: A Panoramic Survey, IEEE Access, 6, 73603–73636, 2018. Sztipanovits J. Model integration and cyber physical systems: A semantics perspective, in Proc. Intl. Conf. on Formal methods, 2011. Singh J., Veeraraghavan P., and Singh S. QoS Multicast Routing Using Explore Best Path, Computer Communications, 29(15), 2881–2894, 2006.

This page intentionally left blank

Chapter 11

Conclusions Vania V. Estrela1, Jude Hemanth2, Osamu Saotome3, George Nikolakopoulos4, and Roberto Sabatini5

The current awareness in unmanned aerial vehicles (UAVs) has prompted not only military applications but also civilian uses. Aerial vehicles’ requirements aspire to guarantee a higher level of safety comparable to see-and-avoid conditions for piloted aeroplanes. The process of probing obstacles in the path of a vehicle and determining whether they pose a threat, alongside measures to avoid these issues, is known as see and avoid or sense and avoid. Other types of decision-making tasks can be accomplished using computer vision and sensor integration since they have a great potential to improve the performance of the UAVs. Macroscopically, UAVs are cyber-physical systems (CPSs) that can benefit from all types of sensing frameworks, despite severe design constraints, such as precision, reliable communication, distributed processing capabilities and data management. This book is paying attention to several issues that are still under discussions in the field of UAV-CPSs. Thus, several trends and needs are discussed to foster criticism from the readers and to provide further food for thought. Some of the significant UAV-CPSs advantages are their capability to withstand the duties and objectives of human beings while accomplishing chores for them. Simultaneously, they also make some decisions and execute some actions independently. Therefore, people and machines must collaborate. Even though these capabilities offer noteworthy paybacks, there is still a tremendous amount of effort necessary to fully master suitable ways of assisting human–machine interaction. Using low-cost, open-source components together with multiple sensors and actuators is quite a challenge in terms of effort and cost. Hence, it is desirable to use open-source software (OSS) and hardware (OSH) in UAV-CPSs. Whenever possible, available OSH and OSS should be used in designs independently of the type of hardware framework and its operating system. The performance of an orthogonal frequency division multiplexing (OFDM) UAV-CPS can be enhanced by adding channel coding (i.e., error correction code) 1

Telecommunications Department, Universidade Federal Fluminense (UFF), Niteroi, RJ, Brazil Department of Electrical and Computer Engineering, Karunya Institute of Technology and Sciences, Coimbatore, India 3 Instituto Tecnologico de Aeronautica (ITA), DCTA-ITA-IEEA, Sao Jose dos Campos, SP, Brazil 4 Lulea˚ University of Technology (LTU), Department of Computer Science, Electrical and Space Engineering (SRT), Lulea˚, Sweden 5 RMIT University, School of Aerospace, Mechanical and Manufacturing Engineering, Melbourne, Australia 2

248

Imaging and sensing for unmanned aircraft systems, volume 2

to identify and correct the errors that happen throughout data transmission. The massive bandwidth of ultra-wideband (UWB) channels can originate new effects compared to conventional wireless channel modelling in several micro aerial vehicle (MAV) applications. UWB technologies based on IEEE 802.15.4a have various uses when there is a need for highly accurate localisation with various sensors for stabilisation and navigation. Still absolute indoor positioning challenges UAVs. Flying a UAV in unstructured settings with changeable conditions is challenging. To support the development of better algorithms, a multipurpose data set for low-altitude UAV flights in a given Brazilian environment is proposed as a benchmark for positioning and other avionics tasks for assessing computer vision procedures in terms of robustness and generalisation with a baseline for depth estimation with and without landmarks. This stage of development can help the advancement of future integration with remote sensing (RS) modules that will bring in more spectral information to the analyses. UAV-CPSs involve a great deal of knowledge on networking—more specifically— on flying ad-hoc networks. The fact that the traffic of high-dimensional multimedia data streams through UAV-CPSs tends to grow exponentially, raising several issues and points towards future research directions. The texture is an important feature to recognise objects or regions of interest (ROIs) in any images, and it has been commonly used for image classification from satellite images to assess biomass, for instance. UAV imagery takes advantage of ultrahigh spatial resolution, which shows that texture is also a paramount source of knowledge. Nevertheless, texture in the UAV imagery was seldom used in surveillance. Moreover, merging ground hyperspectral data could compensate for the limited bands of UAV sensors and increase the estimation precision of analyses. Consequently, the goals of this chapter are to (i) explore UAV-based multispectral imagery and (ii) improve several types of estimation accuracy through hyperspectral information. The application of camera-equipped UAVs for visually observing ecological balance, construction and operation of hydric bodies, buildings, bridges, forest reservations and other types of infrastructure arrangements has grown exponentially. These UAV-CPSs can frequently examine all sorts of sites, monitor work in progress, generate documents/reports for safety and scrutinise the existing structure, chiefly for hard-to-reach regions. Regardless of the sophistication of the on-board sensors, the cloud, RS, computational intelligence and communication advances, super-resolution will be in demand for quite a long time. This will continue to happen in contexts where acquiring imageries is expensive and troublesome like in healthcare, astronomy and disaster relief. Both the quality of service (QoS) and quality of experience (QoE) (besides other qualitative performance metrics) will take a pivotal role to impulse further improvements in all stages of a UAV-CPS. This book intends to be a reference for current and forthcoming applications of UAV-CPSs. It will display fundamental aspects, ongoing research efforts, accomplishments, and challenges faced when it comes to the deployment of imaging capabilities and sensor integration in UAVs. Succinctly, this book addresses the challenges, role, technical issues, and applications of computer vision/image processing frameworks, plus sensors in UAV design. The book chapters help the reader to focus on the most critical factors that relate a subject to computer vision and sensing.

Index

Ada 41 adaptability 117 adaptive neuro-fuzzy inference system (ANFIS) controller 83 ADRSR 194 aerial image database framework 94 database design 94 database requirements 94 aerial imaging and reconstruction of infrastructures by UAVs 157 data-set collection 164–7 experimental results 168 indoor scenario 168 outdoor scenario 168–70 underground scenario 171 future trends 171–3 related studies 158–60 3D reconstruction 162 monocular mapping 163–4 stereo mapping 162–3 visual sensors and mission planner 160 image projection 160–1 path planner 161–2 AERIALTOUR CAMPUS UNIVAP 92–3, 98, 100, 103–4, 106 aeroplane gaming (AG) 68 airborne network from CPS perspective 217–18 Airborne Surveillance Systems (ASSs) 122–3 air-to-ground (AG) UAV communications 68 Amazon 107 Amazon Web Services (AWS) 62 Apache Software Foundation (ASF) 63

application-level management 222–3 application-level QoS (AQoS) 222 application programming interface (API) 58 arbitrary super-resolution rates 198 architectural design 41 Ascending Technologies NEO (AscTec NEO) hexacopter 164 Augmented Reality (AR) 3 applications 3 Autodesk ReCap 360 168 Bayesian filter 163 Beyond VLOS (BVLOS) 59 big data (BD) 1, 150, 186–8 bilateral total variation (BTV) prior 192 blind IQA methods 191 blurring 191 Bregman algorithm 192 British Ministry of Defence Architecture Framework (MODAF) 41–2 C 41 C++ 41 call centres 8 Camenisch–Lysyanskaya signature 235 Canon Hack Development Kit (CHDK) 95 Canon Power-Shot S110 camera 95 Central Processing Unit (CPU) 58 clay tiles mask 97 Cloud Computing (CC) 1–2, 5–7, 172, 188

250

Imaging and sensing for unmanned aircraft systems, volume 2

cloud data centre 62–3 cloud distance, impact of on QoS/QoE 220 cloud distance management 223 cognitive task analysis 39 collaborative filtering (CF) QoS prediction 234 Collective UAV Learning (CUL) 1, 7–8 Common Cause Analysis (CCA) 45 common Internet file system (CIFS) 223 communication infrastructure 137 communications requirements, video streaming, communications links and networked UAVs 113 compact video 69 component-level design 41 Computational Intelligence (CI) 2 computer vision (CV) algorithms 91–2, 113 development of 103–5 Contrast Transfer Function (CTF) 96 control flow analysis 44 Convolution Neural Networks (CNN) 178, 183 cooperative awareness messages (CAMs) 235 coupled deep autoencoder (CDA) 193 critical task analysis (CTA) 39–40 cross-paced partial curriculum learning (CPPCL) framework 194 crowd-sourced data in UAV-CPSs 63 crowdsourcing 2, 8, 225 Cruise Control by LinkedIn 63 curriculum learning 194 CVSRnet 190 cyber-physical system (CPS) 113–14, 134, 150, 247 dark channel prior (DCP) method 182–3 DARPA Agent Markup Language 150 DARPA Robotics Challenge (DRC) 8

Data Access Points (DAPs) 127 Data Acquisition Module (DAM) 52 data centres (DCs) 227 data design 41 data-linking technology 68 data processing toolkits for spatial data 144 data-set collection 164 data set 166–7 experimental setup 164–6 data use analysis 44 deep distillation recursive network (DDRN) 195 deep learning (DL) 3 deep learning (DL) as an alternative to super-resolution imaging 177 critical issues big data 186–8 cloud computing services 188 deep learning and computational intelligence 197–8 efficient metrics and other evaluation strategies 190–1 image acquisition hardware limitations 188–9 multiple priors 191 network design 198–9 novel architectures 193–5 regularisation 192–3 3D SR 195–6 video SR 189–90 experiments and results 185 peak signal-to-noise ratio 186 super-resolution model 178 dehazing 182–3 motion estimation 181–2 patch selection 183 super-resolution 183–5 Department of Defence Architecture Framework (DODAF) 41–2 depth map super-resolution 195–6 digital surface models (DSMs) 58 digital terrain models (DTM) 58 dimensionality reduction (DR) 143 direct sequence (DS) method 70

Index direct sequence UWB (DS-UWB) methods 70–1 Discrete Cosine Transform (DCT) 182 diversity gain 76 DNA-PAINT (Point Accumulation for Imaging in Nanoscale Topography) 196 Doppler’s effect 143 DRPNN 189 ecoquadcopter 10 edge-adaptive RMF 192 electric power grid and energy systems 227 electromagnetic spectrum 135 Elasticsearch 63 Energy Access Points (EAPs) 127 Energy Harvesting (EH) 126 enhanced deep SR (EDSR) network 191 enhanced perceptual SR (EPSR) network 191 event detection and video quality selection algorithms 122–3 Extended Kalman Filter (EKF) 83 FAST 190 Federal Aviation Administration (FAA) 228 field of view (FOV) sensor 161 First-Person View (FPV) mode 57 flight computer 137 Flight Control Module (FCM) 52 flocking-theory model 227 Flying Ad-Hoc Networks (FANETs) protocol 113–15, 215–16, 234–7 adaptability 117 advantages 115 bandwidth requirement 118 future trends data collection and presentation 124 data gathering versus energy harvesting 126–8

251

event detection and video quality selection algorithms 122–3 Flying Nodes (FNs) coordination 123–4 Flying Nodes’ (FNs) placement search algorithms 121 Network Function Virtualisation (NFV) 125–6 onboard video management (UAV) 123 Software-Defined Networking (SDN) 124–5 video-rate adaptation for the fleet platform 123 latency 118 scalability 117–18 streaming and surveillance 119–20 UAV platform constraints 118 Flying Base Station (FBS) 126 Flying Nodes (FNs) 113–14 coordination 123–4 placement search algorithms 121 forward error correction (FEC) procedures 234 FOXEER Box sensor 166 Functional Hazard Assessment (FHA) 45 future scope for UAV 138 gaming 226 generative adversarial network (GAN) framework 191 genetic algorithm (GA) 143 geographical information system (GIS) 142 GitHub platform 61 Global Navigation Satellite System (GNSS) 69, 92 global positioning system (GPS) 157, 159, 177 global-scale data mining 188 Google Earth Engine 188 Google Skybox project 199 Graphical Processing Unit (GPU) 58 graphical user interface (GUI) 58

252

Imaging and sensing for unmanned aircraft systems, volume 2

Grid Computing (GC) 2 Ground Control (GC) 24–25 Ground Control Centre (GCC) 25 Ground Control Station (GCS) 2, 24, 51, 57 Ground Control Unit (GCU) 24 ground terminals (GTs) 236 ground truth (GT) limitations 143 ground truth masks 96 H.264/AVC standard 69, 83 Heterogeneous Airborne Reconnaissance Team (HART) GCS 24 hierarchal task analysis 38–9 High-Definition Image Transmission System (HDIT) 68 High-Dimensional High-Resolution (HDHR) 6 High-Performance Computing (HPC) applications 5 Huber Markov random field (Huber-RMF) 192 human computation 8 Human Factors Engineering (HFE) program 23–4, 29 human–machine collaboration 1 human–machine interface (HMI) 23, 226 future work 44–5 GC Station (GCS) HMI elements 29–33 human factors program (HFE) 33–6 cognitive task analysis 39 critical task analysis (CTA) 39–40 design evaluation 42–3 hierarchal task analysis 38–9 operational sequence diagram 40 requirements definition, capture and refinement 36–8 systems design and development 41–2 task analysis 38

verification and validation (V&V) 43–4 UAS Ground Control (GC) segment 24 UAS HMI functionalities 26–7 mission planning and management 28 multi-platform coordination 28–9 reconfigurable displays 28 sense and avoid (SAA) 28 Hybrid Access Point (HAP) 128 Hyperion imaging spectrometer 140 hyperspectral image (HSI) 133–4, 140, 143, 189 see also multispectral imaging (MSI) and hyperspectral image (HSI) UAV imaging image acquisition hardware limitations 188–9 image capture process 95–8 image database of low-altitude UAV flights 91 aerial image database framework 94 future works 106–7 image capture process 95–8 image processing system for UAVs 92–4 images collected 98–100 use of the image database 100 development of CV algorithms 103–5 mosaics 100–3 image processing system for UAVs 92–4 image quality assessment (IQA) metrics 185 independent component analysis 143 Inertial Measurement Unit (IMU) 52, 83, 159 Inertial Navigation System (INS) 83 information flow analysis 44 Infrastructure as a Service (IaaS) 11 Integrated Development Environments (IDE) 42

Index Intel Core i5 machine 185 intelligent vehicle systems congestion and accident avoidance using (application scenario) 231–2 interface analysis 44 interface design 41 Internet of Things (IoT) 2, 150 Internet protocol (IP) network 219 in-vehicle UAV nodes (IVUAVNs) 231 iterative closest point (ICP) algorithm 162 Java 41 Kaziranga National Park (KNP), Assam 147–8 latency 118 LIDAR cloud points 147, 159 Lightweight Kit 96 Li’s method 193 low-resolution (LR)-to-high-resolution (HR) mappings 189 LPIPS 197 machine learning (ML) 188 Machine Type Communication (MTC) 127 MATLAB2012 185 MAVLink protocol 55 Maximum-Likelihood (ML) decoding 71 mean opinion score (MOS) 190, 220 mean square error (MSE) 76, 186 MFSR techniques 178, 189, 193 micro aerial vehicles (MAVs) 49, 157–8 micro UAV technology 69 Minas Gerais 93 miniature aerial robotics 157 MinimOSD 59–60 minimum noise fraction (MNF) 143 Ministry of Defence Architecture Framework (MODAF) 41–2

253

mission payload controller 137 Mjolkuddsberget mountain 167 Mobile Ad-Hoc Network (MANETs) 113–14 Mobile Base Stations (MBSs) 127, 236 Model-based Systems/Software Engineering 42 MODIS products 188 modus vivendi and operandi 107 monocular mapping 163–4 mosaics 100–3 motion JPEG (MJPEG) 123 motion vectors (MVs) 83 motion vectors extrapolation (MVE) 83 multi-frame (MF) based SR approaches 178, 189 Multimedia and High-Dimensional Data (MHDD) 11 Multimedia-based Cross-Layer (MCL) design 12 Multimedia Cross-Layer Optimisation (MCLO) 11 multimedia wireless communication 69 Multi-Packet Reception (MPR) 116 multiple description coding (MDC) 67, 72–4 multiple-input–multiple-output (MIMO) system 67, 71, 74–6, 127 Multipoint Relay (MPR) 116 multi-population genetic algorithm (MGA) 143 multi-scale purification unit (MSPU) 195 multispectral imaging (MSI) 133–4, 138 high-resolution imaging 139 low-resolution imaging 139 multispectral vs hyperspectral imaging for unmanned aerial vehicles 133 applications of MSI and HSI UAV imaging 147 agriculture monitoring 147 coastal monitoring 147

254

Imaging and sensing for unmanned aircraft systems, volume 2

commercial uses 148 defence applications 148 environmental monitoring 148 forestry 147–8 urban planning 148 data processing toolkits for spatial data 144 future scope 148–50 hyperspectral imaging 140 multispectral imaging 138 high-resolution imaging 139 low-resolution imaging 139 satellite imaging vs UAV imaging 140–1 UAV image processing workflow 141 atmospheric correction 142 computational tasks 143 dimensionality reduction (DR) 143 spectral influence mapping 142–3 UAV imaging architecture and components 136 future scope for UAV 138 UAV open data sets for research 144–7 multi-view stereo (MVS) technique 157 Nakagami-m distribution 71 National Aeronautics and Space Administration (NASA) 228 NATO Architecture Framework (NAF) 42 Navigation Module (NM) 52 Navigation Systems (NSs) 52 Near Infrared (NIR) 53 Network Function Virtualisation (NFV) 125–6 networking systems 227 network-level management 223 network-level QoS (NQoS) 222 Node-ODM 58 Nonorthogonal QoS protocols 229 normalised difference vegetation index (NDVI) processing cameras 138 NVidia processor 54

Objective QoE metrics 224 Object Recognition Server (ORS) 4–5 OctNet 194 onboard video management (UAV) 123 On-Screen Display (OSD) 59 Open Drone Map (ODM) ecosystem 58 open hardware MAV advanced system design 54 open-source and open-access resources 8–9 Open Source Data (OSD) 62 open-source hardware (OSH) 1, 9, 49–50, 247 and software-based UAS 53 open source platform 61 open-source software (OSS) 1, 8–9, 50–1, 247 open source software (OSS) and hardware (OSH) in UAVs 49–50 future work 61 cloud data centre 62–3 control of UAV swarms 63 crowd-sourced data in UAVCPSs 63 OHS challenges 61–2 open data 62 Ground Control Station (GCS) software 57 open source platform 61 open source UAS 51–5 operator information and communication 59–60 processing software 57–8 universal messaging protocol 55–6 Open Source Tool (OST) 49 open source UAS 51 Open Source Unmanned Aerial System (OSUAS) 52 OpenTX 60 Operating System (OS) 56 operational sequence diagram 40 operator information and communication 59–60 Optical Flow (OF) technique 122

Index Optimized Link State Routing (OLSR) protocol 116 OptrisR PI450 G7 96 Orthogonal Space–Time Block Codes (OSTBC) 67, 71, 76, 84 path analysis 44 peak signal to noise ratio (PSNR) 76, 79, 180, 185–7 photogrammetry software 57 PieAPP 197 Pix4D 100–1 Platform as a Service (PaaS) 11 power UWB system 70 Preliminary System Safety Assessment (PSSA) 45 proprietary software (PS) 49–50 ProSR 194 Pulse Position Modulation (PPM) 70 Python 41 Qgis 101 quality of experience (QoE) 113, 216, 218–19 quality of experience (QoE) and quality of service (QoS) in UAV systems 215 airborne network from a CPS perspective 217–18 application-level management 222–3 applications data centres (DCs) 227 electric power grid and energy systems 227 networking systems 227 social networks, gaming and human–machine interfaces 226 surveillance 227–8 cloud distance management 223 congestion and accident avoidance using intelligent vehicle systems (application scenario) 231–2

255

definitions 218 future and open challenges 232 collaborative services 233–4 flying ad hoc networks 235–7 modelling and design 232–3 security 234–5 streaming 234 user emotions 237 impact of cloud distance on 220 mapping of QoS to QoE 224 network-level management 223 parameters that impact 219–20 QoS/QoE metrics in UAV-CPSs 223–4 QoS/QoE monitoring framework in UAV-CPSs 220–2 QoS/QoE service-level management 223 subjective vs objective measurement 224–5 tools to measure QoS/QoE 225–6 UAV-CPSs in traffic congestion management (application scenario) 228 QoS management at the FN 230 QoS management at the UAV-CPS UCN 230–1 quality of service (QoS) 113, 216, 218–19 Radio Control (RC) transmitters 60 Rayleigh/Ricean distributions 71 real-time aerial traffic surveillance system 119 ReCap 360 168 recurrent neural networks (RNNs) 188 Redshift effect 142 regions of interest (ROIs) 248 remote-sensing (RS) technologies 2, 49–50, 133, 248 RGB-D sensors 158 Rikola hyperspectral data set 147 Robot Operating System (ROS) 2, 55 RTABMap SLAM 162

256

Imaging and sensing for unmanned aircraft systems, volume 2

satellite imaging vs UAV imaging 140–1 scalability 117–18 self-paced learning 194 sense and avoid (SAA) 28 service-level agreement (SLA) 218 service providers (SPs) 218 SIGAR 220 signal-to-noise ratio 143 simple network management protocol (SNMP) 220 Simple Object Access Protocol 148 simulations results 77–81 Simultaneous Localisation and Mapping (SLAM) 5, 92 single description coding (SDC) 67, 72 single-frame (SF) SR approaches 178, 189 single-input, single-output (SISO) system 75 single-objective selective-plane illumination microscopy (soSPIM) 196 singular value decomposition (SVD) 164 SISR 178, 180, 193–4 sketch-based image retrieval (SBIR) 193 Sobel filter 182 social networks (SNs) 226 Software as a Service (SaaS) 11 Software-Defined Networking (SDN) 124–5 Software-Defined Radio 150 sparse coding 193 spatial data, data processing toolkits for 144 spatial diversity systems 76 spectral reflectance (SR) graph 142 SRGAN 195 Statistical Machine Learning (ML) 2 stereo mapping 162–3 structural similarity (SSIM) 185–7, 185, 190 structure from motion (SfM) technique 157, 159, 163–4

subjective vs objective measurement 224–5 sum of absolute errors (SAEs) 181–2 super-resolution (SR) model 178 dehazing 182–3 motion estimation 181–2 patch selection 183 super-resolution 183–5 super-resolution (SR) reconstruction 106 surveillance 227–8 System Safety Assessment (SSA) 45 Systems Modeling Language (SysML) 42 task analysis 38 Tata Consultancy Services (TCS) 147–8 TensorFlow 63, 188 The Open Group Architecture Framework (TOGAF) 41 Thermal Infrared (TIR) 53 1080p HD video (HDTV) 68 3D reconstruction 162 monocular mapping 163–4 stereo mapping 162–3 3D super-resolution 195–6 Tikhonov prior 192 time-hopping (TH) methods 70 time-hopping UWB (TH-UWB) technique 70 total variation (TV) model 192 UAV-CPS network (UCN) 231 UAV cyber-physical systems (UAVCPSs) 51, 178 UAV image processing workflow 141 atmospheric correction 142 computational tasks 143 dimensionality reduction (DR) 143 spectral influence mapping 142–3 UAV Small-Cells’ (USCs) networks 127 UGVs 159–60 ultra-dense residual blocks (UDBs) 195

Index ultra-wideband (UWB) channels 247 ultra-wideband (UWB) communication 69 UWB-based localisation workflow 70 Ultra Wideband (UWB) transceiver 12 Unified Modeling Language (UML) 42 UNIVAP Urbanova Campus 94 universal messaging protocol 55–6 unmanned aerial RS facility (UARSF) 147 Unmanned Aerial Vehicle (UAV) 23, 49, 113 unmanned aerial vehicle cyberphysical systems (UAV-CPSs) 1, 215–16, 226, 229 challenges and future directions 10–12 cloud computing 5–7 collective UAV learning 7–8 human computation, crowdsourcing and call centres 8 open-source and open-access resources 8–9 unmanned aircraft system traffic management (UTM) 228 Vehicular Ad-hoc Networks (VANETs) 113 verification and validation (V&V) 43–4 VESPCN 190

257

VICON Motion-capture (Mo-cap) system 168 video-rate adaptation for fleet platform 123 video SR 189–90 video streams 119 Visual Flight Rules (VFR) 92 visual-inertial (VI) navigation framework 159 visual information fidelity (VIFP) 186–7 Visual Line of Sight (VLOS) operator 24, 59 Visual Sensor Networks (VSNs) 11 visual sensors and mission planner 160 image projection 160–1 path planner 161–2 VSRnet 190 WebODM 58 web-of-things (WoT) structure for UAV-CPSs 229, 231 Web Service Definition Language (WSDL) 149 Wilderness Search and Rescue (WiSAR) 120 wind turbine site 166 wireless communication 69 Wireless Multimedia VSNs (WMVSNs) 12 wireless visual area networks (WVAN) 83 ZigBee 12