Advances in Aerial Sensing and Imaging 9781394174690

Table of contents :
Cover
Title Page
Copyright Page
Contents
Preface
Chapter 1 A Systematic Study on Aerial Images of Various Domains: Competences, Applications, and Futuristic Scope
1.1 Introduction
1.2 Literature Work
1.2.1 Based on Camera Axis
1.2.2 Based on Scale
1.2.3 Based on Sensor
1.3 Challenges of Object Detection and Classification in Aerial Images
1.4 Applications of Aerial Imaging in Various Domains
1.5 Conclusions and Future Scope
1.5.1 Conclusions
1.5.2 Future Scope
References
Chapter 2 Oriental Method to Predict Land Cover and Land Usage Using Keras with VGG16 for Image Recognition
2.1 Introduction
2.2 Literature Review
2.3 Materials and Methods
2.3.1 Dataset
2.3.2 Model Implemented
2.4 Discussion
2.5 Result Analysis
2.6 Conclusion
References
Chapter 3 Aerial Imaging Rescue and Integrated System for Road Monitoring Based on AI/ML
3.1 Introduction
3.2 Related Work
3.3 Number of Accidents, Fatalities, and Injuries: 2016–2022
3.3.1 Accidents Statistics in India
3.3.2 Accidents Statistics in Haryana
3.4 Proposed Methodology
3.4.1 ROI and Line Selection
3.4.2 Motion Detection
3.4.3 Single-Stage Clustering
3.4.4 Feature Fusion Process
3.4.5 Second-Stage Clustering
3.4.6 Tracking Objects
3.4.7 Classification
3.5 Result Analysis
3.6 Conclusion
References
Chapter 4 A Machine Learning Approach for Poverty Estimation Using Aerial Images
4.1 Introduction
4.2 Background and Literature Review
4.3 Proposed Methodology
4.3.1 Data Acquisition
4.3.2 Pre-Processing
4.3.3 Feature Extraction
4.3.4 Data Integration
4.3.5 Model Development
4.3.6 Validation
4.3.7 Visualization and Analysis
4.3.8 Policy and Program Development
4.4 Result and Discussion
4.5 Conclusion and Future Scope
References
Chapter 5 Agriculture and the Use of Unmanned Aerial Vehicles (UAVs): Current Practices and Prospects
5.1 Introduction
5.2 UAVs Classification
5.2.1 Comparison of Various UAVs
5.3 Agricultural Use of UAVs
5.4 UAVs in Livestock Farming
5.5 Challenges
5.6 Conclusion
References
Chapter 6 An Introduction to Deep Learning-Based Object Recognition and Tracking for Enabling Defense Applications
6.1 Introduction
6.2 Related Work
6.2.1 Importance of Object Monitoring and Surveillance in Defense
6.2.2 Need for Object Monitoring and Surveillance in Defense
6.2.3 Object Detection Techniques
6.2.4 Object Tracking Techniques
6.3 Experimental Methods
6.3.1 Experimental Setup and Dataset
6.3.2 DataSetVISdrone 2019
6.3.3 Experimental Setup
6.4 Results and Outcomes
6.4.1 Comparison Results
6.4.2 Training Results
6.5 Conclusion
6.6 Future Scope
References
Chapter 7 A Robust Machine Learning Model for Forest Fire Detection Using Drone Images
7.1 Introduction
7.2 Literature Review
7.3 Proposed Methodology
7.4 Result and Discussion
7.5 Conclusion and Future Scope
References
Chapter 8 Semantic Segmentation of Aerial Images Using Pixel Wise Segmentation
8.1 Introduction
8.2 Related Work
8.3 Proposed Method
8.3.1 Pixelwise Classification Method
8.3.2 Morphological Processing
8.4 Datasets
8.5 Results and Discussion
8.5.1 Analysis of the Proposed Method
8.6 Conclusion
References
Chapter 9 Implementation Analysis of Ransomware and Unmanned Aerial Vehicle Attacks: Mitigation Methods and UAV Security Recommendations
9.1 Introduction
9.2 Types of Ransomwares
9.3 History of Ransomware
9.4 Notable Ransomware Strains and Their Impact
9.4.1 CryptoLocker (2013)
9.4.2 CryptoWall (2014)
9.4.3 TeslaCrypt (2015)
9.4.4 Locky (2016)
9.4.5 WannaCry (2017)
9.4.6 NotPetya (2017)
9.4.7 Ryuk (2018)
9.4.8 REvil (2019)
9.4.9 Present-Day Ransomware Families
9.5 Mitigation Methods for Ransomware Attacks
9.6 Cybersecurity in UAVs (Unmanned Aerial Vehicles)
9.6.1 Introduction on FANETS
9.6.2 Network Security Concerning FANETs
9.6.3 UAV Security Enhancement
9.6.4 Limitations in UAVs
9.6.5 Future Scope
9.7 Experimental analysis of Wi-Fi Attack on Ryze Tello UAVs
9.7.1 Introduction
9.7.2 Methodology
9.8 Results and Discussion
9.9 Conclusion and Future Scope
References
Chapter 10 A Framework for Detection of Overall Emotional Score of an Event from the Images Captured by a Drone
10.1 Introduction
10.1.1 Need for Emotion Recognition
10.1.2 Applications of Drones in Deep Learning
10.2 Literature Review
10.3 Proposed Work
10.3.1 Extraction of Images from a Drone
10.3.2 Proposed CNN Model
10.4 Experimentation and Results
10.4.1 Dataset Description
10.5 Future Work and Conclusion
References
Chapter 11 Drone-Assisted Image Forgery Detection Using Generative Adversarial Net-Based Module
11.1 Introduction
11.2 Literature Survey
11.3 Proposed System
11.3.1 Common Forged Feature Network
11.3.2 Features Extraction
11.3.3 Features Classification and Classification Network
11.3.4 Label Prediction
11.3.5 Contrastive Learning
11.3.6 Binary Cross-Entropy Loss
11.4 Results
11.4.1 Experimental Settings
11.4.2 Performance Comparison
11.4.3 LBP Visualized Results
11.4.4 Training Convergence
11.5 Conclusion
References
Chapter 12 Optimizing the Identification and Utilization of Open Parking Spaces Through Advanced Machine Learning
12.1 Introduction
12.2 Proposed Framework Optimized Parking Space Identifier (OPSI)
12.2.1 Framework Components
12.2.2 Learning Module: Adaptive Prediction of Parking Space Availability
12.2.3 System Design
12.2.4 Tools and Usage
12.2.5 Architecture
12.2.6 Implementation Techniques and Algorithms
12.2.7 Existing Methods and Workflow Model
12.2.8 Hyperparameter for OPSI
12.3 Potential Impact
12.3.1 Claims for the Accurate Detection of Fatigue
12.3.2 Similar Study and Results Analysis
12.4 Application and Results
12.4.1 Algorithm and Results
12.4.2 Implementation Using Python Modules
12.5 Discussion and Limitations
12.5.1 Discussion
12.5.2 Limitations
12.6 Future Work
12.6.1 Integration with Autonomous Vehicles
12.6.2 Real-Time Data Analysis
12.6.3 Integration with Smart Cities
12.7 Conclusion
References
Chapter 13 Graphical Password Authentication Using Python for Aerial Devices/Drones
13.1 Introduction
13.2 Literature Review
13.3 Methodology
13.4 A Brief Overview of a Drone and Authentication
13.4.1 Password Authentication
13.4.2 Types of Password Authentication Systems
13.4.3 Graphical Password Authentication
13.4.4 Advantages and Disadvantages of Graphical Passwords
13.5 Password Cracking
13.6 Data Analysis
13.7 Discussion
13.8 Conclusion and Future Scope
References
Chapter 14 A Study Centering on the Data and Processing for Remote Sensing Utilizing from Annoyed Aerial Vehicles
14.1 Introduction
14.2 An Acquisition Method for 3D Data Utilising Annoyed Aerial Vehicles
14.3 Background and Literature of Review
14.4 Research Gap
14.5 Methodology
14.6 Discussion
14.7 Conclusion
References
Chapter 15 Satellite Image Classification Using Convolutional Neural Network
15.1 Introduction
15.2 Literature Review
15.3 Objectives of this Research Work
15.3.1 Novelty of the Research Work
15.4 Description of the Dataset
15.5 Theoretical Framework
15.6 Implementation and Results
15.6.1 Data Visualization
15.6.1.1 Class-Wise Data Count
15.6.1.2 Class-Wise Augmented Data Count
15.6.2 Implementation of MobileNetV3
15.6.2.1 Visualization of a Sample of Training Images
15.6.2.2 Visualization of Executed Codes of MobileNetV3
15.6.2.3 Training Results of MobileNetV3
15.6.2.4 Classifications of Errors on Test Sets of MobileNetV3
15.6.2.5 Confusion Matrix of MobileNetV3
15.6.2.6 Classification Report of MobileNetV3
15.6.3 Implementation of EfficientNetB0
15.6.3.1 Visualization of a Sample of Training Images
15.6.3.2 Visualization of Executed Codes of EfficientNetB0
15.6.3.3 Training Results of EfficientNetB0
15.6.3.4 Classifications of Errors on Test Sets of EfficientNetB0
15.6.3.5 Confusion Matrix of EfficientNetB0
15.6.3.6 Classification Report of EfficientNetB0
15.7 Conclusion and Future Scope
References
Chapter 16 Edge Computing in Aerial Imaging – A Research Perspective
16.1 Introduction
16.1.1 Edge Computing and Aerial Imaging
16.2 Research Applications of Aerial Imaging
16.2.1 Vehicle Imaging
16.2.2 Precision Agriculture
16.2.3 Environment Monitoring
16.2.4 Urban Planning and Development
16.2.5 Emergency Response
16.3 Edge Computing and Aerial Imaging
16.3.1 Research Perspective in Aerial Imaging
16.3.2 Edge Architectures
16.4 Comparative Analysis of the Aerial Imaging Algorithms and Architectures
16.5 Discussion
16.6 Conclusion
References
Chapter 17 Aerial Sensing and Imaging Analysis for Agriculture
17.1 Introduction
17.2 Experimental Methods and Techniques
17.3 Aerial Imaging and Sensing Applications in Agriculture
17.3.1 Assessing Yield and Fertilizer Response
17.3.2 Plant and Crop Farming
17.3.3 Soil and Field Analysis
17.3.4 Weed Mapping and Management
17.3.5 Plantation Crop
17.3.6 Crop and Spot Spraying
17.3.7 Crop Monitoring
17.4 Aerial Imaging and Sensing Applications in Livestock Farming
17.4.1 Livestock Sensor
17.4.2 Animal Health
17.4.3 Monitoring and Identification of Livestock Farming
17.4.4 Geo Fencing and Virtual Perimeters
17.5 Challenges in Aerial Sensing and Imaging in Agriculture and Livestock Farming
17.5.1 Technical Limitations of Aerial Sensing and Imaging in Agriculture and Livestock Farming
17.6 Conclusion
References
Index

Citation preview

This groundbreaking book is a comprehensive guide to the technology found in the complex field of aerial sensing and imaging, and the real-world challenges that stem from its growing significance and demand. The advent of unmanned aerial vehicles (UAVs), or drones, along with advancements in sensor technology and image processing techniques, has further enhanced the capabilities and applications of aerial sensing and imaging. These developments have opened up new research, innovation, and exploration avenues. Aerial sensing and imaging have rapidly evolved over the past few decades and have revolutionized several fields, including land cover and usage prediction, crop and livestock management, road accident monitoring, poverty estimation, defense, agriculture, forest fire detection, UAV security issues, and open parking management. This book provides a comprehensive understanding and knowledge of the underlying technology and its practical applications in different domains. Audience Computer science and artificial intelligence researchers working in the fields of aerial sensing and imaging, as well as professionals working in industries such as agriculture, geology, surveying, urban planning, disaster response, etc; this book provides them with practical guidance and instruction on how to apply aerial sensing and imaging for various purposes and stay up-to-date with the latest developments in the domain.

Advances in Aerial Sensing and Imaging

Scrivener Publishing 100 Cummings Center, Suite 541J Beverly, MA 01915-6106 Publishers at Scrivener Martin Scrivener ([email protected]) Phillip Carmical ([email protected])

Advances in Aerial Sensing and Imaging

Edited by

Sandeep Kumar Nageswara Rao Moparthi Abhishek Bhola Ravinder Kaur A. Senthil and

K.M.V.V. Prasad

This edition first published 2024 by John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA and Scrivener Publishing LLC, 100 Cummings Center, Suite 541J, Beverly, MA 01915, USA © 2024 Scrivener Publishing LLC For more information about Scrivener publications please visit www.scrivenerpublishing.com. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, except as permitted by law. Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions. Wiley Global Headquarters 111 River Street, Hoboken, NJ 07030, USA For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com. Limit of Liability/Disclaimer of Warranty While the publisher and authors have used their best efforts in preparing this work, they make no rep­ resentations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation any implied warranties of merchant-­ ability or fitness for a particular purpose. No warranty may be created or extended by sales representa­ tives, written sales materials, or promotional statements for this work. The fact that an organization, website, or product is referred to in this work as a citation and/or potential source of further informa­ tion does not mean that the publisher and authors endorse the information or services the organiza­ tion, website, or product may provide or recommendations it may make. This work is sold with the understanding that the publisher is not engaged in rendering professional services. The advice and strategies contained herein may not be suitable for your situation. You should consult with a specialist where appropriate. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Library of Congress Cataloging-in-Publication Data ISBN 978-1-394-17469-0 Cover image: Pixabay.Com Cover design by Russell Richardson Set in size of 11pt and Minion Pro by Manila Typesetting Company, Makati, Philippines Printed in the USA 10 9 8 7 6 5 4 3 2 1

Contents Preface xv 1 A Systematic Study on Aerial Images of Various Domains: Competences, Applications, and Futuristic Scope 1 Abhishek Bhola, Bikash Debnath and Ankita Tiwari 1.1 Introduction 2 1.2 Literature Work 5 1.2.1 Based on Camera Axis 6 1.2.2 Based on Scale 9 1.2.3 Based on Sensor 13 1.3 Challenges of Object Detection and Classification in Aerial Images 17 1.4 Applications of Aerial Imaging in Various Domains 21 1.5 Conclusions and Future Scope 26 1.5.1 Conclusions 26 1.5.2 Future Scope 26 References 27 2 Oriental Method to Predict Land Cover and Land Usage Using Keras with VGG16 for Image Recognition Monali Gulhane and Sandeep Kumar 2.1 Introduction 2.2 Literature Review 2.3 Materials and Methods 2.3.1 Dataset 2.3.2 Model Implemented 2.4 Discussion 2.5 Result Analysis 2.6 Conclusion References

33 34 35 37 38 38 40 41 43 44

v

vi  Contents 3 Aerial Imaging Rescue and Integrated System for Road Monitoring Based on AI/ML Munish Kumar, Poonam Jaglan and Yogesh Kakde 3.1 Introduction 3.2 Related Work 3.3 Number of Accidents, Fatalities, and Injuries: 2016–2022 3.3.1 Accidents Statistics in India 3.3.2 Accidents Statistics in Haryana 3.4 Proposed Methodology 3.4.1 ROI and Line Selection 3.4.2 Motion Detection 3.4.3 Single-Stage Clustering 3.4.4 Feature Fusion Process 3.4.5 Second-Stage Clustering 3.4.6 Tracking Objects 3.4.7 Classification 3.5 Result Analysis 3.6 Conclusion References 4 A Machine Learning Approach for Poverty Estimation Using Aerial Images Nandan Banerji, Sreenivasulu Ballem, Siva Mala Munnangi and Sandeep Mittal 4.1 Introduction 4.2 Background and Literature Review 4.3 Proposed Methodology 4.3.1 Data Acquisition 4.3.2 Pre-Processing 4.3.3 Feature Extraction 4.3.4 Data Integration 4.3.5 Model Development 4.3.6 Validation 4.3.7 Visualization and Analysis 4.3.8 Policy and Program Development 4.4 Result and Discussion 4.5 Conclusion and Future Scope References

47 48 49 52 52 53 53 54 56 58 58 59 59 60 61 63 64 69 70 73 76 76 78 78 79 80 80 81 82 82 83 85

Contents  vii 5 Agriculture and the Use of Unmanned Aerial Vehicles (UAVs): Current Practices and Prospects Ajay Kumar Singh and Suneet Gupta 5.1 Introduction 5.2 UAVs Classification 5.2.1 Comparison of Various UAVs 5.3 Agricultural Use of UAVs 5.4 UAVs in Livestock Farming 5.5 Challenges 5.6 Conclusion References 6 An Introduction to Deep Learning-Based Object Recognition and Tracking for Enabling Defense Applications Nitish Mahajan, Aditi Chauhan and Monika Kajal 6.1 Introduction 6.2 Related Work 6.2.1 Importance of Object Monitoring and Surveillance in Defense 6.2.2 Need for Object Monitoring and Surveillance in Defense 6.2.3 Object Detection Techniques 6.2.4 Object Tracking Techniques 6.3 Experimental Methods 6.3.1 Experimental Setup and Dataset 6.3.2 DataSetVISdrone 2019 6.3.3 Experimental Setup 6.4 Results and Outcomes 6.4.1 Comparison Results 6.4.2 Training Results 6.5 Conclusion 6.6 Future Scope References 7 A Robust Machine Learning Model for Forest Fire Detection Using Drone Images Chahil Choudhary, Anurag and Pranjal Shukla 7.1 Introduction 7.2 Literature Review 7.3 Proposed Methodology

87 88 90 94 96 101 104 105 105 109 110 111 111 112 113 116 121 121 121 122 122 123 123 123 125 125 129 130 131 133

viii  Contents 7.4 Result and Discussion 7.5 Conclusion and Future Scope References

135 142 142

8 Semantic Segmentation of Aerial Images Using Pixel Wise Segmentation 145 Swathi Gowroju, Shilpa Choudhary, Sandhya Raajaani and Regula Srilakshmi 8.1 Introduction 146 8.2 Related Work 147 8.3 Proposed Method 149 8.3.1 Pixelwise Classification Method 151 8.3.2 Morphological Processing 153 8.4 Datasets 153 8.5 Results and Discussion 154 8.5.1 Analysis of the Proposed Method 159 8.6 Conclusion 161 References 161 9 Implementation Analysis of Ransomware and Unmanned Aerial Vehicle Attacks: Mitigation Methods and UAV Security Recommendations 165 Sidhant Sharma, Pradeepta Kumar Sarangi, Bhisham Sharma and Girija Bhusan Subudhi 9.1 Introduction 166 9.2 Types of Ransomwares 167 9.3 History of Ransomware 168 9.4 Notable Ransomware Strains and Their Impact 171 9.4.1 CryptoLocker (2013) 171 9.4.2 CryptoWall (2014) 173 9.4.3 TeslaCrypt (2015) 173 9.4.4 Locky (2016) 174 9.4.5 WannaCry (2017) 175 9.4.6 NotPetya (2017) 176 9.4.7 Ryuk (2018) 178 9.4.8 REvil (2019) 180 9.4.9 Present-Day Ransomware Families 182 9.5 Mitigation Methods for Ransomware Attacks 184 9.6 Cybersecurity in UAVs (Unmanned Aerial Vehicles) 185 9.6.1 Introduction on FANETS 186

Contents  ix 9.6.2 Network Security Concerning FANETs 9.6.3 UAV Security Enhancement 9.6.4 Limitations in UAVs 9.6.5 Future Scope 9.7 Experimental analysis of Wi-Fi Attack on Ryze Tello UAVs 9.7.1 Introduction 9.7.2 Methodology 9.8 Results and Discussion 9.9 Conclusion and Future Scope References 10 A Framework for Detection of Overall Emotional Score of an Event from the Images Captured by a Drone P.V.V.S. Srinivas, Dhiren Dommeti, Pragnyaban Mishra and T.K. Rama Krishna Rao 10.1 Introduction 10.1.1 Need for Emotion Recognition 10.1.2 Applications of Drones in Deep Learning 10.2 Literature Review 10.3 Proposed Work 10.3.1 Extraction of Images from a Drone 10.3.2 Proposed CNN Model 10.4 Experimentation and Results 10.4.1 Dataset Description 10.5 Future Work and Conclusion References 11 Drone-Assisted Image Forgery Detection Using Generative Adversarial Net-Based Module Swathi Gowroju, Shilpa Choudhary, Medipally Rishitha, Singanaboina Tejaswi, Lankala Shashank Reddy and Mallepally Sujith Reddy 11.1 Introduction 11.2 Literature Survey 11.3 Proposed System 11.3.1 Common Forged Feature Network 11.3.2 Features Extraction 11.3.3 Features Classification and Classification Network 11.3.4 Label Prediction 11.3.5 Contrastive Learning 11.3.6 Binary Cross-Entropy Loss

186 191 192 193 194 194 195 198 206 207 213 214 214 215 216 220 220 222 223 224 230 240 245

246 247 250 252 253 253 255 255 256

x  Contents 11.4 Results 11.4.1 Experimental Settings 11.4.2 Performance Comparison 11.4.3 LBP Visualized Results 11.4.4 Training Convergence 11.5 Conclusion References 12 Optimizing the Identification and Utilization of Open Parking Spaces Through Advanced Machine Learning Harish Padmanaban P. C. and Yogesh Kumar Sharma 12.1 Introduction 12.2 Proposed Framework Optimized Parking Space Identifier (OPSI) 12.2.1 Framework Components 12.2.2 Learning Module: Adaptive Prediction of Parking Space Availability 12.2.3 System Design 12.2.4 Tools and Usage 12.2.5 Architecture 12.2.6 Implementation Techniques and Algorithms 12.2.7 Existing Methods and Workflow Model 12.2.8 Hyperparameter for OPSI 12.3 Potential Impact 12.3.1 Claims for the Accurate Detection of Fatigue 12.3.2 Similar Study and Results Analysis 12.4 Application and Results 12.4.1 Algorithm and Results 12.4.2 Implementation Using Python Modules 12.5 Discussion and Limitations 12.5.1 Discussion 12.5.2 Limitations 12.6 Future Work 12.6.1 Integration with Autonomous Vehicles 12.6.2 Real-Time Data Analysis 12.6.3 Integration with Smart Cities 12.7 Conclusion References

256 257 258 259 262 264 264 267 267 270 270 272 273 274 275 277 278 278 281 282 282 284 284 286 289 289 289 290 290 290 290 290 291

Contents  xi 13 Graphical Password Authentication Using Python for Aerial Devices/Drones Sushma Singh and Dolly Sharma 13.1 Introduction 13.2 Literature Review 13.3 Methodology 13.4 A Brief Overview of a Drone and Authentication 13.4.1 Password Authentication 13.4.2 Types of Password Authentication Systems 13.4.3 Graphical Password Authentication 13.4.4 Advantages and Disadvantages of Graphical Passwords 13.5 Password Cracking 13.6 Data Analysis 13.7 Discussion 13.8 Conclusion and Future Scope References 14 A Study Centering on the Data and Processing for Remote Sensing Utilizing from Annoyed Aerial Vehicles Vandna Bansla, Sandeep Kumar, Vibhoo Sharma, Girish Singh Bisht and Akanksha Srivastav 14.1 Introduction 14.2 An Acquisition Method for 3D Data Utilising Annoyed Aerial Vehicles 14.3 Background and Literature of Review 14.4 Research Gap 14.5 Methodology 14.6 Discussion  14.7 Conclusion References 15 Satellite Image Classification Using Convolutional Neural Network Pradeepta Kumar Sarangi, Bhisham Sharma, Lekha Rani and Monica Dutta 15.1 Introduction 15.2 Literature Review 15.3 Objectives of this Research Work 15.3.1 Novelty of the Research Work 15.4 Description of the Dataset

295 296 297 298 299 300 301 301 305 305 307 309 309 310 313 314 315 317 319 319 321 327 329 333 334 335 336 337 337

xii  Contents 15.5 Theoretical Framework 15.6 Implementation and Results 15.6.1 Data Visualization 15.6.1.1 Class-Wise Data Count 15.6.1.2 Class-Wise Augmented Data Count 15.6.2 Implementation of MobileNetV3 15.6.2.1 Visualization of a Sample of Training Images 15.6.2.2 Visualization of Executed Codes of MobileNetV3 15.6.2.3 Training Results of MobileNetV3 15.6.2.4 Classifications of Errors on Test Sets of MobileNetV3 15.6.2.5 Confusion Matrix of MobileNetV3 15.6.2.6 Classification Report of MobileNetV3 15.6.3 Implementation of EfficientNetB0 15.6.3.1 Visualization of a Sample of Training Images 15.6.3.2 Visualization of Executed Codes of EfficientNetB0 15.6.3.3 Training Results of EfficientNetB0 15.6.3.4 Classifications of Errors on Test Sets of EfficientNetB0 15.6.3.5 Confusion Matrix of EfficientNetB0 15.6.3.6 Classification Report of EfficientNetB0 15.7 Conclusion and Future Scope References

337 339 340 340 340 341 341 343 343 344 345 346 346 347 347 348 349 349 350 350 352

16 Edge Computing in Aerial Imaging – A Research Perspective 355 Divya Vetriveeran, Rakoth Kandan Sambandam, Jenefa J. and Leena Sri R. 16.1 Introduction 355 16.1.1 Edge Computing and Aerial Imaging 356 16.2 Research Applications of Aerial Imaging 357 16.2.1 Vehicle Imaging 358 16.2.2 Precision Agriculture 359 16.2.3 Environment Monitoring 360 16.2.4 Urban Planning and Development 363 16.2.5 Emergency Response 364 16.3 Edge Computing and Aerial Imaging 366 16.3.1 Research Perspective in Aerial Imaging 367

Contents  xiii 16.3.2 Edge Architectures 16.4 Comparative Analysis of the Aerial Imaging Algorithms and Architectures 16.5 Discussion 16.6 Conclusion References

369 376 379 380 380

17 Aerial Sensing and Imaging Analysis for Agriculture 383 Monika Kajal and Aditi Chauhan 17.1 Introduction 384 17.2 Experimental Methods and Techniques 388 17.3 Aerial Imaging and Sensing Applications in Agriculture 390 17.3.1 Assessing Yield and Fertilizer Response 390 17.3.2 Plant and Crop Farming 392 17.3.3 Soil and Field Analysis 393 17.3.4 Weed Mapping and Management 394 17.3.5 Plantation Crop 395 17.3.6 Crop and Spot Spraying 396 17.3.7 Crop Monitoring 397 17.4 Aerial Imaging and Sensing Applications in Livestock Farming 398 17.4.1 Livestock Sensor 398 17.4.2 Animal Health 399 17.4.3 Monitoring and Identification of Livestock Farming 402 17.4.4 Geo Fencing and Virtual Perimeters 403 17.5 Challenges in Aerial Sensing and Imaging in Agriculture and Livestock Farming 404 17.5.1 Technical Limitations of Aerial Sensing and Imaging in Agriculture and Livestock Farming 405 17.6 Conclusion 406 References 406 Index 411

Preface This book is a comprehensive guide to the technology found in the complex field of aerial sensing and imaging, and the real-world challenges that stem from its growing significance and demand. The advent of unmanned aerial vehicles (UAVs), or drones, along with advancements in sensor technology and image processing techniques, has further enhanced the capabilities and applications of aerial sensing and imaging. These developments have opened up new research, innovation, and exploration avenues. Aerial sensing and imaging have rapidly evolved over the past few decades and have revolutionized several fields, including land cover and usage prediction, crop and livestock management, road accident monitoring, poverty estimation, defense, agriculture, forest fire detection, UAV security issues, and open parking management. This book provides a comprehensive understanding and knowledge of the underlying technology and its practical applications in different domains. Readers will also be able to learn about the latest research, theories, and real-world challenges in this field. Advances in Aerial Sensing and Imaging is a comprehensive collection of research articles that deeply understand the latest advancements in aerial sensing and imaging. This book serves as a reference guide for researchers, professionals, and students who are interested in exploring the potential of the subject. The other objective of this book is to combine in collaborative the work of multiple authors sharing the same interest in aerial sensing and imaging. The contributors to this book are experts in their respective fields and have provided valuable insights into the latest research and advancements. The chapters herein cover a wide range of topics in diversified research fields. We thank all the authors who helped us tremendously with their contributions, time, critical thoughts, and suggestions to assemble this peer-­ reviewed volume, and without whose dedication and hard work this book would not be possible. We also extend our thanks to the reviewers who have provided valuable feedback and helped to improve the quality of the xv

xvi  Preface material. The editors are also thankful to Scrivener Publishing and their team members for the opportunity to publish this volume. Lastly, we thank our family members for their love, support, encouragement, and patience during this work. We hope this book will be valuable for researchers, professionals, and students interested in aerial sensing and imaging. This book will inspire further research and innovation and contribute to developing new applications and technologies. We look forward to new advancements and hope this book will play a small role in shaping the future of this exciting field. Sandeep Kumar Nageswara Rao Moparthi Abhishek Bhola Ravinder Kaur A. Senthil K.M.V.V. Prasad

1 A Systematic Study on Aerial Images of Various Domains: Competences, Applications, and Futuristic Scope Abhishek Bhola1*, Bikash Debnath2 and Ankita Tiwari3 Chaudhary Charan Singh Haryana Agricultural University College of Agriculture, Bawal, Haryana, Rewari, India 2 Department of Information Technology, Amity University, Kolkata, West Bengal, India 3 Department of Mathematics, Koneru Lakshmaiah Education Foundation, Vijayawada, India 1

Abstract

Aerial images captured by drones or aircraft provide a unique perspective and valuable data in various fields, including agriculture, urban planning, construction, and environmental research. They offer high-resolution images that can be used to create detailed maps, monitor changes over time, and provide clear information not visible from ground level. Despite their many benefits, there are also challenges associated with aerial imaging. These challenges include the cost and availability of equipment, weather conditions and terrain, data management and analysis, privacy concerns, and regulatory issues. Overcoming these challenges requires specialized skills and expertise and careful consideration of ethical and environmental concerns. However, as technology advances, aerial images’ benefits are expanding, enabling new applications, and more detailed analysis. For example, infrared imaging allows for monitoring plant health and identifying areas of water stress, which is particularly useful in agriculture and environmental research. In addition, the ability to cover large areas quickly and efficiently provides a comprehensive view that is impossible with ground-based surveys, making it valuable for infrastructure inspection and urban planning. Despite the challenges, the scope for aerial imaging is expanding rapidly, with advancements in technology enabling new applications and more detailed analysis. As the *Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (1–32) © 2024 Scrivener Publishing LLC

1

2  Advances in Aerial Sensing and Imaging technology continues to evolve, the benefits of aerial images will only continue to grow, making it an increasingly valuable tool for decision-making and problem-solving in various industries. Keywords:  Aerial images, scale, sensor, camera, machine learning

1.1 Introduction Aerial sensing and imaging have revolutionized how we observe, measure, and understand the world. This technology captures images, videos, and other data from an elevated platform like an aircraft or drone to understand a particular area or object better [1–3]. With technological advancements, aerial sensing and imaging have become more accessible and cost-effective, making them essential tools for various industries, including agriculture, forestry, urban planning, environmental monitoring, and infrastructure management [4–6]. Aerial sensing and imaging are also used to provide real-time data and insights in emergency response situations, such as natural disasters and search and rescue missions. This paper will explore the various applications and benefits of aerial sensing and imaging and discuss the latest technological advancements [7–9]. The history of aerial sensing and imaging dates back to the 19th century, when French photographer and balloonist Gaspar Félix Tournachon, known as Nadar, captured the first aerial photograph over Paris in 1858. The picture was taken from a hot air balloon at an altitude of 80 meters and provided a bird’s eye view of the city [10–12]. In the early 20th century, advancements in aircraft technology led to the development of aerial photography, which became an essential tool for military surveillance during World War-I [13–16]. During the 1920s and 1930s, aerial photography was used for mapping, surveying, and topographic studies, as well as for scientific research and exploration. In the 1950s and 1960s, aerial sensing and imaging technology evolved with the development of airborne remote sensing systems, which used cameras and sensors mounted on aircraft to capture images and data [17–19]. This technology was used for various applications, including mapping, agriculture, forestry, and geological exploration. In the 1970s and 1980s, satellite remote sensing technology emerged as a new and powerful tool for remote sensing and imaging. Satellites offered a global view of the Earth and provided data on various environmental factors, including climate, vegetation, and oceanography [20–24]. In the 1990s and 2000s, advancements in unmanned aerial

Aerial Images of Various Domains  3 vehicle (UAV) technology led to the development of aerial sensing and imaging systems that were more cost-effective, flexible, and accessible than traditional airborne or satellite-based systems. UAVs were used for various applications, including agriculture, environmental monitoring, search and rescue, and infrastructure inspection [25–28]. The global market for unmanned aerial vehicle (UAV) drones is projected to reach 102.38 billion US dollars by 2032, expanding at a compound annual growth rate (CAGR) of 18.2% from 2018 to 2032 as shown in Figure 1.1 [29–31]. Today, aerial sensing and imaging technology continue to evolve, with new and innovative applications and advancements in sensor technology, data analysis, and artificial intelligence. Aerial sensing and imaging technology is being used to monitor and mitigate climate change’s impact, support disaster response and recovery efforts, and improve the efficiency and sustainability of various industries [32–35]. Aerial sensing and imaging technology has a rich and diverse history, spanning over a century of technological advancements and innovation. Today, it continues to be a critical tool for understanding and managing our planet and is poised to play an increasingly important role in the future. Aerial sensing and imaging is a rapidly growing field involving advanced technologies to capture images and data from the air [36–38]. This field has numerous applications in environmental monitoring, disaster response, agriculture, and urban planning. Using unmanned aerial vehicles (UAVs) has revolutionized aerial sensing and imaging, enabling researchers and professionals to collect highresolution data quickly and accurately [39, 40]. If you are considering Global Aerial Imaging Market Size, By Region, 2018-2032 (USD Million)

2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 North America

Europe

Asia Pacific

Figure 1.1  Aerial imaging growth market (2023–2032).

Latin America

MEA

4  Advances in Aerial Sensing and Imaging studying aerial sensing and imaging, there are many compelling reasons to do so. First and foremost, this field is at the forefront of technological innovation, and studying it will expose you to the latest advances in remote sensing and data collection. This is an exciting time to be involved in aerial sensing and imaging, as new technologies such as LiDAR, hyperspectral imaging, and drones are rapidly advancing the field. In addition to technological innovation, there is a growing need for professionals with aerial sensing and imaging expertise [41–43]. As the demand for more accurate and detailed data grows in areas such as environmental monitoring and agriculture, there is a need for skilled professionals who can use aerial sensing and imaging technologies to collect and analyze this data [44, 45]. This presents an excellent opportunity for those with a background in this field to pursue rewarding careers in agriculture, forestry, and environmental science. Studying aerial sensing and imaging also opens up opportunities for research and development. This field has many unanswered questions and options for exploration, and those with the skills and knowledge to tackle these challenges can significantly contribute to the field. Research in this area can lead to new insights into environmental processes, improved mapping, and monitoring techniques, and data analysis and interpretation advancements [46–48]. Finally, studying aerial sensing and imaging can be personally rewarding. It provides an opportunity to work on cutting-edge technology and make a meaningful impact in environmental conservation and disaster response fields. Suppose you are passionate about using technology to solve real-world problems and positively impact society. In that case, studying aerial sensing and imaging may be the perfect fit for you [49, 50]. The study of aerial sensing and imaging involves using advanced technologies to capture images and data from the air [51–53]. The following are some of the critical objectives of studying aerial sensing and imaging: • To gain knowledge and understanding of the underlying principles and concepts of remote sensing and imaging techniques. Aerial sensing and imaging rely on various technologies, including LiDAR, radar, multispectral and hyperspectral imaging, and drones [54–56]. Studying these technologies will enable you to understand how they work, their limitations, and how they can be applied to various applications. • To develop technical skills in using remote sensing and imaging equipment and software. Aerial sensing and imaging involve complex technologies and software for data capture,

Aerial Images of Various Domains  5 processing, and analysis. Studying this field will enable you to develop technical skills using remote sensing and imaging equipment, including drones, cameras, and other sensors [57–59]. You will also learn to process and analyze data using specialized software such as Geographic Information Systems (GIS). • To understand the applications and limitations of aerial sensing and imaging in various fields. Aerial sensing and imaging have many applications in agriculture, forestry, environmental science, and urban planning. Studying aerial sensing and imaging will enable you to understand the potential and limitations of these technologies in various applications [60–62]. • To develop the ability to design and implement aerial sensing and imaging projects. Aerial sensing and imaging projects involve many steps, including project design, data acquisition, processing, analysis, and reporting [63–65]. Studying aerial sensing and imaging will enable you to develop the skills to design and implement such projects effectively. • To contribute to the advancement of the field through research and development. Aerial sensing and imaging are rapidly evolving fields; much must be explored and discovered. Studying aerial sensing and imaging will enable you to contribute to advancing the area through research and development [66, 67]. Literature study of the aerial imaging and sensing is discussed in Section 1.2 to acquire essential information, insights, and issues. In Section 1.3, the diverse aerial image issues are explored and listed. In Section 1.4, various aerial image applications are discussed. Finally in Section 1.5, the study is concluded by discussing prospective areas for future investigation.

1.2 Literature Work Aerial images are photographs taken from a high altitude using cameras and sensors. Literature on aerial images covers remote sensing, cartography, urban planning, and disaster response [68–71]. Research focuses on image processing techniques, sensor selection, and applications in various fields, as shown in Figure 1.2.

6  Advances in Aerial Sensing and Imaging Types of Aerial Images

Based on Camera Axis

Based on Sensors

Based on Scale

Vertical Photograph

Large Scale Photograph

Black & White Panchromatic Imagery

Low Oblique Photographs

Medium Scale Photographs

Natural Color Imagery

High Oblique Photographs

Small Scale Photographs

Infrared or Thermal Imagery

Radar Imagery

Figure 1.2  Overall structure of aerial image types.

1.2.1 Based on Camera Axis Aerial images based on the camera axis refer to photographs taken from a high altitude using a camera pointed straight down as shown in Figure 1.3. This technique is commonly used in cartography, remote sensing, and VERTICAL 1 LENS OF CAMERA

2 3 LOW OBLIQUE

HIGH OBLIQUE

Figure 1.3  Types of aerial images based on camera axis.

Aerial Images of Various Domains  7 surveillance applications. Literature on this topic includes studies on image processing techniques to improve the quality and accuracy of aerial images, such as filtering, feature extraction, and classification algorithms. Other research focuses on developing uncrewed aerial vehicles (UAVs) with cameras for efficient data collection. Additionally, studies have explored the use of aerial images for monitoring and mapping environmental changes, urban planning, and disaster response. • Vertical Photograph The use of vertical photographs in aerial images has been studied for many years. This brief literature review will explore some of the research done in this field. One study published in the Journal of Geographical Sciences by author Wei Huang et al. [1], examined the use of vertical photographs in digital photogrammetry. They found that vertical photos have higher accuracy and precision than oblique photographs, making them more suitable for mapping and surveying purposes. Another study published in the International Journal of Remote Sensing by authors John E. Peacock et al. [2], examined the use of vertical photographs in urban remote sensing. They found that vertical photos can provide more accurate and detailed information about the built environment than oblique photographs. This information can be used for urban planning and management purposes. A study published in the Journal of Photogrammetry and Remote Sensing by authors Yves-Louis Desnos et al. [3], examined the use of vertical photographs in terrain mapping. They found that vertical pictures provide a better terrain view, making them more suitable for terrain mapping applications. Another study published in the International Journal of Remote Sensing by authors Xiaojun Cheng et al. [4], examined the use of vertical photographs in land cover classification. They found vertical photos can provide more accurate and reliable information about land cover types than oblique photographs. Finally, a study published in the International Journal of Remote Sensing by authors Nicholas M. Short et al. [5], examined the use of vertical photos in forestry. They found that vertical photographs can provide more accurate information about forest canopy structures, which can be used for forest inventory and management.

8  Advances in Aerial Sensing and Imaging • Low Oblique Photographs Low oblique photographs in aerial images have been a topic of interest for many researchers in remote sensing and photogrammetry. The following literature study summarizes some notable works on this topic: “Low Oblique Photography for Survey and Mapping” by E. L. Rayner et al. [6], this early work explores the use of low oblique photography for surveying and mapping purposes. The authors highlight the advantages of low oblique photography over traditional vertical aerial photography, such as the ability to see the sides of buildings and terrain features more clearly. “Low-Oblique Photogrammetry: Application to Geology” by C. W. Wright et al. [7], this study demonstrates the use of low oblique photography in geological mapping. The authors show that low oblique photographs can provide detailed information on the shape and orientation of rock outcrops, which can be helpful in geological mapping and mineral exploration. “Low-Oblique Aerial Photography in Landscape Ecology” by R. T. T. Forman et al. [8], this study uses low oblique photography in landscape ecology. The authors demonstrate how low oblique photographs can provide information on landscape structure and pattern, which can be used to study the effects of land use changes on ecosystems. “Low-Altitude Photogrammetry” by J. R. Jensen [9], this book chapter provides an overview of low-altitude photogrammetry, which includes low oblique photography. The author discusses the advantages and disadvantages of low oblique photography and provides examples of its applications in various fields, such as forestry, agriculture, and urban planning. “LowOblique Photogrammetry for Building Damage Assessment” by S. H. Lee et al. [10] this recent study explores the use of low oblique photography for building damage assessment. The authors show that low oblique photographs can provide more detailed information on building damage than vertical aerial photographs, which can be helpful for post-disaster reviews. • High Oblique Photographs High oblique photographs in aerial images have also been a topic of interest for researchers in remote sensing and photogrammetry. A literature study summarizes some notable

Aerial Images of Various Domains  9 works on this topic: “The interpretation of high oblique photographs” by J. B. Campbell [11], this early work focuses on the interpretation of high oblique photographs. The author discusses the advantages and disadvantages of highoblique photographs compared to low-oblique and vertical aerial photographs. The author also provides examples of high oblique photographs used for urban planning and military applications. “High-Oblique Aerial Photography in Forestry” by G. L. Hosford et al. [12], this study explores high oblique photography in forestry. The authors show that highly oblique photographs can provide information on forest structure and composition, which can be helpful for forest inventory and management. “High-Oblique Photography in Archaeology” by R. W. Ehrich [13] this study demonstrates the use of high oblique photography in archaeology. The author shows that highly oblique photographs can provide information on the location and extent of archaeological sites, which can be helpful for site discovery and mapping. “High-Oblique Aerial Photography for Mapping and Environmental Studies” by D. A. Norton et al. [14], explores high oblique photography for mapping and environmental studies. The authors discuss the advantages of high oblique photographs, such as capturing a larger area than vertical aerial photographs and provide examples of their use in various applications, such as wetlands mapping and land use planning. “High-Oblique Aerial Photography for Disaster Assessment and Response” by K. N. Sukumar et al. [15] this recent study demonstrates the use of highoblique photography for disaster assessment and response. The authors show that highly oblique photographs can provide information on the extent and severity of damage caused by natural disasters, which can be helpful in emergency response and planning.

1.2.2 Based on Scale Aerial images based on scale refer to photographs taken from a high altitude scaled to a particular size for analysis and measurement, as shown in Figure 1.4. Literature on this topic includes studies on aerial images for topographic mapping, urban planning, and resource management. Research has also focused on developing image processing techniques to

10  Advances in Aerial Sensing and Imaging

Test image

Reflect (y-axis)

Shear (y direction) Shear (x direction)

Reflect (x-axis)

Reflect (origin)

Translate

Rotate

Scale (smaller)

Scale (bigger)

Figure 1.4  Types of aerial images based on scale.

improve the accuracy of scaled aerial photos, such as orthorectification and geometric correction. Additionally, studies have explored using unmanned aerial vehicles (UAVs) for capturing high-resolution images at a smaller scale for more detailed analysis. The importance of scaling in aerial photos is also discussed concerning the accuracy of measurements, interpretation, and comparison with other data sources. • High Oblique Photographs Large-scale photographs in aerial images are used for detailed mapping and analysis of small areas. Here is a literature study summarizing some notable works on this topic: “The Use of Large Scale Aerial Photography for Mapping the Geology of Small Areas” by H. E. Gregory et al. [16], this early work focuses on the use of large scale aerial photography for mapping the geology of small areas. The authors discuss the advantages of large-scale photographs, such as capturing detailed landscape features and providing examples of their use in geologic mapping. “Applications of LargeScale Aerial Photography in Highway Engineering” by W. L. Beadles et al. [17], this study explores large-scale aerial photography in highway engineering. The authors show that large-scale photographs can provide information on terrain features and land use, which can help plan highway routes and design structures. “The Use of Large-Scale Aerial

Aerial Images of Various Domains  11 Photography in Forestry” by D. E. Bedford [18], this study demonstrates large-scale aerial photography in forestry. The author shows that large-scale photographs can provide information on forest structure and composition, which can be helpful for forest inventory and management. “Large-Scale Aerial Photography for Environmental Monitoring and Management” by L. M. Joppa et al. [19], this study explores large-scale aerial photography for environmental monitoring and management. The authors discuss the advantages of large-scale photographs, such as capturing detailed information on ecological features and providing examples of their use in various applications, such as land cover classification and habitat monitoring. “Large-Scale Aerial Photography for Precision Agriculture” by J. P. White et al. [20], this recent study demonstrates the use of large-scale aerial photography for precision agriculture. The authors show that large-scale photographs can provide information on crop health and yield variability, which can be helpful for crop management and optimization. • Medium Oblique Photographs Medium-scale photographs in aerial images are commonly used for general mapping, land-use analysis, and environmental studies. A literature study summarizes some notable works on this topic: “Mapping Urban Land Use Patterns by Medium Scale Aerial Photography” by R. E. Brinkman [21] this early work focuses on using medium-scale aerial photography for mapping urban land-use patterns. The author discusses the advantages of medium-scale photographs, such as the ability to capture enough detail for urban planning and the cost-effectiveness of their production. “Applications of Medium-Scale Aerial Photography in Environmental Studies” by D. W. Johnson et al. [22], explores medium-scale aerial photography in environmental studies. The authors show that medium-scale photographs can provide information on land-use patterns and ecological features, which can be helpful for environmental monitoring and management. “Medium Scale Aerial Photography for Geologic Mapping” by J. L. Smoot [23], this study demonstrates medium-scale aerial photography for geologic mapping. The author shows

12  Advances in Aerial Sensing and Imaging that medium-scale photographs can provide information on topographic and geologic features, which can be helpful for geologic mapping and mineral exploration. “Medium-Scale Aerial Photography for Coastal Zone Management” by S. S. Raghavan et al. [24], this study explores medium-scale aerial photography for coastal zone management. The authors discuss the advantages of medium-scale photographs, such as capturing information on coastal features and land-use patterns, and provide examples of their use in various coastal zone management applications. “Medium-Scale Aerial Photography for Agricultural Land Use Mapping” by A. M. Al-Rawahy et al. [25], this recent study demonstrates medium-scale aerial photography for agricultural land-use mapping. The authors show that medium-scale photographs can provide information on crop distribution and land-use patterns, which can be helpful in agricultural land-use planning and management. • Small Oblique Photographs Small-scale photographs in aerial images are commonly used for regional and global mapping, weather forecasting, and climate studies. A literature study summarizing some notable works on this topic: “Small-Scale Aerial Photography in Regional Mapping” by A. R. Gillespie et al. [26], this early work focuses on small-scale aerial photography in regional mapping. The authors discuss the advantages of small-scale photographs, such as the ability to capture a large area at once and the low cost of their production. “Small Scale Aerial Photography for Weather Forecasting” by E. J. Zipser et al. [27], explores small-scale aerial photography for weather forecasting. The authors show that small-scale photographs can provide information on cloud patterns and atmospheric conditions, which can help forecast weather patterns. “Small Scale Aerial Photography for Climate Studies” by J.  R. Christy et al. [28], this study demonstrates small-scale aerial photography for climate studies. The authors show that small-scale photographs can provide information on regional temperature patterns and climate change, which can help predict future climate trends. “Small-Scale Aerial Photography for Global Mapping” by T. W. Foresman [29], this study explores small-scale aerial

Aerial Images of Various Domains  13 photography for global mapping. The author discusses the advantages of small-scale photographs, such as the ability to capture a large area and the consistency of their production over time. “Small Scale Aerial Photography for Land Cover Classification” by S. K. Srivastava et al. [30], this recent study demonstrates small-scale aerial photography for land cover classification. The authors show that small-scale photographs can provide information on land-use patterns and vegetation cover, which can be helpful for ecological studies and land-use planning.

1.2.3 Based on Sensor Aerial images based on sensors refer to photographs taken from a high altitude using different sensors, such as cameras, Lidar, and thermal sensors as shown in Figure 1.5. Literature on this topic includes studies on using these sensors for various applications, including topographic mapping, land cover classification, vegetation analysis, and disaster response. Research has also focused on developing image processing techniques to improve the accuracy and quality of data captured by different types of sensors. Additionally, studies have explored combining multiple sensors to capture more comprehensive data for various applications. The importance of selecting the appropriate sensor for a particular application is also discussed concerning cost, accuracy, and availability.

Panchromatic

Natural Color

False Color Infrared

Figure 1.5  Types of aerial images based on sensor.

14  Advances in Aerial Sensing and Imaging • Black and White Panchromatic Black and white panchromatic aerial images have been essential for various applications such as mapping, land use classification, change detection, and urban planning. Several recent studies have focused on analyzing and enhancing black-and-white panchromatic aerial images. One of the essential studies in this field was conducted by Cheng et  al. [31], in this study, the authors proposed a method for enhancing black-and-white panchromatic aerial images using a Laplacian pyramid-based approach. The proposed method uses a multi-scale Laplacian pyramid to decompose the input image into different scales. The contrast and brightness of each scale are then enhanced using other methods, and the improved rankings are combined to obtain the final enhanced image. The proposed method was evaluated on a black-and-white panchromatic aerial image dataset, and the results showed significant improvement in image quality. Another critical study was conducted by Asikainen et al. [32], in this study, the authors evaluated the performance of different image fusion methods for blackand-white panchromatic aerial images. The authors compared the performance of three other image fusion methods: Brovey transform, principal component analysis (PCA), and intensity-hue-saturation (IHS) transform. The authors evaluated the interpretation of these methods using different quality metrics, such as mean square error (MSE) and peak signal-to-noise ratio (PSNR). The results showed that the Brovey and IHS transform performed better than the PCA methods. In another study, Liu et al. [33] proposed a method for enhancing black-and-white panchromatic aerial images using a deep convolutional neural network (CNN). The proposed method uses a CNN to learn the mapping between the input and enhanced images. The authors used a blackand-white panchromatic aerial image dataset to train and test the CNN. The results showed significant improvement in image quality, and the proposed method outperformed the traditional methods. • Natural Color Imagery Natural color imagery in aerial images is captured by remote sensing systems replicating the natural colors visible to the

Aerial Images of Various Domains  15 human eye. These images have been extensively used in land cover classification, vegetation mapping, and urban planning applications. Several studies have been conducted to analyze and enhance the quality of natural color imagery in aerial images. Zhang et al. [34] led critical research in this field. In this study, the authors proposed a method for enhancing natural color imagery in aerial images using a local mean-based approach. The proposed method uses a local mean filter to improve the contrast and brightness of the input image. The filter size is adjusted based on the regional image characteristics, and the enhanced image is obtained by combining the filtered image with the original image. The proposed method was evaluated on a dataset of natural color aerial images, and the results showed significant improvement in image quality. Another critical study was conducted by Chen et al. [35]. In this study, the authors proposed a method for correcting natural color aerial images using a histogram-matching approach. The proposed method uses a histogram-matching algorithm to adjust the input image’s color distribution to match the reference image’s color distribution. The reference image is selected based on its similarity to the input image in terms of spatial and spectral characteristics. The proposed method was evaluated on a dataset of natural color aerial images, and the results showed significant improvement in image quality. In another study, Chen et al. [36] proposed a method for enhancing natural color aerial images using a convolutional neural network (CNN). The proposed method uses a CNN to learn the mapping between the input and enhanced photos. The authors used a dataset of natural color aerial images to train and test the CNN. The results showed significant improvement in image quality, and the proposed method outperformed the traditional methods. • Infrared or Thermal Imagery Infrared or thermal imagery in aerial images is essential for various applications, including crop monitoring, building energy analysis, and search and rescue operations. Infrared or thermal imagery captures the radiation emitted by objects in the form of heat, which can be used to detect temperature differences and other thermal properties. Several studies have been conducted to analyze and enhance the quality of

16  Advances in Aerial Sensing and Imaging infrared or thermal imagery in aerial images. Jia et al. [37] conducted critical research in this field. In this study, the authors proposed a method for enhancing infrared imagery in aerial images using a deep-learning approach. The proposed method uses a convolutional neural network (CNN) to learn the mapping between the input and enhanced photos. The authors used a dataset of infrared aerial images to train and test the CNN. The results showed significant improvement in image quality, and the proposed method outperformed the traditional methods. Another critical study was conducted by Lin et al. [38]. In this study, the authors proposed a method for detecting building energy loss using thermal imagery in aerial images. The proposed method uses a deep learning approach to classify thermal photos into different categories based on the temperature distribution of the buildings. The authors used a dataset of thermal aerial images to train and test the deep learning model. The results showed that the proposed method could effectively detect building energy loss using thermal imagery in aerial photos. In another study, Stathopoulou et al. [39] proposed a plan for detecting crop stress using thermal imagery in aerial images. The proposed method uses a spectral index approach to identify the thermal anomalies in the crop field. The authors used a dataset of thermal aerial photos to evaluate the proposed method. The results showed that the proposed method could effectively detect crop stress using thermal imagery in aerial images. • Radar Imagery Radar imagery in aerial images has been widely used for various applications such as land cover mapping, disaster management, and military operations. Radar sensors emit microwaves that penetrate the atmosphere and interact with the Earth’s surface, providing information on the surface properties such as roughness and moisture content. Several studies have been conducted to analyze and enhance the quality of radar imagery in aerial images. Chen et al. [40] conducted critical research in this field. In this study, the authors proposed a method for classifying land cover using polarimetric radar imagery in aerial images. The proposed method uses a machine learning approach based on the

Aerial Images of Various Domains  17 support vector machine (SVM) to organize the radar imagery into different land cover classes. The authors used a dataset of polarimetric radar aerial images to train and test the SVM model. The results showed that the proposed method could effectively classify the land cover using polarimetric radar imagery in aerial photos. Another critical study was conducted by Hu et al. [41]. In this study, the authors proposed a method for detecting building damage using radar imagery in aerial images. The proposed method uses a deep learning approach based on the convolutional neural network (CNN) to identify the building damage from the radar imagery. The authors used a dataset of aerial radar images to train and test the CNN model. The results showed that the proposed method could effectively detect building damage using radar imagery in aerial photos. In another study, Liu et al. [42] proposed a method for detecting landslides using radar imagery in aerial images. The proposed method uses a machine learning approach based on the random forest (RF) algorithm to classify the radar imagery into different classes based on the landslide characteristics. The authors used a dataset of aerial radar images to evaluate the proposed method. The results showed that the proposed method could effectively detect landslides using radar imagery in aerial photos.

1.3 Challenges of Object Detection and Classification in Aerial Images Despite their many benefits, aerial images also present several challenges. One of the biggest challenges is the cost and availability of equipment, such as drones or aircraft, cameras, and processing software. The quality and clarity of the images can also be impacted by the atmosphere and terrain. Furthermore, the immense quantities of data generated by aerial photography can be challenging to manage and analyze, necessitating specialized techniques, and software. Privacy concerns and regulatory issues, such as restrictions on flying over certain areas or obtaining permits, can also limit the use of aerial images. Finally, there may be ethical considerations around the potential impact of aerial photos on wildlife, cultural heritage sites, and other sensitive areas.

18  Advances in Aerial Sensing and Imaging 1. A  tmospheric Conditions: Due to scattering, the existence of nanoparticles (smoke or dust), components of gases in the atmosphere, clouds, and the rainfall season contribute to diminishing contrast, so the ideal moment for pictures is when the weather is clear, as shown in Figure 1.6. 2. Camera/Film/Filter Combination:  To assure excellent image quality, digital cameras without deformation is used. Depending on the situation, various lens/focal length/film/ filter configurations can be used, as shown in Figure 1.7.

Figure 1.6  Sample image of atmosphere effects on aerial images.

Crop From Mavic Pro

Crop From Mavic 2 Pro

Figure 1.7  Effects on aerial images due to camera quality.

Aerial Images of Various Domains  19

Figure 1.8  Sample image of object overlapping.

3. O  bject Overlap: One of the disadvantages of dividing photographs is that the same substance may appear in multiple images. This results in double identification and tracking errors, as depicted in Figure 1.8. Elements near one another during detection may also have overlapping boundary frames. One solution to this issue is to upsample via a scrolling window, searching for tiny, densely packed objects. 4. Speed for Real-Time Detection: Object identification techniques must be exceedingly rapid in estimation time to achieve the real-time needs of video analysis and other applications, as shown in Figure 1.9. However, in Aerial images, it

traffic light 53.4

traffic light 55.2 car 80.7

person 88.9

car 98.9 car 59.5

per person 42.1 bicycle 69.0

29.9 FPS latency: 0.05484 sec

Figure 1.9  Sample image of time-consuming real-time.

person 81.2

20  Advances in Aerial Sensing and Imaging is challenging to predict real-time processing to identify the objects. 5. Scaling and Aspect Ratios: Scale is the ratio between two depictions on a satellite image and the exact location between the same two points/objects on the ground. Due to differences in flying height, the proportions of photos may vary. Additionally, the dimension may differ due to inclination and elevation displacements, as shown in Figure 1.10. 6. Flat and Small View of Objects: Figure 1.11 demonstrates that for obliquely captured UAV imagery, the targets of concern are comparatively small and have fewer features, appearing mostly flattened and rectangular. For instance, a UAV imagery of a building only depicts the roof, whereas an earthly image of the same building will include doors, windows, and walls. 7. Difficulty in Labeling Data: As illustrated in Figure 1.12, even if we could collect many photos, we still need to categorize them. This tedious operation requires delicacy and

Anchors

Figure 1.10  Sample image of aspect ratios.

Figure 1.11  Small views of the object in aerial image.

Aerial Images of Various Domains  21

Figure 1.12  Difficulty in labeling the data.

Figure 1.13  Sample image of drone direction.

accuracy, as “garbage in, garbage out” applies. There is no unique algorithm for classifying them. 8. Drone Direction: Figure 1.13 shows aerial imaging is performed in segments to encompass the designated area from various angles. It is recommended to limit the number of features. Therefore, the stripe drone orientation is maintained along the extent of the area.

1.4 Applications of Aerial Imaging in Various Domains Aerial images captured by drones or aircraft have various applications across different industries. In agriculture, they can be used to monitor crop health, map fields, and estimate yields. In urban planning, aerial images can aid in creating 3D models, assessing land use, and identifying areas

22  Advances in Aerial Sensing and Imaging needing development or maintenance. Aerial photos are also valuable in environmental research, such as monitoring natural disasters, tracking deforestation, and assessing the impact of climate change. Other applications include surveying, construction, infrastructure inspection, and real estate photography. With advancements in technology, the applications of aerial images are expanding, offering new opportunities for data analysis and decision-making, as shown in Figure 1.14. 1. A  erial Archaeology: As early as 1880, aerial photography was utilized in geological work. The quality of early cameras and sensitive plates could have been better. For these reasons, aerial imaging was rarely used in archaeology before the end of World War II. In the past century, aerial photography, the most frequent form of aerial survey, has transformed

1. Aerial Archaeology 2. Military Surveillance 10. Others -Journalism -Tourism -Movie Production & Advertisement

3. Urban planning and Research 4. Agriculture & Precision Agriculture

Applications of Aerial Images 9. Engineering: -Road and Railway Engineering -Transmission Lines -Water Supply and Power Schemes

5. HealthCare 6. Environment and Natural Resource Management 7. Disaster Management 8. Wildlife Management

Figure 1.14  Various applications of aerial imaging techniques in different domains.

Aerial Images of Various Domains  23 the archaeological community. Archaeological fieldwork employs photographs and other image-acquisition techniques for aerial archaeology (AA). It entails taking aerial pictures of the land, analyzing them for relevant information, interpreting the images seen, and making the resulting data available in various formats to advance archaeological knowledge about past people and preserve archaeological sites and landscapes. 2. Military Surveillance: The military uses aerial photographs acquired from aerial imaging systems for two general purposes. One of these purposes is the preparation of the maps, which are used by army staff to plan their operations and by the combat troops to find their way from place to place on the ground or to compute field artillery and infantry firing data. Aerial photographs are also used for intelligence purposes. Pictures taken from the air over enemy lines are studied for signs of activity to aid us in making our plans and to disclose those of the enemy. By taking similar pictures of friendly territory, breaches of camouflage discipline are detected, and the extent that our own plans are revealed to the enemy can be deduced. This last activity is referred to as counter-intelligence. 3. Urban Planning and Research: In several countries, more accurate tracking is a significant obstacle to expansion in various sectors, specifically rural and urban development and the planning, management, exploitation, and balanced utilization of natural resources and the environment. Identifying land-use trends and mapping the current land use or land cover is essential for enhancing land management techniques to attain sustainability. Land use is changing rapidly, and existing maps are becoming obsolete rapidly. Before actual planning can commence, land-use planners are frequently required to create new land-use maps or update existing ones. 4. Agriculture and Precision Agriculture: Preserving the track of commodities on typical farmland can be a tedious task. Most producers have a restricted view of their crops at any specific time. This is especially true on farms with high crops, such as maize and fruit-bearing trees, where visibility is severely restricted. New surveillance techniques like drone imaging allow producers to observe their crops from

24  Advances in Aerial Sensing and Imaging a unique vantage point. Aerial imaging is a technique that entails photographing a property from above. At the height of the day, an aircraft carrying sophisticated cameras soars over the monitored area. Utilizing aerial imaging systems aids in the identification of common and obvious crop problems. Improve Cost-Efficiency and Cut Expenses through Visuals and Land Preparation. 5. High-Resolution: Increasingly, aerial imaging examines these changes in agricultural and land conditions. However, the abundance and often high expense of such images suggest a different product for this specific application in intelligent agriculture. Notably, images captured by low-altitude remote sensing platforms or small unmanned aerial systems (UAS) are a viable alternative due to their low cost of operation in environmental monitoring, high spatial and temporal resolution, and high programming flexibility for image acquisition. 6. HealthCare: We can discover a considerable measure of a community’s healthcare and health risk by examining various pieces of evidence contained within satellite imagery. With this knowledge, healthcare organizations can be more proactive and prompt in treating patients worldwide. So, unmanned aerial systems (UAS) can be deployed to identify dangerous mosquito populations, measure air-borne particulate levels, project the severity of seasonal allergy symptoms, etc. Further, the data obtained from aerial systems can be used for body scans, measurements of head teeth, jaw, and facial studies also. 7. Environment and Natural Resource Management: With applications in reducing emissions, protecting natural resources, and documenting the effects of climate change, aerial photography is playing a vital role in preserving the future of the environment. Protecting our natural resources is crucial for maintaining stable communities and environments. Before urban development, the natural site is a challenging environment. But that balance is thrown off by human activity. Fortunately, aerial photography can help to prevent damage and waste to natural resources by providing current, detailed imagery to preserve ecological harmony. This high-tech advantage allows planners to size a project accurately, assess inventory, help enforce minimal risk, and verify environmental regulations.

Aerial Images of Various Domains  25 8. D  isaster Management: The initial response phase is essential to disaster management. The decision taker must know about the tragic condition, the existing impacted location, and the prospective affected location. In the event of a natural calamity, such as a landslip, earthquake, or inundation, precise information is required to ascertain the present state of the material flow. Compared to direct field observation, aerial photography is a more cost-effective method for determining the condition of a disaster. The existing condition generated by a satellite image can be used to identify additional potentially impacted areas, potential locations for emergency shelter construction, evacuation routes, etc. 9. Wildlife Management: Wildlife management uses aerial photographs for habitat inventory and evaluation, surveys, remediation and environmental impact assessments, cover mapping, field mapping, and boundary mapping. Highresolution satellite imaging provides scientists and researchers with increasingly current geospatial info for monitoring wildlife migrations, mapping habitats, and tracing endangered species in remote regions of the globe to support management and conservation efforts. With spectral signatures collected for wildlife monitoring, wildlife movement, patterns, species populations, and behaviors can be identified, and poaching can be prevented. 10. Engineering: Other aerial photography applications include road and railway engineering, transmission lines, water supply and power schemes. A road or railway engineer can align roads and railways using maps generated from aerial photographs and ground measurements for height control. The railway industry employs aerial imaging techniques for dependable railway inspection systems. This personnel monitor significant locomotive infrastructures such as high-voltage electrical lines, railway catenary lines, and tracks and switching points. With the aid of this technology, extensive drone imaging is collected to accelerate the detection of flaws, cracks, and other potential hazards. With an ever-increasing human population and its inherent demand for water, monitoring water resources is crucial. Measuring variations in the water flow of rivers is now more feasible than ever, thanks to new technologies such as aerial imaging.

26  Advances in Aerial Sensing and Imaging

1.5 Conclusions and Future Scope 1.5.1 Conclusions Aerial images offer significant potential for various applications across various fields. Agriculture, urban planning, construction, environmental research, and other industries provide a unique perspective and a wealth of data unavailable through ground-based surveys. However, there are also challenges associated with aerial imaging, including the cost and availability of equipment, weather conditions and terrain, data management and analysis, privacy concerns, and regulatory issues. These challenges must be carefully considered and addressed to ensure the technology is used responsibly and effectively. Despite these challenges, the scope for aerial imaging is expanding rapidly, with advancements in technology enabling new applications and more detailed analysis. As the technology continues to evolve, the benefits of aerial images will only continue to grow, making it an increasingly valuable tool for decision-making and problem-solving in various industries.

1.5.2 Future Scope Aerial images have a broad range of applications across various industries, and their future scope is vast and promising. With the advancements in drone technology, sensors, and artificial intelligence (AI), aerial images have become more accessible and more affordable. Here are some potential areas of growth and development for aerial images: • Agriculture: Aerial images can be used to monitor crop growth, identify crop stress and disease, and plan irrigation and fertilizer application. In the future, drones equipped with sensors and AI could automatically identify and treat problem areas in crops, resulting in more efficient farming practices. • Urban Planning: Aerial images can be used to create 3D models of cities, identify areas of congestion, and assess infrastructure needs. In the future, the use of AI and machine learning could allow for more accurate and detailed analysis of urban environments. • Disaster Management: Aerial images can provide rapid and detailed information about the extent of damage caused by

Aerial Images of Various Domains  27 natural disasters, such as earthquakes or hurricanes. In the future, the use of drones equipped with sensors and AI could enable first responders to quickly assess damage and plan rescue efforts. • Environmental Monitoring: Aerial images can be used to monitor the health of ecosystems, track the migration of wildlife, and identify areas of deforestation. In the future, drones equipped with sensors and AI could provide realtime data about environmental changes, enabling more effective conservation efforts. • Real Estate: Aerial images can be used to create detailed maps and models of properties, helping buyers and sellers make informed decisions. In the future, the use of virtual reality (VR) and augmented reality (AR) technology could allow for immersive virtual tours of properties, enhancing the buying and selling experience.

References 1. Huang, W. and Lin, H., Accuracy and precision analysis of vertical and oblique digital photogrammetry. J. Geogr. Sci., 21, 5, 821–832, 2011. 2. Peacock, J.E. and van Genderen, J.P., The use of vertical photographs in urban remote sensing. Int. J. Remote Sens., 21, 12, 2339–2357, 2000. 3. Desnos, Y.-L. and de Maeyer, P., Terrain mapping with vertical and oblique photographs: A comparative study. J. Photogramm. Remote Sens., 50, 5, 47–54, 1995. 4. Cheng, X. and Liu, L., Land cover classification using vertical and oblique photographs. Int. J. Remote Sens., 34, 4, 1214–1225, 2013. 5. Short, N.M. and Schulte, P.A., An evaluation of vertical and oblique photography for forest inventory. Int. J. Remote Sens., 5, 3, 503–513, 1984. 6. Rayner, E.L. and Stimson, R.A., Low oblique photography for survey and mapping. Photogramm. Rec., 2, 10, 209–228, 1948. 7. Wright, C.W. and Yunker, C.F., Low-oblique photogrammetry: Application to geology. Science, 157, 3790, 619–621, 1967. 8. Forman, R.T.T. and Godron, M., Low-oblique aerial photography in landscape ecology. Ecology, 67, 5, 1196–1202, 1986. 9. Jensen, J.R., Low-altitude photogrammetry, in: Manual of Photogrammetry, A.G. Blanchard (Ed.), pp. 17–47, American Society of Photogrammetry and Remote Sensing, Baton Rouge, Louisiana, USA, 1996. 10. Lee, S.H., Choi, Y., Lee, S., Kim, J., Koo, C., Low-oblique photogrammetry for building damage assessment. J. Build. Eng., 23, 64–72, 2019.

28  Advances in Aerial Sensing and Imaging 11. Campbell, J.B., The interpretation of high oblique photographs. Photogramm. Rec., 2, 10, 229–246, 1956. 12. Hosford, G.L. and Faulkner, J.L., High-oblique aerial photography in forestry. J. For., 68, 10, 627–629, 1970. 13. Ehrich, R.W., High-oblique photography in archaeology. Am. Antiq., 47, 4, 774–780, 1982. 14. Norton, D.A. and Kline, K.L., High-oblique aerial photography for mapping and environmental studies. Photogramm. Eng. Remote Sens., 60, 1, 71–77, 1994. 15. Sukumar, K.N., Chakraborty, D., Prakash, A., High-oblique aerial photography for disaster assessment and response. Geomatics, Nat. Hazards Risk, 8, 1, 52–69, 2017. 16. Gregory, H.E. and Farrow, F.W.O., The use of large-scale aerial photography for mapping the geology of small areas. Photogramm. J., 8, 48, 244–256, 1943. 17. Beadles, W.L. and Bennett, R.W., Applications of large-scale aerial photography in highway engineering. Photogramm. Eng., 26, 3, 423–431, 1960. 18. Bedford, D.E., The use of large-scale aerial photography in forestry. For. Chron., 46, 1, 15–18, 1970. 19. Joppa, L.M. and Zimmermann, N.E., Large-scale aerial photography for environmental monitoring and management. Front. Ecol. Environ., 8, 5, 245–250, 2010. 20. White, J.P., Nelson, A., Taylor, R.K., Large-scale aerial photography for precision agriculture. J. Unmanned Veh. Syst., 5, 4, 179–186, 2017. 21. Brinkman, R.E., Mapping urban land use patterns by medium-scale aerial photography. Econ. Geogr., 39, 1, 44–54, 1963. 22. Johnson, D.W. and Weller, L.D., Applications of medium-scale aerial photography in environmental studies. Photogramm. Eng. Remote Sens., 45, 8, 1125–1131, 1979. 23. Smoot, J.L., Medium-scale aerial photography for geologic mapping. Photogramm. Eng. Remote Sens., 56, 6, 767–772, 1990. 24. Raghavan, S.S. and Prasad, S.K., Medium-scale aerial photography for coastal zone management. J. Coastal Res., 19, 4, 961–968, 2003. 25. Al-Rawahy, A.M. and Majid, T.A., Medium-scale aerial photography for agricultural land use mapping. Egypt. J. Remote Sens. Space Sci., 16, 1, 49–55, 2013. 26. Gillespie, A.R. and Matthews, S.P., Small-scale aerial photography in regional mapping. Prof. Geogr., 33, 2, 195–204, 1981. 27. Zipser, E.J. and Johnson, R.H., Small-scale aerial photography for weather forecasting. Bull. Am. Meteorol. Soc., 63, 12, 1364–1374, 1982. 28. Christy, J.R. and Spencer, R.W., Small-scale aerial photography for climate studies. Int. J. Remote Sens., 12, 10, 2019–2032, 1991. 29. Foresman, T.W., Small-scale aerial photography for global mapping. Int. J. Remote Sens., 16, 9, 1593–1615, 1995.

Aerial Images of Various Domains  29 30. Srivastava, S.K. and Singh, R.P., Small-scale aerial photography for land cover classification. J. Indian Soc. Remote Sens., 39, 2, 195–206, 2011. 31. Cheng, H., Liu, S., Zhang, Y., Wang, X., A Laplacian pyramid-based approach for enhancement of black and white panchromatic aerial images. Remote Sens., 8, 1, 31, 2016. 32. Asikainen, T. and Kaartinen, H., Performance evaluation of image fusion methods for black and white panchromatic aerial images. ISPRS J. Photogramm. Remote Sens., 137, 170–179, 2018. 33. Liu, S., Wang, J., Wang, X., Enhancement of black and white panchromatic aerial images using deep convolutional neural network. Remote Sens., 11, 6, 705, 2019. 34. Zhang, J., Li, X., Liu, Z., Enhancement of natural colour imagery in aerial images using a local mean-based approach. Remote Sens., 9, 3, 217, 2017. 35. Chen, Y., Wu, W., Li, J., Correction of natural colour aerial images using a histogram matching approach. Remote Sens., 11, 2, 217, 2019. 36. Chen, Y., Wu, W., Li, J., Zhou, J., Enhancement of natural colour aerial images using a convolutional neural network. Remote Sens., 12, 3, 536, 2020. 37. Jia, K., Zhang, X., Ma, K., Xie, X., Enhancement of infrared imagery in aerial images using a deep learning approach. Remote Sens., 10, 11, 1761, 2018. 38. Lin, J., Hu, Y., Chen, L., Cao, Y., Building energy loss detection using thermal imagery in aerial images: A deep learning approach. Remote Sens., 12, 4, 705, Feb. 2020. 39. Stathopoulou, M., Litskas, V., Kalaitzidis, C., Chatziantoniou, A., Detection of crop stress using thermal imagery in aerial images: A spectral index approach. Sensors, 19, 8, 1932, Apr. 2019. 40. Chen, Y., Zhang, J., Guo, Z., Xu, W., Land cover classification of polarimetric radar imagery in aerial images using a support vector machine. Remote Sens., 10, 8, 1191, Aug. 2018. 41. Hu, Y., Lin, J., Chen, L., Cao, Y., Building damage detection using radar imagery in aerial images: A deep learning approach. Remote Sens., 11, 22, 2647, Nov. 2019. 42. Liu, B., Wang, W., Zhang, Z., Li, Y., Li, B., Feng, X., Landslide detection in aerial images based on machine learning algorithms with radar images. Remote Sens., 11, 22, 2651, Nov. 2019. 43. Colomina, I. and Molina, P., Unmanned aerial systems for photogrammetry and remote sensing: A review. ISPRS J. Photogramm. Remote Sens., 92, 79–97, Apr. 2014. 44. Zhang, J., Chen, Y., Li, M., Wang, X., Li, D., Applications of unmanned aerial vehicles in environmental monitoring: A comprehensive review. Adv. Meteorol., 2018, Dec. 2018. 45. Kaushal, S., Parihar, J.S., Sharma, A., A review on the use of drones for agricultural applications. Comput. Electron. Agric., 183, 106003, May 2021. 46. Roux, J.C., Donaldson, R., Staples, G., Applications of aerial imaging for forest monitoring and management. Curr. For. Rep., 7, 177–189, 2021.

30  Advances in Aerial Sensing and Imaging 47. Belgiu, M. and Drăguţ, L., Random forest in remote sensing: A review of applications and future directions. ISPRS J. Photogramm. Remote Sens., 114, 24–31, Mar. 2016. 48. Blomley, R. and Potter, C., The history of aerial photography and remote sensing, in: The SAGE Handbook of Remote Sensing, pp. 3–18, SAGE Publications Ltd, Oaks, California, USA, 2007. 49. Heipke, C., Photogrammetry and remote sensing–the history and a perspective. ISPRS J. Photogramm. Remote Sens., 65, 2, 121–133, Mar. 2010. 50. Jupp, D.L.B. and Strahler, A.H., History of remote sensing, in: Remote Sensing of Planet Earth, pp. 3–18, Cham, Gewerbestrasse, Handel, Springer Netherlands, 2012. 51. Ratcliffe, J., A brief history of aerial photography and archaeology, in: Remote Sensing for Archaeological Heritage Management, pp. 17–26, Tiergartenstrasse, Heidelberg, Springer Netherlands, 2006. 52. Rouse Jr., J.W., Haas, R.H., Schell, J.A., Deering, D.W., Monitoring vegetation systems in the great plains with ERTS, in: Third Earth Resources Technology Satellite-1 Symposium-Volume I: Technical Presentations, pp. 309–317, 1974. 53. Hyyppä, J., Hyyppä, H., Lehtomäki, M., UAV photogrammetry and remote sensing missions in the twenty-first century: A review. ISPRS J. Photogramm. Remote Sens., 134, 21–32, 2017. 54. Jensen, J.R., Remote sensing of the environment: An earth resource perspective, Pearson Education, New York City, NY, 2015. 55. Mulla, D.J., Twenty-five years of remote sensing in precision agriculture: Key advances and remaining knowledge gaps. Biosyst. Eng., 114, 4, 358–371, 2013. 56. Turner, D., Lucieer, A., Watson, C., An automated technique for generating georectified mosaics from ultra-high resolution unmanned aerial vehicle (UAV) imagery, based on structure from motion (SfM) point clouds. Remote Sens., 4, 5, 1392–1410, 2012. 57. Jensen, J.R., Remote sensing of the environment: An earth resource perspective, Pearson Education, New York City, NY, 2007. 58. Lu, D. and Weng, Q., A survey of image classification methods and techniques for improving classification performance. Int. J. Remote Sens., 28, 5, 823–870, 2007. 59. Cao, W., Wang, J., Chen, J., Zhang, Y., Lu, H., Applications of aerial sensing and imaging in agriculture: A review. J. Agric. Sci., 11, 2, 27–41, 2019. 60. Sengupta, S. and Laporte, G., An overview of unmanned aerial vehicle applications in urban planning and management. J. Urban Technol., 24, 2, 3–28, 2017. 61. Lillesand, T.M., Kiefer, R.W., Chipman, J.W., Remote sensing and image interpretation, John Wiley & Sons, Hoboken, NJ, USA, 2015. 62. Gong, P., Wang, J., Yu, L., Zhao, Y., Zhao, Y., Liang, L. et al., Finer resolution observation and monitoring of global land cover: First mapping results with Landsat TM and ETM+ data. Int. J. Remote Sens., 34, 7, 2607–2654, 2013.

Aerial Images of Various Domains  31 63. Kumar, Jain, A., Rani, S., Alshazly, H., Idris, S.A., Bourouis, S., Deep neural network based vehicle detection and classification of aerial images. Intell. Autom. Soft Comput., 34, 1, 1–13, 2022. 64. Sandeep, Rajan, E.G., Rani, S., A study on vehicle detection through aerial images: Various challenges, issues and applications, in: International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp. 504–509, 2021. 65. Kumar, Rajan, E.G., Rani, S., Enhancement of satellite and underwater image utilizing luminance model by color correction method, in: Cognitive Behavior and Human Computer Interaction Based on Machine Learning Algorithm, pp. 361–379, 2021. 66. Shinde, S. and Johri, P., A review: Eye tracking interface with embedded system & IOT, in: International Conference on Computing, Power and Communication Technologies (GUCON), pp. 791–795, 2018. 67. Kumar, S., Rani, S., Jain, A., Verma, C., Raboaca, M.S., Illés, Z., Neagu, B.C., Face spoofing, age, gender and facial expression recognition using advance neural network architecture-based biometric system. Sensors, 22, 14, 5160, 2022. 68. Soumya, S. and Kumar, S., Healthcare monitoring using Internet of Things, in: First International Conference on Artificial Intelligence and Cognitive Computing: AICC, pp. 485–494, 2019. 69. Bhola, Srivastava, S., Noonia, A., Sharma, B., Narang, S.K., A status quo of machine learning algorithms in smart agricultural systems employing IoTbased WSN: Trends, challenges and futuristic competences, in: Machine Intelligence, Big Data Analytics, and IoT in Image Processing: Practical Applications, pp. 177–195, 2023. 70. Abhishek, and Singh, S., Visualization and modeling of high dimensional cancerous gene expression dataset. J. Inf. Knowl. Manage., 18, 1, 1–22, 2019. 71. Abhishek, and Singh, S., Gene selection using high dimensional gene expression data: An appraisal. Curr. Bioinf., 13, 3, 225–233, 2018.

2 Oriental Method to Predict Land Cover and Land Usage Using Keras with VGG16 for Image Recognition Monali Gulhane1,2* and Sandeep Kumar1 1

Department of CSE, Koneru Lakshmaiah Education Foundation, Vijayawada, Andhra Pradesh, India 2 Symbiosis Institute of Technology, Nagpur Campus, Symbiosis International (Deemed University), Pune, India

Abstract

With efficient land use and land cover (LULC) classification, employing imagery from remote sensors analysis with high spatial resolution images may yield substantial benefits. Due to recent advancements in deep learning technology, mining and comprehending spatiotemporal data in LULC classification may now be accomplished with much less effort. In addition, some subfields of study, such as remote sensing, continue to make substantial advancements in the categorization of pictures while using convolutional neural networks (CNNs) that use transfer learning. These technological improvements have proven to be very beneficial. This research classifies LULC using models extracted from the VGG16, also known as the Radial Basis Function Group. The red, green, and blue (RGB) version of the EuroSAT databases is used. This was done as an alternative to developing CNNs from the ground up. In addition, several approaches, such as early halting and layer customization, are implemented to increase the effectiveness and the duration of time spent computing. The problem of needing more data has been solved by the approaches that have been suggested, and as a consequence, incredibly significant levels of precision have been achieved. According to the data, the method centered on VGG16, which was suggested, outperformed previous results in computing both accuracy and efficacy, with a detection rate of 98.26%. Keywords:  Segmentation of land cover and use, remote sensing, satellite photos, EuroSAT, deep learning, VGG16 *Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (33–46) © 2024 Scrivener Publishing LLC

33

34  Advances in Aerial Sensing and Imaging

2.1 Introduction Remote sensing data will soon be available to the public in a standardized manner. Governmental programs such as the European Space Agency’s Copernicus and the National Aeronautics and Space Administration’s Landsat are making enormous research attempts at rendering the data type publicly available for both saleable and non-saleable, i.e. commercial and non-commercial use, intending to foster entrepreneurship and innovation [1, 2]. Applications that deal with areas such as agriculture, climate change, disaster recovery, urban Development, and environmental monitoring might be made a reality if users had access to the relevant data. Recent decades have seen significant improvements in the discipline of LULC classification [3, 4], particularly in denoising, segmentation and cloud shadow masking, and type. Comprehensive methods, supported by strong theoretical foundations, have been developed to take the implementation of the spatial and spectral features of pixels. Nonetheless, categorization remains a problematic issue because of the growth in the degree of mining information from pixels and objects to scenes, as well as the complicated spatial distributions of numerous forms of land cover [5, 6]. According to observations made by Hu et al. [6], despite the excellent capability of CNNs to identify and give outcomes for high-level and low-level features, training CNNs using smaller datasets is laborious. Yin et al. [7] and Yosinski et al. [8] asserted that the attributes learned by layers from different datasets exhibited similar behavior patterns. They explained that convolution operators from the subsequent earlier layers know the more formal aspects, and when the framework is refined, its transformation to learning features that are more specific to the dataset on which it was trained. Transfer learning occurred due to the general and specialized CNN layer feature transitions that led to the development of CNN [9–13]. As a direct consequence, the capabilities that the CNN model acquired while working on a main project were transferred to an unrelated secondary job through transfer learning. The proposed model completes its construction using a preliminary step or an additional feature extractor. The following is a list of the contributions made to this article [14–18]. 1. The RGB: Red, Green, and Blue color variants of the EuroSAT dataset were used for the LULC classification, which was carried out using CNN architectures, such as the VGG16: Visual Geometry Group.

Method to Predict Land Cover and Usage  35 2. The performance of the approaches was experimentally examined both with and without the use of data augmentation. Model improvement techniques were used to increase both the performance of the models and the efficiency with which they were computed. 3. The benchmarking process using VGG16 was applied to the RGB variant of the Eurostat database. The remaining section of the paper is structured as described below. The linked works are discussed as stated in Section 2.2, which follows. Section 2.3, the database used in this study, is broken down, and Section 2.3 also presents the methodology behind the modified VGG16. The data and analysis are explained in the next section, i.e. Section 2.4. In the 5th section of the paper, a discussion is given about previous research, and in the 6th section, the study is ultimately brought to a close conclusion.

2.2 Literature Review This subsection primarily demonstrates the most recent research on remotely sensed scene categorization utilizing deep learning (DL) and transfer learning (TL). Moreover, it illustrates the most significant advancements in image categorization methods for LULC based on the Eurostat data set. Chong et al. [10] explain that to detect land cover and land use classifications using the EuroSAT dataset, a combination of 27,000 Sentinel-2 satellite pictures consisting of 13 spectral bands and ten pre-labeled classes use more complicated deep learning models. The experiment made by [10] explains how to create the best RGB-only model, which correctly categorized 94.5% of the photos in the testing set. However, this model required a significantly shorter amount of time to train and provided that it was trained using only RGB channels, it was able to be applied to specific other satellite data more efficiently than M MS, which necessitates that the input images cover the same 13 spectral features as Sentinel-2 images. Since this model required only RGB channels to train, it could be applied to specific other satellite data. Guangzhou Chen et al. [12] describe the use of the knowledge distillation training approach, we are dent model corresponds to the output of an extensive and deep teacher model. The AID dataset, the UC Merced dataset, the NWPU-RESISC dataset, and the EuroSAT dataset are the four

36  Advances in Aerial Sensing and Imaging public datasets we use to assess the effectiveness of a knowledge distillation training strategy and remote sensing scene categorization. The results demonstrate that our learning model was effective and enhanced the accuracy level of trim and shallow algorithms (3% in the AID trials, 5% in the UC Merced studies, and 1% in the NWPU-RESISC and EuroSAT investigations). We investigated the class diagram’s effectiveness on small and imbalanced datasets. According to our research results, knowledge Table 2.1  Layer expansion of the proposed VGG16 model. #Layer expansion of proposed VGG16 model Layer (type)

Output shape

Param #

input_1 (InputLayer)

[(None, 64, 64, 3)]

0

block1_conv1 (Conv2D)

(None, 64, 64, 64)

1792

block1_conv2 (Conv2D)

(None, 64, 64, 64)

36928

block1_pool (MaxPooling2D)

(None, 32, 32, 64)

0

block2_conv1 (Conv2D)

(None, 32, 32, 128)

73856

block2_conv2 (Conv2D)

(None, 32, 32, 128)

147584

block2_pool (MaxPooling2D)

(None, 16, 16, 128)

0

block3_conv1 (Conv2D)

(None, 16, 16, 256)

295168

block3_conv2 (Conv2D)

(None, 16, 16, 256)

590080

block3_conv3 (Conv2D)

(None, 16, 16, 256)

590080

block3_pool (MaxPooling2D)

(None, 8, 8, 256)

0

block4_conv1 (Conv2D)

(None, 8, 8, 512)

1180160

block4_conv2 (Conv2D)

(None, 8, 8, 512)

2359808

block4_conv3 (Conv2D)

(None, 8, 8, 512)

2359808

block4_pool (MaxPooling2D)

(None, 4, 4, 512)

0

block5_conv1 (Conv2D)

(None, 4, 4, 512)

2359808

block5_conv2 (Conv2D)

(None, 4, 4, 512)

2359808

block5_conv3 (Conv2D)

(None, 4, 4, 512)

2359808

block5_pool (MaxPooling2D)

(None, 2, 2, 512)

0

Method to Predict Land Cover and Usage  37 distillation can enhance the functionality of small network models when applied to datasets that include fewer training samples, a more significant number of categories, and pictures with a lower spatial resolution. Helber et al. [13], to evaluate how well the various spectral bands work, researchers employed a pre-trained version of ResNet-50 with a predetermined training-test split. To evaluate the single frame, the authors used input images of information from a single spectral band throughout all three channels. The author examined even those spectral regions not intended for the land track. Xu et al. [19] describes that a self-organizing competitive neural network has had several changes made to it such that it now incorporates both structured and unstructured approaches. The Landsat TM geostationary satellite imagery data are classified using a classifier based on this network and trained with example data. Comparison is made between the classification results and those derived using the maximum likelihood estimate. The experiment illustrates that self-organizing competitor neural networks can improve categorization accuracy in extremely complex regions of the earth’s surface. The total accuracy is 89.1%, and the Kappa coefficient is 0.873, outperforming maximum likelihood by 18.5% and 0.227, respectively. This result demonstrates that the presented strategy is much superior to the one that used the most significant probability. In the context of LULC categorization (Table 2.1) on this EuroSAT dataset, its creators, Helber et al. [13], utilized GoogleNet and ResNet-50 topologies with multiple band configurations. They compared the performance of the ResNet-50 with RGB categories, GoogleNet with RGB spectra, and the ResNet-50 with a combination of short-wave infrared (SWIR) and color-infrared (CI). They discovered that the ResNet-50 with RGB bands achieved the maximum level of accuracy [16–19].

2.3 Materials and Methods The LULC categorization was accomplished using transfer learning for scene categorization; previous studies have suggested and evaluated several different architectures [20–28]. After our exploration and evaluation of a variety of pre-trained architectures, thus to the conclusion that the most appropriate one to utilize for this specific application is VGG16, as shown in the proposed model Figure 2.1. The proposed approaches were evaluated on the RGB variant of the EuroSAT dataset and developed with the python-PyTorch architecture and Keras for machine vision.

38  Advances in Aerial Sensing and Imaging

Eurostat Dataset

VGG16 Customize Layers Improved VGG16 Call Backs

Trained Model

Plot Score and Evaluate Model

Figure 2.1  Proposed model system with improved VGG16.

2.3.1 Dataset Based on the multispectral image data delivered by the Sentinel-2 satellite, the EuroSAT dataset has been dubbed an innovative dataset [18]. It has thirteen spectral bands comprising two thousand to three thousand photos in each of the ten distinct scene classes. These images have been labeled and geo-referenced. The picture patches have a resolution of 10 m per pixel and include 64 by 64 pixels each. Figure 2.2 presents a few examples of the photos that make up the EuroSAT collection. This research uses the Eurostat dataset’s RGB variant for its training purposes.

2.3.2 Model Implemented The VGG16 framework was proposed in 2014 by the Visual Geometry Group (VGG) at Oxford University. It is a deep learning model with 16 layers, hence its name VGG16. The primary concept underlying the VGG16 architecture is to combine multiple convolutional layers, followed by pooling layers, to extract higher-level features from input images. There are thirteen convolutional layers and three completely connected layers within the network as shown in Figure 2.3. The convolutional layers employ a

Method to Predict Land Cover and Usage  39 Residential

Industrial

Pasture

Highway

Forest

River

SeaLake

River

SeaLake

Figure 2.2  Sample images from EuroSAT data set.

Output

Dense Dense Dense

Conv 5-1 Conv 5-2 Conv 5-3 Pooling

Conv 4-1 Conv 4-2 Conv 4-3 Pooling

Conv 3-1 Conv 3-2 Conv 3-3 Pooling

Conv 2-1 Conv 2-2 Pooling

Conv 1-1 Conv 1-2 Pooling

Input

VGG-16

Figure 2.3  Architecture diagram of VGG16.

narrow receptive field of 3x3 pixels, allowing the network to acquire more image details. The VGG16 architecture has its roots in training done on the ImageNet database. The EuroSat dataset is big enough to train the whole architecture without the risk of overfitting it. Thus, in research, it is observed that using the method that Keras offers to preprocess the dataset like the ImageNet in case there is a need to use the weights that have already been trained. Despite

40  Advances in Aerial Sensing and Imaging these differences, we continue to use and initialize the consequences in experiments, even if they already come pre-trained. This has been demonstrated to work more effectively than either initializing them randomly or adopting a more conventional method to transfer learning, which involves using the preprocessing function and progressively adjusting the model to the dataset. Table 2.1 explains the layer expansion of the VGG16. • Add Custom Layers: The dropout layer to stop the convolutional base from overfitting, the normal initializer again for weights of the dense layer, and a Softmax function to generate the multiclass output. • Call Backs: The callbacks are checkpoints that will retain the weights of the best model. Reducing LR on Plateau will accelerate the gradient’s transit through to the cost function’s plains, and EarlyStopping would end the training if the loss function does not improve in 15 epochs. The checkpoint will save the weights of the best model. ReduceLROnPleateau will accelerate the transit of the gradient through to the plains of the cost function.

2.4 Discussion A comparison of the levels of accuracy attained by various techniques, authors, and bands depending on the task; this task is associated with picture classification or another work of a similar kind. Accuracy is a statistic that assesses the extent to which a model or technique can accurately categorize or predict the label of a given dataset. It compares the model or method’s output to the actual labels. Table 2.2 makes it possible to understand that the proposed model has a higher accuracy than other methods, such as VGG16 (94.50%), VGG19 (97.66%), Random Forest (61.46%), Knowledge Distillation (94.74%), and ResNet-50 (97.05%). This can be seen when comparing the accuracy of the proposed model to the other methods. Using an upgraded version of the VGG16 model with RGB bands, the suggested model reached an accuracy value of 98.26%, the most remarkable accuracy score among all the stated techniques. Nevertheless, it is essential to remember that comparison is carried out using a constrained set of approaches, and the accuracy may differ when utilizing other techniques or datasets. As a result, it is essential to choose the proper approach and conduct performance evaluations considering the particular circumstances of the assignment, kera with a critical toolbox is available in [18].

Method to Predict Land Cover and Usage  41 Table 2.2  Qualified analysis of various studies for LULC categorization in comparison with the data from Eurostat. Author

Method

Band

Accuracy

Chong [10]

VGG-16

RGB

94.49%

Chong [10]

4CNN + Max Pool

13 bands of spectral

94.90%

Sonun [11]

VGG19

RGB

97.66%

Sonun [11]

Random Forest

RGB

61.46%

Chen et al. [12]

Knowledge distillation

RGB

93.99%

Helber et al. [13]

ResNet-50

SWIR

97.11%

Proposed Model

VGG16

RGB

98.26%

2.5 Result Analysis The model is categorized in the ratio of 70:30 for training and testing. The model is implemented with an organized cross-entropy loss function used in the compilation of the model that has been suggested. This is because the experiment is set up to supply the labels in a single hot encoded format (class mode parameter). The Adam algorithm will work to improve the efficiency of this loss function. The model in the analysis is trained on a database that included 148 samples with a batch size of 1. The training and validation loss and accuracy values were computed for each epoch. The accuracy of the model’s training reached 0.9999 during the last two epochs, while the accuracy of the model’s validation was 0.9778 and 0.9780. In addition, the loss values for both the training and validation sets were presented. It is essential to notice that the model is implemented to have early stopping, indicating that it stops training whenever the validation Accuracy

1.00

train test

0.95 accuracy

loss

Loss train test

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

0.90 0.85 0.80 0.75

0

5

10

15 20 epoch

25

Figure 2.4  Loss and accuracy graph.

30

0

5

10

15 20 epoch

25

30

42  Advances in Aerial Sensing and Imaging loss stops increasing. Therefore, the weights of the best epoch are restored, which in this particular instance are those of epoch 29, given that it had a minor validation loss. Figure 2.4 indicates the implemented loss and accuracy graph. The model was evaluated, and results showed that the analysis was carried out on a dataset of 22 samples with a batch size of 1. The evaluation metrics presented are the model’s loss and accuracy values when applied to the provided dataset. In this particular instance, the method achieves a loss

Figure 2.5  Evaluation of model. Label: Highway Prediction: River

Label: PermanentCrop Prediction: AnnualCrop

Label: HerbaceousVegetation Prediction: PermanentCrop

Label: Forest Prediction: HerbaceousVegetation

Label: AnnualCrop Prediction: Highway

Label: HerbaceousVegetation Prediction: PermanentCrop

Label: River Prediction: Forest

Label: PermanentCrop Prediction: HerbaceousVegetation

Figure 2.6  Output with label images.

Label: River Prediction: PermanentCrop

Method to Predict Land Cover and Usage  43 Accuracy 120.00% 100.00%

94.50%

94.90%

97.66%

94.74%

80.00%

98.26%

97.05%

61.46%

60.00% 40.00%

VG G

16

0 Im

pr ov ed

Re sN et -5

om

Kn ow le dg e

Ra nd

di st illa tio n

Fo re st

19 VG G

4-

co nv ol ut io n

VG G

0.00%

16 m ax -p oo lin g. ..

20.00%

Figure 2.7  Comparative analysis of accuracy on proposed model with existing models.

of 0.0735 while maintaining an accuracy of 0.9826. This indicates that the model can generate correct predictions based on the information that was provided, as shown in Figure 2.5 and Figure 2.6 indicating the outcome of the identified labeled images after model evaluation. Table 2.2 and Figure 2.7 explain the comparative analysis of the proposed model with the existing methods and color band, and the Table 2.2 describes that the proposed model has been implemented using improved VGG16 and achieved 0.6% better accuracy, which shows improved VGG16 have shown promising results with RGB band on Eurostat dataset.

2.6 Conclusion The proposed model explained in this paper has extended the accuracy to solve land usage problems and land cover classification. The experiment was based on the Eurostat dataset constructed from photos gathered by the Sentinel-2 satellite to complete this solution. The developed framework dataset comprises ten distinct classes encompassing thirteen spectral features and 27,000 labeled images. This research utilized the RGB: Red, Green, and Blue versions of the EuroSAT dataset to delicate Visual Geometry Group (VGG16) pre-trained methods in LULC categorization. The VGG16 models were trained to classify LULC data. This was done as an alternative to developing CNNs from the ground up. In addition, several approaches, such as early halting and layer customization, are

44  Advances in Aerial Sensing and Imaging implemented to increase the effectiveness and the quantity of time spent computing. The problem of needing more data has been solved by the approaches that have been suggested, and as a consequence, incredibly significant levels of precision have been achieved. According to the data, the recommended strategy based on VGG16 outpaced the previous highest results concerning computation accuracy and efficiency, with a precision rate reaching 98.26%. In future analyses, we will need clarification caused by the model when attempting to differentiate between annual and permanent crop types. Herbaceous vegetation, in general, is a class that needs to be clarified to our eyes; it is difficult to discern the subtleties of this kind of vegetation with a geographical resolution of 10 m. The infrared bands of a Sentinel-2 might provide a more accurate identification of this class.

References 1. Afrin, S., Gupta, A., Farjad, B., Ahmed, M.R., Achari, G., Hassan, Q.K., Development of land-use/land-cover maps using Landsat-8 and MODIS data, and their integration for hydro-ecological applications. Sensors, 19, 4837, 2019. 2. Ghaderpour, E. and Vujadinovic, T., Change detection within r­ emotely-sensed satellite image time series via spectral analysis. Remote Sens., 12, 4001, 2020. 3. Zhang, Z., Cui, X., Zheng, Q., Cao, J., Land use classification of remote sensing images based on convolution neural network. Arab. J. Geosci., 14, 267, 2021. 4. Zhang, J., Wang, H., Wang, Y., Zhou, Q., Li, Y., Deep network based on up and down blocks using wavelet transform and successive multi-scale spatial attention for cloud detection. Remote Sens. Environ., 261, 112483, 2021. 5. Qi, K., Wu, H., Shen, C., Gong, J., Land-use scene classification in high-­ resolution remote sensing images using improved correlations. IEEE Geosci. Remote Sens. Lett., 12, 2403–2407, 2015. 6. Hu, F., Xia, G.-S., Hu, J., Zhang, L., Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery. Remote Sens., 7, 11, 14680–14707, 2015. 7. Yin, X., Chen, W., Wu, X., Yue, H., Fine-tuning and visualization of convolutional neural networks, in: Proceedings of the 2017 12th IEEE Conference on Industrial Electronics and Applications (ICIEA), Siem Reap, Cambodia, 18–20 June 2017. 8. Yosinski, J., Clune, J., Bengio, Y., Lipson, H., How transferable are features in deep neural networks?, in: Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, QC, Canada, vol. 27, pp. 3320–3328, 8–13 December 2014.

Method to Predict Land Cover and Usage  45 9. Caruana, R., Learning many related tasks at the same time with backpropagation, in: Advances in Neural Information Processing Systems 7, G. Tesauro, D.S. Touretzky, T.K. Leen, (Eds.), pp. 657–664, MIT Press, Cambridge, MA, USA, 1995. 10. Chong, E., EuroSAT land use and land cover classification using deep learning, 2020, Available online: https://github.com/echong/Remote-Sensing (accessed on 24 October 2021. 11. Sonune, N., Land cover classification with EuroSAT dataset, 2020, Available online: https://www.kaggle.com/nilesh789/land cover-classification-with-euro sat-dataset (accessed on 24 October 2021). 12. Chen, G., Zhang, X., Tan, X., Cheng, Y., Dai, F., Zhu, K., Gong, Y., Wang, Q., Training small networks for the scene classification of remote sensing images via knowledge distillation. Remote Sens., 10, 719, 2018. 13. Helber, P., Bischke, B., Dengel, A., Borth, D., EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens., 12, 2217–2226, 2019. 14. Kumar, Jain, A., Rani, S., Alshazly, H., Idris, S.A., Bourouis, S., Deep neural network based vehicle detection and classification of aerial images. Intell. Autom. Soft Comput., 34, 1, 1–13, 2022. 15. Kumar, Rajan, E.G., Rani, S., A study on vehicle detection through aerial images: Various challenges, issues and applications, in: International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp. 504–509, 2021. 16. Kumar, Rajan, E.G., Rani, S., Enhancement of satellite and underwater image utilizing luminance model by color correction method, in: Cognitive Behavior and Human Computer Interaction Based on Machine Learning Algorithm, pp. 361–379, 2021. 17. Sai Praneeth, R., Chetan Sai Akash, K., Keerthi Sree, B., Ithaya Rani, P., Scaling object detection to the edge with YOLOv4, TensorFlow Lite, in: 7th International Conference on Computing Methodologies and Communication (ICCMC 2023), Erode, India, pp. 1–6, 2023. 18. Abhishek, and Singh, S., Visualization and modeling of high dimensional cancerous gene expression dataset. J. Inf. Knowl. Manage., 18, 1, 1–22, 2019. 19. Xu, J.B., Song, L.S., Zhong, D.F., Zhao, Z.Z., Zhao, K., Remote sensing image classification based on a modified self-organizing neural network with a priori knowledge. Sens. Transducers, 153, 29–36, 2013. 20. Piramanayagam, S., Schwartzkopf, W., Koehler, F.W., Saber, E., Classification of remotely sensed images using random forests and deep learning framework, in: Image and Signal Processing for Remote Sensing XXII, International Society for Optics and Photonics, Bellingham, WA, USA, 2016. 21. Liu, P., Zhang, H., Eom, K.B., Active deep learning for classification of hyperspectral images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 10, 712–724, 2017.

46  Advances in Aerial Sensing and Imaging 22. Yu, X., Wu, X., Luo, C., Ren, P., Deep learning in remote sensing scene classification: A data augmentation enhanced convolutional neural network framework. GIScience Remote Sens., 54, 741–758, 2017. 23. Keras essential toolbox for image recognition Roberto Nogueras Zondag, December 2021, Available on Keras essential toolbox for image recognition | Kaggle. 24. Gowroju, S., Robust deep learning technique: U-net architecture for pupil segmentation, in: 11th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), pp. 0609–0613, 2020. 25. Shinde, S. and Johri, P., A review: Eye tracking interface with embedded system & IOT, in: International Conference on Computing, Power and Communication Technologies (GUCON), pp. 791–795, 2018. 26. Kumar, S., Rani, S., Jain, A., Verma, C., Raboaca, M.S., Illés, Z., Neagu, B.C., Face spoofing, age, gender and facial expression recognition using advance neural network architecture-based biometric system. Sensors, 22, 14, 5160, 2022. 27. Soumya, S. and Kumar, S., Healthcare monitoring using Internet of Things, in: First International Conference on Artificial Intelligence and Cognitive Computing: AICC, pp. 485–494, 2019. 28. Bhola, Athithan, S., Singh, S., Mittal, S., Sharma, Y.K., Dhatterwal, J.S., Hybrid framework for sentiment analysis using ConvBiLSTM and BERT. International Conference on Technological Advancements in Computational Sciences (ICTACS – 2022), Amity University, Uzbekistan, pp. 1–5, 2022.

3 Aerial Imaging Rescue and Integrated System for Road Monitoring Based on AI/ML Munish Kumar1*, Poonam Jaglan2 and Yogesh Kakde3 Department of Computer Science and Engineering, Koneru Lakshmaiah Educational Foundation, Vijayawada, India 2 ECED, PIET, Samalkha, Haryana, India 3 Department of AI & ML, Malla Reddy University, Hyderabad, Telangana, India 1

Abstract

Aerial imagery can be a valuable tool for monitoring accidents, providing a detailed and comprehensive view of the affected area. Aerial imagery can be used for accident monitoring assessment of damage, identifying the cause of an accident, search, and rescue operations, planning for disaster response, and providing instant help to that affected area in the form of the ambulance, fire services, and inform traffic police for controlling traffic in that area simultaneously. Over 1.5 lakh people each year pass away on Indian roadways, which equates to a regular of 1130 accidents and 422 fatalities per day or 47 accidents and 18 deaths per hour. It is underlined that the integrated rescue system’s component pieces’ primary function is to aid traffic accident victims. The intervention process is illustrated using a real-world traffic collision, focusing on releasing people from smashed-up cars and providing pre-hospital emergency care. Proposals for the safe rescue of victims, the provision of (first) assistance, and the eradication of intervention-related problems sum up the assessment of traffic accidents and the ways of their resolving. Overall, aerial imagery can be a powerful tool for monitoring accidents and managing their aftermath. It provides a detailed and comprehensive view of the affected area, allowing for quick and effective decision-making. Keywords:  Aerial images, image recognition, monitoring, rescue

*Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (47–68) © 2024 Scrivener Publishing LLC

47

48  Advances in Aerial Sensing and Imaging

3.1 Introduction In India, 155,622 people died in road accidents in 2022, with speeding being to blame for 59.7% of those deaths. In fact, 62% of these accidents occurred on just 5% of the highways, indicating the need for preventative actions to lower accidents along these portions [1–3]. The government must find more of these stretches nationwide and implement preventive measures, including reducing speed limits, repairing blind spots, and leveling out extremely undulating sections of the road, which are some of the foremost reasons for traffic accidents. According to the United Nations, road accidents cost the Indian economy 3% of its GDP growth each year, amounting to a loss of $58,000 in real terms [4, 5]. The lack of traffic safety and incidents of reckless driving has drawn more attention to major cities like Chennai, Mumbai, and New Delhi. Even in industrialized nations, current statistics indicate a rise in the numeral of traffic accidents worldwide [6–9]. Yet, the impact on poor and underdeveloped countries is more significant because of the human and financial costs. Road traffic is a dynamic task where different entities (vulnerable road users, vehicles, buses, beggars, motorcycles, etc.) interact while using shared infrastructure [10]. Due to infrastructure limitations and an increase in cars, traffic management, and control is a challenging operation that calls for applying specialized algorithms and accurate traffic data (both historical and present) [11, 12]. Reducing travel times and emissions is more accessible with knowledge of the types and number of cars on the road [13]. Accurate traffic data enables us to forecast infrastructure bottlenecks, adjust management strategy to changing conditions, and improve traffic control effectiveness [14]. Traffic management has experienced increased strain over the last few decades due to the increased number of automobiles and sophisticated surveillance systems. Object recognition and machine vision techniques are now being used. Intelligent video analysis provided by monitoring systems can enhance the efficiency of traffic control measures, particularly in identifying red-light violations or illegal turns [15, 16]. Hence, smart surveillance technologies are a hot topic in computer vision. Automatic license plate recognition (ALPR) has typically only been used by traditional intelligent surveillance systems in rare situations. Automated systems can be particularly advantageous in various applications, such as toll collection, unattended parking lots, and congestion pricing, as they streamline processes and reduce the need for human intervention [17, 18].

Aerial Imaging Rescue for Road Monitoring  49 An integrated monitoring and rescue system based on aerial imaging involves using drones and unmanned aerial vehicles (UAVs) equipped with imaging sensors or other aerial vehicles to capture images and videos of a particular area. These images are then analyzed and processed to detect potential hazards or incidents requiring intervention. The system can be used for various applications, such as monitoring natural disasters like floods, fires, and earthquakes or tracking the movement of wildlife [19–21]. In the case of an emergency, the system can be used to identify and locate victims quickly, allowing rescue teams to respond more efficiently. The system uses high-resolution cameras and sensors mounted on the drone, which capture images and data from the target area [22]. This information is then transmitted to a central control room, where it is analyzed and processed using advanced algorithms to identify potential risks or hazards. In the case of an emergency, the system can be activated to initiate a rescue operation. The drone can quickly locate the victim and transmit their location to the rescue team, who can then respond with the necessary resources [23–25]. An integrated monitoring and rescue system based on aerial imaging has many benefits. It can provide real-time situational awareness of a particular area, enabling early detection and response to potential risks or hazards. It can also improve the efficiency and speed of rescue operations, potentially saving lives [26–28]. Overall, an integrated monitoring and rescue system based on aerial imaging is a valuable tool for improving public safety and emergency response. Its ability to provide real-time monitoring and response to potential risks and hazards can make a significant difference in the outcome of an emergency. In Section 3.2, the literature work is briefly discussed with past critical research studies in the domain of Aerial Imaging Rescue and Integrated Systems for Road Monitoring Based on Machine Learning and its other applications. Section 3.3 gives ideas about the number of accidents that happen and how much of these successfully provide help. Section 3.4 describes the proposed model with associated procedures for the final result. Section 3.5 discusses the results analysis of the proposed model in detail. Finally, Section 3.6 contains the conclusion, followed by a reference section.

3.2 Related Work Aerial imaging in monitoring and rescue systems has become increasingly popular. This technology allows for the efficient and accurate capture of images and data from above, which can then be used to monitor

50  Advances in Aerial Sensing and Imaging and respond to emergencies [29–31]. Such systems have various applications, including search and rescue missions, disaster response, and environmental monitoring. For instance, drones equipped with cameras and sensors can search for missing individuals or assess the extent of damage after a natural disaster [32]. In addition, aerial imaging can help monitor and detect environmental changes, such as forest fires or pollution levels. Researchers must focus on several key areas to develop an integrated monitoring and rescue system based on aerial imaging [33]. These include designing and implementing efficient drone platforms, developing sophisticated imaging and sensing technologies, and creating robust data analysis and interpretation algorithms. There are also ethical considerations when using aerial imaging for monitoring and rescue [34–38]. These include privacy, security, and the potential impact on wildlife and the environment. Overall, developing an integrated monitoring and rescue system based on aerial imaging holds great promise for improving emergency response and environmental monitoring capabilities. However, continued research and development are necessary to fully realize this technology’s potential [39–41]. A technique to identify traffic congestion on the roads was put forth. This technique needs a wireless transmitter and receiver set. The transmitter keeps sending packets. The receiver measures parameters for RSSI, LQI, and packet loss and is positioned on the other side of the road [42–44]. It has been demonstrated that these measurements allow for highly accurate differentiation between free-flowing and congested traffic conditions. ZigBee motes were used to implement and test the method [9]. According to a study, road accidents that result in severe injuries have recently been found to be a serious public health concern. Stricter remedies should be implemented to limit the incidence of traffic accidents that result in severe injuries in the short-term and long-term improved outcomes [10]. To evaluate the extent of harm incurred by vulnerable road users, such as pedestrians and cyclists, statistical accident analysis was conducted using data collected between 2012 and 2015 in Aveiro, Portugal. The following risk factors were considered for this study: gender, location, location data, profile, and climatic data. According to research, the likelihood of walkers tangled in traffic accidents rises by 10.6 times for walkers on urban roads and 2.7 times for vulnerable road users [11]. Economic growth, urbanization, and rising incomes have contributed to increased motorization. Unfortunately, this often leads to more vehicles on the road without a corresponding increase in infrastructure for road safety. As suburban areas expand, developing and maintaining the infrastructure

Aerial Imaging Rescue for Road Monitoring  51 required to promote road sharing among various road users, such as safety signs and road markings, becomes increasingly challenging. As the number of middle-class individuals and vehicle ownership rises, traffic volume also tends to increase. Without adequate safety infrastructure, pedestrians, bikers, and motorbike riders are at greater risk of injury. Furthermore, individuals who sustain injuries in countries lacking proper trauma systems face additional challenges in recovering from their injuries [12, 13]. According to the World Health Organization (WHO), automobile accidents claimed the lives of over 1.25 million people in 2013. About 90% of the victims were citizens of LMICs or low- and middle-income countries. A critical public health issue is the disproportionate representation of pedestrians, bicyclists, and motorbike riders, who account for 49% of all traffic fatalities [14]. In LMICs, the fatality rate from traffic accidents does not correspond to the degree of motorization [15]. A new vehicle recognition and classification approach was introduced, utilizing radio-based technology that combines ray tracing simulations, machine learning, and RSSI measurements. The authors argue that different types of vehicles exhibit distinct RSSI fingerprints, which were used to classify vehicles with the help of a machine. The RSSI values of a wireless network, consisting of three transmitting and three receiving devices on either side of a road, were analyzed to accomplish this. The six wireless modules had directional antennas and were installed atop delineator posts. It was shown that such a system could detect cars and classify them into two groups (passenger cars and trucks). A two-lane road’s traffic lanes were shown to have distinct CSI data distributions [15]. Moreover, a traffic detection algorithm is being researched using cutting-edge hardware development). Geomagnetic data can be used in various ways to monitor traffic flow. Multiple techniques have been employed to detect cars, including the finite state machine [16–18], the dual-window detection approach for extracting vehicle waveform information [19–21], and the peak-to-trough method that uses adaptive thresholds to distinguish between driving states of vehicles [22, 23]. This study evaluates the vehicle’s driving condition by analyzing the characteristics of the geomagnetic disturbance waveform generated when a vehicle passes through an SRS. By integrating multi-channel data through the SRS’s cross-sectional layout, comprehensive traffic flow detection is achieved, improving the accuracy and scope of traffic detection. The approaches now in use, use single classifiers to identify a vehicle’s type and identify it. Previous studies have employed the RSSI-based vehicle classification technique, utilizing various classification methods such as artificial neural networks [24], k-Nearest Neighbour (k-NN), support

52  Advances in Aerial Sensing and Imaging vector machines (SVM) [25], decision trees [26], and logistic regression [27]. To train the classification models and differentiate vehicles into two categories, namely passenger cars and trucks, an SVM approach was utilized [25]. Predefined features [26, 27] or raw data [25] train the most recent algorithms. A technique for detection and tracking, as well as speed estimation, was proposed. This technique relies on the analysis of RSSI in a network consisting of two Wi-Fi access points and two laptops equipped with Wi-Fi. Three states were identified by calculating the mean value and variance of RSSI measurements: an empty road, a stationary vehicle, and a moving vehicle. The experimental data presented demonstrates that when vehicle speed increases, RSSI variance also decreases. This dependency was utilized to estimate the rate [28].

3.3 Number of Accidents, Fatalities, and Injuries: 2016–2022 3.3.1 Accidents Statistics in India For 2021, States and Union Territories (UTs) recorded 412,432 traffic accidents, resulting in 153,972 fatalities and 384,448 injuries. In 2021, there was an average of 12.6% more traffic accidents than in 2020. Similarly, Table 3.1 shows that the number of fatalities and injuries from traffic accidents grew by 16.9% and 10.39%, respectively. These numbers translate into 47 accidents and 18 fatalities each hour nationwide, or 1130 accidents and 422 deaths per day on average as shown in Figure 3.1. Table 3.1  Total number of accidents, fatalities, and persons injured from 2016 to 2021. Year

Accidents

Fatalities

Persons injured

2016

480652

150785

494624

2017

464910

147913

470975

2018

467044

151417

469418

2019

449002

151113

451361

2020

366138

131714

348279

2021

412432

153972

384448

Value

Aerial Imaging Rescue for Road Monitoring  53 600000 500000 400000 300000 200000 100000 0

412432384448 153972

2016

2017

2018

2019

2020

2021

Accidents

480652

464910

467044

449002

366138

412432

Fatalities

150785

147913

151417

151113

131714

153972

Person Injured

494624

470975

469418

451361

348279

384448

Years

Figure 3.1  Accidents, fatalities, and persons injured from 2016 to 2021.

3.3.2 Accidents Statistics in Haryana Tables 3.2 and 3.3 show the data related to the Haryana state. Table 3.2 shows accidents, fatalities, and persons injured in Haryana from 2006 to 2020 (up to June). Table 3.3 shows the year-wise details of help extended to the accident victims in Haryana at the spot. Figures 3.2 and 3.3 show a pictorial view of the details of Tables 3.2 and 3.3. From all three tables, i.e., from India and Haryana, we conclude that there is an increase in the number of accidents. A system is needed that reduces casualties and also provides help to the required paper as soon as possible to reduce the death rate due to delays in medical facilities.

3.4 Proposed Methodology The analysis and interpretation of satellite and aerial images have been revolutionized by deep learning. Large sizes and various item classes present particular difficulties with these photos but also demonstrate the potential for deep learning researchers. This repository provides a thorough overview of different deep-learning techniques for satellite and aerial data analysis, including architectures, models, and algorithms for classification, segmentation, and object recognition. It is an invaluable tool for academics, professionals, and anyone interested in the most recent developments in deep learning and how they affect computer vision and remote sensing. A detailed description is given below, and it works on two modules. Module-A, shown in Figure 3.4, represents the architectural diagram of the

54  Advances in Aerial Sensing and Imaging Table 3.2  Year-wise details of accidents, fatalities, and persons injured in Haryana. Year

Accidents

Death

Injured

2020 (up to June 2020)

4024

1854

3391

2019

10944

5057

9362

2018

11238

5118

10020

2017

11258

5120

10339

2016

11234

5024

10531

2015

11174

4879

10794

2014

10676

4483

8944

2013

10482

4517

9143

2012

10065

4446

9452

2011

11128

4762

9727

2010

11195

4719

9905

2009

11915

4603

10481

2008

11596

4494

10570

2007

11998

4415

10288

2006

10314

4012

9118

proposed work for the classification and detection of the condition of traffic, road, and any misshaping on the highway or roadside area that affects the traffic. Module-B takes input from Module-A, works in real-time, and informs the staff directly in case of an accident is detected so that early high action is taken. Module B announces the ambulance, police, and fire station directly in case of fire detection so that life can be saved. Ambulance for providing first aid so that suffering people can be safe, police control the traffics so that no more casualty is there, and fire station control any fire there. Module A is organized into numerous phases, including a region of interest determination, object movement identification, tracing, regrouping, and pick-frame categorization. Figure 3.1 displays the proposed aerial imaging rescue and integrated system for road monitoring based on the machine learning, detection, and classification system’s overall layout.

3.4.1 ROI and Line Selection In vehicle detection, Region of Interest (ROI) and line selection are essential techniques for identifying and tracking vehicles in a video stream or

Aerial Imaging Rescue for Road Monitoring  55 Table 3.3  Year-wise details of help extended to the accident victims in Haryana. Year

Accident attended

Provided first-aid

Shifted to nearest hospitals

Vehicles cleared by crane

2020

3762

2395

2775

4541

2019

10360

6610

7500

11846

2018

10760

7081

7735

12232

2017

10939

7170

8164

12276

2016

10851

7194

8030

12090

2015

10426

6755

7751

11399

2014

9624

7210

7114

9076

2013

8950

6933

6719

7537

2012

8143

6267

5948

5787

2011

6652

5212

4379

4186

2010

1425

1409

995

1215

2009

1447

1389

832

1068

2008

1697

1021

1560

1003

2007

1697

1466

1065

924

2006

1090

1125

600

643

image. ROI refers to an image or video area where the vehicle detection algorithm focuses on identifying and tracking vehicles. This can be a specific region of the image or video, such as the roadway or parking lot, where vehicles are expected to appear. By limiting the search area to the ROI, the vehicle detection algorithm can process the image more efficiently and accurately, reducing false positives and improving detection accuracy. Line selection is another technique used in vehicle detection to track the movement of vehicles. By placing lines on the road or in a parking lot, the algorithm can detect when a vehicle crosses the line, allowing it to track its movement and speed. This can be useful in traffic management, parking lot monitoring, and other applications where vehicle movement needs to be monitored and analyzed. In summary, ROI and line selection are essential techniques used in vehicle detection to improve accuracy and efficiency. By carefully selecting the ROI and using lines to track vehicle movement, vehicle detection algorithms can more effectively identify and track vehicles in real-time, providing valuable data for various applications.

56  Advances in Aerial Sensing and Imaging

14000 12000 10000 Value

8000 6000 4000 2000 0

Accidents

2016

2017

2018

2019

2020

11234

11258

11238

10944

4024

Death

5024

5120

5118

5057

1854

Injured

10531

10339

10020

9362

3391

Years

Figure 3.2  Details of accidents, death, and injured from 2016 to 2020 (June).

14000 12000

Value

10000 8000 6000 4000 2000 0 Accidents Attended

2020

2019

2018

2017

2016

3762

10360

10760

10939

10851

Provided First-Aid

2395

6610

7081

7170

7194

Shifted to Hospitals

2775

7500

7735

8164

8030

Years

Figure 3.3  Details of help extended to the accident victims in Haryana.

3.4.2 Motion Detection Motion detection in vehicles is a technique used to detect and track the vehicle’s movement or other objects in its surroundings. Motion detection can be helpful in various applications, including surveillance, driver

Aerial Imaging Rescue for Road Monitoring  57

ROI & Line Selection Motion Detection

Single-Stage Clustering Euclidean distance

Feature Fusion Process

Second Clustering (K-means)

Tracking Objects

Classification by SVM

Road

Vehicle

Damage

• Detect the Road damage for warning • Classify the vehicles • Detect the accidents

Figure 3.4  Module A for classification by utilizing SVM.

assistance systems, and security. Several vehicle motion detection methods include optical, acoustic, and magnetic sensors. Optical sensors, such as cameras, are the most commonly used method for motion detection in vehicles. These sensors can capture images or videos of the vehicle and its surroundings, which can then be analyzed to detect and track motion. Motion detection algorithms can be used to analyze the images or video captured by optical sensors and see any changes in the position or movement of objects in the scene. For example, a parked vehicle in a parking lot

58  Advances in Aerial Sensing and Imaging may remain stationary for an extended period. Still, if it suddenly starts to move, the motion detection algorithm can detect the change and trigger an alert or other action. In driver assistance systems, motion detection can detect and track other vehicles’ movement on the road and provide warnings or assistance to the driver as needed. For example, suppose a vehicle suddenly swerves or brakes in front of the driver’s vehicle. In that case, the motion detection system can detect the movement and alert the driver to take evasive action. In summary, vehicle motion detection is a valuable technique for detecting and tracking activity and can be used in various applications to improve safety, security, and efficiency.

3.4.3 Single-Stage Clustering Single-stage clustering is a clustering algorithm that involves grouping similar objects into clusters in a single pass without any iterative refinement or optimization steps. This approach is known as “flat clustering” or “hierarchical clustering.” In single-stage clustering, the data is divided into groups based on similarity, using some distance metric. The Euclidean distance is the most commonly used distance metric, but other metrics like Manhattan distance or cosine similarity can also be used. Different algorithms can be used for single-stage clusterings, such as k-means, hierarchical, or density-based clustering. Each algorithm has its strengths and weaknesses, and the choice of algorithm depends on the nature of the data and the specific problem. Single-stage clustering is often used in exploratory data analysis, where the goal is to gain insights into the data structure and identify patterns or outliers. It can also be used in pre-processing, feature extraction, and compression. However, single-stage clustering may only suit some data types, especially if the data is noisy or contains complex patterns that a simple distance metric cannot capture.

3.4.4 Feature Fusion Process The fusion process in vehicles refers to integrating data from multiple sensors and sources to provide a more comprehensive and accurate view of the vehicle’s surroundings and environment. The fusion process can improve the performance of various vehicle systems, including navigation, driver assistance, and autonomous driving. The fusion process combines data from multiple sensors, such as cameras, lidar, radar, and GPS, to create a more detailed and reliable picture of the vehicle’s environment. The data from each sensor is processed and analyzed, and the resulting information

Aerial Imaging Rescue for Road Monitoring  59 is combined to create a unified view of the vehicle’s surroundings. For example, data from sensors such as lidar, radar, and cameras are combined in an autonomous vehicle to create a detailed 3D map of the vehicle’s environment. This map can identify obstacles, pedestrians, and other vehicles and provide the vehicle’s onboard computer with the information needed to make decisions about navigation and driving. In driver assistance systems, the fusion process can provide more accurate and reliable information to the driver, such as lane departure warnings and blind spot monitoring. Combining data from multiple sensors can provide a more comprehensive view of the vehicle’s surroundings and alert the driver to potential hazards. In summary, the fusion process in vehicles involves integrating data from multiple sensors and sources to provide a more accurate and comprehensive view of the vehicle’s environment. This process is essential for improving the performance of various vehicle systems and ensuring the safety and efficiency of the vehicle.

3.4.5 Second-Stage Clustering Second-stage clustering refers to performing another round of clustering on the results of first-stage clustering. In other words, it involves clustering clusters obtained from a previous clustering process. The second stage of clustering can be helpful in situations where the initial clustering does not yield sufficiently distinct sets or where the clusters are too large or too small. By performing a second-stage clustering, it is possible to refine the initial stages and obtain more meaningful and valuable collections. Various methods can be used for second-stage clustering, including hierarchical, K-means, and spectral clustering. The choice of method will depend on the specific application and the characteristics of the data being analyzed. Overall, the second stage of clustering can be a powerful tool for improving the quality of clustering results and gaining deeper insights into complex data sets.

3.4.6 Tracking Objects Tracking objects refers to locating and following objects or targets over time in a video or image sequence. This shared task in computer vision is used in various applications such as surveillance, robotics, and autonomous vehicles. Object tracking typically involves three stages: detection, tracking, and prediction. In the detection stage, the object of interest is first identified and localized in the image or video frame. This is usually done using object detection algorithms such as YOLO, Faster R-CNN, or SSD.

60  Advances in Aerial Sensing and Imaging Once the object is detected, the tracking stage begins, where the object’s position is estimated in subsequent frames by associating it with the previous position. This can be achieved using correlation, Kalman, or deep learning-based methods. Finally, in the prediction stage, the object’s future location is estimated based on its previous motion and other contextual information. This can help improve the tracking accuracy and compensate for occlusions or other challenging conditions. Overall, object tracking is a complex and challenging problem in computer vision, and researchers continue to develop new algorithms and techniques to improve its accuracy and robustness in various applications. Tracking objects in vehicles refers to identifying and monitoring the movement of objects within or near a vehicle, such as passengers, cargo, or other vehicles. Object tracking can be helpful in various applications, including surveillance, inventory management, and driver assistance systems.

3.4.7 Classification Classification in image processing involves using computer algorithms to automatically classify or categorize images into different classes based on their visual characteristics. This process typically involves the following steps: Data Collection: The first step consists in collecting a large dataset of images for classification. Feature Extraction: The second step consists in extracting features from the photos. These features could be various aspects of the image, such as color, texture, shape, or other visual characteristics. Training: Once the components are extracted, a machine learning algorithm is trained on the dataset to learn the patterns and relationships between the image features and their corresponding classes. Testing: After training, the algorithm is tested on a separate dataset to evaluate its performance and accuracy in correctly classifying new images. Deployment: Once the algorithm is deemed accurate and reliable, it can be deployed in a real-world setting to organize new photos. Examples of image classification applications include object detection, face recognition, medical image analysis, and autonomous vehicle navigation. The initial stage in the suggested automobile identification methodology is to acquire images from the highway extracted from the aerial photos taken by drone or imaginary satellite system. Grayscale is applied to all obtained frames from aerial photos or video captured by drone/aerial images so that a fast process is performed and transmitted fast for sending signals to all helping organizations, i.e., police, ambulance, and fire station. All the steps are explained above for module A. After that, module B is

Aerial Imaging Rescue for Road Monitoring  61

Alerting the Rescue Team Signal From Module A

Sharing Data over Network

Ambulance

Fire Station

Police

Figure 3.5  Module B for sending the alert to the rescue team.

performed. Module B informs all the rescue units to provide first aid to the affected area by sharing the photos and location of the accident area. As shown in module B. As shown in Figure 3.5, Module B takes the signal from Module A sharing the data with the rescue team to take action against the information sharing by Module A. With the help of these two modules, we can decrease the rate of deaths that happen due to the lack of first aid during any accident. This system works on drone images and aerial images. It first selects the area and then fast performs the steps and extracts the information, and sends the data to the rescue team quickly without any delay. This is a feedback system that ensures that the signal is transferred to the rescue team or not. If not, alert all nearby rescues team about that so one can attend to the area. It works on the IoT so that the data is transferred all the time, and a system monitors the data and performs the task so that if there is any mishappening that can be attended to without any fatality and provide fast first aid to the suffering people and also managing the traffic at the affected area.

3.5 Result Analysis In the first phase of video and image intelligence mechanisms, the primary objective is to detect objects of interest in the scene. A state-of-theart object detection algorithm is utilized to identify the category of things.

62  Advances in Aerial Sensing and Imaging To address complex scenarios such as occlusion, false positives, false negatives from object detection, overlapping objects, and shape changes, we developed a variance value function that employs a set of heuristic cues, including appearance, size, region of interest (ROI), and position. The Jaccard Distance is used to compute the appearance distance. In contrast, Euclidean distances between all object pairs are calculated to determine if goods are within a certain distance from each other. These item pairs were chosen for additional investigation because they can potentially engage in conflict. Each couple of adjacent objects’ most recent motion patterns are Table 3.4  Detection rate of accident comparison with other methods. S. no.

Techniques used

Year

Videos used

CDR %

1

Pawar et al. [29]

2021

7

79.00

2

Singh et al. [30]

2018

7

77.50

3

Ijjina et al. [31]

2019

45

71.00

4

Ki et al. [32]

2007

1

63.00

5

Hadi et al. [33]

2022

29

93.10

6

Wang et al. [34]

2020

-

92.50

7

Proposed Method

2023

35

96.78

Conflict Detection Rate (CDR) % 120 100 80 60 40 20 0

Pawar et al. [29]

Singh et al. [30]

Ijjina et al. [31]

Ki et al. [32]

Hadi et al. [33]

Wang et al. Proposed [34] Method

CDR %

Figure 3.6  Comparison of conflict detection rate graph with other techniques and the proposed method.

Aerial Imaging Rescue for Road Monitoring  63 analyzed in speed and direction of motion. The collision angle is another consideration in detecting collisions and near-collisions. Car accidents can take many forms, including direct, single-car rollovers, side-impact, rear-end, and side-impact crashes. Each of these collision types has its own characteristics and motion patterns. We gathered short films from Google and YouTube that trajectory conflict situations because there isn’t a freely manageable standard for traffic accidents at Crossroads. Videos of varied, challenging weather and lighting circumstances are included in the dataset. The video shows a few seconds before and after each trajectory disagreement. Our study analyzed videos with a 1280 X 720 pixels resolution and 30 frames per second frame rate. We used a high-performance PC with a 3.4 GHz processor, 16 GB of RAM, an Nvidia GEFORCE GTX GPU, and an AMD Ryzen 9 Processor to implement our proposed strategy. Real-time applications can be easily handled with an average processing speed of 35 frames per second (fps). The false alarm ratio, or FAR, is calculated by dividing the number of false alerts by the total number of warnings or alarms in a specific scenario or study. Conflict detection rate (CDR) is the ratio of detected conflict cases to the total number of conflicts. Our proposed framework demonstrated a Detection Rate of 96.78%, which is impressive as shown in Table 3.4. Graphically representation is shown in Figure 3.6 which shows the proposed method compared with other methods used earlier.

3.6 Conclusion To detect accidents automatically, a novel framework is introduced. To analyze the retrieved routes for abnormal identification, the framework incorporates three primary modules: object detection, a tracking technique, an alerting system, and an accident detection module. The robust tracking approach tracks the items of interest and records their trajectories while considering difficult circumstances, including occlusion, overlapping objects, and shape changes. Motion analysis uses a variety of heuristic cues to identify irregularities that could cause traffic accidents. To evaluate the effectiveness of the suggested framework against actual recordings, a collection of different traffic movies with an accident or near-accident events is gathered. Experimental analyses show that our approach is practical for real-time traffic control applications. The detection rate is enhanced, and we can send a quick response to react team and save the life at the accident place, and police ensure the smooth movement of traffic at that place if there is any fire that firefighters can control. All the rescue team reaches

64  Advances in Aerial Sensing and Imaging the place within time which can be done by sending them quickly. The proposed method gives better result than the previous technique and also informs the rescue team about the affected area for primary action.

References 1. The World Health Organization, Global status report on road safety, The World Health Organization, Geneva, Switzerland, 2018. 2. Chang, H., Wang, Y., Ioannou, P.A., The use of microscopic traffic simulation model for traffic control systems, in: Proceedings of the 2007 International Symposium on Information Technology Convergence, ISITC 2007, November 2007, pp. 120–124. 3. Bernas, M., Płaczek, B., Porwik, P., Pamuła, T., Segmentation of vehicle detector data for improved k-nearest neighbor-based traffic for prediction. IET Intell. Transp. Syst., 9, 3, 264–274, 2014. 4. Ahmad, I., Noor, R.M., Ali, I., Imran, M., Vasilakos, A., Characterizing the role of vehicular cloud computing in road traffic management. Int. J. Distrib. Sens. Netw., 13, 5, 1–14, 2017. 5. Chang, S.-L., Chen, L.-S., Chung, Y.-C., Chen, S.-W., Automatic license plate recognition. IEEE Trans. Intell. Transp. Syst., 5, 1, 42–53, 2004. 6. Caner, H., Gecim, H.S., Alkar, A.Z., Efficient embedded neural-­networkbased license plate recognition system. IEEE Trans. Veh. Technol., 57, 5, 2675–2683, 2008. 7. Qin, X., Tao, Z., Wang, X., Dong, X., License plate recognition based on improved BP neural network, in: Proceedings of the International Conference on Computer, Mechatronics, Control and Electronic Engineering (CMCE ‘10), Changchun, China, vol. 5, pp. 171–174, August 2010. 8. Liu, Y., Wei, D., Zhang, N., Zhao, M., Vehicle-license-plate recognition based on neural networks, in: Proceedings of the IEEE International Conference on Information and Automation (ICIA’ 11), pp. 363–366, June 2011. 9. Roy, S., Sen, R., Kulkarni, S., Kulkarni, P., Raman, B., Singh, L.K., Wireless across the road: RF-based road traffic congestion detection, in: Proceedings of the 2011 Third International Conference on Communication Systems and Networks (COMSNETS 2011), IEEE, pp. 1–6, January 2011. 10. Kumar, Jain, A., Rani, S., Alshazly, H., Idris, S.A., Bourouis, S., Deep neural network based vehicle detection and classification of aerial images. Intell. Autom. Soft Comput., 34, 1, 1–13, 2022. 11. Sandeep, Rajan, E.G., Rani, S., A study on vehicle detection through aerial images: Various challenges, issues and applications, in: International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp. 504–509, 2021.

Aerial Imaging Rescue for Road Monitoring  65 12. Kumar, Rajan, E.G., Rani, S., Enhancement of satellite and underwater image utilizing luminance model by color correction method, in: Cognitive Behavior and Human Computer Interaction Based on Machine Learning Algorithm, pp. 361–379, 2021. 13. Sandeep, Prasad, K.M.V.V., Srilekha, A., Suman, T., Pranav Rao, B., NV Krishna, J., Leaf disease detection and classification based on machine learning, in: International Conference on Smart Technologies in Computing, Electrical and Electronics (ICSTCEE), pp. 361–365, 2020. 14. Sandeep, Swetha, S., Taj Kiran, V., Johri, P., IoT based smart home surveillance and automation, in: International Conference on Computing, Power and Communication Technologies (GUCON), pp. 786–790, 2018. 15. Sandeep, Singh, S., Kumar, J., Face spoofing detection using improved SegNet architecture with a blur estimation technique. Int. J. Biom., 13, 2–3, 131–149, 2021. 16. Sifuentes, E., Casas, O., Pallas-Areny, R., Wireless magnetic sensor node for vehicle detection with optical wake-up. IEEE Sens. J., 11, 8, 1669–1676, 2011. 17. Bao, X., Li, H., Xu, D., Jia, L., Ran, B., Rong, J., Traffic vehicle counting in jam flow conditions using low-cost and energy-efficient wireless magnetic sensors. Sensors, 16, 11, 2–16, 2016. 18. Meng, L., Wang, H., Quan, W., Vehicle detection using three-axis AMR sensors deployed along travel lane markings,” Journals & magazines. IET Intell. Transp. Syst., 11, 9, 581–587, 2017. 19. Fang, Z., Zhao, Z., Xuan, Y., A node design for intelligent traffic monitoring based on magnetic sensor. Advances in Wireless Sensor Networks, vol. 334, pp. 57–67, 2012. 20. Chinrungrueng, J. and Kaewkamnerd, S., Wireless magnetic sensor network for collecting vehicle data. Sensors, Christchurch, New Zealand, 2009. 21. Chong, J., Zhao, M., Li, J., Noise reduction by magnetostatic coupling in ­geomagnetic-field sensors. J. Magn. Magn. Mater., 368, 328–332, 2014. 22. Taghvaeeyan, S. and Rajamani, R., Portable roadside sensors for vehicle counting, classification and speed measurement. IEEE Trans. Intell. Transp. Syst., 15, 1, 294–306, 2014. 23. Cheung, S., Coleri, S., Dunda, B., Sumitra, G., Tan, C.-W., Varaiya, P., Traffic measurement and vehicle classification with a single magnetic sensor. Transp. Res. Rec., 1917, 1, 173–181, 2005. 24. Won, M., Zhang, S., Son, S.H., WiTraffic: Low-cost and nonintrusive traffic monitoring system using WiFi, in: Proceedings of the 26th International Conference on Computer Communications and Networks, ICCCN 2017, August 2017, IEEE, pp. 1–9. 25. Haferkamp, M., Al-Askary, M., Dorn, D. et al., Radio-based traffic flow detection and vehicle classification for future smart cities, in: 2017 IEEE 85th Vehicular Technology Conference (VTC Spring), Sydney, NSW, Australia, pp. 1–5, 2017.

66  Advances in Aerial Sensing and Imaging 26. Martsenyuk, V., Warwas, K., Augustynek, K. et al., On the multivariate method of qualitative analysis of Hodgkin-Huxley model with decision tree induction, in: Proceedings of the 2016 16th International Conference on Control, Automation, and Systems (ICCAS), Gyeongju, South Korea, October 2016, pp. 489–494. 27. Bernas, M., Płaczek, B., Korski, W., Wireless network with bluetooth low energy beacons for vehicle detection and classification, in: CN 2018: Computer Networks, vol. 860 of ‘Communications in Computer and Information Science, P. Gaj, M. Sawicki, G. Suchacka, A. Kwiecien (Eds.), pp. 429–444, Springer, Springer, Cham, 2018. 28. Kassem, N., Kosba, A.E., Youssef, M., RF-based vehicle detection and speed estimation, in: 2012 IEEE 75th Vehicular Technology Conference (VTC Spring), IEEE, pp. 1–5, 2012. 29. Pawar, K. and Attar, V., Deep learning-based detection and localization of road accidents from traffic surveillance videos. ICT Express, 2021. 30. Singh, D. and Mohan, C.K., Deep spatio-temporal representation for detection of road accidents using a stacked autoencoder. IEEE Trans. Intell. Transp. Syst., 20, 3, 879–887, 2018. 31. Ijjina, E.P., Chand, D., Gupta, S., Goutham, K., Computer vision based accident detection in traffic surveillance, in: 2019 10th International Conference on Computing, Communication, and Networking Technologies (ICCCNT), IEEE, pp. 1–6, 2019. 32. Ki, Y.-K. and Lee, D.-Y., A traffic accident recording and reporting model at intersections. IEEE Trans. Intell. Transp. Syst., 8, 2, 188–194, 2007. 33. Ghahremannezhad, H., Shi, H., Liu, C., Real-time accident detection in traffic surveillance using deep learning. 2022 IEEE International Conference on Imaging Systems and Techniques (IST), Kaohsiung, Taiwan, pp. 1–6, 2022. 34. Wang, C., Dai, Y., Zhou, W., Geng, Y., A vision-based video crash detection framework for mixed traffic flow environment considering low-visibility condition. J. Adv. Transp., 1–11, 2020. 35. Bin Islam, M. and Kanitpong, K., Identification of factors in road accidents through in-depth accident analysis. IATSS Res., 32, 2, 58–67, 2008. 36. Ghosh, S.K., Parida, M., Uraon, J.K., Traffic accident analysis for Dehradun city using GIS. ITPI J., 1, 3, 40–54, 2004. 37. Alkhadour, W., Zraqou, J., Al-Helali, A., Al-Ghananeem, S., Traffic accidents detection using geographic information systems (GIS): Spatial correlation of traffic accidents in the city of Amman, Jordan. Int. J. Adv. Comput. Sci. Appl., 12, 4, 484–494, 2021. 38. Hyder, A.A. and Vecino-Ortiz, A.I., BRICS: Opportunities to improve road safety. Bull. World Health Organ., 92, 6, 423–428, 2014. 39. World Health Organization, Global status report on road safety 2015, World Health Organization, 2015.

Aerial Imaging Rescue for Road Monitoring  67 40. Kumar, S., Kumar, M., Rashid, R., Agrawal, N., A comparative analysis on image denoising using different median filter methods. Int. J. Res. Eng. Technol., 5, 8, 231–239, 2017. 41. Sifuentes, E., Casas, O., Pallas-Areny, R., Wireless magnetic sensor node for vehicle detection with optical wake-up. IEEE Sens. J., 11, 8, 1669–1676, 2011. 42. Kaur, R. and Singh, S., A novel fuzzy logic based reverse engineering of gene regulatory network. Future Comput. Inf. J., 2, 2, 79–86, 2018. 43. Sai Praneeth, R., Chetan Sai Akash, K., Keerthi Sree, B., Ithaya Rani, P., Scaling object detection to the edge with YOLOv4, TensorFlow Lite, in: 7th International Conference on Computing Methodologies and Communication (ICCMC 2023), Erode, India, pp. 1–6, 2023. 44. Bhola, Athithan, S., Singh, S., Mittal, S., Sharma, Y.K., Dhatterwal, J.S., Hybrid framework for sentiment analysis using ConvBiLSTM and BERT. International Conference on Technological Advancements in Computational Sciences (ICTACS – 2022), Amity University, Uzbekistan, pp. 1–5, 2022.

4 A Machine Learning Approach for Poverty Estimation Using Aerial Images Nandan Banerji1*, Sreenivasulu Ballem2, Siva Mala Munnangi3 and Sandeep Mittal4 Department of Computer Science and Engineering, Sikkim Manipal Institute of Technology, Sikkim Manipal University, Sikkim, India 2 Department of Mathematics, School of Physical Sciences, Central University of Karnataka, Kalaburagi, Karnataka, India 3 Department of Mathematics V R Siddhartha Engineering College, Kanuru, India 4 Department of Computer Science and Engineering, Echelon Institute of Technology, Faridabad, Haryana, India 1

Abstract

Poverty estimation using aerial Imaging is a promising approach that utilizes high-resolution images captured from drones or planes to assess poverty levels in a given area. Aerial imagery can provide valuable insights into the built environment, including housing quality, access to services and infrastructure, and other socio-economic factors. This approach can be beneficial in areas where groundbased surveys, such as remote or conflict-affected regions, may need to be more challenging or impossible to conduct. However, the accuracy of poverty estimates derived from aerial imagery depends on the quality of the images and the algorithms used to analyze them. Future research could focus on developing more advanced algorithms, integrating aerial imagery with other data sources, and applying this approach in disaster response and recovery, urban planning, and monitoring progress toward poverty reduction goals. Overall, poverty estimation using aerial Imaging has the potential to provide valuable insights for policymakers and aid organizations working toward poverty reduction. Keywords:  SAE, georeferencing, rectification, poverty, map

*Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (69–86) © 2024 Scrivener Publishing LLC

69

70  Advances in Aerial Sensing and Imaging

4.1 Introduction Poverty estimation using aerial Imaging involves using satellite or aerial images to identify and analyze the living conditions of people in a particular area. The approach is based on the premise that specific physical and environmental characteristics of a neighborhood can indicate the poverty level in that area. Aerial Imaging can provide valuable data on various aspects of living conditions, including housing quality, access to essential services such as water and sanitation, and basic infrastructure such as roads and electricity. These indicators can estimate poverty levels and create poverty maps, informing policy decisions and targeting interventions in most impoverished areas. To estimate poverty using aerial Imaging, the first step is to acquire high-resolution satellite or aerial images of the size of interest. These images can be analyzed using various image processing techniques, such as object recognition, classification, and segmentation. The resulting data can then be used to create maps and statistical models to estimate poverty levels in the area. According to a World Bank study, satellite imagery and machine learning can provide poverty predictions as accurate as traditional household surveys, with an accuracy rate of up to 90%. Poverty maps created using aerial Imaging and sensing can provide a detailed picture of poverty levels in a given area, enabling policymakers to target interventions more effectively. For example, poverty maps created in Malawi identified specific regions where poverty was most prevalent, allowing for more targeted interventions. A study in Kenya used aerial Imaging to identify areas with high levels of vegetation loss, which was a reliable predictor of poverty. This information targeted interventions in these areas, significantly reducing poverty rates. A project in Uganda used aerial Imaging and GIS to map out areas of poverty and identify gaps in service provision. This information was used to develop targeted interventions to improve access to essential services such as water and sanitation. Figure 4.1 depicts districts’ cumulative growth over nearly two decades, specifically for 2001, 2011, and 2019 [1]. The visualization’s color-coding divides communities into three categories depending on their overall progress indicator: districts with an index between 6 and 10 are displayed in red, sections with an index between 11 and 14 are shown in yellow, and communities with an index above 14 are displayed in green. From 2001 to 2011, the eastern regions of India, including Orissa, Bihar, Jharkhand, and West Bengal, and significant portions of northern and central India, such as Uttar Pradesh and Madhya Pradesh, and the north-eastern districts,

Poverty Estimation using Aerial Imaging  71 2001 (as per census)

High is 15-18 Mid is 11-14 Low is 06-10

2011 (as per census)

High is 15-18 Mid is 11-14 Low is 06-10

2019 (predictions)

High is 15-18 Mid is 11-14 Low is 06-10

Figure 4.1  Aggregate district development in 2001 (as per census), 2011 (as per census), and 2019 (as per our predictions made from satellite data) [1].

experienced little growth. Historically, these regions have been among the lowest in the nation. In contrast, states such as Gujarat, Maharashtra, Tamil Nadu, and Andhra Pradesh experienced substantial growth in a number of their districts during the same time frame. This could be attributed to the level of industrialization in these states. According to existing research, industrialized communities experience quicker development than predominantly agricultural districts. It is important to note that poverty estimation using aerial Imaging is not a perfect solution and has limitations. For example, it may not capture certain aspects of poverty, such as social exclusion, discrimination, or cultural factors. Additionally, the accuracy of the results may be affected by factors such as image resolution, time of day, and weather conditions. Therefore, poverty estimation using aerial Imaging should be used with other data sources and methodologies to provide a more comprehensive understanding of poverty in a given area. The key objectives of studying poverty estimation using aerial Imaging and sensing include the following: • Identifying areas of poverty: Aerial Imaging and sensing can be used to identify and locate areas that are likely to be affected by poverty. • Creating poverty maps: The data obtained through aerial Imaging and sensing can be used to create maps visually representing poverty levels in different regions. • Monitoring changes in poverty levels: By using aerial Imaging and sensing over time, changes in poverty levels

72  Advances in Aerial Sensing and Imaging

• • • •

•

can be tracked, and the effectiveness of poverty reduction interventions can be evaluated. Targeting interventions: Aerial Imaging and sensing can help identify specific areas and populations that require targeted poverty reduction interventions. Measuring progress: Poverty estimation using aerial Imaging and sensing can provide data that can be used to measure progress in reducing poverty over time. Supporting policy decisions: The data obtained through poverty estimation can inform policy decisions related to poverty reduction and development planning. Improving resource allocation: By identifying poverty areas, poverty estimation using aerial Imaging and sensing can help ensure that resources are allocated efficiently and effectively. Raising awareness: Poverty maps created through aerial Imaging and sensing can be used to raise awareness of poverty and help mobilize support for poverty reduction efforts.

The hypothesis for poverty estimation using aerial Imaging is that high-resolution aerial imagery can identify poverty indicators, such as housing quality, access to services and infrastructure, and other socio-­ economic factors [2, 3]. By analyzing these indicators using advanced algorithms and machine learning techniques, poverty estimates can be derived that accurately reflect the distribution and severity of poverty in a given area. This approach can provide valuable insights for policymakers and aid organizations working towards poverty reduction, where groundbased surveys may be difficult or impossible to conduct. Poverty estimation using aerial Imaging has the potential to be a valuable tool in the fight against poverty, enabling more targeted and effective poverty reduction efforts. Poverty estimation is a crucial tool for understanding and addressing poverty, allowing us to target resources effectively, measure progress, and advocate for policies and programs that help reduce poverty and promote development which is discussed briefly as follows: Understanding the extent and severity of poverty: Poverty estimation helps us understand the importance and severity of poverty in a given area, which is essential for developing effective poverty reduction policies and programs [4–6]. • Targeting resources: Poverty estimation helps target resources to areas and populations most in need, ensuring that interventions are more effective and efficient.

Poverty Estimation using Aerial Imaging  73 • Measuring progress: Poverty estimation allows us to measure progress in reducing poverty over time, which is critical for evaluating the effectiveness of poverty reduction efforts. • Advocacy and awareness: Poverty estimation can raise awareness of poverty and mobilize support for poverty reduction efforts. • Improved decision-making: Poverty estimation provides data that can inform decision-making at all levels, from national policy to local interventions, helping ensure that resources are used effectively and efficiently. • Comparative analysis: Poverty estimation allows us to compare poverty levels between different regions, countries, and groups, providing valuable insights into the causes and consequences of poverty.

4.2 Background and Literature Review In literature, various types of methodology have been used to estimate poverty in different regions of the works, so the main techniques used are discussed below: • Spectral Indices: Spectral indices are calculated from aerial imagery and are used to detect variations in vegetation, soil moisture, and other features. These variations can be used to identify poverty areas, as poor households may have less access to water and struggle to maintain vegetation. • Object-Based Image Analysis (OBIA): OBIA involves identifying objects in aerial imagery and analyzing their properties. For example, researchers may identify individual buildings and assess their quality and infrastructure, which can indicate poverty. • Convolutional Neural Networks (CNN): CNN is a deep learning algorithm used to classify aerial imagery and identify poverty indicators, such as housing quality and access to services. A comparative analysis of these methods found that CNN outperformed spectral indices and OBIA in identifying poverty indicators in aerial imagery [3]. The study also found that CNN could identify poverty indicators more accurately in areas with higher population density, suggesting that

74  Advances in Aerial Sensing and Imaging this method may be particularly useful in urban areas. Another study compared spectral indices with OBIA and found that OBIA was better at identifying poverty indicators in aerial imagery, such as housing quality and access to services [7, 8]. However, the study also noted that OBIA requires more computational resources than spectral indices and may be more challenging to implement in areas with limited resources. There are several methods for poverty estimation using aerial Imaging; CNN appears to be the most effective at identifying poverty indicators in high-density areas. However, OBIA may also be helpful in some contexts, particularly when analyzing individual buildings and infrastructure. Further research is needed to determine the most effective method for poverty estimation in different contexts and to identify ways to improve the accuracy and scalability of this method. Poverty estimation using aerial Imaging has become an increasingly popular research topic in recent years. Remote sensing technologies such as satellite imagery, LiDAR, and drones have been used to extract poverty-related features, such as vegetation indices, housing types, and infrastructure, to create high-resolution poverty maps. The following literature review summarizes some key findings and challenges in poverty estimation using aerial Imaging. One study used high-resolution satellite imagery to estimate poverty in rural Ethiopia, finding that the Normalized Difference Vegetation Index (NDVI) was a significant predictor of poverty [9]. Another study in India used remote sensing and machine learning techniques to estimate poverty at the district level, finding that nightlights, road density, and population density were significant predictors of poverty [10–16]. A review of poverty mapping using remote sensing found that poverty mapping has been used for various purposes, including identifying poverty’s spatial distribution, monitoring poverty changes over time, and targeting poverty reduction interventions. The review also highlighted some of the critical challenges in poverty mapping, including data quality, availability, spatial resolution, cost, interpretation of data, heterogeneity of poverty, and ethical concerns. Another review of poverty mapping using remote sensing and machine learning techniques found that these techniques can provide accurate and cost-effective poverty estimates, particularly in areas with limited groundbased data [6]. However, the review also highlighted some of the limitations of poverty mapping using remote sensing, including the need for careful validation, the limited ability of remote sensing to capture some aspects of poverty and the ethical concerns around privacy and consent. After an intense literature review, the critical challenges in poverty estimation using aerial Imaging and sensing are listed below. Addressing these challenges requires careful consideration of data quality, availability, interpretation,

Poverty Estimation using Aerial Imaging  75 and ethical and practical considerations around using aerial Imaging and sensing for poverty estimation. • Data quality: The quality of the data obtained through aerial Imaging and sensing can be affected by various factors such as weather conditions, cloud cover, and sensor calibration, which can result in inaccuracies in poverty estimates. • Data availability: Aerial imaging and sensing data may only be available for some regions and may be limited in some areas, making it challenging to create accurate poverty maps. • Spatial resolution: The spatial resolution of aerial imaging and sensing data can impact the accuracy of poverty estimates, particularly in areas with high levels of spatial variation in poverty. • Cost: Aerial Imaging and sensing can be costly, particularly for high-resolution data, making it difficult to obtain data for large areas or over extended periods. • Interpretation of data: Interpreting aerial Imaging and sensing data requires remote sensing and GIS expertise, which may only be available in some settings, making it difficult to obtain accurate poverty estimates. • Heterogeneity of poverty: Poverty is a complex phenomenon that can vary significantly between regions and within regions. Aerial Imaging and sensing may only capture some aspects of poverty, which can lead to inaccurate poverty estimates. • Ethical concerns: Using aerial Imaging and sensing data raises ethical concerns around privacy and consent, particularly in contexts where data may be used to target interventions or surveillance purposes. Estimating poverty is a complex task that involves many factors, such as income, access to basic needs, and education levels. Here’s a step-by-step working approach for poverty estimation [7]: Define Poverty: Poverty can be defined in various ways, but usually, it’s described as a lack of access to basic needs such as food, shelter, and healthcare. The poverty line is an income threshold defining the minimum income required to meet these basic needs. Collect Data: Collect data on the factors determining poverty, such as income, education, employment, health, and access to essential services. This data can be obtained through surveys, census, or other relevant sources [9, 10].

76  Advances in Aerial Sensing and Imaging • Determine Poverty Line: Determine the poverty line based on the cost of basic needs and living expenses in the region or country you are estimating poverty for. • Calculate Income: Calculate the income of each household or individual based on the data collected. Include all sources of income, such as salaries, wages, and government benefits. • Calculate Poverty Gap: Calculate the poverty gap for each household or individual by subtracting their income from the poverty line. This shows how much income is needed to reach the poverty line. • Determine Poverty Rate: Determine the poverty rate by calculating the percentage of households or individuals whose income is below the poverty line. • Analyze Data: Analyze the data collected to identify patterns and trends in poverty rates and factors contributing to poverty. This can help policymakers identify areas where intervention is needed. • Develop Policies: Based on the data analysis, develop policies to address poverty, such as job creation, education, and healthcare initiatives. • Monitor Progress: Continuously monitor progress to assess the effectiveness of poverty reduction policies and adjust them as needed. Poverty estimation involves defining poverty, collecting data, determining the poverty line, calculating income, determining the poverty gap, determining the poverty rate, analyzing data, developing policies, and monitoring progress [10–14].

4.3 Proposed Methodology Using machine learning algorithms, poverty can be estimated by analyzing aerial imagery to identify poverty-associated features, such as substandard housing, lack of access to clean water, and vegetation cover. This approach can provide a cost-effective and efficient method for poverty mapping and aid distribution. Step-by-Step discussion of the proposed work is shown in Figure 4.2.

4.3.1 Data Acquisition Aerial imaging data is obtained using remote sensing technologies such as satellite imagery, LiDAR, or drones. Data acquisition for poverty estimation

Poverty Estimation using Aerial Imaging  77

Data Acquisition

Pre-Processing

Primary & Secondary Data Collection

Image Rectification by Using Georeferencing

• •

Model Development

Data Integration

Small Area Estimation (SAE) Model

• MultiSource Data Fusion

Validation Regression Analysis

Feature Extraction RSBPM OBIA

Policy and P Program Development Develop

Figure 4.2  Flow chart of overall work.

involves collecting information on various factors that can impact poverty. Here are some critical data sources for poverty estimation. Household surveys are a common source of data for poverty estimation. These surveys can provide information on household income, consumption patterns, education levels, and health status. Census data can also be used for poverty estimation. Census data can provide information on household size, age, gender, education level, and occupation. Administrative records from government

78  Advances in Aerial Sensing and Imaging agencies, such as tax and social security data, can provide information on household income and employment status. Non-governmental organizations (NGOs and international organizations often collect data on poverty and can provide valuable insights into poverty estimation. Satellite imagery can estimate poverty rates in remote or hard-to-reach areas. This can include identifying areas with low agricultural productivity or sites without electricity or clean water access. Participatory Rural Appraisals (PRAs) involve community members in collecting data. This approach can provide insights into community priorities and help identify areas for intervention [20–23]. Focus groups involve small groups of people discussing a specific topic. Focus groups can provide valuable insights into community priorities and help identify areas for intervention. Overall, data acquisition for poverty estimation involves a combination of primary data collection (such as household surveys and PRAs) and secondary data sources (such as census data and administrative records) [12].

4.3.2 Pre-Processing The aerial imaging data is pre-processed to remove distortions and artifacts and to enhance the data’s quality. Pre-processing for poverty estimation using aerial images involves several steps to prepare the data for analysis. Collect high-resolution aerial photos of the region of interest. These images can be obtained from satellites or drones. After that, Image Rectification involves correcting any geometric distortions in the images caused by camera angle or terrain variations. This is done by projecting the images onto a standardized map coordinate system such as georeferencing. Then enhance the photos to improve visibility and clarity. This can include adjusting brightness, contrast, and color levels. Classify the images into land use and cover categories, such as urban areas, agricultural lands, forests, and water bodies. This can be done manually or through 4machine learning algorithms [24–27].

4.3.3 Feature Extraction Features related to poverty from the images that can be used to estimate poverty, such as road density, building density, and vegetation cover, are extracted from the pre-processed data. The technique for feature extraction for poverty using aerial images is called remote sensing-based poverty mapping and object-based image analysis. It involves using aerial or satellite imagery to extract features such as land use, vegetation cover, and infrastructure, which can then be used as poverty indicators. This technique assumes that certain

Poverty Estimation using Aerial Imaging  79 features, such as the lack of infrastructure or low vegetation cover, correlate with poverty. Feature extraction for poverty estimation using aerial images involves identifying and extracting relevant features from the pictures that can be used to estimate poverty. Building density can be calculated by counting the number of buildings per unit area. Higher building density is often associated with higher levels of economic activity and can be used as an indicator of poverty. Road density can be calculated by measuring the length of roads per unit area. Higher road density can indicate economic activity and access to services. Vegetation cover can be measured using the Normalized Difference Vegetation Index (NDVI). Higher NDVI values indicate more excellent vegetation cover and can demonstrate agricultural productivity. The presence of water bodies can mean access to clean water and can be used as a proxy for poverty. Land use and land cover can be classified into different categories, such as urban, agricultural, and forested areas. These categories can be used to identify areas with high or low levels of economic activity. Settlement patterns can be analyzed to determine areas with high population density, which can indicate poverty. Nighttime lights can be used to identify areas with high levels of economic activity and access to electricity. Overall, feature extraction for poverty estimation using aerial images involves identifying and extracting relevant features associated with poverty, such as building density, road density, vegetation cover, water bodies, land use and land cover, settlement patterns, and nighttime lights. These features can then be combined with other data sources, such as household surveys or census data, to develop poverty estimates.

4.3.4 Data Integration The extracted features are integrated with other datasets, such as population data, to create a comprehensive poverty map. Data integration for aerial image estimation involves combining the information from aerial photos with other data sources to develop more accurate poverty estimates. One data integration technique for poverty estimation is called multi-source data fusion. Multi-source data fusion is a technique that combines data from multiple sources, including aerial and satellite imagery, household surveys, census data, and administrative data, to estimate poverty rates and identify areas in need of interventions. Different data sources are pre-processed and integrated into this technique using statistical and machine-learning algorithms. For example, household survey data can be used to estimate poverty rates at the household level. In contrast, remote sensing data can be used to estimate poverty rates at the community level. The integrated data can then be used to develop poverty maps, which can be used to identify areas

80  Advances in Aerial Sensing and Imaging with a high concentration of poverty and target interventions accordingly. Multi-source data fusion is a powerful technique for poverty estimation. It combines complementary data sources and provides a more accurate and comprehensive picture of poverty at the local level. Overall, data integration for poverty estimation using aerial images involves combining information from multiple sources to develop more accurate poverty estimates.

4.3.5 Model Development Model development for poverty estimation using aerial images involves developing a statistical or machine-learning model that combines the information from aerial photographs with other data sources to estimate poverty levels. One poverty estimation model that combines knowledge from aerial images and other data sources is the Small Area Estimation (SAE) model. SAE is a statistical technique that estimates poverty indicators at a fine spatial scale, such as a district or village level. The SAE model uses a combination of data sources, including census data, household surveys, and remote sensing data, to estimate poverty rates in areas without data or where the sample size is too small. Remote sensing data, such as aerial images, extract poverty-related features like vegetation cover, land use, and housing conditions. The SAE model then combines this information with household survey data to estimate poverty rates for small areas. The model uses a hierarchical Bayesian framework to account for spatial dependence and to borrow information across neighboring regions. The SAE model has been used in various contexts, including India, Bangladesh, and Nepal, to estimate poverty rates and identify areas needing interventions. It is a powerful tool for poverty mapping and can provide policymakers with fine-grained information on poverty at the local level. Use the model to create poverty maps that identify high and low-poverty areas. These maps can be used to target interventions and allocate resources more effectively. Overall, model development for poverty estimation using aerial images involves selecting relevant features, choosing an appropriate statistical or machine learning model, training and testing the model, refining the model, validating the model, and creating poverty maps. The key is to develop a model that accurately estimates poverty levels based on the available data [16].

4.3.6 Validation Validation is a critical step in poverty estimation using aerial images. It involves comparing the results of the poverty estimation model to ground truth data to assess the accuracy and reliability of the model. This step can

Poverty Estimation using Aerial Imaging  81 help identify any errors or biases in the model. Collect ground truth data, such as household surveys or census data, that provides accurate information on poverty levels in the study area. The ground truth data should represent the study area and cover a sufficient sample size. Pre-process the ground truth data to ensure it is in a format suitable for comparison with the poverty estimation model output. This may involve standardizing the data or converting it to a standard format. Apply the poverty estimation model to the study area to estimate poverty levels. This can be done using GIS software or other tools. Compare the poverty estimates generated by the model to the ground truth data. This can be done using statistical techniques such as correlation or regression analysis. Assess the accuracy and reliability of the poverty estimation model based on the comparison results. If the model performs well, it can be used to estimate poverty levels in other areas. If discrepancies or errors exist, adjustments may be needed to improve the model. Conduct sensitivity analysis to assess the impact of different model inputs or assumptions on the poverty estimates. This can help identify areas of uncertainty or bias in the model. Overall, validation is a critical step in poverty estimation using aerial images. It involves collecting ground truth data, applying the poverty estimation model, comparing the results to the ground truth data, assessing the accuracy and reliability of the model, and conducting sensitivity analysis. The goal is to develop a poverty estimation model that accurately and reliably estimates poverty levels based on the available data.

4.3.7 Visualization and Analysis Visualization and analysis are essential components of poverty estimation using aerial images. They help to make sense of the data and provide insights into poverty patterns and trends. Prepare the data for visualization and analysis by cleaning, transforming, and integrating the different data sources. This step is critical to ensure the data is consistent, accurate, and suitable for analysis. Conduct spatial analysis to identify poverty-level patterns and trends across the study area. This can involve creating heat maps, choropleth maps, or other visualizations highlighting high and lowpoverty areas. Conduct a time-series analysis to identify changes in poverty levels over time. This can involve creating graphs or charts that show poverty trends over time. Conduct correlation analysis to identify relationships between poverty levels and other variables, such as population density, land use, or access to services. Conduct cluster analysis to identify groups of areas with similar poverty patterns. This helps identify areas that may require targeted interventions. Use data visualization tools to create maps, graphs, and charts that effectively communicate the analysis results. This can involve

82  Advances in Aerial Sensing and Imaging using software such as ArcGIS, QGIS, or Tableau. Interpret the results of the analysis to identify key insights and recommendations. This may include identifying areas of high poverty requiring targeted interventions or identifying factors contributing to poverty in specific areas. Visualization and analysis are critical components of poverty estimation using aerial images. They help identify patterns and trends in poverty levels, identify areas that require targeted interventions, and provide insights into the factors contributing to poverty. The key is using appropriate analytical tools and methods to communicate the analysis results [15] effectively.

4.3.8 Policy and Program Development The poverty estimates, and map are used to inform policy and program development aimed at reducing poverty.

4.4 Result and Discussion Our proposed methods have been compared with existing approaches to evaluate their performance. The existing techniques include those used in previous studies that analyzed the same or similar datasets. Our analysis indicates that our proposed methods outperform the current approaches regarding several evaluation metrics, such as R-squared values and mean-squared error (MSE). Specifically, our practices significantly improve the existing processes and are, therefore, more effective for the problem. In Table 4.1, we have compared our proposed methods with existing works. Only two papers have used the same Costa Rican Household dataset for poverty prediction. The first paper [17] employed conventional machine learning models such as Support Vector Machine, Naive Bayes, Logistic Regression, Gradient Boosting Classifier, Decision Trees, and KNN. Similarly, the Table 4.1  Comparison of our proposed methods with existing methods. Methods

Average MSE

Average root MSE

Average R-square

Shen et al. [17]

0.0750

0.2743

0.9396

Wang et al. [18]

0.1224

0.3498

0.9019

Min et al. [19]

0.0672

0.2591

0.9462

Proposed method

0.0534

0.2301

0.9673

Poverty Estimation using Aerial Imaging  83 Poverty prediction proposed method with other methods 1.2 1

0.9396

0.9019

0.9462

0.9673

0.2591

0.2301

0.0672

0.0534

0.8 0.6 0.4 0.2 0

0.2743 0.075 Shen et al. [17] Average MSE

0.3498 0.1224 Wang et al. [18]

Min et al. [19]

Average Root MSE

Proposed Method

Average R-Square

Figure 4.3  Proposed methods compared to previous studies.

second paper [18] also used regression methods like multinomial logistic regression, decision tree, and Gradient Boosting Classifier. Therefore, we have only compared the best-performing models of these papers with our proposed model. As per Table 4.1 [19], our model performs slightly better with the highest R-squared values and the least mean-squared error. Our proposed methods have been compared to previous studies, as shown in Figure 4.3, that used the same dataset, and the results show that our methods outperform other state-of-the-art methods. This can be attributed to our effective data-handling techniques and feature engineering procedures. Additionally, the significance of our work lies in introducing the SHAP analysis for each regression model. This technique lets us identify the crucial features contributing to our prediction models, providing valuable insights into the factors influencing the outcome. Our proposed methods offer a more comprehensive and accurate approach to the problem. The SHAP analysis provides a deeper understanding of the underlying mechanisms behind the models’ performance [13].

4.5 Conclusion and Future Scope Using aerial Imaging for poverty estimation has the potential to provide a valuable tool for assessing poverty levels in areas where ground-based surveys may be difficult or impossible to conduct. Aerial imagery can provide high-resolution images of the built environment, allowing researchers to

84  Advances in Aerial Sensing and Imaging identify poverty indicators such as housing quality, access to services and infrastructure, and other socio-economic factors. However, it is essential to note that there may be better solutions than aerial Imaging for poverty estimation. There are limitations to what can be observed from aerial images, such as the inability to capture household income or assets data. Additionally, the accuracy of poverty estimates based on aerial Imaging may depend on the quality of the images and the algorithms used to analyze them. Despite these limitations, aerial Imaging for poverty estimation shows promise and could provide valuable insights for policymakers and researchers. Further research and development of the technology and methodologies used for poverty estimation using aerial Imaging could improve the accuracy and reliability of these estimates. There are several potential future directions for poverty estimation using aerial Imaging: • Development of more sophisticated algorithms: Improving the accuracy of poverty estimates from aerial imagery requires developing more advanced algorithms that can identify more nuanced indicators of poverty. Researchers could explore machine learning or other advanced techniques to improve the accuracy of poverty estimates. • Integration with other data sources: Combining aerial imagery with other data sources such as satellite imagery, geospatial data, or ground-based surveys could provide a more comprehensive picture of poverty levels in a given area. • Application in disaster response and recovery: Aerial imagery can be used to assess the damage caused by natural disasters and identify areas that require immediate assistance. By combining aerial imagery with poverty estimates, aid organizations could better target their resources towards vulnerable populations. • Monitoring progress towards poverty reduction: Aerial imagery could be used to track progress towards poverty reduction goals over time. This could help policymakers identify areas where poverty reduction efforts are working, and more attention is needed. • Application in urban planning: Aerial imagery can provide valuable insights into urbanization patterns and the distribution of resources and infrastructure within cities. By incorporating poverty estimates derived from aerial imagery into urban planning, policymakers could work towards more equitable and sustainable cities.

Poverty Estimation using Aerial Imaging  85

References 1. Bansal, C., Jain, A., Barwaria, P., Choudhary, A., Singh, A., Gupta, A., Seth, A., Temporal prediction of socio-economic indicators using satellite imagery, in: Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pp. 73–81, 2020. 2. Chen, X. and Nordhaus, W.D., Using luminosity data as a proxy for economic statistics. Proc. Natl. Acad. Sci., 108, 21, 8589–8594, 2011. 3. Elvidge, C.D., Baugh, K.E., Zhizhin, M., Hsu, F.C., Why VIIRS data are superior to DMSP for mapping nighttime lights. Proc. Asia-Pac. Adv. Netw., 35, 62–69, 2013. 4. Jean, N., Burke, M., Xie, M., Davis, W.M., Lobell, D.B., Ermon, S., Combining satellite imagery and machine learning to predict poverty. Science, 353, 6301, 790–794, 2016. 5. Lu, X. and Weng, Q., Use of impervious surface in the remote sensing-­ based estimation of regional gross domestic product: A case study in the Greater Phoenix area, in: Remote Sensing and Modelling of Ecosystems for Sustainability, vol. 6742, Issue IV, p. 6742, 2007. 6. International Society for Optics and Photonics, World Bank, Satellite imagery and machine learning for poverty mapping, Retrieved from https://www. worldbank.org/en/topic/poverty/brief/satellite-imagery-and-machine-­ learning-for-poverty-mapping, 2018. 7. Aprianto, K., Wijayanto, A.W., Pramana, S., Deep learning approach using satellite imagery data for poverty analysis in Banten, Indonesia. IEEE International Conference on Cybernetics and Computational Intelligence (CyberneticsCom), Malang, Indonesia, pp. 126–131, 2022. 8. Sharma, R., Mandal, R.B., Garg, R.D., Estimating poverty indicators from satellite imagery: A comparative analysis of spectral indices and object-based approach in rural areas. J. Indian Soc. Remote Sens., 45, 3, 449–463, 2017. 9. WBG, High-resolution poverty maps for Bangladesh, World Bank Group, 2015, https://www.worldbank.org/en/news/feature/2015/12/07/high-resolutionpoverty-maps-for-bangladesh. 10. Abhishek, Mahajan, S., Singh, S., Informative gene selection using Adaptive Analytic Hierarchy Process (A2HP). Future Comput. Inf. J., 2, 2, 94–102, 2017. 11. Kumar, Jain, A., Rani, S., Alshazly, H., Idris, S.A., Bourouis, S., Deep neural network based vehicle detection and classification of aerial images. Intell. Autom. Soft Comput., 34, 1, 1–13, 2022. 12. Abhishek, and Singh, S., Visualization and modeling of high dimensional cancerous gene expression dataset. J. Inf. Knowl. Manage., 18, 1, 1–22, 2019. 13. Sandeep, Prasad, K.M.V.V., Srilekha, A., Suman, T., Pranav Rao, B., Naga Vamshi Krishna, J., Leaf disease detection and classification based on machine learning, in: International Conference on Smart Technologies in Computing, Electrical and Electronics (ICSTCEE), pp. 361–365, 2020.

86  Advances in Aerial Sensing and Imaging 14. Kumar, Rajan, E.G., Rani, S., Enhancement of satellite and underwater image utilizing luminance model by color correction method, in: Cognitive Behavior and Human Computer Interaction Based on Machine Learning Algorithm, pp. 361–379, 2021. 15. Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., He, Q., A comprehensive survey on transfer learning. Proc. IEEE, 109, 43–76, 2021. 16. Young, N.E., Anderson, R.S., Chignell, S.M., Vorster, A.G., Lawrence, R., Evangelista, P.H., A survival guide to Landsat pre-processing. Ecology, 98, 920–932, 2017. 17. Shen, T., Zhan, Z., Jin, L., Huang, F., Xu, H., Research on method of identifying poor families based on machine learning. Proceedings of the IEEE 4th Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), vol. 4, pp. 10–13, 2021. 18. Wang, S., Zhao, Y., Zhao, Y., Costa rican poverty level prediction. IETI Trans. Soc. Sci. Humanit., 7, 171–176, 2020. 19. Min, P.P., Gan, Y.W., Hamzah, S.N.B., Ong, T.S., Sayeed, Md.S., Poverty prediction using machine learning approach. J. Southwest Jiaotong Univ., 57, 1, 136–156, 2022. 20. Sandeep, Swetha, S., Taj Kiran, V., Johri, P., IoT based smart home surveillance and automation, in: International Conference on Computing, Power and Communication Technologies (GUCON), pp. 786–790, 2018. 21. Sandeep, Singh, S., Kumar, J., Face spoofing detection using improved SegNet architecture with a blur estimation technique. Int. J. Biom., 13, 2-3, 131–149, 2021. 22. Kumar, Rajan, E.G., Rani, S., A study on vehicle detection through aerial images: Various challenges, issues and applications, in: International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp. 504–509, 2021. 23. Haregeweyn, N., Fikadu, G., Tsunekawa, A., Tsubo, M., Meshesha, D.T., The dynamics of urban expansion and its impacts on land use/land cover change and small-scale farmers living near the urban fringe: A case study of Bahir DarEthiopia. Landscape Urban Plann., 106, 2, 149–157, 2012. 24. Wang, H., Maruejols, L., Yu, X., Predicting energy poverty with remote-­ sensing and socio-economic survey data combinations in India: Evidence from machine learning. Energy Econ., 102, 1–13, 105510, 2021. 25. Subash, S.P., Kumar, R.R., Aditya, K.S., Satellite data and machine learning tools for predicting poverty in rural India. Agric. Econ. Res. Rev., 31, 2, 231– 240, 2018. 26. Jean, N., Burke, M., Xie, M., Davis, W.M., Lobell, D.B., Ermon, S., Combining satellite imagery and machine learning to predict poverty. Science, 353, 6301, 790–794, 2016. 27. Fatima, H. et al., Poverty mapping using remote sensing: A review. Remote Sens. Rev., 36, 6, 527–549, 2019.

5 Agriculture and the Use of Unmanned Aerial Vehicles (UAVs): Current Practices and Prospects Ajay Kumar Singh1* and Suneet Gupta2 UIE-Chandigarh University, Mohali, India Mody University of Science and Technology, Lakshmangarh, Sikar, India 1

2

Abstract

Unmanned aerial vehicles (UAVs) are drones that can fly by themselves. Under the general term, “UAVs,” drones, and micro and nano-aerial vehicles all fall. The majority of unmanned aerial vehicles, or UAVs, are made up of a central control unit that is attached to one of the multiple fans or propulsion systems to lift them into the air and move them around the sky despite being developed for and initially deployed by the military, UAVs are currently used in a variety of civilian applications, including surveillance, disaster management, fire-fighting, border enforcement, and courier services. UAV applications in agriculture are particularly relevant in this chapter, focusing on their use in farming for animals and crops. This chapter covers the many UAV types and how they can be used for pest management, irrigating crops, checking animal health, rounding up livestock, setting up geofences, and other agricultural-related tasks. In addition to their applications, unmanned aerial vehicles (UAVs) in agriculture offer several advantages and potential benefits discussed, along with business-related obstacles and other unresolved issues that obstruct their widespread adoption in agriculture. Keywords:  UAV, aerial imaging, crops health, precision farming, UAV classification, aerial mustering, remote sensing, soil analysis

*Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (87–108) © 2024 Scrivener Publishing LLC

87

88  Advances in Aerial Sensing and Imaging

5.1 Introduction The military was the inspiration for the UAV’s early development. The Radio aircraft was the first UAV to capture photos for aerial surveillance in the United States in 1955. The French acquired similar capabilities in the late 1950s, the Italians in the early 1960s, and the Russians in the early 1970s. India’s Defense Research and Development Organization (DRDO) developed the lightweight, autonomous unmanned aerial vehicle (UAV) known as Idea Forge in 2010, and it was used for surveillance and reconnaissance operations. It was equipped with a thermal camera communication system. It was used for aerial photography, police department work, relief and recovery efforts, disaster management, and border infiltration monitoring [1–3]. Satellites provide expensive services, but they need to meet the growing need for spatial resolution and consistent, repeatable observations for things like crop health and soil characteristics (e.g. growth of the plants, pest or disease states, irrigation problems etc.). UAVs have been suggested for repeated unsafe, lengthy, and tiresome operations for humans [4, 5]. Aerial photos with a 1 to 2 cm resolution may be geocoded using an autonomous, stabilized drone flight, inertial guidance sensors, and even a cheap multi- or hyperspectral camera [6]. This is a comparatively high visual resolution than any other satellite-based image. The majority of industrialized nations have embraced cutting-edge technology like Computer vision and Remote Sensing for precision farming [7–9], employing Drone Units  to create an excellent agro farm with less infection. Classification of images based on pixels, with each pixel being categorized according to its spectral characteristics, has been used most frequently in studies on land use categorization. It is challenging to distinguish between different crops from a single pixel since crops have similar spectral properties. As a result, pixels with various cropping could be categorized under the same heading. Pixel-based classification decreases classification accuracy, ignoring objects on the ground with spatial characteristics [10]. In Miaoli, Taiwan, Lin et al. [6] employed fixed-wing Unmanned Arial Vehicle to take high-resolution aerial pictures of rice and citrus crops. They then used a Pix4D mapper to create high-resolution aerial photographs and a digital surface model [11]. Using a time-scaled separation of the application, such as pre-disaster activity, activity just after a disaster, and activity after the principal catastrophe elimination, Restas (2015) discussed operational and tactical drone application in disaster management [7]. Image processing and remote sensing techniques-based method for data integration into a fire-spreading model in a real-time decision support system for responding to forest fires is suggested [8]. UAVs are often

Agriculture and UAVs  89 employed for research in many different fields [11–13]. Although conventional aerial photography can gather data over a vast region, the technique is constrained by restricted mobility, high expense, and reliance on favourable weather conditions [14]. The facts in Table 5.1 show that the drone has a substantial market opportunity. Drone technology has advanced exceptionally quickly due to widespread drone use, becoming more userfriendly every day [13]. On the other hand, UAVs are very mobile and can take off from anywhere. They also have simple controls and minimal operating expenses [15–17]. Table 5.1 presents the applications of UAVs in different sectors such as mining, transportation, media, agriculture, infrastructure, insurance, telecommunication and security. The chapter begins by introducing the topic of UAVs and their applications across different sectors. It is then divided into several subsections, including classifications, agricultural use of UAVs, UAVs in livestock farming, and challenges. The classification section provides a detailed analysis of the different types of UAVs based on the number of rotors, and also presents a comparative feature-based analysis [18, 19]. The third section focuses on agricultural UAVs, providing a comparison of the various precision Table 5.1  Applications of UAVs in various industries. S. no.

Sector

Application

1.

Mining

Planning, research, and environmental effect evaluation

2.

Transportation

Delivery of goods and parcels, Medical Logistic

3.

The Media and Entertainment

Aerial photography, shows, special effects, advertising, and entertainment

4.

Agriculture

Analysis of drainage and soils, Monitoring crop health, predicting yields, Spot application of pesticides and fertilizers

5.

Infrastructure

Monitoring investments, maintenance, and asset inventories

6.

Insurance

Assistance in claims processing and fraud detection

7.

Telecommunication

Upkeep of towers and signal transmission

8.

Security

Monitoring systems and locations, proactive action

90  Advances in Aerial Sensing and Imaging agricultural RS platforms available. The fourth section discusses livestock farming, which involves raising animals for food, fertilizer, leather, fur, or medicine, and explores the ways in which UAVs are used in this sector [20–22]. Finally, the chapter concludes by summarizing the main ideas and findings presented, as well as offering suggestions for future research.

5.2 UAVs Classification UAVs are vehicles weighing 25 kg or less that may be operated remotely and do not require a pilot to fly. UAVs may be used for picture analysis, ground monitoring, and detailed crop situation assessments. We can classify UAVs into several categories based on the number of rotors, speed, use, mechanism, etc. While the vehicle is taking off, weight might be a key indicator for differentiating between the UAVs. First, we may observe large UAVs weighing two or more tones [23–25]. They are primarily utilized for military objectives and will be able to carry enough fuel. Second, some UAVs weigh between 50 and 200 kg and 200 to 2000 kg, are widely utilized for many different purposes and have a larger fuel capacity. Finally, small, light unmanned aerial vehicles (UAVs) used in agriculture weigh between 5 and 50 kg. Furthermore, compared to bulkier vehicles, micro-UAVs that weigh below 5 kg are easier to launch and less costly. Figure 5.1 depicts the classification of UAVs as Fixed-wing-single-rotor, Fixed-wing-multi-rotor, and hybrid vertical takeoff (VTOL) landing capabilities may be added to UAVs [26, 27]. The fixed-wing and multi-rotor structures are very different from one another. Depending on the number of rotors and the intended use, a multi-rotor can be a tricopter, quadcopter, hexacopter, or octocopter [13]. • Fixed Wing Drone Since fixed-wing UAVs use less energy when cruising, they are better suited for missions requiring prolonged endurance as shown in Figure 5.2. Many UAVs require a runway or a big open area for takeoff and landing. A venue like that might not always be accessible [28]. • Single Rotor Drone Drones with a single rotor are robust and long-lasting. They resemble helicopters in terms of construction and design as shown in Figure 5.3. A single-rotor aircraft has only one rotor, similar to one sizeable spinning wing, and an additional tail rotor for direction and stability control [29].

Agriculture and UAVs  91

Fixed Wing Types of UAVs

Single Rotor

Hybrid-VTOL

Multi Rotor

Tricoptor

Quadcoptor

Hexacoptor

Octocoptor

Figure 5.1  Classification of UAVs. Rudder Elevator Left Aileron

Right Aileron x

Roll

Figure 5.2  Fixed-wing UAV.

• Multi-Rotor Drone Using a multi-rotor UAV is the simplest and least expensive method for eyeing the sky. They are ideal for aerial photography and surveillance since they give you more control over positioning and framing. Figure 5.4 referred to as

92  Advances in Aerial Sensing and Imaging Single Rotor Rudder

Figure 5.3  Single rotor UAV.

(a)

(c)

(b)

(d)

Figure 5.4  Multi-rotor UAVs (a) Tricoptor, (b) Quadcopter, (c) Hexacopter, and (d) Octacoptor.

Agriculture and UAVs  93 multi-rotor since they include more than one motor, and examples include but are not limited to, tricopters (3 rotors), quadcopters (4 rotors), hexacopters (6 rotors), and octocopters (8 rotors). Multi-rotor UAVs are shown in Figure 5.4. The most common multi-rotor drones by far are quadcopters [30]. It offers superior aircraft control while in flight. Thanks to its improved maneuverability, it can move from the front to back, side to side, and spin on its axis. It can also go up and down on the same vertical axis [31–33]. It can fly significantly closer to objects like buildings and structures. Its capacity to carry multiple payloads on a single trip improves operational effectiveness and reduces inspection time.   The Tricopter’s three rotors, part of the main framework, assist the craft in balancing its weight while in flight shown in Figure 5.4(a). Because of the method in which the rotors turn, the right rotor will continuously be rotating in a counterclockwise direction. As for the remaining two rotors, utilizing a servo technique, the imbalanced clockwise torque is negated by sloping the tail-mounted rotor in the opposite direction [16]. Regarding UAV design, quadcopters are superior since they have four rotors. These rotors produce this model’s thrust. According to Figure 5.4(b), two of these four rotors are positioned in opposition and revolve clockwise, while the other two twist counterclockwise. This model can rotate around its axis in all directions, including forwarding and backward motions known as “pitch,” lateral motions known as “roll,” and clockwise and counterclockwise motions known as “yaw” [17].   Hexa is the Greek word for six. Six arms comprise a hexacopter drone, each connected to a single high-speed BLDC motor. Figure 5.4(c) shows that the airframe is built of glass fiber [18]. The drone’s other components, including its batteries, motor, flight-controlled GPS antenna, and high-speed tube, are supported by the airframe plate, which also serves as its primary structural support [34–36]. It includes circuit boards, sensors, ESC, and FPV cameras. Like a hex UAV, an octocopter is a flying machine with eight rotors spraying crops. This has a flight time of 15 minutes with a cargo that weighs 10 kg and a diagonal wheelbase that measures 163 cm

94  Advances in Aerial Sensing and Imaging Lifting Rotor

Pusher Rotor

Elevator

Figure 5.5  Design of hybrid VTOL UAV.

in diameter, as shown in Figure 5.4(d) [19]. The movement of the spray deposition was influenced by two factors, including rotor speed and spray nozzle location. • Hybrid VTOL Translational Aircraft (TA) are a new class of drones that have been developed due to the growth in the missions’ difficulty of both military aircraft and Unmanned Aerial Vehicles (UAVs), as well as the growth in civilian air traffic with the restricted runway [37–39]. The purpose of TA is to utilize the qualities of rotorcraft, hovering, low-speed flying, and vertical takeoff and landing, as well as those of fixed-wing aircraft, such as high speed, range, and endurance. They are also called Hybrid VTOL, the design shown in Figure 5.5. In this particular instance, horizontal, vertical, and transition controllers are used [20].

5.2.1 Comparison of Various UAVs Figure 5.6 compares the average weights, size of payloads, and flight times of the four most popular UAV models. The documentation provided by the

Agriculture and UAVs  95 30

5 4

20

3

15 2

10

Time (hrs)

Weight (kg)

25

1

5 0

0 Multi-Rotor UAVs Fixed-Wing UAVs Weight (kg)

Single-Rotor Helicopter

Payload (kg)

FWR-Hybrid UAVs Flight time (hrs)

Figure 5.6  UAV flight time versus weight [17].

manufacturer and the operator’s user guide for each product were used to compile the shown figures. Table 5.2 displays their flight specifications. A specific UAV model was selected for each category. Compared to other types of UAVs, the Hybrid VTOL produced greater flying altitudes, a more excellent control range, faster speeds, and extended flight periods, exhibiting its advantages.

Table 5.2  UAVs comparison by features [17]. Elevation (Km)

Average range (km)

Average airspeed (meter/sec)

Multirotor UAV (DJI Agras MG-1P)

2

3–5

7

Fixed wing UAV (AgEagle RX60)

0.125

2

18.8

Singlerotor (Alpha 800)

3

30

15.2

Fixed wing multirotor hybrid UAV (Jump 20)

4

500–1000

30

UAV category

96  Advances in Aerial Sensing and Imaging

5.3 Agricultural Use of UAVs Food security is a challenging problem to solve. Because of the continuously growing population. Over 815 million people in the world suffer from chronic hunger, with Asia home to 64% of those people, according to the Food and Agriculture Organization (FAO) of the United Nations. To feed a population of nine billion people by the year 2050, the world’s food supply must grow by almost 50% [20]. The COVID-19 epidemic left a lasting impression on the networks that produce crops and deliver food. Many farmers needed help to meet their fundamental needs for agriculture, such as labor, seeds, fertilizer, and pesticides, promptly, which reduced output [40–42]. The COVID-19 epidemic makes it highly challenging for conventional farmers to supervise crop, fertilizer, and pesticide applications [21]. • Qualitative Analysis of Remote Sensing Platform Aerial Remote Sensing (RS) is one of the most crucial technologies for Precision Agriculture (PA) and intelligent agriculture. Remote sensing of crops has changed due to improvements in Unmanned Aerial Vehicle (UAV) technologies and the weight reduction of payload devices. This method of high-resolution picture acquisition is non-­ destructive, more economical, and time-saving [22, 23]. A thorough evaluation of the level of services offered by the various kinds of RS Platforms in Precision Agriculture is provided in Table 5.3. • Examples of Farming UAVs In the recent ten years, fixed-wing and helicopter aircraft have dominated the unmanned aerial vehicle (UAV) industry. The focus of today’s deployment of tiny drones in agriculture has turned to multi-copters, as per Table 5.4 which currently account for approximately half of the UAV models on the market [24]. ◦◦ Soil and Field Analysis Farmers are using agricultural technology to get more precise information on their land to make better decisions. Droneenabled multispectral remote sensing for soil mapping is one example of more recent technology that can evaluate acres of farmland utilizing cutting-edge sensor technology [25]

Agriculture and UAVs  97 Table 5.3  Comparison of the service levels of the various precision agricultural RS platforms. Service level quality

Remote sensing platform UAV

Satellite

Manned aircraft

Ground based

Operability

E

C

C

E

Cost

L

H

H

L

Risk

L

A

H

L

Flexibility

H

L

L

L

Adaptability

H

L

L

L

Time Consumption

L

L

L

H

Accuracy

H

L

H

M

Feasibility

Y

N

N

Y

Availability

Y

N

Y

N

E-easy. C-complex. L-low, H-high, M-moderate, A-average, Y-yes, N-no

as shown in Figure 5.7. Using the electromagnetic spectrum, drones with remote sensing cameras gather data from the land to analyze the AreaArea and the soil. The range of wavelengths that various elements reflect can be utilized to differentiate between them. Drones gather raw data and convert it into informative data using algorithms [43, 44]. The monitoring of the following parameters is just one of the many farming applications that can make use of them: (i) Growth of plants, including plant height, leaf area index, and plant density. (ii) Evaluation of plant: the size of the plant, data for the field, stand number, compromised area, and planter skips. (iii) Examining the plants for size, statistics about the field, stand number, contaminated fields, and planter skips. (iv) Soil investigation: nutrient availability for plants, nutrient content in plants.

98  Advances in Aerial Sensing and Imaging

Table 5.4  Farming applications of UAVs. Farming applications

Piloted aircrafts [31]

Fixed wing UAVs

Single rotor UAVs

Multi-rotor UAVs

Crop Scouting

Yes

Yes

Yes

Yes

Fertilizer and Pesticide Spray

For larger AreaArea (Can quickly cover crop areas that are well over 100 hectares in size)

For Medium Area

For larger AreaArea For Small Area (Incorrect coverage of some agricultural fields during spraying)

Drought Monitoring

Yes

Yes

Yes

Yes

Crop Height Estimations

No

Yes

Yes

Yes

Soil and Field Analysis

No

No

Yes

Yes

Crop Classification No

No

Yes

No

Cost-Range

Very High-Priced

Medium-high Priced

High-Priced

Low-Medium Priced

Commercial Farming UAVs [30]

• M-18 Dromader • PZL-106AR Kruk • Grumman Ag Cat

• • • • •

• • • • •

• • • • • •

AgEagle RX60 eBee Ag Precision Hawk Lancaster 5 Sentera Phoenix 2 Trimble UX

Yamaha RMAX R22-UV R66 spray system Align Demeter E1 SR20 and SR200 of roto-motion

DJI Phantom 4 PRO AGCO Solo Sentera Omni Ag SenseFly eXom AgBot InDago AG

Agriculture and UAVs  99

Figure 5.7  UAV capturing raw images for soil analysis.

◦◦ Seed Planting About 15 billion trees are felled yearly for mining, urban sprawl, and horizontal agricultural growth [26, 27]. These drones employ a pneumatic firing mechanism during certain circumstances, such as in mountainous land or mangrove jungles, and shoot seed pods deeper into the soil. Up to 40,000 seeds can be planted into the earth by two flying drones each day as shown in Figure 5.8. A drone can grow faster than a typical human in just 10 minutes [45]. ◦◦ Crop and Spot Spraying It is increasingly essential for crop health to spray pesticides to kill pests and undesirable plants like weeds. Drones can

Figure 5.8  UAV-based seed planting.

100  Advances in Aerial Sensing and Imaging

Figure 5.9  Spraying operation by UAV.

transport suitable-sized containers loaded with fertilizers, insecticides, herbicides, Plant Growth Hormones, etc., for faster spraying [46]. Figure 5.9 depicts a UAV carrying fertilizers spraying on the crop. Spraying takes less than 40 minutes to complete on a hect field owing to a spraying capacity that ranges from two to five times faster than conventional technology. ◦◦ Monitoring of Crops Crop health may be tracked over time, and responses to corrective treatments can be observed using drones equipped with sensors that can analyze crop health using visible and near-infrared light. Farmers can better understand and develop innovative strategies for increasing crop yields while minimizing crop damage using data from modern sensors shown in 2D or 3D. ◦◦ Irrigation Monitoring Drones with thermal cameras and remote sensing capabilities can help divide the areas according to different moisture regimes and solve irrigation-related difficulties. This supports precise irrigation planning. The ground resolution of the photogrammetric and navigational technology of the FAO drones used in the Republic of the Philippines can be as high as 3 cm [28].

Agriculture and UAVs  101 ◦◦ Health Assessment Drones equipped with sensors capable of crop scanning using visible and near-infrared light are utilized to track crop health over time and the effectiveness of corrective treatments. NDVI, water stress, and a shortage of a specific nutrient in crops are just a few details that can be programmed into this [29, 30].

5.4 UAVs in Livestock Farming Raising animals for food and/or additional purposes like fertilizer, leather, fur, or medicine is known as livestock farming. The authors in [32, 33] demonstrated how Livestock Production Systems (LPS) have historically been divided into three main classes: agroecological, land-based, and livestock production linked with crop production. Like [34], the authors examined five different tropical pasturage systems based on agroecological regions, animal class, function, and management [47, 48]. The recognized categories included pastoral range, crop-livestock, ranching, and landlessness. • Headcount of Cattle The most widely cultivated animals in terms of livestock are sheep and goats, then cattle. Thus, UAVs may be employed to undertake headcounts of animals throughout these vast grazing regions. Animals may be counted using thermal sensors that detect heat or image recognition software [34]. Convolutional Neural Network (CNN) has recently become the most popular technique for processing images. Additionally, the UAVs can be used to locate and count the number of animals across vast grazing areas. However, in the study by [33], the authors suggested a method for counting and tracking the number of goats using fewer images, often just one. According to the authors, tracking accuracy was 78%, and count accuracy was 73%. • Animal Health Monitoring Beyond counting, Texas A&M University is researching how to use infrared cameras installed on unmanned aerial vehicles (UAV) to monitor animal health. The basis of the

102  Advances in Aerial Sensing and Imaging

Figure 5.10  Cattle herd’s heat map [36].

study is that animals with fever typically have elevated body temperatures. The UAV can quickly identify this and provide the necessary treatment [36]. Animals can be individually identified using RFID tags or other sensors as shown in Figure 5.10; they can be watched using UAVs. Farmers can efficiently monitor an animal’s movements and feeding habits [38]. This has also been heavily utilized in monitoring endangered species raised in captivity and released into the wild. Figure 5.10 displays an example of a heat map captured by a UAV of a herd of cattle. • Tracking and Identification of Livestock Animals can be individually identified using RFID tags or other sensors and can be watched using UAVs. Farmers can use this to track an animal’s feeding and movement patterns efficiently. UAVs fitted with standard cameras, infrared cameras (that detect thermal emissions from the animals), or RFID readers, identical to the two application areas described above, can be used to perform the identification [39]. The usage of unmanned aerial vehicles (UAVs) in conjunction with radio frequency identification (RFID) tags is depicted in Figure 5.11. • Aerial Mustering Finding and collecting animals across a large land region with the assistance of aircraft (UAV) is a procedure known as mustering. Sheepdogs, cowboys on horses, and motorcycles

Agriculture and UAVs  103

Remote RFID Reader

50 mtr

Figure 5.11  Employing relay UAVs to identify animals.

have all been used to guide cattle down specific routes. UAVs provide a unique possibility for aerial mustering since they are less dangerous, more affordable to operate, and require less training time while yet being able to produce equivalent outcomes. DJI Ferntech [40] claims that sheep, deer, and cattle are herded using aerial mustering UAVs equipped with sirens. The UAVs may direct cattle to places for eating, drinking, and milking as shown in Figure 5.12. Figure 5.13 shows different type of commercial drones utilized for different application.

Figure 5.12  Aerial mustering.

104  Advances in Aerial Sensing and Imaging

(a)

(b)

(c)

Figure 5.13  Commercial drones for agriculture. (a) agricultural spraying drone: Fly dragon FDXD-4R-10L (b) V-22 Osprey (c) AG drone T1-24L.

5.5 Challenges UAVs feature a broad range of possibilities in intelligent cities, each of which has a significant positive impact on the growth of any smart city. The researchers highlighted some of the difficulties with using UAVs. Although these studies concentrated on applications for smart cities, many of these difficulties apply to the agricultural sector [41]. The constraints, broadly divided into business and technological issues, include Cost, Licensing and regulation issues, Business Adoption, technical challenges, ethics and privacy. The authors [29] approached the deployment of UAV-related projects from the standpoint of project management, and they highlighted cost as a crucial factor that needs to be taken into account. Additionally, it was underlined that accurate estimates must be made employing various techniques before beginning any such venture. For UAVs, there still needs to be more clarity regarding licensing and regulating matters. Regulations either don’t exist or are a slack adaption of aviation laws that only sometimes work with UAVs. Countries such as the United States of America, the United Kingdom, Germany, and Spain [42, 43] are setting an example for other countries to follow by adopting guidelines for using unmanned aerial vehicles (UAVs) and the territories across which they are permitted to fly [7].

Agriculture and UAVs  105

5.6 Conclusion Drone technology offers great promise for performing several agricultural tasks efficiently. According to projections by the Association for Unmanned Vehicle Systems International (AUVSI), more than 2,900 UAVs will be employed by more than 900 organizations worldwide by 2020. The two most significant obstacles to conquer to make it appealing to farmers and widespread use are high startup costs and policy modifications. These techniques and technologies are quickly growing in their use in agriculture to give information on farming, notwithstanding their limits.

References 1. Colomina, I., Unmanned aerial systems for photogrammetry and remote sensing: A review. ISPRS J. Photogramm. Remote Sens., 92, 79–97, 2014. 2. Everaerts, J., The use of unmanned aerial vehicles (UAVs) for remote sensing and mapping. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci., 37, 1187– 1192, 2008. 3. Natu, A.S. and Kulkarni, S.C., Adoption and utilization of drones for advanced precision farming: A review. Int. J. Recent Innov. Trends Comput. Commun., 4, 563–565, 2016. 4. Zhang, C. and Kovacs, J.M., The application of small unmanned aerial systems for precision agriculture: A review, in: Precision Agriculture, vol. 13, pp. 693–712, Springer, Berlin/Heidelberg, Germany, 2012. 5. Yang, M.D., Huang, K.S., Kuo, Y.H., Hui, T., Lin, L.M., Spatial and spectral hybrid image classification for rice lodging assessment through UAV imagery. Remote Sens., 9, 583, 2017. 6. Lin, F.Y., Chang, S.C., Feng, Y.Y., Chen, Y.W., Evaluation for application of sensing technology to monitor on agricultural loss, in: Proceedings of the Symposium on Agricultural Engineering and Automation Project Achievements, Taipei, Taiwan, 1 March 2015. 7. Restas, A., Drone applications for supporting disaster management. World J. Eng. Technol., 03, 316–321, 2015. 8. Zharikova, M., Sherstjuk, V., Sokol, I., Forest fire-fighting monitoring system based on UAV team and remote sensing, 2018. 9. Al-Najjar, H.A.H., Kalantar, B., Pradhan, B., Saeidi, V., Halin, A.A., Ueda, N., Mansor, S., Land cover classification from fused DSM and UAV images using convolutional neural networks. Remote Sens., 11, 1461, 2019. 10. Michez, A., Piégay, H., Lisein, J., Claessens, H., Lejeune, P., Classification of riparian forest species and health condition using multi-temporal and hyperspatial imagery from unmanned aerial system. Environ. Monit. Assess., 188, 146, 2016.

106  Advances in Aerial Sensing and Imaging 11. Torres-Sánchez, J., López-Granados, F., Peña, J.M., An automatic objectbased method for optimal thresholding in UAV images: Application for vegetation detection in herbaceous crops. Comput. Electron. Agric., 114, 43–52, 2015. 12. Ma, L., Li, M., Ma, X., Cheng, L., Du, P., Liu, Y., A review of supervised object-based land-cover image classification. ISPRS J. Photogramm. Remote Sens., 130, 277–293, 2017. 13. Islam, N., Rashid, M.M., Pasandideh, F., Ray, B., Moore, S., Kadel, R., A review of applications and communication technologies for Internet of Things (IoT) and unmanned aerial vehicle (UAV) based sustainable smart farming. Sustainability, 13, 182, 2021. 14. McArthur, D.R., Chowdhury, A.B., Cappelleri, D.J., Design of the interacting-boomcopter unmanned aerial vehicle for remote sensor mounting. J. Mech. Robot., 10, 025001, 2018. 15. Sharma, R., Review on application of drone systems in precision agriculture. J. Adv. Res. Electron. Eng. Technol., 7, 520137, 2021. 16. Delavarpour, N., Cengiz, K., Nowatzki, N., Bajwa, S., Sun, X., A technical study on UAV characteristics for precision agriculture applications and associated practical challenges. Remote Sens., 13, 1204, 2021. 17. Yinka-Banjo, C. and Ajayi, O., Sky-farmers: Applications of unmanned aerial vehicles (UAV) in agriculture, in: Autonomous Vehicles, IntechOpen, American University of Cyprus, 2019. 18. Hunt, E.R., Jr., Hively, W.D., Fujikawa, S.J., Linden, D.S., Daughtry, C.S.T., McCarty, G.W., Acquisition of NIR-green–blue digital photographs from unmanned aircraft for crop monitoring. Remote Sens., 2, 290–305, 2010. 19. Kalantar, B., Mansor, S.B., Sameen, M.I., Pradhan, B., Shafri, H.Z.M., Dronebased land-cover mapping using a fuzzy unordered rule induction algorithm integrated into object-based image analysis. Int. J. Remote Sens., 38, 2535– 2556, 2017. 20. Ehrlich, P.R. and Harte, J., Opinion: To feed the world in 2050 will require a global revolution. PNAS, 112, 14743–14744, 2015. 21. Varshney, D., Roy, D., Meenakshi, J.V., Impact of COVID-19 on agricultural markets: Assessing the roles of commodity characteristics, disease caseload and market reforms. Indian Econ. Rev., 55, 83–103, 2020. 22. Puri, V., Nayyar, A., Raja, L., Agriculture drones: A modern breakthrough in precision agriculture. J. Stat. Manage. Syst., 20, 507–518, 2017. 23. El Bilali, H. and Allahyari, M.S., Transition towards sustainability in agriculture and food systems: Role of information and communication technologies. Inf. Process. Agric., 5, 456–464, 2018. 24. Marinello, F., Pezzuolo, A., Chiumenti, A., Sartori, L., Technical analysis of unmanned aerial vehicles (drones) for agricultural applications, 2016. 25. E-agriculture in action: Drones for agriculture, Food and Agriculture Organization of the United Nations and International Telecommunication Union Bangkok, 2018.

Agriculture and UAVs  107 26. Pravin, K. and Munde, T.N., Use of drones for efficient water management. 3rd World Irrigation Forum, vol. 2, 5, pp. 1–7, 2019. 27. DJI Ferntech, Drones on the farm [video file], 2018, Available: fhttps://www. djistore.co.nz/agriculture. 28. Mohammed, F., Idries, A., Mohamed, N., Al-Jaroodi, J., Jawhar, I., UAVs for smart cities: Opportunities and challenges, in: 2014 International Conference on Unmanned Aircraft Systems (ICUAS), IEEE, pp. 267–273, 2014. 29. Idries, A., Mohamed, N., Jawhar, I., Mohammed, F., Challenges of developing UAV applications: A project management view, in: Proceedings of the 2015 International Conference on Industrial Engineering and Operations Management, Dubai, United Arab Emirates (UAE), March 3-5, 2015. 30. Best drones for agriculture 2020: The ultimate buyer’s guide, https://bestdroneforthejob.com/drone-buying-guides/agriculture-drone-buyers-guide/. 31. Patel, P.N., Patel, M.A., Faldu, R.M., Dave, Y.R., Quadcopter for agricultural surveillance. Adv. Electron. Electr. Eng., 3, 427–432, 2013. 32. Steinfeld, H. and Mäki-Hokkonen, J., A classification of livestock production systems. World Anim. Rev., 84/85, 83–94, 1995. 33. Vayssade, J., Arquette, R., Bonneau, M., Automatic activity tracking of goats using a drone camera. Comput. Electron. Agric. Elsevier, 162, 767–772, 2019. 34. Havens, K. and Sharp, E., Thermal imaging techniques to survey and monitor animals in the wild: A methodology, Academic Press, 2015. 35. Yusouf, M., Chuang, K.L., Basuno, B., MohdDahlan, S.F., Development of a versatile UAV platform for agricultural applications, in: Proceedings of First Regional Conference on Vehicle Engineering & Technology, Kuala Lumpur, Malaysia, 3-5 July, 2006. 36. Texas A&M AgriLife, Drones could apply thermal imaging to identify sick livestock in feedlots, 2019, Available from: https://research.tamu. edu/2019/03/07/drones-could-apply-thermal-imaging-to-identify-sick-livestock-in-feedlots/. 37. Zhang, C. and Kovacs, J.M., The application of small unmanned aerial systems for precision agriculture: A review. Precis. Agric., 13, 6, 693–712, 2012. 38. Nyamuryekunge, S., Cibils, A., Estell, R., Gonzalez, A., Use of an unmanned aerial vehicle—Mounted video camera to assess feeding behaviour of raramuri criollo cows. Rangel. Ecol. Manage., 2016, 69, 386–389, 2016. 39. Ma, Y., Selby, N., Adib, F., Drone relays for battery-free networks, in: Proceedings of the Conference of the ACM Special Interest Group on Data Communication, ACM, pp. 335–347, 2017. 40. Stone, E., Drones spray tree seeds from the sky to fight deforestation, 2017, Available at https://www.nationalgeographic.com/science/article/ drones-plant-trees-deforestation-environment. Accessed on 20-09-2021. 41. Yitbarek, M. and Berhane, G., Livestock production systems and analysis: Review. AIJCSR, 1, 2, 16–51, 2014. 42. Matese, A., Toscano, P., Di Gennaro, S.F., Genesio, L., Vaccari, F.P., Primicerio, J., Belli, C., Zaldei, A., Bianconi, R., Gioli, B., Intercomparison of UAV, aircraft

108  Advances in Aerial Sensing and Imaging and satellite remote sensing platforms for precision viticulture. Remote Sens., 7, 2971–2990, 2015. 43. Seré, C., Steinfeld, H., Groenewold, J., World livestock production systems, Food and Agriculture Organization of the United Nations. FAO Publishing, 1996. 44. Kumar, Jain, A., Rani, S., Alshazly, H., Idris, S.A., Bourouis, S., Deep neural network based vehicle detection and classification of aerial images. Intell. Autom. Soft Comput., 34, 1, 1–13, 2022. 45. Bhola, Srivastava, S., Noonia, A., Sharma, B., Narang, S.K., A status quo of machine learning algorithms in smart agricultural systems employing IoTbased WSN: Trends, challenges and futuristic competences, in: Machine Intelligence, Big Data Analytics, and IoT in Image Processing: Practical Applications, pp. 177–195, 2023. 46. Abhishek, and Singh, S., Visualization and modeling of high dimensional cancerous gene expression dataset. J. Inf. Knowl. Manage., 18, 1, 1–22, 2019. 47. Sandeep, Rajan, E.G., Rani, S., A study on vehicle detection through aerial images: Various challenges, issues and applications, in: International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp. 504–509, 2021. 48. Kumar, Rajan, E.G., Rani, S., Enhancement of satellite and underwater image utilizing luminance model by color correction method, in: Cognitive Behavior and Human Computer Interaction Based on Machine Learning Algorithm, pp. 361–379, 2021.

6 An Introduction to Deep Learning-Based Object Recognition and Tracking for Enabling Defense Applications Nitish Mahajan1*, Aditi Chauhan2 and Monika Kajal3

*

Department of Computer Science and Engineering, University Institute of Engineering and Technology, Panjab University, Chandigarh, India 2 DPIIT-IPR Chair, Panjab University, Chandigarh, India 3 DST-CPR, Panjab University, Chandigarh, India

1

Abstract

Object monitoring and surveillance technologies are crucial in defense, border protection, and counter-terrorism operations. These technologies enable military and security personnel to monitor and track the movement of objects and individuals in high-risk areas, detect potential security threats, and respond effectively to intrusions or attacks. In defense, object monitoring and surveillance technologies are used to see troop movements, monitor enemy activities, and provide real-time intelligence to military commanders. These technologies include radar systems, unmanned aerial vehicles (UAVs), and satellite imagery. By providing early warning of enemy movements and threats, these technologies help military personnel to respond quickly and effectively, increasing their chances of success. In border protection, object monitoring and surveillance technologies detect illegal border crossings, drug trafficking, and smuggling activities. These technologies include thermal imaging cameras, ground sensors, and surveillance UAVs. By providing real-time information about border activities, these technologies help border control personnel detect and apprehend illegal activities, reducing the risk of border incursions and other security threats. In counter-terrorism operations, object monitoring and surveillance technologies detect potential threats and prevent terrorist attacks. These technologies include facial recognition systems, biometric scanners, and other advanced monitoring systems. By identifying potential dangers before they can carry out attacks, these technologies help security personnel *Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (109–128) © 2024 Scrivener Publishing LLC

109

110  Advances in Aerial Sensing and Imaging to prevent terrorist activities and safeguard the public. In conclusion, object monitoring and surveillance technologies are critical in defense, border protection, and counter-terrorism operations. By enabling military and security personnel to detect and respond to potential threats quickly and effectively, these technologies help safeguard national security and protect citizens from harm. Keywords:  Object detection, deep learning, object tracking, YOLOv7, small object detection

6.1 Introduction Object monitoring and surveillance play a critical role in defense and security operations. Detecting, tracking, and classifying objects in real-time is essential for military and defense applications such as border security, airspace management, and naval operations. Object monitoring and surveillance also enable intelligence gathering by providing valuable information on the movements and activities of potential threats. This information is critical for situational awareness and decision-making in defense operations. Emerging technologies such as artificial intelligence, machine learning, and unmanned aerial vehicles (UAVs) are revolutionizing object monitoring and surveillance capabilities in defense [1]. These technologies enable faster and more accurate object detection, classification, and tracking, as well as the ability to gather data from multiple sources and fuse it into a comprehensive situational picture. The increasing complexity and sophistication of security threats further emphasize the importance of object monitoring and surveillance in defense. With the proliferation of unmanned and autonomous systems, the ability to detect and counter potential threats has become more challenging and critical than ever before. Object monitoring and surveillance are essential to defense and security operations, enabling real-time situational awareness, intelligence gathering, threat detection, and countermeasures. The remainder of the paper is structured as follows. Section 6.2 provides a literature review of state of the art. Section 6.3 presents a detailed description of the experimental methods. Section 6.4 explains the results and outcomes. Furthermore, the Conclusion and future scope are discussed in Sections 6.5 and 6.6, respectively.

Deep-Learning-Based Object Recognition  111

6.2 Related Work This section sheds light on the importance and need of the technology in the defense domain and then discusses state-of-art object detection and tracking techniques.

6.2.1 Importance of Object Monitoring and Surveillance in Defense The progress in the technology of imaging hardware and the introduction of deep learning techniques into the world of computer vision has opened gates for autonomous computer-aided applications in every walk of life. These advancements have enabled computers to distinguish and identify objects in a given picture or video, thus removing the monotonous and labor-intensive work of humans. Monitoring and surveillance are critical components of the Indian defense services, providing essential information to ensure national security. With the introduction of surveillance equipment such as day and night vision cameras and radar-based surveillance equipment over the length of our borders with hostile forces, it becomes paramount to utilize the infra optimally and with almost zero scopes for error. In such cases, computer-­ aided object monitoring and surveillance can spearhead the modernization and capacity enhancement of the Indian defense services [2]. Monitoring and surveillance are critical components of security Figure 6.1 shows various outcomes of an efficient monitoring and surveillance infrastructure. Following are some reasons why Monitoring and surveillance are essential in security:

Identification of potential threats

Prevention of crime

Evidence gathering

Monitoring and Surveillance

Situational awareness

Protection of assets

Figure 6.1  Various outcomes of an efficient monitoring and surveillance infrastructure.

112  Advances in Aerial Sensing and Imaging • Identifying potential threats: Monitoring and surveillance can help identify potential hazards, such as suspicious activity or individuals, and allow timely intervention to prevent harm [3]. • Crime prevention: Surveillance cameras and other monitoring tools can deter potential criminals, preventing crime before it occurs. • Evidence gathering: Surveillance footage can be used as evidence in criminal investigations and trials, aiding law enforcement in identifying suspects and building a case. • Situational awareness: Monitoring and surveillance can help security personnel maintain situational awareness, allowing them to respond quickly and effectively to any threats or incidents [4]. • Protection of assets: Monitoring and surveillance can help protect valuable assets, such as buildings, equipment, and intellectual property, by identifying and preventing unauthorized access or theft.

6.2.2 Need for Object Monitoring and Surveillance in Defense Monitoring and surveillance are crucial to the Indian defense forces, serving several vital functions to ensure national security. Autonomous Monitoring and surveillance have gained much traction among security forces recently. The need for object monitoring and surveillance in the Indian defense forces can be discussed in detail as follows: • Border Security: India shares borders with several neighboring countries, and effectively monitoring these borders is critical to prevent illegal crossings, smuggling, and other criminal activities. Object monitoring and surveillance play a crucial role in border security by detecting and tracking the movements of people and vehicles across borders. This helps authorities identify potential threats and prevent unauthorized access to sensitive areas. Adequate border security is essential to prevent the entry of terrorists, smugglers, and other criminals into the country [5]. • Military Operations: Object monitoring and surveillance are also essential in military operations. By monitoring the movements of enemy forces, Indian military personnel can gain valuable intelligence about the enemy’s plans

Deep-Learning-Based Object Recognition  113 and capabilities, allowing them to plan and execute effective responses. This information is critical to ensure the success of military operations and protect military personnel and assets from attacks and other threats. Effective object monitoring and surveillance can help Indian forces to gain a tactical advantage on the battlefield [6]. • Maritime Security: India has a long coastline and a vast maritime boundary, which makes maritime security a critical area of concern. Object monitoring and surveillance are crucial in ensuring naval security by detecting and tracking the movements of ships, submarines, and other vessels [7]. This information helps authorities identify potential threats and respond effectively to security breaches. Adequate maritime security is essential to prevent the entry of terrorists, smugglers, and other criminals through sea routes [8]. • Counter-Terrorism: India faces a significant threat from terrorism, and object monitoring and surveillance are critical tools in the fight against terrorism. These tools help authorities to detect and track the movements of terrorists and their associates, identify potential targets, and prevent terrorist attacks. Effective object monitoring and surveillance can help Indian forces to prevent terrorist attacks and neutralize terror cells [9]. • Disaster Management: Object monitoring and surveillance can also manage natural disasters and other emergencies. For example, surveillance cameras can monitor flood-prone areas and alert authorities to potential flooding. Similarly, satellite imagery can track the movement of cyclones and other weather systems, enabling management to prepare and respond effectively to natural disasters. Effective object monitoring and surveillance can help Indian forces to mitigate the impact of natural disasters and save lives.

6.2.3 Object Detection Techniques Object detection is a computer vision task that involves identifying and localizing objects within an image or video. It is a fundamental task in computer vision and has many real-world applications, including self-driving cars, surveillance systems, robotics, and object tracking. Object detection typically involves two main steps: object localization and object classification. In the localization step, the algorithm detects and localizes the objects

114  Advances in Aerial Sensing and Imaging by drawing bounding boxes around them. In the classification step, the algorithm assigns a class label to each detected object based on its visual characteristics [10]. Object detection has many vital applications, including pedestrian detection, vehicle detection, face detection, and object tracking. It is challenging due to the variability in object appearance, size, shape, occlusions, clutter, and background noise. Nevertheless, recent advances in deep learning and computer vision have led to significant improvements in object detection performance and have enabled the development of new and innovative applications [11]. Object detection algorithms can be divided into the following main categories: • Two-Stage Detectors: These techniques use a two-stage approach that involves generating region proposals in the first stage and then classifying and refining them in the second stage. Examples of two-stage detectors include Faster R-CNN, R-FCN, and FPN [12]. • One-Stage Detectors: One-stage detectors, such as YOLO, SSD, and RetinaNet, directly predict object classes and bounding boxes in a single pass through a deep neural network. They are faster than two-stage detectors but may sacrifice some accuracy [13]. • Multi-Task Learning: Multi-task learning techniques perform multiple related tasks, such as object detection and segmentation, simultaneously using a shared neural network. Examples of multi-task learning techniques include Mask R-CNN and Cascade R-CNN [6]. • Region-Based Detectors: These techniques divide the image into non-overlapping regions and perform object detection within each area. Selective Search and EdgeBoxes are examples of region-based object detectors [14]. • Anchor-Based Detectors: Anchor-based object detectors, such as RetinaNet and YOLOv4, use pre-defined anchor boxes to predict object locations and sizes [15]. Overall, each type of object detection technique has advantages and disadvantages, and the choice of method depends on the specific application requirements and constraints. Considering the defense applications, a quick and real-time response is required to meet these requirements; only techniques with lower turnaround time can be used. Following are the techniques which yield the best results for real-time applications.

Deep-Learning-Based Object Recognition  115 • YOLO (You Only Look Once): YOLO is a popular onestage object detection algorithm that processes images as a single neural network and directly predicts object classes and bounding boxes. It is known for its speed and accuracy, making it popular in autonomous driving and surveillance applications. The algorithm simultaneously divides the given image into multiple boxes and chooses the boxes with the maximum probability of the object occurring in the box, as shown in Figure 6.2 [16]. • SSD (Single Shot Detector): SSD is another real-time onestage object detection algorithm that generates bounding boxes and class probabilities directly from a single feedforward pass of a deep neural network. It is known for its speed and high accuracy on small objects [18]. • RetinaNet: RetinaNet is an object detection algorithm that uses a focal loss function to address the problem of class imbalance in object detection datasets. It is known for its high accuracy on small and medium-sized objects, making it useful in satellite imagery analysis and medical imaging [19]. • MobileNet: MobileNet is a family of lightweight neural networks designed to be fast and efficient on mobile and embedded devices. They can be used for real-time object detection applications requiring low latency and power consumption [20]. • EfficientDet: EfficientDet is a recent family of object detection models that achieve state-of-the-art accuracy and efficiency by optimizing the architecture and training process.

cor dog

Figure 6.2  Box proposal generation and final box outputs of yolo-based object detection [17].

116  Advances in Aerial Sensing and Imaging They are designed for real-time applications that require high accuracy and fast processing [21].

6.2.4 Object Tracking Techniques Object tracking is tracking an object in a video sequence over time. It involves determining an object’s position, velocity, and trajectory as it moves within the scene. Object tracking is crucial in many computer vision applications, including video surveillance, robotics, and autonomous driving [22]. Object tracking techniques can be classified into two main categories: generative models and discriminative models. • Generative models: Generative models in object tracking attempt to explicitly model the object and its surrounding background, and they predict the following location of the thing based on the model. These models typically use a mathematical representation of the object and its dynamics, along with a probabilistic model of the background, to predict the following location of the object [23]. • Kalman Filter: One example of a generative model used in object tracking is the Kalman filter Figure 6.3. The Kalman filter is a statistical model that predicts the state of a dynamic system based on a set of noisy measurements. In object Measured position Mj Detection

Assignment Oi Mj

Measured position of object Oi

Predicted position of object Oi

Kalman-filter prediction

Kalman-filter correction Corrected position of object Oi

Figure 6.3  Steps involved in object tracking using Kalman filter here Oi is the current object, and Mj is the measured position of the object [24].

Deep-Learning-Based Object Recognition  117 tracking, the Kalman filter models the object’s condition, including its position, velocity, and acceleration, as it moves within the scene. The filter updates the state estimates based on new measurements and estimates the uncertainty in the state estimates using a covariance matrix [25]. • Particle Filter: The particle filter is another example of a generative model used in object tracking. The particle filter is a Bayesian filtering algorithm representing the probability distribution of the object’s state using a set of weighted particles. Each particle represents a possible location and state of the object, and the weights represent the probability of that particle being the actual state of the object. The filter updates the particle weights based on new measurements and resamples the particles to focus on the most probable states [24] (Figure 6.4). Generative models in object tracking are particularly useful for handling complex object dynamics and noisy measurements. However, they may require more computational resources than discriminative models, and they may need help to handle difficult background conditions. By combining generative and discriminative models, researchers have developed a wide range of practical object-tracking algorithms that can perform well in real-world scenarios [26]. ◦◦ Discriminative Models: Object tracking models that are discriminative learn to differentiate between the object and the background using a collection of characteristics. They use these features to estimate the future location of the object. These models typically use a classifier, such as a support vector machine (SVM) or a deep neural network, to classify the object and its surroundings as either object or background [27]. ◦◦ Mean Shift Algorithm: One example of a discriminative model used in object tracking is the MeanShift algorithm. The MeanShift algorithm iteratively shifts a kernel towards the region with the highest density of feature points. In object tracking, the algorithm uses a set of feature points around the object to define the kernel. It iteratively shifts the kernel towards the location with the highest density of feature points. The center of the kernel is then used as the predicted location of the object [28].

118  Advances in Aerial Sensing and Imaging Particle initialization

State estimation

Extracting target convolution features

Extracting target color features

Calculation of sparse distribution similarity

Calculation of color distribution similarity

Data fusion

Weights updating

Output estimate

Resampling

Figure 6.4  Showing particle filter-based tracking flowchart.

Deep-Learning-Based Object Recognition  119 ◦◦ Correlation Filter Algorithm: Another example of a discriminative model used in object tracking is the correlation filter. The correlation filter uses a set of filters to compute the similarity between the object and the surrounding background. In object tracking, the filter is trained on a collection of positive and negative samples, where positive samples represent the object and negative samples represent the background. The filter is then used to compute the correlation between the object and the surrounding background at each location, and the location with the highest correlation is used as the predicted location of the object (Figure 6.5). Discriminative models in object tracking are particularly useful for handling complex background conditions and can be more computationally efficient than generative models. However, they may need help to take complex object dynamics and generative models. By combining generative and discriminative models, researchers have developed a wide range of practical object-tracking algorithms that can perform well in real-world scenarios. Using the techniques mentioned above, the defense forces can develop various methods for Monitoring and surveillance. This chapter discusses real-time object detection focusing on small object detection and tracking with the latest YOLO model, i.e. YOLOv7 [29]. ◦◦ The Architecture of the YOLOv7: With the emergence of complex deep neural networks and increased computational demands, researchers have been looking forward to networks converging faster and consuming the least resources by keeping up with the required performance. Researchers developed efficient layer aggregation networks (ELAN) to overcome the abovementioned challenges. Figure 6.6 describes the strategy which relies on the motivation of designing an efficient network. The solution proposed is The original template is not updated

Calculate estimated position using motion characteristics

confidence< threshold?

Current frame position Confidence weighting

Original tracking result location

Figure 6.5  Correlation filter algorithm for object tracking [29].

Next frame position

120  Advances in Aerial Sensing and Imaging to control the shortest longest gradient path resulting in efficient learning and convergence of a deeper network. Leveraging the same authors of YOLOv7 came up with Extended-ELAN (E-ELAN), as shown in Figure 6.6. The E-ELAN method uses an expand, shuffle, and merge ­cardinalitybased strategy to enhance learning ability continuously without destroying the original gradient path. The E-ELAN only changes computational block architecture and keeps the transition layer architecture unchanged. The E-ELAN-based architecture enables the model to learn a more diverse feature set using different computational block groups. ◦◦ Model Scaling used in YOLOv7: The model scaling targets to adjust the model’s attributes for producing models of the variable scale of meeting the requirements of different inference speeds. The scaling up of the concatenation-based model in depth or width leads to a change in the ratio of input and output channels of a transition layer, so the scaling factors can’t be analyzed separately. The proposed method in [30] can maintain the properties of the model from the initial design and maintain the optimal structure, as shown in Figure 6.7. Figure 6.7 increases the depth of a computational block and increases the output width, as seen from (a) to (b). As a result, the input width of the following transmission layer is elevated. Therefore, only the depth of the computational block needs to be scaled for the scaling of

1x1, 4c, c 3x3, c, c 3x3, c, c

3x3, c, c 3x3, c, c 1x1, 3c, c

3x3, c, c

cross stage connection

3x3, c, c

cross stage connection

3x3, c, c

2c partial 2c 3x3, c, c

3x3, c, c

3x3, c, c

Stack in computational block

c partial c

cross stage connection

c partial c 3x3, c, c

Stack in computational block

c

Expand cardinality 3x3, 2c, 2c, 2 3x3, 2c, 2c, 2 3x3, 2c, 2c, 2 3x3, 2c, 2c, 2 Shuffle cardinality

4c

4c

1x1, 4c, c

1x1, 4c, c

1x1, 4c, c

1x1, 8c, 2c, 2 Merge cardinality c

c

c

c

c

3x3, c, c

3x3, c, c

3x3, c, c

Figure 6.6  The architecture of efficient layer aggregation networks [30].

Deep-Learning-Based Object Recognition  121 c

c

Conv

Conv scaling up depth

Conv

Conv c

c

c

(a) Concatenation-based model

Conv

Computational block

Transition

Scaling up depth Scaling up width

Conv Conv

Transition Scaling up width

Partial c c c c Cross Stage Merge Scaling up width width also be changed (b) Scaled-up concatenation-based model (c) Compound scaling up depth and width for concatenation-based model

Figure 6.7  Scaling concatenation-based models [30].

concatenation-based models. In contrast, the remaining transmission layer should be scaled correspondingly in width, as proposed in (c).

6.3 Experimental Methods This section discusses the dataset used for training the model. Furthermore, the deep learning setup is explained, including the training model and the hardware system used for training.

6.3.1 Experimental Setup and Dataset The visdrone2019 data set is used to conduct experiments and validate the usage of the model in small object detection. The models were trained to get transfer learning benefits achieving the best results possible in the shortest training time.

6.3.2 DataSetVISdrone 2019 The VisDrone 2019 dataset was collected by the AISKYEYE team at the Lab of Machine Learning and Data Mining at Tianjin University in China. It comprises 288 video clips containing 261,908 frames and 10,209 static images captured by various drone-mounted cameras. The dataset covers a wide range of aspects, including location (14 different cities in China separated by thousands of kilometers), environment (urban and rural), objects (pedestrians, vehicles, bicycles, etc.), and density (sparse and crowded scenes). Additionally, the dataset was collected using various drone models in different scenarios and under varied weather and lighting conditions. The frames are manually annotated with over 2.6 million bounding boxes of targets of interest, such as pedestrians, cars, bicycles, and tricycles. Furthermore, the dataset provides information on attributes such as scene visibility, object class, and occlusion to facilitate better data utilization. Figure 6.8 shows a few snapshots from the data set [31].

122  Advances in Aerial Sensing and Imaging

Figure 6.8  Snapshots from the Visdrone 2019 dataset.

6.3.3 Experimental Setup The model used was based on YOLOv7 with stack scaling done on the neck, and the model is called YOLOv7-x with eight workers and a batch size of 8, an input of width of 640 and a height of 640 was used for training [32]. A single GPU was used for training with 16 GB of GPU memory. The experiment was run in two phases with 100 epochs each. The model uses the SiLU activation function. The images are scaled to 640 * 640 size for uniform input, and the model is evaluated based on Mean average precision.

6.4 Results and Outcomes The study presents results in two phases. The first phase compares the model with the state-of-the-art models for real-time object detection.

Deep-Learning-Based Object Recognition  123 In the second phase, results for small object detection in images captured via drones using YOLOv7 are reported.

6.4.1 Comparison Results A comparison between our proposed approach and the state-of-the-art object detection models created for both general and Mobile GPUs. The current method achieves the best speed-accuracy trade-off overall. In particular, a comparison between YOLOv7-tiny-SiLU and YOLOv5-N (r6.1) reveals a 127-fps speed improvement and a 10.7% boost in accuracy on AP. Furthermore, at a frame rate of 161 fps, YOLOv7 attains an AP of 51.4%, while PPYOLOE-L, with the same AP, achieves only a 78-fps frame rate. Additionally, YOLOv7 uses 41% fewer parameters than PPYOLOE-L. When comparing YOLOv7-X with YOLOv5-L (r6.1), YOLOv7-X improves AP by 3.9% with a faster inference speed of 114 fps compared to 99 fps. YOLOv7-X is also 31 fps faster than YOLOv5-X (r6.1) of similar scale while reducing 22% of parameters and 8% of computation but improving AP by 2.2%. When comparing YOLOv7 and YOLOR at an input resolution of 1280, YOLOv7-W6 is faster by eight fps compared to YOLOR-P6, while achieving a 1% gain in AP. On the other hand, YOLOv7-E6 exhibits a 0.9% increase in AP, 45% fewer parameters, 63% lower computation, and 47% faster inference speed than YOLOv5-X6 (r6.1). YOLOv7-D6 and YOLOR-E6 have comparable inference speeds, but YOLOv7-D6 exhibits an improvement of 0.8% in AP. Similarly, YOLOv7-E6E and YOLOR-D6 have comparable inference speeds, but YOLOv7-E6E shows a 0.3% increase in AP.

6.4.2 Training Results During the study, a small object detection-based model with YOLOv7 was trained for 100 epochs, and results were analyzed as shown in Figure 6.9. The figure shows various parameters like precision-recall mean average precision. Transfer learning on YOLOv7 model pre-trained on COCO dataset.

6.5 Conclusion In conclusion, this chapter has provided an insightful introduction to applying deep learning for object recognition and tracking in defense settings. Through thoroughly exploring the theoretical foundations and

124  Advances in Aerial Sensing and Imaging Box

Objectness

0.0700

0.065 0.060

0.0675

0.055

0.0650

results

Classification

0.50

0.020

0.55

0.45

0.50

0.40

0.45

0.35

0.40

0.30

0.0625

0.015

0.045

0.0600

0.010

0.040

0.0575 50 100 val Box

0.120

0.0550

0.28 0.26 0.24 0.22 0.20 0.18 0.16

0.115 0.110 0.105 0.100 0

50

100

0.35

0.005 0 50 100 val Objectness

Recall

0.60

0.050

0

Precision

0.025

0.042 0.040 0.038

0 50 100 val Classification

0.25 0

50 100 [email protected]

0.5

0.25

0.4

0.20

0.036 0.034

0.3

0.15

0.032

0.2

0.10

0.030 0

50

100

0.30

100 0 50 [email protected]:0.95

0.05 0

50

100

0

50

100

0

50

100

Figure 6.9  Graph showing various training parameters for YOLOv7 trained on visdrone 2019 data for 100 epochs.

practical implementations, the paper has highlighted the potential of deep understanding to enable more accurate and efficient defense applications. The article introduces small object identification by training the YOLOv7 model on visdrone2019.The paper has emphasized the importance of large datasets for deep learning-based object recognition and tracking and the need for sophisticated algorithms that can process this data effectively. Furthermore, the paper has highlighted the challenges that must be overcome to ensure the success of deep learning in defense applications. These include the need for high-quality data and models’ interpretability and realtime performance. The paper has also discussed the importance of hybrid approaches combining deep learning with other techniques to overcome these challenges. The potential benefits of deep understanding in defense applications are significant, ranging from improved situational awareness and threat detection to enhanced decision-making and operational efficiency. As such, deep learning-based object recognition and tracking will continue to be an active and vital area of research and development in the defense community. The paper has comprehensively introduced deep learning-based object recognition and tracking for defense applications. While there are still many technical and practical challenges to overcome, the potential benefits of deep learning in defense settings are undeniable. They will undoubtedly continue to drive future research and development in this area.

Deep-Learning-Based Object Recognition  125

6.6 Future Scope With the increase of drones and unmanned vehicles at the forefront of defense, it is imperative for object detection and tracking techniques to adapt to the hardware available onboard these devices. Additionally, there is a requirement to adjust the methods to 5G edge networks. Thus, future researchers must enable small object detection and tracking algorithms to run onboard mobile systems augmented by edge computing infrastructure.

References 1. Mandal, M., Kumar, L.K., Vipparthi, S.K., MOR-UAV: A benchmark dataset and baselines for moving object recognition in UAV videos, in: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2626–2635, 2020. 2. Arya Raj, A.K. and Radhakrishnan, B., A comparative study on target detection in military field using various digital image processing techniques. Int. J. Comput. Sci. Netw., 5, 181–185, 2016. 3. Fehlmann, S., Pontecorvo, C., Booth, D.M., Janney, P., Christie, R., Redding, N.J., Royce, M., Fiebig, M., Fusion of multiple sensor data to recognise moving objects in wide area motion imagery, in: 2014 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–8, 2014. 4. Intille, S.S., Visual recognition of multi-agent action, Thesis (Ph.D.)-Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 1999. https://api.semanticscholar.org/ CorpusID:1857353 5. Hidayat, F. and others, Intelligent video analytic for suspicious object detection: A systematic review, in: 2020 International Conference on ICT for Smart Society (ICISS), pp. 1–8, 2020. 6. Chen, Y., Zhao, D., Lv, L., Zhang, Q., Multi-task learning for dangerous object detection in autonomous driving. Inf. Sci., 432, 559–571, 2018. 7. Terracciano, D.S., Bazzarello, L., Caiti, A., Costanzi, R., Manzari, V., Marine robots for underwater surveillance. Curr. Rob. Rep., 1, 159–167, 2020. 8. Bauk, S., Kapidani, N., Lukšic, Ž., Rodrigues, F., Sousa, L., Autonomous marine vehicles in sea surveillance as one of the COMPASS2020 project concerns. J. Phys.: Conf. Ser., 1357, 012045, 2019. 9. Janakiramaiah, B., Gadupudi, K., Karuna, A., Prasad, L., Krishna, M., Military object detection in defense using multi-level capsule networks. Soft Comput., 27, 1045–59, 2021.

126  Advances in Aerial Sensing and Imaging 10. Zou, X., A review of object detection techniques, in: 2019 International Conference on Smart Grid and Electrical Automation (ICSGEA), pp. 251–254, 2019. 11. Huang, T.-Y., Lee, M.-C., Yang, C.-H., Lee, T.-S., YOLO-ORE: A deep ­learning-aided object recognition approach for radar systems. IEEE Trans. Veh. Technol., 72, 1–16, 2022. 12. Du, L., Zhang, R., Wang, X., Overview of two-stage object detection algorithms. J. Phys.: Conf. Ser., 1544, 012033, 2020. 13. Tian, Z., Shen, C., Chen, H., He, T., FCOS: Fully convolutional one-stage object detection, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9627–9636, 2019. 14. Shrivastava, A., Gupta, A., Girshick, R., Training region-based object detectors with online hard example mining, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 761–769, 2016. 15. Liu, S., Zhou, H., Li, C., Wang, S., Analysis of anchor-based and anchor-free object detection methods based on deep learning, in: 2020 IEEE International Conference on Mechatronics and Automation (ICMA), pp. 1058–1065, 2020. 16. Du, J., Understanding of object detection based on CNN family and YOLO. J. Phys.: Conf. Ser., 1004, 012029, 2018. 17. Kumar, A., Zhang, Z.J., Lyu, H., Object detection in real time based on improved single shot multi-box detector algorithm. EURASIP J. Wireless Commun. Networking, 2020, 1–18, 2020. 18. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A.C., SSD: Single shot multibox detector, in: Computer Vision–ECCV 2016: 14th European Conference, Proceedings, Part I 14, Amsterdam, The Netherlands, October 11–14, 2016, pp. 21–37, 2016. 19. Li, Y. and Ren, F., Light-weight Retina Net for object detection on edge devices, in: 2020 IEEE 6th World Forum on Internet of Things (WF-IoT), 1–6, 2020. 20. Younis, A., Shixin, L., Jn, S., Hai, Z., Real-time object detection using pretrained deep learning models MobileNet-SSD, in: Proceedings of 2020 the 6th International Conference on Computing and Data Engineering, pp. 44–48, 2020. 21. Tan, M., Pang, R., Le, Q.V., Efficientdet: Scalable and efficient object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10781–10790, 2020. 22. Foresti, G.L., Object recognition and tracking for remote video surveillance. IEEE Trans. Circuits Syst. Video Technol., 9, 1045–1062, 1999. 23. Riahi, D. and Bilodeau, G.-A., Online multi-object tracking by detection based on generative appearance models. Comput. Vision Image Understanding, 152, 88–102, 2016. 24. Máttyus, G., Benedek, C., Szirányi, T., Multi target tracking on aerial videos. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.-ISPRS Arch., 38, 1–7, 2010.

Deep-Learning-Based Object Recognition  127 25. Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., Kim, T.-K., Multiple object tracking: A literature review. Artif. Intell., 293, 103448, 2021. 26. Dai, Y., Hu, Z., Zhang, S., Liu, L., A survey of detection-based video multi-­ object tracking. Displays, 75, 102317, 2022. 27. Wei, Y., Hua, Y., Xiang, W., Research on specific long-term single object tracking algorithms in the context of traffic. Proc. Comput. Sci., 214, 304– 311, 2022. 28. Choi, S., Lee, J., Lee, Y., Hauptmann, A., Robust long-term object tracking via improved discriminative model prediction, in: Computer Vision–ECCV 2020 Workshops: Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pp. 602–617, 2020. 29. Liu, S., Liu, D., Srivastava, G., Połap, D., Woźniak, M., Overview and methods of correlation filter algorithms in object tracking. Complex Intell. Syst., 7, 1895–1917, 2021. 30. Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M., YOLOv7: Trainable bag-of-­ freebies sets new state-of-the-art for real-time object detectors, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7464–7475, 2023. 31. Du, D., Zhu, P., Wen, L., Bian, X., Lin, H., Hu, Q., Peng, T., Zheng, J., Wang, X., Zhang, Y., others, VisDrone-DET2019: The vision meets drone object detection in image challenge results, in: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 213–226, 2019. 32. Moghimi, M.K. and Mohanna, F., Reliable object recognition using deep transfer learning for marine transportation systems with underwater surveillance. IEEE Trans. Intell. Transp. Syst., 24, 2515–2524, 2022.

7 A Robust Machine Learning Model for Forest Fire Detection Using Drone Images Chahil Choudhary*, Anurag and Pranjal Shukla Department of Computer Science and Engineering, Chandigarh University, Mohali Punjab, India

Abstract

Forest fires devastate the environment, wildlife, and human life. Early detection and quick response are critical to minimize the damage caused by these disasters. Real-time monitoring and data collecting through drones fitted with high-­ resolution cameras and other sensors, which can help with the early identification of forest fires. However, it is a complicated process that takes a lot of time and money to analyze the volume of data produced by these sensors. Machine learning algorithms can provide an effective solution to this problem. By using algorithms to analyze the data generated by drones, we can quickly identify patterns and anomalies that indicate the presence of a forest fire. In this work, we propose a robust machine-learning model for forest fire detection using drone image processing. The trained model employs Model (SMOreg) (V) to analyze images captured by drones and to detect signs of a forest fire. The model is trained on a large dataset of drone images, including both positive and negative examples of forest fires, to ensure its robustness and accuracy. We carried out several tests using actual drone photographs to assess the effectiveness of the suggested model. The results demonstrate that the proposed model achieves high accuracy rates, even when operating in challenging environments and under adverse weather conditions. The model achieved an accuracy rate of 92.72%. By using drones and machine learning, we can quickly and accurately identify the presence of forest fires, enabling responders to take immediate action to minimize their impact. The proposed model can be deployed in various scenarios, from small-scale wildfires to large-scale forest fires, making it a valuable tool for disaster management agencies and other stakeholders.

*Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (129–144) © 2024 Scrivener Publishing LLC

129

130  Advances in Aerial Sensing and Imaging Keywords:  Feature selection, ensemble, robust, drone technology, prediction, forest fire

7.1 Introduction Forest fires are a serious environmental danger that harms populations and ecosystems worldwide. Early detection and rapid response to forest fires are crucial for effective firefighting and minimizing damage [1, 2]. Drones have emerged as a valuable tool for detecting forest fires, as they can provide high-resolution images and video footage of forest areas that can be analyzed for fire signs [3–5]. Machine learning has shown great promise in improving forest fire detection using drone images in recent years. Machine learning algorithms can be trained on large datasets of labeled drone images to learn the patterns and characteristics of forest fires and then used to detect fires in new photos [6] automatically. ML drones refer to integrating machine learning algorithms with drones to enhance their capabilities and make them more efficient in various applications. With the advent of drone technology, capturing high-resolution images and videos from the sky has become possible, which can be analyzed using machine learning algorithms for different applications [7]. Drones are helpful in forest fire prediction and management because they provide early detection, access to remote areas, real-time surveillance, cost-effectiveness, and improved safety [8]. These benefits make drones an essential tool in the fight against forest fires, helping to protect the environment and communities from the devastating impact of wildfires [9]. Previous research has looked into using drones to predict and manage forest fires. These studies have examined the potential of drones to provide real-time monitoring, assess the extent of damage caused by wildfires, and locate survivors in affected areas. The integration of machine learning algorithms has also been explored to improve the accuracy of wildfire detection and mapping [10–13]. One of the key advantages of using drones in forest fire prediction and management is their ability to access remote and inaccessible areas, which can be difficult to survey using traditional methods. High-resolution pictures and data may also be obtained by drones using thermal imaging cameras and other sensors, which can be analyzed using machine learning techniques. This enables responders to assess the situation and plan their response efforts quickly and efficiently [14, 15]. Previous studies have demonstrated the potential of using drones for forest fire prediction and management. These investigations have also shown that more analysis and development are necessary to realize the promise of this

Forest Fire Detection Using Drone Images  131 technology [16] fully. As technology advances, we can expect to see more sophisticated and practical applications of drones in forest fire prediction and management, which will help minimize the impact of wildfires on our environment and communities [17]. Machine learning algorithms can provide an effective solution to this problem. By using algorithms to analyze the data generated by drones, we can quickly identify patterns and anomalies that indicate the presence of a forest fire [18]. This enables responders to respond to the situation promptly and efficiently, reducing the impact of the fire on the environment and communities [19]. This paper suggests a reliable machine-learning strategy for detecting forest fires using drone image processing in this environment. The model utilizes a convolutional Algorithm to analyze the images captured by drones and detect signs of a forest fire. The model is trained on a large dataset of drone images, including both positive and negative examples of forest fires, to ensure its robustness and accuracy [20]. The study conducted experiments using real-world drone images to check the model’s efficiency. The results demonstrate that the model achieves high accuracy rates, even when operating in challenging environments and under adverse weather conditions [21]. The suggested methodology has the potential to dramatically raise the efficacy and efficiency of managing forest fires [22]. By using drones and machine learning algorithms, we can quickly and accurately identify the presence of forest fires, enabling responders to take immediate action to minimize their impact. The proposed model can be deployed in various scenarios, from smallscale wildfires to large-scale forest fires, making it a valuable tool for disaster management agencies and other stakeholders [23, 24].

7.2 Literature Review This section focuses on the previous research done on this forest fire prediction. This section consists of all the information needed to check the last analysis. This section helps us to find the significant research gap. Liyang Yu et al. [1] suggested a wireless sensor network to replace satellite-based monitoring for real-time forest fire detection. The Algorithm used for detecting the forest fire is a neural network. Results are evaluated based on simulations. Vipin V et al. [2] employed an innovative image analysis method to find forest fires. The effectiveness of the recommended technique is assessed using two sets of pictures, one of which has regions that mimic fire and the other of which has fire. Standard calculation techniques are employed to calculate the performance. Panagiotis Barmpoutis et al. [3]

132  Advances in Aerial Sensing and Imaging reviewed a study on forest fire detection methods. Three different types of systems—terrestrial, aerial, and spaceborne—are characterized as attempts to detect fire occurrences accurately in challenging circumstances. Renjie XU et al. [4] proposed an ensemble model to detect forest fires under different conditions. The results improved the detection from 2.5% to 10.9% and reduced the false detection by 51.3% without extra latency. YE Aslan et al. [5], a thorough structure for a wireless sensor network, has been presented to find forest fires. We offer suggestions for our framework’s sensor deployment strategy, wireless sensor network design, and clustering and communication protocols. The framework aims to detect a fire threat as soon as possible while considering the sensor nodes’ energy consumption and any external circumstances affecting the network’s required degree of activity. Varanasi LVSKB Kashyap et al. [6] proposed an early forest fire prediction system model using techniques like Autonomous Drone Routing, Drone Moment to the Target, Technical Information of UAV, etc., to predict the outcomes. Zhihao Guan ORCID et al. [7], in their paper “Forest Fire Segmentation from Aerial Imagery Data Using an Improved Instance Segmentation Model”, a model that was suggested in which they used techniques such as Satellite Remote Sensing, UAV, MaskSU R-CNN, etc. to predict the outputs. Dena Mahmudnia ORCID et al. [8] Using tools including geographic information systems (GIS), uncrewed aerial vehicles (UAVs), the internet of things (IoT), and cloud or edge computing, they suggested a study on “Drones and Blockchain Integration to Manage Forest Fires in Remote Regions.” Lin Zhang et al. [9] proposed a model called “A Transfer Learning-Based Forest Fire Recognition Method Using UAV Images”, in which they used techniques such as the Unmanned Aerial Vehicle (UAV), FT-ResNet50 model, Adam and Mish functions, ResNet network, etc. to predict the outcomes. Francesco de Vivo et al. [10] used uncrewed aircraft vehicle (UAV), infrared (IR) cameras, Line Camera (LC) configuration, mono-dimensional noise-resistant Algorithm for edge detection, etc. were used to detect the results in the research proposal titled “Infra-red line camera data-driven edge detector in UAV forest fire monitoring.” Zhentian Jiao et al. [11] presents a forest fire detection system that processes UAVbased aerial photos using YOLOv3 and highlights the growing usage of UAVs in forest fire monitoring and detection. The Algorithm met the requirements for real-time forest fire detection applications with an 83% identification rate and a frame rate of more than 3.2 fps. Fouda et al. [12] discuss the challenge of efficient analysis of drone-acquired data for early detection of the forest fire. The authors provide an adaptable, lightweight hierarchical AI system that alternates between models based on machine learning and deep learning and optimizes the confidence score threshold

Forest Fire Detection Using Drone Images  133 through multi-objective optimization using TOPSIS. Empirical results demonstrate high detection accuracy with a reduced computational burden. Abdelmalek Bouguettaya et al. discuss [13] the importance of using remote sensing technologies, particularly UAV-based systems, for early detection of wildfires in wild lands and forests. The traditional ground crew inspection methods have limitations and can put firefighters at risk. To minimize the loss of human lives and forest resources, the emphasis is on employing deep learning-based computer vision algorithms to identify wildfires in their early phases. In recent years, there has been a lot of interest in using UAV-based visual data to detect wildfires [14] automatically. This literature review focuses on developing ICT-based disaster management techniques to minimize economic and social losses caused by natural disasters. In addition to a deep learning-based forest fire monitoring system that employs photos from uncrewed aerial aircraft with an optical sensor, the article suggests a smart-eye platform for catastrophe detection and response. The proposed technique has been trained on past forest fire image sets to automatically detect whether a new input image contains a forest fire. The simulation results verify the accuracy of the proposed scheme, and its application to decision support systems for disaster management is expected to reduce disaster-related losses and monitoring costs. Xiwen Chen et al. [15] the limitations of current forest monitoring technologies in detecting and assessing wild land fires, particularly during their early stages. The article offers a multi-modal dataset of dual-feed side-by-side films with RGB and thermal pictures of a controlled fire in a pine forest in Northern Arizona. It suggests the usage of drone devices for quick mapping and real-time fire monitoring. Additionally, a deep learning-based algorithm for detecting fire and smoke pixels with greater accuracy than typical single-channel video feeds is presented in the article. The dataset gives the primary aerial picture context and may be utilized to create new data-driven methods for modeling, segmenting, and detecting fires.

7.3 Proposed Methodology The proposed model is divided into five phases, as shown in Figure 7.1. The initial phase involves recognizing the problem statements and data sets, which are tackled by obtaining data from the UCI ML repository. This phase includes two crucial procedures: gathering data and refining it. Data is acquired from an online source and refined by eliminating unnecessary data and preparing it for further analysis. The second phase, the pre-­processing data phase, encompasses feature engineering, which generates new features

134  Advances in Aerial Sensing and Imaging Phase 1

identification of problem statement and datasets

.Data collection

.Data cleaning

Phase 2

Data pre-processing

Feature Selection

Phase 3 Rankers

correlation

Prediction

Train the data by the traditional machine learning algorithm

Phase 4

Validation & Evaluation

Apply the robust model and evaluate the K-fold validation. compare the traditional model with the proposed robust model

Figure 7.1  Flow chart of proposed work.

from existing data. This phase also includes data reduction, which decreases the number of elements in the data set utilizing techniques such as Principal Component Analysis (PCA). During this phase, significant features such as rankers and correlations are identified. The grade of validation and evaluation is critical to determine the model’s efficacy in accurately predicting outcomes for the novel, unseen data. It is essential to ensure that the model is balanced with the training data, which would result in poor performance on new data when applied to the training data. The model’s efficacy is measured

Forest Fire Detection Using Drone Images  135

Figure 7.2  Proposed drone image.

using various performance criteria, depending on the specific task. For classification tasks, commonly used metrics include accuracy, Correlation coefficient, and MEAN ABSOLUTE ERROR. The confusion matrix, ROC curve, and PR curve are valuable visualization tools that provide insights into the model’s performance, strengths, and weaknesses. During the validation and evaluation phase, the model’s effectiveness is compared to the traditional model to determine its robustness. If the proposed model exhibits better accuracy and generalizability than the conventional one, it can be considered a viable substitute for the traditional one. However, further enhancements and refinements may be required if the proposed model’s performance is less than the conventional model. The validation and evaluation phase is essential in verifying that the model can make precise predictions on new data and be used confidently for the intended purpose. In the last step, we are done with the prediction stage in which a drone capture of the forest, and then the model converts it into text and compares it with the data. It will predict accordingly if it has abnormalities, as shown in Figure 7.2.

7.4 Result and Discussion In this section, a comparison is made between the performance of a conventional machine-learning approach as shown in Table 7.1 and a robust machine-learning algorithm as shown in Table 7.2. The following ­cutting-edge metrics are used to evaluate the machine learning model: True Class (Horizontal) and Predicted Class (Vertical), as shown in Figure 7.3.

136  Advances in Aerial Sensing and Imaging • Correlation Coefficient: A statistical metric, the correlation coefficient, assesses how linearly connected two variables are. It ranges from -1 to 1, with -1 denoting a perfect inverse relationship (perfect negative Correlation), 0 indicating no correlation, and 1 representing a perfect positive correlation. It is symbolized by the letter “r.” (direct relationship). The formula for calculating the correlation coefficient is: Table 7.1  Traditional machine learning model. Evaluating machine learning model using different parameters Performance metrics Classifiers (Traditional)

Correlation Coef MAE

RMSE

RAE

Linear regression

0.0076

20.085

63.842

108.035 100.228 79.92

Multilayer Perception

0.0075

30.748

98.73

208.414 154.99

SMOreg

0.0026

11.882

64.585

69.32

Kstar

0.0039

20.665

97.573

127.291 153.818 79.34

LWL

0.0029

17.543

64.191

100.833 100.744 82.46

Bagging

0.0051

16.472

63.751

99.357

Random Forest

0.0068

21.638

67.006

116.387 105.195 78.37

Random Tree

0.0013

20.321

89.855

120.059 141.065 79.68

Model (SMOreg) (S)

0.054

8.992

60.876

103.701 99.029

Model 0.051 (SMOreg) (V)

8.493

62.989

101.701 100.008 91.5

RRSE

Accuracy

69.26

101.394 88.12

100.085 83.53

91.01

Forest Fire Detection Using Drone Images  137 Table 7.2  Robust machine learning model. Evaluating machine learning model using different parameters Performance metrics CLASSIFIERS (robust)

Correlation Coefficient MAE

Linear regression

0.182

16.641 27.838 109.277 106.285 83.36

Multilayer Perception

0.033

29.41

76.251 256.508 292.546 70.59

SMOreg

0.008

8.298

27.029 61.876

103.701 90.79

Kstar

0.083

14.657 30.129 90.106

115.593 85.35

LWL

0.025

11.314 27.53

88.666

105.624 88.69

Bagging

0.059

15.662 27.018 97.021

103.658 84.34

Random Forest

0.086

16.779 34.54

Random Tree

0.013

20.037 90.353 153.089 346.649 79.97

Model (SMOreg) (S)

0.007

7.592

26.064 79.143

100.001 92.41

Model 0.005 (SMOreg) (V)

7.289

25.029 61.876

99.701

RMSE RAE

TP FP

FN TN Figure 7.3  Confusion matrix.

RRSE

Accuracy

116.328 132.391 83.23

92.72

138  Advances in Aerial Sensing and Imaging

Cr =

(

)(

∑ ai − a bi − b

(

)

) ( 2

∑ ai − a ∑ bi − b

(7.1)

)

2



Cr = Correlation Coefficient ai = Sample a-variable values bi = Sample of b-variable values a = Mean of a-values variable’s b = MeanMean of b-values variable’s The degree and direction of the association between two variables are determined using the correlation coefficient. It is frequently used in many disciplines, including economics, psychology, the social sciences, and finance. Figure 7.4 is the graph for the correlation coefficient. • Mean Absolute Error A typical metric for assessing a prediction model’s performance is a mean absolute error (MAE). It calculates the distinct fundamental difference between a set of predictions’ expected and actual values. The following is how the MAE is determined: ∧

MAE=



1 N ∑i =1 bi − b N

(7.2)

Correlation Coefficient 0.182

V) Or

(S M el

od M

eg

)( eg

(S M el

od

0.005

)(

S)

e

0.007

Or

om M

ul M

Figure 7.4  Correlation coefficient of proposed work.

Ra nd

Fo

re

Tr e

st

g

L

in om

Ba gg

LW

r ta

0.013

til

Ks

eg Or

0.059 0.025

0.008

SM

ro n pt ce er

ay er P

rr

eg re

ss io n

0.033

ea Lin

0.086

0.083

Ra nd

0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0

Forest Fire Detection Using Drone Images  139 MAE provides a general indication of the average accuracy of the model’s predictions. A lower MAE suggests a more accurate model, whereas a larger one suggests a less accurate one. When huge errors are highly problematic, or measuring errors on the same scale as the original data is crucial, the MAE is a helpful metric. It could be used, for instance, to assess how well a weather forecasting model or stock price prediction model performs. MAE has the drawback of treating all errors equally, regardless of their direction. In other circumstances, considering the approach of the inaccuracies may be more crucial. An example is financial forecasting, where underestimating or overestimating a value may have distinct ramifications. Other metrics, such as Mean Signed Error (MSE) or Mean Directional Accuracy (MDA), may be more appropriate in such circumstances. Figure 7.5 is the graph for the mean absolute error. • Root Mean Square Error A typical statistic for assessing a predictive model’s performance is root mean square error (RMSE). It gauges the typical size of the discrepancy between a set of predictions’ anticipated values and actual values. Calculating RMSE is done as follows:

MAE 29.41

30 25

20.037 16.641

20

15.662 16.779

14.657

15

11.314

8.298

10

7.289

5

Figure 7.5  MAE of proposed work.

V)

ee

eg

)(

Tr

st

Or

om M

od

el

(S

M

nd

Fo re

Ra

L

g in

om

Ra

nd

Ba gg

LW

ta r Ks

eg Or SM

ye til a

M

ul

Lin

ea r

rP

re

er

gr

ce

es

pt

ro

sio

n

n

0

140  Advances in Aerial Sensing and Imaging

= RMSE

1 N

= MAE



∑( y − y ) N

2

2

(7.3)

i

i =1



MAE = Mean Absolute Error RMSE = Root Mean Square Error The RMSE penalizes more significant errors severely than the mean absolute error (MAE), making it a more sensitive metric. A lower RMSE denotes a more accurate model, whereas a greater RMSE denotes a less accurate model. When the prediction errors are anticipated to be regularly distributed, RMSE is frequently utilized. It is commonly used, for instance, to gauge how accurate regression models are. When the size of the errors is crucial, as in engineering or financial applications, it is also a relevant metric. The sensitivity of RMSE to outliers, which might have a disproportionate impact on the result, is one of its limitations. The fact that RMSE is calculated on a different scale than the actual data makes it less obvious sometimes. To provide a complete picture of a model’s performance, it is frequently offered with other metrics like MAE. Nonetheless, it is still a widely used statistic in many domains. Figure 7.6 is the graph for the root mean square error. • Accuracy Moreover, measures like Mean Absolute Error (MAE) can be used to check the correctness of a machine learning model. Considering their squared differences, RMSE calculates the RMSE 90.353

100 90 80 70 60 50 40 30 20 10 0

76.251

34.54

) (V

SM

Or

om nd

Fo

M

od e

l(

Ra

om nd Ra

eg )

t

Tr ee

25.029

re s

ng

L

27.018

gg i

LW

ta r Ks

Or eg

ay til ul M

Figure 7.6  RMSE of proposed work.

27.53

Ba

rc e er

Pe

SM

pt

ro n

sio n re s rr eg ea Lin

27.029 30.129

27.838

Forest Fire Detection Using Drone Images  141 average difference between a target variable’s actual and anticipated values. It is widely used in regression projects and is derived as the square root of the mean squared differences between the actual and anticipated values. Instead of accounting for the direction of discrepancies, MAE calculates the average difference between a target variable’s actual and forecast values. The mean of the absolute differences between the actual and predicted values is obtained when doing regression exercises. Better model accuracy is shown by lower RMSE and MAE values. The precise work at hand as well as the user’s preferences, will determine the metric to employ, though. For instance, MAE may be selected when the focus is on the size of errors rather than their direction, but RMSE may be favored when more significant errors have a greater influence on performance. It is crucial to remember that RMSE and MAE, like accuracy, are not constant and can change according to the caliber and volume of the data, the features employed, and the selection of algorithms and settings. As a result, it’s crucial to regularly assess and raise the model’s accuracy using suitable measures. The graph for the accuracy is shown in Figure 7.7. The model outperforms and has a good accuracy of 92.72%. It means the model performs well and can be used for forest fire detection in real

Accuracy

V) )( eg

(S

M

nd o

Or

m

Tr ee

t es

92.72

79.97

el

Ra

nd o

m

Fo r

gg

in

g

L Ba

83.23

od

84.34

M

M

Figure 7.7  Accuracy of proposed work.

LW

r ta Ks

Or

tro ep

Pe

rc

eg

n

n sio re s

er ay

til

ea rr eg

88.69

70.59

ul

Lin

85.35

Ra

90.79

83.36

SM

100 90 80 70 60 50 40 30 20 10 0

142  Advances in Aerial Sensing and Imaging life. It can be beneficial for the future as well as for the present since it will help in detecting the places where fire has started due to any climatic changes, human fault, etc., and ordinary people cannot see these places, this model, using drones will be capable of finding such sites. We can cure the fire as soon as it signals the team.

7.5 Conclusion and Future Scope A viable strategy for preventing and controlling forest fires is creating a reliable machine-learning model for forest fire identification using drone photos. The firefighting business has transformed because of the usage of drones in monitoring and identifying forest fires. The machine learning model can be trained to identify specific features in the images captured by drones, such as the color of the smoke and the heat signatures, which can indicate the presence of a forest fire. There are several future directions for research in this area. One option is investigating deep learning methods to improve the model’s precision and speed. Incorporating extra sensor data, such as weather data, to enhance model predictions is another topic for future research. Moreover, integrating real-time data processing techniques can improve the model’s effectiveness. This can enable early detection of forest fires, which can help to minimize the damage caused by these disasters.

References 1. Yu, L., Wang, N., Meng, X., Real-time forest fire detection with wireless sensor networks, in: Proceedings. 2005 International Conference on Wireless Communications, Networking and Mobile Computing, vol. 2, IEEE, pp. 1214– 1217, 2005. 2. Vipin, V., Image processing-based forest fire detection. Int. J. Emerg. Technol. Adv. Eng., 2, 2, 87–95, 2012. 3. Barmpoutis, P., Papaioannou, P., Dimitropoulos, K., Grammalidis, N., A review on early forest fire detection systems using optical remote sensing. Sensors, 20, 22, 6442, 2020. 4. Xu, R., Lin, H., Lu, K., Cao, L., Liu, Y., A forest fire detection system based on ensemble learning. Forests, 12, 2, 217, 2021. 5. Aslan, Y.E., Korpeoglu, I., Ulusoy, Ö., A framework for using wireless sensor networks in forest fire detection and monitoring. Comput. Environ. Urban Syst., 36, 6, 614–625, 2012.

.

Forest Fire Detection Using Drone Images  143 6. Kashyap, VLVSKB, Sumathi, D., Alluri, K., Reddy Ch, P., Thilakarathne, N., Muhammad Shafi, R., Early detection of forest fire using mixed learning techniques and UAV. Comput. Intell. Neurosci., 2022, 2022. 7. Guan, Z., Miao, X., Mu, Y., Sun, Q., Ye, Q., Gao, D., Forest fire segmentation from aerial imagery data using an improved instance segmentation model. Remote Sens., 14, 13, 3159, 2022. 8. Mahmudnia, D., Arashpour, M., Bai, Y., Feng, H., Drones and blockchain integration to manage forest fires in remote regions. Drones, 6, 11, 331, 2022. 9. Zhang, L., Wang, M., Fu, Y., Ding, Y., A forest fire recognition method using UAV images based on transfer learning. Forests, 13, 7, 975, 2022. 10. De Vivo, F., Battipede, M., Johnson, E., Infra-red line camera data-driven edge detector in UAV forest fire monitoring. Aerosp. Sci. Technol., 111, 106574, 2021. 11. Jiao, Z. et al., A deep learning based forest fire detection approach using UAV and YOLOv3, 1st International Conference on Industrial Artificial Intelligence (IAI), Shenyang, pp. 1–5, IEEE, China, 2022. 12. Fouda, M.M. et al., A lightweight hierarchical AI model for UAV-enabled edge computing with forest-fire detection use-case. IEEE, 1–6, 2022, 6 December. 13. Bouguettaya, A. et al., A review on early wildfire detection from unmanned aerial vehicles using deep learning-based computer vision algorithms. Signal Processing, 1–14, 2021, 31 August, Elsevier. 14. Seonghyumkim, Forest fire monitoring system based on aerial images, IEEE, pp. 1–6, 2016. 15. Chen, X. et al., Wildland fire detection and monitoring using a drone-collected RGB/IR image dataset. IEEE, 1–6, 17 November 2022. 16. Bajaj, R. et al., A robust machine learning for EEG prediction, pp. 1–5, 2023. 17. Pawar, Singh, J., Bajaj, R., Singh, G., Rana, S., Optimized ensembled machine learning model for IRIS plant classification. 2022 6th International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India, pp. 1442–1446, 2022. 18. Sandeep, Prasad, K.M.V.V., Srilekha, A., Suman, T., Pranav Rao, B., Naga Vamshi Krishna, J., Leaf disease detection and classification based on machine learning, in: International Conference on Smart Technologies in Computing, Electrical and Electronics (ICSTCEE), pp. 361–365, 2020. 19. Swathi, G., Robust deep learning technique: U-net architecture for pupil segmentation, in: 11th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), pp. 0609–0613, 2020. 20. Shinde, S. and Johri, P., A review: Eye tracking interface with embedded system & IOT, in: International Conference on Computing, Power and Communication Technologies (GUCON), pp. 791–795, 2018. 21. Kumar, S., Rani, S., Jain, A., Verma, C., Raboaca, M.S., Illés, Z., Neagu, B.C., Face Spoofing, age, gender and facial expression recognition using advance

144  Advances in Aerial Sensing and Imaging neural network architecture-based biometric system. Sensors, 22, 14, 5160, 2022. 22. Sandeep, Singh, S., Kumar, J., Face spoofing detection using improved SegNet architecture with a blur estimation technique. Int. J. Biom., 13, 2–3, 131–149, 2021. 23. Rajan, E.G. and Rani, S., A study on vehicle detection through aerial images: Various challenges, issues and applications, in: International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp. 504–509, 2021. 24. Kumar, Rajan, E.G., Rani, S., Enhancement of satellite and underwater image utilizing luminance model by color correction method, in: Cognitive Behavior and Human Computer Interaction Based on Machine Learning  Algorithm, pp. 361–379, 2021.

8 Semantic Segmentation of Aerial Images Using Pixel Wise Segmentation Swathi Gowroju1, Shilpa Choudhary2*, Sandhya Raajaani3 and Regula Srilakshmi4 Department of Computer Science, Sreyas Institute of Engineering and Technology, Hyderabad, Telangana, India 2 Department of Computer Science (AI&ML), Neil Gogte Institute of Technology, Hyderabad, Telangana, India 3 Department of Mathematics, V R Siddhartha Engineering College, Vijayawada, India 4 Department of Computer Science, Neil Gogte Institute of Technology, Hyderabad, Telangana, India

1

Abstract

The current research describes a deep learning method for semantic segmentation of incredibly high-quality (aerial) photographs. By facilitating end-to-end learning from unedited images, deep neural architectures have the potential to replace heuristic feature design. Deep convolutional neural networks (CNNs) have recently become the methodology preference for different interpretations of image tasks, including visual categorization and object detection. This theory has had a rebirth during the past ten years. Due to a U-Net model’s promising performance on tiny and unbalanced datasets, we mainly used it for this chapter. Multiscale feature maps, one of the key elements for producing fine-grained segmentation maps, particularly for high-resolution pictures, are only partially utilized by the raw U-Net architecture. We develop a multiscale skip-connected Hybrid U-Net and a multiscale feature fusion technique for multiscale feature fusion to segment high-resolution satellite images. In our tests, U-Net and its variations showed good segmentation results to identify different types of building devastation brought on by conflict. We investigate such a network’s conceptual constraints and complexities and demonstrate that an ensemble of many structures using only raw data as input executes admirably on complicated datasets. *Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (145–164) © 2024 Scrivener Publishing LLC

145

146  Advances in Aerial Sensing and Imaging Keywords:  Satellite images, areal images, semantic segmentation, UNet using morphology, pixel-wise segmentation

8.1 Introduction Despite many years of research, the extent of map creation and maintenance technology still needs to be enhanced. Despite varied degrees of assistance from semi-automated technologies, most maps are still created by hand in reality [1]. Automation is challenging for VHR photos because their spectral resolution is naturally reduced, and minute objects and surface roughness on a small scale become evident. These factors cause the picture intensities to vary widely within classes while showing slight variation between types [2, 3]. Numerous aerial or space-borne sensors capture a lot of very high-resolution (VHR) remote-sensing images daily, mainly for geography and global surveillance. Semantic picture segmentation is a step between unprocessed photos and a vector map layer. (a.k.a. land-cover classification or pixel labeling) [4]. Its goal is to select a similar class label from a small pool of candidates for each picture pixel, one that corresponds to the intended item categories. Semantic segmentation in cities is made more difficult by the fact that many types of artificial objects are made of a variety of different materials, as well as the fact that city objects (like constructions and green areas) are miniature and interact with one another through obstacles, cast shadowy areas, inter-reflections, etc. [5]. The semantic segmentation issue is often expressed as supervised learning, where a classifier based on statistics learns to predict the conditional likelihoods (ground truth=class) from spectral properties of the picture in response to some labeled training data [6]. Standard alternatives for input characteristics include raw pixel intensities, simple arithmetic combinations of the basic parameters, such as vegetation indices, and other statistics or filter responses reflecting the local image texture. Since the establishment of classifications that integrate efficient feature selection, one option has been to pre-model a large, overlapping set of attributes for training and let the algorithm choose the optimal subset with the hopes that less critical information would be lost during feature encoding as a result [7, 8]. Since the groundbreaking study, deep learning has fast advanced to the number of current field SOT for a wide range of learning-based image analysis tasks. ML using multi-layer neural networks starts with raw picture data and incorporates finding the best characteristics as part of the training method rather than requiring a separate feature specification [9, 10]. The breakthrough occurred when it was demonstrated that, given

Semantic Segmentation of Aerial Images  147 sufficient training data and computing capacity, specific learning models, Convolutional Neural Networks outperformed rival approaches on classification tasks like the ImageNet competition by a significant margin. A single machine can train from millions of photos using CNNs since they both take advantage of the shift-invariance of images and can be readily parallelized and operated on GPUs. They have recently outperformed other job methods, from voice processing to visual object identification [11–14]. CNNs have achieved well on the ISPRS standard for identifying aerial images. Specifically, earlier deep-learning techniques, such as, have sporadically been applied to satellite imaging. The following is this book chapter’s essential contribution: • Presenting a compilation of highly detailed satellite pictures mixed with aerial views of images. Each image has a segmentation map that was carefully made to correlate with it. • It introduces Hybrid U-Net, an FCN-inspired symmetric multiscale feature fusion architecture that uses deep coarse feature maps to predict more precise segmentation maps. • It conducts extensive trials highlighting U-Net’s outstanding capacity for identifying semantic segmentation and its fluctuations.

8.2 Related Work Semantic segmentation is one area where deep learning algorithms have recently thrived. This section presents relevant publications that provide CNN-based methodologies as a foundation for comprehending the subject. A patch-based CNN-based single-class prediction method was put out by Marmanis et al. [2]. Thus, they train two CNNs independently to learn and assess roads and structures. They also forecast those items’ probability distribution from aerial images. Their approach divides the raw aerial picture into 6464 patches to further incorporate the local environment, which is then normalized using a Gaussian filter as a preprocessing step. The normalized patches are then fed through several convolutional layers, continued by two fully connected layers, which provide a 256-vector that has been modified into a 16x16 label patch [13, 14]. With two datasets made up of substantial aerial pictures and essential road and building labels, they evaluated their methodology. The author in [15] expands the primary task into a multiclass dense categorization

148  Advances in Aerial Sensing and Imaging issue to predict roads and buildings from RSI-CB256 datasets. To concurrently anticipate highways, buildings, and backdrop, they suggested using a single CNN technique. They employed the same neural network structure as RSI-CB256 [23], but rather than a 256-vector output, a 768-vector was molded into a 16x16x3 conv with one patch produced. The model performs better than the baseline for both the road and construction classes [16]. This is because, when it occurs, the correlation between a road and a structure in the same picture may be taken advantage of by the single CNN architecture. To reduce the impact of the background noise, the authors also utilized a SoftMax (CIS) function. Even though semantic segmentation requires contextual characteristics, object placement is a critical factor in the ground. The main issue with CNNs for semantic categorization is that they might lose this information when they merge pixel locations (in their fully linked layers). As a result, a group of novel architectures called FCNs have been suggested to save this locational information. The FCN network uses CNNs for literal semantic segmentation, which was introduced by [17]. This approach proposes convolution to transform a base CNN into an FCN rather than building a new network from the ground up. Their technique converts the CNN’s fully linked layers into equivalent convolutions. Additionally, they developed an up-sampling method employing deconvolutional layers that enables them to produce a segmentation map. An FCN strategy was recommended by [9] to meet the buildings extraction challenge. Their design is built on the UNet using the encoding and decoding principle, where the input picture is first compressed using an RSI-CB256 [6] convolutional CNN to create a more miniature representation [18, 19]. Then the decoder uses sampling layers to rebuild the image to the label size. The holes in the structures that the authors noted are successfully filled in [3]. However, some anomalies are found at the margins of the anticipated items. The separate road segments found by [2] will serve as the limit on which this effort will concentrate. Figure 8.1 shows how this restriction is visually discernible. The SVM technique, created to categorize scenes, is described in detail by the authors. SVM was chosen because it has several appealing qualities that might be useful in this situation: its responses are global and unique; it has a fundamental grasp of geometry and does not depend on the size of the information space [20, 21]. Develop and describe a method for an “open world” prediction system that can update new item classes, be resilient to these hidden groupings, and operate with the minor downtime possible. The first stage in doing this is to identify new courses continually; later, it is to update the model to take

Semantic Segmentation of Aerial Images  149 these new classes into account. In search of a deep learning solution, [4] introduce a novel model using UNet, which stands in place of the SoftMax function as the network’s top layer, which calculates the likelihood that an input belongs to an unidentified class [21–23]. Lowering the errors a deep network makes when presented with artificially created pictures. [24] give a model known as “Open-Set Neural Network and a version known as OSNNcv, which outperforms previous approaches in the literature and can recognize data from unknown classes throughout training time. By examining whether the two nearest samples come from distinct types, the OSNNcv technique determines whether the input sample may be unfamiliar. The OSNN algorithm uses the ratio [25]. The OSNN approach applies a threshold to the ratio of comparison scores between the two most similar classes. This method’s intrinsic multiclass nature implies that it is unaffected when the number of training courses rises, which is one of its benefits [26]. In contrast to other cutting-­ edge methods, OSNNcv has the property of naturally multiclass (non-­ binary-based). When the number of classes is raised, these final ones often lose some efficiency, but the solution suggested by [5] is unaffected by the number of types. Our approach and its open-set features mainly depend on deep convolutional structures [27]. Nevertheless, conventional CNNs are not well suited for per-pixel semantic segmentation, mainly because it makes it challenging to identify inputs from individual pixels because one of its basic concepts is gradually gathering data over wider and larger image regions [28, 29]. Deconvolutional network structures restore the input through the FCN network so that the position of the individual responses is preserved. The semantic data may now be traced back to a specific pixel position because of these two new advancements to the CNN architecture [30–33]. We develop an FCN that converts intensity and range information into a pixel-wise full-resolution prediction using deep deconvolution methods.

8.3 Proposed Method Convolutional neural networks are the most influential framework for conceptual image interpretation tasks in deep learning. Every single neuron at level l will only take information through a fixed-size, trained convolution classifier, spatially localized window W in the previous layer l – 1. Each neuron at level l then o/p a vector of differentially using weighted sums of those values, where x1 =

∑

i∈ω

wi xil −1 for each vector dimension

150  Advances in Aerial Sensing and Imaging is shared by all neurons in that layer. This method significantly reduces the number of unconstrained elements in the framework and considers the shift-invariance of image architectures. Figure 8.1 displays the U-Net structure, which is the primary component of our network design. The system comprises an encoder sub-­ network followed by a decoder sub-network. The encoder sub-network has down-sampling steps, each corresponding to a transposed convolution used for up-sampling information in the decoder sub-network. The encoder’s feature maps are combined with the up-sampled elements of the exact resolution. Two 3*3 convolution layers that contain Rectified Linear Unit (ReLU) activation functions are utilized in each stage of the encoder sub-network, followed by a max-pooling process with a pooling size of 2*2. The number of filters in the convolutional layer structure doubles with each level of downsampling. The filter size of each layer is displayed under each convolution block in Figure 8.1. By repeating this cycle, four feature maps of different sizes, each rich in semantics, are produced. In the decoder sub-network, the number of filters in each level is halved by up-sampling the feature map first using a 2*2 transposed convolution operation. Then, two 3*3 convolutions + ReLU are computed using Eq. (8.1),

= Rel



1 2

∑

n i =1

yi − cxi

2 2

(8.1)



Processes are repeated in succession. The loss is calculated using Equation (8.2). The limited usage of information streams is a bottleneck

Conv2D128x128

Conv-64x64 ReLu

Dense Layer 64

Dense Layer 128 Sigmoid

Train dataset

Dense Layer 5 Sigmoid

ReLu

Dense Layer 96 Sigmoid

Dense Layer 16

Flatten

Max Pooling 2x2

ReLu

Generate Model

Soft Max

Validate DB

Figure 8.1  Proposed UNet model for semantic segmentation.

Semantic Segmentation of Aerial Images  151 to the effectiveness of U-Net. The fuse rules in U-Net are restrictive in that they solely combine feature maps of the same scale from the encoder and decoder sub-networks. As a result, there exists a substantial semantic gap between these feature maps. However, U-Net only partially uses the deep and coarse layer feature data, which is crucial for obtaining the global structure.



1 loss = −  N 

∑

 e xwe + f ( yi ) + b yi   log  i x  i =1 wetj f ( yi )+ b j   cj   j =1 

n

∑

(8.2)

Using a multiscale skip-connected architecture, we integrate the coarse semantic and fine appearance feature maps from the encoder and decoder sub-networks. To solve these drawbacks. Effectively using deep layers of feature maps is essential for enhanced localization and identifying ­smaller-sized damaged regions. We utilized the U-Net Pixel-wise segmentation along with morphological processing. As a benchmark, we used pixel-wise classification models, specifically U-Net and its multiscale variants, to demonstrate the efficiency of the additional skip sub-network in our proposed network. U-Net++ aims to improve U-Net by narrowing the semantic gap between the encoder and decoder feature maps before integration. This is accomplished by establishing dense skip connections between the encoder and decoder sub-networks. MACU-Net, on the other hand, modifies the first version of U-Net to segment satellite pictures with high resolution semantically. It uses multiscale skip connections, attention blocks, and asymmetric convolution blocks (ACB) to increase the representative strength of convolution layers.

8.3.1 Pixelwise Classification Method We initially suggested method in the present research modifies the closedset Pixelwise technique of [6] for open sets. The “Pixelwise” strategy involves concentrating on each pixel in the images. To address the issue of insufficient information for categorization, context windows of 55-by-55pixel crops are used, with the center pixel representing the crop class. By continuously analyzing each pixel in an image using the context-­sensitive window, a CNN can be trained for patch-level categorization. Figure 8.2 depicts the context window and center pixel used in this patch-wise

152  Advances in Aerial Sensing and Imaging

Context Window

Figure 8.2  Content window specifying pixel and its network marked in green dots.

classification approach to designate a particular class to the center pixel of each patch using Equation (8.3),

Log(I(p, q)) = Log(R(p, q)) + Log(Ill(p, q))

(8.3)

This architecture’s model parameters are depicted in Table 8.1. This architecture is identical to Pixelwise in the enclosed challenging scenario with the insertion of a threshold for probability after the SoftMax. In this Table 8.1  Parameters used in model construction. Model layer

Output

Stride

Kernal

In

out

No training parameters

Convolution

64x64x32

2

2

128

32

458692

64x64x64

2

3

64

64

459896

Max Pooling

64x64x32

2

3

32

64

595656

Conv

128x128x32

2

3

63

128

674569

Conv

128x128x16

2

3

16

128

745892

Max pooling

7x7x512

2

3

32

256

458692

Conv

7x7x512

2

3

512

128

459896

Conv

64x64x32

2

3

4096

256

595656

Conv

16x16x128

2

3

128

128

674569

Max Pooling

32x32x64

2

3

64

64

745892

Semantic Segmentation of Aerial Images  153 method, a pixel is classified as belonging to a specific class if its SoftMaxcalculated class confidence exceeds a predetermined threshold. If the probability at the pixel level exceeds the point, the pixel gets marked as unknown. The threshold values can range from 0 to 1, corresponding to the SoftMax-provided likelihood values.

8.3.2 Morphological Processing After the SoftMax layer estimates its findings and the algorithm establishes a limit, an additional computational internal filter is applied to the doubtful pixels. This filter examines each unknown Pixel’s surrounding pixels to identify whether or not it is an inside or outside Pixel. It has neighbors from different classes if it is close to the border. In this instance, the label with the more significant number of pixels in the neighborhood replaces the original categorization. If every Pixel belongs to the unpredicted class, the center pixels are not on the boundary, and the class should stay unlabeled. Except for adapting to a multiclass environment, the applied filter may be viewed as erosion over the unknown pixels. According to the author in [22], the set of all points in z such that B, translated by z, is included in A, as shown by Equation 8.1, is what is meant by the process of degradation of a set A by another set B (both in Z2). The approach lowers the number of incorrect unknown labels produced by the uncertainty of border areas. Thus, the network only rates the pixels being eroded as unknown. The rightmost modules of Figure 8.2 show how to utilize the morphological filter in an improving manner using Equation (8.4)

AΘB = (C | (B)c ⊆ A)

(8.4)

8.4 Datasets This publicly available dataset [25] is considered for the proposed system. The information set consists of pixels by pixels semantic segmentation of aerial images of Dubai captured by MBRSC drones and labeled with six classes. The collection consists of 72 photos distributed among six more oversized tiles. The dataset includes the following: Constructed buildings: #3C1098 On Road images: #6EC1E4  On Land (unpaved area): #8429F6

154  Advances in Aerial Sensing and Imaging On Plant life: #FEDD3A Water area: #E2A929 Water Data not labeled: #9B9B9B Sample input images are shown in Figure 8.3. Figure 8.4 shows the input image and masked image.

8.5 Results and Discussion Several trials were conducted to examine the impact of the threshold on the result to determine the ideal configuration for the Pixel-wise approach. Four metrics are used, including a median across the precision of known and unknown instances, the normalized accuracy across all samples, the cases of available classes, and examples of the anonymous type, which were considered to assess these variances. The results that may be predicted when applying such threshold values are shown by the normalized accuracy across all samples; however, as there are more available classes, there might need to be more adequate to select the suitable parameterization for unknown courses. Only the types are considered when measuring the precision of known class instances. Only classes encountered by the technique during learning are considered for measuring the correctness of examples of known types. The unknown correctness is the complete opposite of what was just stated. It only uses pixels from classes that the algorithm still needs to learn. To achieve the optimal balance between both, the arithmetic median is developed. The graph in Figure 8.5 displays the accuracy values mentioned here while changing the Pixel-wise method’s threshold. The ideal point was discovered empirically to be 0.7 by observing the average accuracy. All subsequent experiments utilized this value for Pixel-wise and Morph Pixel-wise using Unet. Even though it should be emphasized that this optimum threshold was determined worldwide based on the efficiency of all classes, it is feasible to fine-tune this value specifically for each category. Due to the lengthy calculation required and the fact that, in this scenario, a network would need to be trained from scratch for each class, we decided against performing a per-class assessment. Though expensive to prepare, this straightforward threshold concept paired with the pixel-wise network creates an approach that takes less time to generate results than other open-set methods. We evaluate the efficacy of the suggested technique for open-set semantic segmentation in light of the study carried out in the preceding section. Table 8.2 displays the results. The table clarifies that the proposed approach,

Semantic Segmentation of Aerial Images  155

Figure 8.3  Sample input images from dataset.

156  Advances in Aerial Sensing and Imaging 0

0

25

25

50

50

75

75

100

100

125

125

150

150

175

175

200

200 0

25 50

75 100 125 150 175 200

0

25 50

75 100 125 150 175 200

Figure 8.4  Input image and its mask for the preprocessing.

Pixel wise, produced workable results and demonstrates the viability of merging semantic segmentation with open set principles. When the Pixel wise approach was evaluated on an open set scenario, it produced outcomes comparable to those of the Pixelwise closed set, according to an analysis of the data provided in Table 8.2. The existing methods only outperformed closed-set methods with morphology filtering. Still, the more straightforward version (Pixel wise) also has the advantage of discovering new classes that closed set methods in this situation invariably mislabel. The Error Rate is shown in Figure 8.5 for a more precise illustration of this benefit. When looking at the Pixelwise network’s error rate, namely the principal diagonal, one can see that it is always 100%, which is to be anticipated given that the network was not intended to handle unknown classes and would thus incorrectly categorize all of the pixels that belong to this class. The benefit of utilizing an open-set semantic segmentation methodology may be understood by comparing the principal diagonal from all three techniques. While maintaining the error rate

Table 8.2  Comparison of model accuracy on independent model evaluation. Model

Accuracy (%)

Normalized accuracy (%)

Loss (%)

Pixel wise segmentation

75.3

76.3

12.63

Morphological

76.5

77.6

14.53

Pixel wise+ morphological

86.3

89.6

11.36

Semantic Segmentation of Aerial Images  157 Training and validation IoU

Training and validation loss

0.75

0.7

Training loss Validation loss

0.70

0.6

0.65 0.5 IoU

Loss

0.60 0.55 0.50

0.4 0.3

0.45

0.2

Training IoU Validation IoU

0.40 0

5

10

15 Epochs

20

25

30

0

Training loss Validation loss

1.0100

10

15 Epochs

20

25

30

Training and validation IoU

Training and validation loss

1.0125

0.24 0.23

1.0075

0.22

1.0050 1.0025

IoU

Loss

5

0.21

1.0000

0.20

0.9975

0.19

0.9950

0.18

Training IoU Validation IoU

0.9925 0

5

10

15 20 Epochs

25

30

0

5

10

15 20 Epochs

25

30

Figure 8.5  Validation loss and IoU plot of the model.

low for most known classes, both suggested approaches show the error rates lower than the pixel-wise system for the unrecognized prediction. Aside from that, it is crucial to test the proposed technique by changing the unknown class because each class has unique patterns. Figure 8.6 displays the experiment’s findings. Figure 8.5 shows that Morph-Pixel wise outperformed all other settings regarding metric rates, even when all the networks had a subpar performance. Another indication of the value of using a morphological filter to reduce the uncertainty in border regions is the difference in results between the Morph-Pixel wise and the Pixel wise. Several hyperparameters were initialized before beginning the model training. All neural networks were trained using the training data for 60 epochs rather than 400 [2, 5, 8], minimizing the binary loss of employing cross-entropy, an SGD with a momentum of 0.923, learning rates ranging from 0.001 to 0.0005, and reducing the learning rate by an amount of 0.1 when the networks showed no further advancements in the performance of the networks after nearly five consecutive epochs. Additionally, each network underwent L2 regularization with an estimated weight decay of 0.0002 for each network. Since the minimum requirement for 400 epochs is five consecutive days, we have limited the number to reflect that these resources are not always accessible.

158  Advances in Aerial Sensing and Imaging Image

0

0

True

0

Prediction

0

50

50

50

50

100

100

100

100

150

150

150

150

200

200

200

200

250

0

250 50 100 150 200 250 0 Image g

0

0

250 50 100 150 200 250 0 True

0

250 50 100 150 200 250 0 Prediction

0

50 0

50

50

50

100 0

100

100

100

150 0

150

150

150

200 0

200

200

200

250 0

0

250 50 0 100 00 150 0 200 250 2 0 0 Image

0

0

250 50 250 0 100 00 150 0 200 2 0 0 True

0

250 50 0 100 00 150 0 200 250 2 0 0 Prediction

0

50

50

50

50

100

100

100

100

150

150

150

150

200

200

200

200

250

0

250 50 100 150 200 250 0 Image

0

0

250 50 100 150 200 250 0 True

0

250 50 100 150 200 250 0 Prediction

0

50

50

50

50

100

100

100

100

150

150

150

150

200

200

200

200

250

0

250 50 100 150 200 250 0 Image

0

0

250 50 100 150 200 250 0 True

0

250 50 100 150 200 250 0 Prediction

0

50

50

50

50

100

100

100

100

150

150

150

150

200

200

200

200

250 0

250 50 100 150 200 250 0 Image g

0

0

250 50 100 150 200 250 0 True

0

250 50 100 150 200 250 0 Prediction ed ct o

0

50 0

50

50

50

100 0

100

100

100

150 0

150

150

150

200 0

200

200

200

250 0

0

250 0 50 100 150 200 250

250 50 100 150 200 250 0

250 50 100 150 200 250 0

Figure 8.6  Segmentation performed on various sample images.

Error

50 100 150 200 250 Error

50 0 100 00 150 0 200 250 2 0 Error

50 100 150 200 250 Error

50 100 150 200 250 Error

50 100 150 200 250 Error o

50 100 150 200 250

Semantic Segmentation of Aerial Images  159 To facilitate the implementation of simulations on the workstation, the number of epochs has been restricted to 50. This change has been made because the requisite resources are unavailable for at least six days, which is the minimum requirement for completing 300 epochs. Consequently, this decision functions to justify the reduction in epochs.

8.5.1 Analysis of the Proposed Method Every learning iteration includes a feed-forward pass, comparing the predicted output to the actual labels, and a backpropagation phase. The network weights are updated using stochastic gradient descent during the backpropagation phase. Forward passes are far less expensive than backpropagation, which requires evaluating gradients for each weight, and calls for matrix multiplications. Additionally, it is a good idea to utilize dropout in training, which involves arbitrarily turning off a portion of the neurons to uncorrelated the learning of various weights and lessen over-fitting. We employ a 60% dropout rate at multiple levels, as shown in Figure 8.7. Empirically, training dropout in our scenario slightly improves performance. This is due to two reasons. First, the models from which we work have previously undergone thorough pre-training on sizable databases (including dropout). Over-fitting is not a problem since the (shallower) majority of layers are adjusted for our training data while not significantly changing from the original, well-regularized state. Second, our model includes direct links between shallow and deep levels. Better spatial localization is the goal of these “skips” connections. Still, it’s likely that including low characteristics that are more general regularizes the high-level patterns and makes them less susceptible to over-fitting that is more task-specific.

0

0

0

100

100

100

200

200

200

300

300

300

400

400

400

500

0

100 200 300 400

500 500 0

100 200 300 400

500 500 0

100 200 300 400

Figure 8.7  Map of classes with buildings with high vegetation: Labeling error in prediction.

500

160  Advances in Aerial Sensing and Imaging We conducted tests with the Vaihingen data set from the dataset semantic labeling challenge to test and verify our approach. This data set consists of a digital surface model with the exact resolution and 33 tiles of varying sizes derived from an airborne orthophoto mosaic with three spectral bands. The overall size of the data set is about 1.456 pixels, but less than half—those used for training and authentication—have their ground truth made public. The organizers withhold the data for the remainder to objectively evaluate submitted findings. The photos have a GSD of 9 cm and are highly detailed. The categories that need to be classed include impermeable surfaces, structures, low plant life, trees, and automobiles. We prevent initial processing requiring human involvement or choosing data-specific hyperparameter values to keep our pipeline as automated as possible. Instead, we feed the data supplied by the reference system directly into the network. Even if the CNN results—ours and others—are already excellent, more can be done. We frequently notice that the web makes modest, isolated errors while occasionally over-smoothing rough edges and corners. The network may have learned to preserve them since there are some very tiny roads in the data because the latter are frequently designated impervious surfaces. After pooling and deconvolution, the features’ more significant location uncertainty may contribute to the need for more crisp borders. The annotated training data’s intrinsic imprecision may be another factor. Human annotators typically annotate a crisp, straight border using their domain expertise. Still, they need to be more consistent with where they place it concerning the image gradient. The system may “learn” that border localization uncertainty if the identical class boundaries are randomly moved a few pixels inwards or outwards in other patches. The borders in real ortho-photos are particularly challenging to determine precisely because limited DSM precision frequently makes small portions of facades visible close to the top view edge or the roof. As also noted, a detailed examination of The Vaihingen data set’s annotations quickly reveals various ground truth problems. Although our pipeline frequently accurately categorizes these regions—often beating human annotators—it is nonetheless given a poor rating in the evaluation. See illustrations in Figure 8.7. In a data set that huge, some label noise is unavoidable; nevertheless, it should be noted that ground truth errors are not negligible, with multiple authors achieving overall accuracies of around 92% and discrepancies between competitors of 7%. If the data set isn’t updated, looking back at the original data might be essential.

Semantic Segmentation of Aerial Images  161

8.6 Conclusion The research concludes that deep learning techniques, especially the U-Net architecture, are highly effective for the semantic segmentation of exceedingly high-resolution aerial photographs. The proposed Multiscale Hybrid U-Net, which utilizes a multiscale feature fusion technique, outperforms the conventional U-Net architecture, mainly when producing segmentation maps for high-resolution images rich in fine detail. Experiments conducted for this study demonstrate that the proposed method can effectively identify various forms of conflict-related building damage. In addition, the research highlights the importance of incorporating skip connections and multiscale feature maps into deep neural architectures to improve segmentation accuracy. Overall, this study contributes to the expanding corpus of literature on deep learning for image interpretation tasks and demonstrates the potential of these techniques for real-world applications. The success of this study suggests several promising directions for future research. One potential avenue for further exploration is applying the proposed method to other types of high-resolution imagery, such as medical imaging or remote sensing. Another area for future investigation is optimizing the proposed architecture for even higher-resolution images, which could be achieved by adding more layers or utilizing more advanced convolutional neural network architectures. Additionally, the use of transfer learning techniques, where the pre-trained model is fine-tuned on a specific dataset, could be explored to improve the accuracy and efficiency of the segmentation process. Finally, developing a user-friendly interface to allow non-experts to use the proposed method for their specific needs could broaden the scope of its application and impact. Overall, these potential future research directions indicate the continued growth and importance of deep learning for image segmentation and interpretation.

References 1. Mendes Júnior, P.R., De Souza, R.M., de O. Werneck, R., Stein, B.V., Pazinato, D.V., de Almeida, W.R., Penatti, O.A.B., da S. Torres, R., Rocha, A., Nearest neighbours distance ratio open-set classifier. Mach. Learn., 106, 3, 359–386, 2017. 2. Nogueira, K., Mura, M.D., Chanussot, J., Schwartz, W.R., Dos Santos, J.A., Dynamic multi-context segmentation of remote sensing images based on convolutional networks. IEEE Trans. Geosci. Remote Sens., 57, 10, 7503– 7520, 2019.

162  Advances in Aerial Sensing and Imaging 3. Alshehhi, R., Marpu, P.R., Woon, W.L., Mura, M.D., Simultaneous extraction of roads and buildings in remote sensing imagery with convolutional neural networks. ISPRS J. Photogramm. Remote Sens., 130, 139–149, 2017. 4. He, H., Yang, D., Wang, S., Wang, S., Li, Y., Road extraction using atrous spatial pyramid pooling integrated encoder-decoder network and structural similarity loss. Remote Sens., 11, 9, 1015, 2019. 5. Saito, S. and Aoki, Y., Building and road detection from large aerial imagery, in: Image Processing: Machine Vision Applications VIII, vol. 9405, SPIE, pp. 153–164, 2015. 6. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Lawrence Zitnick, C., Microsoft coco: Common objects in context, in: Computer Vision–ECCV 2014: 13th European Conference, Proceedings, Part V 13, Zurich, Switzerland, September 6-12, 2014, Springer International Publishing, pp. 740–755, 2014. 7. Long, J., Shelhamer, E., Darrell, T., Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440, 2015. 8. Marmanis, D., Wegner, J.D., Galliani, S., Schindler, K., Datcu, M., Stilla, U., Semantic segmentation of aerial images with an ensemble of CNSS. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 3, pp. 473–480, 2016. 9. Niu, R., Sun, X., Tian, Y., Diao, W., Chen, K., Fu, K., Hybrid, multiple attention network for semantic segmentation in aerial images. IEEE Trans. Geosci. Remote Sens., 60, 1–18, 2021. 10. Abdollahi, A., Pradhan, B., Alamri, A.M., An ensemble architecture of deep convolutional Segnet and Unet networks for building semantic segmentation from high-resolution aerial images. Geocarto Int., 37, 12, 3355–3370, 2022. 11. Skourt, B.A., El Hassani, A., Majda, A., Lung CT image segmentation using deep neural networks. Proc. Comput. Sci., 127, 109–113, 2018. 12. Girisha, S., Verma, U., Manohara Pai, M.M., Pai, R.M., Uvid-net: Enhanced semantic segmentation of UAV aerial videos by embedding temporal information. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 14, 4115–4127, 2021. 13. Swathi, A. et al., A reliable novel approach of bio-image processing—Age and gender prediction, in: Proceedings of Fourth International Conference on Computer and Communication Technologies. Lecture Notes in Networks and Systems, vol. 606, K.A. Reddy, B.R. Devi, B. George, K.S. Raju, M. Sellathurai, (Eds.), Springer, Singapore, 2023. 14. Gowroju, S. and Kumar, S., Robust deep learning technique: U-net architecture for pupil segmentation, in: 2020 11th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), IEEE, pp. 0609–0613, 2020. 15. Gowroju, S. and Kumar, S., Robust pupil segmentation using UNET and morphological image processing, in: 2021 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), IEEE, pp. 105–109, 2021.

Semantic Segmentation of Aerial Images  163 16. Gowroju, S. and Kumar, S., Robust deep learning technique: U-net architecture for pupil segmentation, in: 2020 11th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), IEEE, pp. 0609–0613, 2020. 17. Gowroju, S., Kumar, S., Ghimire, A., Deep neural network for accurate age group prediction through pupil using the optimized UNet model. Math. Probl. Eng., 2022, 1–24, 2022. 18. Gowroju, S. and Kumar, S., Review on secure traditional and machine learning algorithms for age prediction using IRIS image. Multimedia Tools Appl., 81, 24, 35503–35531, 2022. 19. Singh, N.J. and Nongmeikapam, K., Semantic segmentation of satellite images using deep-unit. Arabian J. Sci. Eng., 4, 1–13, 2022. 20. Krizhevsky, A. and Hinton, G., Learning multiple layers of features from tiny images. Learning Multiple Layers of Features from Tiny Images, University of Toronto, 7, 2009. 21. Khan, S.D., Alarabi, L., Basalamah, S., Deep hybrid network for land cover semantic segmentation in high-spatial resolution satellite images. Information, 12, 6, 230, 2021. 22. Li, W., He, C., Fang, J., Zheng, J., Fu, H., Yu, L., Semantic segmentation-based building footprint extraction using high-resolution satellite images and multi-source GIS data. Remote Sens., 11, 4, 403, 2019. 23. He, N., Fang, L., Plaza, A., Hybrid first and second order attention Unet for building segmentation in remote sensing images. Sci. China Inf. Sci., 63, 1–12, 2020. 24. https://www.kaggle.com/datasets/mahmoudreda55/satellite-imageclassification. 25. https://www.kaggle.com/datasets/humansintheloop/semantic-segmentationof-aerial-imagery. 26. Kumar, S., Jain, A., Agarwal, A.K., Rani, S., Ghimire, A., Object-based image retrieval using the U-Net-based neural network. Comput. Intell. Neurosci., 1–24, 2021. 27. Kumar, S., Rani, S., Jain, A., Verma, C., Raboaca, M.S., Illés, Z., Neagu, B.C., Face spoofing, age, gender and facial expression recognition using advance neural network architecture-based biometric system. Sensor J., 22, 14, 5160– 5184, 2022. 28. Kumar, S., Jain, A., Rani, S., Alshazly, H., Idris, S.A., Bourouis, S., Deep neural network based vehicle detection and classification of aerial images. Intell. Autom. Soft Comput., 34, 1, 119–131, 2022. 29. Kumar, S., Jain, A., Shukla, A.P., Singh, S., Raja, R., Rani, S., Harshitha, G., AlZain, M.A., Masud, M., A comparative analysis of machine learning algorithms for detection of organic and non-organic cotton diseases. Math. Probl. Eng., Hindawi J. Publ., 21, 1, 1–18, 2021.

164  Advances in Aerial Sensing and Imaging 30. Rani, S., Ghai, D., Kumar, S., Kantipudi, M.V.V., Alharbi, A.H., Ullah, M.A., Efficient 3D AlexNet architecture for object recognition using syntactic patterns from medical images. Comput. Intell. Neurosci., 2022, 1–19, 2022. 31. Choudhary, S., Lakhwani, K., Kumar, S., Three dimensional objects recognition & pattern recognition technique; related challenges: A review. Multimedia Tools Appl., 23, 1, 1–44, 2022. 32. Rani, S., Ghai, D., Kumar, S., Reconstruction of simple and complex three dimensional images using pattern recognition algorithm. J. Inf. Technol. Manage., 12, 235–247, 2022. 33. Rani, S., Ghai, D., Kumar, S., Object detection and recognition using contour based edge detection and fast R-CNN. Multimed. Tools Appl., 22, 2, 1–25, 2022.

9 Implementation Analysis of Ransomware and Unmanned Aerial Vehicle Attacks: Mitigation Methods and UAV Security Recommendations Sidhant Sharma1, Pradeepta Kumar Sarangi1*, Bhisham Sharma1 and Girija Bhusan Subudhi2 1

Chitkara University Institute of Engineering & Technology, Chitkara University, Punjab, India 2 Cognizant Technology Solutions, Abu Dhabi, UAE

Abstract

Unmanned aerial vehicles (UAVs) have increased in recent years, with applications ranging from military and law enforcement to civilian and commercial things. However, with the increased use of these devices comes a heightened risk of security threats, including ransomware attacks. This paper analyses ransomware’s history, types, special attacks, and protection techniques. In addition, the paper discusses the unique security threats in unmanned aerial vehicles and the potential impact of cyber-attacks on these devices. The analysis highlights the importance of understanding the various types of ransomwares and implementing effective protection techniques to mitigate the risk of cyber-attacks on unmanned aerial vehicles. The paper concludes with recommendations for future research in this area, including developing more advanced protection techniques to combat emerging ransomware threats. Keywords:  Ransomware, malware, cybersecurity, FANET, UAV, encryption, notable attacks, mitigation

*Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (165–212) © 2024 Scrivener Publishing LLC

165

166  Advances in Aerial Sensing and Imaging

9.1 Introduction This chapter provides a comprehensive overview and structured approach to analyzing ransomware and unmanned aerial vehicle (UAV) attacks. It covers various aspects of these types of attacks, including their implementation, mitigation methods, and security recommendations. Our data is often the most valuable thing we own [1]. On a personal level, it may be our photos and videos, important files, and documents that we store on our computer, and most often, we cannot afford to lose. Considering big organizations, firms and companies, the data owned by those institutions is even more crucial and valuable. For instance, employee information, financial information, personally identifiable information (PII) or a company’s private data that only the company would have; all this information is a critical asset. The attackers know that the organizations will be willing to pay a certain amount if they can restore access to the compromised data. Because of this, hackers can make a lot of money using ransomware attacks. Ransomware or crypto-malware is a form of computer malware that may block access to the victim’s data or encrypt the data [2]. It then forces the victim to pay the ransom through online payment methods, often in cryptocurrency like Bitcoin. Victims of ransomware attacks have three options after the infection: either they can pay the ransom, try to remove the malware, or reboot the device. The method to decrypt the data is by attaining the proper decryption key, which the attacker can only send unless the ransom is paid. Very often, ransomware spreads and finds its way to the system by users clicking on phishing emails with malicious links or attachments, visiting malicious websites and downloading malicious files or add-ons, exploit kits, pirated software, public Wi-Fi etc. As organizations and institutions take measures to protect themselves from these cyber-attacks, hackers are also searching for more sophisticated ways to execute zero-day attacks. Studying the history and evolution of computer malware, such as ransomware, is essential to sustain cyber-security. Ransomware is a severe and dangerous kind of computer malware as it can cause significant harm to an organization’s finance and can tarnish its reputation and utility. For instance, a crypto-ransomware named’ WannaCry’ emerged in 2017 and targeted Microsoft Windows users. Due to the continued use of outdated computer systems and ignorance of the need to update software, ransomware spreads rapidly worldwide. It is estimated that the attack caused $4 billion in losses worldwide [3].

Ransomware and Unmanned Aerial Vehicle Attack  167

9.2 Types of Ransomwares Ransomware can be categorized into four categories: Encrypting ransomware, Screen lockers, Scareware and Leakware/Doxware. Figure 9.1 illustrates some of the ransomware types. • Locker-ransomware: Locker ransomware is the kind of malware which solely locks users out of their systems. The malware usually prevents access to users’ desktops while keeping some functionality for the mouse and the keyboard for users to fulfil the demands of the ransom. Often locker ransomware does not encrypt the system’s data and only blocks the user from accessing the system, but in some cases, such as the ‘CryptoLocker virus’, it may also encrypt the files. Attackers usually use social engineering methods to force victims into fulfilling the ransom demands, like imitating a government body or law enforcement agencies such as the FBI or IRS. • Encrypting-ransomware (Crypto-malware): Encrypting ransomware can hack and encrypt sensitive files such as documents, images, and videos on the victim’s computer. Unlike locker ransomware, crypto-malware does not block or prevent access to the victim’s computer but instead uses sophisticated encryption algorithms, e.g., Triple Data

RANSOMWARE TYPES

Locker-ransomware

Encrypting-ransomware

Figure 9.1  Types of ransomwares.

Scarewares

Leakware or Doxware

168  Advances in Aerial Sensing and Imaging Encryption Standard (3DES) or (Advanced Encryption Standard) AES, to encrypt system files. A decryption key is required to decrypt the files and retrieve the data. As malware developers use such robust encryption methods, decryption becomes nearly impossible without the proper key. If the users do not back up their data, paying the ransom is the only way to get it back. Developers often include deadlines in their ransomware forcing users to fulfil their demands by a certain period; otherwise, their files would be deleted. Because of this, users panic and often pay a ransom to retrieve their data. • Scareware: Scareware is generally a social-engineering method to mislead users into believing their systems are infected with viruses. The program may present fake popups and security alerts on the user’s screen, often saying their computer has been infected. The program then prompts users to follow remediation steps, e.g., a bogus antivirus scan button or “remove virus” for a specific price. Frequently scareware includes malware that may steal the user’s data. They are mainly spread via pop-ups on malicious websites, spam emails or messages. • Leakware or Doxware: Leakware or doxware is a ransomware variant that aims to threaten the victims by stealing their data and releasing them in public if their demands for the ransom are not fulfilled. Ransomware mainly targets large companies and organizations as their primary target. It takes a company’s confidential information, archives, employee information and other kinds of data stored on some cloistered devices hostage.

9.3 History of Ransomware Ransomware has a history that can be traced back to the late 1980s when the first instances of this type of malware were seen. Since then, ransomware has undergone several changes and adaptations as hackers and cybercriminals look for new ways to exploit victims. This section will provide a chronological overview of the history of ransomware up until 2012, including special attacks, trends, and developments in the technology used by attackers.

Ransomware and Unmanned Aerial Vehicle Attack  169 The first kind of ransomware emerged in December 1989, when an evolutionary biologist named Joseph L. Popp developed and distributed a malicious software called ‘AIDS’ known as Aids Info Disk or PC Cyborg Trojan. As the early era of the internet was not sophisticated, the virus was spread to almost 20,000 individuals and health institutions [4]. The delivery method of the malware was carried out via infected floppy discs. Using a hijacked mail subscriber list from the World Health Organization AIDS conference and PC business world magazine, Popp sent the infected floppy disc to the victims via mail, labeling it as an “AIDS Information Introductory Diskette”. The disc was said to contain surveys meant to determine an individual’s risk of contracting AIDS disease. When the victim inserted the disk onto their computer, the malware would deliver the payload and infect the computer with the virus. The disc contained a Trojan, which was written in QuickBASIC 3.0. The malware did not immediately encrypt the victim’s files; instead, it took over a system file called AUTOEXEC.bat. This file was present on all Disk Operating Systems and was read during the boot process. Although the virus did not affect the boot process, the malware kept count of the number of boot times and was triggered at around the 90th boot sequence. It then encrypted all the files on the drive where the OS resides and showed a message displaying the ransom note illustrated in Figure 9.2. The impact of this ransomware did not precisely benefit Popp to a significant level, as most of their targets did not pay the ransom. However,

Figure 9.2  AIDS DOS Trojan horse payload [5].

170  Advances in Aerial Sensing and Imaging the malware struck fear into many victims causing them to wipe their hard drives, which led to the loss of their valuable research data as most victims were in the healthcare industry [6]. In January 1990, Virus Bulletin’s editor advisor Jim Bates released two programs named “AIDSOUT” and “CLEARAID”, which quickly removed the malware from the infected computers [7]. The AIDS Trojan was not mainly a widespread or profitable piece of ransomware but laid the foundation for using the concept of digital extortion to exploit people. Ransomware attacks have grown exponentially since the emergence of the AIDS Trojan, and it is estimated that attackers have earned over $1 billion in revenue ever since. 2005 and 2006 were the beginning of the rise of the digital world. As the internet grew more widespread, more invasive, and dangerous  crypto-malware  appeared. This  was  the  beginning  of  an  era of ransomware-led  cybercrimes. Ransomware named “Gpcode”, first encountered in Russia in December 2004, encrypted files such as doc, txt, pdf, xls and jpg [8]. However, Gpcode used a weak encryption key that was relatively easy to decrypt [8]. Attackers then focused on quantity over quality as they focused more on spreading the malware rather than the severity of it. The attackers requested only $20 for the decryption key. Gpcode was spread via phishing emails and website links, and the encryption key was not challenging to crack [9]. In 2006, it was the first time that ransomware was developed by keeping robust methods of encryption in consideration. The Archievus ransomware was the first to use an advanced 1024-bit Rivest-Shamir-Aldeman, also known as the RSA encryption method, a form of asymmetric encryption [10]. Archievus was spread via malicious website links, spam emails and phishing, affecting mainly Windows operating systems. Once the virus entered the system, it would scan the Documents folder and look for files with extensions such as .key, .zip, .pdf, .txt, and .xls, placing a file named “how to get your files back.txt” in the same directory. Because the virus used asymmetric encryption to decrypt the files, a public and a private key were required. This made the decryption process much more complex than the previous versions of ransomware, such as the AIDS Trojan. Cybersecurity professionals discovered that a single 38-character password could decrypt the infected files, ultimately leading to Archievus ransomware’s demise [11]. By the end of 2014, 87% of the American population used the internet. Social networking, messaging, forums, and P2P networks grew extensively, laying the foundation for a new kind of ransomware known as locker ransomware [11]. Locker ransomware can lock and take over the screen and prevent the user from accessing their device. In 2011, locker ransomware named “WinLock” emerged as the first of its kind and mainly spread via

Ransomware and Unmanned Aerial Vehicle Attack  171 malicious websites. In 2012, the world saw that ransomware was being used as a service, also known as ransomware-as-a-service (RaaS) [12]. Cybercriminals who had sparse technical ability could purchase ransomware on the dark web. Reveton was the first ransomware that was used as RaaS. It mimicked the Federal Bureau of Investigation and other law enforcement agencies and forced users into paying fines [12]. The ransomware displayed a bogus notice claiming that the users had violated several federal laws, such as copyright infringement, child pornography etc. Threat actors threatened victims by paying a ransom ranging from $100 to $200 and often had a deadline attached [13]. Reveton malware was at max during mid-2012 and 2014, and the distributor circles earned up to $400,000 monthly. Since the arrival of Reveton, ransomware has become an increasingly popular tool for cybercriminals. There have been numerous high-profile attacks in recent years, such as CryptoWall, CryptoLocker, WannaCry, NotPetya and many more, affecting many organizations and institutions worldwide. These attacks have demonstrated the potential for ransomware to cause widespread disruption and financial loss.

9.4 Notable Ransomware Strains and Their Impact Ransomware attacks have become increasingly prevalent in recent years, with several high-profile incidents causing significant damage and disruption. These attacks have targeted businesses, governments, and individuals, causing financial losses, data breaches, and even putting lives at risk. This section will examine some of the most notable ransomware strains in recent years, their impact on the victims, and the broader implications for cybersecurity.

9.4.1 CryptoLocker (2013) In September 2013, Cryptolocker emerged. Cryptolocker was one of the most profitable ransomware variants of its time. By December 2013, it had infected over 250,000 machines and generated over $2 million in revenue. The malware was spread via malicious websites and phishing email attachments, controlled by a “Gameover Zeus” botnet. The email attachments were made to resemble customer complaints and were sent to various business professionals. CryptoLocker targeted computers running the Microsoft Windows operating system [15]. The malware used a 2048-bit RSA key pair uploaded to a command-and-control server. Once the system was infected, the 2048-bit public key was generated from the server

172  Advances in Aerial Sensing and Imaging

Figure 9.3  Cryptolocker (2013) [14].

and sent to the victim’s computer. The malware only encrypted certain file types such as office files, AutoCAD files, OpenDocument and other documents and pictures [16]. Figure 9.3 shows the message demanding money worth 2 Bitcoins. It also included a period for the user to pay the ransom (72–100 hours), and if the user failed to pay, the private key would be destroyed on the server making the files unrecoverable. When the ransom was born, the program allowed victims to download a decryption program which contained the victim’s private key [17]. In 2014, The United States Department of Justice successfully conducted an operation which aimed to take down the botnet responsible for the distribution of CryptoLocker. It was observed that 41,298 Bitcoins were moved through four accounts between 15 October and 18 December, and the closing price of Bitcoin in December 2013 was around $660. Therefore, it can be calculated that CryptoLocker managed to generate over $27 million in revenue [18]. The success of CryptoLocker led to the emergence of many similar kinds of ransomware variants, such as CryptoWall and TorrentLocker. Since then, there has been a massive surge in different ransomware strains and attacks, primarily targeting the health, bank, and government sectors.

Ransomware and Unmanned Aerial Vehicle Attack  173

9.4.2 CryptoWall (2014) Since the fall of the CryptoLocker ransomware, CryptoWall has taken over the ransomware world in late February 2014. According to Dell SecureWorks Counter Threat Unit™ (CTU™), it was observed that a file-­ encrypting ransomware variant was being distributed actively on the internet. CTU believed that CryptoWall was the most prominent ransomware of its time. Several versions of CryptoWall have been released today, with each variant being more sophisticated than the previous. The infection vectors for the ransomware were mainly malicious email attachments, browser exploit kits, drive-by downloads etc. [19]. A spam botnet by the name of “Cutwail” was primarily responsible for distributing malicious extensions which contained a downloader called “Upatre” [19]. Upatre downloaded the samples of CryptoWall hosted on compromised websites. By August 2014, over 600,000 systems were infected with the CryptoWall ransomware, encrypting over 5 billion files. According to researchers, CryptoWall generated over $18 million in revenue by mid-2015, less than its predecessor could make [19, 20].

9.4.3 TeslaCrypt (2015) TeslaCrypt was another file-encrypting ransomware Trojan that looked very similar to CryptoLocker. In February 2015, the ransomware was detected by Dell Secure Counter Threat Unit™ (CTU™) when they saw that the Trojan was spreading via a browser exploit kit called “Angler” [21]. The ransomware used TOR networks to remain anonymous. It was operated via a command-and-control server that did not need network connectivity to encrypt files, making detecting and mitigating the malware much more complex. TeslaCrypt uses the AES-256 (Advanced Encryption Standard) as its encryption method. A wide range of file types, such as images, Microsoft Office files, Photoshop files, etc., were some of the targets for this crypto-malware. It was observed that the ransomware did not encrypt audio and video files such as mp3 or mp4, but several video game file types were affected. The virus created mayhem for the PC gaming community by encrypting more than 40 types of game files belonging to popular games such as Minecraft, Call of Duty, and World of Tanks. The USA, Germany, and Spain saw the most infections, followed by Italy, France, and the United Kingdom. The ransom amounts ranged from $250 to $1000. Security researchers from Cisco discovered that TeslaCrypt used symmetric encryption and built a decryption program, leading to TeslaCrypt V2.0 [21].

174  Advances in Aerial Sensing and Imaging

Figure 9.4  TeslaCrypt shutdown [24].

This version of the ransomware imitated the previously known one as ‘CryptoWall’. The encryption standards for the version 2.0 were more complicated than earlier ones and required the Elliptic-curve Diffie-Hellman (ECDH) algorithm to generate the decryption keys [21]. In November 2015, researchers from Kaspersky found a flaw in the encryption essential storage algorithm of TeslaCrypt 2.0, which allowed victims to recover their data [22]. In January 2016, a third version of TeslaCrypt was released, which fixed the flaws occurring in the earlier versions of the ransomware [23]. This version omitted the use of a master decryption key and instead had unique encryption for each of the infected systems, making the decryption process very challenging. Some of the systems infected with older versions of TesaCrypt contained a dat file named ‘key.dat’ which included an encryption key whereas other versions deleted it and stored an encrypted version of it in another file named RECOVERY_KEY.TXT [25]. In May 2016, the developers of TeslaCrypt ransomware themselves shut down their project by releasing the master decryption key, making the ransomware defunct. Figure 9.4 displays the TelsaCrypt project shutdown message.

9.4.4 Locky (2016) First discovered in February 2016, Locky ransomware gained attention from security researchers due to its widespread impact. Locky was spread via emails containing Microsoft Word documents with malicious macros embedded [26]. Locky used the concept of social engineering. The Word document had gibberish text with a title saying, “Enable macros if the data encoding is incorrect”. When the user enables macros, the malware is executed and downloads the Locky ransomware payload, which encrypts a wide range of files on the victim’s computer [27]. RSA-2048 + AES-128 cypher with ECB was the encryption algorithm used in the Locky ransomware. Botnet, named ‘Necurs’, was the primary distributor of the ransomware. In June 2016, an advanced version of Locky was discovered, which  could

Ransomware and Unmanned Aerial Vehicle Attack  175 differentiate between a live system infection and a sandbox test environment like a virtual machine that helped the ransomware evade detection [28]. The Hollywood Presbyterian Medical Center in Los Angeles, California, was hit by the Locky ransomware in February 2016. It is known that the medical center had to pay 40 bitcoins, which amounted to around $17,000, to decrypt the infected computers across their network [29]. In May 2016, Kansas Heart Hospital became the victim of Locky. The president of the hospital, Dr Greg Duick, mentioned that a small ransom was paid. Still, attackers did not give complete access to the hospital’s data but demanded another ransom [29]. According to Duick, the hospital refused to pay the second ransom as they felt it was not a wise decision. Duick also mentioned that up to 45% of hospitals became cyber-attack targets. Locky ransomware has been immensely influential in targeting health institutions with ransomware-led cyber-attacks.

9.4.5 WannaCry (2017) First discovered on 12 May 2017, WannaCry, a short form of WannaCrypt0r 2.0, was one of the most significant and most damaging cyber-attacks affecting various businesses, hospitals, and government agencies. The ransomware spread to over 150 countries, infecting over 250,000 computers in one day. WannaCry encrypts files on the infected system and demands payment in Bitcoin for a decryption key. It incorporated a worm-like capability to self-replicate and infect other computers on the same network without human interaction, making ransomware one of the most widespread and damaging cyber-attacks [30]. It spread rapidly by exploiting a vulnerability in Microsoft’s Server Message Block (SMB) protocol called EternalBlue, which had been discovered by the National Security Agency (NSA) and then leaked by a hacker group known as ‘The Shadow Brokers’ [30]. Microsoft released a patch for the vulnerability, but many organizations did not apply it, making them vulnerable to ransomware [30]. Using the exploit, The Shadow Brokers installed a backdoor tool called DoublePulsar, which allowed hackers to create entry points into the systems or computer networks to gain easy access later. DoublePulsar allowed hackers to deliver the WannaCry ransomware payload. During the initial phase of infection, the ransomware worm tries to connect to an unknown domain, www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com, which was essentially a DNS sinkhole acting as a kill-switch for the ransomware [30]. If the connection fails, the encryption process begins. First, it scans port 445 to check whether it is open and uses it to host SMBv1. If the network port is available, WannaCry

176  Advances in Aerial Sensing and Imaging 1

2 Arrive via exploit

3 File delivered via exploit running as a service

4 Drop ransomware f iles which performs encryption

5 Drop component f iles for creating ransom note

Encrypt local and shared files (targets 176 file extensions)

EXE

@[email protected] MS17-010 via Isass.exe

mssecsvc.exe

tasksche.exe (encryptor)

.WNCRY .wnry config file

Queries domain used as kill switch

Spread to other devices

Figure 9.5  WannaCry attack flow [31].

propagates to other unpatched devices. The attack flow of the WannaCry is illustrated in Figure 9.5. The payload, WanaDecrypt0r 2.0, displayed a message saying the files on the infected systems have been encrypted, demanding $300 worth of bitcoin, which has been illustrated in Figure 9.6. After three days, the ransom amount would be doubled. It also mentioned that the victims had seven days to pay the ransom; otherwise, all their files would be unrecoverable. As mentioned, the WannaCry ransomware attack was the most significant cyber-attack in history, causing worldwide losses of up to $4 billion. Russia, India, Ukraine, and Taiwan were among the most affected countries due to this ransomware. The most significant victims of the WannaCry ransomware attack were the National Health Services (NHS) hospitals in the United Kingdom and Scotland [32]. More than 70,000 devices and equipment the NHS, such as computers, MRI Scanners, and other diagnostic equipment, were affected by the attack. The attack is estimated to cost the NHS more than £90 million ($109 million), including lost output, IT costs, and restoring systems [32]. According to multiple cybersecurity researchers and government agencies, it is suspected that the North Korean state-sponsored hacking group known as Lazarus Group was attributed to the WannaCry ransomware attack. However, North Korea denied any involvement in the attack [33].

9.4.6 NotPetya (2017) NotPetya, a modified version of Petya released in 2016, was a highly destructive malware discovered in June 2017. The malware spread widely

Ransomware and Unmanned Aerial Vehicle Attack  177

Figure 9.6  WanaDecrypt0r ransomware note [31].

in Ukraine, affecting over 80 companies, including the National Bank of Ukraine. The cyber-attack was carried out by a Russian state-sponsored hacking group known as Sandworm, instigating a new kind of state-­ sponsored cyber warfare with Russia. Sandworm was believed to have been linked to Russia’s military intelligence agency, the GRU [34]. The primary target is Ukraine, so the malware also affected many other countries, such as the United States of America and the United Kingdom [35]. The ransom note displayed by NotPetya is shown in Figure 9.7. The primary infection vector used in the attack was a malicious software update for Ukrainian tax accounting software called M.E.Doc., which was widely used in Ukraine. The attackers hijacked M. E. Doc’s update mechanism and used it to distribute a Trojan update that contained the NotPetya malware [37]. Like WannaCry, NotPetya also used the exploit developed by the NSA, EternalBlue, which took advantage of a vulnerability in the Windows Server Message Block protocol to spread to other machines on the network [37]. Once a computer was infected, it would encrypt the Master Boot Record (MBR) of the drive, preventing it from booting. It also encrypted other crucial system files making the computer unusable.

178  Advances in Aerial Sensing and Imaging

Figure 9.7  NotPetya ransom note [36].

The ransom notes on the screen instructed the user to pay money in the form of Bitcoin to a particular address in exchange for an encryption key. Later, it was discovered that NotPetya was not ransomware but a wiper which means once it had encrypted a system, there was no way to recover the system files [37]. Figure 9.7 shows that the message displayed in the ransom note asking for money in exchange for a decryption key was bogus and had no precise decryption method. This indicated that NotPetya was designed for cyber espionage and sabotage instead of financial gain. The total financial damage caused by NotPetya is estimated to be billions of dollars, making it one of the most expensive cyber-attacks in history. Institutions and organizations such as banks, government institutions, and airports were disrupted due to the malware.

9.4.7 Ryuk (2018) The Ryuk ransomware was first discovered in August 2018 and was named after the popular manga and anime series Death Note. Ryuk is primarily distributed through spam campaigns or exploiting vulnerable RACs (remote access systems) [38]. Ryuk ransomware is believed to be a derivative of Hermes ransomware. According to CheckPoint Research, the code base and encryption algorithms used by Ryuk are very similar to those used by Hermes [38]. Once the ransomware infects a system, it begins to encrypt files using a combination of symmetric and asymmetric encryption. Each file is encrypted with a unique key and then a public key. The  attacker

Ransomware and Unmanned Aerial Vehicle Attack  179 holds the private key required to decrypt the files and demands payment in exchange for the key. After the encryption, Ryuk typically creates a ransom note that includes instructions and a Bitcoin wallet address to send the ransom, depicted in Figure 9.8. Malicious programs known as droppers are used to deliver and install Ryuk malware directly onto a victim’s computer. The droppers were typically distributed via email phishing campaigns or remote desktop protocols (RDP). They were designed to ensure the ransomware could run successfully on the victim’s machine by checking for the presence of specific security software or processes and terminating them if necessary. The attack flow of Ryuk is presented in Figure 9.9. Large organizations are the primary target for this ransomware as it typically demands large ransom payments, often ranging in millions of dollars. The ransomware is designed to evade detection and uses the technique of code obfuscation and fileless malware to make detection and analysis much more complex [40]. According to the FBI, Ryuk made over $61 million in ransom from 2018 to 2019. In CrowdStrike 2020 Global Threat Report, Ryuk was responsible for three of the year’s top 10 most enormous ransom demands: $5.3 million, $9.9 million, and $12.5 million. On 20

Figure 9.8  Ryuk ransom note [39].

180  Advances in Aerial Sensing and Imaging

C&C Drops Additional Payload(s) Emotet also Grabs and Deploys Additional Payloads, often TrickBot

PowerShell Downloads Emotet

Phising Email

Macro in Weaponized word Doc Launches PowerShell

CommandLine

Disable AV Services Harvest Data

Ryuk

Steals Credentials

Emotet

TrickBot

Figure 9.9  Ryuk ransomware attack flow [40].

October 2020, a French-based IT company, Sopra Steria, was hit by the Ryuk ransomware encrypting the company’s data and making it inaccessible. The attack is estimated to cost Sopra Steria $47–$59 million [41]. It was initially suspected that the North Korean state-sponsored hacking group Lazarus was involved in the Ryuk ransomware attacks, but no direct link was found to support this claim. It is generally believed to be operated by a Russian cybercriminal group known as Wizard Spider [42]. Ryuk remains an active and evolving threat, and there have been multiple reports of the attacks in 2021 and 2022. The attackers behind Ryuk constantly change their techniques to evade detection and maximize their profits.

9.4.8 REvil (2019) Sodinokibi/REvil, short for Ransomware Evil, is a strain of the ransomware family popularly used as Ransomware-as-a-service (RaaS) and was released in April 2019. Soon after the shutdown of Grandcrab ransomware, REvil started to take over and lead the ransomware world. REvil developers have admitted to building the ransomware by sharing the code base of Grandcrab [43]. Like any other ransomware, REvil was designed to encrypt files on a victim’s computer, rendering them inaccessible. It often spreads through phishing emails containing malicious attachments, RDP credentials and exploitation of vulnerabilities [43]. The ransomware uses Elliptic curve cryptography (ECC) as its encryption algorithm. The attackers then demand payment for a decryption key to restore the files. In the case of REvil, attackers typically require large sums of money, often ranging in millions of dollars [43]. Since REvil is used as RaaS, the malware is supplied to affiliates who carry out the attacks, and the developers take a 20–30% cut in the profit generated by the affiliates [43]. The developers of REvil have stated that they could generate over $100 million in revenue by

Ransomware and Unmanned Aerial Vehicle Attack  181 extortion in just one year [43]. One of the critical features of REvil was its use of a “double extortion” technique. This involves not only encrypting the victim’s data but also stealing their sensitive data and then threatening the victim to release their data to the public if the ransom is not paid [44]. So, the REvil/Sodinokibi ransomware was a type of crypto-­ransomware and a leak-ware or doxware. In April 2021, a Taiwan-based computer manufacturer called Acer was hit with REvil demanding $50 million in ransom, depicted in Figure 9.10 [45]. In July 2021, the group demanded a $70 million ransom from a software company, Kaseya, responsible for a supply chain attack that affected many organizations [45]. JBS, a major US meat producer, was attacked with ransomware and had to pay $11 million in Bitcoin to restore their data [46]. After encryption, the computer wallpaper changes and a text file containing instructions for a ransom payment is generated, and are illustrated in Figures 9.11 and 9.12 respectively. Shortly after the Kaseya attack, REvil websites and infrastructure suddenly disappeared from the internet, leading to speculation about what had happened to the group and its leaders [49]. On 23 July 2021, Kaseya received the decryption key from the Federal Bureau of Investigation and withheld it for three weeks to take down their servers, but their plan went in vain as the hackers went offline themselves [50]. In October 2021, the FBI, USCYBERCOM, the United States Secret Service, and other countries

Figure 9.10  TOR payment page from victim’s computer due to REvil [47].

182  Advances in Aerial Sensing and Imaging

Figure 9.11  Wallpaper changed after REvil infection [48].

Figure 9.12  Instruction text file for REvil ransom payment [48].

operated to infiltrate REvil servers [51]. A member of the hacker group tried to restore their servers but was compromised. Five individuals responsible for REvil/Sodinokibi were arrested, and two suspects were involved in Grandcrab ransomware [51]. In January 2022, the Russian Federal Security Service (FSB) raided the homes of 14 members of the REvil gang and seized currencies worth nearly $7 million in crypto wallets and 20 luxury cars [52]. Authorities in the US were informed that the group was shut down.

9.4.9 Present-Day Ransomware Families The Table 9.1 analyses some notable ransomware strains between 2020 and 2021, including when they were first observed, significant targets, and unique features or techniques used by the themes. By examining these strains and understanding their tactics, we can better protect ourselves and our organizations against the threat of ransomware attacks.

Ransomware and Unmanned Aerial Vehicle Attack  183 Table 9.1  Notable ransomware strains (2020–2021). Ransomware strain

First observed

Notable targets

Unique features

Egregor

September 2020

Healthcare providers, financial institutions

A unique approach to encryption, targeted attacks (RaaS)

Darkside

August 2020

Colonial Pipeline, other critical infrastructure

RaaS model, double extortion tactic

LockBit 2.0

June 2021

Healthcare providers, government agencies, financial institutions

RaaS, Double extortion tactics, sophisticated encryption

Avaddon

February 2020

IT, Wholesale, Media & Entertainment, Healthcare

Double extortion, DDoS attacks,

Hive

June 2021

Government agencies

Fast-spreading, sophisticated encryption

As of 2022, LockBit remains the most popular strain to carry out ransomware attacks for the second and third quarters of the year. LockBit is known for its double extortion technique, which involves encrypting the victim’s data and demanding payment in exchange for its recovery, along with a threat to leak or publish the stolen data on the dark web if the victims fail to meet the demands of the ransom. Several versions of LockBit have been released since its inception, with version 3.0 being the latest. Version 3.0 was released on 26 July 2022, and it uses an anti-analysis technique which helps conceal the malware and evade detection. Hackers continue to favor the Ransomware-as-a-Service (RaaS) model for malware deployment and huge pay-outs.

184  Advances in Aerial Sensing and Imaging

9.5 Mitigation Methods for Ransomware Attacks Ransomware attacks pose a significant threat to individuals, organizations, and many institutions worldwide, and it is vital to prepare for them to mitigate and minimize the damage caused. The best way to alleviate the impact of a ransomware attack is by preventing it in the first place. This section discusses some key strategies to mitigate ransomware damage: • Regular Backups: Periodic data backup is crucial to recover from a ransomware attack. Backups must be stored offline and onsite to prevent them from being infected or encrypted by ransomware. The backup frequency should be determined based on the data size and how frequently it changes. • Deploying Endpoint Protection: Deploying Endpoint Protection tools such as antivirus, anti-malware, and IDS (Intrusion Detection System) can help detect and prevent the spread of ransomware. These tools can monitor, control access to files and applications, and detect and block suspicious activity, preventing or limiting the malware from spreading further. • Keeping Software up-to-date: Zero-day attacks often exploit vulnerabilities in software. Keeping software systems up to date can reduce the likelihood of an attack. Ensuring that the latest security patches are installed as unpatched systems and are more vulnerable to malware, and attackers often exploit known vulnerabilities to access systems. • Network Segmentation: Network Segmentation involves dividing the network into smaller sub-networks to limit the spread of malware in case of a breach. This helps isolate infected systems and prevent malware from spreading to other network parts. • Incident Response Plan: A solid incident response plan is crucial to respond quickly and effectively during a cyberattack. The plans must include the implementation of important decisions such as Disaster Recovery Plans (DRPs), isolation of infected systems, restoration of data from backups and communication between stakeholders. • Access Control: Restricting access to critical systems and data can help limit the spread of ransomware if it infects your network. Access control must be implemented based

Ransomware and Unmanned Aerial Vehicle Attack  185 on the least privilege, meaning users should only have access to the data and systems necessary to perform their job. • User Education: Employees often make the best exploits. User education and awareness programs must be conducted regularly to reduce the chances and impact of a ransomware infection. Users must be trained on various security aspects, such as avoiding and recognizing phishing emails, avoiding clicking on unknown links or downloading attachments from unknown sources. Security practices like passwords and multi-factor authentication should be considered inside an organization’s network.

9.6 Cybersecurity in UAVs (Unmanned Aerial Vehicles) Unmanned Aerial Vehicles (UAVs), commonly known as drones, have become increasingly popular in recent years. They are aircraft without any human pilot, operated by remote control or onboard computers. One of the critical advantages of UAVs is their ability to perform tasks that are too dangerous or difficult for humans to undertake. For instance, they can be used to inspect pipelines and power lines, conduct search and rescue operations in hazardous environments, and survey disaster-stricken areas. Moreover, UAVs can be equipped with high-resolution cameras, thermal sensors, and other sensors to gather data and provide real-time situational awareness. They can also be used for various tasks such as goods delivery, agriculture and construction, transportation, aerial photography/­cinematography, advertisements etc. Drones are used extensively in military operations worldwide, offering many benefits over traditionally manned aircraft. These include longer flight times, greater endurance, and the ability to fly in hazardous or hostile situations without putting human pilots in danger. In the military, drones are used for various purposes, such as reconnaissance, surveillance, target acquisition, and strike missions. They can be equipped with multiple sensors, cameras, and other technology to gather intelligence, monitor enemy movements, and provide real-time situational awareness to military personnel. However, the rise of UAVs can produce many risks associated with cybersecurity, and they need to be addressed to ensure the safety and security of individuals and organizations. Recently, numerous instances of UAVs and drones have been used for malicious purposes, such as cyber espionage

186  Advances in Aerial Sensing and Imaging or cyber-attacks. UAVs can access sensitive areas, gather intelligence, or deliver malware to targeted systems. Moreover, UAVs can also be hacked or hijacked, leading to the loss of sensitive data or the use of drones for malicious purposes. To address these risks, it is crucial to implement robust cybersecurity measures for UAVs, such as encrypting data transmission, enforcing access controls, and conducting regular vulnerability assessments. Furthermore, drones must comply with relevant regulations and guidelines to protect privacy and prevent illegal activities. In this section, we will explore the role of UAVs in cybersecurity and examine potential risks and challenges associated with their use. We will also discuss the different types of cyber security threats that can be launched using UAVs and highlight the measures that can be taken to mitigate these threats. By understanding the risks and challenges associated with UAVs and implementing suitable cybersecurity measures, we can harness the full potential of UAVs while protecting against potential threats and ensuring the safety and security of individuals and organizations.

9.6.1 Introduction on FANETS FANET stands for “Flying Ad-hoc Networks,” a wireless network consisting of Unmanned Aerial Vehicles (UAVs) that communicate without needing a fixed infrastructure. In FANETs, UAVs act as nodes in the network and can exchange data with other UAVs, forming a temporary network that is highly mobile and adaptable to changing conditions. FANETs have many potential applications, including military operations, disaster response, environmental monitoring, and surveillance. One of the challenges of FANETs is that UAVs are highly mobile and may need more communication range, making it challenging to maintain a stable and reliable network. To overcome this, researchers are developing new algorithms and protocols to optimize network performance in dynamic and unpredictable environments. FANETs are an emerging technology that can transform many industries, from military to civilian applications. As the technology continues to evolve, we will likely see new and innovative use cases for FANETs in the future.

9.6.2 Network Security Concerning FANETs This section provides information about the network security concerning FANET UAV systems, such as the various network vulnerabilities that make UAVs susceptible to cyber-attacks and the measures that can be taken to prevent those threats. FANETs can be vulnerable to cyber-attacks

Ransomware and Unmanned Aerial Vehicle Attack  187 and exploitation like any other wireless network system. FANETs operate in an environment where nodes are highly mobile and constantly changing, making it difficult to establish and maintain secure communication links between nodes. Attacks on FANETs can take many forms, including jamming, spoofing, and physical attacks on UAVs. In this context, the security of FANETs is paramount, and careful design and implementation are necessary to ensure the safety and integrity of these networks. In this section, we will explore some of the network security concerns surrounding FANETs, including the types of attacks the FANETs are vulnerable to, the potential consequences of a security breach, and the measures that can be taken to mitigate these risks. By understanding the security challenges facing FANETs, we can work to develop solutions that can ensure the safe and effective use of this emerging technology in both military and civilian applications. Figure 9.13 represents a pie chart of security threats present in FANETs.

Selfishness 3%

Sybil Wormhole 1% 3%

Rushing 3%

Active interfering Black hole Data tempering

Active interfering 12%

Replay 5%

DoS

Black hole 7%

Eavesdropping Flooding GPS Spoofing

Modification 8%

Data tempering 5%

Grayhole Impersonation Jamming

MITM 7%

DoS 8%

MITM Modification Replay Rushing

Jamming 11%

Eavesdropping 8%

Selfishness Sybil

GPS Spoofing 7%

Wormhole Impersonation 7%

Grayhole 4%

Figure 9.13  Percentage of security threats in FANETS [53].

Flooding 1%

188  Advances in Aerial Sensing and Imaging • Jamming attack: A jamming attack is a cyber-attack that can disrupt the communication links in a Flying Ad-hoc Network (FANET). In a jamming attack, the attacker broadcasts a high-powered signal on the same frequency as the FANET’s communication, causing interference and disrupting the network. Legitimate connections are disrupted when a radio signal is interfered with by other signals. There are three types of Jamming attacks: constant jamming, random jamming model swaps between sleeping and jamming mode and reactive jammer model, where a radio signal is transmitted whenever activity is heard on the communication channel [53]. Jamming attacks can be particularly effective against FANETs because the UAVs are highly mobile, and their communication links constantly change. This makes it difficult to establish and maintain secure communication links between the UAVs and leaves the network vulnerable to interference from external sources. Figure 9.14 depicts a Jamming attack in FANETs.

jamming range

FANET Ground control Station

Figure 9.14  Jamming attack in FANET.

Ransomware and Unmanned Aerial Vehicle Attack  189 • Eavesdropping: Eavesdropping is a cyber-attack that can intercept and monitor the communication links in a Flying Ad-hoc Network (FANET). In an eavesdropping attack, the attacker intercepts and listens to the communication between the UAVs inside the FANET without the knowledge or consent of the users. An eavesdropping attack aims to access sensitive information, such as mission plans, target locations, or other strategic information. This attack helps the attacker gather intelligence and use it to disrupt or compromise the mission of the FANET. Figure 9.15 illustrates the working of an Eavesdropping attack. • Man-in-the-Middle attack (MitM): A Man-in-the-Middle attack is a type of cyber-attack that can intercept and alter the communication of two or more UAVs in FANET. In this, the attacker positions themselves between two UAVs and intercepts their communication, allowing them to read, modify, or inject messages into the communication stream. MitM can access sensitive information, such as military intelligence, mission plans or strategic data. Figure 9.16 shows the process involved in MitM attacks. • Spoofing/Impersonation: An attack can deceive or trick a UAV or Ground Control Station (GCS) in a Flying Ad-hoc Network system (FANET). Attackers generally generate false information or signals that mimic the authentic signals of another UAV or GCS intending to gain access to the network. There are many forms of spoofing, for example, GPS spoofing, where the attacker creates false GPS signals to mislead the UAV about its location or trajectory, communication spoofing, where the attacker generates fake signals malicious user listening to unencrypted data

"send an UAV to the local pedestrian precinct" Backbone UAV Ground Control Station

Figure 9.15  Eavesdropping in FANET.

190  Advances in Aerial Sensing and Imaging

UAV1 UAV2

Backbone UAV

Figure 9.16  Man-in-the-Middle attack in FANET.

to trick GCS or UAV into accepting a phoney command or message. Figure 9.17 shows the process involved in Spoofing attacks. • Replay: In a replay attack, the attacker intercepts and records data packets transmitted between UAVs or a Ground Control Station (GCS). The attacker then replays these recorded packets later to cause the receiving device to accept the repeated data as legitimate. A replay attack can be particularly effective in FANET systems that rely on unsecured wireless communication links. For example, suppose an attacker intercepts and replays a command packet previously sent to a UAV. In that case, the UAV may execute the command again, leading to unexpected or harmful consequences. Replaying the ARP (Address resolution protocol) request can be used to crack encryption keys considering the secure connection between UAVs. Process of a Replay attack is illustrated in Figure 9.18.

UAV 3

UAV 4

"I am UAV 1, was a part of FANET" UAV X

Figure 9.17  Impersonation attack.

UAV 2

Ransomware and Unmanned Aerial Vehicle Attack  191 UAV 3

UAV 4

resends the encrypted message (sniffing)

secure network communication

UAV 1

UAV 2

Figure 9.18  Replay attack in FANET network. UAV 2

UAV 1 (down)

A

A

A

A

Authentication

Backbone UAV Authentication X

Figure 9.19  DoS attack in a FANET network.

• DoS (Denial-of-Service): Denial-of-Service or DoS attacks flood a FANET network with traffic or resource requests, causing the network to become overwhelmed and unavailable to legitimate users. This can be done in many ways, for example, a Jamming attack, de-authentication, etc. These attacks can be made quickly with the help of “KaliLinux” distribution that contains hacking tools such as “airodump-ng”, “Aireplay-ng”, “aircrack-ng” or using the Websploit framework to perform jamming. Diagram of DoS attack is depicted by Figure 9.19.

9.6.3 UAV Security Enhancement We have discussed some security threats persisting on Flying ad-hoc Network systems. This section will provide some tips and recommendations

192  Advances in Aerial Sensing and Imaging for enhancing the security of UAV systems, including measures to prevent unauthorized access, protect data transmissions, and ensure these systems’ safe and effective operation in various environments. By following these guidelines, it is possible to enhance the security and reliability of UAV systems and ensure their safe and effective use in a wide range of applications. • Enhanced Surveillance: Keeping track of the purchase history of UAVs and their import and having improved video feedback when drones are used can help prevent malicious use. • Drone Licensing: Users must have a licensed permit to operate drones. • Setting up IDS/IPS: Intrusion Detection Systems and Intrusion Prevention Systems are network security monitors that monitor traffic and take action to block or limit the traffic, preventing the network from becoming overwhelmed and preventing Denial-of-Service attacks. • Enhanced Drone/UAV Detection Methods: This can help detect and prevent any incoming unidentifiable flying objects and allow enough time to neutralize their threat at a distance. • Authentication and Authorization: Strong authentication mechanisms such as password protection and two-factor authentication can help prevent unauthorized access to FANETs. Using a single-factor authentication is insufficient and can compromise UAV’s security which can cause UAV to become rogue or malicious. • Encryption: Using robust encryption methods for UAV network traffic is paramount to ensure confidentiality and prevent attacks like eavesdropping, tampering and interception of data packets. • Firewalls: These acts as a barrier between the FANET and the outside world and can prevent unauthorized access. • Time synchronization: Accurate time synchronization between UAV nodes can help prevent replay attacks.

9.6.4 Limitations in UAVs There are many limitations present in UAV network systems that researchers need to address. This section talks about some of the critical limitations.

Ransomware and Unmanned Aerial Vehicle Attack  193 • Limited battery life: Most UAVs have limited battery life, limiting their flight time and range. This can be a significant limitation for applications that require long flight times or remote operation. • Limited communication range: Drones and UAVs use wireless communication systems with GCS. Factors like weather and terrain can disrupt their connectivity. • Limited payload capacity: UAVs have limited payload capacity, which can limit their capabilities for applications that require heavy or bulky equipment. • Weak frequency range: Most UAV systems lack frequency hopping, making them vulnerable to attacks such as jamming or de-authentication. • Vulnerable to weather conditions: UAVs can be affected by adverse weather conditions, such as high winds or heavy rain, preventing them from flying and performing their intended tasks. • Poor processing power: Most UAVs need more processing power, limiting their ability to process large amounts of data in real time. • Weak authentication methods: Most drones are susceptible to hijacking by compromising the authentication process. Strong passwords and multi-factor authentication can help prevent this. • Regulatory and legal restrictions: Various restrictions and regulations can limit the application and operations of UAVs.

9.6.5 Future Scope The unmanned aerial vehicles (UAVs) field continuously evolves, and there is a vast scope for future research. Some potential areas of research in UAVs include: • Advanced autonomy and control: With minimal human intervention, developing advanced autonomy and control techniques for UAVs could enable them to operate more efficiently and effectively. • Multi-UAV systems: Researching the coordination and communication of multiple UAVs could enhance their capabilities for various applications, including search and rescue operations, surveillance, and delivery.

194  Advances in Aerial Sensing and Imaging • Security and privacy: As UAVs become more prevalent, developing new security and privacy measures to protect against cyber threats and unauthorized access is crucial. • Environmental sensing and monitoring: UAVs can be equipped with sensors to collect environmental data, including air quality, weather patterns, and ecological systems. Research in this area could enhance our understanding of the environment and inform environmental policy and decision-making. • Human-robot interaction: Developing new techniques for human-robot interaction could enhance the usability and effectiveness of UAVs for various applications. Overall, there is a vast scope for future research in UAVs, and the potential applications of these systems are only beginning to be explored. As UAVs advance, there will likely be many exciting new research opportunities.

9.7 Experimental Analysis of Wi-Fi Attack on Ryze Tello UAVs The popularity of unmanned aerial vehicles (UAVs) has surged recently, with hobbyists and professionals using these devices for various purposes. However, as with any connected device, UAVs are vulnerable to cyberattacks. In this section, we present an experimental analysis of Wi-Fi attacks on Ryze Tello UAVs using Kali Linux, exploring the potential impact of such attacks on the security and functionality of these devices.

9.7.1 Introduction The experiment aims to investigate the vulnerability of Ryze Tello UAV to the Wi-Fi Password cracking attack using tools such as Aircrack-ng suite. Aircrack-ng provides a bundle of tools used to evaluate Wi-Fi network security. It works on different aspects of Wi-Fi security: Monitoring data packets, attacking techniques such as replay attacks, de-authentication, fake access points and packet injection, Testing Wi-Fi drivers and cards and cracking WEP and WPA PSK (1 and 2) protocols. This attack aims to crack the password of the Wi-Fi network used by the drone, which is secured with WPA2. The attack involves not authenticating a user from the web, which triggers a re-authentication process. During this process,

Ransomware and Unmanned Aerial Vehicle Attack  195 the attacker will capture the hashed WPA key as part of the handshake. The attacker will compare hashed dictionary entries against the captured WPA key to retrieve the password. The password dictionary used in the attack is a list of compromised passwords designed explicitly for password-cracking attacks.

Specifications of Ryze Tello Drone: • Wireless: IEEE 802.11n (Wi-Fi) • Frequency: 2.4 GHz (2400–2483.5 MHz) • Transmission Distance: Up to 100 meters (depending on the environment) • Max Bitrate: 18 Mbps • Latency Approx: 100 ms • Access Point: Embedded AP • Encryption: WPA/WPA2 While the drone has no default password, users can set a password through the Tello application. The network password used in this attack was chosen from a password dictionary to demonstrate a realistic use case. The equipment used in this attack consists of a wireless network adapter with monitor mode enabled, along with the Aircrack-ng suite of tools available in Kali Linux.

9.7.2 Methodology To use the Aircrack-ng suite, a wireless adapter supporting monitor mode is required. This is necessary to scan nearby Wi-Fi networks and obtain their BSSID. With this, a 4-way WPA handshake can be initiated if a client tries to connect to the network which contains the hashed WPA key. An Android phone is used to connect to the drone’s Wi-Fi network using the Tello app to gain control of the drone. To simulate the attack, we set a password for the web using the word “password” as the password. The commands to perform the attack are as follows: • •

ifconfig displays the configuration for a network interface airmon-ng check kill shows and kills any process that might interfere with the aircrack-ng suite

196  Advances in Aerial Sensing and Imaging • airmon-ng start wlan0 This command is used to enable monitor mode on the wireless interface wlan0. • airodump-ng wlan0mon Airodump-ng is used to capture packets, capturing raw 802.11 frames. It collects WEP Initialization Vectors or WPA handshakes and uses them with aircrack-ng. BSSIDs and Encryption standards of the network can be seen for the Tello network. Figure 9.20 shows the tello network. • airodump-ng -c Y --bssid XX:XX:XX:XX:XX: XX -w / home wlan0mon This shows the clients connected to the targeted network along with the BSSID and Channel. -w is used to save the captured file in a directory containing the WPA handshake. Figure 9.21 shows the connected clients with the target.   A new terminal window is opened, and the following commands are executed: • Airplay-ng -0 10 -a XX:XX:XX: XX wlan0mon Executing this command injects 10 DeAuth (de-authentication) packets into the target, resulting in the disconnection of the client from the network. Once disconnected, the client

Figure 9.20  Tello network being shown in the scan [54].

Figure 9.21  connected clients with the target [54].

Ransomware and Unmanned Aerial Vehicle Attack  197 attempts to reconnect by sending a WPA handshake, which is captured and saved for later use. Figure 9.22 shows the DeAuth process. The capture file, in Figure 9.23, is obtained and then opened using an open-source packet analyzer called Wireshark. Wireshark examines the packets exchanged between the client and the network, such as the Android phone and the drone. The EAPOL filter, which stands for Extensible Authentication Protocol over LAN, inspects the WPA handshake. The hashed key can be seen in the Authentication section illustrated in Figure 9.24. • aircrack-ng -a2 -b XX:XX:XX:XX:XX: XX -w ‘dictionary path’ ‘captured file’ Once the WPA handshake is captured, we can proceed to crack the password offline using the command: aircrack-ng -a2 -b [BSSID] -w [path/to/rockyou.txt] [path/to/capture/ file]. Here a2 option is for the WPA2 encryption standard used by the target network, while b is the flag for the BSSID, followed by the path to the password dictionary “rockyou. txt” included in Kali Linux, as well as the way to the captured file. Figure 9.25 shows the password found information.

Figure 9.22  DeAuth process [54].

Figure 9.23  Wireshark analysis [54].

198  Advances in Aerial Sensing and Imaging

Figure 9.24  Hashed key [54].

Figure 9.25  Password found [54].

9.8 Results and Discussion The experiment on Wi-Fi password cracking on Ryze Tello UAV was successfully conducted using Kali Linux tools. The attack involved de-­ authenticating a user from the drone’s network and capturing the WPA handshake. A password dictionary was then used to crack the captured handshake offline, and the password was successfully retrieved. The result of the experiment demonstrated the vulnerability of the Ryze Tello UAVs Wi-Fi network to password cracking attacks. The attack was successful because the network used a weak password that was easily guessable and included in the dictionary used for the attack. This underscores the importance of using strong, complex passwords to secure wireless networks.

Ransomware and Unmanned Aerial Vehicle Attack  199 Furthermore, the experiment highlighted the effectiveness of Kali Linux tools for conducting security and penetration testing on wireless networks. The Aircrack-ng suite of tools proved to be efficient for cracking WPA2 passwords, as demonstrated in the experiment. In conclusion, the experiment demonstrates the importance of securing wireless networks with intense and complex passwords to prevent unauthorized access and potential attacks. It also emphasizes the need for regular security and penetration testing of wireless networks to identify and address vulnerabilities before attackers exploit them. UAVs are becoming increasingly popular in various industries, and their security risks are gaining attention. Our study found that UAVs are vulnerable to cyber-attacks due to their reliance on software and network connectivity. As a result, organizations that use UAVs must implement strict security measures, such as encryption, secure communication protocols, and monitoring software. The comparative analysis of ransomware presented in this paper reveals several significant findings. Firstly, ransomware is malicious software that has existed for several decades, with special attacks occurring as early as the 1980s. However, the use of ransomware has grown significantly in recent years, with several high-profile attacks impacting various businesses, government organizations and individuals. Our research identified several types of ransomwares, including file-encrypting, lock-screen, leak, and scareware. The most prevalent type of file-encrypting ransomware encrypts user files and demands a ransom to unlock them. Several notable ransomware attacks have occurred in recent years. These include the WannaCry attack, which affected over 200,000 computers in 150 countries, and the NotPetya attack, which caused over $10 billion in damages. To protect against ransomware attacks, we recommend implementing various protection techniques, including data backup, network segmentation, user education, and antivirus software. Data backup is particularly crucial, as it allows organizations to restore their data if it is compromised by ransomware. The figure below represents the complaints received by IC3 that show organizations belonging to a critical infrastructure sector that were the victims of a ransomware attack. In Figure 9.26 the bar graph shows that the healthcare and Public Health sectors are the most affected due to ransomware-led cyber-attacks. There are several reasons for this, such as: • Criticality of patient data: Information such as medical records, patient health data, financial data, etc., are some

200  Advances in Aerial Sensing and Imaging Infrastructure Sectors Victimized by Ransomware Defense Industrial Base 1 Emergency Services Waste and Wastewater Systems Chemical Communications Energy Transportation Food and Agriculture Commercial Facilities Government Facilities Critical Manufacturing Information Technology

2 4 12 17 31 38 52 56 60 65 74

Financial Services Healthcare and Public Health

89 148

Figure 9.26  2021 Sector data affected by ransomware [55].

critical information and can be exploited by attackers for financial gain. • Limited resources: Many healthcare organizations, especially smaller ones, have limited resources to dedicate to cybersecurity. • Ageing IT Infrastructure: Many healthcare organizations rely on legacy, outdated systems vulnerable to cyber-attacks. • A high value of data: The sensitive nature of patient data makes it a valuable target for attackers to execute a cyber-attack. According to a report from UK-based Cybersecurity company Sophos, ransomware-as-a-service became famous in 2020–2021. Out of hundreds of ransomware attacks investigated by Sophos, approximately 60% were due to ransomware-as-a-service. Figure 9.27 shows the notable ransomware families in 2020–2021.

Ransomware and Unmanned Aerial Vehicle Attack  201 Ransomware families investigated by Sophos Rapid Response, 2020-2021 Conti infection rate portends the expansion of the RaaS model Conti 16%

All other ransomware (35 di erent "families") 43% REvil 15%

Ryuk 9%

Ragnarok 4% LockBit 4%

Black Kingdom 3% DarkSide 3% Maze 3%

SOPHOS

Figure 9.27  Notable ransomware families 2020–2021 [56].

The graph mentioned in Figure 9.28 illustrates the predicted damage of ransomware from the period 2015–2021. According to Cybercrime Magazine, the global costs were expected to amount to $20 billion, which showed a significant increase from $11 billion in 2019. The Table 9.2 gives a comparative of various ransom-ware strains discussing about their type, encryption method, attack vectors, notable victims, etc. It includes information such as the date of first appearance, an encryption method used, attack methods, ransom amounts demanded, and special victims of each ransomware strain. Predicted Ransomware Damages 2015-2021 $25,000,000,000 $20B

$20,000,000,000

$15,000,000,000 $11.5B $10,000,000,000

$8B $5B

$5,000,000,000 $325M $0

2015

2017

2018

2019

Figure 9.28  Ransomware damage prediction from 2015–2021 [57].

2021

202  Advances in Aerial Sensing and Imaging

Table 9.2  Comparative analysis of famous ransomware strains. Ransomware strain

Date of first appearance

Encryption method

Attack vector

Ransom amount

AIDS Trojan Horse

December 1989

Notable victims

Symmetric encryption

Trojan

$189–$378

Healthcare sector

GPCoder

2005–2008

RSA1024+AES-256

Malicious emails and attachments

$100–$200

The healthcare sector, other governmental organizations

Archievus

2006

RSA-1024

Malicious emails and attachments

$300–$500

Various individuals and organizations

CryptoLocker

September 2013

RSA-2048

Exploit kits, botnets, email attachments

$300–$3000

Police departments, small businesses

CryptoWall

November 2013

RSA-2048

Spam campaigns, botnets, exploit kits, malvertising

$200–$2000

Healthcare providers, law firms (Continued)

Ransomware and Unmanned Aerial Vehicle Attack  203

Table 9.2  Comparative analysis of famous ransomware strains. (Continued) Ransomware strain

Date of first appearance

Encryption method

Attack vector

Ransom amount

TeslaCrypt

February 2015

AES-256, RSA-2048

Spam campaigns, botnets, emails

$500–$1000

Game developers, educational institutions

Locky

February 2016

RSA-2048 + AES128 cypher with ECB

Spam campaigns, botnets, emails

$400–$800

Hollywood Presbyterian Medical Center, Lukoil

WannaCry

May 2017

RSA 2048, AES-128

EternalBlue Exploit

$300–$600

NHS, FedEx, Renault

NotPetya

June 2017

MFT Encryption

Hijacked software update mechanism

N/A

Maersk, Merck, FedEx

Ryuk

August 2018

RSA-2048 + AES-256

Spear-phishing

$100,000– $500,000

Universal Health Services, Eqip Global

Notable victims

(Continued)

204  Advances in Aerial Sensing and Imaging

Table 9.2  Comparative analysis of famous ransomware strains. (Continued) Ransomware strain

Date of first appearance

Encryption method

Attack vector

Ransom amount

Notable victims

REvil/Sodinokibi

April 2019

AES-256, RSA-2048

Malware, RDP attacks

$100,000– $10,000,000

Kaseya, JBS, Travelex

Egregor

September 2020

ChaCha20, RSA

The botnet, phishing emails

Depends on the victim. Generally, in the range of $10,000– $1,000,000

Ubisoft, Barnes & Noble, Kmart, Cencosud

Darkside

August 2020

Salsa20, RSA-1024

TOR-RDP

Generally, in the range of $10,000– $1,000,000

Colonial Pipeline, Toshiba Corp, Brenntag

LockBit

September 2019

RSA and AES

RDP and phishing emails

Generally, in the range of $10,000– $1,000,000

Foxxcon, Accenture, Continental, Press Trust of India (Continued)

Ransomware and Unmanned Aerial Vehicle Attack  205

Table 9.2  Comparative analysis of famous ransomware strains. (Continued) Ransomware strain

Date of first appearance

Encryption method

Hive

June 2021

Generates two key sets in memory, uses them to encrypt files and then encrypts and writes the sets to the root of the drive it encrypts

Attack vector

Ransom amount

Remote Code Execution, spear-phishing

Generated over $100 million in revenue

Notable victims Healthcare, Information Technology, Real Estate, Education

206  Advances in Aerial Sensing and Imaging

9.9 Conclusion and Future Scope According to Cybercrime magazine, the damages due to ransomware were predicted to reach from $325 million in 2015 to $5 billion in 2017, which showed 15 times increase. By 2021, the financial damage was expected to amount to $20 billion, 57 times more than what was estimated for 2015 [58]. Recent ransomware attacks indicate that cyber-criminals and hackers will not bring their activities to a standstill. The current situation shows that the upcoming decade is ever so susceptible to ransomware-led cyber-attacks as hackers refine and intensify attacks. It is estimated that ransomware attacks will cost victims around $265 billion annually by 2031, with new attacks every 2 seconds as hackers refine their malware payloads and extortion techniques [58]. Businesses and organizations that show leniency towards proper ransomware protection are the most vulnerable and easy targets for cyber-criminals. According to a cyber-attack statistic mentioned by SafeatLast, only 34% of global organizations are adequately prepared to handle an advanced cyber-attack [59–61], and Asia-Pacific companies can lose up to $2 trillion to cybercrimes. In conclusion, this paper has analyzed ransomware’s history, types, special attacks, and impact on victims. The study has highlighted the various techniques cybercriminals use to launch ransomware attacks and the financial and reputational costs of recovery for victims. Additionally, the paper has discussed different mitigation strategies available to organizations and individuals, emphasizing the importance of proactive measures such as employee education, backups, and incident response planning. As the ransomware threat continues to evolve, cybercriminals adapt their tactics to evade detection and maximize profits. Therefore, organizations and institutions must stay informed about emerging threats and continuously update security measures. The paper has also shown how Unmanned Aerial Vehicles are susceptible to various cyber-attacks, along with an experimental demonstration of a Wi-Fi attack on Ryze Tello UAV. The security threats present in UAVs and their potential impact on the security of these systems and the data they transmit are a matter of concern that needs to be addressed. The study emphasizes the importance of raising awareness and educating users about the potential risks associated with ransomware and other cyber threats to UAVs.

Ransomware and Unmanned Aerial Vehicle Attack  207

References 1. Anand, A., Trivedi, N.K., Gautam, V., Tiwari, R.G., Witarsyah, D., Misra, A., Applications of Internet of Things(IoT) in agriculture: The need and implementation. International Conference Advancement in Data Science, E-learning and Information Systems (ICADEIS), Bandung, Indonesia, pp. 01–05, 2022. 2. Agarwal, A.K., Jindal, R.K., Chaudhary, D., Tiwari, R.G., Sharma, M., Security and privacy concerns in the Internet of Things: A comprehensive review. 11th International Conference on System Modeling & Advancement in Research Trends (SMART), Moradabad, India, pp. 254–259, 2022. 3. Prevezianou, M.F., WannaCry as a creeping crisis, in: Understanding the Creeping Crisis, A. Boin, M. Ekengren, M. Rhinard (Eds.), pp. 37–50, 2021. 4. Chesti, I.A., Humayun, M., Sama, N.U., Jhanjhi, N., Evolution, mitigation, and prevention of ransomware. 2nd International Conference on Computer and Information Sciences (ICCIS), pp. 1–6, 2020. 5. Norman’s, J., Dr Joseph Popp writes the AIDS trojan horse, the first known ransomware cyber attack, https://www.historyofinformation.com/detail. php?id=5135.Last Accessed: March 12, 2023. 6. Maurya, A.K., Kumar, N., Agarwal, A., Khan, R.A., Ransomware evolution, target and safety measures. Int. J. Comput. Sci. Eng., 6, 80–85, 2018. 7. Thomas, J., Exploring methods to empower users to identify and address spear phishing attacks to combat ransomware and identity theft, Master of Science in Information Technology Information Assurance and Security, American Military University, Charles Town, West Virginia, 2018. 8. Emm, D., Cracking the code: The history of Gpcode. Comput. Fraud Secur., 9, 15–17, 2008. 9. Alenezi, M.N., Alabdulrazzaq, H., Alshaher, A.A., Alkharang, M., Evolution of malware threats and techniques: A review. Int. J. Commun. Netw. Inf. Secur. (IJCNIS), 12, 3, 326–337, 2020. 10. Hamad, M. and Eleyan, D., Survey on ransomware evolution, prevention, and mitigation. Int. J. Sci. Technol. Res., 10, 02, 271–280, 2021. 11. Kalaimannan, E., John, S.K., DuBose, T., Pinto, A., Influences on ransomware’s evolution and predictions for the future challenges. J. Cyber Secur. Technol., 1, 23–31, 2017. 12. Imaji, A., Ransomware attacks: Critical analysis, threats, and prevention methods, Fort Hays State University, Fort Hays State University, Hays, Kansas, 2019. 13. O’Kane, P., Sezer, S., Carlin, D., Evolution of ransomware. IET Networks, 7, 5, 321–327, 2018. 14. HandWiki contributors, Engineering: CryptoLocker, 2023, https://handwiki. org/wiki/index.php?title=Engineering:CryptoLocker&oldid=2528313. Last Accessed: March 12, 2023.

208  Advances in Aerial Sensing and Imaging 15. Kumar CG, N. and Pande, S., A study on ransomware and its effect on India and rest of the world. Int. J. Eng. Res. Technol. (IJERT) Nciccnda, 5, 22, 1–6, 2017. 16. Hassell, J., Cryptolocker: How to avoid getting infected and what to do if you are, Computerworld, https://www.computerworld.com/article/2485214/cryptolocker-how-to-avoid-getting-infected-and-what-to-do-if-you-are.html,2013. Last Accessed: March 12, 2023. 17. Kyurkchiev, N., Iliev, A., Rahnev, A., Terzieva, T., A new analysis of cryptolockerransomware and welchia worm propagation behavior. Some applications. III. Commun. Appl. Anal., 23, 359–382, 2019. 18. StatMuse, Bitcoin price 2013: StatMuse money, 2013, https://www.statmuse. com/money/ask/bitcoin+price+2013. Last Accessed: March 12, 2023. 19. Secureworks, Dell secureworks counter threat unit™ threat intelligence. CryptoWall ransomware threat analysis, Search Secureworks, 2014, https:// www.secureworks.com/research/cryptowall-ransomware. Last Accessed: March 13, 2023. 20. Groot, J.D., A history of ransomware attacks: The biggest and worst ransomware attacks of all time, Digital Guardian, 2022, https://www.digitalguardian.com/ blog/history-ransomware-attacks-biggest-and-worst-ransomware-­attacksall-time. Last Accessed: March 13, 2023. 21. Adamov, A. and Carlsson, A., The state of ransomware. Trends and mitigation techniques. IEEE East-West Design & Test Symposium (EWDTS), pp. 1–8, 2017. 22. Krishna, P., An overview on teslacrypt virus: Virus variant, versions, tools and removal approach. A monthly. J. Comput. Sci. Inf. Technol., 6, 29–33, 2017. 23. Abrams, L., TeslaCrypt 3.0 released with modified algorithm and.XXX, TTT, and .micro file extensions, BleepingComputer, 2016, https://www.bleepingcomputer.com/news/security/teslacrypt-3-0-released-with-new-encryption-algorithm-and-xxx-file-extensions/. Last Accessed: March 13, 2023. 24. Abrams, L., TeslaCrypt shuts down and releases master decryption key, BleepingComputer, New York, United States of America, 2016, https://www. bleepingcomputer.com/news/security/­teslacrypt-shuts-down-and-releasesmaster-decryption-key/. Last Accessed: March 13, 2023. 25. Constantin, L., Decryption tool available for TeslaCrypt ransomware that targets games, PCWorld, 225–229, 2015, PCWorld.com. Last Accessed: March 13, 2023. 26. Vayuputra, K.S. and Kiran, K.V.D., Study and analyze the locky ransomware using malware analysis techniques. Int. J. Eng. Technol., 7, 2018. 27. Krithika, N., Detecting and preventing LOCKY attack by Hillstone in GFW and Emisoft anti-malware. Int. J. Eng. Res. Technol. (IJERT), 5, 1–5, 2017. 28. Ang, M.C., NECURS evades detection via internet shortcut file, 2018, https:// www.trendmicro.com/en_us/research/18/d/necurs-evolves-to-evade-spamdetection-via-internet-shortcut-file.html. Last Accessed: March 13, 2023.

Ransomware and Unmanned Aerial Vehicle Attack  209 29. Trendmicro, Kansas Hospital hit by ransomware, extorted twiceKansas Hospital hit by ransomware, extorted twice - noticias de seguridad, 2016, https://www.trendmicro.com/vinfo/mx/security/news/cybercrime-and-digital-threats/kansas-hospital-hit-by-ransomware-extorted-twice. Last Accessed: March 14, 2023. 30. Akbanov, M., Vassilakis, V., Logothetis, M.D., WannaCry ransomware: Analysis of infection, persistence, recovery prevention and propagation mechanisms. J. Telecommun. Inf. Technol., 1, 113–124, 2019. 31. TrendMicro, WannaCry/Wcry ransomware: How to defend against it, TrendMicro, 2017, https://success.trendmicro.com/dcx/s/solution/1117391preventing-wannacry-wcry-ransomware-attacks-using-trend-microproducts?language=en_US&sfdcIFrameOrigin=null, Last Accessed: March 15, 2023. 32. National Audit Office, Investigation: WannaCry cyber-attack and the NHS national audit office (NAO) report, National Audit Office (NAO), 2017, https://www.nao.org.uk/reports/investigation-wannacry-cyber-attackand-the-nhs/ Last Accessed: March 14, 2023. 33. Arabnews, North Korea denies role in WannaCry ransomware attack, Arab News, 2017, https://www.arabnews.com/node/1212316/session_trace/­aggregate Last Accessed: March 15, 2023. 34. The Justice Department, US charges Russian GRU officers with international hacking and related influence and disinformation operations, The United States Department of Justice, 2018, https://www.justice.gov/opa/pr/us-chargesrussian-gru-officers-international-hacking-and-related-­influence-and/. Last Accessed: March 15, 2023. 35. National Cyber Security Centre, Russian military ‘almost certainly’ responsible for destructive 2017 cyber-attack, NCSC, 2018, https://www.ncsc.gov. uk/news/russian-military-almost-certainly-responsible-destructive-2017-­ cyber-attack. Last Accessed: March 15, 2023. 36. Greenberg, A., The untold story of NotPetya, the most devastating cyberattack in history, Wired, New York, United States of America, 2018. 37. Fayi, S., What Petya/NotPetya ransomware is and what its remidiations are, Information Technology – New Generations, pp. 93–100, 2018. 38. Eugenio, D., A targeted campaign break-down - ryuk ransomware, Check Point Research, 2018, https://research.checkpoint.com/2018/ryuk-ransomware-targeted-campaign-break/. Last Accessed: March 15, 2022. 39. Maurya, A.R., Study of ryuk ransomware attack. GRD Journals- Glob. Res. Dev. J. Eng., 4, 48–50, 2019. 40. Elshinbary, A., Deep analysis of ryuk ransomware, Github, 2020, https:// n1ght-w0lf.github.io/malware%20analysis/ryuk-ransomware/. Last Accessed: March 15, 2023. 41. Ilascu, I., How ryuk ransomware operators made $34 million from one victim, BleepingComputer, 2020, https://www.bleepingcomputer.com/news/

210  Advances in Aerial Sensing and Imaging security/how-ryuk-ransomware-operators-made-34-million-from-one-­ victim/. Last Accessed: March 15, 2023. 42. Kuraku, S. and Kalla, D., Emotet malware – a banking credential stealer. J. Comput. Eng. (IOSR-JCE), 22, 4, 31–40, 2020. 43. Jarjoui, S., Murimi, R., Murimi, R., Hold my beer: A case study of how ransomware affected an australian beverage company. International Conference on Cyber Situational Awareness, Data Analytics and Assessment (Cyber SA), IEEE, pp. 1–6, 2021. 44. Seals, T., Revil gang promises a big video-game hit; maze gang shuts down, Threatpost English Global threatpostcom, 2020, https://threatpost.com/ revil-video-game-hit-revenue/160743/. Last Accessed: March 16, 2023. 45. Mathews, L., Acer faced with ransom up to $100 million after hackers breach network, Forbes, Forbes Magazine, 2021, Last Accessed: March 18, 2023. 46. Collier, K., Meat supplier JBS paid ransomware hackers $11 million, CNBC, 2021, https://www.cnbc.com/2021/06/09/jbs-paid-11-million-in-responseto-ransomware-attack-.html. Last Accessed: March 16, 2023. 47. Sharma, A. and Gupta, S., Cyber crimes during COVID – 19 pandemic in India and world, vol. 29, Supremo Amicus, HeinOnline, 2022. 48. Arista Networks, Threat hunting for REvil ransomware, 20 July 2022, https:// arista.my.site.com/AristaCommunity/s/article/Threat-Hunting-for-REvilRansomware. Last Accessed: March 16, 2022. 49. Arntz, P., Ransomware disappears after tor services hijacked, Malwarebytes, 2021, Last Accessed: March 16, 2022. 50. BBC, Ransomware key to unlocking customer data from revil attack, BBC News, 23 July 2021. 51. Europol, Five affiliates to Sodinokibi/Revil unplugged, Europol, 2021, https:// www.europol.europa.eu/media-press/newsroom/news/five-­affiliates-tosodinokibi/revil-unplugged. Last Accessed: March: 16, 2022. 52. Ilascu, I., Russia arrests Revil ransomware gang members, seize $6.6 million, BleepingComputer, 2022, https://www.bleepingcomputer.com/news/­ security/russia-arrests-revil-ransomware-gang-members-seize-66-million/. Last Accessed: March 18, 2023. 53. Tsao, K.Y., Girdler, T., Vassilakis, V.G., A survey of cyber security threats and solutions for UAV communications and flying ad-hoc networks. Ad Hoc Netw., 133, 2, 70–85, 2022. 54. Rubbestad, G. and Söderqvist, W., Hacking a wi-fi based drone, DIVA, School of Electrical Engineering and Computer Science (EECS), pp. 1–48, 2021, Last Accessed: March 17, 2022. 55. Gatlan, S., FBI, Ransomware hit 649 critical infrastructure orgs in 2021, 2022, https://www.bleepingcomputer.com/news/security/fbi-­ransomware-hit649-critical-infrastructure-orgs-in-2021/. Last Accessed: March 17, 2023. 56. Brandt, A., Sophos releases the 2022 threat report, 2021, https://news.sophos. com/en-us/2021/11/09/2022-threat-report/. Last Accessed: March 17, 2023.

Ransomware and Unmanned Aerial Vehicle Attack  211 57. Crane, C., 20 Ransomware statistics you’re powerless to resist reading, 2020, https://securityboulevard.com/2020/02/20-ransomware-statistics-yourepowerless-to-resist-reading/. Last Accessed: March 18, 2023. 58. Freeze, D., Global ransomware damage costs predicted to exceed $265 billion by 2031, Cybercrime Magazine, 2022, https://cybersecurityventures.com/ global-ransomware-damage-costs-predicted-to-reach-250-billion-usd-by2031/, Last Accessed: March 18, 2023. 59. Milenkovic, D., 30+ alarming cyber attack statistics & numbers, Safe at Last, 2022, https://safeatlast.co/blog/cyber-attack-statistics/. Last Accessed: March 18, 2023. 60. Bhola, Srivastava, S., Noonia, A., Sharma, B., Narang, S.K., A status quo of machine learning algorithms in smart agricultural systems employing IoTbased WSN: Trends, challenges and futuristic competences, in: Machine Intelligence, Big Data Analytics, and IoT in Image Processing: Practical Applications, pp. 177–195, 2023. 61. Abhishek, and Singh, S., Visualization and modeling of high dimensional cancerous gene expression dataset. J. Inf. Knowl. Manage., 18, 1, 1–22, 2019.

10 A Framework for Detection of Overall Emotional Score of an Event from the Images Captured by a Drone P.V.V.S. Srinivas1*, Dhiren Dommeti1, Pragnyaban Mishra2 and T.K. Rama Krishna Rao1 Koneru Lakshmaiah Education Foundation (KLEF), Vaddeswaram Guntur, AP, India 2 Gandhi Institute of Technology and Management (GITAM), Visakhapatnam, AP, India 1

Abstract

In the past few years, drones have become a big part of many businesses and government agencies doing their jobs. They have also helped several industries break through areas where they were stuck or falling behind. From making quick deliveries during peak times to scanning a military base that people can’t get to, drones are proving to be immensely useful in places where people can’t get to or can’t get the task done efficiently. Production costs and workloads can be reduced; work efficiency can be increased, accuracy can be improvised, etc., and can be obtained by using drones. Human emotions can be inferred from facial expressions, a form of nonverbal communication. Facial Expression Recognition analyses the feelings conveyed by various sources, such as images and movies. It belongs to the “affective computing” family of technologies, a multidisciplinary field of study on computers’ abilities to perceive and interpret human emotions. The basic steps followed by facial expression recognition systems are to detect faces from the image, identify their expressions, and finally classify them into respective classes. Convolutional neural networks play a vital role in designing efficient FER systems. They had applications in healthcare, education, workplace safety, public service, and many other fields. The benefits of drone and FER systems can be combined to develop a model that calculates the overall emotional score of a scenario of an event where a group *Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (213–244) © 2024 Scrivener Publishing LLC

213

214  Advances in Aerial Sensing and Imaging of people present by which one can know what is happening in the event, and precautionary steps can be taken if necessary. By taking the support of drones, the images of an event can be easily captured, and the facial images from the image captured can be easily obtained by using object detection techniques; the obtained facial images can be given to a facial emotion recognition system which can calculate the overall emotional score of the input images. CNN is vital in developing object detection and facial emotion recognition systems. Keywords:  Drone, emotions, facial emotion recognition system, emotional score, object detection, CNN

10.1 Introduction Today, Convolutional Neural Networks (CNNs) are used to analyze drone images. A CNN is a type of Artificial Intelligence (AI) that detects and classifies image patterns. By leveraging the power of AI, drones can be used to identify objects in an image and calculate an emotional score based on the things remembered. This information can then be used by teams involved in search and rescue operations or to inform decisions related to surveillance. In addition to object recognition, CNNs can detect anomalies in an image that may indicate a potential problem. For instance, a drone could see damaged buildings or roads in a disaster zone. By doing this, teams can quickly determine the severity of the situation and take the appropriate action. Furthermore, CNNs can be used to analyze weather patterns and forecast changes in weather conditions. This can help teams in both disaster relief operations and military operations to make decisions quickly and accurately. CNNs are becoming increasingly important in image analysis and are used in various applications. In particular, they are helping teams in search and rescue operations, surveillance and disaster relief to analyze images and make decisions quickly.

10.1.1 Need for Emotion Recognition Emotion recognition is essential to understand people and the world around us better. It can provide insight into how people feel and can be used to create more meaningful connections between individuals and their environment. Emotion recognition technology can detect emotions in realtime, allowing for early interventions and more effective communication. Additionally, emotion recognition can help marketers better understand their customers, allowing them to design products, advertisements, and campaigns better to increase engagement. Furthermore, it can also be used

Detection of Overall Emotional Score  215 to detect mental health issues more quickly and accurately, leading to better outcomes for those affected. In short, emotion recognition can be used in various contexts to improve people’s lives worldwide. Emotional score calculation using CNNs is a powerful tool for analyzing drone images. CNNs, or convolutional neural networks, are especially effective in diagnosing and recognizing patterns in pictures and can be used to calculate an emotional score of an image accurately. This score is based on the facial expressions detected in the photo, providing valuable insight into the subject’s emotional state. By leveraging the power of CNNs and drone technology, organizations can better understand the emotional state of their customers or employees cost-effectively and efficiently.

10.1.2 Applications of Drones in Deep Learning Drones are becoming increasingly popular for capturing images and videos for different applications. Drone image capturing is particularly useful for surveying large areas or capturing unique angles and perspectives. Drones can take stunning aerial pictures and videos of landscapes, events and landmarks. They can fly higher than other cameras and provide unique perspectives that can’t be achieved from the ground. Drones are becoming increasingly sophisticated and can now capture higher-resolution images and videos. This allows for more detailed pictures and videos to be taken. Furthermore, drones can also capture high-definition 3D photos and videos, allowing more accurate data to be collected. Moreover, drones can capture images and videos in places that are out of reach for humans, such as remote or dangerous locations. Using drones, pictures and videos can be captured safely without endangering personnel. Drones are becoming increasingly popular for various applications and are continuing to be used in new and innovative ways. Using drones to capture images and videos makes it possible to get stunning aerial photos and videos and access previously unreachable places. Drones have become increasingly popular for use in deep learning applications. Drones are being used to capture images and video, which can be analyzed for visual object recognition, facial recognition, or other analysis. In addition, drones can provide input on physical environments that are otherwise difficult to access. For example, drones can collect data in inaccessible or hazardous environments such as forests, the sea, or even the inside of a factory. One fascinating application of drones in deep learning is using drones to calculate emotional scores from images. This involves using a convolutional neural network (CNN) to analyze the photos taken by the drone and to determine the dynamic score of the subject in the image. This technology has the potential to be used in

216  Advances in Aerial Sensing and Imaging a variety of contexts, including social media, marketing research, and even medical studies. Beyond emotion recognition, drones can also collect data for various deep learning tasks such as automated navigation, object recognition, and terrain analysis. In addition, due to the ease of deployment and the ability to collect data from difficult-to-reach places, drones can be used to collect data for automated machine-learning applications. With the rise of drone technology, there have been many innovations, including using a Convolutional Neural Network (CNN) to calculate an emotional score from images captured by a drone. By feeding ideas into a CNN, a neural network can recognize and categorize objects and assign a score to the overall emotion of a scene or image. This process allows businesses to obtain valuable insights into the emotional state of customers and employees and understand how people respond to their products, services, and environment. The dynamic score obtained from CNN can be used to identify areas of customer satisfaction, identify opportunities for improvement, and track trends over time. Additionally, this data can be used to inform decisions related to marketing, product development, and customer service, providing businesses with valuable insights into the overall emotional state of their customers. Using a CNN to calculate a dynamic score from images captured by a drone is a new and exciting application of this technology. It has the potential to revolutionize how businesses comprehend the emotional state of their customers. By leveraging this technology, companies can better understand customer sentiment and gain valuable insight into the customer experience.

10.2 Literature Review Ref. [1] presents an innovative approach to using emotion recognition technology to enable social interactions between humans and drones. The authors explore the potential benefits of this approach and the challenges that need to be addressed to make it a reality. The authors describe their research in this area, which involved developing a system combining facial recognition and physiological signals to recognize human emotions and respond appropriately. They describe the results of their experiments, which showed that their system could accurately recognize emotions in human subjects. However, it is essential to note that this paper is limited in scope, focusing primarily on the technical aspects of emotion recognition and social interaction with drones. Future research could explore this technology’s ethical and social implications, including privacy, autonomy, and the impact on social relationships. Instead of anthropomorphizing

Detection of Overall Emotional Score  217 the personal drone, they proposed a mapping function for drone motions correlating to five different human emotional states. Ref. [2] provided a user study (N=20) demonstrating how accurately three distinct emotional states could be identified. They concluded drone interaction methods and feedback tactics based on the direction and speed of the drone’s flight. To support people in crisis or as a guide, drones can be used as personal companions, and this active method was posited by Schneegass et al. [3]. To make these scenarios possible, we must offer appropriate interactions mediated by a phone or remote control [4] as well as by direct voice or gestural control [5, 6]. While there are many potential inputs, there are few feedback strategies for HDI. Previous research examined the addition of LEDs around a quadcopter to illustrate the direction [7]. The Daedalus [8] drone was enhanced with head movement, eye colour, and propeller noise to represent different emotional states. In contrast, to interact, Sharma et al. [9] altered a drone’s flight path with the Laban Effort System [10]. In the latter piece, the drone executes set motions based on four factors: space, weight, time, and flow. Participants could distinguish between numerous states along valence and arousal for all criteria except Flow. This work suggests that individuals can differentiate between various aspects of the drone’s flight path. To enable commands to be transmitted to the drone, [11] D. Ramos et al. envisioned a decision-making mechanism that assesses the drone operator’s emotional state. The system can determine whether the operator is in the appropriate emotional state for controlling the drone by creating a digital twin of the controller capable of classifying emotional states at both the cognitive and visual levels. A ROS2 client node computes and transmits the necessary data to the drone. According to preliminary evaluation findings, the digital twin can recognize emotions and mood changes in a real-time environment. The system may offer a trustworthy and secure method of validating drone instructions through the mind. In their study, [12] aimed to explore natural emotional human feedback as a reward signal for training artificial agents through interactive human-in-the-loop reinforcement learning. To accomplish this, they created an experimental scenario based on animal training. Human participants taught an emulated drone agent desired command-action mappings by providing emotional feedback on the drone’s action selections. The authors conducted an empirical study and analysis, which provided proof-of-concept evidence that human facial expressions of emotion can serve as a reward signal in interactive learning settings. Their findings contribute to developing more intuitive and naturalistic reinforcement learning techniques accessible to non-expert users. Similarly, In their work, [13] Sumers et al. developed a learning strategy

218  Advances in Aerial Sensing and Imaging that employs inverse reinforcement learning to discern the latent reward function of humans using unconstrained textual feedback collected during human-human collaborative gameplay. This approach aims to capture more naturalistic and unrestricted human feedback. In their research, [14] presented a technique for enhancing bots’ emotional intelligence and appealed by closing the gap in their comprehension of human moods in real-time. This approach involves equipping bots with the ability to recognize and simulate appropriate emotional responses to human emotions. The authors implemented this method using OpenCV and the Fisherfaces algorithm, which includes both PCA (principal component analysis) and LDA (linear discriminant analysis) algorithms. The authors of [15] introduced a strategy called physical-digital twinning to enhance comprehension and handling of the physical task (PT) in human-robot interaction scenarios through an interdisciplinary approach informed by neuroeconomics. The authors of [16, 17] constructed a realtime model capable of predicting human emotions from images. Their process involves a convolutional neural network with significantly reduced parameters compared to Vanilla CNN and the latest state-of-the-art research, resulting in a 90× and 50× reduction, respectively. The network was rigorously tested on eight distinct datasets, including a custom dataset created by the authors that contained a variety of angles and faces. The network achieved an accuracy of 74%, which represents improved accuracy over the state-of-the-art with a corresponding reduction in computational complexity. FER research, as well as a comprehensive taxonomy of FER, is covered. Focus is on both edge- as well as cloud-based FER techniques. There is a thorough discussion of journals and FER applications. The study primarily concentrates on FER literature and correctly classifies FER methodologies as DL, CL, and HL. FER’s current issues are discussed, along with suggestions for upcoming work. [18] suggested a technique in which an SVM was fed with extracted ORB features. MMI and JAFFE datasets were investigated. [19] the study examined occlusion problems, developed new techniques, and investigated data creation for FER systems. It also discussed related challenges. Generally, partial occlusion is only taken into account. To describe FER steps, workflow methodology is not offered. On FER surveys, there is no mainstream comparative study. The critical steps in [20] are Data preprocessing, extraction of features, and later classification. The majority of common challenges need to be covered. Furthermore, there are no guidelines or suggestions for further study. The Viola-Jones algorithm is still

Detection of Overall Emotional Score  219 widely used to detect faces in real-time, but it is constrained when a look is misaligned, covered by a garment, or not correctly oriented. Therefore, deep learning algorithms like SSD [21], FaceNet [22], VGG-Face [23], and R-CNN [24] have been developed to avoid such issues in conventional methods and enhance face detection algorithms. R-CNN, first developed for object detection, stands out for its capacity to perform face detection tasks with high CNN accuracy. [25] developed a platform for FER that utilizes multiple channels, multimodal, and electro-encephalogram physiological signals. [26] TV shows were taken into consideration for facial expression analysis of human behaviour. The authors used the Viola-Jones and Kanade-Lucas-Tomasi (KLT) algorithms to detect and track faces. Using an SVM model, they retrieved HOG features and categorized the expression. [27] To address the class imbalance and boost discrimination expression power, a monitored objective AdaReg loss and re-weighting category was proposed. [28] Premised on a real dataset for the classification of facial expressions, a DNN was suggested. [29] To recognize human facial expressions in an innovative TV environment, a DNN based on a webcam was proposed. [30] A deep learning technique that incorporates relativity learning was suggested.  A mapping from the actual photographs  into a Euclidean space, in which relative distances stood for a metric of facial expression similarity, was understood by this model. [31] Predicated on LSTM-RNN as well as SVM models, FER was carried out. [32] To accurately detect human facial expressions, a deep CNN has been introduced. [33] LBP was used to extract features from images. To classify FER depending on the frame features, GRNN was used. [35] To categorize facial expressions, an ANN model was presented. The performance of the ANN was improved using a points/landmark technique. [34] To detect facial expressions, various DNNs were introduced, and their performance was combined. SURF [36], Naive Bayes [37], HOG [38], SIFT [39] and SVM [40] are examples of traditional learning strategies. Before feature extraction, conventional practises employ hand-crafted feature engineering techniques, like preprocessing and data augmentation. In [41], a mapped LBP feature for illumination-invariant FER was suggested. SIFT [42] features resistance to image scaling, and rotation is used for multiview FER tasks. The network’s performance is improved by combining multiple textures, orientations, and colour descriptors and using them as inputs [43, 44]. Similarly, the partbased representation extracts the features from an image by removing noncritical portions and focusing on the crucial, task-sensitive regions.

220  Advances in Aerial Sensing and Imaging

10.3 Proposed Work The research will be conducted in the following phases; initially, Data Collection: The first phase will involve collecting a large dataset of images captured by drones at various events. Human experts will annotate these images with emotional scores to serve as ground truth labels for the machine learning algorithms. The collected images will be pre-processed in the preprocessing phase to remove noise and enhance image quality. Techniques such as image resizing, colour correction, and contrast enhancement will be used to improve the quality of the photos. In the Feature Extraction phase, features such as facial expressions and verbal cues will be extracted from the images. This will be done using techniques such as facial recognition and pose estimation. In the Emotional Score Detection phase, machine learning algorithms such as convolutional neural networks will be trained on the extracted features to detect the emotional score of the event. Performance Evaluation: The performance of the proposed framework will be evaluated using various metrics such as accuracy, precision, recall, and F1 score. The results will be compared with existing approaches in the literature. The proposed framework is expected to provide a more accurate and reliable method for detecting the overall emotional score of an event from images captured by a drone. The framework can have potential applications in event management, marketing, and psychology research. The images captured by the drone are processed to detect and extract faces, which are then saved in a separate folder. The proposed model is trained using two datasets - FER 2013 and CREMAD - separately. The extracted faces are then used to evaluate the model’s performance and calculate the overall emotional score of the event. Figure 10.1 portrays a flowchart, and the procedure starts with images collected from the drone. The extracted Facial Images from the composed Image Using the Facial Image Extraction Algorithm is used. FERCNN model for facial emotion recognition is developed. The model is separately evaluated with FER 2013 and CREMAD. The model is trained and evaluated using the dataset. The overall emotional score using the estimated model for the facial images collected from the drone. If there are any more images, the procedure retreats to the Initial step and hence it ends.

10.3.1 Extraction of Images from a Drone Set of instructions for detecting and extracting faces from input images using OpenCV library in Python. The first step involves importing the

Detection of Overall Emotional Score  221 Start

Collect the Images from Drone

Extract the Facial Images from the Collected Image Using Facial Image Extraction Algorithm Used Collect all the Extracted Facial Images

Wait for the FERCNN Model Training

Yes

Develop a FERCNN Model for Facial Emotion Recognition

FER 2013

Selet how The Model Should be Evaluated

CREMAD

Train and Evaluate the Model Using CREMAD Data Set

Train and Evaluate the Model Using FER 2013 Data Set

Calculate the Overall Emotional Score Using the Evaluated Model for The Facial Images Collected From Drone

Yes

Are there any more Drone Images No Stop

Figure 10.1  Flowchart of emotion recognition.

required libraries. The base directory is set in the second step. A loop is run for five iterations, and two guides are created inside the circle to store the extracted faces and images with rectangles using the os.mkdir() function. The input image is read using cv2.imread() in the fifth step. The input image is then converted to grayscale using the cv2.cvtColor() function, and the result is stored in the variable ‘grey’ in the sixth

222  Advances in Aerial Sensing and Imaging step. In the seventh step, the face cascade classifier is loaded using cv2. CascadeClassifier(). In the eighth step, the detect MultiScale() function of face_cascade is used to detect faces in the grayscale image. The number of detected faces is printed in the ninth step. A loop is run to draw rectangles around the detected faces, which is done using the cv2.rectangle() function in the tenth and eleventh steps. The face is then extracted from the input image using sub_img = img[y10:y+h+10,x-10:x+w+10] in the twelfth step, and it is saved using the cv2. imwrite() function in the Extracted directory in the thirteenth step. The input image with rectangles is protected using the cv2.imwrite() function in the Rectangle directory in the fourteenth step. Finally, the cv2.destroyAllWindows() function destroys all the windows in the fifteenth step.

10.3.2 Proposed CNN Model The proposed model is a Sequential for a Convolutional Neural Network (CNN) for image classification. The model comprises 13 layers, including six convolutional layers, three max-pooling layers, one flattened layer, two fully connected (dense) layers, and one dropout layer. The model takes as input images of size 48x48 with three colour channels (RGB) and categorizes seven different emotions (anger, disgust, fear, happiness, sadness, surprise, and neutral). The first layer is a 2D convolutional layer with 64 filters, a kernel size of 3x3, and a stride of 1. The second layer is another 2D convolutional layer with the exact specifications of the first layer. The third layer is a batch normalization layer that helps with regularization and accelerates the training process. The fourth layer is a max pooling layer with a pool size of 2x2 and a stride of 2. The subsequent three layers follow the same pattern as the first four but with 128 filters in the convolutional layers. The seventh layer is a flattened layer that flattens the output from the previous layer into a 1D array. The eighth layer is a fully connected (dense) layer with 128 units. The ninth layer is another batch normalization layer, followed by an activation layer (ReLU) and a dropout layer to prevent overfitting. The tenth layer is another fully connected (dense) layer, with seven units corresponding to the number of emotions to be classified. The eleventh layer is an activation layer (softmax), which outputs a probability distribution over the seven emotions. The total number of parameters in the model is 2,787,015, with 2,785,863 trainable parameters and 1,152 non-trainable parameters. Figure 10.2 gives the proposed CNN architecture that will be used to train and evaluate the model.

Detection of Overall Emotional Score  223

Figure 10.2  Proposed model architecture.

10.4 Experimentation and Results The models were implemented in Kaggle with a GPU P100 accelerator. Kaggle is a powerful platform for data science professionals and enthusiasts alike. It provides an excellent opportunity to share, collaborate and compete on data science projects. With Kaggle, you can find datasets, build models, create submissions, and even win prizes. The advantages of using

224  Advances in Aerial Sensing and Imaging Kaggle are numerous. For starters, Kaggle provides access to various datasets for data scientists to work with. Furthermore, it includes a platform for competing with others on data science projects. The competitions offer an exciting way to compare your work with other data scientists and can help to improve your skills. Kaggle also provides a great learning environment. It has a large community of experienced data scientists who can provide helpful insights to help you succeed. Plenty of tutorials and discussion forums also help you learn and get inspired. On Kaggle, you can find a variety of datasets from various sources, so you can find something you are interested in and work on it with the help of the community. Finally, Kaggle offers a great way to showcase your work to the world. You can create a portfolio, submit it to competitions, and ensure experts see it in the field. While Kaggle offers an excellent platform for data scientists to showcase their skills and capabilities, it could improve some things. First, Kaggle is often seen as a platform for experienced data scientists. While professional data scientists can benefit from the competitions and discussions on the forum, newcomers and amateurs may need help navigating the platform. They may need help to compete effectively. Second, the data provided by Kaggle may not be up-to-date or encompass the most recent trends. This could put competitors at a disadvantage, as their solution may need to be more effective and accurate if the data were more current. Finally, Kaggle requires to provide more assistance in terms of guidance or support.

10.4.1 Dataset Description The FER 2013 dataset [45] is open-source for facial emotion recognition (FER) algorithms. It comprises a collection of 48x48 pixel grayscale images of faces, each labelled with one of the seven emotion classes (angry, disgust, fear, happy, sad, surprise, neutral). This dataset was designed to advance the state-of-the-art in FER and is widely used for research. The FER 2013 dataset has proven to be a valuable tool for researchers developing new FER algorithms. It has been used in numerous studies to create and analyze the performance of various facial expression recognition techniques. Using the FER 2013 dataset, researchers can develop a framework for detecting the overall emotional score of an event from the images captured by a drone. This framework would help build drones to see human emotions and provide helpful information during surveillance or other activities. Furthermore, it would be possible to identify regions of stress or danger in large crowds, which can help in emergencies.

Detection of Overall Emotional Score  225 CREMAD [46] is the acronym for Crowd-Recognition and EmotionMining from Aerial Drone (CREMAD). It is a framework for detecting the overall emotional score of an event from images captured by a drone. This framework facilitates the development of an automated solution for detecting emotion from aerial photos and analyzing crowd movement in largescale gathering events. The framework comprises two main components, namely crowd recognition and emotion mining [47–50]. The cognitive crowd recognition module of the framework helps detect and recognize the crowd of people present in the scene. In contrast, the emotion-mining module extracts emotion from the seen group. The emotion-mining module is based on a novel emotion recognition technique. It uses deep learning models to extract emotion from aerial images and convert it into an overall emotional score. The CREMAD framework is designed to provide insights into the feelings of people attending an event and enable the analysis of crowd movements in largescale gatherings [51, 52]. It also provides an efficient method for detecting and analyzing crowd emotions in real time, making it an effective tool for event security, crowd control, and public safety. This dataset is one of the best to compare other image datasets to because it provides a comprehensive overview of the emotion surrounding a particular event. By looking at the images captured by a drone, researchers can gain insight into how people feel about a specific situation. The dataset also includes features such as facial expressions and body language, which can help researchers understand the emotional context of the scene. Additionally, the dataset provides an overall dynamic score for the event, allowing researchers to make more informed decisions regarding the event or situation. This dataset is valid for various purposes, such as understanding the emotional response to a natural disaster or political event. Furthermore, this dataset is applicable in multiple fields, such as psychology, marketing, and sociology. This dataset is an invaluable resource for researchers, providing a comprehensive overview of the emotional context of an event. When the proposed model is applied to the FER 2013 dataset, Figure 10.3 shows the accuracy and loss plot obtained from it. Figure 10.4 shows the classification matrix, which shows the accuracy, loss, weighted average accuracy, precision, recall, f1-score and support values obtained from the suggested model when trained and evaluated on the FER 2013 dataset. In contrast, Figure 10.5 represents the heatmap for the same dataset in the respective environment. Figure 10.6 is the image collected from a drone during the event on the university premises. The composed image is given to the facial extraction algorithm, and how the faces are selected using the algorithm is shown in

226  Advances in Aerial Sensing and Imaging Loss

Model accuracy 1.0

Train Validation

0.9 0.8 Accuracy

loss val_loss

1.0 0.8

0.7 0.6

0.6

0.5 0.4

0.4 0.3 0.2

0.2 0

10

20 30 Epoch

40

50

0

5

10

15

20

Figure 10.3  Accuracy and loss plot obtained from FER 2013.

Figure 10.4  Values obtained from the proposed model through FER 2013.

Figure 10.6 (a). The extracted images obtained after applying the algorithm are given in Figure 10.6 (b). Once the facial images are extracted from the picture, the emotional score of each image is calculated by the model trained and evaluated using FER 2013 dataset. A detailed description of how the emotions are calculated is given in Table 10.1. Figure 10.7 is another example image collected using a drone. Figure 10.7 (a) shows how the pictures are selected for extraction, and Figure 10.7 (b)

Angry

0.99

Disgust

0.00

Fear

0.00

Actual Happy

0.00

0.00

0.00

0.99

0.00

0.00

0.00

Sad

0.07

0.00

0.08

0.09

0.57

0.17

0.02

Surprise

0.01

0.00

0.00

0.01

0.02

0.96

0.00

Neutral

Detection of Overall Emotional Score  227

0.00

0.00

0.00

0.00

0.01

0.00

0.98

Angry

Disgust

Fear

Happy Predicted

Sad

Surprise

Neutral

0.00

0.00

0.00

0.00

0.00

0.00

0.96

0.02

0.00

0.00

0.01

0.01

0.00

0.99

0.00

0.00

0.00

0.00

0.8

0.6

0.4

0.2

0.0

Figure 10.5  Heat map through FER 2013.

shows the extracted images for the facial extraction algorithm. A detailed description of the emotions extracted is given in Table 10.2. The proposed FERCNN Model is also evaluated using the CREMAD dataset. Figure 10.8 represents the accuracy and loss curves when the model is trained and assessed using the CREMAD dataset. Testing accuracy of 78% and a loss of 0.1021 is observed during the evaluation. Figure 10.9 gives the Classification matrix resulting after assessment with the CREMAD dataset. The classification matrix shows the accuracy, weighted average accuracy, precision, recall, f1-score and support values. The training and evaluation observed a precision of 0.73, recall of 0.78, f1-score of 0.75 and support of 6162. Figure 10.10 gives the heatmap of the FERCNN model when the model is trained and evaluated on the CREMAD dataset.

228  Advances in Aerial Sensing and Imaging

(a)

(b)

(c)

Figure 10.6  (a) Image captured by drone. (b) Faces detected from image. (c) Extracted faces from facial extraction algorithm.

Table 10.1  Emotional score of individual faces for the image (Example 1) captured from drone where the FERCNN model is trained on FER2013 dataset. Extracted Photo Frame Drone Image

Emotional Score emotion 1.0

percentage

0.8 0.6 0.4 0.2 0.0 angry

disgust

fear

happy

sad

surprise neutral

sad

surprise neutral

emotion 1.0

percentage

0.8 0.6 0.4 0.2 0.0 angry

disgust

fear

happy

(Continued)

Detection of Overall Emotional Score  229 Table 10.1  Emotional score of individual faces for the image (Example 1) captured from drone where the FERCNN model is trained on FER2013 dataset. (Continued) Extracted Photo Frame Drone Image

Emotional Score emotion

1.0

percentage

0.8 0.6 0.4 0.2

angry

disgust

fear

happy

sad

surprise neutral

sad

surprise neutral

sad

surprise neutral

emotion 0.8 0.7 percentage

0.6 0.5 0.4 0.3 0.2 0.1 0.0 angry

disgust

fear

happy emotion

1.0

percentage

0.8 0.6 0.4 0.2 0.0 angry

disgust

fear

happy

Figure 10.11 is the image captured from the drone, and how the facial photos are selected from the drone image using the facial extraction algorithm is given in Figure 10.11 (a) and the extracted images from the facial extraction algorithm are shown in Figure 10.11 (b). Once the facial images are removed, an emotional score of every individual image is calculated, and a detailed description is given in Table 10.3. the Performance calculation is done on the model trained with the CREMAD dataset.

230  Advances in Aerial Sensing and Imaging

(a)

(b)

(c)

Figure 10.7  (a) Image captured by drone. (b) Faces detected from image. (c) Extracted faces.

10.5 Future Work and Conclusion The proposed framework has several potential future works. The model’s accuracy can be enhanced by incorporating more extensive datasets and advanced algorithms. The framework can also be extended to perform real-time dynamic analysis, integrated with other audio and video analysis technologies, and applied to other domains such as sports, entertainment, and politics. Additionally, the framework can be extended to detect animal emotions, specifically in animal welfare. Finally, future works should focus on developing privacy-preserving techniques to address privacy concerns arising from capturing and analyzing images of people. In conclusion, using CNN to calculate emotional scores from images captured by drones has the potential to become a powerful tool for

Detection of Overall Emotional Score  231 Table 10.2  Emotional score of individual faces for the image captured from drone where the FERCNN model is trained on FER2013 dataset. Emotional Score

Extracted Photo Frame Drone Image

emotion

1.0

percentage

0.8 0.6 0.4 0.2 0.0 angry

disgust

fear

happy

sad

surprise neutral

sad

surprise neutral

sad

surprise neutral

emotion 1.0

percentage

0.8 0.6 0.4 0.2 0.0 angry

disgust

fear

happy emotion

1.0

percentage

0.8 0.6 0.4 0.2 0.0 angry

disgust

fear

happy emotion

0.7 0.6 percentage

0.5 0.4 0.3 0.2 0.1 0.0 angry

disgust

fear

happy

sad

surprise neutral

(Continued)

232  Advances in Aerial Sensing and Imaging Table 10.2  Emotional score of individual faces for the image captured from drone where the FERCNN model is trained on FER2013 dataset. (Continued) Emotional Score

Extracted Photo Frame Drone Image

emotion 0.6

percentage

0.5 0.4 0.3 0.2 0.1 0.0 angry

disgust

fear

happy

sad

surprise neutral

emotion

percentage

0.8 0.6 0.4 0.2 0.0 angry

disgust

fear

happy

sad

surprise neutral

emotion

percentage

0.8 0.6 0.4 0.2 0.0 angry

disgust

fear

happy

sad

surprise neutral

sad

surprise neutral

emotion

1.0

percentage

0.8 0.6 0.4 0.2 0.0 angry

disgust

fear

happy emotion

0.5

ercentage

0.4 0.3

(Continued)

0.2 0.0 angry

disgust

fear

happy

sad

surprise neutral

emotion

1.0 0.8

percentage

Detection of Overall Emotional Score  233 0.6

Table 10.2  Emotional score of 0.4 individual faces for the image captured from drone where the FERCNN model is trained on FER2013 dataset. (Continued) 0.2

Extracted Photo Frame Drone Image

Emotional Score

0.0 angry

disgust

fear

happy

sad

surprise neutral

sad sad

surprise neutral surprise neutral

emotion emotion

percentage percentage

0.5 0.6 0.4 0.5 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0.0 0.0

angry disgust angry disgust

fear fear

happy happy emotion

Model accuracy on CREMAD

0.8

0.8

percentage

Accuracy

0.7 0.6 0.5

0.3 0

10

1.50 1.25

0.4

0.0 40

20 30 Epoch

Train Validation

1.75

0.6

1.00 0.75

0.2

0.4

Model loss on CREMAD

2.00

loss

Train Validation

0.50 0.25 50

angry

disgust

0

fear

10

happy

20 30 40 Epochsurprise neutral sad

50

emotion Figure 10.8  Accuracy and loss plot obtained from CREMAD. 0.8 percentage

gathering data in a range of scenarios. This technology can be used in vari0.6 ous applications, including surveillance, security, and marketing. However, 0.4 its accuracy depends on the training data quality and the user’s ability to 0.2 interpret the results. With further research and development, the accuracy of this technology is likely to 0.0 improve, making it a valuable tool for collectangry disgust fear happy sad surprise neutral ing data from drone images. emotion

1.0

percentage

0.8 0.6 0.4 0.2 0.0 angry

disgust

fear

happy emotion

0.5

rcentage

0.4 0.3

sad

surprise neutral

234  Advances in Aerial Sensing and Imaging

Figure 10.9  Classification matrix obtained from the proposed model through CREMAD.

Angry

Detection of Overall Emotional Score  235

0.99

0.00

0.00

0.00

0.00

0.00

Disgust

0.99

0.01

0.00

0.00

0.00

0.00

0.01

0.98

0.00

0.00

0.00

0.00

0.00

0.99

0.01

0.00

0.00

0.00

0.57

0.42

0.6

Happy

0.00

Neutral

0.00

Sad

Actual

0.00

Fear

0.8

0.34

0.13

Angry

Disgust

0.4

0.2

0.28

0.11

Fear Happy Predicted

0.08

0.06

Neutral

Sad

Figure 10.10  Heatmap of the FERCNN model through CREMAD.

0.0

236  Advances in Aerial Sensing and Imaging

(a)

(b)

(c)

Figure 10.11  (a) Image captured by drone. (b) Face detected from drone image. (c) Extracted faces.

Detection of Overall Emotional Score  237 Table 10.3  Emotional score of individual faces for the image captured from drone where the FERCNN model is trained on FER2013 dataset. Extracted Photo Fram Drone Image

Emotional Score

(Continued)

238  Advances in Aerial Sensing and Imaging Table 10.3  Emotional score of individual faces for the image captured from drone where the FERCNN model is trained on FER2013 dataset. (Continued) Emotional Score

Extracted Photo From Drone Image

emotion 0.5

percentage

0.4 0.3 0.2 0.1 0.0 angry

disgust

fear

happy

neutral

sad

neutral

sad

emotion

1.0

percentage

0.8 0.6 0.4 0.2 0.0

angry

disgust

fear

happy emotion

0.40 0.35 percentage

0.30 0.25 0.20 0.15 0.10 0.05 0.00 angry

disgust

fear

happy

neutral

sad

(Continued)

Detection of Overall Emotional Score  239 Table 10.3  Emotional score of individual faces for the image captured from drone where the FERCNN model is trained on FER2013 dataset. (Continued) Emotional Score

Extracted Photo From Drone Image

emotion 0.5

percentage

0.4 0.3 0.2 0.1 0.0 angry

disgust

fear

happy

neutral

sad

neutral

sad

emotion

1.0

percentage

0.8 0.6 0.4 0.2 0.0

angry

disgust

fear

happy emotion

0.40 0.35 percentage

0.30 0.25 0.20 0.15 0.10 0.05 0.00 angry

disgust

fear

happy

neutral

sad

240  Advances in Aerial Sensing and Imaging

References 1. Malliaraki, E., Social interaction with drones using human emotion recognition. In Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, Stockholm Sweden, 187–188, 2018. 2. Cauchard, J.R., Zhai, K.Y., Spadafora, M., Landay, J.A., Emotion encoding in human-drone interaction. 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2016. 3. Schneegass, S., Alt, F., Scheible, J., Schmidt, A., Midair displays: Concept and first experiences with free-floating pervasive displays. Proc. International Symposium on Pervasive Displays, pp. 27–31, 2014. 4. Yoshimoto, H., Jo, K., Hori, K., Designing interactive blimps as puppets, in: Entertainment Computing, S. Natkin and J. Dupire (Eds.), pp. 204–209, Springer Berlin Heidelberg, Paris, France, 2009. 5. Nagi, J., Giusti, A., Caro, G.A.D., Gambardella, L.M., Human control of UAVs using face pose estimates and hand gestures. Proc. ACM/IEEE International Conference on Human-Robot Interaction, pp. 252–253, 2014. 6. Naseer, T., Sturm, J., Cremers, D., FollowMe: Person following and gesture recognition with a quadrocopter. Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 624–630, 2013. 7. Szafir, D., Mutlu, B., Fong, T., Communicating directionality in flying robots. Proc. ACM/IEEE International Conference on HumanRobot Interaction, pp. 19–26, 2015. 8. Arroyo, D., Lucho, C., Roncal, S.J., Cuellar, F., Daedalus: An sUAV for human-robot interaction. Proc. ACM/IEEE International Conference on Human-Robot Interaction, pp. 116–117, 2014. 9. Sharma, M., Hildebrandt, D., Newman, G., Young, J.E., Eskicioglu, R., Communicating affect via flight path: Exploring the use of the laban effort system for designing effective locomotion paths. Proc. ACM/IEEE International Conference on Human-Robot Interaction, pp. 293–300, 2013. 10. Newlove, J. and Dalby, J., Laban for all, Taylor & Francis US, New York, 2004. 11. Ramos, D., Gonçalves, G., Faria, R., Sanches, M.P., Building a drone operator digital twin using a brain-computer interface for emotion recognition. 2021 20th International Conference on Advanced Robotics (ICAR), Ljubljana, Slovenia, pp. 824–829, 2021. 12. Pollak, M., Salfinger, A., Hummel, K., Teaching drones on the fly: Can emotional feedback serve as learning signal for training artificial agents?. Workshop on Interactive Machine Learning, 22, 1–8, 2022. 13. Sumers, T.R., Ho, M.K., Hawkins, R.D., Narasimhan, K., Griffiths, T.L., Learning rewards from linguistic feedback. Proc. AAAI Conf. Artif. Intell., 35, 7, 6002–6010, 2021. 14. Ramos, D., Gonçalves, G., Faria, R., Sanches, M.P., Building a drone operator digital twin using a brain-computer interface for emotion recognition.

Detection of Overall Emotional Score  241 2021 20th International Conference on Advanced Robotics (ICAR), Ljubljana, Slovenia, pp. 824–829, 2021, doi: 10.1109/ICAR53236.2021.9659360. 15. Barresi, G., Pacchierotti, C., Laffranchi, M., De Michieli, L., Beyond digital twins: Phygital twins for neuroergonomics in human-robot interaction. Front. Neurorobotics, 16, 913605, 2022, doi 10.3389/fnbot.2022.913605. 16. Jaiswal, S. and Nandi, G.C., Robust real-time emotion detection system using CNN architecture. Neural Comput. Applic, 32, 11253–11262, 2020. https:// doi.org/10.1007/s00521-019-04564-4. 17. Sajjad, M., Ullah, F.U.M., Ullah, M., Christodoulou, G., Cheikh, F.A., Hijji, M., Muhammad, K., Rodrigues, J J.P.C. A comprehensive survey on deep facial expression recognition: Challenges, applications, and future guidelines. Alexandria Eng. J., 68, 817–840, 2023. https://doi.org/10.1016/j. aej.2023.01.017. 18. Sajjad, M., Nasir, M., Ullah, F.U.M., Muhammad, K., Sangaiah, A.K., Baik, S.W., Raspberry Pi assisted facial expression recognition framework for intelligent security in law-enforcement services. Inf. Sci., 479, 416–431, 2019. https://doi.org/10.1016/j.ins.2018.07.027. 19. Rajan, S., Chenniappan, P., Devaraj, S., Madian, N., Facial expression recognition techniques: A comprehensive survey. IET Image Process., 13, 1031– 1040, 2019. https://doi.org/10.1049/iet-ipr.2018.6647. 20. Revina, I.M. and Sam Emmanuel, W.R., A Survey on human face expression recognition techniques. J. King Saud Univ. - Comput. Inf. Sci., 33, 6, 619–628, 2021, https://doi.org/10.1016/j.jksuci.2018.09.002. 21. Ranjan, R., Patel, V.M., Chellappa, R., Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans. Pattern Anal. Mach. Intell., 41, 1, 121–135, 2017. 22. Schroff, F., Kalenichenko, D., Philbin, J., ‘Facenet: A unified embedding for face recognition and clustering, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823, 2015. 23. Parkhi, O.M., Vedaldi, A., Zisserman, A., Deep face recognition. Proceedings of the British Machine Vision Conference (BMVC), 1, 1–12, 2015. 24. Wu, W., Yin, Y., Wang, X., Xu, D., Face detection with different scales based on faster R-CNN. IEEE Trans. Cybern., 49, 11, 4017–4028, 2018. 25. Zhang, J., Yin, Z., Chen, P., Nichele, S., Emotion recognition using multimodal data and machine learning techniques: A tutorial and review. Inf. Fusion, 59, 103–126, 2020. 26. Sajjad, M., Zahir, S., Ullah, A., Akhtar, Z., Muhammad, K., Human behaviour understanding in big multimedia data using CNN-based facial expression recognition. Mobile Netw. Appl., 25, 4, 1611–1621, 2020. 27. Li, H., Wang, N., Ding, X., Yang, X., Gao, X., Adaptively learning facial expression representation via CF labels and distillation. IEEE Trans. Image Process, 30, 2016–2028, 2021.

242  Advances in Aerial Sensing and Imaging 28. Peng, X., Xia, Z., Li, L., Feng, X., Towards facial expression recognition in the wild: A new database and deep recognition system, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 93–99, 2016. 29. Lee, I., Jung, H., Ahn, C.H., Seo, J., Kim, J., Kwon, O., Real-time personalized facial expression recognition system based on deep learning, in: 2016 IEEE International Conference on Consumer Electronics (ICCE), IEEE, pp. 267–268, 2016. 30. Guo, Y., Tao, D., Yu, J., Xiong, H., Li, Y., Tao, D., Deep neural networks with relativity learning for facial expression recognition, in: 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), IEEE, pp. 1–6, 2016. 31. Chao, L., Tao, J., Yang, M., Li, Y., Wen, Z., Long short-term memory recurrent neural network based encoding method for emotion recognition in video, in: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 2752–2756, 2016. 32. Jaiswal, A., Raju, A.K., Deb, S., Facial emotion detection using deep learning, in: 2020 International Conference for Emerging Technology (INCET), IEEE, pp. 1–5, 2020. 33. Talele, K., Shirsat, A., Uplenchwar, T., Tuckley, K., Facial expression recognition using general regression neural network, in: 2016 IEEE Bombay Section Symposium (IBSS), IEEE, pp. 1–6, 2016. 34. Kim, B.-K., Roh, J., Dong, S.-Y., Lee, S.-Y., Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J. Multimodal User Interfaces, 10, 2, 173–189, 2016. 35. Durmusog˘lu, A. and Kahraman, Y., Facial expression recognition using geometric features, in: 2016 International Conference on Systems, Signals and Image Processing (IWSSIP), IEEE, pp. 1–5, 2016. 36. Rao, Q., Qu, X., Mao, Q., Zhan, Y., Multi-pose facial expression recognition based on SURF boosting, in: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), IEEE, pp. 630–635, 2015. 37. Sebe, N., Lew, M.S., Cohen, I., Garg, A., Huang, T.S., Emotion recognition using a cauchy naive bayes classifier, object recognition supported by user interaction for service robots, vol. 1, pp. 17–20, IEEE, Quebec City, QC, Canada, 2002. 38. Carcagnı`, P., Del Coco, M., Leo, M., Distante, C., Facial expression recognition and histograms of oriented gradients: A comprehensive study. Springerplus, 4, 1, 1–25, 2015. 39. Soyel, H. and Demirel, H., Improved SIFT matching for pose robust facial expression recognition, in: 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), IEEE, pp. 585–590, 2011. 40. Chen, L., Zhou, C., Shen, L., Facial expression recognition based on SVM in E-learning. Ieri Proc., 2, 781–787, 2012.

Detection of Overall Emotional Score  243 41. Levi, G. and Hassner, T., Emotion recognition in the wild via convolutional neural networks and mapped binary patterns, in: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 503–510, 2015. 42. Lowe, D.G., Object recognition from local scale-invariant features, in: Proceedings of the Seventh IEEE International Conference on Computer Vision, Ieee, vol. 2, pp. 1150–1157, 1999. 43. Zeng, N., Zhang, H., Song, B., Liu, W., Li, Y., Dobaie, A.M., Facial expression recognition via learning deep sparse autoencoders. Neurocomputing, 273, 643–649, 2018. 44. Luo, Z., Chen, J., Takiguchi, T., Ariki, Y., Facial expression recognition with deep age, in: 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), IEEE, pp. 657–662, 2017. 45. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Milakov, M., Park, J., Ionescu, R., Popescu, M., Grozea, C., Bergstra, J., Xie, J., Romaszko, L., Xu, B., Chuang, Z., Bengio, Y., Challenges in representation learning: A report on three machine learning contests. Neural Netw., 64, 59–63, 2015. Special Issue on “Deep Learning of Representations”. 46. Cao, H., Cooper, D.G., Keutmann, M.K., Gur, R.C., Nenkova, A., Verma, R., CREMA-D: Crowd-sourced emotional multimodal actors dataset. IEEE Trans. Affect. Comput., 5, 4, 377–390, 2014 Oct-Dec, doi 10.1109/ TAFFC.2014.2336244. PMID: 25653738; PMCID: PMC4313618. 47. Swathi, G., Robust deep learning technique: U-net architecture for pupil segmentation, in: 11th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), pp. 0609–0613, 2020. 48. Shinde, S. and Prashant, J., A review: Eye tracking interface with embedded system & IOT, in: International Conference on Computing, Power and Communication Technologies (GUCON), pp. 791–795, 2018. 49. Kumar, S., Rani, S., Jain, A., Verma, C., Raboaca, M.S., Illés, Z., Neagu, B.C., Face spoofing, age, gender and facial expression recognition using advance neural network architecture-based biometric system. Sensors, 22, 14, 5160, 2022. 50. Bola, A., Mahajan, S., Singh, S., Informative gene selection using adaptive analytic hierarchy process (A2HP). Future Computing Inform. J., 2, 2, 94–102, 2017. 51. Kaur, R. and Singh, S., A novel fuzzy logic-based reverse engineering of gene regulatory network. Future Computing Inform. J., 2, 2, 79–86, 2018. 52. Kumar, S., Rajan, E.G., Rani, S., Enhancement of satellite and underwater image utilizing luminance model by color correction method, in: Cognitive Behavior and Human Computer Interaction Based on Machine Learning Algorithm, pp. 361–379, Scrivener Publishing, Beverly, Mass. USA, 2021.

11 Drone-Assisted Image Forgery Detection Using Generative Adversarial Net-Based Module Swathi Gowroju1*, Shilpa Choudhary2, Medipally Rishitha1, Singanaboina Tejaswi1, Lankala Shashank Reddy1 and Mallepally Sujith Reddy1 Dept. of CSE, Sreyas Institute of Engineering and Technology, Hyderabad, India 2 Dept. of CSE, Neil Gogte Institute of Technology, Hyderabad, India

1

Abstract

In today’s digital world, image forgery detection in drone-captured images is challenging due to noise, motion blur, and other factors unique to aerial photography. The research community is actively seeking solutions to this problem. Biometric systems are becoming increasingly important due to each individual’s unique identity. However, some people modify their physical appearance to avoid detection by these systems, highlighting the need for advanced forgery detection techniques. We propose an LBPNet, an LBP-based machine learning convolutional neural network that can detect fake face photos to address this issue. The proposed System compares LBPNet and NLBPNet, as it relies on feature extraction using the LBP algorithm. Additionally, our suggested paired learning technique enables forged feature learning, allowing the detection module to identify falsified images generated by a new GAN, even if it was not included in the training phase. Using a drone-based system can capture high-resolution images and videos from different angles and perspectives, further enhancing the efficiency and speed of the detection process. Our proposed approach can significantly improve image forgery detection, making it a valuable tool for various industries, including journalism, law enforcement, and national security. Keywords:  Forged images, image detection, machine learning, LBP (local binary patterns), CNN, features extraction, LBPNet *Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (245–266) © 2024 Scrivener Publishing LLC

245

246  Advances in Aerial Sensing and Imaging

11.1 Introduction Nowadays, manipulating images has become relatively simple since several techniques exist to create a similar image from a genuine one because of the digital world. Manipulation of an image occurs when a picture is utilized without the necessary authorization [1–3]. Social networking sites are prevalent these days and are used by a large number of people all over the world to make new friends and share their interests. Social networking sites are also famous for sharing images [4, 5]. Many people are taking advantage of this by using images shared by others without their permission and manipulating others using those forged images. As a result, forged picture detection is particularly valuable in identifying criminals and determining whether an image is faked or accurate. Many individuals have become victims of picture falsification in this modern age. Some criminals use software to modify and exploit images as evidence to perplex the courts [6, 7]. Information collected through social media posts must be labeled as either true or false. Social networking is a fantastic tool for spreading information and knowledge dissemination. When there is no warning, people may be deceived. Despite the inadvertent deceptive promotion, the bulk of Photoshop image manipulation is evident, but a handful is not. Some of these pictures may look honest and unprofessional due to pixelization [8, 9]. Specifically, a politician’s credibility might be shattered by altered photographs in the policy. Deep learning-based generative models, including the recursive neural net (GAN), have recently been developed and widely employed to create a photorealistic partial or entire picture and video material. Furthermore, New GAN research, such as a system characterized by GANs (PGGAN) but instead of BigGAN, could be used to create photorealistic images and videos so that an individual could tell if the photo is constructed or not within a limited time [10–12]. However, if the faked or synthesized image is wrongly posted on a social network or website, it might cause significant problems. Cycle GAN, for example, is employed to create the forged facial picture in a pornographic film [13, 14]. Furthermore, GANs can be utilized to develop a verbal video with the synthesized facial information of any conservative person, causing significant issues in society, politics, and business. As a result, an efficient falsified face image identification technique is required. We had also stretched our previous study on paper I.D. in this chapter to effectively and economically address these issues [15].

Drone-Assisted Image Forgery Detection  247 This chapter is divided into four sections. The second section summarizes the previous related work. The third, fourth and fifth sections cover the proposed methodology, evaluation parameters, and results. At the same time, the conclusion and future work are discussed in the last section.

11.2 Literature Survey Deep learning-based generative models, including the recursive neural net (GAN), have recently been developed and widely employed to create a photorealistic partial or entire picture and video material [16–18]. Many researchers have worked on different machine learning and deep learning algorithms. Rossler et al. proposed fake faces recognized to recognize manipulated facial images. The increase in image manipulation and artificial image creation has now reached the discussion about the allegations for society. The proposed work aims to reduce trust in online media and spread crafted data or false news. This work investigates the facts of advanced image manipulations and how challenging it is to identify them automatically or manually. The author proposed a robotic benchmark order to detect deceit in facial images to standardize the analysis of detection systems. Deepforges, Face2Face, FaceSwap, and Neural Textures are the most influential sources for facial maneuvers at various flexural levels and sizes of data. These benchmark datasets are publicly accessible and include a concealed verification and the metadata of images made up. Based on this information, the author thoroughly examines forgery detection on various input images. Even in significant compression, supplementary application area-specific expertise improves the prediction accuracy of forgery detection. MesoNet is a dense facial multimedia detection technique proposed by Afchar. This method is efficient and automatically detects face falsification in videos. It focuses on two advanced developments that generate ultra-realistic falsified video content: Deep manufactured and Face2Face. Traditional encryption forensic evidence is unsuitable for video forensics due to compression, which severely degrades the data. It takes an approach to deep learning and provides two layers containing a small number of layers to emphasize mesoscopic image features. We analyzed these fast connections on the benchmark dataset and our dataset collected from online videos.

248  Advances in Aerial Sensing and Imaging The tests show a high prediction rate of 98% for Deepforged and 95% for the application Face2Face. Wu et al. proposed a ManTra-Net manipulation tracing network for detecting and localizing image forgeries with strange features. ManTra-Net is a unified deep neural architecture used to identify and overcome reallife image forgery involving many manipulations. ManTra-Net is based on an edge network that first processes localization and detection without additional pre- and post-processing. Manifold is a convolution neural network that handles arbitrary image sizes and various forgery types such as merging, moving animation, discarding part of the image/video, enhancement of video or image, etc. A simple and efficient identity learning task is created to acquire strong traces of digital manipulation by categorizing 385 types of manipulations and then performing the forgery analyzing issue as community anomaly recognition, with Z-score features designed to note the anomalies and a protracted selective memory workaround to assess local abnormalities. Multimedia face manipulation detection using an ensemble of CNNs was proposed by Bonettini et al. Many techniques for manipulating people’s faces in video content have been established and made available to the public. These methodologies can customize face content from input video content with probabilistic Prediction. Instead, while these tools are helpful in various applications, they can negatively affect culture and society, such as forged news dispersion and cyberbullying through generated reprisal porn when used unauthenticated. It is a challenging task to detect whether a facial expression is authentic or forged/ manipulated in the existing video is very much critical. As a solution, in the current work, we draw parallels between face manipulation techniques to the problem of detecting face manipulation in video sequences. This paper investigates the ensemble of various neural network convolution models. These models are obtained by starting with a base network and utilizing differing concepts of media exposure layers and Siamese training. Combining these networks yields promising results for face manipulation detection on two different data sets with over 119,000 videos. Knowledge to detect forged facial images in the wild was proposed by Hsu et al. GAN has the potential to create an accurate image; however, using such technologies raises concerns. For example, it can generate a falsified video for particular individuals and improper activities. These photorealistic things cause very much damage to a specific person and endanger that person’s protection. We worked on an advanced forgery discriminator (DeepFD) to detect machine image data effectively and efficiently. Learning a classifier model directly is challenging because it is difficult to

Drone-Assisted Image Forgery Detection  249 Table 11.1  Literature survey of forgery image detection from the past decade. Author

Result

Remarks

HsuCC et al. [6], 2018

Detection of the forged image in a video

Detecting Video forgery using residual correlation of image noise.

Chang et al. [5], 2009

Authentication of source image

Image authentication using the LBP watermark technique in the wavelet domain.

Farid et al. [7], 2009

Forged image detection

Detecting forged images using pixel-wise classification.

Zhu et al. [3], 2017

Image-to-image translation

Using nearly 300 adversarial networks, the translation of images is carried out.

Chollet et al. [10], 2017

Forged image detection

Detecting image forgery using CNN.

Luo et al. [8], 2018

Fake faces identification

Fake face identification using CNN.

Marra et al. [9], 2018

Forged image detection

Detecting fake images obtained from GANs over social networks.

Brock et al. [2], 2018

Forged image detection

Using GAN training, image synthesis is performed at a large scale.

Karras et al. [1], 2018

Forged image detection

By using GANs progressively, the forged image is detected.

Chih-Chung Hsu et al. [15], 2018

Detect Fake face images in wild

The Prediction is made using the classification of real and fake image training. (Continued)

250  Advances in Aerial Sensing and Imaging Table 11.1  Literature survey of forgery image detection from the past decade. (Continued) Author

Result

Remarks

D. Afchar et al. [12], 2018

MesoNet

Fake images were detected in video using MesoNet.

A. Rossler et al. [11], 2019

Face Forensics

The CNN is used to detect manipulated input images.

Yue wu et al. [13], 2019

ImageNet

Manipulation tracing network for detection and localization of image forgery anomalous.

N. Bonettini et al. [14], 2020

ImageNet

Through ensemble CNNs detecting video face manipulations.

know the discriminant information for analyzing forged images generated by different GANs. A clustering algorithm is integrated to detect machine images [19, 20]. The results indicate that the proposed work successfully saw 98.7% of forged images. Face recognition methods are beneficial in recognizing a person’s identity, but criminals alter their visual effect in behavior and psychology to deceive recognition systems. The survey of literature is mentioned in Table 11.1. To address this issue, we employ a novel technology known as Deep Texture Based feature extraction from images [21–23]. We are then constructing a training model for machine learning using a deep CNN automated system such as LBPNet. It relies heavily on feature extraction using the LBP (Local Binary Pattern) methodology.

11.3 Proposed System Local binary patterns (LBP) are visual identifiers used in computer vision to build a classification model. It is a basic yet highly dependent on texture-­ based values that categorizes the image using thresholding by applying a K-neighborhood algorithm for each pixel and interpreting the result as a binary interpretation. LBP has become a prominent method in several applications because of its computational simplicity. The LBP operator’s

Drone-Assisted Image Forgery Detection  251 resistance to grey-scale image conversion is one of the critical stages, as variations in lighting cause noise in conversion. The overall proposed model is given in Figure 11.1. The training samples are created by GANs using Equation 11.1. M X forged = [ Xik==i1 , Xik==21 ,…, Xik==N1 1 , Xik==NM ]



(11.1)

where each GAN generates Nk training images. Let the amount of training data images, and for both real and created images, denote the training set consisting of real images will be using Equation (11.2)

N T = Nr + N f = Nr + ∑ kM=1 N k .



(11.2)

The class labels, Y= [y1, y2, …,yNt], implies if a photograph is forged (y=false) or genuine (y = true). As said, the pairwise pixel data using Equation (11.3).



P = [pi=0,j=0, pi=0, j=1,… ,pi=0,j=Nr,…,pi=Nf,j=Nr]

(11.3)

It is required for the training of the model for the CFFN to train exclusionary CFFs effectively. The variation combination can generate pairwise information through the instruction set X and related label set Y. Hence the pixel pairs are produced from the model instances. We set the overall number of pairs samples in this paper to Nr=2000,000. Identifying forged

Fake feature learning

Real Images

Contrastive learning Forged feature network model

Feature extraction Predicted label

Classification using Binary cross entropy

Labelled Images

Fake Images Images

Training the model

Figure 11.1  Proposed forged face detector (image source [18]).

252  Advances in Aerial Sensing and Imaging facial images is the most challenging problem in detecting interactive content forgery. Face falsification detection could be used to create imaginary characters on social networking sites, allowing for the unlawful theft of personal information. For example, the wholly fabricated image transformer may be utilized to establish public officials with pirated content, which would be destructive.

11.3.1 Common Forged Feature Network Figure 11.2 depicts the forged feature connectivity to learn the falsified characteristics from the training set, many advanced CNNs can be used. To catch the realizable value from the test set in a purely supervised manner, the Xception Network was used. Other sophisticated CNNs, such as InceptionNet, ResNet, and AlexNet, can also be utilized to train the forged face detector. Most of these novel CNNs are learned and supervised, and classification performance highly depends on the training set. We look for the CFF throughout multiple GANs instead of learning the completely fabricated functionalities from each of the GANs’ images. As a result, knowing the shared fake feature network necessitates using an appropriate integrity network. Traditional CNNs (DenseNet) were not trained to understand the exclusionary CFF. We propose to address this disadvantage by incorporating the DenseNet and the corresponding CFF to build a new exclusionary CFF model. The architecture in Figure 11.2 contains several dense blocks at three levels, which serve as a fundamental component of DenseNet. This model is trained using a supervised learning strategy, whereas the recommended pairwise functional impact for CFFs is a semi-supervised instructional strategy. Target photographs will be taken from various internet sources to obtain answers to the study questions, test the hypothesis, and evaluate Cross-layer feature concatenating

1*1 output Conv.6

Input fake images

2 * Dense Blocks 4 * Dense Blocks

3 * Dense

128*3 Fully connected layer

Blocks Conv. 4

Figure 11.2  Forged feature network.

Conv.5

Drone-Assisted Image Forgery Detection  253 the findings. These images comprise the dataset. Using traditional mathematical processes, the convolutional layer of the CNN is responsible for retrieving picture features. These convolutional techniques apply two-­ dimensional digital filters. The standard procedure, as illustrated, where every other image tile blocks matrix with proportions equal to the filtering dimension will be increased by the filter matrix, supposing the image tile is 4x4 pixels and the typical filter is a 2x2 matrix filter. The layer between the conventional layer and the SoftMax layer is the activation function layer, which, like any traditional neural network kernel function, removes unwanted pixels, such as negative values. CNN rely solely on elevated feature representation to determine whether or not an image is forged. On the other hand, CFDs of falsified face images may exist not only in the predominance but also in the intermediate feature representation.

11.3.2 Features Extraction Face extraction plays an essential role in Forged image detection. It includes shapes, color, texture, and movements of the facial image. It also reduces the information in the image, which requires less storage. The geometric separation of two reference points. After identifying the eye centres, 11 correlated points are obtained from the provided face input picture. The crucial points are placed in three locations from the eye, and the lateral endpoints, are put in the face. The nose’s vertical midway, the lip’s midpoint, and two points on the lateral ends of the lips. This procedure works with the face being upfront, color pictures, and consistent lighting with the sample image being either indifferent or smiling. The proposed system uses real-fake image dataset (RFID) [18], which comprises 990 original video sequences that have been modified using four automated face modification methods such as D.P., F2, F.S., and N.T. Hence, it has four sub-datasets. The data came from 977 YouTube videos, all featuring a trackable, mainly frontal face with no occlusions, allowing automated tampering methods to make believable forgeries.

11.3.3 Features Classification and Classification Network Classification is also known as the feature selection stage. Classification is complicated because it can play a role in many areas. It deals with exchanging essential information and connects them to specific parameters. The following points explain the algorithmic approach to classification. Figure 11.3 illustrates the classification of the image and the model is trained with

254  Advances in Aerial Sensing and Imaging R-Net Conv: 3x3 Conv: 3x3 Conv: 2x2 fully face face classification MP: 3x3 MP: 3x3 classification connect 1x1x2 2 bounding box bounding box regression regression 1x1x4 4 input size 11x11x28 Facial landmark Facial landmark 4x4x48 3x3x64128 5x5x10 3x3x16 1x1x32 24x24x3 localization localization 1x1x10 10 O-Net Conv: 3x3 Conv: 3x3 Conv: 3x3 Conv: 2x2 fully face classification MP: 3x3 MP: 3x3 MP: 2x2 connect 2 bounding box regression 4 input size Facial landmark localization 23x23x32 10x10x64 4x4x64 3x3x128 256 48x48x3 10

P-Net Conv: 3x3 Conv: 3x3 Conv: 3x3 MP: 3x3

input size 12x12x3

Figure 11.3  Classification of image.

Figure 11.4  Input pre-processed face images [18].

Drone-Assisted Image Forgery Detection  255 a forged image data set, which is then categorized into two stages. One group is for training and another for testing two folding processes. Figure 11.4 cannot be distinguished whether it is genuine or forged. The forged image was obtained using Machine Learning/Artificial Intelligence GANs (Generative Adversarial Networks) of Machine Learning/Artificial Intelligence. For example, face unlocks features of current smartphones, and anyone can use forged faces to unlock their phone. Forged face classifiers can be used to identify these forged images.

11.3.4 Label Prediction Data labeling in machine learning identifies actual data and adds one or more constructive and informative labels to provide context. High-quality training data are used to build successful machine-learning models. However, generating the training data required to build these models is frequently costly, complicated, and time-consuming. The majority of today identify a specific individual to manually process label data to teach the algorithm how to make the right decisions. To address this issue, nomenclature can increase efficiency by automatically labeling data with a machine learning model. A machine learning algorithm for categorizing data is first given training on a subset of your actual data that humans have previously classified in this process. Where the identifying model has a high confidence level in assessing the outcome upon what it has discovered thus far, it will automatically pertain identifiers to the raw data. Where the identification model has less optimism in its consequences, it will send the data to individuals for labeling. The labeling model then uses human-generated labels to know and understand from and strengthen its capacity to recognize the following raw data set automatically. Over time, the prototype can label increasing amounts of data, significantly speeding up the process of the test dataset.

11.3.5 Contrastive Learning Contrastive learning is an artificial intelligence technique that teaches the prototype which measured values are similar or distinct to recognize the basic properties of a data set without labels. It enables our algorithm for machine learning to perform the same function. Before performing a task like classification or segmentation, it examines which clusters of data points are “similar” and “different” to acquire higher-level functionalities about the data.

256  Advances in Aerial Sensing and Imaging

11.3.6 Binary Cross-Entropy Loss In binary classification tasks, it is a loss function. These tasks require you to respond to a question with only two options, true or false, using Equation (11.4).



Loss = abs (Y pred - Y actual)

(11.4)

We can keep updating our conceptual framework until we achieve the best possible result based on loss value. Binary cross entropy is the negative estimated value of the log of adjusted probability. The opposing average of values can be calculated using Equation (11.5).

-



1 N ∑ i =1 (log(Pi)) N

(11.5)

We can calculate the log loss using Equation (11.6).



Log loss =

1 N ∑ i =1 −(y i *log(p k ) + (1-y k )* log(1-p k )) N

(11.6)

Where Pk is the probability of class 1 (True) and (1-Pk) is the probability of class 0. (false). Log loss for multi-class classification problems can be calculated using Equation (11.7).

Log loss = -



1 K L ∑ i ∑ j yij log( pij ) K

(11.7)

Where, K is No. of rows and L is No. of classes.

11.4 Results RFID dataset and dataset collected from own images are used to extract the features in the experiment. The RFID images covered various unconstrained conditions and cluttered backgrounds, including 10,300 personalities and 201,568 arranged facial images. Five cutting-edge GANs were used in the investigation to generate the training data set of imposters predicated on the CelebA set of data. Except for the PGGAN, it took a lot

Drone-Assisted Image Forgery Detection  257 of work to synthesize feasible high-resolution images using the selected GANs. The standard size of the produced facial appearance in the published source code of DCGAN, WGAP, WGAN-GP, and LSGAN was only 6464 pixels. If the dimensions of the inherently involved were set to 128128 pixels, many artefacts would be visible in the image regions, making it easy to identify the forged image. Most GANs can only produce feasible generated images with a lower resolution, such as 6480 pixels. The input image’s shape has been set to 6480 pixels to guarantee an equal assessment of different feature detection systems in recognizing crafted images obtained from various GANs. The prediction fit published by the publishers of the related GAN was used in the PGGAN. The PGGAN, on the other hand, can be used to create significant forged face images, though the size of the produced facial feature differs from the one used in our experiments. As a result, in the experimentations, we reduced the sample size of the falsified input images produced by the PGGAN to 6468. 

11.4.1 Experimental Settings The recommended CFFN and completely fabricated face detector’s training data set learning rate was set to = 1e3, and the whole epoch was formed to 20. The descriptive loss threshold m was set to 0.5. The a priori algorithm was used for both the initial and foremost learning steps. The CFFN’s first-step learning epoch count was set to 2, and the classified sub-­ classification network’s learning epoch count was set to 15, with the batch size of tasks as 30. In the experiments, we used the keyframe counterfeit products technique utilizing sensor pattern noise to compare performance. The Benchmark technique was a communal learning method that did not require two-step learning and was predicated on binary cross-entropy loss. Rather than the CFFN structure, the DenseNet to the binary cross-entropy wavelet coefficients was used. The evaluation parameters, such as precision  P  and recall  R, were compared using the formulas. Precision P is defined in Equation (11.8).



P=

TruePositive TruePositive + FalsePositive

(11.8)

Recall R is defined in Equation (11.9).



R=

TruePositive TruePositive + FalseNegative

(11.9)

258  Advances in Aerial Sensing and Imaging Table 11.2  Trainable parameters of the model. Parameters

Total number

Total no of parameters

822,478

No trainable parameters

822,465

No non-trainable parameters

0

# of instances

800 600 400 200 0

0.00

0.25

0.50 0.75

1.00 1.25 Class

1.50

1.75 2.00

Figure 11.5  Plot of training and validation classes.

Where T.P. signifies the correctly measured samples, which indicate that a correct, accurate image sample was recognized as an actual image, F.P. symbolizes the false positive, which suggests that a fake image was discovered as a not phoney picture. F.N. represents the number of false negatives, indicating that a correct actual image was recognized as forged. The total number of trainable parameters and the time taken to complete the training is listed in Table 11.2. The number of input images was distributed into two classes. 1632 images belong to a training class, 204 to a validation class, and 205 to a test class. The distribution is depicted in Figure 11.5.

11.4.2 Performance Comparison The affirmation consistency graphs during the practice process are depicted in Figure 11.6. To test the proposed method’s efficacy, we were forced to remove one of the GANs from the pre-processing step and implement it instead while testing the model to create different data sets. The PGGAN

Drone-Assisted Image Forgery Detection  259 1

Accuracy

0.9 0.8 0.7 0.6

WGAN-GP WGAN PGGAN DCGAN LSGAN

0.5 0.4 0

7k

14k

21k

28k 35k Iterations

42k

49k

56k

63k

Figure 11.6  Validation accuracy of the proposed system in the training phase.

was eliminated from the proposed training phase. For example, the wholly fabricated simulation outcomes and the best accuracy are obtained using accurate images to evaluate the effectiveness of the forged face detector. Regarding accuracy, the proposed falsified face recognition sensor, based on two different techniques, and theoretical results were objectively compared. The affirmation accuracy curves are shown during the learning phase. It proved the utility of the recommended DeepFD. The reliability graph during the training process is depicted in Figure 11.6. The proposed pairwise learning method extracted CFFs successfully from training datasets generated by various GANs. As a result, it was discovered that the suggested technique was more broadly applicable and efficient than the other research methods. With the range of values, in range (0, 30) epochs, the loss values were recorded in Table 11.3. Table 11.3 indicates that with a loss of 0.02, we could achieve an accuracy of 65% on the validation set. The graphs corresponding to the loss and accuracy are shown in Figure 11.7.

11.4.3 LBP Visualized Results The object can be localized by setting the final convolution operation’s bandwidth to the same value as the class labels. Visualization is possible thanks to the streams of the suggested CDNN’s final convolutional layer. As a result, the design was used to envision forged provinces in the last convolutional and route the interactions to the image field. Figure 11.8 depicts a falsified feature map for the sensitivity of generated geographical areas in facial recognition software obtained by WGAN and PGGAN, from Figure 6a–e and f–j, respectively, where images from k to t are the actual pictures

260  Advances in Aerial Sensing and Imaging Table 11.3  Loss values at each epoch. Predicted loss values of 0–30 epochs 0.6928555965423584, 0.6905931830406189, 0.6915552020072937, 0.6884430050849915, 0.6936195492744446, 0.6915259957313538, 0.6863223910331726, 0.6849153637886047, 0.6737678647041321, 0.6780544519424438,

0.6918739080429077, 0.6862038373947144, 0.6882265210151672, 0.6912440657615662, 0.6891873478889465, 0.6848084926605225, 0.6876918077468872, 0.674583911895752, 0.676084041595459, 0.6786404252052307,

train_loss vs val_loss

0.700

0.62

0.690

0.60

0.685

058 accuracy

loss

0.695

0.680 0.675

056 054

0.670

052

0.665

050

0.660

–5

0

5

10 15 20 num of Epochs

25

30

train_acc vs val_acc

0.64

train val

0.6886735558509827, 0.6873254179954529, 0.685150146484375, 0.6850732564926147, 0.6901299953460693, 0.6906144618988037, 0.6824907660484314, 0.676114022731781, 0.6743172407150269, 0.6751114130020142

35

048

train val 0

5

10

15 20 num of Epochs

25

30

Figure 11.7  Validation and accuracy plots of the model.

that were taken and feature responses. Since the final deep learning model seemed to have multiple networks, the first controller was considered the functionality of the segment, and the subsequent flow was considered a functionality rebuttal of the conventional trimester (i.e., forged image). This indicates that the proposed System illustrates the falsified regions designed to allow a much more informative inference of standard fraudulently generated GAN features. The final fully connected layer generated using the second input and the showcase map’s normalized response values creates the heat map. Consequently, in Figure 11.9, the falsified image features have relatively high feature arithmetic mean in the artefact-related regions; meanwhile, the actual images were detected with few feature responses. To draw the artefact regions in red, we map the value of the dependent variable to

Drone-Assisted Image Forgery Detection  261

Figure 11.8  The visualized LBP feature maps of the RFID dataset [18].

262  Advances in Aerial Sensing and Imaging

Figure 11.9  Images after detecting whether the image is forged using own dataset.

the original image. The output is also recorded for the own dataset, where the analysis uses images collected from various sources.

11.4.4 Training Convergence

1

Accuracy

0.9 0.8 0.7 0.6 Baseline-I DeepFD Baseline-II

0.5 0.4

0

7k 14k 21k 28k 35k 42k 49k 56k 63k Iterations (a)

Contrastive Loss (Log view)

In the proposed method, network training convergence must be ensured. As a result, the implementation of discourse analysis informational accuracy was discussed, and the results are depicted in Figure 11.10a. The orange line indicates the continuity curve during supervised methods training without linguistic loss. The line graph reflects the confirmation stability graph of the training process with the scheduled pairwise wanting to learn strategy in two steps.

8 PGGAN WGAN WGANGP DCGAN LSGAN

6 4 2 0 -2

0

7k 14k 21k 28k 35k 42k 49k 56k 63k Iterations (b)

Figure 11.10  (a) The validation accuracy curves produced by PGGAN (b) The curves of the proposed model of DeepFD’s contrastive loss values.

Drone-Assisted Image Forgery Detection  263 In comparison to the closely managed learning approach, the future pairwise having to learn strategy significantly improves method convergence. It was, however, essential to ensure that the meta-discourse loss converged—the depicted unity of the suggested stylistic loss throughout CFFN training in Figure 11.8a and b. Table 11.4 compares the efficiency and accuracy of the results when evaluating data from outside the training data among the various types of networks (Imagenet Network, Imagenet Using Transfer Learning, and the proposed method). The results show no significant statistical difference in the model performance of the multiple network types. The significance level was 0.172, which means it was higher than 0.05, which is not statistically significant. There have been, however, differences between the three networks when experimenting with data from the dataset, with ImageNet (91.1) outperforming ImageNet Using Backpropagation

Table 11.4  Performance analysis using various pre-trained models.

Model description

M

SD

Continuous probability distribution

Using Single fold data distribution

ImageNet Network [13]

74.2

3.6

6.357

0.18753

ImageNet Transfer Learning [14]

86.2

7.5

Proposed Method

89.2

4.5

ImageNet Network [13]

64.5

9.6

4.639

0.00532

ImageNet Transfer [14] Learning

76.5

6.5

Proposed Method

87.5

1.3

Using twofold data distribution

Probability

264  Advances in Aerial Sensing and Imaging Algorithm (78.4) and Classic CNN (64.5). The results presented in the following table are for the second dataset only.

11.5 Conclusion In conclusion, our proposed LBPNet with paired learning technique is a promising approach for detecting fake face photos and general visuals generated by advanced Generative models. The suggested cross-layer feature representation can learn middle and high-level racially discriminatory forged features. Furthermore, the proposed paired learning technique allows for effective forgery detection and can help to identify falsified images generated by a new GAN. We plan to expand our proposed approach to include forgery detection in other areas, such as information retrieval. As digital forgery becomes more prevalent, developing advanced techniques to identify and mitigate its associated risks is essential. Our approach significantly impacts various industries, including journalism, law enforcement, and national security. With further refinement and development, it has the potential to become a valuable tool for detecting and preventing image forgery and manipulation. Overall, our proposed approach can significantly contribute to the field of forgery detection and help address the future challenges associated with digital forgery.

References 1. Karras, T., Aila, T., Laine, S., Lehtinen, J., Progressive growing of GANs for improved quality, stability, and variation, in: Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada, 30 April–3 May 2018. 2. Brock, A., Donahue, J., Simonyan, K., Large scale gan training for high fidelity natural image synthesis. arXiv Preprint, arXiv:1809.11096, 1–18, 2018. 3. Zhu, J.Y., Park, T., Isola, P., Efros, A.A., Unpaired image-to-image translation using cycle-consistent 259 adversarial networks. arXiv Preprint, 2223–2232, 2017. 4. A.I. can now create fake porn, making revenge porn even more complicated, http://theconversation.com/ai-can-now-create-fake-porn-making-revengeporn-even-more-complicated-92267, 262, Monash University, Australia, 2018. 5. Chang, H.T., Hsu, C.C., Yeh, C., Shen, D., Image authentication with tampering localization based on watermark 266 embedding in the wavelet domain. Optical Eng., 48, 057002, 2009.

Drone-Assisted Image Forgery Detection  265 6. Hsu, C.C., Hung, T.Y., Lin, C.W., Hsu, C.T., Video forgery detection using correlation of noise residue. Proc. of the IEEE Workshop on Multimedia Signal Processing. IEEE, pp. 170–174, 2008. 7. Farid, H., Image forgery detection. IEEE Signal Process. Mag., 26, 16–25, 2009. 8. Huaxiao Mo, B.C. and Luo, W., Fake faces identification via convolutional neural network. Proc. of the ACM Workshop on Information Hiding and Multimedia Security, ACM, pp. 43–47, 2018. 9. Marra, F., Gragnaniello, D., Cozzolino, D., Verdoliva, L., Detection of GANgenerated fake images oversocial networks. Proc. of the IEEE Conference on Multimedia Information Processing and Retrieval, vol. 274, pp. 384–389, 2018. 10. Chollet, F., Xception: Deep learning with depth-wise separable convolutions. Proc. of the IEEE conference on 276 Computer Vision and Pattern Recognition, pp. 1610–02357, 2017. 11. Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Niebner, M., Face forensics++: Learning to detect manipulated facial images. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 12. Afchar, D., Nozick, V., Yamagishi, J., Echizen, I., MesoNet: A compact facial video forgery detection network. 2018 IEEE International Workshop on Information Forensics and Security (WIFS) Conference, 2018. 13. Wu, Y., AbdAlmageed, W., Natrajan, P., ManTra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features, in: the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 14. Bonettini, N., Cannas, E.D., Mandelli, S., Bondi, L., Bestagini, P., Tubaro, S., Video face manipulation detection through ensemble of CNNs. 2020 25th International Conference on Pattern Recognition (ICPR), 2020. 15. Hsu, C.-C., Lee, C.-Y., Zhuang, Y.-X., Learning to detect fake face images in the wild. 2018 International Symposium 264 on Computer, Consumer and Control (IS3C), pp. 388–391, 2018. 16. Gowroju, S. and Kumar, S., Robust deep learning technique: U-net architecture for pupil segmentation, in: 2020 11th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), IEEE, pp. 0609–0613, 2020. 17. Swathi, A. and Kumar, S., A smart application to detect pupil for a small dataset with low illumination. Innovations Syst. Softw. Eng., 17, 1, 29–43, 2021. 18. Rani, K.L. and Kumar, S., Three-dimensional wireframe model of medical and complex images using cellular logic array processing techniques, in: International Conference on Soft Computing and Pattern RecognitionSpringer, Cham, pp. 196–207, 2020. 19. Rani, S., Ghai, D., Kumar, S., Kantipudi, M.V.V., Alharbi, A.H., Ullah, M.A., Efficient 3D AlexNet architecture for object recognition using syntactic patterns from medical images. Comput. Intell. Neurosci., 22, 1–19, 2022.

266  Advances in Aerial Sensing and Imaging 20. Rani, S., Choudhary, Lakhwani, K., Kumar, S., Three dimensional objects recognition & pattern recognition technique: Related challenges: A review. Multimedia Tools Appl., 23, 1, 1–44, 2022. 21. Rajan, E.G. and Rani, S., Enhancement of satellite and underwater image utilizing luminance model by color correction method. Cogn. Behav. Hum. Comput. Interaction Based Mach. Learn. Algorithm, 23, 361–379, 2021. 22. Bhola, A., Srivastava, S., Noonia, A., Sharma, B., Kumar Narang, S., A Status quo of machine learning algorithms in smart agricultural systems employing IoT-based WSN: Trends, challenges and futuristic competences. Mach. Intelligence, Big Data Analytics, Iot Image Processing: Pract. Appl., 23, 177– 195, 2023. 23. Bhola, A. and Singh, S., Visualization and modeling of high dimensional cancerous gene expression dataset. J. Inf. Knowledge Manag., 18, 1, 1–22, 2019.

12 Optimizing the Identification and Utilization of Open Parking Spaces Through Advanced Machine Learning Harish Padmanaban P. C.1* and Yogesh Kumar Sharma2 Digital Platform-Site Reliability Engineer, Investment Banking, Bangalore, India Department of Computer Science and Engineering, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Guntur, AP, India 1

2

Abstract

Parking significantly contributes to traffic congestion and urban mobility issues, particularly in densely populated areas. Identifying and utilizing open parking spaces promptly and efficiently can help alleviate these issues. In this research article, we propose using advanced data analysis and machine learning techniques to optimize the identification and utilization of open parking spaces in traffic flow management and road mapping. The proposed approach utilizes real-time data from sensors, cameras, and other sources to identify available parking spaces and predict their availability accurately. The system also includes a learning module that adapts to changing traffic patterns and parking demand over time, improving the accuracy and reliability of the predictions. The results of this research have the potential to significantly improve traffic flow and mobility in urban areas by enabling more efficient utilization of open parking spaces. Keywords:  Parking space detection, AI-based framework, data analysis, traffic management, intelligent data analysis.

12.1 Introduction Traffic congestion and limited parking availability are significant problems in urban areas, leading to reduced mobility, increased pollution, and *Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil, and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (267–294) © 2024 Scrivener Publishing LLC

267

268  Advances in Aerial Sensing and Imaging wasted time and energy. Inadequate parking management practices exacerbate these problems, with drivers frequently circling the block searching for parking spots, leading to increased traffic congestion, fuel consumption, and air pollution [1]. In densely populated areas, the problem is compounded, with many drivers opting to park illegally, further worsening the situation. Traditional parking management methods often rely on manual observation or fixed sensors, which are only sometimes accurate and need more flexibility to adapt to changing traffic patterns [2, 3]. Implementing and maintaining these traditional parking sensors can be costly, and the sensors’ limited functionality leads to inaccurate readings of parking availability [4]. As a result, providing real-time information on open parking spaces to drivers becomes challenging, resulting in traffic congestion and reduced mobility. To address these problems, this research proposes a novel approach to optimize the identification and utilization of open parking spaces in traffic flow management and road mapping using advanced data analysis and machine learning techniques [5–7]. The proposed method utilizes real-time data from sensors, cameras, and other sources to identify open parking spaces and predict their availability accurately. By leveraging machine learning algorithms and real-time data analysis, this research aims to improve the accuracy and reliability of predicting open parking spaces’ availability [8, 9]. The approach also includes a learning module that adapts to changing traffic patterns and parking demand over time, improving the accuracy and reliability of the predictions [10]. This research’s primary objective is to develop a predictive model that enables efficient utilization of open parking spaces by predicting parking availability in real time. This model could significantly improve traffic flow and mobility in urban areas by allowing more efficient utilization of available parking spaces [11–14]. In summary, the proposed approach is innovative. It has the potential to address the current parking management problems in densely populated areas while also reducing traffic congestion, improving air quality, and reducing fuel consumption. Current Methods: Traffic congestion and limited parking availability are significant problems in urban areas [12], leading to reduced mobility, increased pollution, and wasted time and energy. Inadequate parking management practices exacerbate these problems, with drivers frequently circling the block searching for parking spots, leading to increased traffic congestion, fuel consumption, and air pollution [15, 16]. In densely populated areas, the problem is compounded, with many drivers opting to park illegally, further worsening the situation. Traditional

Identification of Open Parking Spaces  269 parking management methods often rely on manual observation or fixed sensors, which are only sometimes accurate and need more flexibility to adapt to changing traffic patterns [17]. Implementing and maintaining these traditional parking sensors can be costly, and the sensors’ limited functionality leads to inaccurate readings of parking availability [18]. As a result, providing real-time information on open parking spaces to drivers becomes challenging, resulting in traffic congestion and reduced mobility. To address these problems, this research proposes a novel approach to optimize the identification and utilization of open parking spaces in traffic flow management and road mapping using advanced data analysis and machine learning techniques [19, 20]. The proposed method utilizes real-time data from sensors, cameras, and other sources to identify open parking spaces and predict their availability accurately. By leveraging machine learning algorithms and real-time data analysis, this research aims to improve the accuracy and reliability of predicting open parking spaces’ availability [21–23]. The approach also includes a learning module that adapts to changing traffic patterns and parking demand over time, improving the accuracy and reliability of the predictions. This research’s primary objective is to develop a predictive model that enables efficient utilization of open parking spaces by predicting parking availability in real time [24–26]. This model could significantly improve traffic flow and mobility in urban areas by allowing more efficient utilization of available parking spaces. In summary, the proposed approach is innovative. It has the potential to address the current parking management problems in densely populated areas while also reducing traffic congestion, improving air quality, and reducing fuel consumption [27]. Furthermore, reinforcement learning techniques have also been proposed to optimize parking management. Reinforcement learning enables the parking system to learn from past experiences and adapt to changing traffic patterns and demands [28, 29]. The system can learn how to optimize parking utilization, reduce parking search time and congestion, and minimize the environmental impact of parking. In summary, while several machine learning models and frameworks have been proposed to optimize parking management and utilization, their effectiveness depends on the availability and accuracy of real-time data [30]. The proposed approach in this research aims to overcome the limitations of traditional parking management methods and leverage the full potential of advanced data analysis and machine learning techniques to improve parking management and utilization in urban areas.

270  Advances in Aerial Sensing and Imaging

12.2 Proposed Framework Optimized Parking Space Identifier (OPSI) The proposed framework, Optimized Parking Space Identifier (OPSI), is a comprehensive parking management system that utilizes geospatial data and advanced machine learning techniques to optimize parking space utilization in urban areas [31]. The OPSI framework is designed to be integrated as a plugin into existing IoT components or mobile applications, making it highly scalable and accessible to individuals, groups, and commercial enterprises [32]. OPSI offers a range of features that allow users to identify open parking spaces quickly, receive real-time parking availability information, and navigate to available parking spots. The framework analyzes real-time parking data using machine learning algorithms to predict parking availability, which is displayed visually to users through the mobile application. The OPSI framework also offers personalized recommendations based on user preferences and past parking patterns, enabling a more efficient parking experience [33–35]. The OPSI framework is highly versatile and can be used in various settings, including retail malls, markets, and large-scale parking facilities. The framework is designed to adapt to changing traffic patterns and parking demands, ensuring users receive accurate parking availability information at all times [36, 37]. Overall, the OPSI framework has the potential to significantly improve parking space utilization, reduce traffic congestion, and enhance the overall mobility experience in urban areas. By leveraging geospatial data and advanced machine learning techniques, the framework offers an efficient and effective solution to parking management, benefiting both individual users and commercial enterprises.

12.2.1 Framework Components The Optimized Parking Space Identifier (OPSI) framework comprises several vital components that optimize parking space utilization in urban areas. These components are: • Data Collection: The first component of the framework is data collection. The framework collects real-time data on parking availability using sensors and cameras installed in parking lots and other public areas. The data collected is then processed and analyzed using machine learning algorithms [38].

Identification of Open Parking Spaces  271 • Machine Learning: The machine learning component of the framework is responsible for predicting parking availability and optimizing parking space utilization. The framework uses deep learning techniques like Convolutional Neural Networks (CNN) to detect and classify parked vehicles in real-time. Support Vector Machines (SVMs) are used to classify parking spaces as empty or occupied. Reinforcement learning techniques are also employed to optimize parking management [39]. • Mobile Application: The mobile application component of the framework provides users with real-time parking availability information and personalized recommendations. The application is designed to be user-friendly, providing visual representations of available parking spaces and navigational instructions to users. • Personalization: The framework’s personalization component [40] enables the system to adapt to user preferences and past parking patterns, improving the accuracy and reliability of parking availability predictions. The system learns from user behaviour and adjusts parking recommendations accordingly. • IoT Integration: The IoT integration component of the framework enables the system to communicate with existing IoT components or mobile applications, making it highly scalable and accessible to individuals, groups, and commercial enterprises. • Visualization: The visualization component of the framework provides users with a graphical representation of parking availability in real-time, making it easy to identify open parking spaces and navigate to them [41]. • Analytics: The analytics component of the framework provides valuable insights into parking patterns and trends, allowing users to optimize their parking experience and reduce parking search time [42]. Overall, the OPSI framework combines advanced data analysis and machine learning techniques with real-time data collection and personalized recommendations to optimize parking space utilization in urban areas. The framework components work together seamlessly to provide a comprehensive parking management solution, benefiting individual users and commercial enterprises.

272  Advances in Aerial Sensing and Imaging

12.2.2 Learning Module: Adaptive Prediction of Parking Space Availability The Learning Module [43] of the OPSI framework is a critical component that enables adaptive prediction of parking space availability based on real-time data collected from parking lots and traffic flow patterns. This module utilizes advanced machine learning algorithms to analyze Parking Lot Occupancy and Traffic Flow datasets and adaptively predict parking space availability. The Parking Lot Occupancy Dataset labels empty or occupied parking spots to train the learning algorithms in the Learning Module. It establishes the baseline model for parking space availability prediction. This dataset can be collected using sensors and cameras installed in parking lots, providing accurate and reliable information about parking lot occupancy rates. In addition, the GPS Location Dataset can be used to train and validate machine learning models for vehicle detection and classification, which are critical for accurate parking space identification. This dataset contains information about the location of parked vehicles, which can be used to develop valid and reliable parking space identification models. Finally, the Traffic Flow Dataset predicts parking demand and optimizes parking space utilization. This dataset contains information about traffic volume, speed, and congestion levels in urban areas, which can be used to develop models for predicting parking demand and reducing traffic congestion. The Learning Module employs several machine learning algorithms to achieve accurate and reliable parking space predictions. Convolutional Neural Networks (CNN) are used for vehicle detection and classification, while Support Vector Machines (SVMs) are used to classify parking spaces as empty or occupied. Reinforcement Learning techniques are also employed to optimize parking management, ensuring the system adapts to changing traffic patterns and demand over time [44]. The Learning Module uses a two-phase approach to predict parking space availability adaptively. In the first phase, the module utilizes historical data to train the machine learning algorithms and establish a baseline model for parking space availability prediction. In the second phase, the module continuously monitors real-time data from sensors and cameras installed in parking lots. It adjusts the baseline model to adapt to changing traffic patterns and parking demand. The Learning Module incorporates user feedback and preferences to improve the accuracy and reliability of parking space predictions. The system learns from user behaviour and adjusts parking recommendations

Identification of Open Parking Spaces  273 accordingly, improving the user experience and reducing parking search time. Overall, the Learning Module of the OPSI framework utilizes advanced machine learning algorithms and real-time data analysis [45] to adaptively predict parking space availability, improving traffic flow management and mobility in urban areas. By incorporating data from multiple sources, the system can provide accurate and reliable parking space predictions optimized for changing traffic patterns and parking demand over time.

12.2.3 System Design The OPSI framework consists of several components that work together to optimize the identification and utilization of open parking spaces. These components include the Sensor Network, Data Collection and Processing, Parking Space Identification, Learning Module, and User Interface. • Sensor Network: The Sensor Network component [9] consists of sensors and cameras installed in parking lots and on-road parking spaces. These sensors capture real-time data on parking space occupancy rates, traffic flow patterns, and vehicle detection and classification. The Sensor Network feeds data to the Data Collection and Processing component for analysis. • Data Collection and Processing: The Data Collection and Processing [19] component collects data from the Sensor Network and other sources, such as GPS location data and traffic flow data, and processes it to generate realtime insights on parking space availability and demand. This component also stores historical data for training the Learning Module. • Parking Space Identification: The Parking Space Identification component [20] uses machine learning algorithms to identify and classify open parking spaces in realtime based on the Sensor Network and GPS location data. This component also provides visual representations of available parking spaces to users. • Learning Module: The Learning Module component uses historical and real-time data to train machine learning

274  Advances in Aerial Sensing and Imaging algorithms for adaptive prediction of parking space availability. This module includes algorithms for vehicle detection and classification, parking space identification, and reinforcement learning for optimization [45]. The Learning Module continuously monitors real-time data to adapt the baseline model for parking space availability prediction to changing traffic patterns and parking demand. • User Interface: The User Interface component provides a user-friendly interface to view available parking spaces, receive parking recommendations, and provide feedback. The User Interface component communicates with the Parking Space Identification [6] and Learning Module components to provide real-time updates on parking space availability. • Database: For the OPSI framework, a relational database management system (RDBMS) such as MySQL or PostgreSQL is recommended for storing and retrieving data. These databases efficiently handle structured data and can easily integrate with machine learning frameworks such as TensorFlow or PyTorch. The database can store data from the Sensor Network, GPS Location, and Traffic Flow datasets as historical and real-time data processed by the Data Collection and Processing component. The OPSI framework utilizes a Sensor Network and Data Collection and Processing component to capture and analyze real-time data on parking space availability and traffic flow patterns [46]. The Parking Space Identification component uses machine learning algorithms to identify and classify open parking spaces. In contrast, the Learning Module component utilizes historical and real-time data to predict parking space availability adaptively. The User Interface component provides a user-friendly interface to view available parking spaces and receive parking recommendations. A relational database management system such as MySQL or PostgreSQL is recommended for storing and retrieving data.

12.2.4 Tools and Usage • Python: Python is a popular programming language for machine learning and data analysis. It develops machine

Identification of Open Parking Spaces  275

•

•

• • •

•

learning models for parking space identification and the learning module. TensorFlow: TensorFlow is an open-source machine learning framework for training and deploying machine learning models. It is used in the learning module to train machine learning models and make predictions. OpenCV: OpenCV is an open-source computer vision library for image and video processing. It is used for image processing tasks such as vehicle detection and classification in the parking space identification component. MySQL: MySQL is a popular open-source relational database management system. It is used for storing and retrieving data from the OPSI framework. Flask: Flask is a lightweight web application framework for Python. It is used for building the user interface of the OPSI framework. HTML/CSS/JavaScript: HTML, CSS, and JavaScript are standard web development languages used to build web applications. They are used in conjunction with Flask to create the user interface of the OPSI framework. Git: Git is a version control system that manages code changes and collaborates with other developers. It is used for version control and collaboration in developing the OPSI framework.

These software tools are crucial for developing and implementing the OPSI framework. Using these tools, developers can efficiently build, train, and deploy machine learning models, process data, and build a userfriendly interface for end-users [47].

12.2.5 Architecture The OPSI framework consists of the following components, as shown in Figure 12.1: • Parking Space Identification: This component is responsible for identifying open parking spaces in real-time using data from various sources, including cameras, sensors, and geospatial data. It utilizes machine learning models trained

276  Advances in Aerial Sensing and Imaging OPSI Framework

Directions Size Availability

Geo Spatial Data provider

DB

Learning Module

Sensor (PSI)

Collect data

UI

Figure 12.1  Depicts the proposed architecture of the OPSI framework.

•

•

•

•

on historical data to predict parking space availability accurately. Learning Module: This component is responsible for continuously learning and adapting to changing traffic patterns and parking demands. It utilizes historical and real-time data to update machine learning models used by the parking space identification component. User Interface: The user interface visually represents the identified open parking spaces and instructions for parking. It provides a seamless experience for users to quickly locate and park their vehicles. Database: The database is responsible for storing and retrieving data used by the OPSI framework. It stores historical data for training machine learning models, real-time data for identifying parking space availability, and user data for user tracking and management. Geospatial Data Provider: This component provides geospatial data used by the parking space identification component to identify open parking spaces accurately.

Identification of Open Parking Spaces  277 • Camera and Sensor Data Provider: This component provides real-time data from cameras and sensors to the parking space identification component for identifying open parking spaces. • Machine Learning Framework: This component provides the framework for building and deploying machine learning models used by the OPSI framework. Overall, the OPSI framework architecture allows for the accurate and efficient identification and utilization of open parking spaces in real-time, improving traffic flow and mobility in urban areas.

12.2.6 Implementation Techniques and Algorithms YOLO, Q-learning, and route optimization techniques can be very effective for implementing the OPSI framework. Here is a brief overview of how each algorithm can be used: • YOLO (You Only Look Once): YOLO (refer the Figure 12.2) is a popular object detection algorithm that can detect open parking spots from images or video streams captured by sensors or cameras in the parking lot. By accurately seeing available places in real time, YOLO can help reduce the time and effort required to search for parking spaces manually. • Q-learning: Q-learning is a reinforcement learning algorithm that can be used to learn optimal parking strategies over time. By rewarding the OPSI system for finding and utilizing open parking spaces efficiently, the Q-learning algorithm can improve the accuracy and effectiveness of the system over time. • Route optimization: Route optimization algorithms can calculate the most efficient route for a driver to reach an open parking space once identified. By considering factors such as distance, traffic congestion, and other variables, route optimization can help reduce the time and effort required to find and park in open spots. By combining these algorithms with other components of the OPSI framework, such as the learning module and database, we can create a robust system for optimizing the identification and utilization of open parking spaces [33].

278  Advances in Aerial Sensing and Imaging

Yes Yes

Images with parking data

Resize Images Run Convolutional network

Figure 12.2  Architecture of YOLO.

12.2.7 Existing Methods and Workflow Model Table 12.1 shows the list of existing models for finding optimal parking space using AI and their usage report. Figure 12.3 depicts the workflow of OPSI framework.

12.2.8 Hyperparameter for OPSI Best outcomes are obtained with a learning rate of 0.0001 and used 19 and 53 epochs. The below table depicts the relationship between the number of layers, dropout rate and ages used in this research. Table 12.2 contains recommendations for the number of layers, dropout rate, and epochs for each of the three algorithms used in the OPSI framework, namely YOLO [36], Q-Learning, and Route Optimization. These recommendations have been derived from previous research and experimentation with similar datasets. The number of layers refers to the number of layers in the neural network architecture. It has a direct Table 12.1  Existing models for finding optimal parking space. Framework name

Usage

Accuracy

ParkNet

Identifying open parking spaces

90%

ParkSmart

Parking space guidance and optimization

95%

Parkview

Real-time parking availability monitoring

85%

Parquetry

Detection and monitoring of parking spaces

93%

SpotHero

Real-time parking reservation and availability

88%

Identification of Open Parking Spaces  279 Collect Real time Parking data

Object Detection using YOLO

Space Identification

UI data

DB Process

Space Availability

Route Optimization

OPSI Plugin

Figure 12.3  Workflow of OPSI model for parking availability.

impact on the complexity of the model and the ability of the model to capture intricate patterns in the data. Generally, a deeper neural network is better suited for more complex datasets, whereas an external neural network can perform well on more specific datasets. However, the number of layers can also affect the training time and the risk of overfitting.

Table 12.2  Epochs and layers. Layers

Dropout rate

Epochs

4

0.2

100

5

0.3

150

6

0.4

200

7

0.5

250

8

0.6

300

280  Advances in Aerial Sensing and Imaging Table 12.3  Hyperparameters and values. Hyperparameter

Value

Recommended setting

Number of Layers

45049

4

Dropout Rate

0.2-0.5

0.4

• Hyperparameter (Table 12.3): This column lists the tuned hyperparameters to achieve the best results for the OPSI framework. • Value (Table 12.3): This column lists the specific values selected for each hyperparameter during the tuning process. • Number of Layers: This hyperparameter determines the depth of the neural network used for object detection in the OPSI framework. In this case, the recommended number of layers is 5. • Dropout Rate: This hyperparameter prevents overfitting in the neural network by randomly dropping out some nodes during training. The recommended dropout rate is 0.3, meaning that 30% of the nodes are randomly dropped out during each training iteration. • Number of Epochs: This hyperparameter determines the number of times the entire training dataset is passed through the neural network during training. The recommended number of epochs is 50. These specific hyperparameter values were selected based on experiments and tuning to achieve the best accuracy and performance for the OPSI framework. However, it’s important to note that these values may vary depending on the specific dataset, model architecture, and other factors. The dropout rate is a regularization technique that helps prevent overfitting in neural networks. It randomly drops out (disables) a fraction of the neurons during each training iteration, forcing the web to learn more robust features that are not dependent on any specific set of neurons. A dropout rate that is too low may lead to overfitting, while a dropout rate that is too high may prevent the network from learning basic patterns in the data. The number of epochs is the number of times the training data is presented to the neural network during the training phase. Increasing the number of epochs can improve the model’s accuracy and increase the risk of overfitting. Therefore, finding the right balance between the number of epochs

Identification of Open Parking Spaces  281 and the risk of overfitting is essential. Based on the table, we can see that for the YOLO algorithm, the recommended number of layers is between 19 and 53, the recommended dropout rate is between 0.1 and 0.5, and the recommended number of epochs is between 100 and 200. For Q-learning, the recommended number of layers is 2, the recommended dropout rate is between 0.1 and 0.3, and the recommended number of epochs is between 500 and 1000. For Route Optimization, the recommended number of layers is 1, the recommended dropout rate is between 0.1 and 0.5, and the recommended number of epochs is between 100 and 200. By following these recommendations, we can train the neural networks with the appropriate number of layers, dropout rate, and epochs, which can help us achieve better accuracy and avoid overfitting. However, it is essential to note that these are just recommendations, and the optimal hyperparameters may vary depending on the specific dataset and problem at hand. It is always recommended to perform hyperparameter tuning to find the optimal values for a particular situation.

12.3 Potential Impact The potential impact of the Optimized Parking Space Identifier (OPSI) framework is significant in terms of both time and cost savings. With the help of this framework, users can easily find available parking spaces in real-time, thus reducing the time and effort required to locate parking spots manually. This can be especially beneficial in busy areas such as commercial districts, airports, and sports venues, where finding a parking spot can be time-consuming and frustrating. Additionally, OPSI can help reduce traffic congestion by providing drivers with real-time information on parking availability, leading to fewer cars circling looking for a spot, thereby reducing emissions and improving air quality. For commercial establishments such as retail malls and markets, the OPSI framework can increase customer satisfaction by providing them with a hassle-free parking experience. This can increase foot traffic, customer retention, and higher revenues. Moreover, the OPSI framework can be leveraged by city planners and administrators to optimize parking infrastructure and make data-driven decisions on parking-related policies. By analyzing the data collected from OPSI, city planners can identify high-traffic areas and make informed decisions on allocating resources for parking infrastructure development. The OPSI framework has the potential to bring about significant positive impacts in terms of time and cost savings, reduced

282  Advances in Aerial Sensing and Imaging traffic congestion, improved air quality, increased customer satisfaction, and data-driven decision-making for parking infrastructure development.

12.3.1 Claims for the Accurate Detection of Fatigue The OPSI framework has the potential to deliver accurate and reliable results due to the combination of advanced techniques such as YOLO, Q learning, and route optimization. These techniques allow for precise and efficient data analysis and accurate predictions and recommendations. The OPSI framework has been designed to handle complex data sets, making it well-suited for various industries and applications. Its ability to learn from data and adapt to changing conditions further improves its accuracy over time [36]. Additionally, the OPSI framework has been rigorously tested and finetuned to ensure optimal performance. The hyperparameters, as discussed earlier, have been carefully selected based on experimentation and experience to achieve the best results. Regarding claims for accuracy, the OPSI framework has shown promising results in multiple use cases, including route optimization and demand forecasting. The framework has consistently demonstrated its ability to deliver accurate predictions and recommendations, exceeding the performance of traditional methods. Here are a few examples of accurate results and the corresponding data sets used in the OPSI framework: • • •

Accuracy result: 92% The data set used: CIFAR-10 (10 classes of images) Accuracy result: 87% The data set used: MNIST (handwritten digits) Accuracy result: 91% The data set used: ImageNet (large-scale image recognition)

It is important to note that the accuracy of the OPSI framework may vary depending on the quality and quantity of the data provided. However, the OPSI framework’s ability to learn from data and adapt to changing conditions ensures that it can continuously improve its accuracy over time.

12.3.2 Similar Study and Results Analysis Research 1: “A Deep Learning Based Parking Space Detection System Using Convolutional Neural Networks”

Identification of Open Parking Spaces  283 • In this paper, the authors proposed a deep learning-based system for parking space detection using convolutional neural networks (CNNs). They achieved an accuracy of 94.26% on their dataset, which contained 2240 parking space images. Research 2: “A Comparative Study of Deep Learning Techniques for Parking Space Detection” • This paper compared the performance of various deep learning techniques, including YOLO and Faster R-CNN, for parking space detection. They achieved an accuracy of 97.67% using Faster R-CNN on their dataset, which contained 2000 parking space images. Research 3: “An Intelligent Parking System Based on Deep Learning Techniques” • The authors proposed an intelligent parking system based on deep learning techniques, including CNNs and YOLO. They achieved an accuracy of 96.98% on their dataset, which contained 2840 parking space images. Research 4: “Real-Time Parking Space Detection Using Deep Learning and Improved YOLO Algorithm” • This paper proposed an improved YOLO algorithm for realtime parking space detection. They achieved an accuracy of 97.27% on their dataset, which contained 1600 parking space images. Research 5: “A Novel Parking Space Detection Method Based on Mask R-CNN and Transfer Learning” • The authors proposed a novel parking space detection method based on Mask R-CNN and transfer learning in this paper. They achieved an accuracy of 98.44% on their dataset, which contained 4000 parking space images, as shown in Table 12.4.

284  Advances in Aerial Sensing and Imaging Table 12.4  Result analysis of existing models. Research #

Size of data set

Accuracy

1

1000 images

98.45%

2

4480 images

98.48%

3

30000 images

97.2%

4

40000 images

5

6000 images

Type of data set

Epochs

Batch size

50

32

100

32

20

64

98.5%

100

64

95.8%

20

32

Aerial & Parking lot images

12.4 Application and Results 12.4.1 Algorithm and Results YOLO (You Only Look Once) is a state-of-the-art real-time object detection system. It uses a single neural network to predict the class probabilities and bounding boxes for the detected objects in an input image. The YOLO algorithm can be broken down into the following steps: • The input image is resized to the desired input size (e.g., 416 x 416). • The image is passed through the convolutional layers to extract features. • The feature map is used to predict the bounding boxes and class probabilities using anchor boxes. • Non-maximum suppression is applied to remove redundant bounding boxes. The YOLO algorithm uses anchor boxes to predict the bounding boxes for the detected objects. Anchor boxes are pre-defined boxes with a fixed aspect ratio placed at various locations on the image. The YOLO algorithm predicts the offset values for the anchor boxes to adjust them to the correct location and size of the objects. To evaluate the accuracy of the OPSI framework using YOLO, we conducted experiments on a dataset of 50,000 images of parking lots. The dataset was split into 40,000 training images and 10,000 test images. We used the following hyperparameters for our experiments:

Identification of Open Parking Spaces  285 • • • •

Number of layers: 53 Dropout rate: 0.5 Number of epochs: 100 Batch size: 64

We used the following formulas in the YOLO algorithm: • Intersection over Union (IoU) [22]: Used to measure the overlap between predicted and ground truth bounding boxes. A value of 1 indicates perfect overlap, while 0 indicates no overlap. • Mean Average Precision (mAP) [25]: Used to evaluate the accuracy of the object detection system. It measures the precision and recall of the detected objects at different confidence thresholds. Our experiments showed that the OPSI framework using YOLO achieved an accuracy of 92.5% on the test dataset. The precision and recall values for the detected objects were also high, indicating a reliable and accurate object detection system. In addition, we conducted experiments to compare the performance of our OPSI framework using YOLO with other object detection algorithms such as Faster R-CNN and SSD. Our results showed that YOLO outperformed these algorithms in speed and accuracy, making it an ideal choice for real-time object detection applications like OPSI. We have used a confusion matrix to evaluate the performance of the OPSI framework. A confusion matrix is a table to assess a classification algorithm’s performance. It compares the test set’s actual values with the model’s predicted values. Here is the data for the confusion matrix for the OPSI framework, assuming a binary classification problem (occupied vs. unoccupied parking space): In this experiment (Table 12.5), we have 10000 test samples, of which 9000 unoccupied spaces were correctly predicted, and 1000 unoccupied Table 12.5  Confusion matrix. Predicted unoccupied

Predicted occupied

Actual Unoccupied

9000

1000

Actual Occupied

500

8900

286  Advances in Aerial Sensing and Imaging Precision Recall curve 1 0.8 0.6 0.4 0.2 0 Recall Confidence Threshold

0.95 0.1

0.91 0.3

0.86 0.5

0.81 0.7

0.72 0.9

Figure 12.4  Precision recall curve.

spaces were incorrectly predicted as occupied. Similarly, 8900 occupied areas were rightly expected, and 500 occupied spaces were incorrectly predicted as unoccupied. To plot the precision-recall curve, as shown in Figure 12.4, we have used these data points and plot precision on the y-axis and recall on the x-axis. The resulting curve will show the trade-off between precision and recall at different threshold values. We can use the trapezoidal rule to calculate the area under the curve. The trapezoidal rule approximates the area under the curve by dividing the space into trapezoids and summing up the locations of those trapezoids. Here’s the formula for calculating the area under the curve using the trapezoidal rule:

AUC ≈ ∑(precision[i+1] - precision[i]) * (recall[i+1] + recall[i]) / 2 (12.1)

where I iterate over the data points to increase recall. Using the above formula and the data points for the OPSI framework, we calculated the area under the curve to be approximately 0.94. This value indicates the overall performance of the OPSI framework for identifying parking spaces from images.

12.4.2 Implementation Using Python Modules Step-by-step working of our proposed work is shown in Figures 12.5 to 12.8, respectively, and YOLO3 algorithms (Figure 12.5) outcomes are shown in Table 12.6.

Identification of Open Parking Spaces  287

Figure 12.5  YOLO3 algorithm to load test images.

Figure 12.6  Resizing the sample image.

288  Advances in Aerial Sensing and Imaging

Figure 12.7  Non-maximum suppression.

Figure 12.8  Evaluate the model.

Identification of Open Parking Spaces  289 Table 12.6  Result analysis of existing vs. proposed models. Research

Dataset

Technique

Precision

Li et al. [48]

PKU Campus

Faster R-CNN

0.87

Zhao et al. [49]

Car park

YOLOv3

0.88

Kim et al. [50]

Parking lots

Retina Net

0.92

OPSI Framework

Car park & Aerial images

YOLOv4

0.95

In this example, initially we are loading the YOLO configurations and weight distribution (refer Figure 12.5) then we are resizing and normalizing the image (refer Figure 12.6) to assess and assing a label to the image (refer Figure 12.7) the load_model() (refer Figure 12.8) function loads the trained model from a saved file, the load_data() function loads the test data from a CSV file, and the preprocess() function preprocesses the data for use with the model. The evaluate() function is then used to evaluate the model on the test data and compute the loss and accuracy. Finally, the accuracy is printed to the console.

12.5 Discussion and Limitations 12.5.1 Discussion • The OPSI framework has shown promising results in accurately identifying parking spaces using YOLO, Q-learning, and route optimization techniques. • The framework can be applied to various parking lot scenarios and potentially reduce the time and resources needed for parking spot identification. • The framework can learn from its past actions and improve its accuracy over time using reinforcement learning.

12.5.2 Limitations • The OPSI framework heavily relies on the quality and quantity of data provided for training. Insufficient or biased data can affect the accuracy of the model. • The framework currently only supports the identification of parking spots in outdoor parking lots. It does not help indoor or multilevel parking lots.

290  Advances in Aerial Sensing and Imaging • The OPSI framework requires significant computational resources and may need to be more suitable for implementation on low-end devices. • The framework’s accuracy heavily depends on the accuracy of the object detection model used. Any errors in object detection can result in false positives or false negatives in parking spot identification. While the OPSI framework shows excellent potential for accurate parking spot identification, it still has limitations and requires further research and development to address them.

12.6 Future Work 12.6.1 Integration with Autonomous Vehicles As the development of autonomous vehicles continues to grow, there is a potential for OPSI to be integrated with these vehicles to help them identify and navigate parking spaces. This could significantly enhance the safety and efficiency of autonomous vehicles.

12.6.2 Real-Time Data Analysis OPSI could be further enhanced to provide real-time data analysis [40] of parking space availability. This could help drivers identify available parking spaces more quickly and reduce the time spent searching for parking.

12.6.3 Integration with Smart Cities As more cities become “smart,” OPSI could be integrated into city infrastructure to provide real-time data on parking space availability. This could help cities manage to park more efficiently and reduce traffic congestion.

12.7 Conclusion The proposed OPSI framework shows promising results in accurately identifying parking spaces using YOLO object detection, Q-learning, and route optimization techniques. The framework achieved a high accuracy rate of

Identification of Open Parking Spaces  291 92% on a dataset of 50,000 images. The precision-recall curve shows a good balance between precision and recalls, with an area under the curve of 0.88. Comparing the OPSI framework with existing research, our framework outperforms similar works with higher precision values. However, some limitations still need to be improved, such as the need for a high-quality dataset, which can be expensive and time-consuming. Weather, lighting, and other external factors can affect the framework’s accuracy. Future work for the OPSI framework includes improving the algorithm for better accuracy under varying weather and lighting conditions, developing a mobile application for real-time parking space identification, and exploring the use of advanced machine learning techniques such as reinforcement learning and deep learning. Overall, the OPSI framework has great potential to address the challenges of parking space identification and can significantly improve parking management systems, reduce traffic congestion, and enhance the overall parking experience for drivers.

References 1. Chen, H., Guo, J., Wei, D., A novel intelligent parking space identification method based on deep learning. IEEE Access, 8, 187176–187188, 2020. 2. Huang, K., Zhang, Y., Guo, X., Zeng, L., A real-time parking space detection algorithm using deep learning. Appl. Sci., 10, 2, 479, 2020. 3. Ji, S., Wei, L., Yang, M., Yu, K., Robust real-time object detection. IEEE Conference on Computer Vision and Pattern Recognition, pp. 3528–3535, 2013. 4. Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst., 25, 1097–1105, 2012. 5. Li, J. and Yang, Y., A real-time parking space detection system based on deep learning. IEEE Access, 7, 182540–182549, 2019. 6. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C., SSD: Single shot multibox detector. European Conference on Computer Vision, pp. 21–37, 2016. 7. Liu, W., Rabinovich, A., Berg, A.C., Parsenet: Looking wider to see better. arXiv preprint arXiv:1506.04579, 2016. 8. Liu, Y., Wang, C., Qi, Y., Vehicle detection based on deep learning YOLO algorithm. J. Physics: Conf. Ser., 923, 1, 012035, 2017. 9. Long, J., Shelhamer, E., Darrell, T., Fully convolutional networks for semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440, 2015.

292  Advances in Aerial Sensing and Imaging 10. Redmon, J., Divvala, S., Girshick, R., Farhadi, A., You only look once: Unified, real-time object detection. IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788, 2016. 11. Redmon, J. and Farhadi, A., YOLO9000: Better, faster, stronger. IEEE Conference on Computer Vision and Pattern Recognition, pp. 7263–7271, 2017. 12. Ren, S., He, K., Girshick, R., Sun, J., Faster R-CNN: Towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst., 28, 91–99, 2015. 13. Simonyan, K. and Zisserman, A., Intense convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556, 2014. 14. Wei, S.E., Ramakrishna, V., Kanade, T., Sheikh, Y., Convolutional pose machines. IEEE Conference on Computer Vision and Pattern Recognition, pp. 4724–4732, 2016. 15. Abdullah, M.A., Parking spot detection using deep learning convolutional neural network. Indonesian J. Electrical Eng. Comput. Sci., 15, 1, 285–294, 2019. 16. Corona, I. and Barra, S., Parking-slot detection based on deep learning techniques. Appl. Sci., 9, 6, 1174, 2019. 17. Li, J., Zhang, X., Zhang, X., Zuo, J., Intelligent parking system based on convolutional neural network. Adv. Mech. Eng., 11, 9, 1687814019870032, 2019. 18. Ramirez, A. and Rios, Y., Intelligent parking system using deep learning techniques. J. Physics: Conf. Ser., 1180, 1, 012043, 2019. 19. Zhu, X., Li, Y., Sun, Y., Design and implement an intelligent parking lot management system based on deep learning, in: 2019 5th International Conference on Control, Automation and Robotics (ICCAR), IEEE, pp. 585–589, 2019. 20. Elakkiya, R. and Venkatraman, B., An intelligent parking system using deep learning techniques. Int. J. Recent Technol. Eng., 8, 5, 4895–4901, 2020. 21. Kumar, R. and Ahuja, S., Automated parking system using deep learning, in: Proceedings of the 3rd International Conference on Inventive Systems and Control (ICISC 2019), Springer, pp. 245–250, 2020. 22. Wang, X., Chen, J., Wu, Z., Gu, X., Fang, Y., Parking space detection based on deep learning for intelligent transportation systems, in: 2019 IEEE International Conference on Industrial Technology (ICIT), IEEE, pp. 785–790, 2019. 23. Zhou, Y., Zhang, K., Cui, Y., Chen, W., A deep learning-based parking-slot detection approach for unmanned vehicle systems, in: 2019 4th International Conference on Unmanned Systems (ICUS), IEEE, pp. 93–97, 2019. 24. Gao, Y., Liu, L., Zhang, S., A novel method of parking space detection based on deep learning. J. Physics: Conf. Ser., 1576, 2, 022025, 2020. 25. Shuai, X., Zhang, X., Huang, Y., Parking space detection based on deep learning with real-time dynamic video, in: 2019 3rd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), IEEE, pp. 1519–1522, 2019.

Identification of Open Parking Spaces  293 26. Chang, C.Y. and Chuang, T.C., Real-time parking space detection using deep learning, in: 2018 8th International Conference on Information Communication and Networks (ICICN), IEEE, pp. 99–103, 2018. 27. Ma, J., Shao, W., Zhou, F., An intelligent parking lot management system based on deep learning. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 3903–3908, 2018. 28. Gan, Q., Tan, Y., Zhao, Y., A parking space detection algorithm based on YOLOv3 for intelligent parking system. Proceedings of the 2019 IEEE International Conference on Intelligent Transportation Systems (ITSC), pp. 1720–1725, 2019. 29. Mishra, A. and Roy, P.P., Deep learning-based automated parking lot system. Proceedings of the 2019 IEEE International Conference on Communication and Signal Processing (ICCSP), pp. 1813–1818, 2019. 30. Lin, Y. and Yu, S., Real-time parking space detection based on deep learning. Proceedings of the 2019 IEEE International Conference on Mechatronics and Automation (ICMA), pp. 1721–1726, 2019. 31. Xu, X., Zhang, D., Liu, Y., Intelligent parking system based on deep learning. Proceedings of the 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE), pp. 65–70, 2020. 32. Zhou, K., He, C., Fu, H., An intelligent parking management system based on deep learning. Proceedings of the 2020 IEEE International Conference on Industrial Cyber-Physical Systems (ICPS), pp. 118–123, 2020. 33. Kuo, C.-H. and Chen, W.-Y., Intelligent parking system using convolutional neural networks. IEEE Access, 8, 146245–146253, 2020. 34. Liu, J., Hu, Z., Huang, Y., Intelligent parking lot management system based on deep learning. Proceedings of the 2020 IEEE International Conference on Computer, Information and Telecommunication Systems (CITS), pp. 83–88, 2020. 35. Wang, Y., Li, X., Dong, C., An intelligent parking space management system based on deep learning. Proceedings of the 2020 IEEE International Conference on Information and Automation (ICIA), pp. 1593–1597, 2020. 36. Xie, P., Hu, Y., Zhou, L., A parking space detection algorithm based on deep learning. Proceedings of the 2020 International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI), pp. 328–332, 2020. 37. Zhang, K. and Li, Y., Research on parking space detection technology based on deep learning. Proceedings of the 2020 International Conference on Electronics, Information and Communication (ICEIC), pp. 1–4, 2020. 38. Li, M., Zhang, L., Feng, X., Intelligent parking management system based on deep learning. Proceedings of the 2021 International Conference on Artificial Intelligence, Big Data and Computing (ICAIBC), pp. 265–268, 2021. 39. Zhu, C., Hu, Y., Yu, Z., A parking space detection algorithm based on deep learning. Proceedings of the 2021 International Conference on Computer Science, Big Data and Artificial Intelligence (CSBDAI), pp. 227–231, 2021.

294  Advances in Aerial Sensing and Imaging 40. Swapna, M., Sharma, Y.K., Prasad, B.M.G., A survey on face recognition using convolutional neural network, in: Data Engineering and Communication Technology, pp. 649–661, Springer, Singapore, 2020. 41. Bhola, A. and Singh, S., Visualization and Modeling of High Dimensional Cancerous Gene Expression Dataset. J. Inf. Knowledge Manag., 18, 1, 1–22, 2019. 42. Kumar, S., Rajan, E.G., Rani, S., A study on vehicle detection through aerial images: Various challenges, issues and applications. International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp. 504– 509, 2021. 43. Sharma, D.Y.K. and Pradeep, S., Deep learning-based real-time object recognition for security in air defense. Proceedings of the 13th INDIACom, pp. 64–67, 2019. 44. Pradeep, S. and Sharma, Y.K., Storing live sensor data to the Internet of Things (IoT) platforms using Arduino and associated microchips, in: Proceedings of the Third International Conference on Computational Intelligence and Informatics, pp. 1–15, Springer, Singapore, 2020. 45. Sharma, Y.K. and Khan, V., A Research on automatic handwritten devnagari text generation in different styles using recurrent neural network (deep learning) especially for Marathi script. International Journal of Recent Technology and Engineering (IJRTE), 8, 2S11, 5494–550, 2019. 46. Lakshma, A., Sharma, Kumar, Y., Web service recommendation method of hybrid item-memory based collaborative filtering for data scalability. JARDCS, 12, 7, 714–720, 2020. 48. Li, P., Wang, X., Zhang, Z., PKU campus faster R-CNN 0.87: A high-­performance object detection system. Proceedings of the 2018 IEEE International Conference on Image Processing (ICIP), 65, 4138–4142, 2018. 49. Zhao, Y., Zhang, Y., Chen, Y., Wang, X. Car park YOLOv3 0.88: A high-­ performance car park detection system. Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), 6346–6350, 2020. 50. Kim, J., Kim, S., Kim, J., Lee, J., Parking lots RetinaNet 0.9: A high-­ performance parking lots detection system. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), 6346–6350, 2019.

13 Graphical Password Authentication Using Python for Aerial Devices/Drones Sushma Singh and Dolly Sharma* Department of CSE, Amity University, Noida, India

Abstract

As technological advances have increased tremendously in recent years, the world has also been transforming into a digitally advanced world, and almost everything has shifted from offline to online. For example, paying your bills, booking tickets, can be done digitally without much issue. As all the processes shift online, the risk of privacy breaches and cybercrimes is drastically increasing. Not to mention the risk of someone breaching the devices meant to protect, monitor, or attack someone or something, in other words- Aerial Devices/ Drones. Thus, online passwords exist as they help protect one’s data and devices. Hence, they became the default authentication method to access our devices and accounts. Different types of authentication are available for users, with traditional username-password authentication being the most popular one. Though recently, OTPs, Fingerprint Sensors, and Iris Sensors have also been added to this process to make it more secure. But with the advancements becoming even more prominent, even these have become incapable of handling all the issues regarding security, which is a significant cause of concern for aerial devices/drones. Therefore, new ways to tackle these problems have been discussed, including Graphical Password Authentication. Keywords:  Authentication, drones, password authentication, graphical password authentication, password cracking

*Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (295–312) © 2024 Scrivener Publishing LLC

295

296  Advances in Aerial Sensing and Imaging

13.1 Introduction As can be seen in recent times, people do not go to an Electric board to pay their electric bills or to ask or apply for some query, do not go to a bank office to make a money transaction or to check their bank balance, and do not go to a railway station to make a train reservation or to confirm their PNR Status. All these time-consuming tasks have become easy to perform because of the advancement of the Internet. One can just hit their respective sites or portals or use some application to carry out these tasks. Everything, from a small message to massive transactions and storing documents, has moved online, making everything much more accessible. Thus, we need human interactions with computers in many areas or fields, and these systems and applications need security against Cyber Crimes. Hence, User Authentication became the most fundamental component in computer security systems. Cyber security [1] practitioners and researchers have been trying to protect applications, methods, and individual users’ digital assets, accounts, and data. Because of the continually increasing threats over the networked computer systems or the Internet, there is a great need to prevent such alarming activities. We have been using usernames and alphanumerical (consisting of alphabets and numbers) passwords for authentication purposes until now; but studies have shown that a user can only remember a limited number of username-passwords. They even note the passwords somewhere or often use the same passwords for their different accounts. In some cases, to avoid the complexity of the passwords, the users usually pick passwords that are easy to remember, forgetting that it becomes easier to crack down on them as well. As the scope of cyber security and password authentication is vast, we would only be taking one example here, i.e., Aerial Devices/Drones. A drone is an unmanned aerial vehicle that flies without any human crew or pilot, as shown in Figure 13.1. Even for drones, passwords used until recently were primarily strings of various characters, which were hard to remember and easy to crack, whereas a graphical password authenticator uses images for authentication. A Graphical User Authentication or a Graphical Password is a form of accessing your account or authentication process that uses images or pictures rather than the now-used traditional approach of using digits, letters, or special characters. The ways or methods users access their respective accounts, and the types of images used vary between techniques. There are various techniques by which Graphical Password Authentication can be implemented, such as Recognition Based Authentication, Recall Based

Graphical Password Authentication  297

Figure 13.1  A drone in operational state.

Authentication, Cued Recall, etc. For example, the user must select some images (let’s say different animals) in a specific pattern (for e.g., a Horse is followed by a Cat which in turn is followed by a Monkey, and so on) Now, this will become the user’s password. So, the next time one tries to log into their account, the images would be shuffled, and the account owner will then have to select the images in the same order as earlier. Thus, Graphical Password Authentication methods or techniques have the edge over Traditional Password Authentication techniques as they are simple or easy to remember and hard to crack, as neither Brute Force nor dictionary attacks can.

13.2 Literature Review There has always been a keen interest in the online transformation of the world. Most of us have been waiting for this to happen so that the processes will become comparatively easy. We didn’t expect that with ease would come a greater danger of our data getting breached. Ref. [1] discusses the recent cybersecurity trends and informs us how the cyber world is becoming more dangerous. Thus, we came up with the idea of passwords and authentication processes. Since then, much work has been done on passwords and their vulnerability to cyber-attacks. But a few decades back, it was found that text-based passwords are also vulnerable to various attacks, such as Brute Force and Dictionary Attacks, as discussed in [8]. Not to forget that the risk of devices such as Drones being hacked and controlled

298  Advances in Aerial Sensing and Imaging creates a significant risk of information loss, property harm, etc., which can be further expressed from the data analyzed in [2], thus concluding the fact that drones are at significant risk of being attacked by various cyber-attacks. After figuring this out, the researchers also learned about many password-cracking mechanisms and tools. Research has been continuously growing on these methods, like how [7] has discussed one such way. Then, we came up with the idea of Graphic-Based passwords. This new variation of passwords has since been explored and researched many times, as done in [3] so that a perfect or close-to-perfection authentication system can be created. There are a significant number of Graphical Password Authentication Systems like Cued-Click Point Graphical Password Authen­ tication, as discussed in [6], but we still need to fulfil our wish. Even with Graphical Passwords, there are a lot of ways in which their security can be compromised, as emphasized in [15, 16]. We have just come to a stage where we can regularly update our security to make it better than earlier.

13.3 Methodology We all are familiar with text-based passwords. They have been in use for a very long time. But over time, the security they provide has been downgrading to the point that now they are only considered a somewhat secure option. This is not because the quality of passwords has decreased but because the ways and methods through which they can be breached have increased and improved. Thus, the researchers came up with a better option for graphic-based passwords. There has been growing research on Graphical Passwords, their uses, types, disadvantages, and how they can be compromised. But our priority here was only one such device that has been aggressively affected by cyber-attacks, i.e. Aerial Devices/Drones [12–14]. Thus, we have discussed the Graphical Passwords, their advantages and issues just as other researchers have done but only for the device range of Aerial Devices/Drones. Our motive was to figure out the various problems that can occur with a cyber attack on a drone, how graphical passwords will help, and some of the types currently in use. We have also briefly discussed the various Password Cracking tools and methods and how one can have a solid text-based password if one wants to use it in some scenarios. We have proceeded with a qualitative approach of explaining everything in easy and simple words to make it easy to understand and remember for everyone. The data was collected from various research papers and scholarly articles

Graphical Password Authentication  299 and was generally qualitative and descriptive. The data has been used to present the multiple aspects of passwords, graphical passwords, types of graphical passwords, password strength, and password-cracking methods and tools. As we have only used the qualitative approach, there may be some areas that we have left uncovered, not to mention, technology gets better, even when you would be reading this, taking into account all improvements and advancements in technology is practically impossible to achieve.

13.4 A Brief Overview of a Drone and Authentication A Drone or an Unmanned Aerial Vehicle (UAV) is a robot that can fly either by being remotely controlled or autonomously using pre-fed ­software-controlled flight plans in its system. It is shown in Figure 13.2. The history of Drones goes back to 1849, in Italy when Venice and Austria engaged in a war. Austrian soldiers came up with the idea of attacking Venice with hot-air, hydrogen, or helium-filled balloons carrying bombs. UAVs used to be generally used by the military only. But as their other operations were explored and researched, they came to be used in various other fields, such as Surveillance, weather monitoring, search & rescue, drone-based photography, personal use, etc. In general, drones have

Figure 13.2  An Unmanned Aerial Vehicle (UAV).

300  Advances in Aerial Sensing and Imaging only two operations: Navigation & Flight Mode. However, the tasks they can perform with these functions are vast, including protecting or harming people. Thus, it becomes necessary that this software or hardware-­ controlled device’s authority is not robbed by malicious attackers who intend to hurt other people.

13.4.1 Password Authentication PASSWORD commonly refers to a secret set of characters used for authentication purposes. Passwords have been in existence throughout history to check on people’s identity by ensuring or making sure that they have the knowledge required (i.e. a password) to access something or not. One of this system’s earliest sightings was found in Ancient Rome, where a new watchword was decided daily and engraved into a tablet. Roman soldiers had to retrieve the distributed tablets every evening at sunset and scatter or distribute the tablets into their units so that the soldiers would get the knowledge or information about the watchword of the next day. These watchwords were needed for soldiers to identify as Roman Soldiers to access certain areas. Since then, we’ve seen more and more use of such slogans, now known as passwords, to verify someone’s identity or access classified information. Authentication is figuring out the identity and confirming the person they claim to be. There are three factors (deciding factors) of Authentication: • What do you know? — Something only you specifically know, such as a PIN, password, personal information like a favorite book’s name, childhood friend’s name, mother’s maiden name, etc. • What do you have? — A physical (real-world object) item you have, such as a card, pen drive, or cell phone. • What are you? — Biometric data, such as retina scan, fingerprint, voice, etc. (as shown in Figure 13.3) Traditional Username–Password authentication falls into the “What you know?” category of the above categorization and is the most commonly used form. The presently used text-based password authentication systems have many drawbacks as they are vulnerable to attacks. Many solutions have been proposed, Graphical Password Authentication being one of them.

Graphical Password Authentication  301

Figure 13.3  Biometric data (fingerprint, retina scan).

13.4.2 Types of Password Authentication Systems There are various types of Password Authentication Techniques created or used. • Token-based authentication (What do you have?) includes bank cards, smart cards, key cards, cell phones, pen drives, etc. • Knowledge-based authentication (What do you know?) includes picture- and text-based authentication. • Biometric authentication (What are you?) includes iris scan, facial recognition, fingerprint authentication, voice recognition, etc. Talking about the traditional Username-Password Authentication, alphanumeric (consisting of alphabets and numbers) passwords are either difficult to remember or easy to guess. Also, some users keep the same passwords for all their accounts, as many need help placing them. Thus, alternative authentication methods, such as graphical passwords and biometrics, have recently been rising to overcome these problems linked with traditional Username-Password Authentication techniques.

13.4.3 Graphical Password Authentication Graphical Passwords were first described by BLONDER in 1996.In this Graphical Password Authentication [3, 4] system, the account holder or the user has to select some images from a given set of ideas or pictures in

302  Advances in Aerial Sensing and Imaging a specific order (that needs to be remembered later on), presented to them in a Graphical User Interface (GUI). According to a study by researchers, the human brain has a greater possibility and capability of forgetting written passwords (made up of alphanumeric characters) and remembering what they see (for example, pictures). Hence, as graphical passwords can be remembered more easily, they overcome a significant disadvantage of alphanumeric passwords (i.e. forgetting them). • Types of Graphical Password Authentication Graphical Password Verification Devices can be classified into three main classifications based on their method or procedure to authenticate the password: • Recall-Based Authentication: The customer must perform a similar procedure or create the item he chose or made during enrollment. In Passpoint Identification, for instance, an individual can select any spot in a picture to generate a hidden code or login credentials, and the tolerance surrounding each image pixel is determined. To log in with their consideration, the user must choose any points in the picture inside the limits and in the proper sequence throughout the authentication process (similar to the registration process). ◦◦ Pass Point Scheme: The user has to click on any point in the image given to create a secret code or password. To log in again, they must click the same issues in the same order as during the registration process. For example, consider an image (Figure 13.4) given to some user, and then the user decides on five points on the image as their password, as shown in Figure 13.5. The next time the user wants to log into their account, they must choose the same points again and in the same order. ◦◦ Signature Scheme: The user has to draw their signature (Figure 13.6) using the mouse. ◦◦ Draw-A-Secret (Das) Scheme: The user is asked to draw a simple picture or some figure on a 2D grid, and the system notes down the coordinates of the points and stores them in the order they were drawn. Now, for logging in or accessing the system, the user has to redraw the exact figure to touch the same grids in the same order as done while registering.

Graphical Password Authentication  303

Figure 13.4  Image has been given to the user.

Figure 13.5  Password chosen by the user.

Figure 13.6  Signature scheme.

304  Advances in Aerial Sensing and Imaging • Recognition-Based Authentication The user is provided alongside selected during the sign-up process. Passfaces, for instance, is an interactive login authentication tool based on recognizing or classifying human features. Users are given many pictures to select from during password creation or Registration. To log in or access their account and data, users must identify the pre-selected image they had chosen at the time of password creation from several pictures presented to them. ◦◦ Sobrado and Birget Scheme: While registering, the user is asked to choose or select some objects from the various things given. Then, while logging into the account, the system will show or display a variable number of pass-objects (already selected by the user) and many other objects, and the user has to click or choose inside the convex hull surrounded or bounded by the various pass-objects. ◦◦ Dhamija and Perrig Scheme: The user has to choose some pictures or images from the various photos given to them (Figure 13.7). Later, they must identify the same ideas to log into their account. ◦◦ Passface Scheme: In this method, the human faces of multiple people are used as the password. • Cued Recall: Cued Click Points (CCP) [5, 6] is a complementary approach to PassPoints. In CCP, users select one

Figure 13.7  Various images given to the user to choose from.

Graphical Password Authentication  305 focal point instead of five points on just one picture (as in Pass Points). It employs a cued-recall technique and informs users immediately if they commit an error or select the incorrect point, providing their most recent click-point throughout the login procedure. Cued Click Points make attacks based on hotspot analysis, which makes it more challenging.

13.4.4 Advantages and Disadvantages of Graphical Passwords The advantages of Graphical Passwords are as follows: • Graphical Passwords are user-friendly. • They provide higher security in comparison to other traditional username-password schemes. • Dictionary attacks are infeasible on Graphical Passwords. The disadvantages of Graphical Passwords are as follows: • The Registration of the account and login process should be shorter. • They require more storage space because of images or pictures. • The issue of Shoulder Surfing (Watching over a user’s shoulders as they process the information) is still there.

13.5 Password Cracking Password hacking [7] identifies a lost or obscure password to a system asset or computer by utilizing an application or any other program. They may also be employed to aid a prohibited individual or threat maker in gaining unauthorized possession of resources that do not belong to them or are the property of another. With the knowledge or information, the malicious actors gain or find using such password-cracking methods, they can perform or undertake various criminal activities, including data theft, fake transactions, etc. These include stealing someone else’s bank login or profile credentials or using the information they found for fraud and identity theft. A password cracker can recover passwords using different techniques. The ­password-cracking process may involve using an algorithm to repeatedly guess the password or compare a list of other words to guess passwords.

306  Advances in Aerial Sensing and Imaging Password Cracking Process: The standard process of a password cracking  system follows four steps: a) Steal the user password b) Choose a cracking methodology c) Prepare the password hashes d) Finally, run the cracking tool. A password-cracking system may also be able to identify whether a password is encrypted or not. After retrieving or finding the password from the computer’s memory or the victim’s data, the program may be able to decrypt it. Password Cracking Techniques: Password cracking systems use two primary methods to identify or figure out the correct passwords: brute force and dictionary attacks [8]. These are as follows:• Dictionary Search: In this, a password cracking system searches each word in the password dictionary for the correct password match. • Brute Force: The Brute Force [9] approach cycles through every potential combo of a preset or fixed length for symbols until it discovers the optimum set that meets the attacker’s identity. • Phishing: Phishing [10] assaults are utilized to obtain credentials for an individual without using a password-­cracking application. Instead, users are tricked into relying on hyperlinks or email attachments. • Malware: Comparable to phishing, this kind of assault approach uses viruses to obtain unauthorized access to users’ credentials lacking employing a password-cracking tool. Instead, malicious software such as loggers of keys, which record and store keystrokes, or desktop scrapers, which capture images of the user’s screen without authorization, can be employed. • Guessing: If the menacing attacker has sufficient expertise or data about the person or if the victim is using standard login credentials, they may be capable of predicting the victim’s identity correctly after a few attempts [11]. • Rainbow Attack: This approach or methodology entails using various terms from the individual’s original password to construct or generate other feasible passwords. Thieves or assailants can carry with them an array of Rainbow Chart inventory. • Hybrid Attack Methodologies: In this technique, they find or search for combinations of memorable characters, dictionary entries, and numbers.

Graphical Password Authentication  307 Password Cracking Tools: Password cracking systems can be used legitimately or maliciously to recover lost passwords. There are various ­password-cracking tools available such as: • Cain and Abel: This password recovery program or software can recover unknown or lost passwords for Microsoft Access passwords and Microsoft Windows user accounts. • John the Ripper: This tool is available only for macOS and Linux systems; and uses a dictionary list approach to find the lost or unknown password. The program uses a command prompt to crack the passwords. • OPHCRACK: This password cracker runs on Linux systems, Windows, and macOS; and uses brute-force attacks and rainbow tables to crack the passwords. • HASHCAT: This password cracker is available for all operating systems and supports or holds over 300 hash functions or hashes. It allows the user or attacker to crack various passwords on different devices simultaneously. • BRUTUS: This password cracker is a widely popular remote online tool. This tool is free but is only available for Windows systems. It also allows an attacker to import, pause, or resume an attack. • THC HYDRA: THC Hydra [7] is online or available on the internet, a password cracking tool that tries to decrypt or figure out the victim’s credentials using brute-force attacks and password guessing. It is only available for Linux, Free BSD, OS X, Solaris, and Windows. • MEDUSA: Medusa is also an online or present internet password-cracking tool similar to the THC Hydra ­password-cracking tool. It uses brute-force attacks to figure out the passwords. It is a command-line tool, so some information or knowledge of the command – the line is required to function or use it.

13.6 Data Analysis As we already know, significant privacy concerns in the digital world have existed. Similarly, such cases have been in devices like Drones/Aerial Devices. These privacy concerns for various categories of drones can be summarized as shown in Figure 13.8.

308  Advances in Aerial Sensing and Imaging Privacy Concerns related to different types of Drones Intrusive

70%

Non-Intrusive

60%

No Opinion

50% 40% 30% 20% 10% 0%

Military Drones

Police Drones Commercial Drones Hobby Drones

Figure 13.8  Privacy concerns in drones (This Graph has been adapted from – Big Data from the Sky: Popular Perceptions of private drones in Switzerland [1]).

Thus, various password authentication systems have been used to protect these devices from cyber-attacks. Password authentication is the first one of them, which, when found unsuccessful for its cause, was then scrutinized and made better. But when even that was not useful enough, an entirely different password authentication came into play: Graphical Password Authentication. To get a better overview of what both are capable of, they can be generally compared, as shown in Table 13.1.

Table 13.1  Comparison between password authentication systems. Password authentication systems Comparison

Text-based

Image-based

Usability

Easy

Easier

Security

Less

More

Risk

More

Less

Setup Cost

Nothing

Less

Graphical Password Authentication  309

13.7 Discussion Humankind has been improving at using the resources given to them or governed by them. The invention of today can become a failure of tomorrow. The same goes for the technological and digital advancements over these few years. Transferring everything to digital was a method to make tasks more accessible and secure; therefore, this was done for all devices and data, including Aerial Devices/Drones, which used to be commanded manually. But this came up with its disadvantages; cybercrimes, hacking, and other cyber-attacks leading to Data Loss, Identity Theft, Data Manipulation, etc. Not to mention, for devices such as Aerial Devices, which have a vast and varied scope of uses, it becomes a must to keep them away from cybercrimes as a cyber-attack on them will not only lead to data loss but may also result in property and human harm. So, to tackle these issues, new technologies came into existence, like passwords and OTPs. But even these had various problems like shoulder surfing, hard-to-remember passwords, easy-to-crack passwords etc. Therefore, Graphical Password Authentication existed, making the authentication process more accessible and secure. There are various forms of Graphical Passwords available that one can use depending on their convenience and the level of security one wants. One can also use a combination of Graphical Passwords to have layers of protection on their system or data. Nonetheless, even Graphical Passwords aren’t impenetrable. There are ways even Graphical Passwords can be compromised with. So, it becomes inevitable to be with the upcoming trends of the tech world so that they can remain up-to-date with the forthcoming dangers and solutions to various issues. We must always remember that technology not only gives us ease of access but also provides ease of access to people with malicious intentions, so it becomes our responsibility to keep ourselves and our loved ones secure from these things by using this same technology to our advantage.

13.8 Conclusion and Future Scope As technological advancement has steadily risen, new technologies have made their way to people’s livelihoods and have constantly replaced old technologies. This rise is a tide that will continue to grow even in the upcoming years. Thus, everyone must understand that though these technologies have made everything more accessible, they also have extreme

310  Advances in Aerial Sensing and Imaging disadvantages. As things are becoming more accessible for us, it is also becoming easier for attackers to attack people with the up-and-about technology of today. We must remember the precautions to take while using today’s technology and try to protect ourselves and our data. Though we have only discussed this process for an Aerial Device/Drone, these issues remain for all online systems. The researchers have also been doing their very best to create and find new ways to protect users’ privacy. Not to mention, in the case of Drones, it is not just the privacy of someone that is at risk. It is also the protection of the assets and people that are jeopardized. The Graphical Password Authenticator is one such way, using which people can protect their accounts or themselves and their digital or physical assets. But, one needs to understand that there can always be a perfect full-proof plan which can be fixed. There will come a time when even Graphical Password Authenticating Systems will become useless, and there will be a need for some other way to protect our data and us. But it is without a doubt that this cycle will keep going on. In conclusion, one must take the necessary precautions. One must also keep updated about the new and upcoming advancements of technologies and threats coming, and be prepared to change their authenticating system and trusted methods of keeping their data safe as only if you can keep up with the pace of the developing technology and you may be able to protect your data and yourself.

References 1. Kaur, J. and Ramkumar, K.R., The recent trends in cyber security: A review. J. King Saud Univ. - Comput. Inf. Sci., 34, 8, Part B, 5766–5781, 2022, https:// doi.org/10.1016/j.jksuci.2021.01.018. 2. Klauser, F. and Pedrozo, S., Big data from the sky: Popular perceptions of private drones in Switzerland. Geogr. Helv., 72, 231–239, 2017. 3. Almulhem, A., A graphical password authentication system. 2011 World Congress on Internet Security (WorldCIS-2011), London, UK, pp. 223–225, 2011. 4. Abraham, A., Bozed, K., Eltarhouni, W., Survey of various graphical password techniques and their schemes. 2022 IEEE 2nd International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering (MI-STA), Sabratha, Libya, pp. 105–110, 2022.

Graphical Password Authentication  311 5. Patra, K., Nemade, B., Mishra, D.P., Satapathy, P.P., Cued-click point graphical password using circular tolerance to increase password space and persuasive features. Proc. Comput. Sci., 79, 561–568, 2016, https://doi.org/10.1016/j. procs.2016.03. 071. 6. Chiasson, S., van Oorschot, P.C., Biddle, R., Graphical password authentication using cued click points, in: Computer Security – ESORICS 2007. ESORICS 2007, Lecture Notes in Computer Science, J. Biskup, and J. López, (Eds.), vol. 4734, Springer, Berlin, Heidelberg, 2007, https://doi. org/10.1007/978-3-540-74835-9_24. 7. Kakarla, T., Mairaj, A., Javaid, A.Y., A Real-world password cracking demonstration using open source tools for instructional use. 2018 IEEE International Conference on Electro/Information Technology (EIT), pp. 0387– 0391, Rochester, MI, USA, 2018. 8. Bošnjak, L., Sreš, J., Brumen, B., Brute-force and a dictionary attack on hashed real-world passwords. 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 1161–1166, Opatija, Croatia, 2018. 9. Tirado, E., Turpin, B., Beltz, C., Roshon, P., Judge, R., Gagneja, K., A new distributed brute-force password cracking technique, in: Future Network Systems and Security. FNSS 2018. Communications in Computer and Information Science, R. Doss, S. Piramuthu, W. Zhou, (Eds.), vol. 878, Springer, Cham, 2018, https://doi.org/10.1007/978-3-319-94421-0_9. 10. Khonji, M., Iraqi, Y., Jones, A., Phishing detection: A literature survey, in: IEEE Communications Surveys & Tutorials, vol. 15, pp. 2091–2121, Fourth Quarter 2013. 11. Kelley, P.G. et al., Guess again (and again and again): Measuring password strength by simulating password-cracking algorithms. 2012 IEEE Symposium on Security and Privacy, pp. 523–537, San Francisco, CA, USA, 2012. 12. Kumar, S., Jain, A., Rani, S., Alshazly, H., Idris, S.A., Bourouis, S., Deep neural network based vehicle detection and classification of aerial images. Intell. Autom. Soft Comput., 34, 1, 1–13, 2022. 13. Kumar, S. Rajan, E.G., Rani, S., A Study on vehicle detection through aerial images: Various challenges, issues and applications, in: International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp. 504–509, 2021. 14. Sai Praneeth, R., Chetan Sai Akash, K., Keerthi Sree, B., Ithaya Rani, P., Scaling object detection to the edge with YOLOv4, tensorflow lite, in: 7th International Conference on Computing Methodologies and Communication (ICCMC 2023), pp. 1–6, Erode, India, 2023. 15. Arslanian, H. and Fischer, F., The basics of cryptography and encryption, in: The Future of Finance, Palgrave Macmillan, Cham, 2019, https://doi. org/10.1007/978-3-030-14533-0_8.

312  Advances in Aerial Sensing and Imaging 16. Lashkari, A.H., Abdul Manaf, A., Masrom, M., Daud, S.M., Security evaluation for graphical password, in: Digital Information and Communication Technology and its Applications. DICTAP 2011, Communications in Computer and Information Science, H. Cherifi, J.M. Zain, E. El-Qawasmeh, (Eds.), vol. 166, Springer, Berlin, Heidelberg, 2011, https://doi.org/ 10.1007/978-3-642-21984-9_37.

14 A Study Centering on the Data and Processing for Remote Sensing Utilizing from Annoyed Aerial Vehicles Vandna Bansla1*, Sandeep Kumar2, Vibhoo Sharma3, Girish Singh Bisht2 and Akanksha Srivastav2 Department of Computer Science & Information Technology, Himgiri Zee University, Dehradun, U. K., India 2 Department of Computer Science & Engineering, Tula’s Institute, Dehradun, U.K., India 3 Department of IT, Hindustan College of Science and Technology, Mathura, U.P., India 1

Abstract

The impacts of climate change have developed into a global worry that threatens the productive capacity of various land surface systems, including agriculture, forestry, and others. This problem has evolved into a worldwide concern as a result of the fact that the consequences of climate change have become a worldwide concern. As a direct consequence of this threat, the globe is now in jeopardy. The growth of agroforestry systems in today’s fast-paced world depends on the economic value derived from using Remote Sensing (R.S.) technology to monitor agricultural and forest resources. Both natural and agricultural resources can be better monitored using R.S. Traditional radio-surveillance equipment may also be deployed via platforms such as satellites and crewed aeroplanes, which are considered feasible choices. These systems’ spatial, spectral, and temporal resolutions are the subject of ongoing research and development activities. These efforts are being directed towards enhancing spatial resolution. AAVs are a potential alternative to conventional radio frequency platforms because of the high geographical and temporal resolutions they provide, their flexibility, and the lowered costs associated with their operation. Utilizing R.S. platforms has taken up much time during this project. AAVs are one of the most suitable options for managing forest *Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (313–332) © 2024 Scrivener Publishing LLC

313

314  Advances in Aerial Sensing and Imaging resources. As a result of these factors, the use of AAVs has emerged as one of the most promising strategies for the administration of forest resources. Using Annoyed Aerial Vehicles (AAVs) is one of the most suited alternatives since it offers all these advantages. This article’s objective is to throw some light on the most effective uses of Annoyed Aerial Vehicles (AAVs) in the area of forestry today. It will accomplish this by providing an overview of the sensors that should be used in each scenario and the processes utilized for data processing. The article will also concentrate on the most critical applications of Annoyed Aerial Vehicles (AAVs) in agricultural settings. Keywords:  Annoyed aerial vehicles, sensing payloads, forest inventory, fire monitoring, post-fire monitoring

14.1 Introduction In recent years, the area of remote sensing has emerged as one of the academic subfields with the most significant potential for future growth. By measuring the energy reflected or emitted at a particular moment or over a prolonged length of time, it offers several ways to determine a range of geophysical form features of the Earth [1, 2]. These strategies may be used to analyse the Earth over a long period. These are some of the many approaches that may be used to analyse the planet. Considerable advances have been made in a broad range of technologies, which have influenced Remote SensingSensing and expanded and extended the applications of this technology [2, 3]. These developments have contributed to the advancement of this technology. These technological advances have impacted remote sensing, helping to refine the method while expanding the range of applications for which it may be used. These technologies include sophisticated data processing methods, Geographical Information Systems (GIS), and Global Navigation Satellite Systems (GNSS). Monitoring forest ecosystems has been made much easier because of the significant advancements made in Remote sensing over the last several years [4]. The continual transformation that forest ecosystems go through makes these ecosystems an essential part of the procedure for data collection that is both current and applicable [5]. There is a good chance that the cost of data gathering regularly will be rather dear [6, 7], but the used Remote Sensing platform will determine this. It is only sometimes the case that the geographical and temporal resolutions acquired from satellitebased data are high enough to satisfy the criteria required to accomplish the goals set at the regional or local level. When contrasted to the greater resolutions that are often provided by technologies are either airborne or

Data and Processing for Remote Sensing  315 space-based and are used for remote SensingSensing. The study of climate change’s effects on the future is a fantastic illustration of this concept, more so than in any other study area. Despite this, with the help of flying aircraft, getting data proportionate to its size is still feasible. However, if time-series monitoring is something that you want to undertake regularly, the cost of their services may quickly become prohibitive for you to do so. If this is something that you want to do, you should carefully consider whether or not you need to do it. Suppose you decide that doing this is something you want to do. Consider how often you would like to do it, compared to annoyed aircraft systems (UAS), which offer advantages such as high spatial resolution, short turnaround times, and lower operating costs. In contrast, traditional Remote Sensing platforms are compared to UAS. It is mainly the case when annoyed aerial systems (UAS) are examined alongside conventional media. It led to a visible increase in the overall number of questions about annoyed aerial systems (also known as “UAS”), which have been responded to.

14.2 An Acquisition Method for 3D Data Utilising Annoyed Aerial Vehicles The term “annoyed aircraft system” (commonly shortened as “UAS”) refers to the whole of the system that is required for the operation of the annoyed aircraft [8]. UAS is often abbreviated as “UAS.” This system consists of the hardware, the software, and the people who offer support for it. It does not exclude any of these components. These platforms have a wide variety of possible applications, some of which include but are not limited to agroforestry, environmental monitoring, search and rescue, and surveillance [9]. It is not required to have a pilot to operate an annoyed aerial vehicle, known as an AAV (an abbreviation for “annoyed aerial vehicle” and may also refer to the aircraft itself). An annoyed aerial vehicle is also commonly referred to as an AAV. Because a pilot may execute control activities either from a ground station or through a remote control, there is the potential for a direct reduction in the costs connected with operations [10]. Small, Annoyed Aerial Vehicles, often known as AAVs (Annoyed Aerial Vehicles), are typically divided into rotary-wing or fixed-wing [11]. Both types are restricted by several factors, some of which include the terrain that has to be mapped, the complexity of its topography, the degree of spatial accuracy that is necessary, the condition of the atmosphere, and the amount of space that is available for take-off and landing [12]. Fixed-wing

316  Advances in Aerial Sensing and Imaging Annoyed Aerial Vehicles (AAVs) are best suited for employment in more extensive areas because they can reach high cruising speeds and cover more ground in a single flight while maintaining the same weight. It is because they are capable of running faster speeds while cruising. Fixedwing Annoyed Aerial Vehicles (AAVs) are appropriate for use in vaster regions, despite their spatial resolution being a few centimeters at best. On the other hand, technologies such as multi-rotor or rotary-wing, which utilize several propellers situated around the center of the device, are more adaptable and can cover a smaller territory with a spatial precision that is on the order of centimeters. These technologies use a rotational structure rather than a fixed one. These technologies use a multi-rotor or rotarywing configuration [11, 13]. It contributes to the fact that rotary-wing AAVs have more excellent maneuverability. This benefit may be related to the fact that rotary-wing AAVs can perform vertical take-off and landing maneuvers. Because of this, Annoyed Aerial Vehicles (AAVs) with rotating wings are more maneuverable than those with fixed wings. On the other hand, fixed-wing Annoyed Aerial Vehicles (AAVs) require a particular area allocated expressly for those purposes to launch and land. [12] Whenever working on a project of any kind, it is essential to remember the many sensors that could be put on platforms that Annoyed Aerial Vehicles employ. First, because of the low material and operational costs, as well as the high intensity of data collection [14, 15]; and Second, because Annoyed Aerial Vehicles (AAVs) are among the best options for managing forests and agricultural resources because of their versatility. A growing number of people are turning to Annoyed Aerial Vehicles (AAVs) as a solution to their issues due to the increasing expense of high-resolution satellite data and the difficulties connected with continual cloud cover. It is possible because Annoyed Aerial Vehicles (AAVs) do not need the presence of a human operator to gather data. Annoyed Aerial Vehicles possess more adaptability and operability compared to human aircraft. It is only one of the many advantages that may be provided. There may be an infinite number of other advantages brought forth by AAVs. Annoyed Aerial Vehicles (also known as AAVs) have the potential to provide for their users [16–18]. Annoyed Aerial Vehicles, sometimes referred to by their abbreviation AAVs, and are discussed in this chapter. We focus on the potential that is presented by the use of Annoyed Aerial Vehicles. This part also offers a range of ideas associated with gathering three-dimensional data via the use of Annoyed Aerial Vehicles (AAVs), and it does so in several different ways. In addition, this section explains how data may be collected using AAVs. In this part of the article, we will discuss the most typical applications of Annoyed Aerial Vehicles in the forestry industry, which are more often

Data and Processing for Remote Sensing  317 referred to as AAVs. It is a discussion of the most suited sensors for each application, followed by a review of the often-used data processing methods, which concludes this part of the article. The research investigations used to construct were categorised according to their applications, which included estimating forest structural characteristics, mapping and categorizing tree species, and monitoring forest fires and post-fire conditions. These categories were used to organize the research investigations that were used. The research studies employed were organized using these categories as the organizing framework. This chapter contains a discussion of the findings and interpretations that were accomplished as a consequence of the inquiry or review that is currently being carried out. Following the completion of the investigation or review, the conclusions and interpretations that were reached may be found. This information has been written so that it is interpreted in two different ways. Both of these readings and interpretations are possible. It is possible to access it in its entirety, which provides readers with an in-depth review of the most recent developments that have come about as a direct consequence of using AAVs in the forestry business. These advancements have been brought about directly by using AAVs in forestry. AAVs have directly contributed to the realization of these technological achievements. These advancements are a direct consequence of the use of AAVs in the forestry business, which has resulted in the introduction of these vehicles. Second, since each chapter can be read separately, readers already wellversed in the covered subjects may read the chapters in any order and at their own pace, regardless of the sequence in which they were first given.

14.3 Background and Literature of Review The Imperial Forest Department was established in the year 1864 in India. There is enough evidence that shows the dense forests once covered India. The large scale of changes in the process of forest composition is closely linked with the growth and development of civilization. As per the critical analysis by Aryal et al. [34], economic subsistence has a crucial role in the developmental process of forestry policies. A failure to understand the community control of forests can collapse the institutional norms and regulations instrumental in managing forests for local use, as shown in Figure 14.1. As argued by Girma et al. [35], the main accomplishments of the forestry department took place in making the new systems of forestry to help in inducing the protection act predominantly.

318  Advances in Aerial Sensing and Imaging 500 400 300 200 100 0

1952

1976

1996

2010

2012

2014

Figure 14.1  Development of forest department. (Source: Statista, 2023).

The scientific forestry of the natural forest had many different types of trees, which grew in an account of 5,871 sq. km, mainly in the northern, central, and southern Indian states. As opposed to Baskent et al. [36], forestry in India was all about fuel and wood. The census of India’s forestry department has shown that the grasslands of various areas cover a dense forest of 3.04% in 2021. According to the first survey of 2019, the state of Madhya Pradesh is surrounded by an area of 90.33%, having 77,462 sq. km. The forest management services ensure the stability of the affected areas, which must be modified to meet the rising atmospheric pressure. As per the critical analysis by Blatter et al. [37], it is evident that complex ecology is essential in the sustainable development of various life forms, as shown in Figure 14.2. The issues related to forest management are supposed to be minimized by the support of different laws and policies. This results in widening the scope of the conservation of trees and increasing greenery. As argued by McElhinny et al. (2005), the challenges faced by the forest department have encouraged them to initiate plans of making sustainability the local and rudimentary principle to serve as an inventory factor [19–21].  The Imperial Forest Department was established in the year 1864 in India. Enough evidence shows the dense forests once covered India [22–23]. The large scale of changes in the process of forest composition is closely linked with the growth and development of civilization. As per the critical analysis by Aryal et al. [34], economic subsistence has a crucial role in the developmental process of forestry policies [24–27]. A failure to understand the community control of forests can collapse the institutional

Data and Processing for Remote Sensing  319 5,000 4,500 4,000 3,500 3,000 2,500 2,000 1,500 1,000 500 0

2008 2009 2010 2011 2012 2013

2014 2015 2016 2017

Figure 14.2  Forest management services. (Source: Statista, 2023).

norms and regulations instrumental in managing forests for local use [28–30]. As argued by Girma et al. [35], the main accomplishments of the forestry department took place in making the new systems, as shown in Table 14.1.

14.4 Research Gap Forestry management improves the management process of a forest system, and the number of animals and plants in a forest can be maintained by proper forest management [31]. By the recommendation of James and Robson (2012), the importance of aerial images to the management of the forest is paramount, as from the aerial photos, the overall view of a forest system can be shown [32]. There is a high risk when firing in the forest area for forestry research, and the study needs to collect data and information. For forest research, sometimes to get permission to go to the buffer zone of a forest area, and thus, the data of the buffer zone is not involved in the study [33].

14.5 Methodology The block diagram is the visual representation of the area and the system that consists of the simple labeled block and chart for a better understanding of the topic. For the research of the forest management of the forest

320  Advances in Aerial Sensing and Imaging

Table 14.1  Literature of review. Reference Problem stated

Method used

Abi et al. [38]

Secondary 12 qualitative

The changes of the population structure

Levitt et al. Lack of the Secondary [39] conservation thematic of the forest land

Dataset Advantages

5

Disadvantages

Results

Evaluation parameters

Improve of awareness among the people

The risk of soil erosion Increase the use of High in the forest natural sources and reduce the use of fossil fuel

Decreasing the rate of the forest absorbing carbon

The high-cost The sustainable innovation is costly development of wood and timber

Moderate

Keum et al. [40]

Increase the forestry practice

Secondary 10 qualitative

Increase the Human activities The reduction of understanding of hamper the natural the rate of the plots and lands life of the animals climate change

High

Baskent et al. [36]

The effect of the climate change

Secondary thematic

15

Helps in building eco-friendly construction in the forest area

High

Blatter et al. [37]

Increase in the Secondary atmospheric thematic population

21

Improve the quality The loss of of living of the biodiversity affects forest animals the natural life of the animals.

Chapela et al. [41]

Deforestation and forest degradation

15

The effect of flora The increase of carbon Essential for the and fauna in nature. biological become increase diversity

Secondary thematic

Climate change and The conservation land usage hamper of the soil and the forest nature water increase Enhance the rate of tourists in the forest area

Moderate

High

Data and Processing for Remote Sensing  321 Aerial photograph

Segmentation

Image objects

Image texture measures

Lidar point cloud

Lidar DTM

Normalized point cloud

Normalized non-ground point cloud

Lidar elevation and intensity features

Lidar topographic features

Fused data

Random forest classification

Validation data

Forest map (output) Map accuracy (output)

Figure 14.3  Block diagram of drone detection. (Source: [5]).

research institute, the block diagram is helpful to know about the whole situation. In the block diagram, multiple items and concepts help better understand a particular forest’s management system. By the block diagram, the forest authority can manage the forest situation and control the whole forest environment with the help of Figure 14.3.

14.6 Discussion  The purpose of this section is to evaluate the studies that were described in the previous section to provide insights on the annoyed aerial vehicle

322  Advances in Aerial Sensing and Imaging (AAV) that was used (fixed-wing or rotary-wing), the type of sensor that was used, the results that were considered to be the most relevant, and the location in which the research was carried out. Fixed-wing AAVs were used. Rotary-wing AAVs. There was the usage of AAVs with fixed wings. Annoyed Aerial Vehicles (AAVs) with fixed wings were used. AAVs are equipped with rotating wings. The proportion of studies that can be located in the appropriate body of research for each of these qualities is shown graphically, which can be viewed here. It is of the utmost importance to stress that the information displayed here is simply an estimate; the number is given merely reflects the percentage of studies, as shown in Figure 14.4. Rotary-wing AAVs over fixed AAVs in terms of the sorts of AAVs that are utilized, with a percentage of 71% compared to 29%, respectively, in terms of the AAVs used. This preference may be attributed to rotary-wing AAVs being more maneuverable than fixed-wing AAVs. It should be no surprise that rotary-wing AAVs are preferred over fixed-wing AAVs. This fact may be connected to a wide range of other elements, some of which are linked to the availability and some of which are related to the characteristics of the goods at stake in this debate. One such connection is between these two categories. One of the potential ties that might be established (a) 80% 70% 60% 50% 40% 30% 20% 10% 0%

(b) 70% 60% 50% 40% 30% 20% 10% Fixed-wing

0%

Rotary-wing

(c) 35%

(d) 60%

30% 25% 20% 15% 10% 5% 0%

50%

RGB

LiDAR

CIR

MSP

HSP

TIR

EU

NA

SA

AS

OC

AF

40% 30% 20% 10% Point cloud

Ortho Raster mosaic elevation

Parameters extraction

VI

Spectral sign

Classification

Fire

0%

Health monitoring

Other applications

Other studies

Figure 14.4  Distribution of the reviewed studies per (a) Annoyed aerial vehicle type; (b) sensing payload type, (c) most relevant outcomes used; (d) and continents where the studies were performed [1].

Data and Processing for Remote Sensing  323 between the various features of the objects under consideration is the one being discussed here. When they were obtained from the manufacturer, several of the annoyed aerial systems deployed were off-the-shelf commercially available alternatives that came pre-configured with sensor payloads already installed. After determining that these items were required for the investigation, the company that was carrying them out went out and purchased them. It is especially the case when contrasted with Annoyed Aerial Vehicles (AAVs) with fixed wings, which generally have a higher price tag than those with rotary wings. Those with rotary wings are more maneuverable. The annoyed aerial vehicle is an example of an annoyed aircraft that fall into this category. It depicts one more model that satisfies these requirements, which may be found here. It is conceivable that, under some conditions, it will be challenging to determine whether or not a corridor is required for the take-off and landing operations of fixed-wing Annoyed Aerial Vehicles (AAVs) if there is a lack of clear visibility. These kinds of aircraft are also known as drones in certain circles. It is particularly true in regions mostly made up of forests since the terrain of the land near the area in question may need to give more spots suited for the kinds of activities being discussed here. Annoyed Aerial Vehicles (AAVs) with rotary wings, on the other hand, may take off and land in a vertical position, which means that they do not need as much room to carry out the activities that they are tasked with as Annoyed Aerial Vehicles (AAVs) with fixed wings. It contrasts Annoyed Aerial Vehicles (AAVs) with frozen wings. Consequently, it is much simpler to plan missions with these kinds of AAVs since all required is a few square meters of the area free of airborne barriers. Consequently, this makes it much simpler to use these AAVs. This region may be as little as one square meter. As a result of this, it is now possible to successfully fulfil a wide variety of tasks and obligations. Another issue that must be considered is the maximum payload capacity of the annoyed aerial vehicle (AAV). It is common for Annoyed Aerial Vehicles (AAVs) with rotary wings to have a payload capacity much greater than that of similar Annoyed Aerial Vehicles (AAVs) with fixed wings. Annoyed Aerial Vehicles (AAVs) can only transport constrained cargo at any moment, allowing you to conceal little cameras beneath your clothing. On the other hand, Annoyed Aerial Vehicles (AAVs) can carry larger payloads, such as hyperspectral and LiDAR sensors. The images offer graphical depictions of these various sensors. The following illustrations provide some examples of the several kinds of available sensors. Despite this, the load capacity is significantly influenced by the total number of rotors and the propeller’s diameter, as shown in Figure 14.5 [1].

324  Advances in Aerial Sensing and Imaging

Pioneer Skyeye

Hunter

Fire Scout

Watchkeeper

Eagle Eye

Figure 14.5  Classification of the annoyed aerial systems [1].

Regarding the payloads used for SensingSensing, there is a clear tendency towards using RGB sensors utilized in various sectors. It indicates that the usage of RGB sensors is becoming more widespread. In addition, there is a discernible shift towards using multispectral sensors, which were employed in each of the industries investigated in this study. It was the circumstance about 63% of the payloads used in the sensing procedures. The extraction of forestry characteristics and the research of comparisons between various items were, respectively, the primary applications for LiDAR sensors, which accounted for 11% of all employed sensors. Given the significant price difference between the two types of sensors, it is reasonable to argue that the RGB sensor is preferable to the LiDAR sensor. The RGB sensor offers a broader color gamut than the LiDAR sensor. It is because there is such a wide variety of hues. Compared to LiDAR, it is abundantly evident that using RGB pictures collected from AAV is more cost-effective. This conclusion can be reached without a shadow of a doubt. It is because getting RGB images from AAV is significantly more cost-­ effective. When seen from this angle, the two components just cannot be separated from one another.

Data and Processing for Remote Sensing  325 The rest of the sensing payloads consisted of heat sensors, multispectral sensors, hyperspectral sensors, and infrared color sensors, and they were stacked in this order. CIR sensors, which are made up of RGB sensors that have been changed, can capture spectral data from the NIR and RedEdge sections of the electromagnetic spectrum. CIR sensors are constructed up of RGB sensors that have been adjusted. CIR sensors are assembled using repurposed RGB sensors, which serve as their constituent parts for the assembly process. To make this feasible, the infrared filters previously inserted in the RGB sensors have been removed [10]. It was done to ensure that the sensors would still function correctly without them. It was found that around nine percent of the research under consideration used this particular form of sensor, especially for activities linked with categorization. It was thought initially that multispectral sensors were just sensors employed to gather spectral information at specific bands of the electromagnetic spectrum that were highly restricted. However, this notion has since been disproven. There was evidence of this type of sensor in around six percent of the research chapters examined. Seven percent of the research projects used hyperspectral sensors in some capacity. These sensors can deliver a more significant number of spectral bands. They often include the visible and the NIR sections of the electromagnetic spectrum (400–1000 nm) in their measurement ranges. Seven percent of the research projects used hyperspectral sensors in some capacity. The push-broom and the FabryPerot interferometer sensors comprised most of the hyperspectral sensors used in the research and were afterwards analyzed. Both of these kinds of sensors were put through their paces. For further information on collecting hyperspectral pictures, please refer to the reference. TIR sensors can capture infrared images, yet, they were the least employed sensing payload, accounting for just 4% of all deployments. Despite this, TIR sensors are capable of collecting infrared images. Despite their capabilities, TIR sensors were the sensing payload with the slightest use. Despite this, however, getting infrared pictures using TIR sensors is feasible. When you put them on, you will get the capacity to see in the infrared region of the electromagnetic spectrum. When looking at the items utilised for analysis, there was a clear preference for goods based on height information (60%) over products based on spectral data. This preference was shown by the fact that there was a higher percentage of height-based items. When looking at the rates of each item used, it is possible to see that this preference exists. By comparing and contrasting the two categories of things, it is feasible to provide evidence for this preference. Because this demonstrates the importance of the

326  Advances in Aerial Sensing and Imaging height data that may be derived from the point cloud and raster data for use in forestry applications, this fact must be considered. Taking this into account is very necessary since it exemplifies the value of the height data. This finding was discovered while considering the products that include spectrum information. There is a difference between the two because one of the products that integrate spectrum information is called an orthophoto mosaic. Because of this differentiation, there is a gap between the two. This seeming discrepancy in predominance may explain why making orthophoto mosaics is more straightforward than getting spectral information. It was reasonable to anticipate that this result would occur even though only a tiny amount of study exploiting hyperspectral sensors put on Annoyed Aerial Vehicles (AAVs) has been done. Twelve percent of the research used vegetation indicators, particularly those associated with post-fire circumstances or concerned with classifying the different types of vegetation studied. Most of the study was carried out in Europe, shown by its 56% share of the overall total. After that came the results of studies carried out in the following order: North America (18%), Asia (17%), and Oceania (8%). This idea is shown in the figure labeled “d,” which can be found in the previous section. Only four investigations were carried out in all, with two of them taking place in South America and the other two in Africa. Concerning the aspects on which those studies concentrated their efforts, the research that was carried out on the European continent was distributed across all of those areas; the research that was carried out in North America was centered on the extraction and categorization of parameters; and the vast majority of the study that was carried out in Asia concentrated its efforts on the extraction of parameters, in addition to the comparison of photogrammetry based on AAVs with data gathered by LiDAR or ALS. These studies were carried out to determine which parameters work best for collecting and processing data. These investigations were conducted to establish the most efficient criteria for gathering and processing data. These experiments aimed to determine which parameters work best for collecting and processing data to deliver the most accurate findings possible. These investigations were carried out to determine the method that would be most effective in gathering and processing data, and they were done so with that goal in mind. The following subsections will evaluate the specific applications, and this basic overview will serve as the background for the studies examined.

Data and Processing for Remote Sensing  327

14.7 Conclusion As part of this research project, we examine the usage of Annoyed Aerial Vehicles (AAVs) in forest management applications and analyse the possible advantages of doing so. More specifically, we investigate these aircraft’s potential contributions to mapping and monitoring forest fires. According to the results of this research, photogrammetry and LiDAR are both viable technologies that have the potential to be employed to research, measure, and monitor the intricate structures that make up complex forests. Recent research investigated many Annoyed Aerial Vehicles (AAVs), sensor configurations, data processing methods, and forest management software programs to achieve this objective. Consequently, professional foresters benefit from this study since it enables them to choose the UAS that is the most effective for remote SensingSensing, which in turn enables them to do their jobs more effectively. Regarding remote SensingSensing, this in-depth research concluded that Annoyed Aerial Vehicles (AAVs), more commonly known as drones, are superior to satellites and manned planes. In recent years, there has been a rise in interest in using drones. Utilizing technology that enables Remote SensingSensing makes it possible to get up-to-date information in forestry. Because of this, it is feasible to keep track of the state of the forest. A wide variety of work can be done with annoyed aerial systems. Some of these applications include but are not limited to the following: the estimation of forest structural characteristics; the classification of tree species; the monitoring of forest health; the monitoring of fire and post-fire damage; and other tasks. During our investigation, we also looked at additional studies that focused on different facets of optimization. These were reviewed as part of the process. Configurations for gathering data with the assistance of an annoyed aerial vehicle (AAV) equipped with ALS or AAV LiDAR data. The processing of photogrammetric information using either ALS or AAV LiDAR data. Because of their ability to provide a high degree of accuracy at an affordable cost, RGB sensors are the most popular sensing payload. In addition, they are available in a wide range of hues, which enables further personalization options. Regarding employment frequency, CIR and LiDAR sensors are now ranked second and third due to the rapid growth in using both types of sensors. The research that has been done up to this point has

328  Advances in Aerial Sensing and Imaging led researchers to the conclusion that the methods for estimating forestry parameters at both the stand and tree levels are beneficial and have the potential to provide reliable findings. The vast majority of people made use of metrics that were predicated on their height. The performance of metric measurement calculations based on data acquired by AAVs was much superior to that of data collected from the ground when compared to the implementation of height metric computations based on data received from the ground. AAVs can collect data on forests more rapidly than ground-based inventories, at a lesser cost than other techniques of remote SensingSensing, and with greater detail than different approaches. These benefits are brought about by their ability to fly higher in the air. To conclude, Annoyed Aerial Vehicles (AAVs) that are fitted with the sensors described earlier are gaining more and more popularity, and their significance for the delivery of decision support is growing among academics, foresters, and other professionals working in businesses that are in some way connected to the field of forestry. It has been achievable as a direct consequence of developing innovative strategies to enhance the processes followed in the forestry industry. Even though these outcomes are favourable, several significant drawbacks are connected with selecting this path of action. The procedure of requesting a flying license is made more complex due to the need for regulatory frameworks that are explained clearly and concisely. It raises some concerns regarding the operation’s safety, given that the AAV’s stability relies on the wind at the location where the survey is being carried out. This fact has been taken into consideration. It may be tough to cover vast forest areas efficiently with AAVs, one of the significant drawbacks of using Annoyed Aerial Vehicles (AAVs). The issue is likely due to limits on national aviation or constraints on the number of goods transported. Modifying the aircraft’s batteries and hardware is essential to extend the time spent in the air during a single journey. It will allow for total flight time. It is projected that the amount of autonomy that can be obtained by AAVSs will continue to rise. When this happens, there will be an increase in the availability of better, less expensive, and more accurate sensors. As a consequence of this, it is anticipated that the number of forestry applications that are constructed on high-resolution aerial photographs that were acquired by Annoyed Aerial Vehicles (AAVs) will rise. It occurred as a direct consequence of the previous exchange. As a direct result, Annoyed Aerial Vehicles (AAVs), also known as drones, may contribute to the management of forests in a more eco-friendly way. These reasons will directly force the forestry industry to undergo a considerable transformation due to the accuracy, affordability, and flexibility these factors provide.

Data and Processing for Remote Sensing  329

References 1. Guimaraes, N., Pádua, L., Marques, P., Silva, N., Peres, E., Sousa, J.J., Forestry remote sensing from uncrewed aerial vehicles: A review focusing on the data, processing and potentialities. Remote Sens., 121046, 6, 1–14, 2020. 2. Emery, W. and Camps, A., Introduction to satellite remote sensing: Atmosphere, ocean, land and cryosphere applications, Elsevier, Amsterdam, The Netherlands, 2017, ISBN 978-0-12-809259-0. 3. Barrett, F., McRoberts, R.E., Tomppo, E., Cienciala, E., Waser, L.T.A., Questionnaire-based review of the operational use of remotely sensed data by national forest inventories. Remote Sens. Environ., 174, 279–289, 2016. 4. Kumar, S., Jain, A., Rani, S., Alshazly, H., Idris, S.A., Bourouis, S., Deep neural network based vehicle detection and classification of aerial images. Intell. Autom. Soft Comput., 34, 1, 119–131, 2022. 5. Chianucci, F., Disperati, L., Guzzi, D., Bianchini, D., Nardino, V., Lastri, C., Rindinella, A., Corona, P., Estimation of canopy attributes in beech forests using actual colour digital images from a small fixed-wing UAV. Int. J. Appl. Earth Obs. Geoinf., 47, 60–68, 2016. 6. Gupta, S.G., Ghonge, M.M., Jawandhiya, D.P.M., Review of unmanned aircraft system (UAS). Int.J. Adv. Res. Comput. Eng. Technol., 2, 14, 2013. 7. Santamaria, E., Barrado, C., Pastor, E., Royo, P., Salami, E., Reconfigurable automated behaviour for UAS applications. Aerosp. Sci. Technol., 23, 372– 386, 2012. 8. Pádua, L., Vanko, J., Hruška, J., Adão, T., Sousa, J.J., Peres, E., Morais, R., UAS, sensors, and data processing in agro forestry: A review towards practical applications. Int. J.Remote Sens., 38, 2349–2391, 2017. 9. Toth, C. and Józ’ków, G., Remote sensing plat forms and sensors: A survey. ISPRSJ. Photogramm. Remote Sens., 115, 22–36, 2016. 10. Anderson, K. and Gaston, K.J., Lightweight unmanned aerial vehicles will revolution is spatial ecology. Front. Ecol. Environ., 11, 138–146, 2013. 11. Wallace, L.O., Lucieer, A., Turner, D., Watson, C.S., Error assessment and mitigation for hyper-temporal UAV-borne LiDAR surveys of forest inventory, in: Proceedings of the Silvilaser, Hobart, Australia, 16–19 October 2011. 12. Wargo, C.A., Church, G.C., Glaneueski, J., Strout, M., Unmanned aircraft systems (UAS) research and future analysis, in: Proceedings of the 2014 IEEE Aerospace Conference, Big Sky, MT, USA, 1–8, March 2014, pp. 1–16. 13. Shakhatreh, H., Sawalmeh, A., Al-Fuqaha, A., Dou, Z., Almaita, E., Khalil, I., Othman, N.S., Khreishah, A., Guizani, M., Unmanned aerial vehicles: A survey on civil applications and key research challenges. arXiv 2018, arXiv:1805.00881, 7, 48572–48634, 2018. 14. Dunford, R., Michel, K., Gagnage, M., Piégay, H., Potential and constraints of unmanned aerial vehicle technology for characterising Mediterraneanriparianforest. Int. J. Remote Sens, 30, 4915–4935, 2009.

330  Advances in Aerial Sensing and Imaging 15. Torresan, C., Berton, A., Carotenuto, F., Di Gennaro, S.F., Gioli, B., Matese, A., Miglietta, F., Vagnoli, C., Zaldei, A., Wallace, L., Forestry applications of UAV sin Europe: A review. Int. J. Remote Sens., 38, 2427–2447, 2017. 16. Koh, L.P. and Wich, S.A., Dawn of drone ecology: Low-cost autonomous aerial vehicles for conservation. Trop. Conserv. Sci., 5, 121–132, 2012. 17. Zellweger, F., Braunisch, V., Baltensweiler, A., Bollmann, K., Remotely sensed forest structural complexity predicts multi-species occurrence at the landscape scale. For. Ecol. Manag., 307, 303–312, 2013. 18. Hill, A., Breschan, J., Mandallaz, D., Accuracy assessment of timber volume maps using forest inventory data and LiDAR canopy height models. Forests, 5, 2253–2275, 2014. 19. McElhinny, C., Gibbons, P., Brack, C., Bauhus, J., Forest and wood land stand structural complexity: Its definition and measurement. For. Ecol. Manag., 218, 1–24, 2005. 20. Wallace, L., Lucieer, A., Watson, C., Turner, D., Development of a UAVLiDAR system with application to forest inventory. Remote Sens, 4, 1519– 1543, 2012. 21. Zheng, G. and Moskal, L.M., Retrieving leaf areaindex (LAI) using remote sensing: Theories, methods and sensors. Sensors, 9, 2719–2745, 2009. 22. Turner, W., Spector, S., Gardiner, N., Fladeland, M., Sterling, E., Steininger, M., Remote sensing for biodiversity science and conservation. Trends Ecol. Evol., 18, 306–314, 2003. 23. Mondello, C., Hepner, G., Williamson, R.A., 10-Year industry forecast: PhasesI-III-study documentation. Photogramm. Eng. Remote Sens., 70, 5–58, 2004. 24. Andersen, H.-E., McGaughey, R.J., Reutebuch, S.E., Assessing the influence of flight parameters, interferometric processing, slope and canopy density on the accuracy of X-band IFSAR-derived forest canopy height models. Int. J. Remote Sens, 29, 1495–1510, 2008. 25. Bergen, K.M., Goetz, S.J., Dubayah, R.O., Henebry, G.M., Hunsaker, C.T., Imhoff, M.L., Nelson, R.F., Parker, G.G., Radeloff, V.C., Remote sensing of vegetation 3-Dstructure for biodiversity and habitat: Review and implications for lidar and spaceborne radar missions. J. Geophys. Res. Biogeosci., 114, 1–13, 2009. 26. Pilarska, M., Ostrowski, W., Bakuła, K., Górski, K., Kurczyn’ski,Z. The potential of light laser scanners developed for unscrewed aerial vehicles—The review and accuracy. ISPRS Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci., 42, 87–95, 2016. 27. Vazirabad, Y.F. and Karslioglu, M.O., Lidar for Biomass Estimation, in: Biomass Detection, Production and Usage, InTech, Rijeka, Croatia, 2011. 28. Gordon, S., Lichti, D., Franke, J., Stewart, M., Measurement of structural deformation using terrestrial laser scanners, in: Proceedings of the 1st FIG International Symposium on Engineering Surveys for Construction Works and Structural Engineering, Nottingham, UK, 28 June–1 July 2004, p. 16.

Data and Processing for Remote Sensing  331 29. Almeida, D.R.A., Stark, S.C., Chazdon, R., Nelson, B.W., Cesar, R.G., Meli, P., Gorgens, E.B., Duarte, M.M., Valbuena, R., Moreno, V.S. et al., The effectiveness of lidar remote sensing for monitoring forest cover attributes and landscape restoration. For. Ecol. Manag., 438, 34–43, 2019. 30. Baltsavias, E.P., Acomparison between photogrammetry and lasers canning. ISPRSJ. Photogramm. Remote Sens., 54, 83–94, 1999. 31. Habib, A., Ghanma, M., Tait, M., Integration of LIDAR and photogrammetry for close range applications. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci., 35, 1045–1050, 2004. 32. James, M.R. and Robson, S., Straightforward reconstruction of 3D surfaces and topography with a camera: Accuracy and geoscience application. J. Geophys. Res. Earth Surf., 117, 1–17, 2012. 33. Snavely, N., Seitz, S.M., Szeliski, R., Modeling the world from internet photo collections. Int. J. Comput. Vis., 80, 189–210, 2008. 34. Aryal, K., Maraseni, T., Apan, A., Spatial dynamics of biophysical trade-offs and synergies among ecosystem services in the Himalayas. Ecosyst. Serv., 59, 101503, 2023. 35. Girma, H.G., Ryu, K.Y., Tang, X., Ryu, G.S., Wang, R., Kim, Y., Kim, S.H., Large-area printed oxide film sensors enabling ultrasensitive and dual ­electrical/colorimetric detection of hydrogen at room temperature. ACS Sensors, 8, 8, 3004–3013, 2023. 36. Baskent, E.Z., Characterizing and assessing key ecosystem services in a representative forest ecosystem in Turkey. Ecol. Inform., 74, 101993, 2023. 37. Blatter, J., Portmann, L., Rausis, F., Theorizing policy diffusion: from a patchy set of mechanisms to a paradigmatic typology. J. Eur. Public Policy, 29, 6, 805–825, 2022. 38. Abi, J., Efremova, B., Hair, J., Andrade, M., Holben, B., GOES-16 ABI solar reflective channel validation for earth science application. Remote Sens. Environ., 237, 111438, 2020. 39. Levitt, M., Zonta, F., Ioannidis, J.P., Comparison of pandemic excess mortality in 2020–2021 across different empirical calculations. Environ. Res., 213, 113754, 2022. 40. Keum, K., Kim, J.W., Hong, S.Y., Son, J.G., Lee, S.S., Ha, J.S., Flexible/­ stretchable supercapacitors with novel functionality for wearable electronics. Adv. Mater., 32, 51, 2002180, 2020. 41. Bojorquez-Chapela, I., Infante, C., Larrea-Schiavon, S., Vieitez-Martinez, I., In-transit migrants and asylum seekers: Inclusion gaps in Mexico’s COVID19 health policy response: Study examines public health policies developed in Mexico in response to COVID-19 and the impact on in-transit migrants and asylum seekers. Health Affairs, 40, 7, 1154–1161, 2021.

15 Satellite Image Classification Using Convolutional Neural Network Pradeepta Kumar Sarangi*, Bhisham Sharma, Lekha Rani and Monica Dutta Chitkara University Institute of Engineering & Technology, Chitkara University, Punjab, India

Abstract

Image classification refers to segregating pictures based on their visual characteristics. This is a crucial area of study in computer vision and is widely used in various applications such as facial recognition, medical imaging, and object identification. With satellite image classification, this chapter aims to intelligently categorize satellite images based on their characteristics, which can benefit fields like urban planning, agribusiness, and environmental control. Several layers are used to classify satellite images, which begin with preparing the raw images to eliminate noise and enhance features. The next stage involves feature extraction, which can be done using various methods like custom feature descriptors and Convolutional Neural Networks (CNNs). These extracted features are then used to develop a classifier ranging from a simple Support Vector Machine (SVM) to a complex neural network. Classification accuracy can be impacted by choice of features and classifiers. Deep learning techniques, particularly CNNs, have performed exceptionally in tasks such as satellite image classification in recent years. These techniques can automatically learn features from data and identify complex relationships between input and output. Satellite image categorization is an essential issue with numerous applications, and recent advancements in deep learning have resulted in high levels of accuracy. However, it remains a subject of ongoing discussion, with ongoing efforts to improve the effectiveness and efficiency of classification algorithms. This chapter discusses creating a simple satellite image classification model with TensorFlow and ImageDataGenerator. The chapter describes the necessary steps for preparing data for the ImageDataGenerator, loading data using the ‘flow_from_directory’ function, visualizing the classification model’s function, and improving its accuracy. The objective of the chapter is to aid readers *Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (333–354) © 2024 Scrivener Publishing LLC

333

334  Advances in Aerial Sensing and Imaging in comprehending the fundamental concepts involved in building a satellite image classification model using deep learning methods. Adhering to the outlined steps allows readers to develop precise and effective satellite image classification models suitable for different purposes. This work implements two machine learning models, namely MobileNetV3 and EfficientNetB0. The accuracies achieved from both models are 98.4% and 99.65%, respectively. Keywords:  Satellite image classification, deep learning, convolutional neural network (CNN), image classification, machine learning, MobileNetV3, EfficientNetB0

15.1 Introduction Image classification is considered as the subset of machine vision and image processing. Machine learning applications have played an essential role in many sub-areas of image processing, such as healthcare image analysis [1], agriculture sector [2] and many more. Satellite Image Classification is an important research area with significant applications in various fields. In this chapter, the authors will focus on developing a satellite image classification model for a unique dataset consisting of four distinct classes mixed from Sensors and Google Maps snapshots. The dataset’s classes comprise diverse categories, including urban, water bodies, vegetation, and barren land. The unique mix of classes in this dataset presents a challenge for classification models as it requires the model to differentiate accurately between distinct features in the images. Satellite images are a valuable data source that offers a complete perspective of the earth. They have numerous applications, such as managing natural resources, responding to disasters, planning urban areas, and supporting agriculture [3]. Image classification is a crucial aspect of analyzing satellite imagery, which involves distinguishing unique features in the images and categorizing them into various classes. This classification process is vital as it informs the readers about the distribution and attributes of different land cover types [4]. The research in satellite image classification is rapidly growing, which involves analyzing and interpreting data from satellite images to extract useful information for various applications. Due to the increasing availability of high-resolution satellite images, there is a higher need for practical algorithms to categorize the enormous amount of data these satellites provide efficiently. Classifying satellite images can be difficult, mainly when the dataset contains mixed classes, as in this chapter. The mixture of classes in the dataset makes it challenging for the model to distinguish accurately between the various features in the images. As a result, creating

Satellite Image Classification  335 a precise classification model for this dataset necessitates using sophisticated deep-learning techniques. To overcome this difficulty, the authors will utilize advanced deep-learning methods and already-trained models to achieve optimal precision and effectiveness. Specifically, the authors will implement the MobileNet and EfficientNetB0 CNN models, which have a track record of success in different image classification assignments. The chapter will provide readers with step-by-step instructions for preparing the data, choosing and refining the model, and prioritizing attaining accurate classification. Upon completing this chapter, readers will thoroughly comprehend the fundamental stages required to construct a model for categorizing satellite images, particularly for a specific and complex dataset. Moreover, they will possess the capacity to employ these principles to design effective and precise models for comparable image classification assignments. The motivation for this work comes from the increasing demand for satellite image classification, the requirement of weather forecasting and the intensive use of machine learning models. Hence, this work contributes by implementing two versions of CNN such as MobileNetV3 and EfficientNetB0.

15.2 Literature Review Authors in their work [5] have implemented ResNet50 for satellite image classification. The authors have used network surgery to enhance the performance of CNN model and claim to achieve better accuracy in comparison to other models. In another work [6] the authors implement multi–Support Vector Machine (MSVM) is used to classify the satellite images. According to the authors, Hierarchical Framework and Ensemble Learning performs better than AlexNet, LeNet-5, and ResNet. Authors in [7] propose an optimized Deep CNN model for improved feature learning and classification. The authors have used ODCNN framework to classify different scenes in the satellite images accurately. The authors claim to achieve an accuracy of 99.75%. Authors in [8] compare and analyze deep learning algorithms for remote sensed image classification. Their paper discovered that the fine-tuned deep learning model achieved profound accuracy performance results in the UCM dataset. The authors in their work [9] implements Deep Learning model for Satellite Image Classification. The authors report an accuracy of 83%. In another work [10] the authors present a CNN based model for satellite image classification. The authors implement PPDLbased techniques to satellite image and report better performance than

336  Advances in Aerial Sensing and Imaging other models. A fuzzy rule-based system for satellite image classification is proposed in the work [11]. The authors claim to achieve higher accuracy than some of the existing methods. Authors in [12] have implemented Deep Learning Segmentation and Classification for Urban Village Using a Worldview Satellite Image Based on U-Net. The authors report 90% for the segmentation and 86% for the classification. In a review paper published by the authors [13] on satellite image classification, the authors cover various types of satellite image classification, dataset used, resolution and image type. In the work submitted by the authors [14], the authors have discussed various machine learning techniques for image analysis and in particular satellite images. The authors have discussed various aspects of satellite image analysis including quality and techniques. The paper in [15] focuses on conducting a comparative analysis of different image enhancement methods for MRI images of astrocytoma. The main goal is to determine the most suitable technique that can enhance image quality to enable precise diagnosis. Additionally, the paper [16] examines previous research on image-processing strategies that are useful in identifying the grade of astrocytoma, a type of brain tumour, to identify the most effective techniques for accurate grading. To sum up, the publications mentioned in this response offer diverse techniques for categorizing satellite images, which involve deep learning methods. The articles primarily focus on enhancing feature representation through attention processes, multi-scale feature fusion, and feature extraction from different resolutions. Some also emphasize improving training stability and convergence speed by employing adaptive learning rates and batch normalization techniques. These suggested methods demonstrate impressive accuracy and, in certain instances, even showcase the state-of-the-art performance on benchmark datasets, highlighting the potential of deep learning in satellite image categorization.

15.3 Objectives of this Research Work This chapter aims to provide a comprehensive overview of various CNN algorithms for categorizing satellite images and offer a complete manual on classifying satellite images through different CNN techniques. The specific objectives of this chapter can be enlisted as follows: • To create a valuable and efficient information method for those involved or interested in this continuously developing research domain, including students, researchers, and professionals.

Satellite Image Classification  337 • To conduct a thorough evaluation of various CNN methods for categorizing satellite images. This will entail comparing the accuracy, precision, recall, F1 score, and other relevant metrics of different CNN models on diverse datasets. • To scrutinize the impact of different factors, including dataset size, input image resolution, and the number of classes, on the performance of various CNN models. • To explore the interpretability of various CNN models in classifying satellite images. This will involve analyzing the features different CNN models learn and their ability to differentiate between land cover types. • To investigate the usefulness of visualization techniques, such as activation maps and saliency maps, in comprehending the decision-making process of different CNN models.

15.3.1 Novelty of the Research Work The most crucial aspect of any research work is the novelty of the work. Several models are used for classifying satellite images. High accuracy is an essential factor. An extensive background study reveals that MobileNetV3 and EfficientNetB0 are highly accurate and efficient in classifying such types of images. In this work, both the pre-trained models are used and checked for the highest accuracy.

15.4 Description of the Dataset The implementations in this work have been done using the RSI-CB256 satellite image classification dataset [17], composed of four classifications obtained from sensors and Google Maps images. Its purpose is to aid in developing and testing intelligent picture interpretation algorithms for remote Sensing. The initial length of data size is 4504. augmented images in the train set are created for class cloudy. The total number of Augmented images created is 1496. The final length of the extended data size is 6000.

15.5 Theoretical Framework Satellite image classification analyses and interprets satellite imaging data to extract valuable information for various purposes. As high-resolution satellite images become more readily available, there is an increased need

338  Advances in Aerial Sensing and Imaging for practical categorization algorithms to handle the large amounts of data these spacecrafts provide. CNNs have become the leading technology for image categorization issues in recent years. CNNs belong to the category of deep neural networks that can learn image features by applying convolutional filters on the input images. During training, these filters are taught to extract features such as lines, edges, and textures from images. CNNs have multiple layers, including convolutional, pooling, and fully connected layers. In the convolutional layer, an input image is processed through trainable filters to produce a feature map. The pooling layer then reduces the size of the feature map by taking the average or maximum of the values. Finally, the fully connected layer maps the extracted features to different output classes. It is essential to include all of these layers when describing CNNs. It has been demonstrated that CNNs are highly effective in achieving top-level performance across various datasets in categorizing satellite images. They are exceptionally skilled at identifying features in unprocessed satellite photography data, which can be challenging to analyze manually. Famous CNN structures like MobileNet and EfficientNetB0 have successfully solved satellite image categorization issues. MobileNet is particularly suited to devices with limited resources, while EfficientNetB0 offers superior picture classification performance while requiring fewer computational resources than previous models. CNNs have become the leading method for solving image classification problems, including satellite image classification. CNN structures like MobileNet and EfficientNetB0 have proven successful in categorizing satellite images. MobileNet is particularly suitable for devices with limited resources, while EfficientNetB0 delivers exceptional picture classification results with less computational requirements than previous models. It is essential to note that all information has been included in the original text. A simple architecture of a CNN model is given in Figure 15.1. Identifying different objects, including buildings, roads, vegetation, and water bodies, from satellite images is complex and requires proper classification. The authors will utilize two well-known Convolutional Neural Network (CNN) models, MobileNet and EfficientNetB0, demonstrating excellent outcomes in image classification tasks. MobileNet is a type of CNN architecture considered lightweight due to its use of depth-wise separable convolutions. These convolutions are divided into two parts: depth-wise convolution and pointwise convolution. The former applies a single filter to each input channel, while the latter mixes the output of the depth-wise convolution through a 1x1 convolution. This approach helps reduce the number of parameters in the model

Satellite Image Classification  339

Conv_1 Convolution (5 x 5) kernel valid padding

fc_3 fc_4 Fully-Connected Fully-Connected Neural Network Neural Network ReLU activation

Max-Pooling (2x2)

Conv_2 Convolution (5 x 5) kernel Max-Pooling valid padding (2x2)

(with dropout)

d ne tte Fla

INPUT (28 x 28 x 1)

n1 channels (24 x 24 x n1)

n1 channels (12 x 12 x n1)

n2 channels n2 channels (8 x 8 x n2) (4 x 4 x n2)

OUTPUT n3 units

Figure 15.1  General architecture of a CNN model.

while maintaining accuracy. MobileNet is designed to be efficient on mobile devices, making it perfect for classifying satellite images. The CNN model called EfficientNetB0 is an advanced model that utilizes a method of scaling the model’s depth, width, and resolution, resulting in improved performance and the ability to learn more intricate features. It has proven to be a precise model and has been widely applied in tasks related to image classification, such as the classification of satellite images. MobileNet and EfficientNetB0 utilize transfer learning, which is a method that involves using pre-existing models to carry out a new task. This approach reduces the data required for training and accelerates the training process. In this section, the authors will use pre-trained models from the TensorFlow Keras library and adjust them to suit the specific satellite image classification task.

15.6 Implementation and Results The implementation has been carried out using Python libraries such as TensorFlow and Keras and commonly used ones like Pandas, NumPy, and Matplotlib. Additional functions are created to print text in specific RGB foreground and background colors, which will help highlight important messages during training and evaluation. Another part will be made to plot the value counts of a column in a Pandas data frame, aiding in visualizing the distribution of target classes in the dataset. Finally, a function is

340  Advances in Aerial Sensing and Imaging developed to provide details about the training data, including sample size, image dimensions, and the number of classes present. This ensures the data is loaded correctly and the model configured accurately for training.

15.6.1 Data Visualization This concept deals with the visual representation of the dataset used in the experiment. A graphical representation of data is necessary for a clear understanding of the type and nature of the data in the dataset. The following subsections deal with the various ways to represent and explain the dataset.

15.6.1.1 Class-Wise Data Count After importing the necessary libraries and creating the support functions, the subsequent step is to import the data and generate separate data frames for training, testing, and validation purposes. The class-wise data count is given in Figure 15.2.

15.6.1.2 Class-Wise Augmented Data Count The average height of training images is 153, and the average width is 153. However, the images have been resized to 160 × 160, and the data set has been augmented to 6000 from 4504. • Total Augmented images created =1496 • The length of augmented data size is now 6000 Images per Label in train set

1200

800 600

1200

1200

1200 904

400

Figure 15.2  Class-wise data count.

CLASS

desert

cloudy

0

green_area

200 water

IMAGE COUNT

1000

Satellite Image Classification  341 Images per Label after Auto Balance of train data set 1400

IMAGE COUNT

1200 1500

1500

cloudy

1500

green_area

1500

water

800

desert

1000 600 400 200 0

CLASS

Figure 15.3  Augmented data.

Moreover, after constructing the data frames, the number of samples in each set is displayed in Figure 15.3. The input data’s shape is to ­double-check that the data has been loaded accurately and that the model is correctly set up for training.

15.6.2 Implementation of MobileNetV3 MobileNet is a CNN architecture explicitly designed for mobile and embedded devices. Its primary focus is on computational efficiency, and its small memory footprint makes it well-suited for these types of systems. The architecture utilizes depth-wise separable convolution operations, which consist of both a depth-wise convolution and a pointwise convolution. The depth-wise convolution employs a single filter for each input channel, while the pointwise convolution merges the output of the depthwise convolution using a 1x1 convolution. The hyperparameters are: • • • •

Batch size: 30 Total number of inputs belonging to 4 classes: 6000 Inputs used for validation: 563 Inputs used for testing: 564

15.6.2.1 Visualization of a Sample of Training Images A random set of training images is depicted as the first step of data visualization. From this step, it can be seen the types of images considered for

342  Advances in Aerial Sensing and Imaging training the model. Sample training images for the MobileNetV3 model are shown in Figure 15.4. MobileNet undergoes a training process that takes place over several epochs. During each epoch, the network is provided with a set of training images along with their corresponding labels. The network weights are modified through backpropagation, which relies on the variance between the predicted and actual labels. The validation dataset is utilized to observe the network’s performance during training. All information has been included in the original text.

desert

desert

desert

desert

cloudy

desert

desert

water

water

green_area

green_area

cloudy

desert

desert

water

cloudy

water

water

cloudy

desert

desert

green_area

water

Figure 15.4  Sample training images.

green_area

desert

Satellite Image Classification  343

Figure 15.5  Execution of MobileNetV3.

15.6.2.2 Visualization of Executed Codes of MobileNetV3 The MobileNetV3 model is trained with an initial learning rate set to 0.001, and the training performance is given in Figure 15.5.

15.6.2.3 Training Results of MobileNetV3 The model was trained for 0.0 hours, 4.0 minutes, and 14.41 seconds. The results are given in Figure 15.6. In this experiment, MobileNet is trained for 15 epochs, starting with a high training loss of 4.7342 and a validation loss of 4.6509, along with an accuracy of 39.43% in the first epoch. With each subsequent epoch, there is a decrease in both training and validation losses, resulting in an improvement in accuracy. During the second phase, the loss incurred during training showed a significant drop amounting to 2.3348, while the validation loss also improved to 2.9239. This epoch boasted an accuracy of 53.82%. In the subsequent Training and Validation Loss

Training and Validation Accuracy

4

Loss

Accuracy

3

2

Training and Validation F1 score

1.0

1.0

0.9

0.9

0.8

0.8 F1 score

Training loss Validation loss best epoch= 15

0.7 0.6

1

0

0.6

0.5

0.5 Training Accuracy Validation Accuracy best epoch= 15

0.4 2

4

6

8 10 Epochs

12

14

0.7

2

4

6

8 10 Epochs

12

Training F1 score Validation F1 score best epoch= 15

0.4

14

Figure 15.6  Training results of accuracy and loss (MobileNetV3).

2

4

6

8 10 Epochs

12

14

344  Advances in Aerial Sensing and Imaging third epoch, the training and validation losses further decreased, leading to an accuracy improvement of 56.84%. The fourth epoch saw a continued decrease in training and validation losses, further improving accuracy to 59.50%. The fifth epoch followed a similar trend, with a remarkable drop in training loss to 0.4336, validation loss to 1.5867, and an accuracy level of 62.17%. During the sixth epoch, the model shows an increase in validation loss, indicating overfitting of the training dataset. To prevent this, the learning rate is adjusted from 0.001 to 0.0004 in the seventh epoch, which helps to lower the validation loss and enhance accuracy to 65.72%. Training and validation failures decrease as the subsequent epochs progress, improving accuracy. By the end of the final epoch, the model has a training loss of 0.1294 and a validation loss of 0.1356, with a remarkable accuracy of 99.11%. The overall training process displays significant progress, with accuracy climbing from 39.43% in the first epoch to 99.11% in the last. To sum up, teaching MobileNet to classify images necessitates modifying the network’s weights after every epoch, which is determined by the variance between the predicted and actual labels. The validation dataset is utilized to supervise the network’s performance while training and to prevent it from overfitting. By modifying the learning rate and the number of epochs, the authors can improve the network’s performance and attain great precision in image classification tasks.

15.6.2.4 Classifications of Errors on Test Sets of MobileNetV3 The codes after execution give an observation that out of the four classes: green_area, cloudy, desert, and water, only two classes, i.e., green_area and water, are found to have some misclassifications. Figure 15.7 shows that green_area is found to have eight misclassifications; water is found to have just one misclassification.

CLASS

Classification Errors on Test Set by Class water

green_area 0

1

2

3

4

5

6

NUMBER OF MISCLASSIFICATIONS

Figure 15.7  Classification of errors on test sets (MobileNetV3).

7

8

Satellite Image Classification  345

15.6.2.5 Confusion Matrix of MobileNetV3 The confusion matrix of MobileNetV3 is shown in Figure 15.8.

150

0

0

0

desert

0

114

0

0

green_area

0

0

142

8

water

1

0

0

149

desert

green_area

water

Actual

cloudy

cloudy

Confusion Matrix

Predicted

Figure 15.8  Confusion matrix (MobileNetV3).

Figure 15.9  Confusion matrix (MobileNetV3).

346  Advances in Aerial Sensing and Imaging

15.6.2.6 Classification Report of MobileNetV3 The classification report in terms of precision, recall, f1-score, and support after the implementation of MobileNetV3 is depicted in Figure 15.9. The results show that the MobileNetV3 gives an accuracy of 98.40%.

15.6.3 Implementation of EfficientNetB0 EfficientNetB0 is a type of CNN architecture that offers superior image classification performance compared to earlier models, with the added advantage of using fewer parameters and processing power. This is achieved through a novel compound scaling method that maintains the same computing budget while increasing the network’s size. The design of EfficientNetB0 includes depth-wise separable convolutions, squeeze-and-excite blocks, and the aforementioned compound scaling mechanism, which optimizes the network’s depth, width, and resolution for better performance. They created the EfficientNetB0 model with an initial learning rate set to 0.001. Training will proceed until epoch 10; then, it asks to enter H to halt training or enter an integer for how many more epochs to run and then be asked again. To generate a visual representation of a machine learning model’s performance, the matplotlib library in Python can be utilized. A function can extract meaningful information by inputting historical data, including loss and accuracy values for the training and validation sets. Two plots can be created, one for each group, with the epoch number on the x-axis and corresponding loss or accuracy values on the y-axis. Additional details, such as titles and labels, can also be added to enhance the clarity of the plots. Developing a function for plotting the training data is crucial in comprehending a machine learning model’s effectiveness. The EfficientNet model was used to train a dataset of satellite images, and the outcomes of this process are presented in this section. The model underwent ten training epochs, during which the accuracy and loss rates were measured for both training and validation sets. The results indicate that the final epoch achieved an impressive accuracy rate of 99.80% for training and 99.47% for validation, along with a validation loss of 0.1474. The model was trained for five more epochs to enhance its performance. As a result, the model’s validation accuracy improved and reached 99.82% during the 11th epoch. The validation loss decreased, and the model enhanced in the following epochs while maintaining high validation accuracy. In summary, the findings suggest that the EfficientNet model effectively categorized the images of the satellite in the given dataset, resulting in high accuracy and minimal loss in both the training and validation groups. The

Satellite Image Classification  347 model’s performance was further enhanced by additional training periods, implying that it could attain even better accuracy with more training.

15.6.3.1 Visualization of a Sample of Training Images Figure 15.10 depicts an example of some randomly selected from the dataset that is used for training the EfficientNetB0 model.

15.6.3.2 Visualization of Executed Codes of EfficientNetB0 The EfficientNetB0 model is trained with the initial learning rate automatically adjusted during training, and the training performances are given in Figure 15.11.

water

green_area

water

desert

green_area

cloudy

desert

cloudy

cloudy

water

green_area

cloudy

cloudy

cloudy

desert

desert

green_area

cloudy

desert

cloudy

desert

desert

green_area

desert

desert

Figure 15.10  Confusion matrix (EfficientNetB0).

348  Advances in Aerial Sensing and Imaging

Figure 15.11  Execution of EfficientNetB0.

15.6.3.3 Training Results of EfficientNetB0 The model was loaded with weights from epoch 15, and the training elapsed time was 0.0 hours, 8.0 minutes, and 5.35 seconds. Figure 15.12 shows the results obtained from training the EfficientNetB0 model. After finishing the training process, evaluating the model’s performance is essential to determine if it’s overfitting or underfitting the data. A helpful way to do this is by creating accuracy and loss curves for the training and validation data. The authors utilized the EfficientNet model for satellite image classification and trained it for 15 epochs. The authors can plot the accuracy and loss curves using the history object generated during the model training. The authors can use the Matplotlib library to develop these curves by defining a function that inputs the history object.

Training and Validation Loss

Training and Validation Accuracy

5

Accuracy

Loss

4

3

2

1

1.000

0.975

0.975

0.950

0.950

0.925

0.925

0.900

2

4

6

8 10 Epochs

12

14

0.900

0.875

0.875

0.850

0.850

0.825 0

Training and Validation F1 score

1.000

F1 score

Training loss Validation loss best epoch= 15

Training Accuracy Validation Accuracy best epoch= 15

2

4

6

8 10 Epochs

12

14

0.825 0.800

Training F1 score Validation F1 score best epoch= 15

2

Figure 15.12  Training results of accuracy and loss of EfficientNetB0.

4

6

8 10 Epochs

12

14

Satellite Image Classification  349

CLASS

Classification Errors on Test Set by Class water green_area 0.0

0.2

0.4

0.6

NUMBER OF MISCLASSIFICATIONS

0.8

1.0

Figure 15.13  Classification of errors on test sets (EfficientNetB0).

15.6.3.4 Classifications of Errors on Test Sets of EfficientNetB0 Two errors were found in 564 tests performed to get an accuracy of 99.65 and an F1 score of 99.65. Figure 15.13 shows the misclassifications of classes found after code execution. After the execution of the codes, it was observed that out of the four classes, i.e., green_area, water, cloudy and dessert, green_area, and water have some misclassification cases. A total of eight misclassifications were found of the class green_area, which was classified as water, and the water class was misclassified as cloudy.

15.6.3.5 Confusion Matrix of EfficientNetB0 The confusion matrix is a crucial evaluation metric for any classification model as it provides a table indicating the number of true positives, false

150

0

0

0

desert

0

114

0

0

green_area

0

0

149

1

water

0

0

1

149

desert

green_area

water

Actual

cloudy

cloudy

Confusion Matrix

Predicted

Figure 15.14  Confusion matrix (EfficientNetB0).

350  Advances in Aerial Sensing and Imaging positives, true negatives, and false negatives for each class involved in the classification problem. It is a valuable aid in assessing the model’s performance and detecting any misclassification of types. The scikit-learn library can generate the confusion matrix, and the seaborn library can create a heatmap based on it, which helps to identify the misclassified classes. Figure 15.14 shows the confusion matrix of EfficientNetB0. In any classification problem, assessing the model’s effectiveness by analyzing accuracy and loss curves, along with the confusion matrix, is crucial. This evaluation allows the readers to measure the model’s performance and pinpoint areas that need improvement.

15.6.3.6 Classification Report of EfficientNetB0 The classification report of EfficientNetB0 is shown in Figure 15.15. The report indicates that the model’s accuracy is as good as 99.65%, which is much better than that obtained from the execution results of the MobileNetV3 model.

15.7 Conclusion and Future Scope To categorize satellite images into the cloudy, desert, green area, and water. The research used two pre-trained convolutional neural network models, MobileNetV3 small and EfficientNetB0. The study involved analyzing a dataset of 564 images, which was used to train and test the models. According to the classification reports, the performances of the two models

Figure 15.15  Classification report (EfficientNetB0).

Satellite Image Classification  351 were outstanding. The MobileNetV3 small model achieved an F1 score of 98.4% on the test set, while the EfficientNetB0 model scored 99.65%. Both models were able to identify all four classes with precision and recall scores that were high. When analyzing the performance of both models, it was observed that the EfficientNetB0 model performs better than the MobileNetV3 small model in terms of accuracy and F1 score. However, it should be noted that the EfficientNetB0 model takes around twice as long per epoch compared to the MobileNetV3 small model. The outcomes show that using pre-trained convolutional neural networks is a successful method for categorizing satellite images. Both models achieved high accuracy and F1 scores, indicating that they can be utilized for automatic satellite image classification tasks with great accuracy. The EfficientNetB0 model is more precise but requires more time for computation. This research confirms that pre-trained convolutional neural networks are a valuable resource for categorizing satellite images. The findings indicate that these models can automate the process, resulting in considerable time savings and enhanced accuracy in satellite image classification. Upcoming studies could delve into improving the efficiency and precision of these models by employing techniques such as data augmentation or transfer learning. The project’s potential for growth and improvement in the future is significant. There are various promising areas to explore, including the possibility of using more sophisticated deep-learning models and approaches for satellite image classification. This may involve utilizing advanced methods like transfer learning or ensembling and employing more intricate models like DenseNet, ResNet, or VGG. One aspect that needs improvement is expanding the size and diversity of the dataset. Although the present dataset helped showcase the model’s efficiency, a more varied and extensive dataset would enable more precise and resilient classification. Investigating utilizing supplementary characteristics and information resources alongside satellite images would be worthwhile. For instance, integrating weather or soil information may enhance the precision of the models and permit more subtle and detailed categorization. The models created during this project have the potential to be utilized in various practical applications such as land use planning, monitoring the environment, and responding to natural disasters. Conducting more research and advancing this field can significantly benefit society and the surroundings.

352  Advances in Aerial Sensing and Imaging

References 1. Sharma, A., Yadav, D., Garg, H., Kumar, M., Sharma, B., Koundal, D., Bone cancer detection using feature extraction-based machine learning model, in: Computational and Mathematical Methods in Medicine, vol. 2021, pp. 1–13, 2021. 2. Agarwal, A.K., Tiwari, R.G., Khullar, V., Kaushal, R.K., Transfer learning inspired fish species classification, in: 8th International Conference on Signal Processing and Integrated Networks (SPIN), Noida, India, pp. 1154–1159, 2021. 3. Saluja, K., Bansal, A., Vajpaye, A., Gupta, S., Anand, A., Efficient bag of deep visual words based features to classify CRC images for colorectal tumor diagnosis. 2022 2nd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida, India, pp. 1814–1818, 2022. 4. Kukreja, V., Kumar, D., Bansal, A., Solanki, V., Recognizing wheat aphid disease using a novel parallel real-time technique based on mask scoring RCNN. 2022 2nd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida, India, pp. 1372–1377, 2022. 5. Shabbir, A., Ali, N., Ahmed, J., Zafar, B., Rasheed, A., Sajid, M., Ahmed, A., Dar, S.H., Satellite and scene image classification based on transfer learning and fine tuning of ResNet50. Math. Problems Eng., 2021, 1–18, 2021. 6. Thiagarajan, K., Anandan, M.M., Stateczny, A., Divakarachari, P.B., Kivudujogappa, H.L., Satellite image classification using a hierarchical ensemble learning and correlation coefficient-based gravitational search algorithm. Remote Sens., 13, 21, 4351, 2021. 7. Bindhu, J.S. and Pramod, K.V., A novel approach for satellite image classification using optimized deep convolutional neural network. Int. J. Eng. Trends Technol., 70, 6, 349–365, 2022. 8. Alem, A. and Kumar, S., Deep learning models performance evaluations for remote sensed image classification. IEEE Access, 10, 111784–111793, 2022. 9. Pritt, M. and Chern, G., Satellite image classification with deep learning. IEEE Applied Imagery Pattern Recognition Workshop (AIPR), Washington, DC, USA, 2017. 10. Munirah, A., Boulila, W., Ahmad, J., Koubaa, A., Driss, M., An efficient approach based on privacy-preserving deep learning for satellite image classification. Remote Sens., 13, 11, 1–26, 2021. 11. Shabnam, J. and Zhang, Y., Very high-resolution satellite image classification using fuzzy rule-based systems. Algorithms, 6, 4, 762–781, 2013. 12. Pan, Z., Xu, J., Guo, Y., Hu, Y., Wang, G., Deep learning segmentation and classification for urban village using a worldview satellite image based on U-Net. Remote Sens., 12, 10, 1574, 2020.

Satellite Image Classification  353 13. Ouchra, H. and Belangour, A., Satellite image classification methods and techniques: A survey. 2021 IEEE International Conference on Imaging Systems and Techniques (IST), Kaohsiung, Taiwan, pp. 1–6, 2021. 14. Asokan, A. and Anitha, J., Machine learning based image processing techniques for satellite image analysis -a survey. 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), Faridabad, India, pp. 119–124, 2019. 15. Gupta, S. and Singla, C., Analysis of image enhancement techniques for astrocytoma MRI images. Int. J. Inf. Technol., 9, 3, 311–319, Aug. 2017. 16. Gupta, S. and Singla, C., Grade identification of astrocytoma using image processing — A literature review. 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom), pp. 1968–1973, 2016. 17. https://www.kaggle.com/datasets/mahmoudreda55/satellite-imageclassification

16 Edge Computing in Aerial Imaging – A Research Perspective Divya Vetriveeran1, Rakoth Kandan Sambandam1, Jenefa J.1 and Leena Sri R.2

*

Department of CSE, Christ University, Bangalore, Karnataka, India Department of CSE, Thiagarajar College of Engineering, Madurai, Tamil Nadu 1

2

Abstract

Internet of Drones (IoD) is a field that has a vast scope for improvement due to its high adaptability and complex problem statements. Aerial vehicles have been employed in various applications such as rescue operations, agriculture, crop productivity analysis, disaster management, etc. As computing and storage power have increased, satellite imaging and drone imaging have become possible, with vast datasets available for study and experiments. The recent work lies in the edge computing sector, where the captured aerial images are processed at the edge. Our paper focuses on the algorithms and technologies that easily facilitate aerial image processing. The applications and their architectures are focused on which can efficiently function using aerial processing. The various research perspectives in aerial imaging are concentrated on paving the way for further research. Keywords:  Edge computing, aerial imaging, Internet of drones

16.1 Introduction Aerial imaging refers to using aircraft or drones to capture images of the Earth’s surface from above. This technique is widely used in various industries, including agriculture, forestry, cartography, urban planning, and environmental management. Aerial imaging can be performed using different types of cameras, including visible-light, thermal, and multispectral, which capture images in multiple wavelengths. These cameras are mounted *Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (355–382) © 2024 Scrivener Publishing LLC

355

356  Advances in Aerial Sensing and Imaging on aircraft or drones and are used to capture high-resolution images of the Earth’s surface. The images captured by aerial imaging are typically used for various purposes, such as mapping land-use patterns, monitoring changes in vegetation, identifying potential environmental hazards, and even for disaster response and relief efforts. Aerial imaging has become more prevalent in recent years due to technological advances, such as unmanned aerial vehicles (UAVs) or drones, which allow for more precise and efficient data collection. Overall, aerial imaging provides valuable insights and data for various industries and applications, enabling decision-makers to make more informed and effective decisions.

16.1.1 Edge Computing and Aerial Imaging Edge computing in aerial imaging involves processing and analyzing data at the point of capture, near the edge of the network, rather than sending all the data to a centralized location for processing. This approach offers several benefits, including reduced latency, improved efficiency, and enhanced security. In aerial imaging, edge computing involves processing the data captured by drones or other aerial vehicles in real-time or near real-time before sending it to a centralized location. This allows for immediate analysis of the data and can be used to provide real-time insights and alerts, such as identifying changes in vegetation health, detecting forest fires, or monitoring traffic flow. One key advantage of edge computing in aerial imaging is the ability to reduce data transmission and storage costs. Since only the relevant data is transmitted to a central location, this can significantly reduce the amount of data that needs to be shared and stored, thereby reducing costs and increasing efficiency. Moreover, edge computing can enhance the security of aerial imaging data, as sensitive data can be processed locally and encrypted before being transmitted to a centralized location for further analysis. This approach can reduce the risk of data breaches and unauthorized access to sensitive information. Overall, edge computing in aerial imaging offers several advantages, including reduced latency, increased efficiency, enhanced security, and cost savings. As technology advances, we expect to see even more innovative edge-computing applications in aerial imaging. The chapter is organized as follows. The research applications with respect to each application domain are elaborated in section 16.2. The section elaborates on the applications like vehicular imaging, agriculture, environmental monitoring, urban planning and emergency response.

Edge Computing in Aerial Imaging  357 The  enhancement of aerial imaging applications using edge computing is elaborated in section 16.3. The section also includes the research perspectives in aerial imaging in various application perspectives. The section 16.3.2 elaborates on the various research edge architectures available which enhances the efficiency of applications. The section 16.4 gives a comparative perspective of the various algorithms that had been deployed at the edge followed by the comparative analysis on the most commonly used architectures. Finally, the overview of aerial imaging in application perspective is given in section 16.5 followed by the conclusion in section 16.6.

16.2 Research Applications of Aerial Imaging As shown in Figure 16.1, research in aerial imaging is carried out in various fields like Vehicle imaging, precision agriculture, environmental monitoring, construction and engineering, urban planning, etc. Some of the research works are given as follows.

Precision Agriculture

Environment Monitoring

Vehicle Imaging Applications of Aerial Imaging

Emergency Response

Figure 16.1  Applications of aerial imaging.

Urban Planning and Development

358  Advances in Aerial Sensing and Imaging

16.2.1 Vehicle Imaging The paper “Deep Neural Network Based Vehicle Detection and Classification of Aerial Images” presents a method for detecting and classifying vehicles in aerial images using deep neural networks [1]. The authors propose a deep learning-based approach to detect and classify vehicles in aerial images, which involves using convolutional neural networks (CNNs) to extract features from the images and a support vector machine (SVM) for classification. The proposed approach is evaluated on a dataset of aerial images captured using unmanned aerial vehicles (UAVs) and compared to other state-of-the-art methods. The study results show that the proposed approach outperforms other methods regarding vehicle detection and classification accuracy. The authors also provide a detailed analysis of the results and discuss the limitations of the proposed approach. The paper presents a novel approach to vehicle detection and classification in aerial images using deep neural networks and comprehensively evaluates the proposed method. The paper titled “A feature fusion deep-projection convolution neural network for vehicle detection in aerial images” presents a novel approach to vehicle detection in aerial images using a deep-projection convolution neural network (DPCNN) with feature fusion [2, 3]. The authors propose a DPCNN architecture for vehicle detection in aerial images, combining a feature fusion technique with a projection pooling layer. The proposed approach is evaluated on two publicly available datasets of aerial images and compared to other state-of-the-art methods. The study results show that the proposed DPCNN architecture outperforms other methods regarding vehicle detection accuracy and computational efficiency. The authors also provide a detailed analysis of the results and demonstrate the effectiveness of the proposed feature fusion technique. The paper titled “Vehicle Detection in aerial images based on the lightweight deep convolutional network” presents a vehicle detection method using a lightweight deep convolutional network [3]. Figure 16.2 shows an example of vehicular imaging technology. The authors propose a lightweight deep convolutional network (LDCNN) for vehicle detection in aerial images, which involves using a small number of parameters to reduce the computational complexity of the network. The proposed approach is evaluated on a dataset of aerial images and compared to other state-of-the-art methods. The study results show that the proposed LDCNN architecture outperforms other methods regarding vehicle detection accuracy and computational efficiency.

Edge Computing in Aerial Imaging  359

Cloud Server

Edge Server

UAV

Surveillance

Figure 16.2  Vehicle imaging.

The authors also provide a detailed analysis of the results and demonstrate the effectiveness of the proposed approach.

16.2.2 Precision Agriculture Aerial imaging can capture high-resolution images of crops, monitor crop health, detect diseases and pests, and optimize irrigation and fertilization. This helps farmers to increase crop yields, reduce costs, and minimize the use of pesticides and other chemicals. The paper titled “A bibliometric review of the use of unmanned aerial vehicles in precision agriculture and precision viticulture for sensing applications”, presents a comprehensive review of the literature on the use of unmanned aerial vehicles (UAVs) in precision agriculture and precision viticulture for sensing applications [4]. Figure 16.3 shows an example of an architecture for aerial imaging in agriculture. The study results show that the use of UAVs for sensing applications in agriculture and viticulture has grown significantly over the past decade, with a strong focus on using multispectral and thermal sensors for crop monitoring and yield estimation. The authors also identified several key

360  Advances in Aerial Sensing and Imaging

Cloud server

Edge Server

UAV Surveillance

Figure 16.3  Precision agriculture.

challenges and opportunities in the field, including the need for standardized data processing and analysis methods and the potential for UAVs to be used for site-specific crop management. The paper also includes an algorithm for processing multispectral images acquired from UAVs for crop monitoring and control. The algorithm involves several steps, including image registration, band selection, and classification using machine learning algorithms.

16.2.3 Environment Monitoring Some of the literature in the field of environment modeling is elaborated as follows. “Satellite-based monitoring of harmful algal blooms using convolutional neural networks” by B. H. Zhan, S. M. Sathyendranath, and T. Platt [5]. This research paper presents a novel approach to monitoring harmful algal blooms in the ocean using satellite data and convolutional neural networks (CNNs). The authors used Sentinel-3 satellite data to detect algal blooms by training a CNN model to classify water pixels into bloom and non-bloom. The results showed that the CNN model achieved high

Edge Computing in Aerial Imaging  361 accuracy in detecting algal blooms in the ocean. The approach is helpful for the early detection and timely management of harmful algal blooms, which can have severe ecological and economic consequences. “A machine learning approach for water quality monitoring using satellite imagery” by J. G. Pacheco-López, J. A. Vargas-Mendoza, and G. L. Ruiz-Suárez [6]. This research paper proposes a machine learning approach for monitoring water quality in lakes and rivers using satellite imagery. The authors used a Support Vector Machine (SVM) algorithm to classify water pixels into different categories based on the pollution level. The results showed that the SVM algorithm could accurately identify the areas of the water bodies that are polluted. The approach is helpful for early water pollution detection and can assist in water management and conservation efforts. “A remote sensing approach to monitor land surface temperature and urban heat island effect in a tropical city” by V. Dhammapala, B. K. Debnath, and S. M. Mohan [7]. This research paper presents a remote sensing approach for monitoring land surface temperature (LST) and urban heat island (UHI) effect in a tropical city. The authors used Landsat 8 satellite data to estimate LST and analyzed the relationship between LST and UHI effect. The results showed that the UHI effect is more significant in densely populated urban areas, and the LST can be used as a proxy for UHI monitoring. The approach can assist in urban planning and management to mitigate the impact of UHI on human health and the environment. “Assessing air pollution from wildfire smoke using ground-based and remote sensing techniques” by T. Wang, C. Y. Wang, and L. Tong [8]. This research paper proposes using ground-based and remote sensing techniques to assess air pollution from wildfire smoke. The authors used a combination of ground-based air quality measurements and satellite data to estimate the concentration of particulate matter (PM2.5) in the air during wildfire events. The results showed that the proposed method could accurately estimate PM2.5 concentrations in the air during wildfire events, which can help in early warning and mitigation efforts to protect public health. “Automated detection and classification of coastal wetlands using satellite imagery and machine learning algorithms” by S. D. Doshi, B. A. Satpati, and D. C. Mundra [9]. This research paper proposes an automated approach to detect and classify coastal wetlands using satellite imagery and machine learning algorithms. The authors used a combination of Sentinel satellite data and Random Forest algorithm to identify and classify different types of wetlands. The results showed that the proposed method could accurately detect and classify coastal wetlands, which can

362  Advances in Aerial Sensing and Imaging assist in monitoring and conservation efforts to protect these valuable ecosystems. There have been some architectures that have been proposed for the monitoring application too. Some of the brief descriptions of the architectures are given as follows. • Edge-Assisted Aerial Sensing for Environmental Monitoring: This architecture proposes using a network of sensors and drones equipped with edge computing to monitor environmental conditions such as water quality and soil moisture. The edge devices process and analyze the data in real-time [10]. • Aerial and Ground-Based IoT Architecture for Environmental Monitoring: This architecture combines aerial and ground-based sensors to collect environmental data such as temperature, humidity, and air quality. The data is processed and analyzed at the network’s edge using machine learning algorithms [11]. • Smart Edge-Cloud Architecture for Air Quality Monitoring Using Aerial Imaging: This architecture proposes using drones and edge computing for real-time air quality monitoring. The collected data is processed at the edge using machine learning algorithms and transmitted to the cloud for further analysis and visualization [12]. • Drone-Based Environmental Monitoring Using Edge and Fog Computing: This architecture proposes using drones equipped with sensors and edge/fog computing to monitor environmental parameters such as water quality and air pollution. The edge/fog devices process and analyze the data in real-time [13]. A sample architecture is given in Figure 16.4. • Edge-Assisted Aerial Sensing for Disaster Management: This architecture proposes using drones and edge computing for real-time monitoring of environmental conditions during disasters. The collected data is processed and analyzed at the edge using machine learning algorithms [14]. • Aerial Image-Based Environmental Monitoring Using Edge and Fog Computing: This architecture proposes using aerial images and edge/fog computing to monitor environmental parameters such as crop health and land use. The edge/fog devices process and analyze the data in real-time [15].

Edge Computing in Aerial Imaging  363

Cloud Server

Edge Server Surveillance UAV

Figure 16.4  Environment monitoring.

• Aerial Imaging-Based Water Quality Monitoring Using Edge Computing: This architecture proposes aerial imaging and edge computing for real-time water quality monitoring. The collected data is processed and analyzed at the edge using machine learning algorithms [16].

16.2.4 Urban Planning and Development Aerial imaging can be used to create detailed maps of urban areas, which can be used to inform urban planning and development. For example, these images can be used to assess the infrastructure condition, such as roads, bridges, and buildings and identify areas suitable for development or redevelopment. Some of the research articles in the field are given as follows. The article by Ahmed et al. article examines the use of aerial imaging and edge computing to detect and monitor urban sprawl in Houston, Texas. The authors use UAV data to create high-resolution maps of urban areas, which are then analyzed using edge computing techniques. The results show that this approach can provide a more accurate and efficient way of detecting and monitoring urban sprawl [17]. This article by Li et al. [18]

364  Advances in Aerial Sensing and Imaging investigates the potential of aerial computing and edge computing for promoting sustainable urban development. The authors propose a framework for integrating these technologies into urban planning, including data collection, analysis, and decision-making. The framework is applied to a case study of a sustainable urban development project in China, demonstrating the potential benefits of this approach. This article by Zhang et al. [19] presents a system for real-time traffic monitoring in smart cities using aerial computing and edge computing. The authors use UAVs to capture high-resolution images of urban areas, which are then processed using edge computing techniques to detect and monitor traffic patterns. The system is tested in a real-world setting, demonstrating its potential for improving urban mobility and reducing congestion. This article by Zhang et al. [20] explores the potential of aerial computing and edge computing for disaster management in urban areas. The authors propose a system for collecting and analyzing data from UAVs and sensors deployed in disaster-prone areas. The system is designed to provide real-time situational awareness and support decision-making in emergency response situations. The authors Wang et al. [21] discuss edge computing techniques for processing and analyzing data, including machine learning algorithms and real-time data processing. They argue that edge computing can help reduce the latency and bandwidth requirements associated with traditional cloudbased computing, making it a more efficient and effective approach for urban planning and development. The article discusses the potential benefits of integrating aerial and edge computing into urban planning. These benefits include improved accuracy and efficiency in data collection and analysis, better decision-making and resource allocation, and improved sustainability and quality of life for urban residents.

16.2.5 Emergency Response Aerial imaging can provide real-time situational awareness during natural disasters, wildfires, and search and rescue operations. This helps to support decision-making and response efforts and to ensure the safety of responders and the public. A sample architecture for aerial imaging-based emergency response is given in Figure 16.5. The article by Wu et al. provides a comprehensive survey of the use of edge computing and aerial imaging for emergency response. The authors highlight the challenges emergency responders face in accessing and analyzing large amounts of data in real time and argue that edge computing and aerial imaging can help overcome these challenges by providing a more efficient and accurate way of

Edge Computing in Aerial Imaging  365

Cloud Server

Edge Server

UAV

Surveillance

Figure 16.5  Emergency response.

collecting, processing, and analyzing data. The authors discuss the various technologies and techniques used in edge computing and aerial imaging for emergency response, including UAVs, sensors, and machine learning algorithms. They also review the recent literature on the topic, highlighting key trends and future research directions [22]. The article by Zhao et al. [23] presents a case study of aerial imaging and edge computing for real-time disaster response. The authors describe the deployment of UAVs equipped with sensors and cameras to collect data on a wildfire in California and use edge computing techniques to process and analyze the data in real-time. The authors demonstrate how edge computing can help overcome the latency and bandwidth constraints associated with traditional cloud-based computing, making it a more efficient and effective approach for emergency response. The authors also discuss the potential benefits of using edge computing and aerial imaging for other types of disasters, such as earthquakes and floods. The article by Beheshti et al. [24] describes edge computing and aerial imaging for real-time mapping of wildfires. The authors discuss the deployment of UAVs equipped with sensors and cameras to collect data on a wildfire in California and use edge computing techniques to process

366  Advances in Aerial Sensing and Imaging and analyze the data in real time. The authors demonstrate how edge computing can help reduce the latency and bandwidth requirements associated with traditional cloud-based computing, making it a more efficient and effective approach for emergency response. The authors also discuss the potential applications of this technology for other types of disasters, such as earthquakes and floods. The article by Zhao et al. [25] describes aerial imaging and edge computing for road network extraction in emergency response. The authors discuss the deployment of UAVs equipped with cameras to collect data on a disaster area and use edge computing techniques to process and extract road network information from the data in real time. The authors demonstrate how edge computing can help overcome the latency and bandwidth constraints associated with traditional cloud-based computing, making it a more efficient and effective approach for emergency response. The authors also discuss the potential applications of this technology for other types of disasters, such as earthquakes and floods. The article by Wang et al. [26] provides an overview of using edge computing and UAV networks for disaster management. The authors discuss the various challenges emergency responders face in disaster scenarios and argue that edge computing and UAV networks can help overcome these challenges by providing a more efficient and accurate way of collecting, processing, and analyzing data. The authors discuss the various technologies and techniques used in edge computing and UAV networks for disaster management, including sensors, cameras, and machine learning algorithms. They also highlight the potential benefits and challenges of using this technology in disaster scenarios and provide a roadmap for future research. Overall, aerial imaging has various applications across various industries and fields and is increasingly used to improve decision-making and optimize operations.

16.3 Edge Computing and Aerial Imaging Edge computing can be used in aerial imaging to process and analyze the vast amounts of data generated by aerial imaging systems while minimizing the latency and bandwidth requirements of transmitting this data to a centralized location. This is particularly important for real-time monitoring and surveillance applications, where data processing, analysis speed, and reliability are critical. Some specific use cases of edge computing in aerial imaging include:

Edge Computing in Aerial Imaging  367 • Real-time object detection and tracking: Edge computing can object detection and tracking algorithms on the data captured by aerial imaging systems, allowing for real-time monitoring and analysis of moving objects such as vehicles, animals, or people. • Environmental monitoring: Edge computing can process and analyze aerial imaging data for environmental monitoring applications, such as detecting and monitoring changes in land use, vegetation, or water quality. • Precision agriculture: Edge computing can process and analyze aerial imaging data for precision agriculture applications, such as identifying crop health, monitoring soil moisture, and optimizing irrigation and fertilizer application. • Disaster response: Edge computing can process and analyze aerial imaging data in real-time during disaster response operations, such as identifying areas of damage or detecting people needing rescue. By processing and analyzing data on edge, rather than transmitting it to a centralized location for processing, edge computing can reduce the latency and bandwidth requirements of aerial imaging systems and enable more efficient and responsive applications.

16.3.1 Research Perspective in Aerial Imaging As shown in Figure 16.6, from a research perspective, there are several areas where edge computing can be explored and further developed for aerial imaging applications. Some of these areas include: • Efficient and accurate object detection and tracking: There is ongoing research into developing more efficient and accurate object detection and tracking algorithms for aerial imaging applications. This involves exploring new techniques, such as deep learning, and optimizing these algorithms for edge computing platforms. • Integration with other technologies: There is potential for edge computing in aerial imaging to be integrated with other technologies such as drones, IoT sensors, and 5G networks. This could enable more comprehensive and responsive applications, such as real-time monitoring and control of autonomous drones.

368  Advances in Aerial Sensing and Imaging

Real-time Data Analysis

Optimized Drone Flight Paths

Object Detection & Tracking

Integration with other technologies

Resources Optimization Edge Computing in Aerial Imaging

Privacy

Improved Network Efficiency

Security

Real-time Data Processing

Dronw Navigation

Figure 16.6  Edge computing in aerial imaging.

• Resource optimization: There is ongoing research into optimizing computational resources for edge computing in aerial imaging applications. This includes developing more efficient algorithms and architectures for processing and analyzing data and exploring new techniques such as federated learning. • Privacy and security: There are concerns around privacy and security in aerial imaging applications, particularly regarding the storage and transmission of sensitive data. Ongoing research is into developing secure and privacy-­ preserving edge computing solutions for aerial imaging, such as encrypted data storage and decentralized processing. • Drone navigation: Edge computing can play a crucial role in 5G drone navigation by enabling faster and more reliable data processing, reducing latency and improving network efficiency. By processing and analyzing data on edge, rather than transmitting it to a centralized location for processing, edge computing can help drones to navigate and operate more efficiently and effectively in a 5G network

Edge Computing in Aerial Imaging  369

•

•

•

•

environment. Here are some specific ways that edge computing can be used in 5G drone navigation: Real-time data processing and analysis: Edge computing can process and analyze data captured by drone sensors in real-time, such as image and video data. This enables the drone to make more informed decisions and take immediate action without sending the data to a centralized location for processing. For example, edge computing can detect and avoid obstacles in the drone’s path or detect and track moving objects. Improved network efficiency: Edge computing can reduce the amount of data that needs to be transmitted over the 5G network, improving network efficiency and reducing latency. The drone can transmit only the most essential data back to the control center by processing data on edge rather than sending all the raw sensor data. Enhanced security and privacy: Edge computing can provide additional layers of security and confidentiality for drone navigation in a 5G network. For example, edge computing can encrypt and store data locally on the drone, reducing the risk of data breaches or interception. Optimized drone flight paths: Edge computing can analyze real-time data, such as weather conditions, traffic patterns, and other environmental factors, to optimize the drone’s flight path and ensure efficient navigation. This can help to reduce travel time, conserve battery life, and improve overall flight efficiency.

In summary, edge computing can significantly enhance the capabilities of 5G drone navigation by enabling real-time data processing and analysis, improving network efficiency, enhancing security and privacy, and optimizing drone flight paths.

16.3.2 Edge Architectures Here are some of the best edge computing architectures for aerial imaging designed for specific applications. • Cloudlet-based architecture: This architecture involves using a network of cloudlets, which are small-scale data centres closer to the network’s edge, to process and analyze

370  Advances in Aerial Sensing and Imaging data from aerial imaging systems. This reduces the latency and bandwidth requirements of transmitting data to a centralized cloud while still providing the scalability and reliability of cloud computing. The article by [27] is given in Figure 16.7. • Mobile edge computing (MEC) architecture involves deploying computing resources, such as servers and storage, at the network’s edge, such as on mobile base stations or roadside units. MEC can process and analyze data from aerial imaging systems in real-time while providing low-­ latency connectivity. The architecture by the authors [28] is depicted as follows in Figure 16.8. • Distributed edge computing architecture: This architecture involves distributing computing resources across multiple nodes or devices, such as drones or IoT sensors, to enable distributed processing and analysis of aerial imaging data. This reduces the need to transmit data over long distances and can help improve aerial imaging systems’ efficiency and responsiveness. The authors [29] have proposed an architecture for UAV-based early fire detection given as follows in Figure 16.9. • Distributed fog architecture: The fog nodes are spread across the farm to process and analyze data from drones in real time. The architecture uses a hierarchical clustering

Remote Cloud Internet

Network Switch

Edge Cloudlet Base station

Mobile App

Figure 16.7  Cloudlet-based architecture [27].

Edge Cloudlet Base station

Edge Computing in Aerial Imaging  371 WSNs Layer

Drones Layer WSN 1

MECs Server Layer

Offloading Hubs

Drone 1

MEC Server 1

Drone 2

MEC Server 2

Drone n

MEC Server n

Smart Contact

WSN 2 Agent

WSN n

IoT devices

Blockchain Network

Offloading Execution

Figure 16.8  MEC architecture [28].

Cloud service Image Classification Service

Decision Making Service

Context Broker NGSI API

Cloud Classif`ication scores Fog NGSI Client Image Classif`ication Service Edge

Image

Raspberry-Pi

Image

Raspberry-Pi

Figure 16.9  Distributed edge architecture [29].

Image

Raspberry-Pi

372  Advances in Aerial Sensing and Imaging

Clients

REST API Validator

Consensus Engine

Transaction Processor

Validator Node

Blockchain Node

SDN Controller APPs Adapters Protocol

Distributed SDN Controller

IoT Data Platform

Blockchain Client

Network Monitering

Security

Automated Control

Simulated Switch Virtualization Hardware

Fog Node

Sensor 1 Sensor 2 Sensor 3

A/D Converter

Sensor N

Microcontroller

Power Management

IoT Device

Figure 16.10  Fog architecture [30].

Actuator

Edge Computing in Aerial Imaging  373 algorithm to organize the fog nodes into a cluster, with each cluster having a primary node that performs data aggregation and decision-making. The proposed architecture was shown to improve the efficiency and reliability of precision farming systems compared to traditional cloud-based architectures. The architecture by the authors [30] is given as follows and depicted in Figure 16.10. The architecture also deals with the security aspects. • Federated learning architecture: This architecture was proposed in a research paper titled “Learning in the Air: Secure Federated Learning for UAV-Assisted Crowdsensing” by Want. et al. [31]. In this architecture, edge devices, such as drones, collaborate to train machine learning models using a federated learning approach. The trained models are then used to provide decision support for precision farming applications. The proposed architecture was shown in Figure 16.11 to reduce communication overhead and improve the privacy of precision farming systems compared to traditional centralized learning approaches. • Hybrid edge-cloud architecture with Blockchain was proposed in a research paper titled “A Hybrid Edge-Cloud Architecture for Secure and Private Precision Agriculture with Blockchain” by P. Loke et al. [32]. In this architecture, edge devices like drones process and analyze data locally, while Blockchain ensures the data’s security and privacy. Cloud computing is used to provide scalable and reliable processing and analysis. The proposed architecture improved the protection and confidentiality of precision farming systems compared to traditional cloud-based architectures.

UAV

4 local model update

3

1 task request

4

Base station MEC node

3 global model 2

Task publisher 5 trained model

Figure 16.11  Federated learning architecture [31].

Local model training Global model aggregation

374  Advances in Aerial Sensing and Imaging Some architectures that can be used explicitly for application-specific implementation are as follows. • Multi-Agent Edge Architecture: In this architecture, multiple agents, such as drones and robots, are deployed for crop health monitoring. The agents communicate with each other and a central edge server to perform real-time analysis of crop health data. The proposed architecture improves the efficiency and effectiveness of precision agriculture systems by enabling dynamic crop health monitoring. • Edge-Cloud Hybrid Architecture with Deep Learning: In this architecture, edge devices, such as cameras and sensors, perform real-time object detection and tracking using deep learning algorithms, while cloud computing is used for data storage, long-term analysis, and decision-making. The proposed architecture leverages the power of deep learning to improve the accuracy and efficiency of object detection and tracking. • Mobile Edge Computing Architecture with Edge-Cloud Collaboration: In this architecture, edge devices, such as drones and cameras, are connected to mobile edge computing (MEC) servers, which perform local processing and analysis of video streams to detect and track objects. The MEC servers also collaborate with the cloud for data storage and further study. • Decentralized Edge Architecture with Peer-to-Peer Communication: In this architecture, edge devices are connected in a decentralized network and use peer-to-peer communication to share data and perform real-time object detection and tracking. The proposed architecture is scalable, fault-tolerant, and efficient, eliminating the need for a centralized server. • Collaborative Edge Architecture with Multi-Camera Tracking: In this architecture, multiple cameras are deployed in the field to capture video streams, which edge devices then process and analyze. The proposed architecture uses multi-camera tracking algorithms to improve object detection and tracking accuracy and efficiency. • Intelligent Edge Architecture with Transfer Learning: In this architecture, edge devices are equipped with transfer learning algorithms, which enable the devices to learn from

Edge Computing in Aerial Imaging  375

•

•

•

•

•

pre-trained models and perform real-time object detection and tracking. The proposed architecture is efficient and scalable, eliminating the need for large amounts of labeled data for training. Hierarchical Edge Architecture with Adaptive Computing: In this architecture, multiple tiers of edge devices are used to perform real-time object detection and tracking, with each level performing increasingly complex processing and analysis tasks. The proposed architecture uses adaptive computing techniques to optimize the allocation of computing resources between the tiers, improving the overall system performance. Edge-Cloud-Fog Architecture with Reinforcement Learning: In this architecture, edge devices, fog nodes, and cloud servers collaborate to perform real-time object detection and tracking using reinforcement learning algorithms. The proposed architecture leverages the power of edge, cloud, and fog computing to improve object detection and tracking accuracy and efficiency. Multi-Agent Edge Architecture with Blockchain: In this architecture, multiple agents, such as drones and cameras, are deployed for real-time object detection and tracking. The agents communicate with each other and a central edge server using blockchain technology, which ensures data privacy, integrity, and security. The proposed architecture is efficient, secure, and scalable, eliminating the need for a centralized server and enabling real-time agent collaboration. Edge-Cloud Hybrid Architecture with 3D Modeling: In this architecture, edge devices, such as drones and cameras, capture high-resolution images of the city, which are then processed and analyzed by deep learning algorithms to create 3D models of the urban environment. The proposed architecture leverages the power of edge and cloud computing to improve the accuracy and efficiency of 3D modeling. Collaborative Edge Architecture with Multi-Sensor Fusion: In this architecture, multiple sensors, such as cameras and weather stations, are deployed to collect real-time data on different urban parameters. The proposed architecture uses multi-sensor fusion algorithms to combine the data from various sensors and perform real-time analysis of the urban environment.

376  Advances in Aerial Sensing and Imaging • Decentralized Edge Architecture with Blockchain: In this architecture, edge devices are connected in a decentralized network and use blockchain technology to securely share data and perform real-time analysis of the urban environment. The proposed architecture ensures data privacy, integrity, and security, eliminating the need for a centralized server. • Mobile Edge Computing Architecture with Virtual Reality: In this architecture, edge devices, such as smartphones and tablets, access virtual reality applications for urban planning and development. The proposed architecture leverages the power of mobile edge computing to enable real-time interaction with virtual urban environments. • Intelligent Edge Architecture with Reinforcement Learning: In this architecture, edge devices, such as sensors and cameras, are equipped with reinforcement learning algorithms to perform real-time analysis of the urban environment. The proposed architecture leverages the power of intelligent edge computing to improve the accuracy and efficiency of urban planning and development. • Edge-Cloud-Fog Architecture with Autonomous Vehicles: In this architecture, edge devices, fog nodes, and cloud servers collaborate to enable autonomous vehicles for urban transportation. The proposed architecture uses edge computing for real-time traffic data analysis and cloud computing for long-term research and decision-making. • Hierarchical Edge Architecture with Multi-Objective Optimization: In this architecture, multiple tiers of edge devices are used to perform real-time analysis of different urban parameters, with each level performing increasingly complex processing and analysis tasks. The proposed architecture uses multi-objective optimization algorithms to optimize the allocation of computing resources between the tiers, improving the overall system performance.

16.4 Comparative Analysis of the Aerial Imaging Algorithms and Architectures The following table gives a comparison of the most common algorithms available for aerial imaging, as elaborated in Table 16.1. However, it’s

Edge Computing in Aerial Imaging  377 Table 16.1  Algorithm comparison. Algorithm

Description

Pros

Thresholding

Segments an image Simple and fast, into binary suitable for regions based on a segmenting chosen threshold objects from a value background

Ideal for feature Edge Detection Detects the extraction boundaries between objects in and object an image recognition

Cons Sensitive to variations in lighting and image quality Sensitive to image noise and can produce false positives

Feature Matching

Compares features Robust to variations It can be computationally (e.g. corners, in lighting and expensive edges) in different image quality, suitable for object images to identify recognition and everyday objects tracking

Object Detection

Can accurately Uses machine learning detect specific algorithms to objects, suitable identify specific for large-scale objects (e.g. analysis buildings, trees) in an image

Image Divides an image Segmentation into multiple segments or regions based on similarity in pixel values

Requires training data and can be computationally expensive

Can accurately identify objects and their boundaries

It can be sensitive to variations in lighting and image quality and can produce oversegmentation or undersegmentation

Spectral Analysis

Analyzes the spectral Can identify properties of an specific features image (e.g. using (e.g. vegetation NDVI to identify health) vegetation)

Requires knowledge of spectral properties and may not work for all applications

Classification

Assigns pixels to Can accurately different classes classify different based on their land cover types spectral properties

Requires training data and can be computationally expensive

378  Advances in Aerial Sensing and Imaging important to note that the choice of algorithm depends on the specific application and goals of the analysis [33, 34]. The comparison of the most common edge computing architectures is also elaborated in Table 16.2. Again, the choice of architecture depends on Table 16.2  Architecture comparison. Architecture

Description

Pros

Cons

Standalone Edge Device

A single device that performs edge computing tasks

Simple to implement, low latency

Limited processing power and storage capacity

Fog A distributed Computing architecture that extends cloud computing to the edge of the network

Scalable, supports real-time processing

Requires highbandwidth network connectivity

Mobile Edge An architecture that Computing combines edge computing with mobile networks

Low latency, supports mobility

Limited processing power and storage capacity

Cloudlet

High processing power and storage capacity, low latency

Requires highbandwidth network connectivity, may not be suitable for mobile applications

A small data center that is deployed at the edge of the network

Scalable, Requires complex Distributed A decentralized fault-tolerant coordination and Edge architecture that synchronization Computing distributes computing among edge tasks among multiple edge devices devices Hybrid Edge- An architecture that Cloud combines edge Computing computing and cloud computing for optimal performance

Can balance processing tasks between edge devices and cloud servers

Requires careful management of data transfer and workload distribution

Edge Computing in Aerial Imaging  379 the specific requirements and constraints of the application. Factors such as processing power, storage capacity, network connectivity, and mobility requirements should be considered when selecting an edge-computing architecture for aerial imaging.

16.5 Discussion As shown in Table 16.1, each aerial imaging algorithm has pros and cons. According to the requirements, a suitable algorithm can be identified and used for efficiently processing the data. Similarly, per the comparison of the edge computing architectures in Table 16.2, the appropriate architecture can be chosen per the applications’ requirements. Some of the research areas that the upcoming researchers can explore include • Image analysis and processing: This area involves developing algorithms and techniques for processing and analyzing aerial images, including feature extraction, object recognition, and image enhancement. • Remote sensing and geospatial analysis: This area uses aerial imagery for mapping, land-use analysis, and environmental monitoring. • Machine learning and artificial intelligence: This area explores machine learning and artificial intelligence techniques for analyzing and interpreting aerial imagery, including developing deep learning models for object recognition and classification. • Sensor technology: This area involves the development of new sensors and imaging systems for aerial platforms, including high-resolution cameras, lidar, and thermal imaging. • Applications of aerial imaging: This area focuses on developing new applications and uses cases for aerial imaging, including disaster response, urban planning, agriculture, and wildlife conservation. • Data management and storage: This area involves developing new methods for managing and storing large volumes of data generated by aerial imaging systems, including cloudbased solutions and distributed storage systems.

380  Advances in Aerial Sensing and Imaging Overall, the research areas in aerial imaging are diverse and interdisciplinary, spanning fields such as computer vision, remote sensing, geospatial analysis, and data management.

16.6 Conclusion This chapter gives an overview of aerial imaging applications. It overviews the available literature that uses real-time aerial imaging. The concept of edge in the field has also been reviewed. The various edge-based architectures, their applications, and the development components have been given in detail. In conclusion, edge computing technology significantly impacts aerial imaging applications. By processing data closer to the source, edge computing allows for real-time analysis and decision-making, improving the efficiency and effectiveness of aerial imaging tasks. In particular, edge computing enables faster data processing, reduced latency, and enhanced data security, making it an essential technology for surveillance, mapping, and environmental monitoring applications. As edge computing technology continues to evolve, it is expected to play an increasingly important role in the future of aerial imaging.

References 1. Kumar, S., Jain, A., Rani, S., Alshazly, H., Idris, S.A., Bourouis, S., Deep neural network based vehicle detection and classification of aerial images. Intell. Auto. Soft Comput., 34, 1, 2022. 2. Wang, B. and Xu, B., A feature fusion deep-projection convolution neural network for vehicle detection in aerial images. PLoS One, 16, 5, e0250782, 2021. 3. Shen, J., Liu, N., Sun, H., Vehicle detection in aerial images based on lightweight deep convolutional network. IET Image Process., 15, 2, 479–491, 2021. 4. Singh, A.P., Yerudkar, A., Mariani, V., Iannelli, L., Glielmo, L., A bibliometric review of the use of unmanned aerial vehicles in precision agriculture and precision viticulture for sensing applications. Remote Sens., 14, 7, 1604, 2022. 5. Zhan, B.H., Sathyendranath, S.M., Platt, T., Satellite-based monitoring of harmful algal blooms using convolutional neural networks. Remote Sens., 12, 3, 511, 2020. 6. Pacheco-López, J.G., Vargas-Mendoza, J.A., Ruiz-Suárez, G.L., A machine learning approach for water quality monitoring using satellite imagery. Environ. Monitoring Assess., 192, 9, 607, 2020.

Edge Computing in Aerial Imaging  381 7. Dhammapala, V., Debnath, B.K., Mohan, S.M., A remote sensing approach to monitor land surface temperature and urban heat island effect in a tropical city. Sustain. Cities Soc., 55, 102004, 2020. 8. Wang, T., Wang, C.Y., Tong, L., Assessing air pollution from wildfire smoke using ground-based and remote sensing techniques. Environ. Pollution, 265, 114734, 2020. 9. Doshi, S.D., Satpati, B.A., Mundra, D.C., Automated detection and classification of coastal wetlands using satellite imagery and machine learning algorithms. Environ. Monitoring Assess., 192, 11, 700, 2020. 10. Zhang, N. et al., Edge-assisted aerial sensing for environmental monitoring, in: 2021 IEEE International Conference on Edge Computing (EDGE), Dec. 2021. 11. Islam, M.M. et al., Aerial and ground-based IoT architecture for environmental monitoring, in: 2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), Nov. 2020. 12. Lee, K. et al., Smart edge-cloud architecture for air quality monitoring using aerial imaging, in: 2021 IEEE International Conference on Communications (ICC), Jun. 2021. 13. Liu, J. et al., Drone-based environmental monitoring using edge and fog computing, in: 2021 IEEE International Conference on Edge Computing (EDGE), Dec. 2021. 14. Islam, S.S., Kumar, N., Rakib, M.A., Chowdhury, M.S., Bhuiyan, M.Z.H., Edge-assisted aerial sensing for disaster management, in: 2020 IEEE Global Humanitarian Technology Conference (GHTC), IEEE, 2020, October. 15. Huang, R., Fang, X., Li, C., Li, Y., Wang, Y., Li, Y., Aerial image-based environmental monitoring using edge and fog computing, in: 2021 IEEE International Conference on Edge Computing (EDGE), IEEE, 2021, December. 16. Nguyen, N.C., Jiang, C., Ly, V., Cheng, K.C., Chen, C.F., Aerial imaging-based water quality. IEEE Trans. Geosci. Remote Sens., 59, 8, 6261–6275, 2021. 17. Ahmed, M.S., Abid, M., Hussain, A., Zhang, Y., Aslam, A., urban sprawl detection using aerial imaging and edge computing: A case study of Houston, Texas. IEEE Access, 9, 16130–16141, 2021. 18. Li, L., Chen, X., He, Y., Zhang, C., Zhang, X., Exploring the potential of aerial computing and edge computing for sustainable urban development. Sustainability, 12, 7, 2955, 2020. 19. Zhang, J., Liu, H., Xu, X., Yu, L., Wang, C., Using aerial computing and edge computing for real-time traffic monitoring in smart cities. IEEE Access, 7, 45506–45516, 2019. 20. Zhang, Y., Chen, X., Zhang, C., He, Y., Li, L., Aerial computing and edge computing for disaster management in urban areas. Int. J. Environ. Res. Public Health, 15, 8, 1647, 2018. 21. Wang, W., Lu, Y., Liu, K., Optimizing urban planning and development using aerial computing and edge computing. Sustain. Cities Soc., 34, 73–79, 201.

382  Advances in Aerial Sensing and Imaging 22. Wu, Y., Huang, J., Zhang, Y., Liu, J., Edge computing-enabled aerial image processing for emergency response: A comprehensive survey. J. Netw. Comput. Appl., 183, 102980, 2021. 23. Zhao, S., Zhang, Y., Wang, L., Xiong, N., Song, H., Real-time disaster response using aerial imagery and edge computing. J. Parallel Distrib. Comput., 137, 1–12, 2020. 24. Beheshti, R., Vaezi, M., Beheshti, R., Real-time mapping of wildfires using edge computing and aerial imaging. J. Ambient Intell. Humaniz. Comput., 10, 10, 4053–4063, 2019. 25. Zhao, X., Chen, Y., Zhou, Y., Chen, S., Aerial imagery-based road network extraction for emergency response using edge computing. Future Gener. Comput. Syst., 99, 80–88, 2019. 26. Wang, L., Xiong, N., Wang, Z., Edge computing-enabled unmanned aerial vehicle networks for disaster management: Opportunities and challenges. IEEE Commun. Mag., 57, 8, 38–43, 2019. 27. Mahesar, A.R., Lakhan, A., Sajnani, D.K., Jamali, I.A., Hybrid delay optimization and workload assignment in mobile edge cloud networks. Open Access Library J., 5, 9, 1–12, 2018. 28. Luo, S., Li, H., Wen, Z., Qian, B., Morgan, G., Longo, A., Ranjan, R., Blockchain-based task offloading in drone-aided mobile edge computing. IEEE Netw., 35, 1, 124–129, 2021. 29. Kalatzis, N., Avgeris, M., Dechouniotis, D., Papadakis-Vlachopapadopoulos, K., Roussaki, I., Papavassiliou, S., Edge computing in IoT ecosystems for UAV-enabled early fire detection, in: 2018 IEEE International Conference on Smart Computing (SMARTCOMP), IEEE, pp. 106–114, 2018, June. 30. Padhy, S., Alowaidi, M., Dash, S., Alshehri, M., Malla, P.P., Routray, S., Alhumyani, H., AgriSecure: A fog computing-based security framework for agriculture 4.0 via blockchain. Processes, 11, 3, 757, 2023. 31. Wang, Y., Su, Z., Zhang, N., Benslimane, A., Learning in the air: Secure federated learning for UAV-assisted crowdsensing. IEEE Trans. Netw. Sci. Eng., 8, 2, 1055–1069, 2020. 32. Loke, P., Gope, P., Krishnamurthy, P., Doss, R., Li, W., A Hybrid edge-cloud architecture for secure and private precision agriculture with blockchain. IEEE Trans. Ind. Inf., 17, 3, 1962–1972, 2021. 33. Bhola, S.S., Noonia, A., Sharma, B., Kumar Narang, S., A Status quo of machine learning algorithms in smart agricultural systems employing IoTbased WSN: Trends, challenges and futuristic competences, in: Machine Intelligence, Big Data Analytics, and Iot in Image Processing: Practical Applications, pp. 177–195, 2023. 34. Sandeep, S. S., Taj Kiran, V., Prashant, J., IoT based smart home surveillance and automation, in: International Conference on Computing, Power and Communication Technologies (GUCON), pp. 786–790, 2018.

17 Aerial Sensing and Imaging Analysis for Agriculture Monika Kajal1* and Aditi Chauhan2 1

DST-Centre for Policy Research, Panjab University, Chandigarh, India 2 DPIIT-IPR, Panjab University, Cell Chandigarh, India

Abstract

Aerial sensing is gathering information about an area from a distance using aircraft or unmanned aerial vehicles (UAV). UAVs fly approximately 120 meters over the ground, while aircraft fly around 6000 meters. Aerial imaging is more desirable when remote sensing is ineffective and in applications that call for the highest possible spatial and temporal resolutions. There has been a notable advancement in the field of aerial photography brought about by the substitution of digital sensors for cameras. Some advantages of digital sensors over analogue cameras include increased spatial resolution, simultaneous image capture in various electromagnetic spectrum bands, unmanned aerial vehicles (UAVs) cost-effectiveness, and temporal affordability. This technique could be more flawless; there is an opportunity for improvement. To make this technology more practical and progress agriculture, it is essential to examine the effects of technology at every stage of agriculture, such as assessing yield and fertilizer response, plant, and crop farming, planting, weed mapping and management, aerial mustering, geofencing, and virtual perimeters, etc. This study examines how aerial sensing and imaging technology is used worldwide in agriculture and animal husbandry. Assessment will help investigate the challenges posed by aerial sensing applications and make a case for advancements in this area. This section is also vital for understanding its applications in regions that have yet to use these techniques previously. Keywords:  Aerial sensing, unmanned aerial vehicles, precision agriculture, livestock farming

*Corresponding author: [email protected] Sandeep Kumar, Nageswara Rao Moparthi, Abhishek Bhola, Ravinder Kaur, A. Senthil, and K.M.V.V. Prasad (eds.) Advances in Aerial Sensing and Imaging, (383–410) © 2024 Scrivener Publishing LLC

383

384  Advances in Aerial Sensing and Imaging

17.1 Introduction Aerial sensing and imaging analysis are rapidly becoming popular crop and livestock management tools. These technologies use remote sensing systems, such as unmanned aerial vehicles (UAVs), satellites, and aircraft, to collect high-resolution data that can be used to monitor and analyze agricultural systems. This data can help farmers and ranchers make informed decisions about crop and livestock management, leading to increased yields, improved resource use efficiency, and reduced environmental impact [1]. Using aerial sensing and imaging analysis in agriculture is a concept that has been introduced previously. Farmers have relied on satellite imagery for decades to track weather patterns, soil moisture levels, and vegetation growth. However, recent technological advancements have allowed for more detailed and accurate data collection and faster data processing and analysis. One of the critical benefits of aerial sensing and imaging analysis is its ability to provide a comprehensive view of agricultural systems. Farmers must physically inspect their crops and livestock with traditional methods to identify issues such as disease or pest infestations. However, aerial imaging can detect these issues from above, providing a more efficient and accurate way to identify and address problems [2]. In addition to identifying issues, aerial sensing and imaging analysis can monitor crop and livestock health over time. For example, farmers can identify nutrient deficiencies or water stress in their crops by analyzing changes in vegetation growth patterns. This information can then be used to adjust fertilizer or irrigation applications to optimize crop growth. Similarly, aerial imaging can be used to monitor livestock health and behaviour. By tracking animal movements and identifying abnormal behaviour patterns, farmers can identify issues such as disease or injury and take action to prevent further harm. This technology can also monitor grazing patterns, allowing farmers to optimize their use of pasture resources and reduce overgrazing [3]. Aerial sensing and imaging analysis can also provide valuable information for precision agriculture. By collecting data on soil properties and crop growth patterns, farmers can create detailed maps that show variations in these parameters across their fields. This information can then be used to optimize fertilizer and pesticide applications, leading to increased crop yields and reduced input costs. Overall, aerial sensing and imaging analysis can potentially revolutionize managing agricultural systems. By providing detailed and accurate data on crop and livestock health and resource use efficiency, this technology can help farmers and ranchers make informed decisions that improve their bottom line while

Aerial Sensing for Crop and Livestock  385 reducing environmental impact. As this technology advances, it will likely become an increasingly important tool for sustainable agriculture [4]. Recently, there has been a growing interest in using aerial sensing and imaging technologies for crop and livestock management. With the help of drones and other aerial vehicles, farmers and ranchers can obtain high-­ resolution images and data that provide a detailed view of their fields and pastures. Workflow of aerial sensing and imaging is detailed in Figure 17.1. This technology can revolutionize agriculture, enabling farmers to make more informed decisions about planting, harvesting, and animal husbandry. Aerial sensing and imaging technologies offer several advantages over traditional methods of monitoring crops and livestock. For one, they provide a much larger and more comprehensive landscape view. Farmers can quickly and efficiently survey an entire field or pasture with a drone or other aerial vehicle, capturing data on plant health, soil moisture, and other factors affecting crop yield and animal health. This data can be analyzed in real-time using machine learning algorithms, allowing farmers to make rapid decisions about optimizing their operations [5]. Another advantage of aerial sensing and imaging technologies is their ability to capture high-resolution images and data. Farmers can obtain detailed images of their crops and livestock from above with advanced sensors and cameras equipped with drones. These images can reveal information about plant health, crop density, and other factors difficult to detect from the ground. For livestock management, drones can help farmers monitor herd health, track animal movements, and identify potential problems before they become serious. The use of aerial sensing and imaging technologies for crop and livestock management is still in its early stages. Still, a growing body of research and case studies demonstrates its potential. For example, a study by researchers at the University of Illinois found that

User

Maps Tables & Reports

Figure 17.1  Flow of energy and information in aerial sensing and imaging.

386  Advances in Aerial Sensing and Imaging using drones to monitor crop growth and health can increase crop yields by up to 9%. Other studies have shown that using aerial imaging technologies for livestock management can help reduce disease incidence and improve overall animal health [6]. Despite the many benefits of aerial sensing and imaging technologies, challenges must be addressed to realize their potential fully. One of the biggest challenges is the cost of the technology itself. Drones and other aerial vehicles can be expensive to purchase and maintain, and the cost of advanced sensors and cameras can also be prohibitive for some farmers. Table 17.1  Advantages and disadvantages of aerial sensing and imaging in agriculture. Advantages

Disadvantages

• Aerial sensing and imaging can help farmers practice precision agriculture in tiny, targeted regions. This reduces water and fertilizer use, saving money and the environment.

• Small farmers may be unable to afford aerial sensing and imaging equipment and software.

• Aerial sensing and imaging can detect insect infestations, nutritional deficits, and diseases early, allowing farmers to take action before they spread. This reduces crop losses and boosts yields.

• Some farmers need to gain the technical abilities to operate and analyze aerial sensing and imaging equipment.

• Aerial sensing and imaging can help farmers identify fields that are performing better or worse so they can modify management practices. Optimizing inputs and management practices boost yields and profits.

• Aerial sensing and imaging can provide significant crop health and growth data, but interpreting it requires expertise. Experts may help farmers analyze data and make judgments.

• Aerial sensing and imaging speed up field inspections, scouting, and mapping. Farmers can then focus on marketing and business management.

• Wind, rain, and cloud cover can limit aerial sensing and imaging.

• Aerial sensing and imaging can replace dangerous, time-consuming physical inspections. In significant or difficult-toaccess fields, this can increase farmer and worker safety.

• Aerial sensing and imaging can generate privacy concerns for farmers and landowners, primarily if used for surveillance.

Aerial Sensing for Crop and Livestock  387 Table 17.2  Advantages and disadvantages of aerial sensing and imaging in livestock farming. Advantages

Disadvantages

• Aerial sensing and imaging can provide real-time data on cattle health, behaviour, and movement. This helps farmers spot early illness or injury symptoms and prevent disease spread.

• Aerial sensing and imaging can be costly, especially for small-scale farmers.

• Aerial sensing and imaging can help farmers manage resources by giving feed and water usage data, herd number and composition, and pasture condition. This allows farmers to optimize productivity and avoid waste.

• Some farmers may need more technical abilities to operate and analyze aerial sensing and imaging equipment.

• Aerial sensing and imaging can help farmers boost output by recognizing failing or overperforming operations. This data can inform feed, breeding, and management decisions.

• Aerial sensing and imaging can generate privacy concerns for farmers and animal owners, primarily if utilized for surveillance. Consumers wary about livestock management technology may likewise be concerned.

• Real-time cattle behaviour and movement data from aerial sensing and photography reduce human labor. This helps farmers decide when to feed, shift, or manage their livestock, saving work and enhancing efficiency.

• Aerial sensing and imaging can provide essential data on livestock health and behaviour, but interpreting it can be challenging and require specialized skills. Experts may help farmers analyze data and make judgements.

• Aerial sensing and imaging can help farmers monitor cattle health, behaviour, and living circumstances. This can help farmers spot difficulties early and take steps to protect their animals.

• Wind, rain, and cloud cover can hamper aerial sensing and imagery. Extreme weather might also endanger animals and farmers if employed for Monitoring and management.

388  Advances in Aerial Sensing and Imaging Additionally, there are regulatory issues to consider, as many countries restrict using drones for commercial purposes. Another challenge is the processing and analysis of the data obtained through aerial sensing and imaging. While drones and other aerial vehicles can capture vast data, it can be difficult for farmers to understand. Machine learning algorithms and other data analysis tools can help, but they require specialized knowledge and expertise to use effectively [4]. Like any other contemporary technology, aerial sensing and imaging have advantages and disadvantages, as outlined in Tables 17.1 and 17.2. Despite these challenges, the potential benefits of aerial sensing and imaging technologies for crop and livestock management are vibrant. As technology continues to evolve and become more accessible, we can expect to see more and more farmers and ranchers adopting it as a vital tool in their operations. With its ability to provide a comprehensive landscape view and capture detailed images and data, aerial sensing and imaging can revolutionize agriculture, leading to more efficient and sustainable farming practices and healthier crops and livestock.

17.2 Experimental Methods and Techniques This section discusses effective techniques that can be used for aerial sensing and imaging present in the literature as follows: a) Aerial photography: The oldest form of remote sensing remains one of the most reliable and widely used sources of remotely sensed data, despite the increasing availability of more sophisticated imaging systems. Aerial photography using film and digital cameras is still one of the most cost-effective remote sensing methods. High spatial resolution enables one to detect and analyze patterns at excellent spatial scales [7]. b) Unmanned aerial vehicle (UAV): Commonly known as drones, which have become increasingly popular in recent years. These drones can carry a range of sensors, such as optical and thermal cameras, and can be used for various applications, including agriculture, environmental monitoring, and infrastructure inspection. For example, the imagery data layers in ArcGIS Living Atlas of the World include satellite, aerial, and remote sensing content, providing a range of current, forecasted, and historical datasets for

Aerial Sensing for Crop and Livestock  389 researchers and practitioners (https://www.esri.com). There are four significant types of UAVs are identified, comparative analysis of their average speed, altitude, and average control range in depicted in Figure 17.2: • • • •

Multi-rotor UAVs Fixed-wing UAVs Single-rotor Helicopter Fixed-wing-multi-rotor Hybrid UAVs

Average airspeed (m/s) 40 30

18.8

20 10

Altitude (Km) 4 3 2 1 0

30 15.2

7

0

4

500 400 300 200 100 0

3 2 0.125 Altitude (Km)

Average control range (Km) 500

5

2

30

Avg. control range (Km)

Multi-rotor UAVs Fixed-wing UAVs Single-rotor Helicopter Fixed-wing-multi-rotor Hybrid UAVs

Multi-rotor UAVs Fixed-wing UAVs Single-rotor Helicopter Fixed-wing-multi-rotor Hybrid UAVs

Multi-rotor UAVs Fixed-wing UAVs Single-rotor Helicopter Fixed-wing-multi-rotor Hybrid UAVs

Figure 17.2  Feature comparison of different types of UAVs.

0.75

pavement

Conv

Cross-modal Feature Alignment

Feature Maps

grass field

0.70

sand

0.75 0.70 0.70

feature extractor

Label Graph Mining and Label Embedding

pa ld ve m en t

ba r

Figure 17.3  The proposed CM-GM framework [8].

ee

LSTM

fie

LSTM

il

LSTM

Bi-LSTM Predictor

tr

LSTM

so

LSTM

e

LSTM

LSTM

gr as

LSTM

s w at er

Sequence Generated Based on Label Graph

water

390  Advances in Aerial Sensing and Imaging There are various models to interpret aerial imagery. Cross-Modal Representation Learning and Label Graph Mining-based Residual MultiAttentional CNN-LSTM framework is the most recent model proposed in this instance. Within this framework, a residual multi-attentional convolutional neural network is developed to extract object-level image features. Additionally, semantic annotations are incorporated into the language model to create a labeled graph that can be further mapped by advanced graph convolutional networks (GCN). Using these cross-modal feature representations, object-level visual features will be enhanced and aligned with GCN-based label embeddings (image, graph, and text). Figure 17.3 depicts the input of aligned optical signals into a bi-LSTM subnetwork based on the constructed label graph.

17.3 Aerial Imaging and Sensing Applications in Agriculture 17.3.1 Assessing Yield and Fertilizer Response A study entitled ‘World fertilizer trends and outlook to 2020’ (FAO Citation 2017) reported that the global demand for fertilizer nutrients by the end of 2020 (201.66 million tons) was about 10% higher than that in 2016. Types of data that may be relevant for assessing yield and fertilizer response using aerial sensing and imaging: • Vegetation indices: A common way to assess plant health is by using vegetation indices, such as the Normalized Difference Vegetation Index (NDVI) or the Enhanced Vegetation Index (EVI) [9]. Scale and interpretation of the NDVI is depicted in Figure 17.4. These indices are calculated from the reflectance of visible and near-infrared light, which can be captured using aerial sensing and imaging [10]. Comparative analysis of NDVI and EVI are detailed in Table 17.3. NDVI and EVI are helpful tools for monitoring vegetation health and productivity, but the EVI provides a more comprehensive and balanced view of vegetation cover. However, the choice between NDVI and EVI ultimately depends on the research question and available data. • Yield data: To assess yield, you may need to collect data on the crop produced per unit area. This can be done through

Aerial Sensing for Crop and Livestock  391 Sparse vegetation (0.10.5)

Clouds & Water (-ve value)

Bare Soil (Near zero)

Dense green vegetation 90.6 or above)

Figure 17.4  The NDVI scale and interpretation (Range varies from -1.0 to 1.0).

Table 17.3  Comparison of the values/data for the NDVI and the EVI. Attribute

NDVI

EVI

Formula

The NDVI formula is (NIR - Red)/(NIR + Red), where NIR is the nearinfrared band, and Red is the red band.

The EVI formula is a more complex calculation that takes into account the blue band and corrects for atmospheric influences: 2.5 * ((NIR - Red) / (NIR + 6Red - 7.5Blue + 1)).

Sensitivity

NDVI is less sensitive to frequent atmospheric and soil noises.

The EVI is more sensitive to changes in vegetation cover than the NDVI because it corrects for atmospheric influences and provides a more balanced representation of vegetation cover.

Saturation

NDVI values can saturate at high vegetation cover levels, meaning there is no discernible difference in NDVI values between very dense and highly dense vegetation.

EVI values do not saturate as quickly as NDVI values, so they can provide more information about dense vegetation cover

Interpretation

NDVI values range from -1 to 1, with higher values indicating healthier vegetation.

EVI values range from -1 to 1, but the interpretation differs slightly. EVI values closer to zero indicate poor vegetation health, while values closer to one indicate healthy vegetation.

392  Advances in Aerial Sensing and Imaging ground truthing, where crop samples are collected and weighed, or through remote sensing techniques such as drones or satellites [11]. • Soil data: Fertilizer response is often influenced by soil properties such as nutrient content, pH, and texture. Soil data such as soil type, organic matter content, and nutrient levels can be collected through soil sampling or remote sensing techniques [12]. Aerial sensing can be used to study various aspects of soil, such as: ◦◦ Erosion risk assessment and Monitoring. ◦◦ Support of erosion modeling-assessment of model input parameters, most commonly vegetation factors. ◦◦ Indirect erosion mapping (primarily via vegetation assessment as an indirect indicator of land degradation status). ◦◦ Specific soil properties may indicate erosion status, focusing primarily on soil color, iron oxides, clay minerals, and organic matter. ◦◦ Direct mapping of linear erosion features, such as rills and gullies, and natural mapping of areal erosion phenomena (erosion patterns). • Weather data: Weather conditions can also influence crop yield and fertilizer response. Weather stations or remote sensing techniques can collect data such as temperature, precipitation, and wind speed [13]. • Fertilizer application data: To assess fertilizer response, it’s essential to know the amount and timing of fertilizer application. This data can be collected through farmer surveys, fertilizer sales records, or sensors attached to fertilizer applicators[14]. Combining these data types may be possible to assess yield and fertilizer response using aerial sensing and imaging.

17.3.2 Plant and Crop Farming Aerial sensing and imaging have become increasingly popular in plant and crop farming due to their potential to provide farmers with valuable insights into crop health, yield potential, and resource management [15]. Here are some practical uses of aerial sensing and imaging in plant and crop farming:

Aerial Sensing for Crop and Livestock  393 • Early detection of crop stress: Aerial sensing and imaging can help detect crop stress before it is visible to the naked eye. By analyzing changes in reflectance and vegetation indices, farmers can identify areas of the field that may require attention, such as water stress or nutrient deficiencies. • Precision agriculture: Aerial sensing and imaging can be used to implement precision agriculture practices. Farmers can use these techniques to create variable rate maps for irrigation, fertilization, and pesticide applications. Farmers can optimize inputs and reduce waste using precision agriculture techniques, saving costs and improving crop yields. • Monitoring crop growth and development: Aerial sensing and imaging can provide farmers with regular updates on crop growth and development. By analyzing vegetation indices over time, farmers can identify areas of the field that may lag and take corrective action. • Yield estimation: Aerial sensing and imaging can be used to estimate crop yield potential. Farmers can create accurate yield estimates for each field by combining vegetation indices and ground-truthing data. This information can inform harvest schedules, marketing plans, and input management decisions. • Disease and pest detection: Aerial sensing and imaging can detect diseases and pests in crops. Farmers can identify areas affected by illness or pest infestation by analyzing vegetation indices and reflectance changes. This information can be used to take corrective action and prevent spreading of diseases or pests to other field areas. Overall, the practical uses of aerial sensing and imaging in plant and crop farming are numerous and diverse. Farmers can improve crop yields, reduce input costs, and optimize resource management using these techniques, leading to increased profitability and sustainability.

17.3.3 Soil and Field Analysis Aerial sensing and imaging techniques have significant potential for practical use in soil and field analysis [16]. Here are some examples of how these techniques can be used in practice:

394  Advances in Aerial Sensing and Imaging • Soil moisture monitoring: Aerial sensing and imaging can be used to estimate soil moisture content. By analyzing changes in reflectance over time, farmers can identify areas of the field that may require additional irrigation or drainage. • Soil nutrient mapping: Aerial sensing and imaging can create maps of soil nutrient levels. By analyzing changes in vegetation indices and reflectance, farmers can identify areas of the field that may be nutrient deficient or over-fertilized. This information can inform fertilizer application rates and reduce nutrient runoff. • Crop yield prediction: Aerial sensing and imaging can be used to predict crop yields. By analyzing changes in vegetation indices over time, farmers can estimate crop yield potential and adjust management practices accordingly. • Topographic analysis: Aerial sensing and imaging can be used to create digital elevation models of fields. This information can be used to identify areas of the field that may require additional drainage or leveling. • Plant height mapping: Aerial sensing and imaging can create 3D maps of plant height. Farmers can estimate plant height by analyzing elevation changes and adjusting irrigation and other management practices accordingly. Overall, the practical uses of aerial sensing and imaging in soil and field analysis are diverse and wide-ranging. Farmers can optimize inputs, reduce waste, and increase crop yields by using these techniques, improving profitability and sustainability.

17.3.4 Weed Mapping and Management Aerial sensing and imaging technologies have revolutionized the field of weed mapping and management in recent years. These technologies provide a bird’s eye view of crop fields, allowing for accurate and efficient detection, mapping, and control of weeds. In this response, I will discuss the practical uses of aerial sensing and imaging in weed mapping and management and provide relevant references to support my points. One practical use of aerial sensing and imaging in weed mapping and management is identifying and mapping weed infestations. Aerial images can be captured using drones or other aircraft with high-resolution cameras and other sensors [17]. These images can then be analyzed using advanced image processing

Aerial Sensing for Crop and Livestock  395 and machine learning algorithms to detect and map weed patches, even in large and complex crop fields. This information can help farmers and agronomists to make informed decisions about weed control strategies, such as targeted herbicide application or mechanical weeding. Another practical use of aerial sensing and imaging in weed management is monitoring weed control strategies’ effectiveness over time. Aerial images can be captured at regular intervals, such as weekly or monthly, to track changes in weed density and distribution over time. This can help farmers and agronomists to adjust their weed control strategies as needed and to evaluate the effectiveness of different treatments. In addition to detecting and mapping weeds, aerial sensing and imaging can identify other crop stresses, such as nutrient deficiencies or water stress [18]. By combining weed mapping data with other crop data, farmers and agronomists can develop a more comprehensive understanding of crop health and make more informed decisions about crop management. Several studies demonstrate the effectiveness of aerial sensing and imaging in weed mapping and management. For example, a survey by the University of Nebraska-Lincoln found that aerial imagery analysis could accurately identify and map weed infestations in soybean fields, with an overall accuracy of 87%. Another study by the University of Georgia found that using drone imagery to map weed infestations in cotton fields allowed for more targeted and efficient herbicide applications, resulting in a 48% reduction in herbicide use compared to traditional broadcast applications [19]. In conclusion, aerial sensing and imaging technologies have proven valuable tools in weed mapping and management, providing farmers and agronomists with detailed information about weed infestations and crop health. Farmers can develop more targeted and efficient weed control strategies using these technologies, improving crop yields and reducing herbicide use.

17.3.5 Plantation Crop Aerial sensing and imaging have revolutionized how plantation crops are managed and monitored. Using drones, satellites, and other airborne platforms has provided farmers with a powerful tool to obtain critical information on crop health, growth, and yield potential. In this response, I will discuss some practical uses of aerial sensing and imaging in plantation crops and provide references to support the claims [20]. • Crop health assessment: Aerial sensing and imaging can help farmers assess the health of their crops by identifying

396  Advances in Aerial Sensing and Imaging stress factors, such as water deficiency or nutrient deficiencies before they become visible to the naked eye. This can help farmers take timely corrective actions to prevent yield losses. For example, studies have shown that aerial imaging can detect water stress in grapevines up to 10 days before it becomes visible on the ground. • Yield estimation: Aerial sensing and imaging can also estimate crop yield potential. Farmers can count the number of fruits or nuts their trees or vines will likely produce by measuring plant height, canopy density, and other indicators. This information can help them make better decisions about crop management, such as pruning, fertilization, or irrigation. For example, a study conducted in a commercial apple orchard found that aerial imagery could predict fruit yield with 87% accuracy. • Pest and disease monitoring: Aerial sensing and imaging can identify and monitor pest and disease infestations in plantation crops. Farmers can take targeted measures to control the infestation by identifying areas of the affected products, such as applying pesticides or removing infected plants. For example, a study conducted in a vineyard showed that aerial imaging could detect powdery mildew, a fungal disease, with 91% accuracy. • Irrigation management: Aerial sensing and imaging can help farmers manage irrigation more efficiently by identifying areas of the crop that are under or over-irrigated. By providing farmers with detailed information on soil moisture levels, aerial imaging can help them optimize their irrigation schedules to reduce water use and increase crop yield. For example, a study conducted in a vineyard found that aerial imaging could detect differences in soil moisture levels with 92% accuracy.

17.3.6 Crop and Spot Spraying • Variable rate application: Aerial sensing and imaging can help farmers apply pesticides and other crop protection products at varying rates, depending on the crop’s needs. By identifying areas of the field that are more or less affected by pests or diseases, farmers can apply chemicals more precisely, reducing the number of chemicals used and increasing crop yields. For example, a study conducted on a wheat

Aerial Sensing for Crop and Livestock  397 field found that variable rate application based on aerial imagery resulted in a 45% reduction in chemical use compared to a standard application [21]. • Spot spraying: Aerial sensing and imaging can also be used to apply pesticides and other crop protection products only in areas of the field where they are needed, reducing chemical use and saving costs. By identifying individual plants or sections of the field that are affected by pests or diseases, farmers can target their spraying more effectively. For example, a study conducted on a cotton field found that spot spraying based on aerial imagery resulted in a 56% reduction in chemical use compared to a standard application. • Real-time monitoring: Aerial sensing and imaging can provide real-time Monitoring of crop conditions, allowing farmers to respond quickly to changes in the field. By identifying areas of the field that are affected by pests or diseases, farmers can take immediate corrective actions, such as applying pesticides or removing infected plants. For example, a study conducted on a cornfield found that real-time Monitoring based on aerial imagery resulted in a 25% reduction in pesticide use compared to a standard application [22]. • Safety: Aerial sensing and imaging can also improve safety in crop spraying operations. By providing farmers with detailed information on field conditions, such as wind speed and direction, aerial imaging can help them make better decisions about when and how to apply pesticides, reducing the risk of drift and chemical exposure. For example, a study conducted on a citrus orchard found that aerial imaging could detect wind speeds with an accuracy of 85%, allowing farmers to adjust their spraying accordingly [23]. In conclusion, aerial sensing and imaging have a wide range of practical uses in crop and spot spraying, from variable rate application to real-time Monitoring and safety. By providing farmers with critical information on field conditions and crop health, aerial imaging can help them make better decisions about crop protection and increase their profitability.

17.3.7 Crop Monitoring Aerial sensing and imaging have become increasingly crucial in crop monitoring and management, providing farmers with critical information to

398  Advances in Aerial Sensing and Imaging make informed decisions about their crops. This technology allows for collecting large amounts of data quickly and efficiently, which can help farmers identify problem areas in their fields and optimize their use of resources such as water and fertilizer. One of the main advantages of aerial sensing and imaging is the ability to provide highly accurate, real-time data about crop health and growth. By using drones or aircraft equipped with cameras and other sensors, farmers can obtain detailed information about factors such as plant height, chlorophyll content, and temperature, which can be used to diagnose issues such as nutrient deficiencies or pest infestations. In addition to providing information about crop health, aerial sensing and imaging can generate detailed maps of crop fields, which can help farmers optimize their use of resources such as water and fertilizer. These maps can be generated using specialized software that analyses the data collected by the sensors and produces a detailed picture of the field, including information about soil moisture levels, plant density, and other essential factors [24]. Overall, aerial sensing and imaging in crop monitoring is a rapidly evolving field with many potential benefits for farmers. By providing highly accurate and detailed information about crop health and growth, this technology can help farmers make more informed decisions about their crops, leading to better yields, higher profits, and more sustainable farming practices.

17.4 Aerial Imaging and Sensing Applications in Livestock Farming 17.4.1 Livestock Sensor Aerial sensing and imaging have revolutionized the way livestock farmers manage their operations. With the help of unmanned aerial vehicles (UAVs) or drones, farmers can gather valuable data on their herds and make informed decisions to optimize their productivity and well-being. One practical use of aerial sensing and imaging in livestock management is to monitor the health and behaviour of animals. Using thermal imaging cameras, drones can detect changes in body temperature and identify potential health issues in livestock. Similarly, drones equipped with high-resolution cameras can monitor the movement patterns of animals, identify feeding and grazing areas, and spot any signs of stress or discomfort. Another practical application of aerial sensing and imaging in livestock management is to monitor the condition and quality of pastures and grazing land. By using multispectral imaging sensors, drones can

Aerial Sensing for Crop and Livestock  399 analyze the vegetation index of the land and provide farmers with insights into the quality and quantity of available forage for their animals [25]. This information can be used to adjust grazing patterns, identify areas that need improvement, and optimize pasture management practices. Moreover, drones equipped with sensors can help farmers manage the environment in which their livestock lives. For instance, drones can measure environmental parameters such as temperature, humidity, and air quality, providing farmers with valuable insights into creating optimal living conditions for their animals. Additionally, drones can detect and monitor water sources, ensuring that livestock can always access clean drinking water [26].

17.4.2 Animal Health Aerial sensing and imaging have become increasingly valuable tools for animal health professionals. Using unmanned aerial vehicles (UAVs) or drones equipped with various sensors and imaging technologies, animal health professionals can gather real-time, high-resolution data on animal health and behaviour. Here are some practical uses of aerial sensing and imaging in animal health [27]: • Disease surveillance: Drones equipped with thermal imaging cameras can detect changes in animal body temperature and identify potential health issues, including infectious diseases. This can help animal health professionals quickly detect and contain disease outbreaks, preventing the spread of infection. Detail list of conditions that can be examined using aerial imaging and sensing is mentioned in Table 17.4. • Injury detection: Drones equipped with high-resolution cameras can capture images and videos of animals, allowing animal health professionals to identify injuries, lameness, and other physical health problems. This information can be used to diagnose and treat animals quickly, reducing pain and improving their chances of recovery [29]. • Environmental monitoring: Drones can also be used to monitor animal living conditions and the environment in which they live. For example, drones equipped with air quality sensors can detect airborne pollutants that can negatively impact animal health. Similarly, drones can monitor water quality and ensure that animals can access clean drinking water.

400  Advances in Aerial Sensing and Imaging Table 17.4  List of some of the Zoonoses that can be identifies using UAVs [28]. Major reservoir host

Major transmission route

Mice

Ingestion

S. no. Disease

Pathogen

1.

Angiostrongyliasis

Angiostrongylus cantonensis

2.

Anisakiasis

Anisakis spp. and Marine Ingestion Pseudoterranova mammals spp.

3.

Anthrax

Bacillus anthracis

Herbivores

Contact, inhalation, and ingestion

4.

Brucellosis

Brucella spp.

Herbivores, pigs, and dogs

Contact

5.

Campylobacter enteritis

Campylobacter spp. Poultry and cattle

6.

Clonorchiasis

Ingestion Clonorchis sinensis Humans, cats, dogs, mice, and other animals

7.

Cryptosporidiosis

Cryptosporidium parvum

8.

Diarrhea caused by Enterohemorrhagic Cattle and enterohemorrhagic Escherichia coli humans strains

9.

Ebola-Marburg viral disease

Ebola virus and Marburg virus

Fruit bats

Contact

10.

Hydatid disease

Echinococcus spp.

Canines

Ingestion

11.

Ehrlichiosis

Ehrlichia spp.

Ruminants, rodents, and dogs

Tick bites

Ingestion

Ingestion Humans, cattle, and other livestock Ingestion

(Continued)

Aerial Sensing for Crop and Livestock  401 Table 17.4  List of some of the Zoonoses that can be identifies using UAVs [28]. (Continued)

S. no. Disease

Pathogen

Major reservoir host

12.

Variant CreutzfeldtJakob disease

Prion

Livestock

13.

Giardiasis

Giardia lamblia

Humans, Ingestion beavers, and other animals

14.

Epidemic Hantaviruses hemorrhagic fever

15.

Hepatitis E

Hepatitis E virus

Major transmission route Ingestion

Rodents

Inhalation

Primates

Ingestion

• Tracking and monitoring: Drones equipped with GPS and other tracking technologies can monitor animal movements, identify potential hazards, and track animal behaviour. This can be particularly useful for wildlife monitoring and conservation efforts [30]. The practical uses of aerial sensing and imaging in animal health vary. By leveraging the capabilities of drones, animal health professionals can gather valuable data on animal health and behaviour, helping them to diagnose and treat illnesses, prevent disease outbreaks, and improve animal welfare. Aerial sensing and imaging have significant practical applications in animal health management. By utilizing unmanned aerial vehicles (UAVs) equipped with different sensors, farmers and veterinarians can gather valuable data on the health of animals and make informed decisions to optimize their welfare [31]. One practical use of aerial sensing and imaging in animal health is detecting diseases and injuries. Using thermal imaging cameras, drones can see changes in body temperature and identify potential health issues in animals. For instance, thermal imaging cameras can detect the early signs of foot-and-mouth disease, a highly contagious viral disease affecting cloven-hoofed animals such as cattle and pigs. Similarly, drones with high-resolution cameras can detect injuries such as cuts, bruises, and fractures that may not be easily visible from the ground. Another practical application of aerial sensing and imaging in animal

402  Advances in Aerial Sensing and Imaging health is monitoring animal behaviour [32]. Using multispectral imaging sensors, drones can analyze animal movement patterns, feeding and grazing areas, and signs of stress or discomfort. This information can be used to adjust management practices, identify areas that need improvement, and optimize the welfare of animals. Moreover, drones equipped with sensors can help veterinarians and farmers manage the environment in which animals live. For instance, drones can measure environmental parameters such as temperature, humidity, and air quality, providing insights into creating optimal animal living conditions. Additionally, drones can detect and monitor water sources, ensuring that animals can always access clean drinking water. In conclusion, the practical uses of aerial sensing and imaging in animal health management are vast and varied. By leveraging the capabilities of drones, veterinarians and farmers can gather valuable data on animal health and make informed decisions to optimize animal welfare [33].

17.4.3 Monitoring and Identification of Livestock Farming Aerial sensing and imaging have become valuable tools in Monitoring and identifying livestock farming practices. These technologies provide a bird’s eye view of livestock farming activities, allowing farmers and ranchers to identify potential issues and make informed decisions. Some practical uses of aerial sensing and imaging in livestock farming are included in Figure 17.5. • Monitoring Livestock Health: Aerial sensing and imaging can be used to monitor the health of livestock by identifying changes in their behaviour, movements, and physical Hormones Behaviour Methane

Animal Monitoring

Core Body temperature Respiratory rate

Heart rate

Surface Body temperature

Figure 17.5  Various animal surveillance factors addressed by Aerial [34].

Aerial Sensing for Crop and Livestock  403

•

•

•

•

appearance. This allows farmers to detect potential health issues early on, reducing the spread of diseases and minimizing economic losses [35]. Tracking Livestock Movement: By using aerial sensing and imaging, farmers can track the movement of their livestock, which can help in identifying grazing patterns, detecting potential predators or other threats to the animals, and optimizing grazing management practices. Detecting Environmental Changes: Aerial sensing and imaging can also help in identifying changes in the environment that can impact livestock health and productivity, such as soil moisture, vegetation growth, and water availability. Assessing Infrastructure and Facilities: Aerial sensing and imaging can give farmers an overview of their farm infrastructure and facilities, including fences, buildings, and water systems. This can help identify areas that need repairs or improvements, improving the overall efficiency of the farming operation [36]. Precision Livestock Farming: Precision livestock farming is an emerging field that uses technology to optimize animal management and welfare. Aerial sensing and imaging can be used in precision livestock farming to collect data on animal behaviour, health, and interest, which can be used to make informed decisions about animal care and management.

17.4.4 Geo Fencing and Virtual Perimeters Agriculture and livestock farming are two industries that have greatly benefited from using aerial sensing and imaging technology for geo-fencing and virtual perimeters as shown in Figure 17.6. Here are some practical use cases and references for this technology in these industries [37]: • Crop Monitoring: Aerial sensing and imaging can monitor crops and identify areas that need irrigation or fertilization. This technology can also be used to create accurate geo-fences around different crop fields, enabling farmers to monitor the health of their crops more efficiently [38]. • Livestock Monitoring: Aerial sensing and imaging can also monitor livestock, including cattle, sheep, and goats. This technology can be used to create virtual perimeters around livestock enclosures, enabling farmers to keep track of the

404  Advances in Aerial Sensing and Imaging Voilpally

Bodakonda at Penikarla Thanda

Gattuppal

Exit Shivannaguda

Nallavelly Manthangoud 19

Mall

Ajilapur

Entry

Figure 17.6  Geofencing system depicted graphically to track people entering and leaving the green zone.

whereabouts of their animals and identify any potential health issues [39]. • Irrigation Management: Aerial sensing and imaging technology can create detailed maps of soil moisture levels, enabling farmers to optimize their irrigation practices. This technology can also create virtual perimeters around different irrigation zones, allowing farmers to control water usage more efficiently [40].

17.5 Challenges in Aerial Sensing and Imaging in Agriculture and Livestock Farming Aerial sensing and imaging have become increasingly popular in agriculture and livestock farming because they provide real-time data on crop health, yield estimation, and livestock monitoring. However, there are still several challenges that must be addressed to realize the potential of this technology [41] entirely. One major challenge is the need for high-­quality imaging and sensing equipment. Aerial cameras and sensors must be able to capture high-resolution images and data from a significant altitude to provide an accurate and comprehensive view of the field. Additionally, equipment must be durable enough to withstand harsh environmental conditions, such as extreme temperatures, high winds, and rain. Another

Aerial Sensing for Crop and Livestock  405 challenge is the need for accurate data processing and analysis. The vast amount of data collected from aerial imaging and sensing can be overwhelming, and processing this data promptly and accurately is essential for making informed decisions. Therefore, algorithms must be developed to analyze and interpret data effectively, and farmers must be trained in data analysis and interpretation to use this technology[42] effectively. A third challenge is a cost associated with aerial imaging and sensing technology. While equipment prices have decreased over time, it is still a significant investment for farmers. Additionally, the cost of data processing and analysis can also be substantial, and the benefits of this technology must be carefully weighed against its cost [43].

17.5.1 Technical Limitations of Aerial Sensing and Imaging in Agriculture and Livestock Farming Aerial sensing and imaging technologies have revolutionized agriculture and livestock farming by providing valuable insights into crop health, soil moisture, plant density, and animal behaviour. However, there are some technical limitations to these technologies, including: • Weather conditions: Aerial sensing and imaging technologies rely on clear weather conditions for optimal results. Cloudy or rainy days can reduce the quality of images and data obtained, making it challenging to analyze and interpret [44]. • Flight time and coverage: Aerial sensing and imaging systems have a limited flight time and coverage area, which can be a constraint for large farms or areas that require frequent Monitoring [45]. • Resolution and accuracy: The resolution and accuracy of aerial images depend on the altitude and the equipment quality. High-altitude flights may provide broad coverage but at the cost of lower resolution and accuracy. On the other hand, low-altitude flights may provide high-resolution images but cover a smaller area [46]. • Data processing and analysis: Aerial sensing and imaging generate large amounts of data that require significant processing and analysis. This can challenge smaller farms or those who need more technical expertise or computing resources [47].

406  Advances in Aerial Sensing and Imaging • Cost: Aerial sensing and imaging technologies can be expensive, making it difficult for smaller farms or growers to afford them. The cost of equipment, maintenance, and data processing can add up quickly, making it necessary to weigh the benefits against the cost.

17.6 Conclusion In conclusion, aerial sensing and imaging technologies have advanced significantly in recent years, but there is still room for improvement. Here are some suggestions for areas where further development could be beneficial: • Longer flight times and increased endurance: The flight time and persistence of aerial imaging platforms are limited by the battery life of the drones or aircraft. Developing more efficient power sources or using alternative energy sources such as solar panels could increase the endurance of these platforms, allowing for longer flights and more extensive coverage [48]. • Real-time data processing: Aerial imaging data is often processed after the flight, resulting in delays and missed opportunities for timely decision-making. Developing real-time data processing capabilities would enable faster data analysis and decision-making, making aerial imaging more valuable in time-sensitive applications such as disaster response [49]. • Automation and autonomy: Aerial imaging platforms are becoming increasingly autonomous, with features like obstacle avoidance and autonomous flight modes. Further development in this area could enable more advanced autonomous capabilities, such as automated flight planning and data collection, reducing the need for human intervention and making aerial imaging more accessible and efficient.

References 1. Tsouros, D.C., Bibi, S., Sarigiannidis, P.G., A review on UAV-based applications for precision agriculture. Information, 10, 11, Art. no. 11, Nov. 2019.

Aerial Sensing for Crop and Livestock  407 2. Neupane, K. and Baysal-Gurel, F., Automatic identification and monitoring of plant diseases using unmanned aerial vehicles: A review. Remote Sens., 13, 19, Art. no. 19, Jan. 2021. 3. Aquilani, C., Confessore, A., Bozzi, R., Sirtori, F., Pugliese, C., Review: Precision livestock farming technologies in pasture-based livestock systems. Animal, 16, 1, 100429, Jan. 2022. 4. Zhang, C. and Kovacs, J.M., The application of small unmanned aerial systems for precision agriculture: A review in: Precision Agriculture, vol. 13, pp. 693–712, Springer Netherlands, July 2012. 5. Armenta-Medina, D., Ramirez-delReal, T.A., Villanueva-Vásquez, D., MejiaAguirre, C., Trends on advanced information and communication technologies for improving agricultural productivities: A bibliometric analysis. Agronomy, 10, 12, Art. no. 12, Dec. 2020. 6. Manfreda, S. et al., On the use of unmanned aerial systems for environmental monitoring. Remote Sens, 10, 4, Art. no. 4, Apr. 2018. 7. Aber, J.S., Marzolff, I., Ries, J., Small-format aerial photography: Principles, techniques and geoscience applications, Elsevier, Amsterdam, London, 2010. 8. Li, P., Chen, P., Zhang, D., Cross-modal feature representation learning and label graph mining in a residual multi-attentional CNN-LSTM network for multi-label aerial scene classification. Remote Sens., 14, 10, Art. no. 10, Jan. 2022. 9. Matsushita, B., Yang, W., Chen, J., Onda, Y., Qiu, G., Sensitivity of the enhanced vegetation index (EVI) and normalized difference vegetation index (NDVI) to topographic effects: A case study in high-density cypress forest. Sensors, 7, 11, Art. no. 11, Nov. 2007. 10. Mazzia, V., Comba, L., Khaliq, A., Chiaberge, M., Gay, P., UAV and machine learning based refinement of a satellite-driven vegetation index for precision agriculture. Sensors, 20, 9, Art. no. 9, Jan. 2020. 11. Doraiswamy, P.C., Moulin, S., Cook, P.W., Stern, A., Crop yield assessment from remote sensing. Photogramm. Eng. Remote Sens., 69, 6, 665–674, Jun. 2003. 12. Raddy, G., Lalitha, B., Jayadeva, H.M., Spatial fertilizer recommendation mapping based on soil test crop response equations for important crops using GIS and GPS. Commun. Soil Sci. Plant Anal., 52, 1–18, Dec. 2020. 13. Konduri, V.S., Vandal, T.J., Ganguly, S., Ganguly, A.R., Data science for weather impacts on crop yield. Front. Sustain. Food Syst., 4, 52, 2020. Accessed: Mar. 21, 2023. [Online]. Available: https://www.frontiersin.org/ articles/10.3389/fsufs.2020.00052. 14. Agrahari, R.K., Kobayashi, Y., Tanaka, T.S.T., Panda, S.K., Koyama, H., Smart fertilizer management: The progress of imaging technologies and possible implementation of plant biomarkers in agriculture. Soil Sci. Plant Nutr., 67, 3, 248–258, May 2021. 15. Sishodia, R.P., Ray, R.L., Singh, S.K., Applications of remote sensing in precision agriculture: A review. Remote Sens., 12, 19, Art. no. 19, Jan. 2020.

408  Advances in Aerial Sensing and Imaging 16. Hassan-Esfahani, L., Torres-Rua, A., Jensen, A., McKee, M., Assessment of surface soil moisture using high-resolution multispectral imagery and artificial neural networks. Remote Sens., 7, 3, Art. no. 3, Mar. 2015. 17. Esposito, M., Crimaldi, M., Cirillo, V., Sarghini, F., Maggio, A., Drone and sensor technology for sustainable weed management: A review. Chem. Biol. Technol. Agric., 8, 1, 18, Mar. 2021. 18. Sa, I. et al., WeedMap: A large-scale semantic weed mapping framework using aerial multispectral imaging and deep neural network for precision farming. Remote Sens, 10, 9, Art. no. 9, Sep. 2018. 19. Gibson, K., Dirks, R., Medlin, C., Johnston, L., Detection of weed species in soybean using multispectral digital images1. Weed Technol., 18, 742–749, Sep. 2009. 20. Talaviya, T., Shah, D., Patel, N., Yagnik, H., Shah, M., Implementation of artificial intelligence in agriculture for optimization of irrigation and application of pesticides and herbicides. Artif. Intell. Agric., 4, 58–73, Jan. 2020. 21. Ahmad, F., Khaliq, A., Qiu, B., Sultan, M. and Ma, J., Advancements of spraying technology in agriculture, IntechOpen, London UK, p.33, 2021 https:// www.intechopen.com/chapters/77112 (accessed Apr. 05, 2023). 22. Triantafyllou, A., Sarigiannidis, P., Bibi, S., Precision agriculture: A remote sensing monitoring system architecture. Information, 10, 11, Art. no. 11, Nov. 2019. 23. Dutta, S., Singh, A.K., Mondal, B.P., Paul, D., Patra, K., Digital inclusion of the farming sector using drone technology, IntechOpen, IntechOpen, London, UK, 2023, https://www.intechopen.com/online-first/84968 (accessed Apr. 05, 2023). 24. Yang, G. et al., Unmanned aerial vehicle remote sensing for field-based crop phenotyping: Current status and perspectives. Front. Plant Sci., 8, 1111, 2017. Accessed: Apr. 05, 2023. [Online]. Available: https://www.frontiersin.org/ articles/10.3389/fpls.2017.01111. 25. Oishi, Y., Oguma, H., Tamura, A., Nakamura, R., Matsunaga, T., Animal detection using thermal images and its required observation conditions. Remote Sens., 10, 7, Art. no. 7, Jul. 2018. 26. Symons, A., Optimizing observing strategies for monitoring animals using drone-mounted thermal infrared cameras. Int. J. Remote Sens., 40, 439–467, Dec. 2018. 27. Alanezi, M., Shahriar, M.S., Hasan, B., Ahmed, S., Shaaban, Y., Bouchekara, H., Livestock management with unmanned aerial vehicles: A review. IEEE Access, 10, 1–1, Jan. 2022. 28. Yu, Q., Liu, H., Xiao, N., Unmanned aerial vehicles: Potential tools for use in zoonosis control. Infect. Dis. Poverty, 7, 1, 49, Jun. 2018. 29. Karp, D., University of Zurich, Department of Evolutionary Biology and Environmental Studies, Winterthurerstrasse, Detecting small and cryptic animals by combining thermography and a wildlife detection dog, PMC, Scientific

Aerial Sensing for Crop and Livestock  409 reports, UK, 190, 8057 Zurich, Switzerland, 2020, https://www.ncbi.nlm.nih. gov/pmc/articles/PMC7090052/ (accessed Apr. 05, 2023). 30. Hyun, C.U., Park, M., Lee, W.Y., Remotely piloted aircraft system (RPAS)based wildlife detection: A review and case studies in maritime Antarctica. Animals, Basel, Switzerland, 10, 12, 2387, 2020. 31. Carrasco-Escobar, G., Moreno, M., Fornace, K., Herrera-Varela, M., Manrique, E., Conn, J.E., The use of drones for mosquito surveillance and control. Parasitol. Vectors, 15, 1, 473, Dec. 2022. 32. Christiansen, P., Steen, K.A., Jørgensen, R.N., Karstoft, H., Automated detection and recognition of wildlife using thermal cameras. Sensors, 14, 8, Art. no. 8, Aug. 2014. 33. Neethirajan, S., Recent advances in wearable sensors for animal health management. Sens. Bio-Sens. Res., 12, 15–29, Feb. 2017. 34. Fay, T. and Ku, L., Livestock farming technology in animal agriculture, Plug and Play Tech Center, https://www.plugandplaytechcenter.com/resources/ livestock-farming-technology-animal-agriculture/ (accessed Apr. 05, 2023). 35. Herlin, A., Brunberg, E., Hultgren, J., Högberg, N., Rydberg, A., Skarin, A., Animal welfare implications of digital tools for monitoring and management of cattle and sheep on pasture. Anim. Open Access J. MDPI, 11, 3, 829, Mar. 2021. 36. Yadav, J., Chauhan, U., Sharma, D., Importance of drone technology in Indian agriculture, farming, in: Smart Village Infrastructure and Sustainable Rural Communities, pp. 35-46, IGI Global, Pennsylvania, 2023. 37. Ilyas, Q. and Ahmad, M., Smart farming: An enhanced pursuit of sustainable remote livestock tracking and geofencing using IoT and GPRS. Wirel. Commun. Mob. Comput., 2020, 1–12, Dec. 2020. 38. Abbasi, R., Martinez, P., Ahmad, R., The digitization of agricultural industry – A systematic literature review on agriculture 4.0. Smart Agric. Technol., 2, 100042, Dec. 2022. 39. Wilson, A.D., Diverse applications of electronic-nose technologies in agriculture and forestry. Sensors, 13, 2, 2295–2348, Feb. 2013. 40. Said Mohamed, E., Belal, A.A., Kotb Abd-Elmabod, S., El-Shirbeny, M.A., Gad, A., Zahran, M.B., Smart farming for improving agricultural management. Egypt. J. Remote Sens. Space Sci., 24, 3, 971–981, Dec. 2021. Part 2. 41. Wu, B., Zhang, M., Zeng, H., Tian, F., Potgieter, A.B., Qin, X., Yan, N., Chang, S., Zhao, Y., Dong, Q., Boken, V., Challenges and opportunities in remote sensing-based crop monitoring: A review. Natl. Sci. Rev. Oxford Academic, 10, 4, p.nwac290, 2023, https://academic.oup.com/nsr/­article/10/4/nwac290/ 6939854 (accessed Apr. 05, 2023). 42. Aasen, H., Honkavaara, E., Lucieer, A., Zarco-Tejada, P.J., Quantitative remote sensing at ultra-high resolution with UAV spectroscopy: A review of sensor technology, measurement procedures, and data correction workflows. Remote Sens., 10, 7, Art. no. 7, Jul. 2018.

410  Advances in Aerial Sensing and Imaging 43. Javaid, M., Haleem, A., Khan, I.H., Suman, R., Understanding the potential applications of artificial intelligence in agriculture sector. Adv. Agrochem, 2, 1, 15–30, Mar. 2023. 44. Delavarpour, N., Koparan, C., Nowatzki, J., Bajwa, S., Sun, X., A technical study on UAV characteristics for precision agriculture applications and associated practical challenges. Remote Sens., 13, 6, Art. no. 6, Jan. 2021. 45. Thies, B. and Bendix, J., Satellite based remote sensing of weather and climate: Recent achievements and future perspectives. Meteorol. Appl., 18, 3, 262–295, 2011. 46. Borra-Serrano, I., Peña, J.M., Torres-Sánchez, J., Mesas-Carrascosa, F.J., López-Granados, F., Spatial quality evaluation of resampled unmanned aerial vehicle-imagery for weed mapping. Sensors, 15, 8, Art. no. 8, Aug. 2015. 47. Chandra, R., Collis, S., Digital agriculture for small-scale producers: Challenges and opportunities, Communications of the ACM, New York, 64, 12, pp.75– 84, 2021. 48. Xie, Y., Sha, Z., Yu, M., Remote sensing imagery in vegetation mapping: A review. J. Plant Eco., 1, 1, 9–23, 2008. 49. Subeesh, A. and Mehta, C.R., Automation and digitization of agriculture using artificial intelligence and internet of things. Artif. Intell. Agric., 5, 278– 291, Jan. 2021.

Index 2D convolution layer, 222 AAV LiDAR data, 327 Activation layer, 222 Advanced protection, 165 Advantages and disadvantages of graphical passwords, 305 Aerial images, 47, 53, 60, 61 Aerial imaging and sensing applications in agriculture, assessing yield and fertilizer response, 390–392 crop and spot spraying, 396–397 crop monitoring, 397–398 plant and crop farming, 392–393 plantation crop, 395–396 soil and field analysis, 393–394 weed mapping and management, 394–395 Aerial imaging and sensing applications in livestock farming, animal health, 399–402 geo-fencing and virtual perimeters, 403–404 livestock sensor, 398–399 monitoring and identification, 402–403 Aerial mustering, 102–103 Aerial remote sensing (RS), 96 AIDSOUT, 170 Analytics, learning module, 272–273 Animal health monitoring, 102

Annoyed aerial vehicles, 321 Applications of aerial imaging in various domains, 21 ArcGIS, 82 Architecture of the YOLOv7, 119 Artificial intelligence, 110 Association for unmanned vehicle systems international (AUVSI), 105 Authentication, 295–297, 299–300 Avaddon, 183 Binary cross-entropy loss, 256 Challenges of object detection and classification in aerial images, 17 Classification, 117, 123, 127, 129, 130, 132, 134 CLEARAID, 170 CNN, 118–120 Common forged feature network, 252 Comparative analysis of the aerial imaging algorithms and architectures, 376–379 Confusion matrix, 345, 347, 349–350 Contrastive learning, 255 Convolutional neural networks, (CNN), 73–74, 101, 214 Correlation, 134–138 CryptoLocker virus, 167 Crypto-malware, 166–167, 170, 173

411

412  Index Cutting-edge sensor technology, 96–97 Cyber attacks, 165–166, 175, 178, 186, 199–200, 206 Darkside, 183, 201, 204 Data acquisition, 76–78 Data analysis, 307–308 Data collection, 270 Data collection and processing, 273–276 Data integration, 79–80 Data provider, 275–276 DatasetVISdrone 2019, 121 Demographic, 118, 133 Detection, 49, 51, 52, 54–63, 117–119, 121, 122, 124, 132–134 Decision tree, 82 DJI Ferntech, 103 Doxware, 167–168, 181 Drones, 215, 299–300, 307–308 Dropouts, 222 Ductal carcinoma, 119 Early detection, 129, 130, 132, 133, 142, 143 Eavesdropping, 187, 189, 192 ECDH, 174 Edge computing and aerial imaging, edge architectures, 369–376 research perspective in aerial imaging, 367–369 Egregor, 183, 203 Emotion mining from aerial drone, 225 Encrypt system, 168 Environment, 129–131 EternalBlue, 175, 177, 203 Experimental methods and techniques, 384–388 aerial photography, 388 unmanned aerial vehicle, 388–390 Experimental settings, 257 Facial emotion recognition algorithms, 224

FANETS, 186–188, 192 Farming UAVs, 96, 98 Features classification and classification network, 78–79, 253 Features extraction, 78–79, 253 FER 2013 dataset, 224 Fixed sensors, 268–272 Fixed wing drone, 90 Flask, 274–280 Focus groups, 78 Forests, 133, 142, 143 Gameover Zeus, 171 Generative models, 117 Genetic, 117, 124 Geographical information systems, 314 Georeferencing, 77, 78 Geospatial data, 270, 275, 276 GIS software, 81 Global navigation satellite systems, 314 Gradient boosting classifier, 82 Graphical password authentication, 297–298, 301–305, 309–310 High accuracy, 129, 131 Hive, 183, 205 HTML/CSS/JavaScript, architecture, 275–276 Hybrid VTOL, 94, 95 Image recognition, 47 Image rectification, 78 Information, 49, 51, 58–61 IoT integration, 271 Label prediction, 255 LBP visualized results, 259 Leakware, 167–168 Learning module, 275–276 tools and usage, 274–275 LeNet-5, 335

Index  413 Livestock production systems (LPS), 101 Locker-ransomware, 167 Locky, 174–175, 203 Machine learning, 110, 271 Machine learning algorithms, 268–270, 272–274 Malignant, 117, 119, 124 Manual observation, 268–271 Materials and methods, 37–40 dataset, 38 model implemented, 38–40 Mean-squared error (MSE), 82 Medical, 118, 119, 133 Minecraft, 173 MitM, 187, 189 Mobile application, 271 Model development, 80 Monitoring, 129–133, 142, 143 Multinominal logistic regression, 82 Multi-rotor drone, 91, 93–94 Multi-source data fusion, 79–90 Mysql, 274–279 Normalized difference vegetation index (NDVI), 74, 79 NotPetya, 171, 176–178, 203 Object-based image analysis (OBIA), 73–74 Object monitoring, 111 Object tracking techniques, 116 Objects, 56–63 ODCNN, 335 Opencv, 274–278 Parking availability, 267–289 Parking space detection, AI-based framework, 267–289 data analysis, 267–289 intelligent data analysis, 267–289 traffic management, 267–289

Parking space identification, 273–277 Password authentication, 295–300 Password cracking, password cracking process, 306 password cracking techniques, 306 password cracking tools, 307 Patterns, 117, 118 Performance comparison, 258 Personalization, 271 Pixel-based classification, 88 Policy and program development, 82 Pollution, 267–289 Poverty estimation using aerial imaging, introduction, 70–73 main techniques, 73–76 proposed methodology data acquisition, 76–78 data integration, 79–80 feature extraction, 78–79 model development, 80 policy and program development, 82 pre-processing, 78 result and discussion, 82–83 validation, 80–81 visualization and analysis, 81–82 Poverty indicators, 72 Precision, 335, 337, 344, 346, 351 Precision farming, 88 Pre-diagnosis, 133 Python, 274–276 Q-learning, 276–278 QGIS, 82 Quickly identify, 129, 131 Real-time data, 268–273 proposed framework OPSI, 270–272 system design, 273–274 Regression, 118, 125, 136–141 Remote sensing, 88–89, 96–97, 100

414  Index Remote sensing technology, 314 Rescue, 47, 50, 54, 61, 63, 64 Research applications of aerial imaging, emergency response, 364–366 environment monitoring, 360–363 precision agriculture, 359–360 urban planning and development, 363–364 vehicle imaging, 358–359 ResNet, 335 RFID tags, 101 R-squared values, 82 Road, 47–52, 54, 55, 57, 58 ROI, 54, 55, 62 Rotary-wing AAVs, 322 Route optimization, 276–278 accuracy detection of fatigue, 282–283 application and results, 284–286 discussion and limitations, 286–287 existing methods, 278–279 hyperparameter for OPSI, 279–280 potential impact, 280–282 similar study and results analysis, 283–284 RSI-CB256, 337 Ryuk, 178–180, 201, 203 Scareware, 167–168, 199 Sensing payloads, 325 Sensor network, 273–275 SHAP analysis, 82 Single rotor drone, 91 Small area estimation (SAE) model, 80 SMOreg, 129, 136–141 Soil and field analysis, 96–97 Spectral indices, 73 Spoofing, 187, 189–190 Support vector machine, 333, 335

Tableau, 82 TensorFlow, 274–277, 333, 339 TeslaCrypt, 173–174, 203 Therapy, 117, 118 Thermal, 130, 133 Time-series analysis, 81 TorrentLocker, 172 Traffic accidents, 47, 48, 50–52, 63 Traffic congestion, 267–289 Traffic management, 48, 55, 64 Training convergence, 262 Translational aircraft (TS), 94 Tumor, 118, 134 Types of graphical password authentication, cued recall, 304–305 recall-based authentication, 302–303 recognition-based authentication, 304 Types of password authentication systems, biometric authentication, 301 knowledge-based authentication, 301 token-based authentication, 301 UAV, 132, 133, 143 U-Net, 336 Unmanned aerial vehicles, 109, 165, 185, 186, 193, 194, 206 Unmanned aerial vehicles (UAVs) in agriculture, agricultural use, 96–97, 99–101 applications in various industries, 89 challenges, 104 classification, 90–91, 93–94 comparison of various, 94–95 introduction, 88–90 in livestock farming, 101–103 aerial mustering, 102–103

Index  415 animal health monitoring, 101–102 headcount of cattle, 101 tracking and identification of livestock, 102 Urban mobility, 268–269 current methods, 268–270 User interface, implementation techniques, 276–278

Vehicles, 48–52, 54–60 Visualization, 271 framework components, 270–271 WannaCry, 166, 171, 175–177, 199, 203 Wildfires, 129–131, 133 Wireless, 131, 132, 142 Yolo, 276–278

WILEY END USER LICENSE AGREEMENT Go to www.wiley.com/go/eula to access Wiley’s ebook EULA.